* [RFC PATCH v1 0/5] test case blocking and logging
@ 2023-12-20 10:33 Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 1/5] dts: convert dts.py methods to class Juraj Linkeš
` (8 more replies)
0 siblings, 9 replies; 44+ messages in thread
From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
yoan.picchi, Luca.Vizzarro
Cc: dev, Juraj Linkeš
We currently don't store test cases that couldn't be executed because of
a previous failure, such as when a test suite setup failed, resulting in
no executed test cases.
In order to record the test cases that couldn't be executed, we must
know the lists of test suites and test cases ahead of the actual test
suite execution, as an error could occur before we even start executing
test suites.
In addition, the patch series contains two refactors.
The first refactor is closely related. The dts.py was renamed to
runner.py and given a clear purpose - running the test suites and all
other orchestration needed to run test suites. The logic for this was
not all in the original dts.py module and it was brought there. The
runner is also responsible for recording results, which is the blocked
test cases are recorded.
The other refactor, logging, is related to the first refactor. The
logging module was simplified while extending capabilities. Each test
suite logs into its own log file in addition to the main log file which
the runner must handle (as it knows when we start executing particular
test suites). The runner also handles the switching between execution
stages for the purposes of logging.
Juraj Linkeš (5):
dts: convert dts.py methods to class
dts: move test suite execution logic to DTSRunner
dts: process test suites at the beginning of run
dts: block all testcases when earlier setup fails
dts: refactor logging configuration
dts/framework/config/__init__.py | 8 +-
dts/framework/dts.py | 228 --------
dts/framework/logger.py | 162 +++---
dts/framework/remote_session/__init__.py | 4 +-
dts/framework/remote_session/os_session.py | 6 +-
.../remote_session/remote/__init__.py | 7 +-
.../remote/interactive_remote_session.py | 7 +-
.../remote/interactive_shell.py | 7 +-
.../remote_session/remote/remote_session.py | 8 +-
.../remote_session/remote/ssh_session.py | 5 +-
dts/framework/runner.py | 499 ++++++++++++++++++
dts/framework/test_result.py | 365 +++++++------
dts/framework/test_suite.py | 193 +------
dts/framework/testbed_model/node.py | 10 +-
dts/framework/testbed_model/scapy.py | 6 +-
.../testbed_model/traffic_generator.py | 5 +-
dts/main.py | 12 +-
dts/tests/TestSuite_smoke_tests.py | 2 +-
18 files changed, 830 insertions(+), 704 deletions(-)
delete mode 100644 dts/framework/dts.py
create mode 100644 dts/framework/runner.py
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [RFC PATCH v1 1/5] dts: convert dts.py methods to class
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
@ 2023-12-20 10:33 ` Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 2/5] dts: move test suite execution logic to DTSRunner Juraj Linkeš
` (7 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
yoan.picchi, Luca.Vizzarro
Cc: dev, Juraj Linkeš
The dts.py module deviates from the rest of the code without a clear
reason. Converting it into a class and using better naming will improve
organization and code readability.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/config/__init__.py | 3 -
dts/framework/dts.py | 228 -----------------------------
dts/framework/runner.py | 243 +++++++++++++++++++++++++++++++
dts/main.py | 6 +-
4 files changed, 247 insertions(+), 233 deletions(-)
delete mode 100644 dts/framework/dts.py
create mode 100644 dts/framework/runner.py
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 9b32cf0532..497847afb9 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -314,6 +314,3 @@ def load_config() -> Configuration:
config: dict[str, Any] = warlock.model_factory(schema, name="_Config")(config_data)
config_obj: Configuration = Configuration.from_dict(dict(config))
return config_obj
-
-
-CONFIGURATION = load_config()
diff --git a/dts/framework/dts.py b/dts/framework/dts.py
deleted file mode 100644
index 25d6942d81..0000000000
--- a/dts/framework/dts.py
+++ /dev/null
@@ -1,228 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2019 Intel Corporation
-# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
-# Copyright(c) 2022-2023 University of New Hampshire
-
-import sys
-
-from .config import (
- CONFIGURATION,
- BuildTargetConfiguration,
- ExecutionConfiguration,
- TestSuiteConfig,
-)
-from .exception import BlockingTestSuiteError
-from .logger import DTSLOG, getLogger
-from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
-from .test_suite import get_test_suites
-from .testbed_model import SutNode, TGNode
-from .utils import check_dts_python_version
-
-dts_logger: DTSLOG = getLogger("DTSRunner")
-result: DTSResult = DTSResult(dts_logger)
-
-
-def run_all() -> None:
- """
- The main process of DTS. Runs all build targets in all executions from the main
- config file.
- """
- global dts_logger
- global result
-
- # check the python version of the server that run dts
- check_dts_python_version()
-
- sut_nodes: dict[str, SutNode] = {}
- tg_nodes: dict[str, TGNode] = {}
- try:
- # for all Execution sections
- for execution in CONFIGURATION.executions:
- sut_node = sut_nodes.get(execution.system_under_test_node.name)
- tg_node = tg_nodes.get(execution.traffic_generator_node.name)
-
- try:
- if not sut_node:
- sut_node = SutNode(execution.system_under_test_node)
- sut_nodes[sut_node.name] = sut_node
- if not tg_node:
- tg_node = TGNode(execution.traffic_generator_node)
- tg_nodes[tg_node.name] = tg_node
- result.update_setup(Result.PASS)
- except Exception as e:
- failed_node = execution.system_under_test_node.name
- if sut_node:
- failed_node = execution.traffic_generator_node.name
- dts_logger.exception(f"Creation of node {failed_node} failed.")
- result.update_setup(Result.FAIL, e)
-
- else:
- _run_execution(sut_node, tg_node, execution, result)
-
- except Exception as e:
- dts_logger.exception("An unexpected error has occurred.")
- result.add_error(e)
- raise
-
- finally:
- try:
- for node in (sut_nodes | tg_nodes).values():
- node.close()
- result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Final cleanup of nodes failed.")
- result.update_teardown(Result.ERROR, e)
-
- # we need to put the sys.exit call outside the finally clause to make sure
- # that unexpected exceptions will propagate
- # in that case, the error that should be reported is the uncaught exception as
- # that is a severe error originating from the framework
- # at that point, we'll only have partial results which could be impacted by the
- # error causing the uncaught exception, making them uninterpretable
- _exit_dts()
-
-
-def _run_execution(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- result: DTSResult,
-) -> None:
- """
- Run the given execution. This involves running the execution setup as well as
- running all build targets in the given execution.
- """
- dts_logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
- execution_result = result.add_execution(sut_node.config)
- execution_result.add_sut_info(sut_node.node_info)
-
- try:
- sut_node.set_up_execution(execution)
- execution_result.update_setup(Result.PASS)
- except Exception as e:
- dts_logger.exception("Execution setup failed.")
- execution_result.update_setup(Result.FAIL, e)
-
- else:
- for build_target in execution.build_targets:
- _run_build_target(sut_node, tg_node, build_target, execution, execution_result)
-
- finally:
- try:
- sut_node.tear_down_execution()
- execution_result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Execution teardown failed.")
- execution_result.update_teardown(Result.FAIL, e)
-
-
-def _run_build_target(
- sut_node: SutNode,
- tg_node: TGNode,
- build_target: BuildTargetConfiguration,
- execution: ExecutionConfiguration,
- execution_result: ExecutionResult,
-) -> None:
- """
- Run the given build target.
- """
- dts_logger.info(f"Running build target '{build_target.name}'.")
- build_target_result = execution_result.add_build_target(build_target)
-
- try:
- sut_node.set_up_build_target(build_target)
- result.dpdk_version = sut_node.dpdk_version
- build_target_result.add_build_target_info(sut_node.get_build_target_info())
- build_target_result.update_setup(Result.PASS)
- except Exception as e:
- dts_logger.exception("Build target setup failed.")
- build_target_result.update_setup(Result.FAIL, e)
-
- else:
- _run_all_suites(sut_node, tg_node, execution, build_target_result)
-
- finally:
- try:
- sut_node.tear_down_build_target()
- build_target_result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Build target teardown failed.")
- build_target_result.update_teardown(Result.FAIL, e)
-
-
-def _run_all_suites(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
-) -> None:
- """
- Use the given build_target to run execution's test suites
- with possibly only a subset of test cases.
- If no subset is specified, run all test cases.
- """
- end_build_target = False
- if not execution.skip_smoke_tests:
- execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
- for test_suite_config in execution.test_suites:
- try:
- _run_single_suite(sut_node, tg_node, execution, build_target_result, test_suite_config)
- except BlockingTestSuiteError as e:
- dts_logger.exception(
- f"An error occurred within {test_suite_config.test_suite}. Skipping build target."
- )
- result.add_error(e)
- end_build_target = True
- # if a blocking test failed and we need to bail out of suite executions
- if end_build_target:
- break
-
-
-def _run_single_suite(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
- test_suite_config: TestSuiteConfig,
-) -> None:
- """Runs a single test suite.
-
- Args:
- sut_node: Node to run tests on.
- execution: Execution the test case belongs to.
- build_target_result: Build target configuration test case is run on
- test_suite_config: Test suite configuration
-
- Raises:
- BlockingTestSuiteError: If a test suite that was marked as blocking fails.
- """
- try:
- full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
- test_suite_classes = get_test_suites(full_suite_path)
- suites_str = ", ".join((x.__name__ for x in test_suite_classes))
- dts_logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
- except Exception as e:
- dts_logger.exception("An error occurred when searching for test suites.")
- result.update_setup(Result.ERROR, e)
-
- else:
- for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(
- sut_node,
- tg_node,
- test_suite_config.test_cases,
- execution.func,
- build_target_result,
- )
- test_suite.run()
-
-
-def _exit_dts() -> None:
- """
- Process all errors and exit with the proper exit code.
- """
- result.process()
-
- if dts_logger:
- dts_logger.info("DTS execution has ended.")
- sys.exit(result.get_return_code())
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
new file mode 100644
index 0000000000..5b077c5805
--- /dev/null
+++ b/dts/framework/runner.py
@@ -0,0 +1,243 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2019 Intel Corporation
+# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
+# Copyright(c) 2022-2023 University of New Hampshire
+
+import logging
+import sys
+
+from .config import (
+ BuildTargetConfiguration,
+ Configuration,
+ ExecutionConfiguration,
+ TestSuiteConfig,
+)
+from .exception import BlockingTestSuiteError
+from .logger import DTSLOG, getLogger
+from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
+from .test_suite import get_test_suites
+from .testbed_model import SutNode, TGNode
+from .utils import check_dts_python_version
+
+
+class DTSRunner:
+ _logger: DTSLOG
+ _result: DTSResult
+ _configuration: Configuration
+
+ def __init__(self, configuration: Configuration):
+ self._logger = getLogger("DTSRunner")
+ self._result = DTSResult(self._logger)
+ self._configuration = configuration
+
+ def run(self):
+ """
+ The main process of DTS. Runs all build targets in all executions from the main
+ config file.
+ """
+ # check the python version of the server that run dts
+ check_dts_python_version()
+ sut_nodes: dict[str, SutNode] = {}
+ tg_nodes: dict[str, TGNode] = {}
+ try:
+ # for all Execution sections
+ for execution in self._configuration.executions:
+ sut_node = sut_nodes.get(execution.system_under_test_node.name)
+ tg_node = tg_nodes.get(execution.traffic_generator_node.name)
+
+ try:
+ if not sut_node:
+ sut_node = SutNode(execution.system_under_test_node)
+ sut_nodes[sut_node.name] = sut_node
+ if not tg_node:
+ tg_node = TGNode(execution.traffic_generator_node)
+ tg_nodes[tg_node.name] = tg_node
+ self._result.update_setup(Result.PASS)
+ except Exception as e:
+ failed_node = execution.system_under_test_node.name
+ if sut_node:
+ failed_node = execution.traffic_generator_node.name
+ self._logger.exception(
+ f"The Creation of node {failed_node} failed."
+ )
+ self._result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_execution(sut_node, tg_node, execution)
+
+ except Exception as e:
+ self._logger.exception("An unexpected error has occurred.")
+ self._result.add_error(e)
+ raise
+
+ finally:
+ try:
+ for node in (sut_nodes | tg_nodes).values():
+ node.close()
+ self._result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("The final cleanup of nodes failed.")
+ self._result.update_teardown(Result.ERROR, e)
+
+ # we need to put the sys.exit call outside the finally clause to make sure
+ # that unexpected exceptions will propagate
+ # in that case, the error that should be reported is the uncaught exception as
+ # that is a severe error originating from the framework
+ # at that point, we'll only have partial results which could be impacted by the
+ # error causing the uncaught exception, making them uninterpretable
+ self._exit_dts()
+
+ def _run_execution(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ ) -> None:
+ """
+ Run the given execution. This involves running the execution setup as well as
+ running all build targets in the given execution.
+ """
+ self._logger.info(
+ f"Running execution with SUT '{execution.system_under_test_node.name}'."
+ )
+ execution_result = self._result.add_execution(sut_node.config)
+ execution_result.add_sut_info(sut_node.node_info)
+
+ try:
+ sut_node.set_up_execution(execution)
+ execution_result.update_setup(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Execution setup failed.")
+ execution_result.update_setup(Result.FAIL, e)
+
+ else:
+ for build_target in execution.build_targets:
+ self._run_build_target(
+ sut_node, tg_node, build_target, execution, execution_result
+ )
+
+ finally:
+ try:
+ sut_node.tear_down_execution()
+ execution_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Execution teardown failed.")
+ execution_result.update_teardown(Result.FAIL, e)
+
+ def _run_build_target(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ build_target: BuildTargetConfiguration,
+ execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ ) -> None:
+ """
+ Run the given build target.
+ """
+ self._logger.info(f"Running build target '{build_target.name}'.")
+ build_target_result = execution_result.add_build_target(build_target)
+
+ try:
+ sut_node.set_up_build_target(build_target)
+ self._result.dpdk_version = sut_node.dpdk_version
+ build_target_result.add_build_target_info(sut_node.get_build_target_info())
+ build_target_result.update_setup(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Build target setup failed.")
+ build_target_result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_all_suites(sut_node, tg_node, execution, build_target_result)
+
+ finally:
+ try:
+ sut_node.tear_down_build_target()
+ build_target_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Build target teardown failed.")
+ build_target_result.update_teardown(Result.FAIL, e)
+
+ def _run_all_suites(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ build_target_result: BuildTargetResult,
+ ) -> None:
+ """
+ Use the given build_target to run execution's test suites
+ with possibly only a subset of test cases.
+ If no subset is specified, run all test cases.
+ """
+ end_build_target = False
+ if not execution.skip_smoke_tests:
+ execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
+ for test_suite_config in execution.test_suites:
+ try:
+ self._run_single_suite(
+ sut_node, tg_node, execution, build_target_result, test_suite_config
+ )
+ except BlockingTestSuiteError as e:
+ self._logger.exception(
+ f"An error occurred within {test_suite_config.test_suite}. "
+ "Skipping build target..."
+ )
+ self._result.add_error(e)
+ end_build_target = True
+ # if a blocking test failed and we need to bail out of suite executions
+ if end_build_target:
+ break
+
+ def _run_single_suite(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ build_target_result: BuildTargetResult,
+ test_suite_config: TestSuiteConfig,
+ ) -> None:
+ """Runs a single test suite.
+
+ Args:
+ sut_node: Node to run tests on.
+ execution: Execution the test case belongs to.
+ build_target_result: Build target configuration test case is run on
+ test_suite_config: Test suite configuration
+
+ Raises:
+ BlockingTestSuiteError: If a test suite that was marked as blocking fails.
+ """
+ try:
+ full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
+ test_suite_classes = get_test_suites(full_suite_path)
+ suites_str = ", ".join((x.__name__ for x in test_suite_classes))
+ self._logger.debug(
+ f"Found test suites '{suites_str}' in '{full_suite_path}'."
+ )
+ except Exception as e:
+ self._logger.exception("An error occurred when searching for test suites.")
+ self._result.update_setup(Result.ERROR, e)
+
+ else:
+ for test_suite_class in test_suite_classes:
+ test_suite = test_suite_class(
+ sut_node,
+ tg_node,
+ test_suite_config.test_cases,
+ execution.func,
+ build_target_result,
+ )
+ test_suite.run()
+
+ def _exit_dts(self) -> None:
+ """
+ Process all errors and exit with the proper exit code.
+ """
+ self._result.process()
+
+ if self._logger:
+ self._logger.info("DTS execution has ended.")
+
+ logging.shutdown()
+ sys.exit(self._result.get_return_code())
diff --git a/dts/main.py b/dts/main.py
index 43311fa847..879ce5cb89 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -10,11 +10,13 @@
import logging
-from framework import dts
+from framework.config import load_config
+from framework.runner import DTSRunner
def main() -> None:
- dts.run_all()
+ dts = DTSRunner(configuration=load_config())
+ dts.run()
# Main program begins here
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [RFC PATCH v1 2/5] dts: move test suite execution logic to DTSRunner
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 1/5] dts: convert dts.py methods to class Juraj Linkeš
@ 2023-12-20 10:33 ` Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 3/5] dts: process test suites at the beginning of run Juraj Linkeš
` (6 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
yoan.picchi, Luca.Vizzarro
Cc: dev, Juraj Linkeš
Move the code responsible for running the test suite from the
TestSuite class to the DTSRunner class. This restructuring decision
was made to consolidate and unify the related logic into a single unit.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 156 +++++++++++++++++++++++++++++++++---
dts/framework/test_suite.py | 134 +------------------------------
2 files changed, 147 insertions(+), 143 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 5b077c5805..5e145a8066 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -5,6 +5,7 @@
import logging
import sys
+from types import MethodType
from .config import (
BuildTargetConfiguration,
@@ -12,10 +13,18 @@
ExecutionConfiguration,
TestSuiteConfig,
)
-from .exception import BlockingTestSuiteError
+from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
from .logger import DTSLOG, getLogger
-from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
-from .test_suite import get_test_suites
+from .settings import SETTINGS
+from .test_result import (
+ BuildTargetResult,
+ DTSResult,
+ ExecutionResult,
+ Result,
+ TestCaseResult,
+ TestSuiteResult,
+)
+from .test_suite import TestSuite, get_test_suites
from .testbed_model import SutNode, TGNode
from .utils import check_dts_python_version
@@ -148,7 +157,7 @@ def _run_build_target(
build_target_result.update_setup(Result.FAIL, e)
else:
- self._run_all_suites(sut_node, tg_node, execution, build_target_result)
+ self._run_test_suites(sut_node, tg_node, execution, build_target_result)
finally:
try:
@@ -158,7 +167,7 @@ def _run_build_target(
self._logger.exception("Build target teardown failed.")
build_target_result.update_teardown(Result.FAIL, e)
- def _run_all_suites(
+ def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
@@ -175,7 +184,7 @@ def _run_all_suites(
execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
for test_suite_config in execution.test_suites:
try:
- self._run_single_suite(
+ self._run_test_suite(
sut_node, tg_node, execution, build_target_result, test_suite_config
)
except BlockingTestSuiteError as e:
@@ -189,7 +198,7 @@ def _run_all_suites(
if end_build_target:
break
- def _run_single_suite(
+ def _run_test_suite(
self,
sut_node: SutNode,
tg_node: TGNode,
@@ -198,6 +207,9 @@ def _run_single_suite(
test_suite_config: TestSuiteConfig,
) -> None:
"""Runs a single test suite.
+ Setup, execute and teardown the whole suite.
+ Suite execution consists of running all test cases scheduled to be executed.
+ A test cast run consists of setup, execution and teardown of said test case.
Args:
sut_node: Node to run tests on.
@@ -222,13 +234,131 @@ def _run_single_suite(
else:
for test_suite_class in test_suite_classes:
test_suite = test_suite_class(
- sut_node,
- tg_node,
- test_suite_config.test_cases,
- execution.func,
- build_target_result,
+ sut_node, tg_node, test_suite_config.test_cases
+ )
+
+ test_suite_name = test_suite.__class__.__name__
+ test_suite_result = build_target_result.add_test_suite(test_suite_name)
+ try:
+ self._logger.info(f"Starting test suite setup: {test_suite_name}")
+ test_suite.set_up_suite()
+ test_suite_result.update_setup(Result.PASS)
+ self._logger.info(f"Test suite setup successful: {test_suite_name}")
+ except Exception as e:
+ self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
+ test_suite_result.update_setup(Result.ERROR, e)
+
+ else:
+ self._execute_test_suite(
+ execution.func, test_suite, test_suite_result
+ )
+
+ finally:
+ try:
+ test_suite.tear_down_suite()
+ sut_node.kill_cleanup_dpdk_apps()
+ test_suite_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(
+ f"Test suite teardown ERROR: {test_suite_name}"
+ )
+ self._logger.warning(
+ f"Test suite '{test_suite_name}' teardown failed, "
+ f"the next test suite may be affected."
+ )
+ test_suite_result.update_setup(Result.ERROR, e)
+ if (
+ len(test_suite_result.get_errors()) > 0
+ and test_suite.is_blocking
+ ):
+ raise BlockingTestSuiteError(test_suite_name)
+
+ def _execute_test_suite(
+ self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
+ ) -> None:
+ """
+ Execute all test cases scheduled to be executed in this suite.
+ """
+ if func:
+ for test_case_method in test_suite._get_functional_test_cases():
+ test_case_name = test_case_method.__name__
+ test_case_result = test_suite_result.add_test_case(test_case_name)
+ all_attempts = SETTINGS.re_run + 1
+ attempt_nr = 1
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+
+ def _run_test_case(
+ self,
+ test_suite: TestSuite,
+ test_case_method: MethodType,
+ test_case_result: TestCaseResult,
+ ) -> None:
+ """
+ Setup, execute and teardown a test case in this suite.
+ Exceptions are caught and recorded in logs and results.
+ """
+ test_case_name = test_case_method.__name__
+
+ try:
+ # run set_up function for each case
+ test_suite.set_up_test_case()
+ test_case_result.update_setup(Result.PASS)
+ except SSHTimeoutError as e:
+ self._logger.exception(f"Test case setup FAILED: {test_case_name}")
+ test_case_result.update_setup(Result.FAIL, e)
+ except Exception as e:
+ self._logger.exception(f"Test case setup ERROR: {test_case_name}")
+ test_case_result.update_setup(Result.ERROR, e)
+
+ else:
+ # run test case if setup was successful
+ self._execute_test_case(test_case_method, test_case_result)
+
+ finally:
+ try:
+ test_suite.tear_down_test_case()
+ test_case_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test case teardown ERROR: {test_case_name}")
+ self._logger.warning(
+ f"Test case '{test_case_name}' teardown failed, "
+ f"the next test case may be affected."
)
- test_suite.run()
+ test_case_result.update_teardown(Result.ERROR, e)
+ test_case_result.update(Result.ERROR)
+
+ def _execute_test_case(
+ self, test_case_method: MethodType, test_case_result: TestCaseResult
+ ) -> None:
+ """
+ Execute one test case and handle failures.
+ """
+ test_case_name = test_case_method.__name__
+ try:
+ self._logger.info(f"Starting test case execution: {test_case_name}")
+ test_case_method()
+ test_case_result.update(Result.PASS)
+ self._logger.info(f"Test case execution PASSED: {test_case_name}")
+
+ except TestCaseVerifyError as e:
+ self._logger.exception(f"Test case execution FAILED: {test_case_name}")
+ test_case_result.update(Result.FAIL, e)
+ except Exception as e:
+ self._logger.exception(f"Test case execution ERROR: {test_case_name}")
+ test_case_result.update(Result.ERROR, e)
+ except KeyboardInterrupt:
+ self._logger.error(
+ f"Test case execution INTERRUPTED by user: {test_case_name}"
+ )
+ test_case_result.update(Result.SKIP)
+ raise KeyboardInterrupt("Stop DTS")
def _exit_dts(self) -> None:
"""
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 4a7907ec33..e96305deb0 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -17,15 +17,9 @@
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
-from .exception import (
- BlockingTestSuiteError,
- ConfigurationError,
- SSHTimeoutError,
- TestCaseVerifyError,
-)
+from .exception import ConfigurationError, TestCaseVerifyError
from .logger import DTSLOG, getLogger
from .settings import SETTINGS
-from .test_result import BuildTargetResult, Result, TestCaseResult, TestSuiteResult
from .testbed_model import SutNode, TGNode
from .testbed_model.hw.port import Port, PortLink
from .utils import get_packet_summaries
@@ -50,11 +44,10 @@ class TestSuite(object):
"""
sut_node: SutNode
+ tg_node: TGNode
is_blocking = False
_logger: DTSLOG
_test_cases_to_run: list[str]
- _func: bool
- _result: TestSuiteResult
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -69,17 +62,13 @@ def __init__(
self,
sut_node: SutNode,
tg_node: TGNode,
- test_cases: list[str],
- func: bool,
- build_target_result: BuildTargetResult,
+ test_cases_to_run: list[str],
):
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = getLogger(self.__class__.__name__)
- self._test_cases_to_run = test_cases
+ self._test_cases_to_run = test_cases_to_run
self._test_cases_to_run.extend(SETTINGS.test_cases)
- self._func = func
- self._result = build_target_result.add_test_suite(self.__class__.__name__)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
@@ -280,60 +269,6 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
return False
return True
- def run(self) -> None:
- """
- Setup, execute and teardown the whole suite.
- Suite execution consists of running all test cases scheduled to be executed.
- A test cast run consists of setup, execution and teardown of said test case.
- """
- test_suite_name = self.__class__.__name__
-
- try:
- self._logger.info(f"Starting test suite setup: {test_suite_name}")
- self.set_up_suite()
- self._result.update_setup(Result.PASS)
- self._logger.info(f"Test suite setup successful: {test_suite_name}")
- except Exception as e:
- self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
- self._result.update_setup(Result.ERROR, e)
-
- else:
- self._execute_test_suite()
-
- finally:
- try:
- self.tear_down_suite()
- self.sut_node.kill_cleanup_dpdk_apps()
- self._result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
- self._logger.warning(
- f"Test suite '{test_suite_name}' teardown failed, "
- f"the next test suite may be affected."
- )
- self._result.update_setup(Result.ERROR, e)
- if len(self._result.get_errors()) > 0 and self.is_blocking:
- raise BlockingTestSuiteError(test_suite_name)
-
- def _execute_test_suite(self) -> None:
- """
- Execute all test cases scheduled to be executed in this suite.
- """
- if self._func:
- for test_case_method in self._get_functional_test_cases():
- test_case_name = test_case_method.__name__
- test_case_result = self._result.add_test_case(test_case_name)
- all_attempts = SETTINGS.re_run + 1
- attempt_nr = 1
- self._run_test_case(test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
- self._run_test_case(test_case_method, test_case_result)
-
def _get_functional_test_cases(self) -> list[MethodType]:
"""
Get all functional test cases.
@@ -363,67 +298,6 @@ def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool
return match
- def _run_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
- ) -> None:
- """
- Setup, execute and teardown a test case in this suite.
- Exceptions are caught and recorded in logs and results.
- """
- test_case_name = test_case_method.__name__
-
- try:
- # run set_up function for each case
- self.set_up_test_case()
- test_case_result.update_setup(Result.PASS)
- except SSHTimeoutError as e:
- self._logger.exception(f"Test case setup FAILED: {test_case_name}")
- test_case_result.update_setup(Result.FAIL, e)
- except Exception as e:
- self._logger.exception(f"Test case setup ERROR: {test_case_name}")
- test_case_result.update_setup(Result.ERROR, e)
-
- else:
- # run test case if setup was successful
- self._execute_test_case(test_case_method, test_case_result)
-
- finally:
- try:
- self.tear_down_test_case()
- test_case_result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test case teardown ERROR: {test_case_name}")
- self._logger.warning(
- f"Test case '{test_case_name}' teardown failed, "
- f"the next test case may be affected."
- )
- test_case_result.update_teardown(Result.ERROR, e)
- test_case_result.update(Result.ERROR)
-
- def _execute_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
- ) -> None:
- """
- Execute one test case and handle failures.
- """
- test_case_name = test_case_method.__name__
- try:
- self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method()
- test_case_result.update(Result.PASS)
- self._logger.info(f"Test case execution PASSED: {test_case_name}")
-
- except TestCaseVerifyError as e:
- self._logger.exception(f"Test case execution FAILED: {test_case_name}")
- test_case_result.update(Result.FAIL, e)
- except Exception as e:
- self._logger.exception(f"Test case execution ERROR: {test_case_name}")
- test_case_result.update(Result.ERROR, e)
- except KeyboardInterrupt:
- self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}")
- test_case_result.update(Result.SKIP)
- raise KeyboardInterrupt("Stop DTS")
-
def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
def is_test_suite(object) -> bool:
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [RFC PATCH v1 3/5] dts: process test suites at the beginning of run
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 1/5] dts: convert dts.py methods to class Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 2/5] dts: move test suite execution logic to DTSRunner Juraj Linkeš
@ 2023-12-20 10:33 ` Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 4/5] dts: block all testcases when earlier setup fails Juraj Linkeš
` (5 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
yoan.picchi, Luca.Vizzarro
Cc: dev, Juraj Linkeš
We initialize test suite/case objects at the start of
the program and store them with custom execution config
(add test case names) in Execution. This change helps
identify errors with test suites earlier, and we have
access to the right data when programs crash earlier.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/config/__init__.py | 5 +-
dts/framework/runner.py | 309 ++++++++++++++++++++---------
dts/framework/test_suite.py | 59 +-----
dts/tests/TestSuite_smoke_tests.py | 2 +-
4 files changed, 217 insertions(+), 158 deletions(-)
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 497847afb9..d65ac625f8 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -238,7 +238,6 @@ class ExecutionConfiguration:
system_under_test_node: SutNodeConfiguration
traffic_generator_node: TGNodeConfiguration
vdevs: list[str]
- skip_smoke_tests: bool
@staticmethod
def from_dict(
@@ -247,9 +246,10 @@ def from_dict(
build_targets: list[BuildTargetConfiguration] = list(
map(BuildTargetConfiguration.from_dict, d["build_targets"])
)
+ if not d.get("skip_smoke_tests", False):
+ d["test_suites"].insert(0, "smoke_tests")
test_suites: list[TestSuiteConfig] = list(map(TestSuiteConfig.from_dict, d["test_suites"]))
sut_name = d["system_under_test_node"]["node_name"]
- skip_smoke_tests = d.get("skip_smoke_tests", False)
assert sut_name in node_map, f"Unknown SUT {sut_name} in execution {d}"
system_under_test_node = node_map[sut_name]
assert isinstance(
@@ -270,7 +270,6 @@ def from_dict(
build_targets=build_targets,
perf=d["perf"],
func=d["func"],
- skip_smoke_tests=skip_smoke_tests,
test_suites=test_suites,
system_under_test_node=system_under_test_node,
traffic_generator_node=traffic_generator_node,
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 5e145a8066..acc3342f0c 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -3,9 +3,14 @@
# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
# Copyright(c) 2022-2023 University of New Hampshire
+import importlib
+import inspect
import logging
+import re
import sys
-from types import MethodType
+from copy import deepcopy
+from dataclasses import dataclass
+from types import MethodType, ModuleType
from .config import (
BuildTargetConfiguration,
@@ -13,7 +18,12 @@
ExecutionConfiguration,
TestSuiteConfig,
)
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .exception import (
+ BlockingTestSuiteError,
+ ConfigurationError,
+ SSHTimeoutError,
+ TestCaseVerifyError,
+)
from .logger import DTSLOG, getLogger
from .settings import SETTINGS
from .test_result import (
@@ -24,25 +34,55 @@
TestCaseResult,
TestSuiteResult,
)
-from .test_suite import TestSuite, get_test_suites
+from .test_suite import TestSuite
from .testbed_model import SutNode, TGNode
from .utils import check_dts_python_version
+@dataclass
+class TestSuiteSetup:
+ test_suite: type[TestSuite]
+ test_cases: list[MethodType]
+
+ def processed_config(self) -> TestSuiteConfig:
+ return TestSuiteConfig(
+ test_suite=self.test_suite.__name__,
+ test_cases=[test_case.__name__ for test_case in self.test_cases],
+ )
+
+
+@dataclass
+class Execution:
+ config: ExecutionConfiguration
+ test_suite_setups: list[TestSuiteSetup]
+
+ def processed_config(self) -> ExecutionConfiguration:
+ """
+ Creating copy of execution config witch add test-case names.
+ """
+ modified_execution_config = deepcopy(self.config)
+ modified_execution_config.test_suites[:] = [
+ test_suite.processed_config() for test_suite in self.test_suite_setups
+ ]
+ return modified_execution_config
+
+
class DTSRunner:
_logger: DTSLOG
_result: DTSResult
- _configuration: Configuration
+ _executions: list[Execution]
def __init__(self, configuration: Configuration):
self._logger = getLogger("DTSRunner")
self._result = DTSResult(self._logger)
- self._configuration = configuration
+ self._executions = create_executions(configuration.executions)
def run(self):
"""
The main process of DTS. Runs all build targets in all executions from the main
config file.
+ Suite execution consists of running all test cases scheduled to be executed.
+ A test case run consists of setup, execution and teardown of said test case.
"""
# check the python version of the server that run dts
check_dts_python_version()
@@ -50,22 +90,22 @@ def run(self):
tg_nodes: dict[str, TGNode] = {}
try:
# for all Execution sections
- for execution in self._configuration.executions:
- sut_node = sut_nodes.get(execution.system_under_test_node.name)
- tg_node = tg_nodes.get(execution.traffic_generator_node.name)
+ for execution in self._executions:
+ sut_node = sut_nodes.get(execution.config.system_under_test_node.name)
+ tg_node = tg_nodes.get(execution.config.traffic_generator_node.name)
try:
if not sut_node:
- sut_node = SutNode(execution.system_under_test_node)
+ sut_node = SutNode(execution.config.system_under_test_node)
sut_nodes[sut_node.name] = sut_node
if not tg_node:
- tg_node = TGNode(execution.traffic_generator_node)
+ tg_node = TGNode(execution.config.traffic_generator_node)
tg_nodes[tg_node.name] = tg_node
self._result.update_setup(Result.PASS)
except Exception as e:
- failed_node = execution.system_under_test_node.name
+ failed_node = execution.config.system_under_test_node.name
if sut_node:
- failed_node = execution.traffic_generator_node.name
+ failed_node = execution.config.traffic_generator_node.name
self._logger.exception(
f"The Creation of node {failed_node} failed."
)
@@ -100,29 +140,34 @@ def _run_execution(
self,
sut_node: SutNode,
tg_node: TGNode,
- execution: ExecutionConfiguration,
+ execution: Execution,
) -> None:
"""
Run the given execution. This involves running the execution setup as well as
running all build targets in the given execution.
"""
self._logger.info(
- f"Running execution with SUT '{execution.system_under_test_node.name}'."
+ "Running execution with SUT "
+ f"'{execution.config.system_under_test_node.name}'."
)
execution_result = self._result.add_execution(sut_node.config)
execution_result.add_sut_info(sut_node.node_info)
try:
- sut_node.set_up_execution(execution)
+ sut_node.set_up_execution(execution.config)
execution_result.update_setup(Result.PASS)
except Exception as e:
self._logger.exception("Execution setup failed.")
execution_result.update_setup(Result.FAIL, e)
else:
- for build_target in execution.build_targets:
+ for build_target in execution.config.build_targets:
self._run_build_target(
- sut_node, tg_node, build_target, execution, execution_result
+ sut_node,
+ tg_node,
+ build_target,
+ execution,
+ execution_result,
)
finally:
@@ -138,7 +183,7 @@ def _run_build_target(
sut_node: SutNode,
tg_node: TGNode,
build_target: BuildTargetConfiguration,
- execution: ExecutionConfiguration,
+ execution: Execution,
execution_result: ExecutionResult,
) -> None:
"""
@@ -171,7 +216,7 @@ def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
- execution: ExecutionConfiguration,
+ execution: Execution,
build_target_result: BuildTargetResult,
) -> None:
"""
@@ -180,16 +225,18 @@ def _run_test_suites(
If no subset is specified, run all test cases.
"""
end_build_target = False
- if not execution.skip_smoke_tests:
- execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
- for test_suite_config in execution.test_suites:
+ for test_suite_setup in execution.test_suite_setups:
try:
self._run_test_suite(
- sut_node, tg_node, execution, build_target_result, test_suite_config
+ sut_node,
+ tg_node,
+ test_suite_setup,
+ build_target_result,
)
except BlockingTestSuiteError as e:
self._logger.exception(
- f"An error occurred within {test_suite_config.test_suite}. "
+ "An error occurred within "
+ f"{test_suite_setup.test_suite.__name__}. "
"Skipping build target..."
)
self._result.add_error(e)
@@ -202,14 +249,10 @@ def _run_test_suite(
self,
sut_node: SutNode,
tg_node: TGNode,
- execution: ExecutionConfiguration,
+ test_suite_setup: TestSuiteSetup,
build_target_result: BuildTargetResult,
- test_suite_config: TestSuiteConfig,
) -> None:
"""Runs a single test suite.
- Setup, execute and teardown the whole suite.
- Suite execution consists of running all test cases scheduled to be executed.
- A test cast run consists of setup, execution and teardown of said test case.
Args:
sut_node: Node to run tests on.
@@ -220,84 +263,67 @@ def _run_test_suite(
Raises:
BlockingTestSuiteError: If a test suite that was marked as blocking fails.
"""
+ test_suite = test_suite_setup.test_suite(sut_node, tg_node)
+ test_suite_name = test_suite_setup.test_suite.__name__
+ test_suite_result = build_target_result.add_test_suite(test_suite_name)
try:
- full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
- test_suite_classes = get_test_suites(full_suite_path)
- suites_str = ", ".join((x.__name__ for x in test_suite_classes))
- self._logger.debug(
- f"Found test suites '{suites_str}' in '{full_suite_path}'."
- )
+ self._logger.info(f"Starting test suite setup: {test_suite_name}")
+ test_suite.set_up_suite()
+ test_suite_result.update_setup(Result.PASS)
+ self._logger.info(f"Test suite setup successful: {test_suite_name}")
except Exception as e:
- self._logger.exception("An error occurred when searching for test suites.")
- self._result.update_setup(Result.ERROR, e)
+ self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
+ test_suite_result.update_setup(Result.ERROR, e)
else:
- for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(
- sut_node, tg_node, test_suite_config.test_cases
- )
-
- test_suite_name = test_suite.__class__.__name__
- test_suite_result = build_target_result.add_test_suite(test_suite_name)
- try:
- self._logger.info(f"Starting test suite setup: {test_suite_name}")
- test_suite.set_up_suite()
- test_suite_result.update_setup(Result.PASS)
- self._logger.info(f"Test suite setup successful: {test_suite_name}")
- except Exception as e:
- self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
- test_suite_result.update_setup(Result.ERROR, e)
-
- else:
- self._execute_test_suite(
- execution.func, test_suite, test_suite_result
- )
+ self._execute_test_suite(
+ test_suite,
+ test_suite_setup.test_cases,
+ test_suite_result,
+ )
- finally:
- try:
- test_suite.tear_down_suite()
- sut_node.kill_cleanup_dpdk_apps()
- test_suite_result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(
- f"Test suite teardown ERROR: {test_suite_name}"
- )
- self._logger.warning(
- f"Test suite '{test_suite_name}' teardown failed, "
- f"the next test suite may be affected."
- )
- test_suite_result.update_setup(Result.ERROR, e)
- if (
- len(test_suite_result.get_errors()) > 0
- and test_suite.is_blocking
- ):
- raise BlockingTestSuiteError(test_suite_name)
+ finally:
+ try:
+ test_suite.tear_down_suite()
+ sut_node.kill_cleanup_dpdk_apps()
+ test_suite_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
+ self._logger.warning(
+ f"Test suite '{test_suite_name}' teardown failed, "
+ "the next test suite may be affected."
+ )
+ test_suite_result.update_setup(Result.ERROR, e)
+ if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
+ raise BlockingTestSuiteError(test_suite_name)
def _execute_test_suite(
- self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
+ self,
+ test_suite: TestSuite,
+ test_cases: list[MethodType],
+ test_suite_result: TestSuiteResult,
) -> None:
"""
Execute all test cases scheduled to be executed in this suite.
"""
- if func:
- for test_case_method in test_suite._get_functional_test_cases():
- test_case_name = test_case_method.__name__
- test_case_result = test_suite_result.add_test_case(test_case_name)
- all_attempts = SETTINGS.re_run + 1
- attempt_nr = 1
- self._run_test_case(test_suite, test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
- self._run_test_case(test_suite, test_case_method, test_case_result)
+ for test_case_method in test_cases:
+ test_case_name = test_case_method.__name__
+ test_case_result = test_suite_result.add_test_case(test_case_name)
+ all_attempts = SETTINGS.re_run + 1
+ attempt_nr = 1
+ self._run_test_case(test_case_method, test_suite, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
+ self._run_test_case(test_case_method, test_suite, test_case_result)
def _run_test_case(
self,
- test_suite: TestSuite,
test_case_method: MethodType,
+ test_suite: TestSuite,
test_case_result: TestCaseResult,
) -> None:
"""
@@ -305,7 +331,6 @@ def _run_test_case(
Exceptions are caught and recorded in logs and results.
"""
test_case_name = test_case_method.__name__
-
try:
# run set_up function for each case
test_suite.set_up_test_case()
@@ -319,7 +344,7 @@ def _run_test_case(
else:
# run test case if setup was successful
- self._execute_test_case(test_case_method, test_case_result)
+ self._execute_test_case(test_case_method, test_suite, test_case_result)
finally:
try:
@@ -335,7 +360,10 @@ def _run_test_case(
test_case_result.update(Result.ERROR)
def _execute_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
+ self,
+ test_case_method: MethodType,
+ test_suite: TestSuite,
+ test_case_result: TestCaseResult,
) -> None:
"""
Execute one test case and handle failures.
@@ -343,7 +371,7 @@ def _execute_test_case(
test_case_name = test_case_method.__name__
try:
self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method()
+ test_case_method(test_suite)
test_case_result.update(Result.PASS)
self._logger.info(f"Test case execution PASSED: {test_case_name}")
@@ -371,3 +399,92 @@ def _exit_dts(self) -> None:
logging.shutdown()
sys.exit(self._result.get_return_code())
+
+
+def create_executions(
+ execution_configs: list[ExecutionConfiguration],
+) -> list[Execution]:
+ executions: list[Execution] = []
+ for execution_config in execution_configs:
+ test_suite_setups: list[TestSuiteSetup] = []
+
+ for test_suite_config in execution_config.test_suites:
+ testsuite_module_path = f"tests.TestSuite_{test_suite_config.test_suite}"
+ try:
+ suite_module = importlib.import_module(testsuite_module_path)
+ except ModuleNotFoundError as e:
+ raise ConfigurationError(
+ f"Test suite '{testsuite_module_path}' not found."
+ ) from e
+
+ test_suite = _get_suite_class(suite_module, test_suite_config.test_suite)
+
+ test_cases_to_run = test_suite_config.test_cases
+ test_cases_to_run.extend(SETTINGS.test_cases)
+
+ test_cases = []
+ if execution_config.func:
+ # add functional test cases
+ test_cases.extend(
+ _get_test_cases(test_suite, r"test_(?!perf_)", test_cases_to_run)
+ )
+
+ if execution_config.perf:
+ # add performance test cases
+ test_cases.extend(
+ _get_test_cases(test_suite, r"test_perf_", test_cases_to_run)
+ )
+
+ test_suite_setups.append(
+ TestSuiteSetup(test_suite=test_suite, test_cases=test_cases)
+ )
+
+ executions.append(
+ Execution(
+ config=execution_config,
+ test_suite_setups=test_suite_setups,
+ )
+ )
+
+ return executions
+
+
+def _get_suite_class(suite_module: ModuleType, suite_name: str) -> type[TestSuite]:
+ def is_test_suite(object) -> bool:
+ try:
+ if issubclass(object, TestSuite) and object is not TestSuite:
+ return True
+ except TypeError:
+ return False
+ return False
+
+ suite_name_regex = suite_name.replace("_", "").lower()
+ for class_name, suite_class in inspect.getmembers(suite_module, is_test_suite):
+ if not class_name.startswith("Test"):
+ continue
+
+ if suite_name_regex == class_name[4:].lower():
+ return suite_class
+ raise ConfigurationError(
+ f"Cannot find valid test suite in {suite_module.__name__}."
+ )
+
+
+def _get_test_cases(
+ suite_class: type[TestSuite], test_case_regex: str, test_cases_to_run: list[str]
+) -> list[MethodType]:
+ def should_be_executed(test_case_name: str) -> bool:
+ match = bool(re.match(test_case_regex, test_case_name))
+ if test_cases_to_run:
+ return match and test_case_name in test_cases_to_run
+
+ return match
+
+ test_cases = []
+ for test_case_name, test_case_method in inspect.getmembers(
+ suite_class, inspect.isfunction
+ ):
+ if should_be_executed(test_case_name):
+ test_cases.append(test_case_method)
+
+ return test_cases
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index e96305deb0..e73206993d 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -6,20 +6,15 @@
Base class for creating DTS test cases.
"""
-import importlib
-import inspect
-import re
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
-from types import MethodType
from typing import Union
from scapy.layers.inet import IP # type: ignore[import]
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
-from .exception import ConfigurationError, TestCaseVerifyError
+from .exception import TestCaseVerifyError
from .logger import DTSLOG, getLogger
-from .settings import SETTINGS
from .testbed_model import SutNode, TGNode
from .testbed_model.hw.port import Port, PortLink
from .utils import get_packet_summaries
@@ -47,7 +42,6 @@ class TestSuite(object):
tg_node: TGNode
is_blocking = False
_logger: DTSLOG
- _test_cases_to_run: list[str]
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -62,13 +56,10 @@ def __init__(
self,
sut_node: SutNode,
tg_node: TGNode,
- test_cases_to_run: list[str],
):
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = getLogger(self.__class__.__name__)
- self._test_cases_to_run = test_cases_to_run
- self._test_cases_to_run.extend(SETTINGS.test_cases)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
@@ -268,51 +259,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
return False
return True
-
- def _get_functional_test_cases(self) -> list[MethodType]:
- """
- Get all functional test cases.
- """
- return self._get_test_cases(r"test_(?!perf_)")
-
- def _get_test_cases(self, test_case_regex: str) -> list[MethodType]:
- """
- Return a list of test cases matching test_case_regex.
- """
- self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.")
- filtered_test_cases = []
- for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod):
- if self._should_be_executed(test_case_name, test_case_regex):
- filtered_test_cases.append(test_case)
- cases_str = ", ".join((x.__name__ for x in filtered_test_cases))
- self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.")
- return filtered_test_cases
-
- def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool:
- """
- Check whether the test case should be executed.
- """
- match = bool(re.match(test_case_regex, test_case_name))
- if self._test_cases_to_run:
- return match and test_case_name in self._test_cases_to_run
-
- return match
-
-
-def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
- def is_test_suite(object) -> bool:
- try:
- if issubclass(object, TestSuite) and object is not TestSuite:
- return True
- except TypeError:
- return False
- return False
-
- try:
- testcase_module = importlib.import_module(testsuite_module_path)
- except ModuleNotFoundError as e:
- raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e
- return [
- test_suite_class
- for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite)
- ]
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index 8958f58dac..aa4bae5b17 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -10,7 +10,7 @@
from framework.utils import REGEX_FOR_PCI_ADDRESS
-class SmokeTests(TestSuite):
+class TestSmokeTests(TestSuite):
is_blocking = True
# dicts in this list are expected to have two keys:
# "pci_address" and "current_driver"
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [RFC PATCH v1 4/5] dts: block all testcases when earlier setup fails
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
` (2 preceding siblings ...)
2023-12-20 10:33 ` [RFC PATCH v1 3/5] dts: process test suites at the beginning of run Juraj Linkeš
@ 2023-12-20 10:33 ` Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 5/5] dts: refactor logging configuration Juraj Linkeš
` (4 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
yoan.picchi, Luca.Vizzarro
Cc: dev, Juraj Linkeš
In case of a failure during execution, build target or suite setup
the test case results will be recorded as blocked. We also unify
methods add_<result> to add_child_result to be more consistently.
Now we store the corresponding config in each result with child
configs and parent result.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 12 +-
dts/framework/test_result.py | 361 ++++++++++++++++++++---------------
2 files changed, 216 insertions(+), 157 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index acc3342f0c..28570d4a1c 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -74,7 +74,7 @@ class DTSRunner:
def __init__(self, configuration: Configuration):
self._logger = getLogger("DTSRunner")
- self._result = DTSResult(self._logger)
+ self._result = DTSResult(configuration, self._logger)
self._executions = create_executions(configuration.executions)
def run(self):
@@ -150,7 +150,7 @@ def _run_execution(
"Running execution with SUT "
f"'{execution.config.system_under_test_node.name}'."
)
- execution_result = self._result.add_execution(sut_node.config)
+ execution_result = self._result.add_child_result(execution.processed_config())
execution_result.add_sut_info(sut_node.node_info)
try:
@@ -190,7 +190,7 @@ def _run_build_target(
Run the given build target.
"""
self._logger.info(f"Running build target '{build_target.name}'.")
- build_target_result = execution_result.add_build_target(build_target)
+ build_target_result = execution_result.add_child_result(build_target)
try:
sut_node.set_up_build_target(build_target)
@@ -265,7 +265,9 @@ def _run_test_suite(
"""
test_suite = test_suite_setup.test_suite(sut_node, tg_node)
test_suite_name = test_suite_setup.test_suite.__name__
- test_suite_result = build_target_result.add_test_suite(test_suite_name)
+ test_suite_result = build_target_result.add_child_result(
+ test_suite_setup.processed_config()
+ )
try:
self._logger.info(f"Starting test suite setup: {test_suite_name}")
test_suite.set_up_suite()
@@ -308,7 +310,7 @@ def _execute_test_suite(
"""
for test_case_method in test_cases:
test_case_name = test_case_method.__name__
- test_case_result = test_suite_result.add_test_case(test_case_name)
+ test_case_result = test_suite_result.add_child_result(test_case_name)
all_attempts = SETTINGS.re_run + 1
attempt_nr = 1
self._run_test_case(test_case_method, test_suite, test_case_result)
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 4c2e7e2418..dba2c55d36 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -9,6 +9,7 @@
import os.path
from collections.abc import MutableSequence
from enum import Enum, auto
+from typing import Any, Union
from .config import (
OS,
@@ -16,9 +17,11 @@
BuildTargetConfiguration,
BuildTargetInfo,
Compiler,
+ Configuration,
CPUType,
- NodeConfiguration,
+ ExecutionConfiguration,
NodeInfo,
+ TestSuiteConfig,
)
from .exception import DTSError, ErrorSeverity
from .logger import DTSLOG
@@ -35,6 +38,7 @@ class Result(Enum):
FAIL = auto()
ERROR = auto()
SKIP = auto()
+ BLOCK = auto()
def __bool__(self) -> bool:
return self is self.PASS
@@ -63,42 +67,6 @@ def __bool__(self) -> bool:
return bool(self.result)
-class Statistics(dict):
- """
- A helper class used to store the number of test cases by its result
- along a few other basic information.
- Using a dict provides a convenient way to format the data.
- """
-
- def __init__(self, dpdk_version: str | None):
- super(Statistics, self).__init__()
- for result in Result:
- self[result.name] = 0
- self["PASS RATE"] = 0.0
- self["DPDK VERSION"] = dpdk_version
-
- def __iadd__(self, other: Result) -> "Statistics":
- """
- Add a Result to the final count.
- """
- self[other.name] += 1
- self["PASS RATE"] = (
- float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result)
- )
- return self
-
- def __str__(self) -> str:
- """
- Provide a string representation of the data.
- """
- stats_str = ""
- for key, value in self.items():
- stats_str += f"{key:<12} = {value}\n"
- # according to docs, we should use \n when writing to text files
- # on all platforms
- return stats_str
-
-
class BaseResult(object):
"""
The Base class for all results. Stores the results of
@@ -109,6 +77,12 @@ class BaseResult(object):
setup_result: FixtureResult
teardown_result: FixtureResult
_inner_results: MutableSequence["BaseResult"]
+ _child_configs: Union[
+ list[ExecutionConfiguration],
+ list[BuildTargetConfiguration],
+ list[TestSuiteConfig],
+ list[str],
+ ]
def __init__(self):
self.setup_result = FixtureResult()
@@ -119,6 +93,23 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
self.setup_result.result = result
self.setup_result.error = error
+ if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
+ for child_config in self._child_configs:
+ child_result = self.add_child_result(child_config)
+ child_result.block()
+
+ def add_child_result(self, config: Any) -> "BaseResult":
+ """
+ Adding corresponding result for each classes.
+ """
+
+ def block(self):
+ """
+ Mark the result as block on corresponding classes.
+ """
+ self.update_setup(Result.BLOCK)
+ self.update_teardown(Result.BLOCK)
+
def update_teardown(self, result: Result, error: Exception | None = None) -> None:
self.teardown_result.result = result
self.teardown_result.error = error
@@ -139,119 +130,11 @@ def _get_inner_errors(self) -> list[Exception]:
def get_errors(self) -> list[Exception]:
return self._get_setup_teardown_errors() + self._get_inner_errors()
- def add_stats(self, statistics: Statistics) -> None:
+ def add_stats(self, statistics: "Statistics") -> None:
for inner_result in self._inner_results:
inner_result.add_stats(statistics)
-class TestCaseResult(BaseResult, FixtureResult):
- """
- The test case specific result.
- Stores the result of the actual test case.
- Also stores the test case name.
- """
-
- test_case_name: str
-
- def __init__(self, test_case_name: str):
- super(TestCaseResult, self).__init__()
- self.test_case_name = test_case_name
-
- def update(self, result: Result, error: Exception | None = None) -> None:
- self.result = result
- self.error = error
-
- def _get_inner_errors(self) -> list[Exception]:
- if self.error:
- return [self.error]
- return []
-
- def add_stats(self, statistics: Statistics) -> None:
- statistics += self.result
-
- def __bool__(self) -> bool:
- return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
-
-
-class TestSuiteResult(BaseResult):
- """
- The test suite specific result.
- The _inner_results list stores results of test cases in a given test suite.
- Also stores the test suite name.
- """
-
- suite_name: str
-
- def __init__(self, suite_name: str):
- super(TestSuiteResult, self).__init__()
- self.suite_name = suite_name
-
- def add_test_case(self, test_case_name: str) -> TestCaseResult:
- test_case_result = TestCaseResult(test_case_name)
- self._inner_results.append(test_case_result)
- return test_case_result
-
-
-class BuildTargetResult(BaseResult):
- """
- The build target specific result.
- The _inner_results list stores results of test suites in a given build target.
- Also stores build target specifics, such as compiler used to build DPDK.
- """
-
- arch: Architecture
- os: OS
- cpu: CPUType
- compiler: Compiler
- compiler_version: str | None
- dpdk_version: str | None
-
- def __init__(self, build_target: BuildTargetConfiguration):
- super(BuildTargetResult, self).__init__()
- self.arch = build_target.arch
- self.os = build_target.os
- self.cpu = build_target.cpu
- self.compiler = build_target.compiler
- self.compiler_version = None
- self.dpdk_version = None
-
- def add_build_target_info(self, versions: BuildTargetInfo) -> None:
- self.compiler_version = versions.compiler_version
- self.dpdk_version = versions.dpdk_version
-
- def add_test_suite(self, test_suite_name: str) -> TestSuiteResult:
- test_suite_result = TestSuiteResult(test_suite_name)
- self._inner_results.append(test_suite_result)
- return test_suite_result
-
-
-class ExecutionResult(BaseResult):
- """
- The execution specific result.
- The _inner_results list stores results of build targets in a given execution.
- Also stores the SUT node configuration.
- """
-
- sut_node: NodeConfiguration
- sut_os_name: str
- sut_os_version: str
- sut_kernel_version: str
-
- def __init__(self, sut_node: NodeConfiguration):
- super(ExecutionResult, self).__init__()
- self.sut_node = sut_node
-
- def add_build_target(self, build_target: BuildTargetConfiguration) -> BuildTargetResult:
- build_target_result = BuildTargetResult(build_target)
- self._inner_results.append(build_target_result)
- return build_target_result
-
- def add_sut_info(self, sut_info: NodeInfo):
- self.sut_os_name = sut_info.os_name
- self.sut_os_version = sut_info.os_version
- self.sut_kernel_version = sut_info.kernel_version
-
-
class DTSResult(BaseResult):
"""
Stores environment information and test results from a DTS run, which are:
@@ -269,25 +152,27 @@ class DTSResult(BaseResult):
"""
dpdk_version: str | None
+ _child_configs: list[ExecutionConfiguration]
_logger: DTSLOG
_errors: list[Exception]
_return_code: ErrorSeverity
- _stats_result: Statistics | None
+ _stats_result: Union["Statistics", None]
_stats_filename: str
- def __init__(self, logger: DTSLOG):
+ def __init__(self, configuration: Configuration, logger: DTSLOG):
super(DTSResult, self).__init__()
self.dpdk_version = None
+ self._child_configs = configuration.executions
self._logger = logger
self._errors = []
self._return_code = ErrorSeverity.NO_ERR
self._stats_result = None
self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
- def add_execution(self, sut_node: NodeConfiguration) -> ExecutionResult:
- execution_result = ExecutionResult(sut_node)
- self._inner_results.append(execution_result)
- return execution_result
+ def add_child_result(self, config: ExecutionConfiguration) -> "ExecutionResult":
+ result = ExecutionResult(config, self)
+ self._inner_results.append(result)
+ return result
def add_error(self, error) -> None:
self._errors.append(error)
@@ -325,3 +210,175 @@ def get_return_code(self) -> int:
self._return_code = error_return_code
return int(self._return_code)
+
+
+class ExecutionResult(BaseResult):
+ """
+ The execution specific result.
+ The _inner_results list stores results of build targets in a given execution.
+ Also stores the SUT node configuration.
+ """
+
+ sut_os_name: str
+ sut_os_version: str
+ sut_kernel_version: str
+ _config: ExecutionConfiguration
+ _parent_result: DTSResult
+ _child_configs: list[BuildTargetConfiguration]
+
+ def __init__(self, config: ExecutionConfiguration, parent_result: DTSResult):
+ super(ExecutionResult, self).__init__()
+ self._config = config
+ self._parent_result = parent_result
+ self._child_configs = config.build_targets
+
+ def add_sut_info(self, sut_info: NodeInfo):
+ self.sut_os_name = sut_info.os_name
+ self.sut_os_version = sut_info.os_version
+ self.sut_kernel_version = sut_info.kernel_version
+
+ def add_child_result(self, config: BuildTargetConfiguration) -> "BuildTargetResult":
+ result = BuildTargetResult(config, self)
+ self._inner_results.append(result)
+ return result
+
+
+class BuildTargetResult(BaseResult):
+ """
+ The build target specific result.
+ The _inner_results list stores results of test suites in a given build target.
+ Also stores build target specifics, such as compiler used to build DPDK.
+ """
+
+ arch: Architecture
+ os: OS
+ cpu: CPUType
+ compiler: Compiler
+ compiler_version: str | None
+ dpdk_version: str | None
+ _config: BuildTargetConfiguration
+ _parent_result: ExecutionResult
+ _child_configs: list[TestSuiteConfig]
+
+ def __init__(
+ self, config: BuildTargetConfiguration, parent_result: ExecutionResult
+ ):
+ super(BuildTargetResult, self).__init__()
+ self.arch = config.arch
+ self.os = config.os
+ self.cpu = config.cpu
+ self.compiler = config.compiler
+ self.compiler_version = None
+ self.dpdk_version = None
+ self._config = config
+ self._parent_result = parent_result
+ self._child_configs = parent_result._config.test_suites
+
+ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
+ self.compiler_version = versions.compiler_version
+ self.dpdk_version = versions.dpdk_version
+
+ def add_child_result(
+ self,
+ config: TestSuiteConfig,
+ ) -> "TestSuiteResult":
+ result = TestSuiteResult(config, self)
+ self._inner_results.append(result)
+ return result
+
+
+class TestSuiteResult(BaseResult):
+ """
+ The test suite specific result.
+ The _inner_results list stores results of test cases in a given test suite.
+ Also stores the test suite name.
+ """
+
+ _config: TestSuiteConfig
+ _parent_result: BuildTargetResult
+ _child_configs: list[str]
+
+ def __init__(self, config: TestSuiteConfig, parent_result: BuildTargetResult):
+ super(TestSuiteResult, self).__init__()
+ self._config = config
+ self._parent_result = parent_result
+ self._child_configs = config.test_cases
+
+ def add_child_result(self, config: str) -> "TestCaseResult":
+ result = TestCaseResult(config, self)
+ self._inner_results.append(result)
+ return result
+
+
+class TestCaseResult(BaseResult, FixtureResult):
+ """
+ The test case specific result.
+ Stores the result of the actual test case.
+ Also stores the test case name.
+ """
+
+ _config: str
+ _parent_result: TestSuiteResult
+
+ def __init__(self, config: str, parent_result: TestSuiteResult):
+ super(TestCaseResult, self).__init__()
+ self._config = config
+ self._parent_result = parent_result
+
+ def block(self):
+ self.update(Result.BLOCK)
+
+ def update(self, result: Result, error: Exception | None = None) -> None:
+ self.result = result
+ self.error = error
+
+ def _get_inner_errors(self) -> list[Exception]:
+ if self.error:
+ return [self.error]
+ return []
+
+ def add_stats(self, statistics: "Statistics") -> None:
+ statistics += self.result
+
+ def __bool__(self) -> bool:
+ return (
+ bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
+ )
+
+
+class Statistics(dict):
+ """
+ A helper class used to store the number of test cases by its result
+ along a few other basic information.
+ Using a dict provides a convenient way to format the data.
+ """
+
+ def __init__(self, dpdk_version: str | None):
+ super(Statistics, self).__init__()
+ for result in Result:
+ self[result.name] = 0
+ self["PASS RATE"] = 0.0
+ self["DPDK VERSION"] = dpdk_version
+
+ def __iadd__(self, other: Result) -> "Statistics":
+ """
+ Add a Result to the final count.
+ """
+ self[other.name] += 1
+ self["PASS RATE"] = (
+ float(self[Result.PASS.name])
+ * 100
+ / sum(self[result.name] for result in Result)
+ )
+ return self
+
+ def __str__(self) -> str:
+ """
+ Provide a string representation of the data.
+ """
+ stats_str = ""
+ for key, value in self.items():
+ stats_str += f"{key:<12} = {value}\n"
+ # according to docs, we should use \n when writing to text files
+ # on all platforms
+ return stats_str
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [RFC PATCH v1 5/5] dts: refactor logging configuration
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
` (3 preceding siblings ...)
2023-12-20 10:33 ` [RFC PATCH v1 4/5] dts: block all testcases when earlier setup fails Juraj Linkeš
@ 2023-12-20 10:33 ` Juraj Linkeš
2024-01-08 18:47 ` [RFC PATCH v1 0/5] test case blocking and logging Jeremy Spewock
` (3 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
yoan.picchi, Luca.Vizzarro
Cc: dev, Juraj Linkeš
Refactor logging for improved configuration and flexibility,
investigating unclear arguments and exploring alternatives
for logging test suites into separate files. In addition,
efforts were made to ensure that the modules remained
independent from the logger module, enabling potential use
by other consumers.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/logger.py | 162 ++++++++----------
dts/framework/remote_session/__init__.py | 4 +-
dts/framework/remote_session/os_session.py | 6 +-
.../remote_session/remote/__init__.py | 7 +-
.../remote/interactive_remote_session.py | 7 +-
.../remote/interactive_shell.py | 7 +-
.../remote_session/remote/remote_session.py | 8 +-
.../remote_session/remote/ssh_session.py | 5 +-
dts/framework/runner.py | 13 +-
dts/framework/test_result.py | 6 +-
dts/framework/test_suite.py | 6 +-
dts/framework/testbed_model/node.py | 10 +-
dts/framework/testbed_model/scapy.py | 6 +-
.../testbed_model/traffic_generator.py | 5 +-
dts/main.py | 6 +-
15 files changed, 124 insertions(+), 134 deletions(-)
diff --git a/dts/framework/logger.py b/dts/framework/logger.py
index bb2991e994..43c49c2d03 100644
--- a/dts/framework/logger.py
+++ b/dts/framework/logger.py
@@ -10,108 +10,98 @@
import logging
import os.path
-from typing import TypedDict
-
-from .settings import SETTINGS
+from enum import Enum
+from logging import FileHandler, StreamHandler
+from pathlib import Path
date_fmt = "%Y/%m/%d %H:%M:%S"
-stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s"
-class LoggerDictType(TypedDict):
- logger: "DTSLOG"
- name: str
- node: str
+def init_logger(verbose: bool, output_dir: str):
+ logging.raiseExceptions = False
+ DTSLog._output_dir = output_dir
+ logging.setLoggerClass(DTSLog)
-# List for saving all using loggers
-Loggers: list[LoggerDictType] = []
+ root_logger = logging.getLogger()
+ root_logger.setLevel(1)
+ sh = StreamHandler()
+ sh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
+ sh.setLevel(logging.INFO)
+ if verbose:
+ sh.setLevel(logging.DEBUG)
+ root_logger.addHandler(sh)
-class DTSLOG(logging.LoggerAdapter):
- """
- DTS log class for framework and testsuite.
- """
+ if not os.path.exists(output_dir):
+ os.mkdir(output_dir)
- _logger: logging.Logger
- node: str
- sh: logging.StreamHandler
- fh: logging.FileHandler
- verbose_fh: logging.FileHandler
+ add_file_handlers(Path(output_dir, "dts"))
- def __init__(self, logger: logging.Logger, node: str = "suite"):
- self._logger = logger
- # 1 means log everything, this will be used by file handlers if their level
- # is not set
- self._logger.setLevel(1)
- self.node = node
+def add_file_handlers(log_file_path: Path) -> list[FileHandler]:
+ root_logger = logging.getLogger()
- # add handler to emit to stdout
- sh = logging.StreamHandler()
- sh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
- sh.setLevel(logging.INFO) # console handler default level
+ fh = FileHandler(f"{log_file_path}.log")
+ fh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
+ root_logger.addHandler(fh)
- if SETTINGS.verbose is True:
- sh.setLevel(logging.DEBUG)
+ verbose_fh = FileHandler(f"{log_file_path}.verbose.log")
+ verbose_fh.setFormatter(
+ logging.Formatter(
+ "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
+ "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s",
+ datefmt=date_fmt,
+ )
+ )
+ root_logger.addHandler(verbose_fh)
- self._logger.addHandler(sh)
- self.sh = sh
+ return [fh, verbose_fh]
- # prepare the output folder
- if not os.path.exists(SETTINGS.output_dir):
- os.mkdir(SETTINGS.output_dir)
- logging_path_prefix = os.path.join(SETTINGS.output_dir, node)
+class DtsStage(Enum):
+ pre_execution = "pre-execution"
+ execution = "execution"
+ build_target = "build-target"
+ suite = "suite"
+ post_execution = "post-execution"
- fh = logging.FileHandler(f"{logging_path_prefix}.log")
- fh.setFormatter(
- logging.Formatter(
- fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
- datefmt=date_fmt,
- )
- )
+ def __str__(self) -> str:
+ return self.value
- self._logger.addHandler(fh)
- self.fh = fh
-
- # This outputs EVERYTHING, intended for post-mortem debugging
- # Also optimized for processing via AWK (awk -F '|' ...)
- verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log")
- verbose_fh.setFormatter(
- logging.Formatter(
- fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
- "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s",
- datefmt=date_fmt,
- )
- )
- self._logger.addHandler(verbose_fh)
- self.verbose_fh = verbose_fh
-
- super(DTSLOG, self).__init__(self._logger, dict(node=self.node))
-
- def logger_exit(self) -> None:
- """
- Remove stream handler and logfile handler.
- """
- for handler in (self.sh, self.fh, self.verbose_fh):
- handler.flush()
- self._logger.removeHandler(handler)
-
-
-def getLogger(name: str, node: str = "suite") -> DTSLOG:
- """
- Get logger handler and if there's no handler for specified Node will create one.
- """
- global Loggers
- # return saved logger
- logger: LoggerDictType
- for logger in Loggers:
- if logger["name"] == name and logger["node"] == node:
- return logger["logger"]
-
- # return new logger
- dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node)
- Loggers.append({"logger": dts_logger, "name": name, "node": node})
- return dts_logger
+class DTSLog(logging.Logger):
+ _stage: DtsStage = DtsStage.pre_execution
+ _extra_file_handlers: list[FileHandler] = []
+ _output_dir: None | str = None
+
+ def makeRecord(self, *args, **kwargs):
+ record = super().makeRecord(*args, **kwargs)
+ record.stage = DTSLog._stage
+ return record
+
+ def set_stage(self, stage: DtsStage, log_file_name: str | None = None):
+ self._remove_extra_file_handlers()
+
+ if DTSLog._stage != stage:
+ self.info(f"Moving from stage '{DTSLog._stage}' to stage '{stage}'.")
+ DTSLog._stage = stage
+
+ if log_file_name:
+ if DTSLog._output_dir:
+ DTSLog._extra_file_handlers.extend(
+ add_file_handlers(Path(DTSLog._output_dir, log_file_name))
+ )
+ else:
+ self.warning(
+ f"Cannot log '{DTSLog._stage}' stage in separate file, "
+ "output dir is not defined."
+ )
+
+ def _remove_extra_file_handlers(self) -> None:
+ if DTSLog._extra_file_handlers:
+ for extra_file_handler in DTSLog._extra_file_handlers:
+ self.root.removeHandler(extra_file_handler)
+
+ DTSLog._extra_file_handlers = []
diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
index 6124417bd7..a4ec2f40ae 100644
--- a/dts/framework/remote_session/__init__.py
+++ b/dts/framework/remote_session/__init__.py
@@ -11,10 +11,10 @@
"""
# pylama:ignore=W0611
+import logging
from framework.config import OS, NodeConfiguration
from framework.exception import ConfigurationError
-from framework.logger import DTSLOG
from .linux_session import LinuxSession
from .os_session import InteractiveShellType, OSSession
@@ -30,7 +30,7 @@
)
-def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession:
+def create_session(node_config: NodeConfiguration, name: str, logger: logging.Logger) -> OSSession:
match node_config.os:
case OS.linux:
return LinuxSession(node_config, name, logger)
diff --git a/dts/framework/remote_session/os_session.py b/dts/framework/remote_session/os_session.py
index 8a709eac1c..2524a4e669 100644
--- a/dts/framework/remote_session/os_session.py
+++ b/dts/framework/remote_session/os_session.py
@@ -2,6 +2,7 @@
# Copyright(c) 2023 PANTHEON.tech s.r.o.
# Copyright(c) 2023 University of New Hampshire
+import logging
from abc import ABC, abstractmethod
from collections.abc import Iterable
from ipaddress import IPv4Interface, IPv6Interface
@@ -9,7 +10,6 @@
from typing import Type, TypeVar, Union
from framework.config import Architecture, NodeConfiguration, NodeInfo
-from framework.logger import DTSLOG
from framework.remote_session.remote import InteractiveShell
from framework.settings import SETTINGS
from framework.testbed_model import LogicalCore
@@ -36,7 +36,7 @@ class OSSession(ABC):
_config: NodeConfiguration
name: str
- _logger: DTSLOG
+ _logger: logging.Logger
remote_session: RemoteSession
interactive_session: InteractiveRemoteSession
@@ -44,7 +44,7 @@ def __init__(
self,
node_config: NodeConfiguration,
name: str,
- logger: DTSLOG,
+ logger: logging.Logger,
):
self._config = node_config
self.name = name
diff --git a/dts/framework/remote_session/remote/__init__.py b/dts/framework/remote_session/remote/__init__.py
index 06403691a5..4a22155153 100644
--- a/dts/framework/remote_session/remote/__init__.py
+++ b/dts/framework/remote_session/remote/__init__.py
@@ -4,8 +4,9 @@
# pylama:ignore=W0611
+import logging
+
from framework.config import NodeConfiguration
-from framework.logger import DTSLOG
from .interactive_remote_session import InteractiveRemoteSession
from .interactive_shell import InteractiveShell
@@ -16,12 +17,12 @@
def create_remote_session(
- node_config: NodeConfiguration, name: str, logger: DTSLOG
+ node_config: NodeConfiguration, name: str, logger: logging.Logger
) -> RemoteSession:
return SSHSession(node_config, name, logger)
def create_interactive_session(
- node_config: NodeConfiguration, logger: DTSLOG
+ node_config: NodeConfiguration, logger: logging.Logger
) -> InteractiveRemoteSession:
return InteractiveRemoteSession(node_config, logger)
diff --git a/dts/framework/remote_session/remote/interactive_remote_session.py b/dts/framework/remote_session/remote/interactive_remote_session.py
index 098ded1bb0..bf0996a747 100644
--- a/dts/framework/remote_session/remote/interactive_remote_session.py
+++ b/dts/framework/remote_session/remote/interactive_remote_session.py
@@ -2,7 +2,7 @@
# Copyright(c) 2023 University of New Hampshire
"""Handler for an SSH session dedicated to interactive shells."""
-
+import logging
import socket
import traceback
@@ -16,7 +16,6 @@
from framework.config import NodeConfiguration
from framework.exception import SSHConnectionError
-from framework.logger import DTSLOG
class InteractiveRemoteSession:
@@ -54,11 +53,11 @@ class InteractiveRemoteSession:
username: str
password: str
session: SSHClient
- _logger: DTSLOG
+ _logger: logging.Logger
_node_config: NodeConfiguration
_transport: Transport | None
- def __init__(self, node_config: NodeConfiguration, _logger: DTSLOG) -> None:
+ def __init__(self, node_config: NodeConfiguration, _logger: logging.Logger) -> None:
self._node_config = node_config
self._logger = _logger
self.hostname = node_config.hostname
diff --git a/dts/framework/remote_session/remote/interactive_shell.py b/dts/framework/remote_session/remote/interactive_shell.py
index 4db19fb9b3..b6074838c2 100644
--- a/dts/framework/remote_session/remote/interactive_shell.py
+++ b/dts/framework/remote_session/remote/interactive_shell.py
@@ -11,14 +11,13 @@
elevated privileges to start it is expected that the method for gaining those
privileges is provided when initializing the class.
"""
-
+import logging
from abc import ABC
from pathlib import PurePath
from typing import Callable
from paramiko import Channel, SSHClient, channel # type: ignore[import]
-from framework.logger import DTSLOG
from framework.settings import SETTINGS
@@ -58,7 +57,7 @@ class InteractiveShell(ABC):
_stdin: channel.ChannelStdinFile
_stdout: channel.ChannelFile
_ssh_channel: Channel
- _logger: DTSLOG
+ _logger: logging.Logger
_timeout: float
_app_args: str
_default_prompt: str = ""
@@ -69,7 +68,7 @@ class InteractiveShell(ABC):
def __init__(
self,
interactive_session: SSHClient,
- logger: DTSLOG,
+ logger: logging.Logger,
get_privileged_command: Callable[[str], str] | None,
app_args: str = "",
timeout: float = SETTINGS.timeout,
diff --git a/dts/framework/remote_session/remote/remote_session.py b/dts/framework/remote_session/remote/remote_session.py
index 719f7d1ef7..da78e5c921 100644
--- a/dts/framework/remote_session/remote/remote_session.py
+++ b/dts/framework/remote_session/remote/remote_session.py
@@ -2,14 +2,13 @@
# Copyright(c) 2010-2014 Intel Corporation
# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
# Copyright(c) 2022-2023 University of New Hampshire
-
import dataclasses
+import logging
from abc import ABC, abstractmethod
from pathlib import PurePath
from framework.config import NodeConfiguration
from framework.exception import RemoteCommandExecutionError
-from framework.logger import DTSLOG
from framework.settings import SETTINGS
@@ -50,14 +49,14 @@ class RemoteSession(ABC):
username: str
password: str
history: list[CommandResult]
- _logger: DTSLOG
+ _logger: logging.Logger
_node_config: NodeConfiguration
def __init__(
self,
node_config: NodeConfiguration,
session_name: str,
- logger: DTSLOG,
+ logger: logging.Logger,
):
self._node_config = node_config
@@ -120,7 +119,6 @@ def close(self, force: bool = False) -> None:
"""
Close the remote session and free all used resources.
"""
- self._logger.logger_exit()
self._close(force)
@abstractmethod
diff --git a/dts/framework/remote_session/remote/ssh_session.py b/dts/framework/remote_session/remote/ssh_session.py
index 1a7ee649ab..42441c4587 100644
--- a/dts/framework/remote_session/remote/ssh_session.py
+++ b/dts/framework/remote_session/remote/ssh_session.py
@@ -1,6 +1,6 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2023 PANTHEON.tech s.r.o.
-
+import logging
import socket
import traceback
from pathlib import PurePath
@@ -20,7 +20,6 @@
from framework.config import NodeConfiguration
from framework.exception import SSHConnectionError, SSHSessionDeadError, SSHTimeoutError
-from framework.logger import DTSLOG
from .remote_session import CommandResult, RemoteSession
@@ -49,7 +48,7 @@ def __init__(
self,
node_config: NodeConfiguration,
session_name: str,
- logger: DTSLOG,
+ logger: logging.Logger,
):
super(SSHSession, self).__init__(node_config, session_name, logger)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 28570d4a1c..5c06e4ca1a 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -11,6 +11,7 @@
from copy import deepcopy
from dataclasses import dataclass
from types import MethodType, ModuleType
+from typing import cast
from .config import (
BuildTargetConfiguration,
@@ -24,7 +25,7 @@
SSHTimeoutError,
TestCaseVerifyError,
)
-from .logger import DTSLOG, getLogger
+from .logger import DTSLog, DtsStage
from .settings import SETTINGS
from .test_result import (
BuildTargetResult,
@@ -68,12 +69,12 @@ def processed_config(self) -> ExecutionConfiguration:
class DTSRunner:
- _logger: DTSLOG
+ _logger: DTSLog
_result: DTSResult
_executions: list[Execution]
def __init__(self, configuration: Configuration):
- self._logger = getLogger("DTSRunner")
+ self._logger = cast(DTSLog, logging.getLogger("DTSRunner"))
self._result = DTSResult(configuration, self._logger)
self._executions = create_executions(configuration.executions)
@@ -146,6 +147,7 @@ def _run_execution(
Run the given execution. This involves running the execution setup as well as
running all build targets in the given execution.
"""
+ self._logger.set_stage(DtsStage.execution)
self._logger.info(
"Running execution with SUT "
f"'{execution.config.system_under_test_node.name}'."
@@ -175,6 +177,7 @@ def _run_execution(
sut_node.tear_down_execution()
execution_result.update_teardown(Result.PASS)
except Exception as e:
+ self._logger.set_stage(DtsStage.execution)
self._logger.exception("Execution teardown failed.")
execution_result.update_teardown(Result.FAIL, e)
@@ -189,6 +192,7 @@ def _run_build_target(
"""
Run the given build target.
"""
+ self._logger.set_stage(DtsStage.build_target)
self._logger.info(f"Running build target '{build_target.name}'.")
build_target_result = execution_result.add_child_result(build_target)
@@ -209,6 +213,7 @@ def _run_build_target(
sut_node.tear_down_build_target()
build_target_result.update_teardown(Result.PASS)
except Exception as e:
+ self._logger.set_stage(DtsStage.build_target)
self._logger.exception("Build target teardown failed.")
build_target_result.update_teardown(Result.FAIL, e)
@@ -265,6 +270,7 @@ def _run_test_suite(
"""
test_suite = test_suite_setup.test_suite(sut_node, tg_node)
test_suite_name = test_suite_setup.test_suite.__name__
+ self._logger.set_stage(DtsStage.suite, test_suite_name)
test_suite_result = build_target_result.add_child_result(
test_suite_setup.processed_config()
)
@@ -397,6 +403,7 @@ def _exit_dts(self) -> None:
self._result.process()
if self._logger:
+ self._logger.set_stage(DtsStage.post_execution)
self._logger.info("DTS execution has ended.")
logging.shutdown()
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index dba2c55d36..221e75205e 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -6,6 +6,7 @@
Generic result container and reporters
"""
+import logging
import os.path
from collections.abc import MutableSequence
from enum import Enum, auto
@@ -24,7 +25,6 @@
TestSuiteConfig,
)
from .exception import DTSError, ErrorSeverity
-from .logger import DTSLOG
from .settings import SETTINGS
@@ -153,13 +153,13 @@ class DTSResult(BaseResult):
dpdk_version: str | None
_child_configs: list[ExecutionConfiguration]
- _logger: DTSLOG
+ _logger: logging.Logger
_errors: list[Exception]
_return_code: ErrorSeverity
_stats_result: Union["Statistics", None]
_stats_filename: str
- def __init__(self, configuration: Configuration, logger: DTSLOG):
+ def __init__(self, configuration: Configuration, logger: logging.Logger):
super(DTSResult, self).__init__()
self.dpdk_version = None
self._child_configs = configuration.executions
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index e73206993d..9c9a8c1e08 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -6,6 +6,7 @@
Base class for creating DTS test cases.
"""
+import logging
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
from typing import Union
@@ -14,7 +15,6 @@
from scapy.packet import Packet, Padding # type: ignore[import]
from .exception import TestCaseVerifyError
-from .logger import DTSLOG, getLogger
from .testbed_model import SutNode, TGNode
from .testbed_model.hw.port import Port, PortLink
from .utils import get_packet_summaries
@@ -41,7 +41,7 @@ class TestSuite(object):
sut_node: SutNode
tg_node: TGNode
is_blocking = False
- _logger: DTSLOG
+ _logger: logging.Logger
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -59,7 +59,7 @@ def __init__(
):
self.sut_node = sut_node
self.tg_node = tg_node
- self._logger = getLogger(self.__class__.__name__)
+ self._logger = logging.getLogger(self.__class__.__name__)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
index ef700d8114..a98c58df4f 100644
--- a/dts/framework/testbed_model/node.py
+++ b/dts/framework/testbed_model/node.py
@@ -6,7 +6,7 @@
"""
A node is a generic host that DTS connects to and manages.
"""
-
+import logging
from abc import ABC
from ipaddress import IPv4Interface, IPv6Interface
from typing import Any, Callable, Type, Union
@@ -16,7 +16,6 @@
ExecutionConfiguration,
NodeConfiguration,
)
-from framework.logger import DTSLOG, getLogger
from framework.remote_session import InteractiveShellType, OSSession, create_session
from framework.settings import SETTINGS
@@ -43,7 +42,7 @@ class Node(ABC):
name: str
lcores: list[LogicalCore]
ports: list[Port]
- _logger: DTSLOG
+ _logger: logging.Logger
_other_sessions: list[OSSession]
_execution_config: ExecutionConfiguration
virtual_devices: list[VirtualDevice]
@@ -51,7 +50,7 @@ class Node(ABC):
def __init__(self, node_config: NodeConfiguration):
self.config = node_config
self.name = node_config.name
- self._logger = getLogger(self.name)
+ self._logger = logging.getLogger(self.name)
self.main_session = create_session(self.config, self.name, self._logger)
self._logger.info(f"Connected to node: {self.name}")
@@ -137,7 +136,7 @@ def create_session(self, name: str) -> OSSession:
connection = create_session(
self.config,
session_name,
- getLogger(session_name, node=self.name),
+ logging.getLogger(session_name),
)
self._other_sessions.append(connection)
return connection
@@ -237,7 +236,6 @@ def close(self) -> None:
self.main_session.close()
for session in self._other_sessions:
session.close()
- self._logger.logger_exit()
@staticmethod
def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
diff --git a/dts/framework/testbed_model/scapy.py b/dts/framework/testbed_model/scapy.py
index 9083e92b3d..61058cd38a 100644
--- a/dts/framework/testbed_model/scapy.py
+++ b/dts/framework/testbed_model/scapy.py
@@ -13,6 +13,7 @@
"""
import inspect
+import logging
import marshal
import time
import types
@@ -24,7 +25,6 @@
from scapy.packet import Packet # type: ignore[import]
from framework.config import OS, ScapyTrafficGeneratorConfig
-from framework.logger import DTSLOG, getLogger
from framework.remote_session import PythonShell
from framework.settings import SETTINGS
@@ -190,12 +190,12 @@ class ScapyTrafficGenerator(CapturingTrafficGenerator):
rpc_server_proxy: xmlrpc.client.ServerProxy
_config: ScapyTrafficGeneratorConfig
_tg_node: TGNode
- _logger: DTSLOG
+ _logger: logging.Logger
def __init__(self, tg_node: TGNode, config: ScapyTrafficGeneratorConfig):
self._config = config
self._tg_node = tg_node
- self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
+ self._logger = logging.getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
assert (
self._tg_node.config.os == OS.linux
diff --git a/dts/framework/testbed_model/traffic_generator.py b/dts/framework/testbed_model/traffic_generator.py
index 28c35d3ce4..6b0838958a 100644
--- a/dts/framework/testbed_model/traffic_generator.py
+++ b/dts/framework/testbed_model/traffic_generator.py
@@ -7,12 +7,11 @@
These traffic generators can't capture received traffic,
only count the number of received packets.
"""
-
+import logging
from abc import ABC, abstractmethod
from scapy.packet import Packet # type: ignore[import]
-from framework.logger import DTSLOG
from framework.utils import get_packet_summaries
from .hw.port import Port
@@ -24,7 +23,7 @@ class TrafficGenerator(ABC):
Defines the few basic methods that each traffic generator must implement.
"""
- _logger: DTSLOG
+ _logger: logging.Logger
def send_packet(self, packet: Packet, port: Port) -> None:
"""Send a packet and block until it is fully sent.
diff --git a/dts/main.py b/dts/main.py
index 879ce5cb89..f2828148f0 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -8,18 +8,18 @@
A test framework for testing DPDK.
"""
-import logging
-
from framework.config import load_config
+from framework.logger import init_logger
from framework.runner import DTSRunner
+from framework.settings import SETTINGS
def main() -> None:
+ init_logger(SETTINGS.verbose, SETTINGS.output_dir)
dts = DTSRunner(configuration=load_config())
dts.run()
# Main program begins here
if __name__ == "__main__":
- logging.raiseExceptions = True
main()
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [RFC PATCH v1 0/5] test case blocking and logging
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
` (4 preceding siblings ...)
2023-12-20 10:33 ` [RFC PATCH v1 5/5] dts: refactor logging configuration Juraj Linkeš
@ 2024-01-08 18:47 ` Jeremy Spewock
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
` (2 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Jeremy Spewock @ 2024-01-08 18:47 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
yoan.picchi, Luca.Vizzarro, dev
[-- Attachment #1: Type: text/plain, Size: 85 bytes --]
Definitely worth-while changes. I looked them over and it all looks good so
far.
+1
[-- Attachment #2: Type: text/html, Size: 323 bytes --]
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 0/7] test case blocking and logging
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
` (5 preceding siblings ...)
2024-01-08 18:47 ` [RFC PATCH v1 0/5] test case blocking and logging Jeremy Spewock
@ 2024-02-06 14:57 ` Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 1/7] dts: convert dts.py methods to class Juraj Linkeš
` (6 more replies)
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
8 siblings, 7 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
We currently don't record test case results that couldn't be executed
because of a previous failure, such as when a test suite setup failed,
resulting in no executed test cases.
In order to record the test cases that couldn't be executed, we must
know the lists of test suites and test cases ahead of the actual test
suite execution, as an error could occur before we even start executing
test suites.
In addition, the patch series contains two refactors and one
improvement.
The first refactor is closely related. The dts.py was renamed to
runner.py and given a clear purpose - running the test suites and all
other orchestration needed to run test suites. The logic for this was
not all in the original dts.py module and it was brought there. The
runner is also responsible for recording results, which is the blocked
test cases are recorded.
The other refactor, logging, is related to the first refactor. The
logging module was simplified while extending capabilities. Each test
suite logs into its own log file in addition to the main log file which
the runner must handle (as it knows when we start executing particular
test suites). The runner also handles the switching between execution
stages for the purposes of logging.
The one aforementioned improvement is in unifying how we specify test
cases in the conf.yaml file and in the environment variable/command line
argument.
v2:
Rebase and update of the whole patch. There are changes in all parts of
the code, mainly improving the design and logic.
Also added the last patch which improves test suite specification on the
cmdline.
Juraj Linkeš (7):
dts: convert dts.py methods to class
dts: move test suite execution logic to DTSRunner
dts: filter test suites in executions
dts: reorganize test result
dts: block all test cases when earlier setup fails
dts: refactor logging configuration
dts: improve test suite and case filtering
doc/guides/tools/dts.rst | 14 +-
dts/framework/config/__init__.py | 36 +-
dts/framework/config/conf_yaml_schema.json | 2 +-
dts/framework/dts.py | 338 ---------
dts/framework/logger.py | 235 +++---
dts/framework/remote_session/__init__.py | 6 +-
.../interactive_remote_session.py | 6 +-
.../remote_session/interactive_shell.py | 6 +-
.../remote_session/remote_session.py | 8 +-
dts/framework/runner.py | 701 ++++++++++++++++++
dts/framework/settings.py | 188 +++--
dts/framework/test_result.py | 565 ++++++++------
dts/framework/test_suite.py | 239 +-----
dts/framework/testbed_model/node.py | 11 +-
dts/framework/testbed_model/os_session.py | 7 +-
.../traffic_generator/traffic_generator.py | 6 +-
dts/main.py | 9 +-
dts/pyproject.toml | 3 +
dts/tests/TestSuite_smoke_tests.py | 2 +-
19 files changed, 1354 insertions(+), 1028 deletions(-)
delete mode 100644 dts/framework/dts.py
create mode 100644 dts/framework/runner.py
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 1/7] dts: convert dts.py methods to class
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
@ 2024-02-06 14:57 ` Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš
` (5 subsequent siblings)
6 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
The dts.py module deviates from the rest of the code without a clear
reason. Converting it into a class and using better naming will improve
organization and code readability.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/dts.py | 338 ----------------------------------------
dts/framework/runner.py | 333 +++++++++++++++++++++++++++++++++++++++
dts/main.py | 6 +-
3 files changed, 337 insertions(+), 340 deletions(-)
delete mode 100644 dts/framework/dts.py
create mode 100644 dts/framework/runner.py
diff --git a/dts/framework/dts.py b/dts/framework/dts.py
deleted file mode 100644
index e16d4578a0..0000000000
--- a/dts/framework/dts.py
+++ /dev/null
@@ -1,338 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2019 Intel Corporation
-# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
-# Copyright(c) 2022-2023 University of New Hampshire
-
-r"""Test suite runner module.
-
-A DTS run is split into stages:
-
- #. Execution stage,
- #. Build target stage,
- #. Test suite stage,
- #. Test case stage.
-
-The module is responsible for running tests on testbeds defined in the test run configuration.
-Each setup or teardown of each stage is recorded in a :class:`~.test_result.DTSResult` or
-one of its subclasses. The test case results are also recorded.
-
-If an error occurs, the current stage is aborted, the error is recorded and the run continues in
-the next iteration of the same stage. The return code is the highest `severity` of all
-:class:`~.exception.DTSError`\s.
-
-Example:
- An error occurs in a build target setup. The current build target is aborted and the run
- continues with the next build target. If the errored build target was the last one in the given
- execution, the next execution begins.
-
-Attributes:
- dts_logger: The logger instance used in this module.
- result: The top level result used in the module.
-"""
-
-import sys
-
-from .config import (
- BuildTargetConfiguration,
- ExecutionConfiguration,
- TestSuiteConfig,
- load_config,
-)
-from .exception import BlockingTestSuiteError
-from .logger import DTSLOG, getLogger
-from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
-from .test_suite import get_test_suites
-from .testbed_model import SutNode, TGNode
-
-# dummy defaults to satisfy linters
-dts_logger: DTSLOG = None # type: ignore[assignment]
-result: DTSResult = DTSResult(dts_logger)
-
-
-def run_all() -> None:
- """Run all build targets in all executions from the test run configuration.
-
- Before running test suites, executions and build targets are first set up.
- The executions and build targets defined in the test run configuration are iterated over.
- The executions define which tests to run and where to run them and build targets define
- the DPDK build setup.
-
- The tests suites are set up for each execution/build target tuple and each scheduled
- test case within the test suite is set up, executed and torn down. After all test cases
- have been executed, the test suite is torn down and the next build target will be tested.
-
- All the nested steps look like this:
-
- #. Execution setup
-
- #. Build target setup
-
- #. Test suite setup
-
- #. Test case setup
- #. Test case logic
- #. Test case teardown
-
- #. Test suite teardown
-
- #. Build target teardown
-
- #. Execution teardown
-
- The test cases are filtered according to the specification in the test run configuration and
- the :option:`--test-cases` command line argument or
- the :envvar:`DTS_TESTCASES` environment variable.
- """
- global dts_logger
- global result
-
- # create a regular DTS logger and create a new result with it
- dts_logger = getLogger("DTSRunner")
- result = DTSResult(dts_logger)
-
- # check the python version of the server that run dts
- _check_dts_python_version()
-
- sut_nodes: dict[str, SutNode] = {}
- tg_nodes: dict[str, TGNode] = {}
- try:
- # for all Execution sections
- for execution in load_config().executions:
- sut_node = sut_nodes.get(execution.system_under_test_node.name)
- tg_node = tg_nodes.get(execution.traffic_generator_node.name)
-
- try:
- if not sut_node:
- sut_node = SutNode(execution.system_under_test_node)
- sut_nodes[sut_node.name] = sut_node
- if not tg_node:
- tg_node = TGNode(execution.traffic_generator_node)
- tg_nodes[tg_node.name] = tg_node
- result.update_setup(Result.PASS)
- except Exception as e:
- failed_node = execution.system_under_test_node.name
- if sut_node:
- failed_node = execution.traffic_generator_node.name
- dts_logger.exception(f"Creation of node {failed_node} failed.")
- result.update_setup(Result.FAIL, e)
-
- else:
- _run_execution(sut_node, tg_node, execution, result)
-
- except Exception as e:
- dts_logger.exception("An unexpected error has occurred.")
- result.add_error(e)
- raise
-
- finally:
- try:
- for node in (sut_nodes | tg_nodes).values():
- node.close()
- result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Final cleanup of nodes failed.")
- result.update_teardown(Result.ERROR, e)
-
- # we need to put the sys.exit call outside the finally clause to make sure
- # that unexpected exceptions will propagate
- # in that case, the error that should be reported is the uncaught exception as
- # that is a severe error originating from the framework
- # at that point, we'll only have partial results which could be impacted by the
- # error causing the uncaught exception, making them uninterpretable
- _exit_dts()
-
-
-def _check_dts_python_version() -> None:
- """Check the required Python version - v3.10."""
-
- def RED(text: str) -> str:
- return f"\u001B[31;1m{str(text)}\u001B[0m"
-
- if sys.version_info.major < 3 or (sys.version_info.major == 3 and sys.version_info.minor < 10):
- print(
- RED(
- (
- "WARNING: DTS execution node's python version is lower than"
- "python 3.10, is deprecated and will not work in future releases."
- )
- ),
- file=sys.stderr,
- )
- print(RED("Please use Python >= 3.10 instead"), file=sys.stderr)
-
-
-def _run_execution(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- result: DTSResult,
-) -> None:
- """Run the given execution.
-
- This involves running the execution setup as well as running all build targets
- in the given execution. After that, execution teardown is run.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- execution: An execution's test run configuration.
- result: The top level result object.
- """
- dts_logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
- execution_result = result.add_execution(sut_node.config)
- execution_result.add_sut_info(sut_node.node_info)
-
- try:
- sut_node.set_up_execution(execution)
- execution_result.update_setup(Result.PASS)
- except Exception as e:
- dts_logger.exception("Execution setup failed.")
- execution_result.update_setup(Result.FAIL, e)
-
- else:
- for build_target in execution.build_targets:
- _run_build_target(sut_node, tg_node, build_target, execution, execution_result)
-
- finally:
- try:
- sut_node.tear_down_execution()
- execution_result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Execution teardown failed.")
- execution_result.update_teardown(Result.FAIL, e)
-
-
-def _run_build_target(
- sut_node: SutNode,
- tg_node: TGNode,
- build_target: BuildTargetConfiguration,
- execution: ExecutionConfiguration,
- execution_result: ExecutionResult,
-) -> None:
- """Run the given build target.
-
- This involves running the build target setup as well as running all test suites
- in the given execution the build target is defined in.
- After that, build target teardown is run.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- build_target: A build target's test run configuration.
- execution: The build target's execution's test run configuration.
- execution_result: The execution level result object associated with the execution.
- """
- dts_logger.info(f"Running build target '{build_target.name}'.")
- build_target_result = execution_result.add_build_target(build_target)
-
- try:
- sut_node.set_up_build_target(build_target)
- result.dpdk_version = sut_node.dpdk_version
- build_target_result.add_build_target_info(sut_node.get_build_target_info())
- build_target_result.update_setup(Result.PASS)
- except Exception as e:
- dts_logger.exception("Build target setup failed.")
- build_target_result.update_setup(Result.FAIL, e)
-
- else:
- _run_all_suites(sut_node, tg_node, execution, build_target_result)
-
- finally:
- try:
- sut_node.tear_down_build_target()
- build_target_result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Build target teardown failed.")
- build_target_result.update_teardown(Result.FAIL, e)
-
-
-def _run_all_suites(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
-) -> None:
- """Run the execution's (possibly a subset) test suites using the current build target.
-
- The function assumes the build target we're testing has already been built on the SUT node.
- The current build target thus corresponds to the current DPDK build present on the SUT node.
-
- If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
- in the current build target won't be executed.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- execution: The execution's test run configuration associated with the current build target.
- build_target_result: The build target level result object associated
- with the current build target.
- """
- end_build_target = False
- if not execution.skip_smoke_tests:
- execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
- for test_suite_config in execution.test_suites:
- try:
- _run_single_suite(sut_node, tg_node, execution, build_target_result, test_suite_config)
- except BlockingTestSuiteError as e:
- dts_logger.exception(
- f"An error occurred within {test_suite_config.test_suite}. Skipping build target."
- )
- result.add_error(e)
- end_build_target = True
- # if a blocking test failed and we need to bail out of suite executions
- if end_build_target:
- break
-
-
-def _run_single_suite(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
- test_suite_config: TestSuiteConfig,
-) -> None:
- """Run all test suite in a single test suite module.
-
- The function assumes the build target we're testing has already been built on the SUT node.
- The current build target thus corresponds to the current DPDK build present on the SUT node.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- execution: The execution's test run configuration associated with the current build target.
- build_target_result: The build target level result object associated
- with the current build target.
- test_suite_config: Test suite test run configuration specifying the test suite module
- and possibly a subset of test cases of test suites in that module.
-
- Raises:
- BlockingTestSuiteError: If a blocking test suite fails.
- """
- try:
- full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
- test_suite_classes = get_test_suites(full_suite_path)
- suites_str = ", ".join((x.__name__ for x in test_suite_classes))
- dts_logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
- except Exception as e:
- dts_logger.exception("An error occurred when searching for test suites.")
- result.update_setup(Result.ERROR, e)
-
- else:
- for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(
- sut_node,
- tg_node,
- test_suite_config.test_cases,
- execution.func,
- build_target_result,
- )
- test_suite.run()
-
-
-def _exit_dts() -> None:
- """Process all errors and exit with the proper exit code."""
- result.process()
-
- if dts_logger:
- dts_logger.info("DTS execution has ended.")
- sys.exit(result.get_return_code())
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
new file mode 100644
index 0000000000..acc1c4d6db
--- /dev/null
+++ b/dts/framework/runner.py
@@ -0,0 +1,333 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2019 Intel Corporation
+# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
+# Copyright(c) 2022-2023 University of New Hampshire
+
+"""Test suite runner module.
+
+The module is responsible for running DTS in a series of stages:
+
+ #. Execution stage,
+ #. Build target stage,
+ #. Test suite stage,
+ #. Test case stage.
+
+The execution and build target stages set up the environment before running test suites.
+The test suite stage sets up steps common to all test cases
+and the test case stage runs test cases individually.
+"""
+
+import logging
+import sys
+
+from .config import (
+ BuildTargetConfiguration,
+ ExecutionConfiguration,
+ TestSuiteConfig,
+ load_config,
+)
+from .exception import BlockingTestSuiteError
+from .logger import DTSLOG, getLogger
+from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
+from .test_suite import get_test_suites
+from .testbed_model import SutNode, TGNode
+
+
+class DTSRunner:
+ r"""Test suite runner class.
+
+ The class is responsible for running tests on testbeds defined in the test run configuration.
+ Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult`
+ or one of its subclasses. The test case results are also recorded.
+
+ If an error occurs, the current stage is aborted, the error is recorded and the run continues in
+ the next iteration of the same stage. The return code is the highest `severity` of all
+ :class:`~.framework.exception.DTSError`\s.
+
+ Example:
+ An error occurs in a build target setup. The current build target is aborted and the run
+ continues with the next build target. If the errored build target was the last one in the
+ given execution, the next execution begins.
+ """
+
+ _logger: DTSLOG
+ _result: DTSResult
+
+ def __init__(self):
+ """Initialize the instance with logger and result."""
+ self._logger = getLogger("DTSRunner")
+ self._result = DTSResult(self._logger)
+
+ def run(self):
+ """Run all build targets in all executions from the test run configuration.
+
+ Before running test suites, executions and build targets are first set up.
+ The executions and build targets defined in the test run configuration are iterated over.
+ The executions define which tests to run and where to run them and build targets define
+ the DPDK build setup.
+
+ The tests suites are set up for each execution/build target tuple and each discovered
+ test case within the test suite is set up, executed and torn down. After all test cases
+ have been executed, the test suite is torn down and the next build target will be tested.
+
+ All the nested steps look like this:
+
+ #. Execution setup
+
+ #. Build target setup
+
+ #. Test suite setup
+
+ #. Test case setup
+ #. Test case logic
+ #. Test case teardown
+
+ #. Test suite teardown
+
+ #. Build target teardown
+
+ #. Execution teardown
+
+ The test cases are filtered according to the specification in the test run configuration and
+ the :option:`--test-cases` command line argument or
+ the :envvar:`DTS_TESTCASES` environment variable.
+ """
+ sut_nodes: dict[str, SutNode] = {}
+ tg_nodes: dict[str, TGNode] = {}
+ try:
+ # check the python version of the server that runs dts
+ self._check_dts_python_version()
+
+ # for all Execution sections
+ for execution in load_config().executions:
+ sut_node = sut_nodes.get(execution.system_under_test_node.name)
+ tg_node = tg_nodes.get(execution.traffic_generator_node.name)
+
+ try:
+ if not sut_node:
+ sut_node = SutNode(execution.system_under_test_node)
+ sut_nodes[sut_node.name] = sut_node
+ if not tg_node:
+ tg_node = TGNode(execution.traffic_generator_node)
+ tg_nodes[tg_node.name] = tg_node
+ self._result.update_setup(Result.PASS)
+ except Exception as e:
+ failed_node = execution.system_under_test_node.name
+ if sut_node:
+ failed_node = execution.traffic_generator_node.name
+ self._logger.exception(f"The Creation of node {failed_node} failed.")
+ self._result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_execution(sut_node, tg_node, execution)
+
+ except Exception as e:
+ self._logger.exception("An unexpected error has occurred.")
+ self._result.add_error(e)
+ raise
+
+ finally:
+ try:
+ for node in (sut_nodes | tg_nodes).values():
+ node.close()
+ self._result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("The final cleanup of nodes failed.")
+ self._result.update_teardown(Result.ERROR, e)
+
+ # we need to put the sys.exit call outside the finally clause to make sure
+ # that unexpected exceptions will propagate
+ # in that case, the error that should be reported is the uncaught exception as
+ # that is a severe error originating from the framework
+ # at that point, we'll only have partial results which could be impacted by the
+ # error causing the uncaught exception, making them uninterpretable
+ self._exit_dts()
+
+ def _check_dts_python_version(self) -> None:
+ """Check the required Python version - v3.10."""
+ if sys.version_info.major < 3 or (
+ sys.version_info.major == 3 and sys.version_info.minor < 10
+ ):
+ self._logger.warning(
+ "DTS execution node's python version is lower than Python 3.10, "
+ "is deprecated and will not work in future releases."
+ )
+ self._logger.warning("Please use Python >= 3.10 instead.")
+
+ def _run_execution(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ ) -> None:
+ """Run the given execution.
+
+ This involves running the execution setup as well as running all build targets
+ in the given execution. After that, execution teardown is run.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: An execution's test run configuration.
+ """
+ self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
+ execution_result = self._result.add_execution(sut_node.config)
+ execution_result.add_sut_info(sut_node.node_info)
+
+ try:
+ sut_node.set_up_execution(execution)
+ execution_result.update_setup(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Execution setup failed.")
+ execution_result.update_setup(Result.FAIL, e)
+
+ else:
+ for build_target in execution.build_targets:
+ self._run_build_target(sut_node, tg_node, build_target, execution, execution_result)
+
+ finally:
+ try:
+ sut_node.tear_down_execution()
+ execution_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Execution teardown failed.")
+ execution_result.update_teardown(Result.FAIL, e)
+
+ def _run_build_target(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ build_target: BuildTargetConfiguration,
+ execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ ) -> None:
+ """Run the given build target.
+
+ This involves running the build target setup as well as running all test suites
+ of the build target's execution.
+ After that, build target teardown is run.
+
+ Args:
+ sut_node: The execution's sut node.
+ tg_node: The execution's tg node.
+ build_target: A build target's test run configuration.
+ execution: The build target's execution's test run configuration.
+ execution_result: The execution level result object associated with the execution.
+ """
+ self._logger.info(f"Running build target '{build_target.name}'.")
+ build_target_result = execution_result.add_build_target(build_target)
+
+ try:
+ sut_node.set_up_build_target(build_target)
+ self._result.dpdk_version = sut_node.dpdk_version
+ build_target_result.add_build_target_info(sut_node.get_build_target_info())
+ build_target_result.update_setup(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Build target setup failed.")
+ build_target_result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_all_suites(sut_node, tg_node, execution, build_target_result)
+
+ finally:
+ try:
+ sut_node.tear_down_build_target()
+ build_target_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Build target teardown failed.")
+ build_target_result.update_teardown(Result.FAIL, e)
+
+ def _run_all_suites(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ build_target_result: BuildTargetResult,
+ ) -> None:
+ """Run the execution's (possibly a subset of) test suites using the current build target.
+
+ The method assumes the build target we're testing has already been built on the SUT node.
+ The current build target thus corresponds to the current DPDK build present on the SUT node.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: The execution's test run configuration associated
+ with the current build target.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ """
+ end_build_target = False
+ if not execution.skip_smoke_tests:
+ execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
+ for test_suite_config in execution.test_suites:
+ try:
+ self._run_single_suite(
+ sut_node, tg_node, execution, build_target_result, test_suite_config
+ )
+ except BlockingTestSuiteError as e:
+ self._logger.exception(
+ f"An error occurred within {test_suite_config.test_suite}. "
+ "Skipping build target..."
+ )
+ self._result.add_error(e)
+ end_build_target = True
+ # if a blocking test failed and we need to bail out of suite executions
+ if end_build_target:
+ break
+
+ def _run_single_suite(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ build_target_result: BuildTargetResult,
+ test_suite_config: TestSuiteConfig,
+ ) -> None:
+ """Run all test suites in a single test suite module.
+
+ The method assumes the build target we're testing has already been built on the SUT node.
+ The current build target thus corresponds to the current DPDK build present on the SUT node.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: The execution's test run configuration associated
+ with the current build target.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ test_suite_config: Test suite test run configuration specifying the test suite module
+ and possibly a subset of test cases of test suites in that module.
+
+ Raises:
+ BlockingTestSuiteError: If a blocking test suite fails.
+ """
+ try:
+ full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
+ test_suite_classes = get_test_suites(full_suite_path)
+ suites_str = ", ".join((x.__name__ for x in test_suite_classes))
+ self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
+ except Exception as e:
+ self._logger.exception("An error occurred when searching for test suites.")
+ self._result.update_setup(Result.ERROR, e)
+
+ else:
+ for test_suite_class in test_suite_classes:
+ test_suite = test_suite_class(
+ sut_node,
+ tg_node,
+ test_suite_config.test_cases,
+ execution.func,
+ build_target_result,
+ )
+ test_suite.run()
+
+ def _exit_dts(self) -> None:
+ """Process all errors and exit with the proper exit code."""
+ self._result.process()
+
+ if self._logger:
+ self._logger.info("DTS execution has ended.")
+
+ logging.shutdown()
+ sys.exit(self._result.get_return_code())
diff --git a/dts/main.py b/dts/main.py
index f703615d11..1ffe8ff81f 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -21,9 +21,11 @@ def main() -> None:
be modified before the settings module is imported anywhere else in the framework.
"""
settings.SETTINGS = settings.get_settings()
- from framework import dts
- dts.run_all()
+ from framework.runner import DTSRunner
+
+ dts = DTSRunner()
+ dts.run()
# Main program begins here
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 2/7] dts: move test suite execution logic to DTSRunner
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 1/7] dts: convert dts.py methods to class Juraj Linkeš
@ 2024-02-06 14:57 ` Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 3/7] dts: filter test suites in executions Juraj Linkeš
` (4 subsequent siblings)
6 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
Move the code responsible for running the test suite from the
TestSuite class to the DTSRunner class. This restructuring decision
was made to consolidate and unify the related logic into a single unit.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 175 ++++++++++++++++++++++++++++++++----
dts/framework/test_suite.py | 152 ++-----------------------------
2 files changed, 169 insertions(+), 158 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index acc1c4d6db..933685d638 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -19,6 +19,7 @@
import logging
import sys
+from types import MethodType
from .config import (
BuildTargetConfiguration,
@@ -26,10 +27,18 @@
TestSuiteConfig,
load_config,
)
-from .exception import BlockingTestSuiteError
+from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
from .logger import DTSLOG, getLogger
-from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
-from .test_suite import get_test_suites
+from .settings import SETTINGS
+from .test_result import (
+ BuildTargetResult,
+ DTSResult,
+ ExecutionResult,
+ Result,
+ TestCaseResult,
+ TestSuiteResult,
+)
+from .test_suite import TestSuite, get_test_suites
from .testbed_model import SutNode, TGNode
@@ -227,7 +236,7 @@ def _run_build_target(
build_target_result.update_setup(Result.FAIL, e)
else:
- self._run_all_suites(sut_node, tg_node, execution, build_target_result)
+ self._run_test_suites(sut_node, tg_node, execution, build_target_result)
finally:
try:
@@ -237,7 +246,7 @@ def _run_build_target(
self._logger.exception("Build target teardown failed.")
build_target_result.update_teardown(Result.FAIL, e)
- def _run_all_suites(
+ def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
@@ -249,6 +258,9 @@ def _run_all_suites(
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
+ in the current build target won't be executed.
+
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
@@ -262,7 +274,7 @@ def _run_all_suites(
execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
for test_suite_config in execution.test_suites:
try:
- self._run_single_suite(
+ self._run_test_suite_module(
sut_node, tg_node, execution, build_target_result, test_suite_config
)
except BlockingTestSuiteError as e:
@@ -276,7 +288,7 @@ def _run_all_suites(
if end_build_target:
break
- def _run_single_suite(
+ def _run_test_suite_module(
self,
sut_node: SutNode,
tg_node: TGNode,
@@ -284,11 +296,18 @@ def _run_single_suite(
build_target_result: BuildTargetResult,
test_suite_config: TestSuiteConfig,
) -> None:
- """Run all test suites in a single test suite module.
+ """Set up, execute and tear down all test suites in a single test suite module.
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ Test suite execution consists of running the discovered test cases.
+ A test case run consists of setup, execution and teardown of said test case.
+
+ Record the setup and the teardown and handle failures.
+
+ The test cases to execute are discovered when creating the :class:`TestSuite` object.
+
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
@@ -313,14 +332,140 @@ def _run_single_suite(
else:
for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(
- sut_node,
- tg_node,
- test_suite_config.test_cases,
- execution.func,
- build_target_result,
+ test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases)
+
+ test_suite_name = test_suite.__class__.__name__
+ test_suite_result = build_target_result.add_test_suite(test_suite_name)
+ try:
+ self._logger.info(f"Starting test suite setup: {test_suite_name}")
+ test_suite.set_up_suite()
+ test_suite_result.update_setup(Result.PASS)
+ self._logger.info(f"Test suite setup successful: {test_suite_name}")
+ except Exception as e:
+ self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
+ test_suite_result.update_setup(Result.ERROR, e)
+
+ else:
+ self._execute_test_suite(execution.func, test_suite, test_suite_result)
+
+ finally:
+ try:
+ test_suite.tear_down_suite()
+ sut_node.kill_cleanup_dpdk_apps()
+ test_suite_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
+ self._logger.warning(
+ f"Test suite '{test_suite_name}' teardown failed, "
+ f"the next test suite may be affected."
+ )
+ test_suite_result.update_setup(Result.ERROR, e)
+ if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
+ raise BlockingTestSuiteError(test_suite_name)
+
+ def _execute_test_suite(
+ self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
+ ) -> None:
+ """Execute all discovered test cases in `test_suite`.
+
+ If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment
+ variable is set, in case of a test case failure, the test case will be executed again
+ until it passes or it fails that many times in addition of the first failure.
+
+ Args:
+ func: Whether to execute functional test cases.
+ test_suite: The test suite object.
+ test_suite_result: The test suite level result object associated
+ with the current test suite.
+ """
+ if func:
+ for test_case_method in test_suite._get_functional_test_cases():
+ test_case_name = test_case_method.__name__
+ test_case_result = test_suite_result.add_test_case(test_case_name)
+ all_attempts = SETTINGS.re_run + 1
+ attempt_nr = 1
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+
+ def _run_test_case(
+ self,
+ test_suite: TestSuite,
+ test_case_method: MethodType,
+ test_case_result: TestCaseResult,
+ ) -> None:
+ """Setup, execute and teardown a test case in `test_suite`.
+
+ Record the result of the setup and the teardown and handle failures.
+
+ Args:
+ test_suite: The test suite object.
+ test_case_method: The test case method.
+ test_case_result: The test case level result object associated
+ with the current test case.
+ """
+ test_case_name = test_case_method.__name__
+
+ try:
+ # run set_up function for each case
+ test_suite.set_up_test_case()
+ test_case_result.update_setup(Result.PASS)
+ except SSHTimeoutError as e:
+ self._logger.exception(f"Test case setup FAILED: {test_case_name}")
+ test_case_result.update_setup(Result.FAIL, e)
+ except Exception as e:
+ self._logger.exception(f"Test case setup ERROR: {test_case_name}")
+ test_case_result.update_setup(Result.ERROR, e)
+
+ else:
+ # run test case if setup was successful
+ self._execute_test_case(test_case_method, test_case_result)
+
+ finally:
+ try:
+ test_suite.tear_down_test_case()
+ test_case_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test case teardown ERROR: {test_case_name}")
+ self._logger.warning(
+ f"Test case '{test_case_name}' teardown failed, "
+ f"the next test case may be affected."
)
- test_suite.run()
+ test_case_result.update_teardown(Result.ERROR, e)
+ test_case_result.update(Result.ERROR)
+
+ def _execute_test_case(
+ self, test_case_method: MethodType, test_case_result: TestCaseResult
+ ) -> None:
+ """Execute one test case, record the result and handle failures.
+
+ Args:
+ test_case_method: The test case method.
+ test_case_result: The test case level result object associated
+ with the current test case.
+ """
+ test_case_name = test_case_method.__name__
+ try:
+ self._logger.info(f"Starting test case execution: {test_case_name}")
+ test_case_method()
+ test_case_result.update(Result.PASS)
+ self._logger.info(f"Test case execution PASSED: {test_case_name}")
+
+ except TestCaseVerifyError as e:
+ self._logger.exception(f"Test case execution FAILED: {test_case_name}")
+ test_case_result.update(Result.FAIL, e)
+ except Exception as e:
+ self._logger.exception(f"Test case execution ERROR: {test_case_name}")
+ test_case_result.update(Result.ERROR, e)
+ except KeyboardInterrupt:
+ self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}")
+ test_case_result.update(Result.SKIP)
+ raise KeyboardInterrupt("Stop DTS")
def _exit_dts(self) -> None:
"""Process all errors and exit with the proper exit code."""
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index dfb391ffbd..b02fd36147 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -8,7 +8,6 @@
must be extended by subclasses which add test cases. The :class:`TestSuite` contains the basics
needed by subclasses:
- * Test suite and test case execution flow,
* Testbed (SUT, TG) configuration,
* Packet sending and verification,
* Test case verification.
@@ -28,27 +27,22 @@
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
-from .exception import (
- BlockingTestSuiteError,
- ConfigurationError,
- SSHTimeoutError,
- TestCaseVerifyError,
-)
+from .exception import ConfigurationError, TestCaseVerifyError
from .logger import DTSLOG, getLogger
from .settings import SETTINGS
-from .test_result import BuildTargetResult, Result, TestCaseResult, TestSuiteResult
from .testbed_model import Port, PortLink, SutNode, TGNode
from .utils import get_packet_summaries
class TestSuite(object):
- """The base class with methods for handling the basic flow of a test suite.
+ """The base class with building blocks needed by most test cases.
* Test case filtering and collection,
- * Test suite setup/cleanup,
- * Test setup/cleanup,
- * Test case execution,
- * Error handling and results storage.
+ * Test suite setup/cleanup methods to override,
+ * Test case setup/cleanup methods to override,
+ * Test case verification,
+ * Testbed configuration,
+ * Traffic sending and verification.
Test cases are implemented by subclasses. Test cases are all methods starting with ``test_``,
further divided into performance test cases (starting with ``test_perf_``)
@@ -60,10 +54,6 @@ class TestSuite(object):
The union of both lists will be used. Any unknown test cases from the latter lists
will be silently ignored.
- If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable
- is set, in case of a test case failure, the test case will be executed again until it passes
- or it fails that many times in addition of the first failure.
-
The methods named ``[set_up|tear_down]_[suite|test_case]`` should be overridden in subclasses
if the appropriate test suite/test case fixtures are needed.
@@ -82,8 +72,6 @@ class TestSuite(object):
is_blocking: ClassVar[bool] = False
_logger: DTSLOG
_test_cases_to_run: list[str]
- _func: bool
- _result: TestSuiteResult
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -99,30 +87,23 @@ def __init__(
sut_node: SutNode,
tg_node: TGNode,
test_cases: list[str],
- func: bool,
- build_target_result: BuildTargetResult,
):
"""Initialize the test suite testbed information and basic configuration.
- Process what test cases to run, create the associated
- :class:`~.test_result.TestSuiteResult`, find links between ports
- and set up default IP addresses to be used when configuring them.
+ Process what test cases to run, find links between ports and set up
+ default IP addresses to be used when configuring them.
Args:
sut_node: The SUT node where the test suite will run.
tg_node: The TG node where the test suite will run.
test_cases: The list of test cases to execute.
If empty, all test cases will be executed.
- func: Whether to run functional tests.
- build_target_result: The build target result this test suite is run in.
"""
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = getLogger(self.__class__.__name__)
self._test_cases_to_run = test_cases
self._test_cases_to_run.extend(SETTINGS.test_cases)
- self._func = func
- self._result = build_target_result.add_test_suite(self.__class__.__name__)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
@@ -384,62 +365,6 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
return False
return True
- def run(self) -> None:
- """Set up, execute and tear down the whole suite.
-
- Test suite execution consists of running all test cases scheduled to be executed.
- A test case run consists of setup, execution and teardown of said test case.
-
- Record the setup and the teardown and handle failures.
-
- The list of scheduled test cases is constructed when creating the :class:`TestSuite` object.
- """
- test_suite_name = self.__class__.__name__
-
- try:
- self._logger.info(f"Starting test suite setup: {test_suite_name}")
- self.set_up_suite()
- self._result.update_setup(Result.PASS)
- self._logger.info(f"Test suite setup successful: {test_suite_name}")
- except Exception as e:
- self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
- self._result.update_setup(Result.ERROR, e)
-
- else:
- self._execute_test_suite()
-
- finally:
- try:
- self.tear_down_suite()
- self.sut_node.kill_cleanup_dpdk_apps()
- self._result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
- self._logger.warning(
- f"Test suite '{test_suite_name}' teardown failed, "
- f"the next test suite may be affected."
- )
- self._result.update_setup(Result.ERROR, e)
- if len(self._result.get_errors()) > 0 and self.is_blocking:
- raise BlockingTestSuiteError(test_suite_name)
-
- def _execute_test_suite(self) -> None:
- """Execute all test cases scheduled to be executed in this suite."""
- if self._func:
- for test_case_method in self._get_functional_test_cases():
- test_case_name = test_case_method.__name__
- test_case_result = self._result.add_test_case(test_case_name)
- all_attempts = SETTINGS.re_run + 1
- attempt_nr = 1
- self._run_test_case(test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
- self._run_test_case(test_case_method, test_case_result)
-
def _get_functional_test_cases(self) -> list[MethodType]:
"""Get all functional test cases defined in this TestSuite.
@@ -471,65 +396,6 @@ def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool
return match
- def _run_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
- ) -> None:
- """Setup, execute and teardown a test case in this suite.
-
- Record the result of the setup and the teardown and handle failures.
- """
- test_case_name = test_case_method.__name__
-
- try:
- # run set_up function for each case
- self.set_up_test_case()
- test_case_result.update_setup(Result.PASS)
- except SSHTimeoutError as e:
- self._logger.exception(f"Test case setup FAILED: {test_case_name}")
- test_case_result.update_setup(Result.FAIL, e)
- except Exception as e:
- self._logger.exception(f"Test case setup ERROR: {test_case_name}")
- test_case_result.update_setup(Result.ERROR, e)
-
- else:
- # run test case if setup was successful
- self._execute_test_case(test_case_method, test_case_result)
-
- finally:
- try:
- self.tear_down_test_case()
- test_case_result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test case teardown ERROR: {test_case_name}")
- self._logger.warning(
- f"Test case '{test_case_name}' teardown failed, "
- f"the next test case may be affected."
- )
- test_case_result.update_teardown(Result.ERROR, e)
- test_case_result.update(Result.ERROR)
-
- def _execute_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
- ) -> None:
- """Execute one test case, record the result and handle failures."""
- test_case_name = test_case_method.__name__
- try:
- self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method()
- test_case_result.update(Result.PASS)
- self._logger.info(f"Test case execution PASSED: {test_case_name}")
-
- except TestCaseVerifyError as e:
- self._logger.exception(f"Test case execution FAILED: {test_case_name}")
- test_case_result.update(Result.FAIL, e)
- except Exception as e:
- self._logger.exception(f"Test case execution ERROR: {test_case_name}")
- test_case_result.update(Result.ERROR, e)
- except KeyboardInterrupt:
- self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}")
- test_case_result.update(Result.SKIP)
- raise KeyboardInterrupt("Stop DTS")
-
def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
r"""Find all :class:`TestSuite`\s in a Python module.
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 3/7] dts: filter test suites in executions
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 1/7] dts: convert dts.py methods to class Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš
@ 2024-02-06 14:57 ` Juraj Linkeš
2024-02-12 16:44 ` Jeremy Spewock
2024-02-06 14:57 ` [PATCH v2 4/7] dts: reorganize test result Juraj Linkeš
` (3 subsequent siblings)
6 siblings, 1 reply; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
We're currently filtering which test cases to run after some setup
steps, such as DPDK build, have already been taken. This prohibits us to
mark the test suites and cases that were supposed to be run as blocked
when an earlier setup fails, as that information is not available at
that time.
To remedy this, move the filtering to the beginning of each execution.
This is the first action taken in each execution and if we can't filter
the test cases, such as due to invalid inputs, we abort the whole
execution. No test suites nor cases will be marked as blocked as we
don't know which were supposed to be run.
On top of that, the filtering takes place in the TestSuite class, which
should only concern itself with test suite and test case logic, not the
processing behind the scenes. The logic has been moved to DTSRunner
which should do all the processing needed to run test suites.
The filtering itself introduces a few changes/assumptions which are more
sensible than before:
1. Assumption: There is just one TestSuite child class in each test
suite module. This was an implicit assumption before as we couldn't
specify the TestSuite classes in the test run configuration, just the
modules. The name of the TestSuite child class starts with "Test" and
then corresponds to the name of the module with CamelCase naming.
2. Unknown test cases specified both in the test run configuration and
the environment variable/command line argument are no longer silently
ignored. This is a quality of life improvement for users, as they
could easily be not aware of the silent ignoration.
Also, a change in the code results in pycodestyle warning and error:
[E] E203 whitespace before ':'
[W] W503 line break before binary operator
These two are not PEP8 compliant, so they're disabled.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/config/__init__.py | 24 +-
dts/framework/config/conf_yaml_schema.json | 2 +-
dts/framework/runner.py | 426 +++++++++++++++------
dts/framework/settings.py | 3 +-
dts/framework/test_result.py | 34 ++
dts/framework/test_suite.py | 85 +---
dts/pyproject.toml | 3 +
dts/tests/TestSuite_smoke_tests.py | 2 +-
8 files changed, 382 insertions(+), 197 deletions(-)
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 62eded7f04..c6a93b3b89 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -36,7 +36,7 @@
import json
import os.path
import pathlib
-from dataclasses import dataclass
+from dataclasses import dataclass, fields
from enum import auto, unique
from typing import Union
@@ -506,6 +506,28 @@ def from_dict(
vdevs=vdevs,
)
+ def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration":
+ """Create a shallow copy with any of the fields modified.
+
+ The only new data are those passed to this method.
+ The rest are copied from the object's fields calling the method.
+
+ Args:
+ **kwargs: The names and types of keyword arguments are defined
+ by the fields of the :class:`ExecutionConfiguration` class.
+
+ Returns:
+ The copied and modified execution configuration.
+ """
+ new_config = {}
+ for field in fields(self):
+ if field.name in kwargs:
+ new_config[field.name] = kwargs[field.name]
+ else:
+ new_config[field.name] = getattr(self, field.name)
+
+ return ExecutionConfiguration(**new_config)
+
@dataclass(slots=True, frozen=True)
class Configuration:
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 84e45fe3c2..051b079fe4 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -197,7 +197,7 @@
},
"cases": {
"type": "array",
- "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.",
+ "description": "If specified, only this subset of test suite's test cases will be run.",
"items": {
"type": "string"
},
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 933685d638..3e95cf9e26 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -17,17 +17,27 @@
and the test case stage runs test cases individually.
"""
+import importlib
+import inspect
import logging
+import re
import sys
from types import MethodType
+from typing import Iterable
from .config import (
BuildTargetConfiguration,
+ Configuration,
ExecutionConfiguration,
TestSuiteConfig,
load_config,
)
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .exception import (
+ BlockingTestSuiteError,
+ ConfigurationError,
+ SSHTimeoutError,
+ TestCaseVerifyError,
+)
from .logger import DTSLOG, getLogger
from .settings import SETTINGS
from .test_result import (
@@ -37,8 +47,9 @@
Result,
TestCaseResult,
TestSuiteResult,
+ TestSuiteWithCases,
)
-from .test_suite import TestSuite, get_test_suites
+from .test_suite import TestSuite
from .testbed_model import SutNode, TGNode
@@ -59,13 +70,23 @@ class DTSRunner:
given execution, the next execution begins.
"""
+ _configuration: Configuration
_logger: DTSLOG
_result: DTSResult
+ _test_suite_class_prefix: str
+ _test_suite_module_prefix: str
+ _func_test_case_regex: str
+ _perf_test_case_regex: str
def __init__(self):
- """Initialize the instance with logger and result."""
+ """Initialize the instance with configuration, logger, result and string constants."""
+ self._configuration = load_config()
self._logger = getLogger("DTSRunner")
self._result = DTSResult(self._logger)
+ self._test_suite_class_prefix = "Test"
+ self._test_suite_module_prefix = "tests.TestSuite_"
+ self._func_test_case_regex = r"test_(?!perf_)"
+ self._perf_test_case_regex = r"test_perf_"
def run(self):
"""Run all build targets in all executions from the test run configuration.
@@ -106,29 +127,28 @@ def run(self):
try:
# check the python version of the server that runs dts
self._check_dts_python_version()
+ self._result.update_setup(Result.PASS)
# for all Execution sections
- for execution in load_config().executions:
- sut_node = sut_nodes.get(execution.system_under_test_node.name)
- tg_node = tg_nodes.get(execution.traffic_generator_node.name)
-
+ for execution in self._configuration.executions:
+ self._logger.info(
+ f"Running execution with SUT '{execution.system_under_test_node.name}'."
+ )
+ execution_result = self._result.add_execution(execution.system_under_test_node)
try:
- if not sut_node:
- sut_node = SutNode(execution.system_under_test_node)
- sut_nodes[sut_node.name] = sut_node
- if not tg_node:
- tg_node = TGNode(execution.traffic_generator_node)
- tg_nodes[tg_node.name] = tg_node
- self._result.update_setup(Result.PASS)
+ test_suites_with_cases = self._get_test_suites_with_cases(
+ execution.test_suites, execution.func, execution.perf
+ )
except Exception as e:
- failed_node = execution.system_under_test_node.name
- if sut_node:
- failed_node = execution.traffic_generator_node.name
- self._logger.exception(f"The Creation of node {failed_node} failed.")
- self._result.update_setup(Result.FAIL, e)
+ self._logger.exception(
+ f"Invalid test suite configuration found: " f"{execution.test_suites}."
+ )
+ execution_result.update_setup(Result.FAIL, e)
else:
- self._run_execution(sut_node, tg_node, execution)
+ self._connect_nodes_and_run_execution(
+ sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases
+ )
except Exception as e:
self._logger.exception("An unexpected error has occurred.")
@@ -163,11 +183,204 @@ def _check_dts_python_version(self) -> None:
)
self._logger.warning("Please use Python >= 3.10 instead.")
+ def _get_test_suites_with_cases(
+ self,
+ test_suite_configs: list[TestSuiteConfig],
+ func: bool,
+ perf: bool,
+ ) -> list[TestSuiteWithCases]:
+ """Test suites with test cases discovery.
+
+ The test suites with test cases defined in the user configuration are discovered
+ and stored for future use so that we don't import the modules twice and so that
+ the list of test suites with test cases is available for recording right away.
+
+ Args:
+ test_suite_configs: Test suite configurations.
+ func: Whether to include functional test cases in the final list.
+ perf: Whether to include performance test cases in the final list.
+
+ Returns:
+ The discovered test suites, each with test cases.
+ """
+ test_suites_with_cases = []
+
+ for test_suite_config in test_suite_configs:
+ test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
+ test_cases = []
+ func_test_cases, perf_test_cases = self._filter_test_cases(
+ test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases)
+ )
+ if func:
+ test_cases.extend(func_test_cases)
+ if perf:
+ test_cases.extend(perf_test_cases)
+
+ test_suites_with_cases.append(
+ TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases)
+ )
+
+ return test_suites_with_cases
+
+ def _get_test_suite_class(self, test_suite_name: str) -> type[TestSuite]:
+ """Find the :class:`TestSuite` class with `test_suite_name` in the corresponding module.
+
+ The method assumes that the :class:`TestSuite` class starts
+ with `self._test_suite_class_prefix`,
+ continuing with `test_suite_name` with CamelCase convention.
+ It also assumes there's only one test suite in each module and the module name
+ is `test_suite_name` prefixed with `self._test_suite_module_prefix`.
+
+ The CamelCase convention is not tested, only lowercase strings are compared.
+
+ Args:
+ test_suite_name: The name of the test suite to find.
+
+ Returns:
+ The found test suite.
+
+ Raises:
+ ConfigurationError: If the corresponding module is not found or
+ a valid :class:`TestSuite` is not found in the module.
+ """
+
+ def is_test_suite(object) -> bool:
+ """Check whether `object` is a :class:`TestSuite`.
+
+ The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself.
+
+ Args:
+ object: The object to be checked.
+
+ Returns:
+ :data:`True` if `object` is a subclass of `TestSuite`.
+ """
+ try:
+ if issubclass(object, TestSuite) and object is not TestSuite:
+ return True
+ except TypeError:
+ return False
+ return False
+
+ testsuite_module_path = f"{self._test_suite_module_prefix}{test_suite_name}"
+ try:
+ test_suite_module = importlib.import_module(testsuite_module_path)
+ except ModuleNotFoundError as e:
+ raise ConfigurationError(
+ f"Test suite module '{testsuite_module_path}' not found."
+ ) from e
+
+ lowercase_suite_name = test_suite_name.replace("_", "").lower()
+ for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite):
+ if (
+ class_name.startswith(self._test_suite_class_prefix)
+ and lowercase_suite_name == class_name[len(self._test_suite_class_prefix) :].lower()
+ ):
+ return class_obj
+ raise ConfigurationError(
+ f"Couldn't find any valid test suites in {test_suite_module.__name__}."
+ )
+
+ def _filter_test_cases(
+ self, test_suite_class: type[TestSuite], test_cases_to_run: set[str]
+ ) -> tuple[list[MethodType], list[MethodType]]:
+ """Filter `test_cases_to_run` from `test_suite_class`.
+
+ There are two rounds of filtering if `test_cases_to_run` is not empty.
+ The first filters `test_cases_to_run` from all methods of `test_suite_class`.
+ Then the methods are separated into functional and performance test cases.
+ If a method doesn't match neither the functional nor performance name prefix, it's an error.
+
+ Args:
+ test_suite_class: The class of the test suite.
+ test_cases_to_run: Test case names to filter from `test_suite_class`.
+ If empty, return all matching test cases.
+
+ Returns:
+ A list of test case methods that should be executed.
+
+ Raises:
+ ConfigurationError: If a test case from `test_cases_to_run` is not found
+ or it doesn't match either the functional nor performance name prefix.
+ """
+ func_test_cases = []
+ perf_test_cases = []
+ name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
+ if test_cases_to_run:
+ name_method_tuples = [
+ (name, method) for name, method in name_method_tuples if name in test_cases_to_run
+ ]
+ if len(name_method_tuples) < len(test_cases_to_run):
+ missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples}
+ raise ConfigurationError(
+ f"Test cases {missing_test_cases} not found among methods "
+ f"of {test_suite_class.__name__}."
+ )
+
+ for test_case_name, test_case_method in name_method_tuples:
+ if re.match(self._func_test_case_regex, test_case_name):
+ func_test_cases.append(test_case_method)
+ elif re.match(self._perf_test_case_regex, test_case_name):
+ perf_test_cases.append(test_case_method)
+ elif test_cases_to_run:
+ raise ConfigurationError(
+ f"Method '{test_case_name}' doesn't match neither "
+ f"a functional nor a performance test case name."
+ )
+
+ return func_test_cases, perf_test_cases
+
+ def _connect_nodes_and_run_execution(
+ self,
+ sut_nodes: dict[str, SutNode],
+ tg_nodes: dict[str, TGNode],
+ execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
+ ) -> None:
+ """Connect nodes, then continue to run the given execution.
+
+ Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`.
+ If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`,
+ respectively.
+ If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`.
+
+ Args:
+ sut_nodes: A dictionary storing connected/to be connected SUT nodes.
+ tg_nodes: A dictionary storing connected/to be connected TG nodes.
+ execution: An execution's test run configuration.
+ execution_result: The execution's result.
+ test_suites_with_cases: The test suites with test cases to run.
+ """
+ sut_node = sut_nodes.get(execution.system_under_test_node.name)
+ tg_node = tg_nodes.get(execution.traffic_generator_node.name)
+
+ try:
+ if not sut_node:
+ sut_node = SutNode(execution.system_under_test_node)
+ sut_nodes[sut_node.name] = sut_node
+ if not tg_node:
+ tg_node = TGNode(execution.traffic_generator_node)
+ tg_nodes[tg_node.name] = tg_node
+ except Exception as e:
+ failed_node = execution.system_under_test_node.name
+ if sut_node:
+ failed_node = execution.traffic_generator_node.name
+ self._logger.exception(f"The Creation of node {failed_node} failed.")
+ execution_result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_execution(
+ sut_node, tg_node, execution, execution_result, test_suites_with_cases
+ )
+
def _run_execution(
self,
sut_node: SutNode,
tg_node: TGNode,
execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
"""Run the given execution.
@@ -178,11 +391,11 @@ def _run_execution(
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
execution: An execution's test run configuration.
+ execution_result: The execution's result.
+ test_suites_with_cases: The test suites with test cases to run.
"""
self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
- execution_result = self._result.add_execution(sut_node.config)
execution_result.add_sut_info(sut_node.node_info)
-
try:
sut_node.set_up_execution(execution)
execution_result.update_setup(Result.PASS)
@@ -192,7 +405,10 @@ def _run_execution(
else:
for build_target in execution.build_targets:
- self._run_build_target(sut_node, tg_node, build_target, execution, execution_result)
+ build_target_result = execution_result.add_build_target(build_target)
+ self._run_build_target(
+ sut_node, tg_node, build_target, build_target_result, test_suites_with_cases
+ )
finally:
try:
@@ -207,8 +423,8 @@ def _run_build_target(
sut_node: SutNode,
tg_node: TGNode,
build_target: BuildTargetConfiguration,
- execution: ExecutionConfiguration,
- execution_result: ExecutionResult,
+ build_target_result: BuildTargetResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
"""Run the given build target.
@@ -220,11 +436,11 @@ def _run_build_target(
sut_node: The execution's sut node.
tg_node: The execution's tg node.
build_target: A build target's test run configuration.
- execution: The build target's execution's test run configuration.
- execution_result: The execution level result object associated with the execution.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ test_suites_with_cases: The test suites with test cases to run.
"""
self._logger.info(f"Running build target '{build_target.name}'.")
- build_target_result = execution_result.add_build_target(build_target)
try:
sut_node.set_up_build_target(build_target)
@@ -236,7 +452,7 @@ def _run_build_target(
build_target_result.update_setup(Result.FAIL, e)
else:
- self._run_test_suites(sut_node, tg_node, execution, build_target_result)
+ self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases)
finally:
try:
@@ -250,10 +466,10 @@ def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
- execution: ExecutionConfiguration,
build_target_result: BuildTargetResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
- """Run the execution's (possibly a subset of) test suites using the current build target.
+ """Run `test_suites_with_cases` with the current build target.
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
@@ -264,22 +480,20 @@ def _run_test_suites(
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
- execution: The execution's test run configuration associated
- with the current build target.
build_target_result: The build target level result object associated
with the current build target.
+ test_suites_with_cases: The test suites with test cases to run.
"""
end_build_target = False
- if not execution.skip_smoke_tests:
- execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
- for test_suite_config in execution.test_suites:
+ for test_suite_with_cases in test_suites_with_cases:
+ test_suite_result = build_target_result.add_test_suite(
+ test_suite_with_cases.test_suite_class.__name__
+ )
try:
- self._run_test_suite_module(
- sut_node, tg_node, execution, build_target_result, test_suite_config
- )
+ self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
except BlockingTestSuiteError as e:
self._logger.exception(
- f"An error occurred within {test_suite_config.test_suite}. "
+ f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
"Skipping build target..."
)
self._result.add_error(e)
@@ -288,15 +502,14 @@ def _run_test_suites(
if end_build_target:
break
- def _run_test_suite_module(
+ def _run_test_suite(
self,
sut_node: SutNode,
tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
- test_suite_config: TestSuiteConfig,
+ test_suite_result: TestSuiteResult,
+ test_suite_with_cases: TestSuiteWithCases,
) -> None:
- """Set up, execute and tear down all test suites in a single test suite module.
+ """Set up, execute and tear down `test_suite_with_cases`.
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
@@ -306,92 +519,79 @@ def _run_test_suite_module(
Record the setup and the teardown and handle failures.
- The test cases to execute are discovered when creating the :class:`TestSuite` object.
-
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
- execution: The execution's test run configuration associated
- with the current build target.
- build_target_result: The build target level result object associated
- with the current build target.
- test_suite_config: Test suite test run configuration specifying the test suite module
- and possibly a subset of test cases of test suites in that module.
+ test_suite_result: The test suite level result object associated
+ with the current test suite.
+ test_suite_with_cases: The test suite with test cases to run.
Raises:
BlockingTestSuiteError: If a blocking test suite fails.
"""
+ test_suite_name = test_suite_with_cases.test_suite_class.__name__
+ test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
try:
- full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
- test_suite_classes = get_test_suites(full_suite_path)
- suites_str = ", ".join((x.__name__ for x in test_suite_classes))
- self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
+ self._logger.info(f"Starting test suite setup: {test_suite_name}")
+ test_suite.set_up_suite()
+ test_suite_result.update_setup(Result.PASS)
+ self._logger.info(f"Test suite setup successful: {test_suite_name}")
except Exception as e:
- self._logger.exception("An error occurred when searching for test suites.")
- self._result.update_setup(Result.ERROR, e)
+ self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
+ test_suite_result.update_setup(Result.ERROR, e)
else:
- for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases)
-
- test_suite_name = test_suite.__class__.__name__
- test_suite_result = build_target_result.add_test_suite(test_suite_name)
- try:
- self._logger.info(f"Starting test suite setup: {test_suite_name}")
- test_suite.set_up_suite()
- test_suite_result.update_setup(Result.PASS)
- self._logger.info(f"Test suite setup successful: {test_suite_name}")
- except Exception as e:
- self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
- test_suite_result.update_setup(Result.ERROR, e)
-
- else:
- self._execute_test_suite(execution.func, test_suite, test_suite_result)
-
- finally:
- try:
- test_suite.tear_down_suite()
- sut_node.kill_cleanup_dpdk_apps()
- test_suite_result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
- self._logger.warning(
- f"Test suite '{test_suite_name}' teardown failed, "
- f"the next test suite may be affected."
- )
- test_suite_result.update_setup(Result.ERROR, e)
- if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
- raise BlockingTestSuiteError(test_suite_name)
+ self._execute_test_suite(
+ test_suite,
+ test_suite_with_cases.test_cases,
+ test_suite_result,
+ )
+ finally:
+ try:
+ test_suite.tear_down_suite()
+ sut_node.kill_cleanup_dpdk_apps()
+ test_suite_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
+ self._logger.warning(
+ f"Test suite '{test_suite_name}' teardown failed, "
+ "the next test suite may be affected."
+ )
+ test_suite_result.update_setup(Result.ERROR, e)
+ if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
+ raise BlockingTestSuiteError(test_suite_name)
def _execute_test_suite(
- self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
+ self,
+ test_suite: TestSuite,
+ test_cases: Iterable[MethodType],
+ test_suite_result: TestSuiteResult,
) -> None:
- """Execute all discovered test cases in `test_suite`.
+ """Execute all `test_cases` in `test_suite`.
If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment
variable is set, in case of a test case failure, the test case will be executed again
until it passes or it fails that many times in addition of the first failure.
Args:
- func: Whether to execute functional test cases.
test_suite: The test suite object.
+ test_cases: The list of test case methods.
test_suite_result: The test suite level result object associated
with the current test suite.
"""
- if func:
- for test_case_method in test_suite._get_functional_test_cases():
- test_case_name = test_case_method.__name__
- test_case_result = test_suite_result.add_test_case(test_case_name)
- all_attempts = SETTINGS.re_run + 1
- attempt_nr = 1
+ for test_case_method in test_cases:
+ test_case_name = test_case_method.__name__
+ test_case_result = test_suite_result.add_test_case(test_case_name)
+ all_attempts = SETTINGS.re_run + 1
+ attempt_nr = 1
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
self._run_test_case(test_suite, test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
- self._run_test_case(test_suite, test_case_method, test_case_result)
def _run_test_case(
self,
@@ -399,7 +599,7 @@ def _run_test_case(
test_case_method: MethodType,
test_case_result: TestCaseResult,
) -> None:
- """Setup, execute and teardown a test case in `test_suite`.
+ """Setup, execute and teardown `test_case_method` from `test_suite`.
Record the result of the setup and the teardown and handle failures.
@@ -424,7 +624,7 @@ def _run_test_case(
else:
# run test case if setup was successful
- self._execute_test_case(test_case_method, test_case_result)
+ self._execute_test_case(test_suite, test_case_method, test_case_result)
finally:
try:
@@ -440,11 +640,15 @@ def _run_test_case(
test_case_result.update(Result.ERROR)
def _execute_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
+ self,
+ test_suite: TestSuite,
+ test_case_method: MethodType,
+ test_case_result: TestCaseResult,
) -> None:
- """Execute one test case, record the result and handle failures.
+ """Execute `test_case_method` from `test_suite`, record the result and handle failures.
Args:
+ test_suite: The test suite object.
test_case_method: The test case method.
test_case_result: The test case level result object associated
with the current test case.
@@ -452,7 +656,7 @@ def _execute_test_case(
test_case_name = test_case_method.__name__
try:
self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method()
+ test_case_method(test_suite)
test_case_result.update(Result.PASS)
self._logger.info(f"Test case execution PASSED: {test_case_name}")
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 609c8d0e62..2b8bfbe0ed 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser:
"--test-cases",
action=_env_arg("DTS_TESTCASES"),
default="",
- help="[DTS_TESTCASES] Comma-separated list of test cases to execute. "
- "Unknown test cases will be silently ignored.",
+ help="[DTS_TESTCASES] Comma-separated list of test cases to execute.",
)
parser.add_argument(
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 4467749a9d..075195fd5b 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -25,7 +25,9 @@
import os.path
from collections.abc import MutableSequence
+from dataclasses import dataclass
from enum import Enum, auto
+from types import MethodType
from .config import (
OS,
@@ -36,10 +38,42 @@
CPUType,
NodeConfiguration,
NodeInfo,
+ TestSuiteConfig,
)
from .exception import DTSError, ErrorSeverity
from .logger import DTSLOG
from .settings import SETTINGS
+from .test_suite import TestSuite
+
+
+@dataclass(slots=True, frozen=True)
+class TestSuiteWithCases:
+ """A test suite class with test case methods.
+
+ An auxiliary class holding a test case class with test case methods. The intended use of this
+ class is to hold a subset of test cases (which could be all test cases) because we don't have
+ all the data to instantiate the class at the point of inspection. The knowledge of this subset
+ is needed in case an error occurs before the class is instantiated and we need to record
+ which test cases were blocked by the error.
+
+ Attributes:
+ test_suite_class: The test suite class.
+ test_cases: The test case methods.
+ """
+
+ test_suite_class: type[TestSuite]
+ test_cases: list[MethodType]
+
+ def create_config(self) -> TestSuiteConfig:
+ """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
+
+ Returns:
+ The :class:`TestSuiteConfig` representation.
+ """
+ return TestSuiteConfig(
+ test_suite=self.test_suite_class.__name__,
+ test_cases=[test_case.__name__ for test_case in self.test_cases],
+ )
class Result(Enum):
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index b02fd36147..f9fe88093e 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -11,25 +11,17 @@
* Testbed (SUT, TG) configuration,
* Packet sending and verification,
* Test case verification.
-
-The module also defines a function, :func:`get_test_suites`,
-for gathering test suites from a Python module.
"""
-import importlib
-import inspect
-import re
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
-from types import MethodType
-from typing import Any, ClassVar, Union
+from typing import ClassVar, Union
from scapy.layers.inet import IP # type: ignore[import]
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
-from .exception import ConfigurationError, TestCaseVerifyError
+from .exception import TestCaseVerifyError
from .logger import DTSLOG, getLogger
-from .settings import SETTINGS
from .testbed_model import Port, PortLink, SutNode, TGNode
from .utils import get_packet_summaries
@@ -37,7 +29,6 @@
class TestSuite(object):
"""The base class with building blocks needed by most test cases.
- * Test case filtering and collection,
* Test suite setup/cleanup methods to override,
* Test case setup/cleanup methods to override,
* Test case verification,
@@ -71,7 +62,6 @@ class TestSuite(object):
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
_logger: DTSLOG
- _test_cases_to_run: list[str]
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -86,24 +76,19 @@ def __init__(
self,
sut_node: SutNode,
tg_node: TGNode,
- test_cases: list[str],
):
"""Initialize the test suite testbed information and basic configuration.
- Process what test cases to run, find links between ports and set up
- default IP addresses to be used when configuring them.
+ Find links between ports and set up default IP addresses to be used when
+ configuring them.
Args:
sut_node: The SUT node where the test suite will run.
tg_node: The TG node where the test suite will run.
- test_cases: The list of test cases to execute.
- If empty, all test cases will be executed.
"""
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = getLogger(self.__class__.__name__)
- self._test_cases_to_run = test_cases
- self._test_cases_to_run.extend(SETTINGS.test_cases)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
@@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
return False
return True
-
- def _get_functional_test_cases(self) -> list[MethodType]:
- """Get all functional test cases defined in this TestSuite.
-
- Returns:
- The list of functional test cases of this TestSuite.
- """
- return self._get_test_cases(r"test_(?!perf_)")
-
- def _get_test_cases(self, test_case_regex: str) -> list[MethodType]:
- """Return a list of test cases matching test_case_regex.
-
- Returns:
- The list of test cases matching test_case_regex of this TestSuite.
- """
- self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.")
- filtered_test_cases = []
- for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod):
- if self._should_be_executed(test_case_name, test_case_regex):
- filtered_test_cases.append(test_case)
- cases_str = ", ".join((x.__name__ for x in filtered_test_cases))
- self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.")
- return filtered_test_cases
-
- def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool:
- """Check whether the test case should be scheduled to be executed."""
- match = bool(re.match(test_case_regex, test_case_name))
- if self._test_cases_to_run:
- return match and test_case_name in self._test_cases_to_run
-
- return match
-
-
-def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
- r"""Find all :class:`TestSuite`\s in a Python module.
-
- Args:
- testsuite_module_path: The path to the Python module.
-
- Returns:
- The list of :class:`TestSuite`\s found within the Python module.
-
- Raises:
- ConfigurationError: The test suite module was not found.
- """
-
- def is_test_suite(object: Any) -> bool:
- try:
- if issubclass(object, TestSuite) and object is not TestSuite:
- return True
- except TypeError:
- return False
- return False
-
- try:
- testcase_module = importlib.import_module(testsuite_module_path)
- except ModuleNotFoundError as e:
- raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e
- return [
- test_suite_class
- for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite)
- ]
diff --git a/dts/pyproject.toml b/dts/pyproject.toml
index 28bd970ae4..8eb92b4f11 100644
--- a/dts/pyproject.toml
+++ b/dts/pyproject.toml
@@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes"
format = "pylint"
max_line_length = 100
+[tool.pylama.linter.pycodestyle]
+ignore = "E203,W503"
+
[tool.pylama.linter.pydocstyle]
convention = "google"
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index 5e2bac14bd..7b2a0e97f8 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -21,7 +21,7 @@
from framework.utils import REGEX_FOR_PCI_ADDRESS
-class SmokeTests(TestSuite):
+class TestSmokeTests(TestSuite):
"""DPDK and infrastructure smoke test suite.
The test cases validate the most basic DPDK functionality needed for all other test suites.
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 4/7] dts: reorganize test result
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
` (2 preceding siblings ...)
2024-02-06 14:57 ` [PATCH v2 3/7] dts: filter test suites in executions Juraj Linkeš
@ 2024-02-06 14:57 ` Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš
` (2 subsequent siblings)
6 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
The current order of Result classes in the test_suite.py module is
guided by the needs of type hints, which is not as intuitively readable
as ordering them by the occurrences in code. The order goes from the
topmost level to lowermost:
BaseResult
DTSResult
ExecutionResult
BuildTargetResult
TestSuiteResult
TestCaseResult
This is the same order as they're used in the runner module and they're
also used in the same order between themselves in the test_result
module.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/test_result.py | 411 ++++++++++++++++++-----------------
1 file changed, 206 insertions(+), 205 deletions(-)
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 075195fd5b..abdbafab10 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -28,6 +28,7 @@
from dataclasses import dataclass
from enum import Enum, auto
from types import MethodType
+from typing import Union
from .config import (
OS,
@@ -129,58 +130,6 @@ def __bool__(self) -> bool:
return bool(self.result)
-class Statistics(dict):
- """How many test cases ended in which result state along some other basic information.
-
- Subclassing :class:`dict` provides a convenient way to format the data.
-
- The data are stored in the following keys:
-
- * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases.
- * **DPDK VERSION** (:class:`str`) -- The tested DPDK version.
- """
-
- def __init__(self, dpdk_version: str | None):
- """Extend the constructor with keys in which the data are stored.
-
- Args:
- dpdk_version: The version of tested DPDK.
- """
- super(Statistics, self).__init__()
- for result in Result:
- self[result.name] = 0
- self["PASS RATE"] = 0.0
- self["DPDK VERSION"] = dpdk_version
-
- def __iadd__(self, other: Result) -> "Statistics":
- """Add a Result to the final count.
-
- Example:
- stats: Statistics = Statistics() # empty Statistics
- stats += Result.PASS # add a Result to `stats`
-
- Args:
- other: The Result to add to this statistics object.
-
- Returns:
- The modified statistics object.
- """
- self[other.name] += 1
- self["PASS RATE"] = (
- float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result)
- )
- return self
-
- def __str__(self) -> str:
- """Each line contains the formatted key = value pair."""
- stats_str = ""
- for key, value in self.items():
- stats_str += f"{key:<12} = {value}\n"
- # according to docs, we should use \n when writing to text files
- # on all platforms
- return stats_str
-
-
class BaseResult(object):
"""Common data and behavior of DTS results.
@@ -245,7 +194,7 @@ def get_errors(self) -> list[Exception]:
"""
return self._get_setup_teardown_errors() + self._get_inner_errors()
- def add_stats(self, statistics: Statistics) -> None:
+ def add_stats(self, statistics: "Statistics") -> None:
"""Collate stats from the whole result hierarchy.
Args:
@@ -255,91 +204,149 @@ def add_stats(self, statistics: Statistics) -> None:
inner_result.add_stats(statistics)
-class TestCaseResult(BaseResult, FixtureResult):
- r"""The test case specific result.
+class DTSResult(BaseResult):
+ """Stores environment information and test results from a DTS run.
- Stores the result of the actual test case. This is done by adding an extra superclass
- in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and
- the class is itself a record of the test case.
+ * Execution level information, such as testbed and the test suite list,
+ * Build target level information, such as compiler, target OS and cpu,
+ * Test suite and test case results,
+ * All errors that are caught and recorded during DTS execution.
+
+ The information is stored hierarchically. This is the first level of the hierarchy
+ and as such is where the data form the whole hierarchy is collated or processed.
+
+ The internal list stores the results of all executions.
Attributes:
- test_case_name: The test case name.
+ dpdk_version: The DPDK version to record.
"""
- test_case_name: str
+ dpdk_version: str | None
+ _logger: DTSLOG
+ _errors: list[Exception]
+ _return_code: ErrorSeverity
+ _stats_result: Union["Statistics", None]
+ _stats_filename: str
- def __init__(self, test_case_name: str):
- """Extend the constructor with `test_case_name`.
+ def __init__(self, logger: DTSLOG):
+ """Extend the constructor with top-level specifics.
Args:
- test_case_name: The test case's name.
+ logger: The logger instance the whole result will use.
"""
- super(TestCaseResult, self).__init__()
- self.test_case_name = test_case_name
+ super(DTSResult, self).__init__()
+ self.dpdk_version = None
+ self._logger = logger
+ self._errors = []
+ self._return_code = ErrorSeverity.NO_ERR
+ self._stats_result = None
+ self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
- def update(self, result: Result, error: Exception | None = None) -> None:
- """Update the test case result.
+ def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult":
+ """Add and return the inner result (execution).
- This updates the result of the test case itself and doesn't affect
- the results of the setup and teardown steps in any way.
+ Args:
+ sut_node: The SUT node's test run configuration.
+
+ Returns:
+ The execution's result.
+ """
+ execution_result = ExecutionResult(sut_node)
+ self._inner_results.append(execution_result)
+ return execution_result
+
+ def add_error(self, error: Exception) -> None:
+ """Record an error that occurred outside any execution.
Args:
- result: The result of the test case.
- error: The error that occurred in case of a failure.
+ error: The exception to record.
"""
- self.result = result
- self.error = error
+ self._errors.append(error)
- def _get_inner_errors(self) -> list[Exception]:
- if self.error:
- return [self.error]
- return []
+ def process(self) -> None:
+ """Process the data after a whole DTS run.
- def add_stats(self, statistics: Statistics) -> None:
- r"""Add the test case result to statistics.
+ The data is added to inner objects during runtime and this object is not updated
+ at that time. This requires us to process the inner data after it's all been gathered.
- The base method goes through the hierarchy recursively and this method is here to stop
- the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree.
+ The processing gathers all errors and the statistics of test case results.
+ """
+ self._errors += self.get_errors()
+ if self._errors and self._logger:
+ self._logger.debug("Summary of errors:")
+ for error in self._errors:
+ self._logger.debug(repr(error))
- Args:
- statistics: The :class:`Statistics` object where the stats will be added.
+ self._stats_result = Statistics(self.dpdk_version)
+ self.add_stats(self._stats_result)
+ with open(self._stats_filename, "w+") as stats_file:
+ stats_file.write(str(self._stats_result))
+
+ def get_return_code(self) -> int:
+ """Go through all stored Exceptions and return the final DTS error code.
+
+ Returns:
+ The highest error code found.
"""
- statistics += self.result
+ for error in self._errors:
+ error_return_code = ErrorSeverity.GENERIC_ERR
+ if isinstance(error, DTSError):
+ error_return_code = error.severity
- def __bool__(self) -> bool:
- """The test case passed only if setup, teardown and the test case itself passed."""
- return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
+ if error_return_code > self._return_code:
+ self._return_code = error_return_code
+ return int(self._return_code)
-class TestSuiteResult(BaseResult):
- """The test suite specific result.
- The internal list stores the results of all test cases in a given test suite.
+class ExecutionResult(BaseResult):
+ """The execution specific result.
+
+ The internal list stores the results of all build targets in a given execution.
Attributes:
- suite_name: The test suite name.
+ sut_node: The SUT node used in the execution.
+ sut_os_name: The operating system of the SUT node.
+ sut_os_version: The operating system version of the SUT node.
+ sut_kernel_version: The operating system kernel version of the SUT node.
"""
- suite_name: str
+ sut_node: NodeConfiguration
+ sut_os_name: str
+ sut_os_version: str
+ sut_kernel_version: str
- def __init__(self, suite_name: str):
- """Extend the constructor with `suite_name`.
+ def __init__(self, sut_node: NodeConfiguration):
+ """Extend the constructor with the `sut_node`'s config.
Args:
- suite_name: The test suite's name.
+ sut_node: The SUT node's test run configuration used in the execution.
"""
- super(TestSuiteResult, self).__init__()
- self.suite_name = suite_name
+ super(ExecutionResult, self).__init__()
+ self.sut_node = sut_node
- def add_test_case(self, test_case_name: str) -> TestCaseResult:
- """Add and return the inner result (test case).
+ def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult":
+ """Add and return the inner result (build target).
+
+ Args:
+ build_target: The build target's test run configuration.
Returns:
- The test case's result.
+ The build target's result.
"""
- test_case_result = TestCaseResult(test_case_name)
- self._inner_results.append(test_case_result)
- return test_case_result
+ build_target_result = BuildTargetResult(build_target)
+ self._inner_results.append(build_target_result)
+ return build_target_result
+
+ def add_sut_info(self, sut_info: NodeInfo) -> None:
+ """Add SUT information gathered at runtime.
+
+ Args:
+ sut_info: The additional SUT node information.
+ """
+ self.sut_os_name = sut_info.os_name
+ self.sut_os_version = sut_info.os_version
+ self.sut_kernel_version = sut_info.kernel_version
class BuildTargetResult(BaseResult):
@@ -386,7 +393,7 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def add_test_suite(self, test_suite_name: str) -> TestSuiteResult:
+ def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult":
"""Add and return the inner result (test suite).
Returns:
@@ -397,146 +404,140 @@ def add_test_suite(self, test_suite_name: str) -> TestSuiteResult:
return test_suite_result
-class ExecutionResult(BaseResult):
- """The execution specific result.
+class TestSuiteResult(BaseResult):
+ """The test suite specific result.
- The internal list stores the results of all build targets in a given execution.
+ The internal list stores the results of all test cases in a given test suite.
Attributes:
- sut_node: The SUT node used in the execution.
- sut_os_name: The operating system of the SUT node.
- sut_os_version: The operating system version of the SUT node.
- sut_kernel_version: The operating system kernel version of the SUT node.
+ suite_name: The test suite name.
"""
- sut_node: NodeConfiguration
- sut_os_name: str
- sut_os_version: str
- sut_kernel_version: str
+ suite_name: str
- def __init__(self, sut_node: NodeConfiguration):
- """Extend the constructor with the `sut_node`'s config.
+ def __init__(self, suite_name: str):
+ """Extend the constructor with `suite_name`.
Args:
- sut_node: The SUT node's test run configuration used in the execution.
+ suite_name: The test suite's name.
"""
- super(ExecutionResult, self).__init__()
- self.sut_node = sut_node
-
- def add_build_target(self, build_target: BuildTargetConfiguration) -> BuildTargetResult:
- """Add and return the inner result (build target).
+ super(TestSuiteResult, self).__init__()
+ self.suite_name = suite_name
- Args:
- build_target: The build target's test run configuration.
+ def add_test_case(self, test_case_name: str) -> "TestCaseResult":
+ """Add and return the inner result (test case).
Returns:
- The build target's result.
- """
- build_target_result = BuildTargetResult(build_target)
- self._inner_results.append(build_target_result)
- return build_target_result
-
- def add_sut_info(self, sut_info: NodeInfo) -> None:
- """Add SUT information gathered at runtime.
-
- Args:
- sut_info: The additional SUT node information.
+ The test case's result.
"""
- self.sut_os_name = sut_info.os_name
- self.sut_os_version = sut_info.os_version
- self.sut_kernel_version = sut_info.kernel_version
+ test_case_result = TestCaseResult(test_case_name)
+ self._inner_results.append(test_case_result)
+ return test_case_result
-class DTSResult(BaseResult):
- """Stores environment information and test results from a DTS run.
-
- * Execution level information, such as testbed and the test suite list,
- * Build target level information, such as compiler, target OS and cpu,
- * Test suite and test case results,
- * All errors that are caught and recorded during DTS execution.
-
- The information is stored hierarchically. This is the first level of the hierarchy
- and as such is where the data form the whole hierarchy is collated or processed.
+class TestCaseResult(BaseResult, FixtureResult):
+ r"""The test case specific result.
- The internal list stores the results of all executions.
+ Stores the result of the actual test case. This is done by adding an extra superclass
+ in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and
+ the class is itself a record of the test case.
Attributes:
- dpdk_version: The DPDK version to record.
+ test_case_name: The test case name.
"""
- dpdk_version: str | None
- _logger: DTSLOG
- _errors: list[Exception]
- _return_code: ErrorSeverity
- _stats_result: Statistics | None
- _stats_filename: str
+ test_case_name: str
- def __init__(self, logger: DTSLOG):
- """Extend the constructor with top-level specifics.
+ def __init__(self, test_case_name: str):
+ """Extend the constructor with `test_case_name`.
Args:
- logger: The logger instance the whole result will use.
+ test_case_name: The test case's name.
"""
- super(DTSResult, self).__init__()
- self.dpdk_version = None
- self._logger = logger
- self._errors = []
- self._return_code = ErrorSeverity.NO_ERR
- self._stats_result = None
- self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
+ super(TestCaseResult, self).__init__()
+ self.test_case_name = test_case_name
- def add_execution(self, sut_node: NodeConfiguration) -> ExecutionResult:
- """Add and return the inner result (execution).
+ def update(self, result: Result, error: Exception | None = None) -> None:
+ """Update the test case result.
- Args:
- sut_node: The SUT node's test run configuration.
+ This updates the result of the test case itself and doesn't affect
+ the results of the setup and teardown steps in any way.
- Returns:
- The execution's result.
+ Args:
+ result: The result of the test case.
+ error: The error that occurred in case of a failure.
"""
- execution_result = ExecutionResult(sut_node)
- self._inner_results.append(execution_result)
- return execution_result
+ self.result = result
+ self.error = error
- def add_error(self, error: Exception) -> None:
- """Record an error that occurred outside any execution.
+ def _get_inner_errors(self) -> list[Exception]:
+ if self.error:
+ return [self.error]
+ return []
+
+ def add_stats(self, statistics: "Statistics") -> None:
+ r"""Add the test case result to statistics.
+
+ The base method goes through the hierarchy recursively and this method is here to stop
+ the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree.
Args:
- error: The exception to record.
+ statistics: The :class:`Statistics` object where the stats will be added.
"""
- self._errors.append(error)
+ statistics += self.result
- def process(self) -> None:
- """Process the data after a whole DTS run.
+ def __bool__(self) -> bool:
+ """The test case passed only if setup, teardown and the test case itself passed."""
+ return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
- The data is added to inner objects during runtime and this object is not updated
- at that time. This requires us to process the inner data after it's all been gathered.
- The processing gathers all errors and the statistics of test case results.
+class Statistics(dict):
+ """How many test cases ended in which result state along some other basic information.
+
+ Subclassing :class:`dict` provides a convenient way to format the data.
+
+ The data are stored in the following keys:
+
+ * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases.
+ * **DPDK VERSION** (:class:`str`) -- The tested DPDK version.
+ """
+
+ def __init__(self, dpdk_version: str | None):
+ """Extend the constructor with keys in which the data are stored.
+
+ Args:
+ dpdk_version: The version of tested DPDK.
"""
- self._errors += self.get_errors()
- if self._errors and self._logger:
- self._logger.debug("Summary of errors:")
- for error in self._errors:
- self._logger.debug(repr(error))
+ super(Statistics, self).__init__()
+ for result in Result:
+ self[result.name] = 0
+ self["PASS RATE"] = 0.0
+ self["DPDK VERSION"] = dpdk_version
- self._stats_result = Statistics(self.dpdk_version)
- self.add_stats(self._stats_result)
- with open(self._stats_filename, "w+") as stats_file:
- stats_file.write(str(self._stats_result))
+ def __iadd__(self, other: Result) -> "Statistics":
+ """Add a Result to the final count.
- def get_return_code(self) -> int:
- """Go through all stored Exceptions and return the final DTS error code.
+ Example:
+ stats: Statistics = Statistics() # empty Statistics
+ stats += Result.PASS # add a Result to `stats`
+
+ Args:
+ other: The Result to add to this statistics object.
Returns:
- The highest error code found.
+ The modified statistics object.
"""
- for error in self._errors:
- error_return_code = ErrorSeverity.GENERIC_ERR
- if isinstance(error, DTSError):
- error_return_code = error.severity
-
- if error_return_code > self._return_code:
- self._return_code = error_return_code
+ self[other.name] += 1
+ self["PASS RATE"] = (
+ float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result)
+ )
+ return self
- return int(self._return_code)
+ def __str__(self) -> str:
+ """Each line contains the formatted key = value pair."""
+ stats_str = ""
+ for key, value in self.items():
+ stats_str += f"{key:<12} = {value}\n"
+ # according to docs, we should use \n when writing to text files
+ # on all platforms
+ return stats_str
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 5/7] dts: block all test cases when earlier setup fails
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
` (3 preceding siblings ...)
2024-02-06 14:57 ` [PATCH v2 4/7] dts: reorganize test result Juraj Linkeš
@ 2024-02-06 14:57 ` Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 6/7] dts: refactor logging configuration Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 7/7] dts: improve test suite and case filtering Juraj Linkeš
6 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
In case of a failure before a test suite, the child results will be
recursively recorded as blocked, giving us a full report which was
missing previously.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 21 ++--
dts/framework/test_result.py | 186 +++++++++++++++++++++++++----------
2 files changed, 148 insertions(+), 59 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 3e95cf9e26..f58b0adc13 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -60,13 +60,15 @@ class DTSRunner:
Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult`
or one of its subclasses. The test case results are also recorded.
- If an error occurs, the current stage is aborted, the error is recorded and the run continues in
- the next iteration of the same stage. The return code is the highest `severity` of all
+ If an error occurs, the current stage is aborted, the error is recorded, everything in
+ the inner stages is marked as blocked and the run continues in the next iteration
+ of the same stage. The return code is the highest `severity` of all
:class:`~.framework.exception.DTSError`\s.
Example:
- An error occurs in a build target setup. The current build target is aborted and the run
- continues with the next build target. If the errored build target was the last one in the
+ An error occurs in a build target setup. The current build target is aborted,
+ all test suites and their test cases are marked as blocked and the run continues
+ with the next build target. If the errored build target was the last one in the
given execution, the next execution begins.
"""
@@ -100,6 +102,10 @@ def run(self):
test case within the test suite is set up, executed and torn down. After all test cases
have been executed, the test suite is torn down and the next build target will be tested.
+ In order to properly mark test suites and test cases as blocked in case of a failure,
+ we need to have discovered which test suites and test cases to run before any failures
+ happen. The discovery happens at the earliest point at the start of each execution.
+
All the nested steps look like this:
#. Execution setup
@@ -134,11 +140,12 @@ def run(self):
self._logger.info(
f"Running execution with SUT '{execution.system_under_test_node.name}'."
)
- execution_result = self._result.add_execution(execution.system_under_test_node)
+ execution_result = self._result.add_execution(execution)
try:
test_suites_with_cases = self._get_test_suites_with_cases(
execution.test_suites, execution.func, execution.perf
)
+ execution_result.test_suites_with_cases = test_suites_with_cases
except Exception as e:
self._logger.exception(
f"Invalid test suite configuration found: " f"{execution.test_suites}."
@@ -486,9 +493,7 @@ def _run_test_suites(
"""
end_build_target = False
for test_suite_with_cases in test_suites_with_cases:
- test_suite_result = build_target_result.add_test_suite(
- test_suite_with_cases.test_suite_class.__name__
- )
+ test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
try:
self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
except BlockingTestSuiteError as e:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index abdbafab10..eedb2d20ee 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -37,7 +37,7 @@
BuildTargetInfo,
Compiler,
CPUType,
- NodeConfiguration,
+ ExecutionConfiguration,
NodeInfo,
TestSuiteConfig,
)
@@ -88,6 +88,8 @@ class Result(Enum):
ERROR = auto()
#:
SKIP = auto()
+ #:
+ BLOCK = auto()
def __bool__(self) -> bool:
"""Only PASS is True."""
@@ -141,21 +143,26 @@ class BaseResult(object):
Attributes:
setup_result: The result of the setup of the particular stage.
teardown_result: The results of the teardown of the particular stage.
+ child_results: The results of the descendants in the results hierarchy.
"""
setup_result: FixtureResult
teardown_result: FixtureResult
- _inner_results: MutableSequence["BaseResult"]
+ child_results: MutableSequence["BaseResult"]
def __init__(self):
"""Initialize the constructor."""
self.setup_result = FixtureResult()
self.teardown_result = FixtureResult()
- self._inner_results = []
+ self.child_results = []
def update_setup(self, result: Result, error: Exception | None = None) -> None:
"""Store the setup result.
+ If the result is :attr:`~Result.BLOCK`, :attr:`~Result.ERROR` or :attr:`~Result.FAIL`,
+ then the corresponding child results in result hierarchy
+ are also marked with :attr:`~Result.BLOCK`.
+
Args:
result: The result of the setup.
error: The error that occurred in case of a failure.
@@ -163,6 +170,16 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
self.setup_result.result = result
self.setup_result.error = error
+ if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
+ self.update_teardown(Result.BLOCK)
+ self._block_result()
+
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed.
+
+ The blocking of child results should be done in overloaded methods.
+ """
+
def update_teardown(self, result: Result, error: Exception | None = None) -> None:
"""Store the teardown result.
@@ -181,10 +198,8 @@ def _get_setup_teardown_errors(self) -> list[Exception]:
errors.append(self.teardown_result.error)
return errors
- def _get_inner_errors(self) -> list[Exception]:
- return [
- error for inner_result in self._inner_results for error in inner_result.get_errors()
- ]
+ def _get_child_errors(self) -> list[Exception]:
+ return [error for child_result in self.child_results for error in child_result.get_errors()]
def get_errors(self) -> list[Exception]:
"""Compile errors from the whole result hierarchy.
@@ -192,7 +207,7 @@ def get_errors(self) -> list[Exception]:
Returns:
The errors from setup, teardown and all errors found in the whole result hierarchy.
"""
- return self._get_setup_teardown_errors() + self._get_inner_errors()
+ return self._get_setup_teardown_errors() + self._get_child_errors()
def add_stats(self, statistics: "Statistics") -> None:
"""Collate stats from the whole result hierarchy.
@@ -200,8 +215,8 @@ def add_stats(self, statistics: "Statistics") -> None:
Args:
statistics: The :class:`Statistics` object where the stats will be collated.
"""
- for inner_result in self._inner_results:
- inner_result.add_stats(statistics)
+ for child_result in self.child_results:
+ child_result.add_stats(statistics)
class DTSResult(BaseResult):
@@ -242,18 +257,18 @@ def __init__(self, logger: DTSLOG):
self._stats_result = None
self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
- def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult":
- """Add and return the inner result (execution).
+ def add_execution(self, execution: ExecutionConfiguration) -> "ExecutionResult":
+ """Add and return the child result (execution).
Args:
- sut_node: The SUT node's test run configuration.
+ execution: The execution's test run configuration.
Returns:
The execution's result.
"""
- execution_result = ExecutionResult(sut_node)
- self._inner_results.append(execution_result)
- return execution_result
+ result = ExecutionResult(execution)
+ self.child_results.append(result)
+ return result
def add_error(self, error: Exception) -> None:
"""Record an error that occurred outside any execution.
@@ -266,8 +281,8 @@ def add_error(self, error: Exception) -> None:
def process(self) -> None:
"""Process the data after a whole DTS run.
- The data is added to inner objects during runtime and this object is not updated
- at that time. This requires us to process the inner data after it's all been gathered.
+ The data is added to child objects during runtime and this object is not updated
+ at that time. This requires us to process the child data after it's all been gathered.
The processing gathers all errors and the statistics of test case results.
"""
@@ -305,28 +320,30 @@ class ExecutionResult(BaseResult):
The internal list stores the results of all build targets in a given execution.
Attributes:
- sut_node: The SUT node used in the execution.
sut_os_name: The operating system of the SUT node.
sut_os_version: The operating system version of the SUT node.
sut_kernel_version: The operating system kernel version of the SUT node.
"""
- sut_node: NodeConfiguration
sut_os_name: str
sut_os_version: str
sut_kernel_version: str
+ _config: ExecutionConfiguration
+ _parent_result: DTSResult
+ _test_suites_with_cases: list[TestSuiteWithCases]
- def __init__(self, sut_node: NodeConfiguration):
- """Extend the constructor with the `sut_node`'s config.
+ def __init__(self, execution: ExecutionConfiguration):
+ """Extend the constructor with the execution's config and DTSResult.
Args:
- sut_node: The SUT node's test run configuration used in the execution.
+ execution: The execution's test run configuration.
"""
super(ExecutionResult, self).__init__()
- self.sut_node = sut_node
+ self._config = execution
+ self._test_suites_with_cases = []
def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult":
- """Add and return the inner result (build target).
+ """Add and return the child result (build target).
Args:
build_target: The build target's test run configuration.
@@ -334,9 +351,34 @@ def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTarg
Returns:
The build target's result.
"""
- build_target_result = BuildTargetResult(build_target)
- self._inner_results.append(build_target_result)
- return build_target_result
+ result = BuildTargetResult(
+ self._test_suites_with_cases,
+ build_target,
+ )
+ self.child_results.append(result)
+ return result
+
+ @property
+ def test_suites_with_cases(self) -> list[TestSuiteWithCases]:
+ """The test suites with test cases to be executed in this execution.
+
+ The test suites can only be assigned once.
+
+ Returns:
+ The list of test suites with test cases. If an error occurs between
+ the initialization of :class:`ExecutionResult` and assigning test cases to the instance,
+ return an empty list, representing that we don't know what to execute.
+ """
+ return self._test_suites_with_cases
+
+ @test_suites_with_cases.setter
+ def test_suites_with_cases(self, test_suites_with_cases: list[TestSuiteWithCases]) -> None:
+ if self._test_suites_with_cases:
+ raise ValueError(
+ "Attempted to assign test suites to an execution result "
+ "which already has test suites."
+ )
+ self._test_suites_with_cases = test_suites_with_cases
def add_sut_info(self, sut_info: NodeInfo) -> None:
"""Add SUT information gathered at runtime.
@@ -348,6 +390,12 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
self.sut_os_version = sut_info.os_version
self.sut_kernel_version = sut_info.kernel_version
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ for build_target in self._config.build_targets:
+ child_result = self.add_build_target(build_target)
+ child_result.update_setup(Result.BLOCK)
+
class BuildTargetResult(BaseResult):
"""The build target specific result.
@@ -369,11 +417,17 @@ class BuildTargetResult(BaseResult):
compiler: Compiler
compiler_version: str | None
dpdk_version: str | None
+ _test_suites_with_cases: list[TestSuiteWithCases]
- def __init__(self, build_target: BuildTargetConfiguration):
- """Extend the constructor with the `build_target`'s build target config.
+ def __init__(
+ self,
+ test_suites_with_cases: list[TestSuiteWithCases],
+ build_target: BuildTargetConfiguration,
+ ):
+ """Extend the constructor with the build target's config and ExecutionResult.
Args:
+ test_suites_with_cases: The test suites with test cases to be run in this build target.
build_target: The build target's test run configuration.
"""
super(BuildTargetResult, self).__init__()
@@ -383,6 +437,23 @@ def __init__(self, build_target: BuildTargetConfiguration):
self.compiler = build_target.compiler
self.compiler_version = None
self.dpdk_version = None
+ self._test_suites_with_cases = test_suites_with_cases
+
+ def add_test_suite(
+ self,
+ test_suite_with_cases: TestSuiteWithCases,
+ ) -> "TestSuiteResult":
+ """Add and return the child result (test suite).
+
+ Args:
+ test_suite_with_cases: The test suite with test cases.
+
+ Returns:
+ The test suite's result.
+ """
+ result = TestSuiteResult(test_suite_with_cases)
+ self.child_results.append(result)
+ return result
def add_build_target_info(self, versions: BuildTargetInfo) -> None:
"""Add information about the build target gathered at runtime.
@@ -393,15 +464,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult":
- """Add and return the inner result (test suite).
-
- Returns:
- The test suite's result.
- """
- test_suite_result = TestSuiteResult(test_suite_name)
- self._inner_results.append(test_suite_result)
- return test_suite_result
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ for test_suite_with_cases in self._test_suites_with_cases:
+ child_result = self.add_test_suite(test_suite_with_cases)
+ child_result.update_setup(Result.BLOCK)
class TestSuiteResult(BaseResult):
@@ -410,29 +477,42 @@ class TestSuiteResult(BaseResult):
The internal list stores the results of all test cases in a given test suite.
Attributes:
- suite_name: The test suite name.
+ test_suite_name: The test suite name.
"""
- suite_name: str
+ test_suite_name: str
+ _test_suite_with_cases: TestSuiteWithCases
+ _parent_result: BuildTargetResult
+ _child_configs: list[str]
- def __init__(self, suite_name: str):
- """Extend the constructor with `suite_name`.
+ def __init__(self, test_suite_with_cases: TestSuiteWithCases):
+ """Extend the constructor with test suite's config and BuildTargetResult.
Args:
- suite_name: The test suite's name.
+ test_suite_with_cases: The test suite with test cases.
"""
super(TestSuiteResult, self).__init__()
- self.suite_name = suite_name
+ self.test_suite_name = test_suite_with_cases.test_suite_class.__name__
+ self._test_suite_with_cases = test_suite_with_cases
def add_test_case(self, test_case_name: str) -> "TestCaseResult":
- """Add and return the inner result (test case).
+ """Add and return the child result (test case).
+
+ Args:
+ test_case_name: The name of the test case.
Returns:
The test case's result.
"""
- test_case_result = TestCaseResult(test_case_name)
- self._inner_results.append(test_case_result)
- return test_case_result
+ result = TestCaseResult(test_case_name)
+ self.child_results.append(result)
+ return result
+
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ for test_case_method in self._test_suite_with_cases.test_cases:
+ child_result = self.add_test_case(test_case_method.__name__)
+ child_result.update_setup(Result.BLOCK)
class TestCaseResult(BaseResult, FixtureResult):
@@ -449,7 +529,7 @@ class TestCaseResult(BaseResult, FixtureResult):
test_case_name: str
def __init__(self, test_case_name: str):
- """Extend the constructor with `test_case_name`.
+ """Extend the constructor with test case's name and TestSuiteResult.
Args:
test_case_name: The test case's name.
@@ -470,7 +550,7 @@ def update(self, result: Result, error: Exception | None = None) -> None:
self.result = result
self.error = error
- def _get_inner_errors(self) -> list[Exception]:
+ def _get_child_errors(self) -> list[Exception]:
if self.error:
return [self.error]
return []
@@ -486,6 +566,10 @@ def add_stats(self, statistics: "Statistics") -> None:
"""
statistics += self.result
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ self.update(Result.BLOCK)
+
def __bool__(self) -> bool:
"""The test case passed only if setup, teardown and the test case itself passed."""
return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 6/7] dts: refactor logging configuration
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
` (4 preceding siblings ...)
2024-02-06 14:57 ` [PATCH v2 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš
@ 2024-02-06 14:57 ` Juraj Linkeš
2024-02-12 16:45 ` Jeremy Spewock
2024-02-06 14:57 ` [PATCH v2 7/7] dts: improve test suite and case filtering Juraj Linkeš
6 siblings, 1 reply; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
Remove unused parts of the code and add useful features:
1. Add DTS execution stages such as execution and test suite to better
identify where in the DTS lifecycle we are when investigating logs,
2. Logging to separate files in specific stages, which is mainly useful
for having test suite logs in additional separate files.
3. Remove the dependence on the settings module which enhances the
usefulness of the logger module, as it can now be imported in more
modules.
The execution stages and the files to log to are the same for all DTS
loggers. To achieve this, we have one DTS root logger which should be
used for handling stage switching and all other loggers are children of
this DTS root logger. The DTS root logger is the one where we change the
behavior of all loggers (the stage and which files to log to) and the
child loggers just log messages under a different name.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/logger.py | 235 +++++++++++-------
dts/framework/remote_session/__init__.py | 6 +-
.../interactive_remote_session.py | 6 +-
.../remote_session/interactive_shell.py | 6 +-
.../remote_session/remote_session.py | 8 +-
dts/framework/runner.py | 19 +-
dts/framework/test_result.py | 6 +-
dts/framework/test_suite.py | 6 +-
dts/framework/testbed_model/node.py | 11 +-
dts/framework/testbed_model/os_session.py | 7 +-
.../traffic_generator/traffic_generator.py | 6 +-
dts/main.py | 1 -
12 files changed, 183 insertions(+), 134 deletions(-)
diff --git a/dts/framework/logger.py b/dts/framework/logger.py
index cfa6e8cd72..568edad82d 100644
--- a/dts/framework/logger.py
+++ b/dts/framework/logger.py
@@ -5,141 +5,186 @@
"""DTS logger module.
-DTS framework and TestSuite logs are saved in different log files.
+The module provides several additional features:
+
+ * The storage of DTS execution stages,
+ * Logging to console, a human-readable log file and a machine-readable log file,
+ * Optional log files for specific stages.
"""
import logging
-import os.path
-from typing import TypedDict
+from enum import auto
+from logging import FileHandler, StreamHandler
+from pathlib import Path
+from typing import ClassVar
-from .settings import SETTINGS
+from .utils import StrEnum
date_fmt = "%Y/%m/%d %H:%M:%S"
-stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s"
+dts_root_logger_name = "dts"
+
+
+class DtsStage(StrEnum):
+ """The DTS execution stage."""
+ #:
+ pre_execution = auto()
+ #:
+ execution = auto()
+ #:
+ build_target = auto()
+ #:
+ suite = auto()
+ #:
+ post_execution = auto()
-class DTSLOG(logging.LoggerAdapter):
- """DTS logger adapter class for framework and testsuites.
- The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment
- variable control the verbosity of output. If enabled, all messages will be emitted to the
- console.
+class DTSLogger(logging.Logger):
+ """The DTS logger class.
- The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment
- variable modify the directory where the logs will be stored.
+ The class extends the :class:`~logging.Logger` class to add the DTS execution stage information
+ to log records. The stage is common to all loggers, so it's stored in a class variable.
- Attributes:
- node: The additional identifier. Currently unused.
- sh: The handler which emits logs to console.
- fh: The handler which emits logs to a file.
- verbose_fh: Just as fh, but logs with a different, more verbose, format.
+ Any time we switch to a new stage, we have the ability to log to an additional log file along
+ with a supplementary log file with machine-readable format. These two log files are used until
+ a new stage switch occurs. This is useful mainly for logging per test suite.
"""
- _logger: logging.Logger
- node: str
- sh: logging.StreamHandler
- fh: logging.FileHandler
- verbose_fh: logging.FileHandler
+ _stage: ClassVar[DtsStage] = DtsStage.pre_execution
+ _extra_file_handlers: list[FileHandler] = []
- def __init__(self, logger: logging.Logger, node: str = "suite"):
- """Extend the constructor with additional handlers.
+ def __init__(self, *args, **kwargs):
+ """Extend the constructor with extra file handlers."""
+ self._extra_file_handlers = []
+ super().__init__(*args, **kwargs)
- One handler logs to the console, the other one to a file, with either a regular or verbose
- format.
+ def makeRecord(self, *args, **kwargs):
+ """Generates a record with additional stage information.
- Args:
- logger: The logger from which to create the logger adapter.
- node: An additional identifier. Currently unused.
+ This is the default method for the :class:`~logging.Logger` class. We extend it
+ to add stage information to the record.
+
+ :meta private:
+
+ Returns:
+ record: The generated record with the stage information.
"""
- self._logger = logger
- # 1 means log everything, this will be used by file handlers if their level
- # is not set
- self._logger.setLevel(1)
+ record = super().makeRecord(*args, **kwargs)
+ record.stage = DTSLogger._stage
+ return record
+
+ def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None:
+ """Add logger handlers to the DTS root logger.
+
+ This method should be called only on the DTS root logger.
+ The log records from child loggers will propagate to these handlers.
+
+ Three handlers are added:
- self.node = node
+ * A console handler,
+ * A file handler,
+ * A supplementary file handler with machine-readable logs
+ containing more debug information.
- # add handler to emit to stdout
- sh = logging.StreamHandler()
+ All log messages will be logged to files. The log level of the console handler
+ is configurable with `verbose`.
+
+ Args:
+ verbose: If :data:`True`, log all messages to the console.
+ If :data:`False`, log to console with the :data:`logging.INFO` level.
+ output_dir: The directory where the log files will be located.
+ The names of the log files correspond to the name of the logger instance.
+ """
+ self.setLevel(1)
+
+ sh = StreamHandler()
sh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
- sh.setLevel(logging.INFO) # console handler default level
+ if not verbose:
+ sh.setLevel(logging.INFO)
+ self.addHandler(sh)
- if SETTINGS.verbose is True:
- sh.setLevel(logging.DEBUG)
+ self._add_file_handlers(Path(output_dir, self.name))
- self._logger.addHandler(sh)
- self.sh = sh
+ def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None:
+ """Set the DTS execution stage and optionally log to files.
- # prepare the output folder
- if not os.path.exists(SETTINGS.output_dir):
- os.mkdir(SETTINGS.output_dir)
+ Set the DTS execution stage of the DTSLog class and optionally add
+ file handlers to the instance if the log file name is provided.
- logging_path_prefix = os.path.join(SETTINGS.output_dir, node)
+ The file handlers log all messages. One is a regular human-readable log file and
+ the other one is a machine-readable log file with extra debug information.
- fh = logging.FileHandler(f"{logging_path_prefix}.log")
- fh.setFormatter(
- logging.Formatter(
- fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
- datefmt=date_fmt,
- )
- )
+ Args:
+ stage: The DTS stage to set.
+ log_file_path: An optional path of the log file to use. This should be a full path
+ (either relative or absolute) without suffix (which will be appended).
+ """
+ self._remove_extra_file_handlers()
+
+ if DTSLogger._stage != stage:
+ self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.")
+ DTSLogger._stage = stage
+
+ if log_file_path:
+ self._extra_file_handlers.extend(self._add_file_handlers(log_file_path))
+
+ def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]:
+ """Add file handlers to the DTS root logger.
+
+ Add two type of file handlers:
+
+ * A regular file handler with suffix ".log",
+ * A machine-readable file handler with suffix ".verbose.log".
+ This format provides extensive information for debugging and detailed analysis.
+
+ Args:
+ log_file_path: The full path to the log file without suffix.
+
+ Returns:
+ The newly created file handlers.
- self._logger.addHandler(fh)
- self.fh = fh
+ """
+ fh = FileHandler(f"{log_file_path}.log")
+ fh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
+ self.addHandler(fh)
- # This outputs EVERYTHING, intended for post-mortem debugging
- # Also optimized for processing via AWK (awk -F '|' ...)
- verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log")
+ verbose_fh = FileHandler(f"{log_file_path}.verbose.log")
verbose_fh.setFormatter(
logging.Formatter(
- fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
+ "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
"%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s",
datefmt=date_fmt,
)
)
+ self.addHandler(verbose_fh)
- self._logger.addHandler(verbose_fh)
- self.verbose_fh = verbose_fh
-
- super(DTSLOG, self).__init__(self._logger, dict(node=self.node))
-
- def logger_exit(self) -> None:
- """Remove the stream handler and the logfile handler."""
- for handler in (self.sh, self.fh, self.verbose_fh):
- handler.flush()
- self._logger.removeHandler(handler)
-
-
-class _LoggerDictType(TypedDict):
- logger: DTSLOG
- name: str
- node: str
-
+ return [fh, verbose_fh]
-# List for saving all loggers in use
-_Loggers: list[_LoggerDictType] = []
+ def _remove_extra_file_handlers(self) -> None:
+ """Remove any extra file handlers that have been added to the logger."""
+ if self._extra_file_handlers:
+ for extra_file_handler in self._extra_file_handlers:
+ self.removeHandler(extra_file_handler)
+ self._extra_file_handlers = []
-def getLogger(name: str, node: str = "suite") -> DTSLOG:
- """Get DTS logger adapter identified by name and node.
- An existing logger will be returned if one with the exact name and node already exists.
- A new one will be created and stored otherwise.
+def get_dts_logger(name: str = None) -> DTSLogger:
+ """Return a DTS logger instance identified by `name`.
Args:
- name: The name of the logger.
- node: An additional identifier for the logger.
+ name: If :data:`None`, return the DTS root logger.
+ If specified, return a child of the DTS root logger.
Returns:
- A logger uniquely identified by both name and node.
+ The DTS root logger or a child logger identified by `name`.
"""
- global _Loggers
- # return saved logger
- logger: _LoggerDictType
- for logger in _Loggers:
- if logger["name"] == name and logger["node"] == node:
- return logger["logger"]
-
- # return new logger
- dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node)
- _Loggers.append({"logger": dts_logger, "name": name, "node": node})
- return dts_logger
+ logging.setLoggerClass(DTSLogger)
+ if name:
+ name = f"{dts_root_logger_name}.{name}"
+ else:
+ name = dts_root_logger_name
+ logger = logging.getLogger(name)
+ logging.setLoggerClass(logging.Logger)
+ return logger # type: ignore[return-value]
diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
index 51a01d6b5e..1910c81c3c 100644
--- a/dts/framework/remote_session/__init__.py
+++ b/dts/framework/remote_session/__init__.py
@@ -15,7 +15,7 @@
# pylama:ignore=W0611
from framework.config import NodeConfiguration
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from .interactive_remote_session import InteractiveRemoteSession
from .interactive_shell import InteractiveShell
@@ -26,7 +26,7 @@
def create_remote_session(
- node_config: NodeConfiguration, name: str, logger: DTSLOG
+ node_config: NodeConfiguration, name: str, logger: DTSLogger
) -> RemoteSession:
"""Factory for non-interactive remote sessions.
@@ -45,7 +45,7 @@ def create_remote_session(
def create_interactive_session(
- node_config: NodeConfiguration, logger: DTSLOG
+ node_config: NodeConfiguration, logger: DTSLogger
) -> InteractiveRemoteSession:
"""Factory for interactive remote sessions.
diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py
index 1cc82e3377..c50790db79 100644
--- a/dts/framework/remote_session/interactive_remote_session.py
+++ b/dts/framework/remote_session/interactive_remote_session.py
@@ -16,7 +16,7 @@
from framework.config import NodeConfiguration
from framework.exception import SSHConnectionError
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
class InteractiveRemoteSession:
@@ -50,11 +50,11 @@ class InteractiveRemoteSession:
username: str
password: str
session: SSHClient
- _logger: DTSLOG
+ _logger: DTSLogger
_node_config: NodeConfiguration
_transport: Transport | None
- def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None:
+ def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None:
"""Connect to the node during initialization.
Args:
diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py
index b158f963b6..5cfe202e15 100644
--- a/dts/framework/remote_session/interactive_shell.py
+++ b/dts/framework/remote_session/interactive_shell.py
@@ -20,7 +20,7 @@
from paramiko import Channel, SSHClient, channel # type: ignore[import]
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from framework.settings import SETTINGS
@@ -38,7 +38,7 @@ class InteractiveShell(ABC):
_stdin: channel.ChannelStdinFile
_stdout: channel.ChannelFile
_ssh_channel: Channel
- _logger: DTSLOG
+ _logger: DTSLogger
_timeout: float
_app_args: str
@@ -61,7 +61,7 @@ class InteractiveShell(ABC):
def __init__(
self,
interactive_session: SSHClient,
- logger: DTSLOG,
+ logger: DTSLogger,
get_privileged_command: Callable[[str], str] | None,
app_args: str = "",
timeout: float = SETTINGS.timeout,
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
index 2059f9a981..a69dc99400 100644
--- a/dts/framework/remote_session/remote_session.py
+++ b/dts/framework/remote_session/remote_session.py
@@ -9,14 +9,13 @@
the structure of the result of a command execution.
"""
-
import dataclasses
from abc import ABC, abstractmethod
from pathlib import PurePath
from framework.config import NodeConfiguration
from framework.exception import RemoteCommandExecutionError
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from framework.settings import SETTINGS
@@ -75,14 +74,14 @@ class RemoteSession(ABC):
username: str
password: str
history: list[CommandResult]
- _logger: DTSLOG
+ _logger: DTSLogger
_node_config: NodeConfiguration
def __init__(
self,
node_config: NodeConfiguration,
session_name: str,
- logger: DTSLOG,
+ logger: DTSLogger,
):
"""Connect to the node during initialization.
@@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None:
Args:
force: Force the closure of the connection. This may not clean up all resources.
"""
- self._logger.logger_exit()
self._close(force)
@abstractmethod
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index f58b0adc13..035e3368ef 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -19,9 +19,10 @@
import importlib
import inspect
-import logging
+import os
import re
import sys
+from pathlib import Path
from types import MethodType
from typing import Iterable
@@ -38,7 +39,7 @@
SSHTimeoutError,
TestCaseVerifyError,
)
-from .logger import DTSLOG, getLogger
+from .logger import DTSLogger, DtsStage, get_dts_logger
from .settings import SETTINGS
from .test_result import (
BuildTargetResult,
@@ -73,7 +74,7 @@ class DTSRunner:
"""
_configuration: Configuration
- _logger: DTSLOG
+ _logger: DTSLogger
_result: DTSResult
_test_suite_class_prefix: str
_test_suite_module_prefix: str
@@ -83,7 +84,10 @@ class DTSRunner:
def __init__(self):
"""Initialize the instance with configuration, logger, result and string constants."""
self._configuration = load_config()
- self._logger = getLogger("DTSRunner")
+ self._logger = get_dts_logger()
+ if not os.path.exists(SETTINGS.output_dir):
+ os.makedirs(SETTINGS.output_dir)
+ self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir)
self._result = DTSResult(self._logger)
self._test_suite_class_prefix = "Test"
self._test_suite_module_prefix = "tests.TestSuite_"
@@ -137,6 +141,7 @@ def run(self):
# for all Execution sections
for execution in self._configuration.executions:
+ self._logger.set_stage(DtsStage.execution)
self._logger.info(
f"Running execution with SUT '{execution.system_under_test_node.name}'."
)
@@ -164,6 +169,7 @@ def run(self):
finally:
try:
+ self._logger.set_stage(DtsStage.post_execution)
for node in (sut_nodes | tg_nodes).values():
node.close()
self._result.update_teardown(Result.PASS)
@@ -419,6 +425,7 @@ def _run_execution(
finally:
try:
+ self._logger.set_stage(DtsStage.execution)
sut_node.tear_down_execution()
execution_result.update_teardown(Result.PASS)
except Exception as e:
@@ -447,6 +454,7 @@ def _run_build_target(
with the current build target.
test_suites_with_cases: The test suites with test cases to run.
"""
+ self._logger.set_stage(DtsStage.build_target)
self._logger.info(f"Running build target '{build_target.name}'.")
try:
@@ -463,6 +471,7 @@ def _run_build_target(
finally:
try:
+ self._logger.set_stage(DtsStage.build_target)
sut_node.tear_down_build_target()
build_target_result.update_teardown(Result.PASS)
except Exception as e:
@@ -535,6 +544,7 @@ def _run_test_suite(
BlockingTestSuiteError: If a blocking test suite fails.
"""
test_suite_name = test_suite_with_cases.test_suite_class.__name__
+ self._logger.set_stage(DtsStage.suite, Path(SETTINGS.output_dir, test_suite_name))
test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
try:
self._logger.info(f"Starting test suite setup: {test_suite_name}")
@@ -683,5 +693,4 @@ def _exit_dts(self) -> None:
if self._logger:
self._logger.info("DTS execution has ended.")
- logging.shutdown()
sys.exit(self._result.get_return_code())
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index eedb2d20ee..28f84fd793 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -42,7 +42,7 @@
TestSuiteConfig,
)
from .exception import DTSError, ErrorSeverity
-from .logger import DTSLOG
+from .logger import DTSLogger
from .settings import SETTINGS
from .test_suite import TestSuite
@@ -237,13 +237,13 @@ class DTSResult(BaseResult):
"""
dpdk_version: str | None
- _logger: DTSLOG
+ _logger: DTSLogger
_errors: list[Exception]
_return_code: ErrorSeverity
_stats_result: Union["Statistics", None]
_stats_filename: str
- def __init__(self, logger: DTSLOG):
+ def __init__(self, logger: DTSLogger):
"""Extend the constructor with top-level specifics.
Args:
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index f9fe88093e..365f80e21a 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -21,7 +21,7 @@
from scapy.packet import Packet, Padding # type: ignore[import]
from .exception import TestCaseVerifyError
-from .logger import DTSLOG, getLogger
+from .logger import DTSLogger, get_dts_logger
from .testbed_model import Port, PortLink, SutNode, TGNode
from .utils import get_packet_summaries
@@ -61,7 +61,7 @@ class TestSuite(object):
#: Whether the test suite is blocking. A failure of a blocking test suite
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
- _logger: DTSLOG
+ _logger: DTSLogger
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -88,7 +88,7 @@ def __init__(
"""
self.sut_node = sut_node
self.tg_node = tg_node
- self._logger = getLogger(self.__class__.__name__)
+ self._logger = get_dts_logger(self.__class__.__name__)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
index 1a55fadf78..74061f6262 100644
--- a/dts/framework/testbed_model/node.py
+++ b/dts/framework/testbed_model/node.py
@@ -23,7 +23,7 @@
NodeConfiguration,
)
from framework.exception import ConfigurationError
-from framework.logger import DTSLOG, getLogger
+from framework.logger import DTSLogger, get_dts_logger
from framework.settings import SETTINGS
from .cpu import (
@@ -63,7 +63,7 @@ class Node(ABC):
name: str
lcores: list[LogicalCore]
ports: list[Port]
- _logger: DTSLOG
+ _logger: DTSLogger
_other_sessions: list[OSSession]
_execution_config: ExecutionConfiguration
virtual_devices: list[VirtualDevice]
@@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration):
"""
self.config = node_config
self.name = node_config.name
- self._logger = getLogger(self.name)
+ self._logger = get_dts_logger(self.name)
self.main_session = create_session(self.config, self.name, self._logger)
self._logger.info(f"Connected to node: {self.name}")
@@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession:
connection = create_session(
self.config,
session_name,
- getLogger(session_name, node=self.name),
+ get_dts_logger(session_name),
)
self._other_sessions.append(connection)
return connection
@@ -299,7 +299,6 @@ def close(self) -> None:
self.main_session.close()
for session in self._other_sessions:
session.close()
- self._logger.logger_exit()
@staticmethod
def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
@@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
return func
-def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession:
+def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession:
"""Factory for OS-aware sessions.
Args:
diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
index ac6bb5e112..6983aa4a77 100644
--- a/dts/framework/testbed_model/os_session.py
+++ b/dts/framework/testbed_model/os_session.py
@@ -21,7 +21,6 @@
the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux
and other commands for other OSs. It also translates the path to match the underlying OS.
"""
-
from abc import ABC, abstractmethod
from collections.abc import Iterable
from ipaddress import IPv4Interface, IPv6Interface
@@ -29,7 +28,7 @@
from typing import Type, TypeVar, Union
from framework.config import Architecture, NodeConfiguration, NodeInfo
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from framework.remote_session import (
CommandResult,
InteractiveRemoteSession,
@@ -62,7 +61,7 @@ class OSSession(ABC):
_config: NodeConfiguration
name: str
- _logger: DTSLOG
+ _logger: DTSLogger
remote_session: RemoteSession
interactive_session: InteractiveRemoteSession
@@ -70,7 +69,7 @@ def __init__(
self,
node_config: NodeConfiguration,
name: str,
- logger: DTSLOG,
+ logger: DTSLogger,
):
"""Initialize the OS-aware session.
diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
index c49fbff488..d86d7fb532 100644
--- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py
+++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
@@ -13,7 +13,7 @@
from scapy.packet import Packet # type: ignore[import]
from framework.config import TrafficGeneratorConfig
-from framework.logger import DTSLOG, getLogger
+from framework.logger import DTSLogger, get_dts_logger
from framework.testbed_model.node import Node
from framework.testbed_model.port import Port
from framework.utils import get_packet_summaries
@@ -28,7 +28,7 @@ class TrafficGenerator(ABC):
_config: TrafficGeneratorConfig
_tg_node: Node
- _logger: DTSLOG
+ _logger: DTSLogger
def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
"""Initialize the traffic generator.
@@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
"""
self._config = config
self._tg_node = tg_node
- self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
+ self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
def send_packet(self, packet: Packet, port: Port) -> None:
"""Send `packet` and block until it is fully sent.
diff --git a/dts/main.py b/dts/main.py
index 1ffe8ff81f..d30c164b95 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -30,5 +30,4 @@ def main() -> None:
# Main program begins here
if __name__ == "__main__":
- logging.raiseExceptions = True
main()
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 7/7] dts: improve test suite and case filtering
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
` (5 preceding siblings ...)
2024-02-06 14:57 ` [PATCH v2 6/7] dts: refactor logging configuration Juraj Linkeš
@ 2024-02-06 14:57 ` Juraj Linkeš
6 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
The two places where we specify which test suite and test cases to run
are complimentary and not that intuitive to use. A unified way provides
a better user experience.
The syntax in test run configuration file has not changed, but the
environment variable and the command line arguments was changed to match
the config file syntax. This required changes in the settings module
which greatly simplified the parsing of the environment variables while
retaining the same functionality.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
doc/guides/tools/dts.rst | 14 ++-
dts/framework/config/__init__.py | 12 +-
dts/framework/runner.py | 21 ++--
dts/framework/settings.py | 187 ++++++++++++++-----------------
dts/framework/test_suite.py | 2 +-
dts/main.py | 2 -
6 files changed, 116 insertions(+), 122 deletions(-)
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index f686ca487c..d1c3c2af7a 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,28 +215,30 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
.. code-block:: console
(dts-py3.10) $ ./main.py --help
- usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN]
+ usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
options:
-h, --help show this help message and exit
--config-file CONFIG_FILE
- [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
+ [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
--output-dir OUTPUT_DIR, --output OUTPUT_DIR
[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
-t TIMEOUT, --timeout TIMEOUT
[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
- -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None)
+ -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
--tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
--compile-timeout COMPILE_TIMEOUT
[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
- --test-cases TEST_CASES
- [DTS_TESTCASES] Comma-separated list of test cases to execute. Unknown test cases will be silently ignored. (default: )
+ --test-suite TEST_SUITE [TEST_CASES ...]
+ [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
+ variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
+ (default: [])
--re-run RE_RUN, --re_run RE_RUN
- [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs (default: 0)
+ [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index c6a93b3b89..4cb5c74059 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -35,9 +35,9 @@
import json
import os.path
-import pathlib
from dataclasses import dataclass, fields
from enum import auto, unique
+from pathlib import Path
from typing import Union
import warlock # type: ignore[import]
@@ -53,7 +53,6 @@
TrafficGeneratorConfigDict,
)
from framework.exception import ConfigurationError
-from framework.settings import SETTINGS
from framework.utils import StrEnum
@@ -571,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration":
return Configuration(executions=executions)
-def load_config() -> Configuration:
+def load_config(config_file_path: Path) -> Configuration:
"""Load DTS test run configuration from a file.
Load the YAML test run configuration file
@@ -581,13 +580,16 @@ def load_config() -> Configuration:
The YAML test run configuration file is specified in the :option:`--config-file` command line
argument or the :envvar:`DTS_CFG_FILE` environment variable.
+ Args:
+ config_file_path: The path to the YAML test run configuration file.
+
Returns:
The parsed test run configuration.
"""
- with open(SETTINGS.config_file_path, "r") as f:
+ with open(config_file_path, "r") as f:
config_data = yaml.safe_load(f)
- schema_path = os.path.join(pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json")
+ schema_path = os.path.join(Path(__file__).parent.resolve(), "conf_yaml_schema.json")
with open(schema_path, "r") as f:
schema = json.load(f)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 035e3368ef..32c0698cd2 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -24,7 +24,7 @@
import sys
from pathlib import Path
from types import MethodType
-from typing import Iterable
+from typing import Iterable, Sequence
from .config import (
BuildTargetConfiguration,
@@ -83,7 +83,7 @@ class DTSRunner:
def __init__(self):
"""Initialize the instance with configuration, logger, result and string constants."""
- self._configuration = load_config()
+ self._configuration = load_config(SETTINGS.config_file_path)
self._logger = get_dts_logger()
if not os.path.exists(SETTINGS.output_dir):
os.makedirs(SETTINGS.output_dir)
@@ -129,7 +129,7 @@ def run(self):
#. Execution teardown
The test cases are filtered according to the specification in the test run configuration and
- the :option:`--test-cases` command line argument or
+ the :option:`--test-suite` command line argument or
the :envvar:`DTS_TESTCASES` environment variable.
"""
sut_nodes: dict[str, SutNode] = {}
@@ -146,14 +146,17 @@ def run(self):
f"Running execution with SUT '{execution.system_under_test_node.name}'."
)
execution_result = self._result.add_execution(execution)
+ test_suite_configs = (
+ SETTINGS.test_suites if SETTINGS.test_suites else execution.test_suites
+ )
try:
test_suites_with_cases = self._get_test_suites_with_cases(
- execution.test_suites, execution.func, execution.perf
+ test_suite_configs, execution.func, execution.perf
)
execution_result.test_suites_with_cases = test_suites_with_cases
except Exception as e:
self._logger.exception(
- f"Invalid test suite configuration found: " f"{execution.test_suites}."
+ f"Invalid test suite configuration found: " f"{test_suite_configs}."
)
execution_result.update_setup(Result.FAIL, e)
@@ -222,7 +225,7 @@ def _get_test_suites_with_cases(
test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
test_cases = []
func_test_cases, perf_test_cases = self._filter_test_cases(
- test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases)
+ test_suite_class, test_suite_config.test_cases
)
if func:
test_cases.extend(func_test_cases)
@@ -295,7 +298,7 @@ def is_test_suite(object) -> bool:
)
def _filter_test_cases(
- self, test_suite_class: type[TestSuite], test_cases_to_run: set[str]
+ self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
) -> tuple[list[MethodType], list[MethodType]]:
"""Filter `test_cases_to_run` from `test_suite_class`.
@@ -324,7 +327,9 @@ def _filter_test_cases(
(name, method) for name, method in name_method_tuples if name in test_cases_to_run
]
if len(name_method_tuples) < len(test_cases_to_run):
- missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples}
+ missing_test_cases = set(test_cases_to_run) - {
+ name for name, _ in name_method_tuples
+ }
raise ConfigurationError(
f"Test cases {missing_test_cases} not found among methods "
f"of {test_suite_class.__name__}."
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 2b8bfbe0ed..688e8679a7 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -48,10 +48,11 @@
The path to a DPDK tarball, git commit ID, tag ID or tree ID to test.
-.. option:: --test-cases
-.. envvar:: DTS_TESTCASES
+.. option:: --test-suite
+.. envvar:: DTS_TEST_SUITES
- A comma-separated list of test cases to execute. Unknown test cases will be silently ignored.
+ A test suite with test cases which may be specified multiple times.
+ In the environment variable, the suites are joined with a comma.
.. option:: --re-run, --re_run
.. envvar:: DTS_RERUN
@@ -71,83 +72,13 @@
import argparse
import os
-from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass, field
from pathlib import Path
-from typing import Any, TypeVar
+from typing import Any
+from .config import TestSuiteConfig
from .utils import DPDKGitTarball
-_T = TypeVar("_T")
-
-
-def _env_arg(env_var: str) -> Any:
- """A helper method augmenting the argparse Action with environment variables.
-
- If the supplied environment variable is defined, then the default value
- of the argument is modified. This satisfies the priority order of
- command line argument > environment variable > default value.
-
- Arguments with no values (flags) should be defined using the const keyword argument
- (True or False). When the argument is specified, it will be set to const, if not specified,
- the default will be stored (possibly modified by the corresponding environment variable).
-
- Other arguments work the same as default argparse arguments, that is using
- the default 'store' action.
-
- Returns:
- The modified argparse.Action.
- """
-
- class _EnvironmentArgument(argparse.Action):
- def __init__(
- self,
- option_strings: Sequence[str],
- dest: str,
- nargs: str | int | None = None,
- const: bool | None = None,
- default: Any = None,
- type: Callable[[str], _T | argparse.FileType | None] = None,
- choices: Iterable[_T] | None = None,
- required: bool = False,
- help: str | None = None,
- metavar: str | tuple[str, ...] | None = None,
- ) -> None:
- env_var_value = os.environ.get(env_var)
- default = env_var_value or default
- if const is not None:
- nargs = 0
- default = const if env_var_value else default
- type = None
- choices = None
- metavar = None
- super(_EnvironmentArgument, self).__init__(
- option_strings,
- dest,
- nargs=nargs,
- const=const,
- default=default,
- type=type,
- choices=choices,
- required=required,
- help=help,
- metavar=metavar,
- )
-
- def __call__(
- self,
- parser: argparse.ArgumentParser,
- namespace: argparse.Namespace,
- values: Any,
- option_string: str = None,
- ) -> None:
- if self.const is not None:
- setattr(namespace, self.dest, self.const)
- else:
- setattr(namespace, self.dest, values)
-
- return _EnvironmentArgument
-
@dataclass(slots=True)
class Settings:
@@ -171,7 +102,7 @@ class Settings:
#:
compile_timeout: float = 1200
#:
- test_cases: list[str] = field(default_factory=list)
+ test_suites: list[TestSuiteConfig] = field(default_factory=list)
#:
re_run: int = 0
@@ -180,6 +111,31 @@ class Settings:
def _get_parser() -> argparse.ArgumentParser:
+ """Create the argument parser for DTS.
+
+ Command line options take precedence over environment variables, which in turn take precedence
+ over default values.
+
+ Returns:
+ argparse.ArgumentParser: The configured argument parser with defined options.
+ """
+
+ def env_arg(env_var: str, default: Any) -> Any:
+ """A helper function augmenting the argparse with environment variables.
+
+ If the supplied environment variable is defined, then the default value
+ of the argument is modified. This satisfies the priority order of
+ command line argument > environment variable > default value.
+
+ Args:
+ env_var: Environment variable name.
+ default: Default value.
+
+ Returns:
+ Environment variable or default value.
+ """
+ return os.environ.get(env_var) or default
+
parser = argparse.ArgumentParser(
description="Run DPDK test suites. All options may be specified with the environment "
"variables provided in brackets. Command line arguments have higher priority.",
@@ -188,25 +144,23 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--config-file",
- action=_env_arg("DTS_CFG_FILE"),
- default=SETTINGS.config_file_path,
+ default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
type=Path,
- help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets.",
+ help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
+ "SUTs and targets.",
)
parser.add_argument(
"--output-dir",
"--output",
- action=_env_arg("DTS_OUTPUT_DIR"),
- default=SETTINGS.output_dir,
+ default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
)
parser.add_argument(
"-t",
"--timeout",
- action=_env_arg("DTS_TIMEOUT"),
- default=SETTINGS.timeout,
+ default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
type=float,
help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
)
@@ -214,9 +168,8 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"-v",
"--verbose",
- action=_env_arg("DTS_VERBOSE"),
- default=SETTINGS.verbose,
- const=True,
+ action="store_true",
+ default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
"to the console.",
)
@@ -224,8 +177,8 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"-s",
"--skip-setup",
- action=_env_arg("DTS_SKIP_SETUP"),
- const=True,
+ action="store_true",
+ default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
)
@@ -233,8 +186,7 @@ def _get_parser() -> argparse.ArgumentParser:
"--tarball",
"--snapshot",
"--git-ref",
- action=_env_arg("DTS_DPDK_TARBALL"),
- default=SETTINGS.dpdk_tarball_path,
+ default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
type=Path,
help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
"tag ID or tree ID to test. To test local changes, first commit them, "
@@ -243,36 +195,71 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--compile-timeout",
- action=_env_arg("DTS_COMPILE_TIMEOUT"),
- default=SETTINGS.compile_timeout,
+ default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
type=float,
help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
)
parser.add_argument(
- "--test-cases",
- action=_env_arg("DTS_TESTCASES"),
- default="",
- help="[DTS_TESTCASES] Comma-separated list of test cases to execute.",
+ "--test-suite",
+ action="append",
+ nargs="+",
+ metavar=("TEST_SUITE", "TEST_CASES"),
+ default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
+ help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
+ "The first parameter is the test suite name, and the rest are test case names, "
+ "which are optional. May be specified multiple times. To specify multiple test suites in "
+ "the environment variable, join the lists with a comma. "
+ "Examples: "
+ "--test-suite suite case case --test-suite suite case ... | "
+ "DTS_TEST_SUITES='suite case case, suite case, ...' | "
+ "--test-suite suite --test-suite suite case ... | "
+ "DTS_TEST_SUITES='suite, suite case, ...'",
)
parser.add_argument(
"--re-run",
"--re_run",
- action=_env_arg("DTS_RERUN"),
- default=SETTINGS.re_run,
+ default=env_arg("DTS_RERUN", SETTINGS.re_run),
type=int,
help="[DTS_RERUN] Re-run each test case the specified number of times "
- "if a test failure occurs",
+ "if a test failure occurs.",
)
return parser
+def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
+ """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
+
+ Args:
+ args: The arguments to process. The args is a string from an environment variable
+ or a list of from the user input containing tests suites with tests cases,
+ each of which is a list of [test_suite, test_case, test_case, ...].
+
+ Returns:
+ A list of test suite configurations to execute.
+ """
+ if isinstance(args, str):
+ # Environment variable in the form of "suite case case, suite case, suite, ..."
+ args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
+
+ test_suites_to_run = []
+ for suite_with_cases in args:
+ test_suites_to_run.append(
+ TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
+ )
+
+ return test_suites_to_run
+
+
def get_settings() -> Settings:
"""Create new settings with inputs from the user.
The inputs are taken from the command line and from environment variables.
+
+ Returns:
+ The new settings object.
"""
parsed_args = _get_parser().parse_args()
return Settings(
@@ -287,6 +274,6 @@ def get_settings() -> Settings:
else Path(parsed_args.tarball)
),
compile_timeout=parsed_args.compile_timeout,
- test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []),
+ test_suites=_process_test_suites(parsed_args.test_suite),
re_run=parsed_args.re_run,
)
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 365f80e21a..1957ea7328 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -40,7 +40,7 @@ class TestSuite(object):
and functional test cases (all other test cases).
By default, all test cases will be executed. A list of testcase names may be specified
- in the YAML test run configuration file and in the :option:`--test-cases` command line argument
+ in the YAML test run configuration file and in the :option:`--test-suite` command line argument
or in the :envvar:`DTS_TESTCASES` environment variable to filter which test cases to run.
The union of both lists will be used. Any unknown test cases from the latter lists
will be silently ignored.
diff --git a/dts/main.py b/dts/main.py
index d30c164b95..fa878cc16e 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -6,8 +6,6 @@
"""The DTS executable."""
-import logging
-
from framework import settings
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 3/7] dts: filter test suites in executions
2024-02-06 14:57 ` [PATCH v2 3/7] dts: filter test suites in executions Juraj Linkeš
@ 2024-02-12 16:44 ` Jeremy Spewock
2024-02-14 9:55 ` Juraj Linkeš
0 siblings, 1 reply; 44+ messages in thread
From: Jeremy Spewock @ 2024-02-12 16:44 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev
On Tue, Feb 6, 2024 at 9:57 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> We're currently filtering which test cases to run after some setup
> steps, such as DPDK build, have already been taken. This prohibits us to
> mark the test suites and cases that were supposed to be run as blocked
> when an earlier setup fails, as that information is not available at
> that time.
>
> To remedy this, move the filtering to the beginning of each execution.
> This is the first action taken in each execution and if we can't filter
> the test cases, such as due to invalid inputs, we abort the whole
> execution. No test suites nor cases will be marked as blocked as we
> don't know which were supposed to be run.
>
> On top of that, the filtering takes place in the TestSuite class, which
> should only concern itself with test suite and test case logic, not the
> processing behind the scenes. The logic has been moved to DTSRunner
> which should do all the processing needed to run test suites.
>
> The filtering itself introduces a few changes/assumptions which are more
> sensible than before:
> 1. Assumption: There is just one TestSuite child class in each test
> suite module. This was an implicit assumption before as we couldn't
> specify the TestSuite classes in the test run configuration, just the
> modules. The name of the TestSuite child class starts with "Test" and
> then corresponds to the name of the module with CamelCase naming.
> 2. Unknown test cases specified both in the test run configuration and
> the environment variable/command line argument are no longer silently
> ignored. This is a quality of life improvement for users, as they
> could easily be not aware of the silent ignoration.
>
> Also, a change in the code results in pycodestyle warning and error:
> [E] E203 whitespace before ':'
> [W] W503 line break before binary operator
>
> These two are not PEP8 compliant, so they're disabled.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
> dts/framework/config/__init__.py | 24 +-
> dts/framework/config/conf_yaml_schema.json | 2 +-
> dts/framework/runner.py | 426 +++++++++++++++------
> dts/framework/settings.py | 3 +-
> dts/framework/test_result.py | 34 ++
> dts/framework/test_suite.py | 85 +---
> dts/pyproject.toml | 3 +
> dts/tests/TestSuite_smoke_tests.py | 2 +-
> 8 files changed, 382 insertions(+), 197 deletions(-)
>
> diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
> index 62eded7f04..c6a93b3b89 100644
> --- a/dts/framework/config/__init__.py
> +++ b/dts/framework/config/__init__.py
> @@ -36,7 +36,7 @@
> import json
> import os.path
> import pathlib
> -from dataclasses import dataclass
> +from dataclasses import dataclass, fields
> from enum import auto, unique
> from typing import Union
>
> @@ -506,6 +506,28 @@ def from_dict(
> vdevs=vdevs,
> )
>
> + def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration":
> + """Create a shallow copy with any of the fields modified.
> +
> + The only new data are those passed to this method.
> + The rest are copied from the object's fields calling the method.
> +
> + Args:
> + **kwargs: The names and types of keyword arguments are defined
> + by the fields of the :class:`ExecutionConfiguration` class.
> +
> + Returns:
> + The copied and modified execution configuration.
> + """
> + new_config = {}
> + for field in fields(self):
> + if field.name in kwargs:
> + new_config[field.name] = kwargs[field.name]
> + else:
> + new_config[field.name] = getattr(self, field.name)
> +
> + return ExecutionConfiguration(**new_config)
> +
>
> @dataclass(slots=True, frozen=True)
> class Configuration:
> diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
> index 84e45fe3c2..051b079fe4 100644
> --- a/dts/framework/config/conf_yaml_schema.json
> +++ b/dts/framework/config/conf_yaml_schema.json
> @@ -197,7 +197,7 @@
> },
> "cases": {
> "type": "array",
> - "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.",
> + "description": "If specified, only this subset of test suite's test cases will be run.",
> "items": {
> "type": "string"
> },
> diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> index 933685d638..3e95cf9e26 100644
> --- a/dts/framework/runner.py
> +++ b/dts/framework/runner.py
> @@ -17,17 +17,27 @@
> and the test case stage runs test cases individually.
> """
>
> +import importlib
> +import inspect
> import logging
> +import re
> import sys
> from types import MethodType
> +from typing import Iterable
>
> from .config import (
> BuildTargetConfiguration,
> + Configuration,
> ExecutionConfiguration,
> TestSuiteConfig,
> load_config,
> )
> -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
> +from .exception import (
> + BlockingTestSuiteError,
> + ConfigurationError,
> + SSHTimeoutError,
> + TestCaseVerifyError,
> +)
> from .logger import DTSLOG, getLogger
> from .settings import SETTINGS
> from .test_result import (
> @@ -37,8 +47,9 @@
> Result,
> TestCaseResult,
> TestSuiteResult,
> + TestSuiteWithCases,
> )
> -from .test_suite import TestSuite, get_test_suites
> +from .test_suite import TestSuite
> from .testbed_model import SutNode, TGNode
>
>
> @@ -59,13 +70,23 @@ class DTSRunner:
> given execution, the next execution begins.
> """
>
> + _configuration: Configuration
> _logger: DTSLOG
> _result: DTSResult
> + _test_suite_class_prefix: str
> + _test_suite_module_prefix: str
> + _func_test_case_regex: str
> + _perf_test_case_regex: str
>
> def __init__(self):
> - """Initialize the instance with logger and result."""
> + """Initialize the instance with configuration, logger, result and string constants."""
> + self._configuration = load_config()
> self._logger = getLogger("DTSRunner")
> self._result = DTSResult(self._logger)
> + self._test_suite_class_prefix = "Test"
> + self._test_suite_module_prefix = "tests.TestSuite_"
> + self._func_test_case_regex = r"test_(?!perf_)"
> + self._perf_test_case_regex = r"test_perf_"
>
> def run(self):
> """Run all build targets in all executions from the test run configuration.
> @@ -106,29 +127,28 @@ def run(self):
> try:
> # check the python version of the server that runs dts
> self._check_dts_python_version()
> + self._result.update_setup(Result.PASS)
>
> # for all Execution sections
> - for execution in load_config().executions:
> - sut_node = sut_nodes.get(execution.system_under_test_node.name)
> - tg_node = tg_nodes.get(execution.traffic_generator_node.name)
> -
> + for execution in self._configuration.executions:
> + self._logger.info(
> + f"Running execution with SUT '{execution.system_under_test_node.name}'."
> + )
> + execution_result = self._result.add_execution(execution.system_under_test_node)
> try:
> - if not sut_node:
> - sut_node = SutNode(execution.system_under_test_node)
> - sut_nodes[sut_node.name] = sut_node
> - if not tg_node:
> - tg_node = TGNode(execution.traffic_generator_node)
> - tg_nodes[tg_node.name] = tg_node
> - self._result.update_setup(Result.PASS)
> + test_suites_with_cases = self._get_test_suites_with_cases(
> + execution.test_suites, execution.func, execution.perf
> + )
> except Exception as e:
> - failed_node = execution.system_under_test_node.name
> - if sut_node:
> - failed_node = execution.traffic_generator_node.name
> - self._logger.exception(f"The Creation of node {failed_node} failed.")
> - self._result.update_setup(Result.FAIL, e)
> + self._logger.exception(
> + f"Invalid test suite configuration found: " f"{execution.test_suites}."
> + )
> + execution_result.update_setup(Result.FAIL, e)
>
> else:
> - self._run_execution(sut_node, tg_node, execution)
> + self._connect_nodes_and_run_execution(
> + sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases
> + )
>
> except Exception as e:
> self._logger.exception("An unexpected error has occurred.")
> @@ -163,11 +183,204 @@ def _check_dts_python_version(self) -> None:
> )
> self._logger.warning("Please use Python >= 3.10 instead.")
>
> + def _get_test_suites_with_cases(
> + self,
> + test_suite_configs: list[TestSuiteConfig],
> + func: bool,
> + perf: bool,
> + ) -> list[TestSuiteWithCases]:
> + """Test suites with test cases discovery.
> +
> + The test suites with test cases defined in the user configuration are discovered
> + and stored for future use so that we don't import the modules twice and so that
> + the list of test suites with test cases is available for recording right away.
> +
> + Args:
> + test_suite_configs: Test suite configurations.
> + func: Whether to include functional test cases in the final list.
> + perf: Whether to include performance test cases in the final list.
> +
> + Returns:
> + The discovered test suites, each with test cases.
> + """
> + test_suites_with_cases = []
> +
> + for test_suite_config in test_suite_configs:
> + test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
> + test_cases = []
> + func_test_cases, perf_test_cases = self._filter_test_cases(
> + test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases)
> + )
> + if func:
> + test_cases.extend(func_test_cases)
> + if perf:
> + test_cases.extend(perf_test_cases)
> +
> + test_suites_with_cases.append(
> + TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases)
> + )
> +
> + return test_suites_with_cases
> +
> + def _get_test_suite_class(self, test_suite_name: str) -> type[TestSuite]:
> + """Find the :class:`TestSuite` class with `test_suite_name` in the corresponding module.
> +
> + The method assumes that the :class:`TestSuite` class starts
> + with `self._test_suite_class_prefix`,
> + continuing with `test_suite_name` with CamelCase convention.
> + It also assumes there's only one test suite in each module and the module name
> + is `test_suite_name` prefixed with `self._test_suite_module_prefix`.
> +
> + The CamelCase convention is not tested, only lowercase strings are compared.
> +
> + Args:
> + test_suite_name: The name of the test suite to find.
> +
> + Returns:
> + The found test suite.
> +
> + Raises:
> + ConfigurationError: If the corresponding module is not found or
> + a valid :class:`TestSuite` is not found in the module.
> + """
> +
> + def is_test_suite(object) -> bool:
> + """Check whether `object` is a :class:`TestSuite`.
> +
> + The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself.
> +
> + Args:
> + object: The object to be checked.
> +
> + Returns:
> + :data:`True` if `object` is a subclass of `TestSuite`.
> + """
> + try:
> + if issubclass(object, TestSuite) and object is not TestSuite:
> + return True
> + except TypeError:
> + return False
> + return False
> +
> + testsuite_module_path = f"{self._test_suite_module_prefix}{test_suite_name}"
> + try:
> + test_suite_module = importlib.import_module(testsuite_module_path)
> + except ModuleNotFoundError as e:
> + raise ConfigurationError(
> + f"Test suite module '{testsuite_module_path}' not found."
> + ) from e
> +
> + lowercase_suite_name = test_suite_name.replace("_", "").lower()
> + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite):
> + if (
> + class_name.startswith(self._test_suite_class_prefix)
> + and lowercase_suite_name == class_name[len(self._test_suite_class_prefix) :].lower()
> + ):
Would it be simpler to instead just make lowercase_suite_name =
f"{self._test_suite_class_prefix}{test_suite_name.replace("_",
"").lower()}" so that you can just directly compare class_name ==
lowercase_suite_name? Both ways should have the exact same result of
course so it isn't important, I was just curious.
> + return class_obj
> + raise ConfigurationError(
> + f"Couldn't find any valid test suites in {test_suite_module.__name__}."
> + )
> +
> + def _filter_test_cases(
> + self, test_suite_class: type[TestSuite], test_cases_to_run: set[str]
> + ) -> tuple[list[MethodType], list[MethodType]]:
> + """Filter `test_cases_to_run` from `test_suite_class`.
> +
> + There are two rounds of filtering if `test_cases_to_run` is not empty.
> + The first filters `test_cases_to_run` from all methods of `test_suite_class`.
> + Then the methods are separated into functional and performance test cases.
> + If a method doesn't match neither the functional nor performance name prefix, it's an error.
I think this is a double negative but could be either "if a method
doesn't match either ... or ..." or "if a method matches neither ...
nor ...". I have a small preference to the second of the two options
though because the "neither" makes the negative more clear in my mind.
> +
> + Args:
> + test_suite_class: The class of the test suite.
> + test_cases_to_run: Test case names to filter from `test_suite_class`.
> + If empty, return all matching test cases.
> +
> + Returns:
> + A list of test case methods that should be executed.
> +
> + Raises:
> + ConfigurationError: If a test case from `test_cases_to_run` is not found
> + or it doesn't match either the functional nor performance name prefix.
> + """
> + func_test_cases = []
> + perf_test_cases = []
> + name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
> + if test_cases_to_run:
> + name_method_tuples = [
> + (name, method) for name, method in name_method_tuples if name in test_cases_to_run
> + ]
> + if len(name_method_tuples) < len(test_cases_to_run):
> + missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples}
> + raise ConfigurationError(
> + f"Test cases {missing_test_cases} not found among methods "
> + f"of {test_suite_class.__name__}."
> + )
> +
> + for test_case_name, test_case_method in name_method_tuples:
> + if re.match(self._func_test_case_regex, test_case_name):
> + func_test_cases.append(test_case_method)
> + elif re.match(self._perf_test_case_regex, test_case_name):
> + perf_test_cases.append(test_case_method)
> + elif test_cases_to_run:
> + raise ConfigurationError(
> + f"Method '{test_case_name}' doesn't match neither "
> + f"a functional nor a performance test case name."
Same thing here with the double negative.
> + )
> +
> + return func_test_cases, perf_test_cases
> +
> + def _connect_nodes_and_run_execution(
> + self,
> + sut_nodes: dict[str, SutNode],
> + tg_nodes: dict[str, TGNode],
> + execution: ExecutionConfiguration,
> + execution_result: ExecutionResult,
> + test_suites_with_cases: Iterable[TestSuiteWithCases],
> + ) -> None:
> + """Connect nodes, then continue to run the given execution.
> +
> + Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`.
> + If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`,
> + respectively.
> + If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`.
> +
> + Args:
> + sut_nodes: A dictionary storing connected/to be connected SUT nodes.
> + tg_nodes: A dictionary storing connected/to be connected TG nodes.
> + execution: An execution's test run configuration.
> + execution_result: The execution's result.
> + test_suites_with_cases: The test suites with test cases to run.
> + """
> + sut_node = sut_nodes.get(execution.system_under_test_node.name)
> + tg_node = tg_nodes.get(execution.traffic_generator_node.name)
> +
> + try:
> + if not sut_node:
> + sut_node = SutNode(execution.system_under_test_node)
> + sut_nodes[sut_node.name] = sut_node
> + if not tg_node:
> + tg_node = TGNode(execution.traffic_generator_node)
> + tg_nodes[tg_node.name] = tg_node
> + except Exception as e:
> + failed_node = execution.system_under_test_node.name
> + if sut_node:
> + failed_node = execution.traffic_generator_node.name
> + self._logger.exception(f"The Creation of node {failed_node} failed.")
> + execution_result.update_setup(Result.FAIL, e)
> +
> + else:
> + self._run_execution(
> + sut_node, tg_node, execution, execution_result, test_suites_with_cases
> + )
> +
> def _run_execution(
> self,
> sut_node: SutNode,
> tg_node: TGNode,
> execution: ExecutionConfiguration,
> + execution_result: ExecutionResult,
> + test_suites_with_cases: Iterable[TestSuiteWithCases],
> ) -> None:
> """Run the given execution.
>
> @@ -178,11 +391,11 @@ def _run_execution(
> sut_node: The execution's SUT node.
> tg_node: The execution's TG node.
> execution: An execution's test run configuration.
> + execution_result: The execution's result.
> + test_suites_with_cases: The test suites with test cases to run.
> """
> self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
> - execution_result = self._result.add_execution(sut_node.config)
> execution_result.add_sut_info(sut_node.node_info)
> -
> try:
> sut_node.set_up_execution(execution)
> execution_result.update_setup(Result.PASS)
> @@ -192,7 +405,10 @@ def _run_execution(
>
> else:
> for build_target in execution.build_targets:
> - self._run_build_target(sut_node, tg_node, build_target, execution, execution_result)
> + build_target_result = execution_result.add_build_target(build_target)
> + self._run_build_target(
> + sut_node, tg_node, build_target, build_target_result, test_suites_with_cases
> + )
>
> finally:
> try:
> @@ -207,8 +423,8 @@ def _run_build_target(
> sut_node: SutNode,
> tg_node: TGNode,
> build_target: BuildTargetConfiguration,
> - execution: ExecutionConfiguration,
> - execution_result: ExecutionResult,
> + build_target_result: BuildTargetResult,
> + test_suites_with_cases: Iterable[TestSuiteWithCases],
> ) -> None:
> """Run the given build target.
>
> @@ -220,11 +436,11 @@ def _run_build_target(
> sut_node: The execution's sut node.
> tg_node: The execution's tg node.
> build_target: A build target's test run configuration.
> - execution: The build target's execution's test run configuration.
> - execution_result: The execution level result object associated with the execution.
> + build_target_result: The build target level result object associated
> + with the current build target.
> + test_suites_with_cases: The test suites with test cases to run.
> """
> self._logger.info(f"Running build target '{build_target.name}'.")
> - build_target_result = execution_result.add_build_target(build_target)
>
> try:
> sut_node.set_up_build_target(build_target)
> @@ -236,7 +452,7 @@ def _run_build_target(
> build_target_result.update_setup(Result.FAIL, e)
>
> else:
> - self._run_test_suites(sut_node, tg_node, execution, build_target_result)
> + self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases)
>
> finally:
> try:
> @@ -250,10 +466,10 @@ def _run_test_suites(
> self,
> sut_node: SutNode,
> tg_node: TGNode,
> - execution: ExecutionConfiguration,
> build_target_result: BuildTargetResult,
> + test_suites_with_cases: Iterable[TestSuiteWithCases],
> ) -> None:
> - """Run the execution's (possibly a subset of) test suites using the current build target.
> + """Run `test_suites_with_cases` with the current build target.
>
> The method assumes the build target we're testing has already been built on the SUT node.
> The current build target thus corresponds to the current DPDK build present on the SUT node.
> @@ -264,22 +480,20 @@ def _run_test_suites(
> Args:
> sut_node: The execution's SUT node.
> tg_node: The execution's TG node.
> - execution: The execution's test run configuration associated
> - with the current build target.
> build_target_result: The build target level result object associated
> with the current build target.
> + test_suites_with_cases: The test suites with test cases to run.
> """
> end_build_target = False
> - if not execution.skip_smoke_tests:
> - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
> - for test_suite_config in execution.test_suites:
> + for test_suite_with_cases in test_suites_with_cases:
> + test_suite_result = build_target_result.add_test_suite(
> + test_suite_with_cases.test_suite_class.__name__
> + )
> try:
> - self._run_test_suite_module(
> - sut_node, tg_node, execution, build_target_result, test_suite_config
> - )
> + self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
> except BlockingTestSuiteError as e:
> self._logger.exception(
> - f"An error occurred within {test_suite_config.test_suite}. "
> + f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
> "Skipping build target..."
> )
> self._result.add_error(e)
> @@ -288,15 +502,14 @@ def _run_test_suites(
> if end_build_target:
> break
>
> - def _run_test_suite_module(
> + def _run_test_suite(
> self,
> sut_node: SutNode,
> tg_node: TGNode,
> - execution: ExecutionConfiguration,
> - build_target_result: BuildTargetResult,
> - test_suite_config: TestSuiteConfig,
> + test_suite_result: TestSuiteResult,
> + test_suite_with_cases: TestSuiteWithCases,
> ) -> None:
> - """Set up, execute and tear down all test suites in a single test suite module.
> + """Set up, execute and tear down `test_suite_with_cases`.
>
> The method assumes the build target we're testing has already been built on the SUT node.
> The current build target thus corresponds to the current DPDK build present on the SUT node.
> @@ -306,92 +519,79 @@ def _run_test_suite_module(
>
> Record the setup and the teardown and handle failures.
>
> - The test cases to execute are discovered when creating the :class:`TestSuite` object.
> -
> Args:
> sut_node: The execution's SUT node.
> tg_node: The execution's TG node.
> - execution: The execution's test run configuration associated
> - with the current build target.
> - build_target_result: The build target level result object associated
> - with the current build target.
> - test_suite_config: Test suite test run configuration specifying the test suite module
> - and possibly a subset of test cases of test suites in that module.
> + test_suite_result: The test suite level result object associated
> + with the current test suite.
> + test_suite_with_cases: The test suite with test cases to run.
>
> Raises:
> BlockingTestSuiteError: If a blocking test suite fails.
> """
> + test_suite_name = test_suite_with_cases.test_suite_class.__name__
> + test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
> try:
> - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
> - test_suite_classes = get_test_suites(full_suite_path)
> - suites_str = ", ".join((x.__name__ for x in test_suite_classes))
> - self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
> + self._logger.info(f"Starting test suite setup: {test_suite_name}")
> + test_suite.set_up_suite()
> + test_suite_result.update_setup(Result.PASS)
> + self._logger.info(f"Test suite setup successful: {test_suite_name}")
> except Exception as e:
> - self._logger.exception("An error occurred when searching for test suites.")
> - self._result.update_setup(Result.ERROR, e)
> + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
> + test_suite_result.update_setup(Result.ERROR, e)
>
> else:
> - for test_suite_class in test_suite_classes:
> - test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases)
> -
> - test_suite_name = test_suite.__class__.__name__
> - test_suite_result = build_target_result.add_test_suite(test_suite_name)
> - try:
> - self._logger.info(f"Starting test suite setup: {test_suite_name}")
> - test_suite.set_up_suite()
> - test_suite_result.update_setup(Result.PASS)
> - self._logger.info(f"Test suite setup successful: {test_suite_name}")
> - except Exception as e:
> - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
> - test_suite_result.update_setup(Result.ERROR, e)
> -
> - else:
> - self._execute_test_suite(execution.func, test_suite, test_suite_result)
> -
> - finally:
> - try:
> - test_suite.tear_down_suite()
> - sut_node.kill_cleanup_dpdk_apps()
> - test_suite_result.update_teardown(Result.PASS)
> - except Exception as e:
> - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
> - self._logger.warning(
> - f"Test suite '{test_suite_name}' teardown failed, "
> - f"the next test suite may be affected."
> - )
> - test_suite_result.update_setup(Result.ERROR, e)
> - if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
> - raise BlockingTestSuiteError(test_suite_name)
> + self._execute_test_suite(
> + test_suite,
> + test_suite_with_cases.test_cases,
> + test_suite_result,
> + )
> + finally:
> + try:
> + test_suite.tear_down_suite()
> + sut_node.kill_cleanup_dpdk_apps()
> + test_suite_result.update_teardown(Result.PASS)
> + except Exception as e:
> + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
> + self._logger.warning(
> + f"Test suite '{test_suite_name}' teardown failed, "
> + "the next test suite may be affected."
> + )
> + test_suite_result.update_setup(Result.ERROR, e)
> + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
> + raise BlockingTestSuiteError(test_suite_name)
>
> def _execute_test_suite(
> - self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
> + self,
> + test_suite: TestSuite,
> + test_cases: Iterable[MethodType],
> + test_suite_result: TestSuiteResult,
> ) -> None:
> - """Execute all discovered test cases in `test_suite`.
> + """Execute all `test_cases` in `test_suite`.
>
> If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment
> variable is set, in case of a test case failure, the test case will be executed again
> until it passes or it fails that many times in addition of the first failure.
>
> Args:
> - func: Whether to execute functional test cases.
> test_suite: The test suite object.
> + test_cases: The list of test case methods.
> test_suite_result: The test suite level result object associated
> with the current test suite.
> """
> - if func:
> - for test_case_method in test_suite._get_functional_test_cases():
> - test_case_name = test_case_method.__name__
> - test_case_result = test_suite_result.add_test_case(test_case_name)
> - all_attempts = SETTINGS.re_run + 1
> - attempt_nr = 1
> + for test_case_method in test_cases:
> + test_case_name = test_case_method.__name__
> + test_case_result = test_suite_result.add_test_case(test_case_name)
> + all_attempts = SETTINGS.re_run + 1
> + attempt_nr = 1
> + self._run_test_case(test_suite, test_case_method, test_case_result)
> + while not test_case_result and attempt_nr < all_attempts:
> + attempt_nr += 1
> + self._logger.info(
> + f"Re-running FAILED test case '{test_case_name}'. "
> + f"Attempt number {attempt_nr} out of {all_attempts}."
> + )
> self._run_test_case(test_suite, test_case_method, test_case_result)
> - while not test_case_result and attempt_nr < all_attempts:
> - attempt_nr += 1
> - self._logger.info(
> - f"Re-running FAILED test case '{test_case_name}'. "
> - f"Attempt number {attempt_nr} out of {all_attempts}."
> - )
> - self._run_test_case(test_suite, test_case_method, test_case_result)
>
> def _run_test_case(
> self,
> @@ -399,7 +599,7 @@ def _run_test_case(
> test_case_method: MethodType,
> test_case_result: TestCaseResult,
> ) -> None:
> - """Setup, execute and teardown a test case in `test_suite`.
> + """Setup, execute and teardown `test_case_method` from `test_suite`.
>
> Record the result of the setup and the teardown and handle failures.
>
> @@ -424,7 +624,7 @@ def _run_test_case(
>
> else:
> # run test case if setup was successful
> - self._execute_test_case(test_case_method, test_case_result)
> + self._execute_test_case(test_suite, test_case_method, test_case_result)
>
> finally:
> try:
> @@ -440,11 +640,15 @@ def _run_test_case(
> test_case_result.update(Result.ERROR)
>
> def _execute_test_case(
> - self, test_case_method: MethodType, test_case_result: TestCaseResult
> + self,
> + test_suite: TestSuite,
> + test_case_method: MethodType,
> + test_case_result: TestCaseResult,
> ) -> None:
> - """Execute one test case, record the result and handle failures.
> + """Execute `test_case_method` from `test_suite`, record the result and handle failures.
>
> Args:
> + test_suite: The test suite object.
> test_case_method: The test case method.
> test_case_result: The test case level result object associated
> with the current test case.
> @@ -452,7 +656,7 @@ def _execute_test_case(
> test_case_name = test_case_method.__name__
> try:
> self._logger.info(f"Starting test case execution: {test_case_name}")
> - test_case_method()
> + test_case_method(test_suite)
> test_case_result.update(Result.PASS)
> self._logger.info(f"Test case execution PASSED: {test_case_name}")
>
> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> index 609c8d0e62..2b8bfbe0ed 100644
> --- a/dts/framework/settings.py
> +++ b/dts/framework/settings.py
> @@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser:
> "--test-cases",
> action=_env_arg("DTS_TESTCASES"),
> default="",
> - help="[DTS_TESTCASES] Comma-separated list of test cases to execute. "
> - "Unknown test cases will be silently ignored.",
> + help="[DTS_TESTCASES] Comma-separated list of test cases to execute.",
> )
>
> parser.add_argument(
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index 4467749a9d..075195fd5b 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -25,7 +25,9 @@
>
> import os.path
> from collections.abc import MutableSequence
> +from dataclasses import dataclass
> from enum import Enum, auto
> +from types import MethodType
>
> from .config import (
> OS,
> @@ -36,10 +38,42 @@
> CPUType,
> NodeConfiguration,
> NodeInfo,
> + TestSuiteConfig,
> )
> from .exception import DTSError, ErrorSeverity
> from .logger import DTSLOG
> from .settings import SETTINGS
> +from .test_suite import TestSuite
> +
> +
> +@dataclass(slots=True, frozen=True)
> +class TestSuiteWithCases:
> + """A test suite class with test case methods.
> +
> + An auxiliary class holding a test case class with test case methods. The intended use of this
> + class is to hold a subset of test cases (which could be all test cases) because we don't have
> + all the data to instantiate the class at the point of inspection. The knowledge of this subset
> + is needed in case an error occurs before the class is instantiated and we need to record
> + which test cases were blocked by the error.
> +
> + Attributes:
> + test_suite_class: The test suite class.
> + test_cases: The test case methods.
> + """
> +
> + test_suite_class: type[TestSuite]
> + test_cases: list[MethodType]
> +
> + def create_config(self) -> TestSuiteConfig:
> + """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
> +
> + Returns:
> + The :class:`TestSuiteConfig` representation.
> + """
> + return TestSuiteConfig(
> + test_suite=self.test_suite_class.__name__,
> + test_cases=[test_case.__name__ for test_case in self.test_cases],
> + )
>
>
> class Result(Enum):
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index b02fd36147..f9fe88093e 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -11,25 +11,17 @@
> * Testbed (SUT, TG) configuration,
> * Packet sending and verification,
> * Test case verification.
> -
> -The module also defines a function, :func:`get_test_suites`,
> -for gathering test suites from a Python module.
> """
>
> -import importlib
> -import inspect
> -import re
> from ipaddress import IPv4Interface, IPv6Interface, ip_interface
> -from types import MethodType
> -from typing import Any, ClassVar, Union
> +from typing import ClassVar, Union
>
> from scapy.layers.inet import IP # type: ignore[import]
> from scapy.layers.l2 import Ether # type: ignore[import]
> from scapy.packet import Packet, Padding # type: ignore[import]
>
> -from .exception import ConfigurationError, TestCaseVerifyError
> +from .exception import TestCaseVerifyError
> from .logger import DTSLOG, getLogger
> -from .settings import SETTINGS
> from .testbed_model import Port, PortLink, SutNode, TGNode
> from .utils import get_packet_summaries
>
> @@ -37,7 +29,6 @@
> class TestSuite(object):
> """The base class with building blocks needed by most test cases.
>
> - * Test case filtering and collection,
> * Test suite setup/cleanup methods to override,
> * Test case setup/cleanup methods to override,
> * Test case verification,
> @@ -71,7 +62,6 @@ class TestSuite(object):
> #: will block the execution of all subsequent test suites in the current build target.
> is_blocking: ClassVar[bool] = False
> _logger: DTSLOG
> - _test_cases_to_run: list[str]
> _port_links: list[PortLink]
> _sut_port_ingress: Port
> _sut_port_egress: Port
> @@ -86,24 +76,19 @@ def __init__(
> self,
> sut_node: SutNode,
> tg_node: TGNode,
> - test_cases: list[str],
> ):
> """Initialize the test suite testbed information and basic configuration.
>
> - Process what test cases to run, find links between ports and set up
> - default IP addresses to be used when configuring them.
> + Find links between ports and set up default IP addresses to be used when
> + configuring them.
>
> Args:
> sut_node: The SUT node where the test suite will run.
> tg_node: The TG node where the test suite will run.
> - test_cases: The list of test cases to execute.
> - If empty, all test cases will be executed.
> """
> self.sut_node = sut_node
> self.tg_node = tg_node
> self._logger = getLogger(self.__class__.__name__)
> - self._test_cases_to_run = test_cases
> - self._test_cases_to_run.extend(SETTINGS.test_cases)
> self._port_links = []
> self._process_links()
> self._sut_port_ingress, self._tg_port_egress = (
> @@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
> if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
> return False
> return True
> -
> - def _get_functional_test_cases(self) -> list[MethodType]:
> - """Get all functional test cases defined in this TestSuite.
> -
> - Returns:
> - The list of functional test cases of this TestSuite.
> - """
> - return self._get_test_cases(r"test_(?!perf_)")
> -
> - def _get_test_cases(self, test_case_regex: str) -> list[MethodType]:
> - """Return a list of test cases matching test_case_regex.
> -
> - Returns:
> - The list of test cases matching test_case_regex of this TestSuite.
> - """
> - self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.")
> - filtered_test_cases = []
> - for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod):
> - if self._should_be_executed(test_case_name, test_case_regex):
> - filtered_test_cases.append(test_case)
> - cases_str = ", ".join((x.__name__ for x in filtered_test_cases))
> - self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.")
> - return filtered_test_cases
> -
> - def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool:
> - """Check whether the test case should be scheduled to be executed."""
> - match = bool(re.match(test_case_regex, test_case_name))
> - if self._test_cases_to_run:
> - return match and test_case_name in self._test_cases_to_run
> -
> - return match
> -
> -
> -def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
> - r"""Find all :class:`TestSuite`\s in a Python module.
> -
> - Args:
> - testsuite_module_path: The path to the Python module.
> -
> - Returns:
> - The list of :class:`TestSuite`\s found within the Python module.
> -
> - Raises:
> - ConfigurationError: The test suite module was not found.
> - """
> -
> - def is_test_suite(object: Any) -> bool:
> - try:
> - if issubclass(object, TestSuite) and object is not TestSuite:
> - return True
> - except TypeError:
> - return False
> - return False
> -
> - try:
> - testcase_module = importlib.import_module(testsuite_module_path)
> - except ModuleNotFoundError as e:
> - raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e
> - return [
> - test_suite_class
> - for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite)
> - ]
> diff --git a/dts/pyproject.toml b/dts/pyproject.toml
> index 28bd970ae4..8eb92b4f11 100644
> --- a/dts/pyproject.toml
> +++ b/dts/pyproject.toml
> @@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes"
> format = "pylint"
> max_line_length = 100
>
> +[tool.pylama.linter.pycodestyle]
> +ignore = "E203,W503"
> +
> [tool.pylama.linter.pydocstyle]
> convention = "google"
>
> diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
> index 5e2bac14bd..7b2a0e97f8 100644
> --- a/dts/tests/TestSuite_smoke_tests.py
> +++ b/dts/tests/TestSuite_smoke_tests.py
> @@ -21,7 +21,7 @@
> from framework.utils import REGEX_FOR_PCI_ADDRESS
>
>
> -class SmokeTests(TestSuite):
> +class TestSmokeTests(TestSuite):
> """DPDK and infrastructure smoke test suite.
>
> The test cases validate the most basic DPDK functionality needed for all other test suites.
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 6/7] dts: refactor logging configuration
2024-02-06 14:57 ` [PATCH v2 6/7] dts: refactor logging configuration Juraj Linkeš
@ 2024-02-12 16:45 ` Jeremy Spewock
2024-02-14 7:49 ` Juraj Linkeš
0 siblings, 1 reply; 44+ messages in thread
From: Jeremy Spewock @ 2024-02-12 16:45 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev
On Tue, Feb 6, 2024 at 9:57 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> Remove unused parts of the code and add useful features:
> 1. Add DTS execution stages such as execution and test suite to better
> identify where in the DTS lifecycle we are when investigating logs,
> 2. Logging to separate files in specific stages, which is mainly useful
> for having test suite logs in additional separate files.
> 3. Remove the dependence on the settings module which enhances the
> usefulness of the logger module, as it can now be imported in more
> modules.
>
> The execution stages and the files to log to are the same for all DTS
> loggers. To achieve this, we have one DTS root logger which should be
> used for handling stage switching and all other loggers are children of
> this DTS root logger. The DTS root logger is the one where we change the
> behavior of all loggers (the stage and which files to log to) and the
> child loggers just log messages under a different name.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
> dts/framework/logger.py | 235 +++++++++++-------
> dts/framework/remote_session/__init__.py | 6 +-
> .../interactive_remote_session.py | 6 +-
> .../remote_session/interactive_shell.py | 6 +-
> .../remote_session/remote_session.py | 8 +-
> dts/framework/runner.py | 19 +-
> dts/framework/test_result.py | 6 +-
> dts/framework/test_suite.py | 6 +-
> dts/framework/testbed_model/node.py | 11 +-
> dts/framework/testbed_model/os_session.py | 7 +-
> .../traffic_generator/traffic_generator.py | 6 +-
> dts/main.py | 1 -
> 12 files changed, 183 insertions(+), 134 deletions(-)
>
> diff --git a/dts/framework/logger.py b/dts/framework/logger.py
> index cfa6e8cd72..568edad82d 100644
> --- a/dts/framework/logger.py
> +++ b/dts/framework/logger.py
> @@ -5,141 +5,186 @@
>
> """DTS logger module.
>
> -DTS framework and TestSuite logs are saved in different log files.
> +The module provides several additional features:
> +
> + * The storage of DTS execution stages,
> + * Logging to console, a human-readable log file and a machine-readable log file,
> + * Optional log files for specific stages.
> """
>
> import logging
> -import os.path
> -from typing import TypedDict
> +from enum import auto
> +from logging import FileHandler, StreamHandler
> +from pathlib import Path
> +from typing import ClassVar
>
> -from .settings import SETTINGS
> +from .utils import StrEnum
>
> date_fmt = "%Y/%m/%d %H:%M:%S"
> -stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
> +stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s"
> +dts_root_logger_name = "dts"
> +
> +
> +class DtsStage(StrEnum):
> + """The DTS execution stage."""
>
> + #:
> + pre_execution = auto()
> + #:
> + execution = auto()
> + #:
> + build_target = auto()
> + #:
> + suite = auto()
> + #:
> + post_execution = auto()
>
> -class DTSLOG(logging.LoggerAdapter):
> - """DTS logger adapter class for framework and testsuites.
>
> - The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment
> - variable control the verbosity of output. If enabled, all messages will be emitted to the
> - console.
> +class DTSLogger(logging.Logger):
> + """The DTS logger class.
>
> - The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment
> - variable modify the directory where the logs will be stored.
> + The class extends the :class:`~logging.Logger` class to add the DTS execution stage information
> + to log records. The stage is common to all loggers, so it's stored in a class variable.
>
> - Attributes:
> - node: The additional identifier. Currently unused.
> - sh: The handler which emits logs to console.
> - fh: The handler which emits logs to a file.
> - verbose_fh: Just as fh, but logs with a different, more verbose, format.
> + Any time we switch to a new stage, we have the ability to log to an additional log file along
> + with a supplementary log file with machine-readable format. These two log files are used until
> + a new stage switch occurs. This is useful mainly for logging per test suite.
> """
>
> - _logger: logging.Logger
> - node: str
> - sh: logging.StreamHandler
> - fh: logging.FileHandler
> - verbose_fh: logging.FileHandler
> + _stage: ClassVar[DtsStage] = DtsStage.pre_execution
> + _extra_file_handlers: list[FileHandler] = []
>
> - def __init__(self, logger: logging.Logger, node: str = "suite"):
> - """Extend the constructor with additional handlers.
> + def __init__(self, *args, **kwargs):
> + """Extend the constructor with extra file handlers."""
> + self._extra_file_handlers = []
> + super().__init__(*args, **kwargs)
>
> - One handler logs to the console, the other one to a file, with either a regular or verbose
> - format.
> + def makeRecord(self, *args, **kwargs):
Is the return type annotation here skipped because of the `:meta private:`?
> + """Generates a record with additional stage information.
>
> - Args:
> - logger: The logger from which to create the logger adapter.
> - node: An additional identifier. Currently unused.
> + This is the default method for the :class:`~logging.Logger` class. We extend it
> + to add stage information to the record.
> +
> + :meta private:
> +
> + Returns:
> + record: The generated record with the stage information.
> """
> - self._logger = logger
> - # 1 means log everything, this will be used by file handlers if their level
> - # is not set
> - self._logger.setLevel(1)
> + record = super().makeRecord(*args, **kwargs)
> + record.stage = DTSLogger._stage
> + return record
> +
> + def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None:
> + """Add logger handlers to the DTS root logger.
> +
> + This method should be called only on the DTS root logger.
> + The log records from child loggers will propagate to these handlers.
> +
> + Three handlers are added:
>
> - self.node = node
> + * A console handler,
> + * A file handler,
> + * A supplementary file handler with machine-readable logs
> + containing more debug information.
>
> - # add handler to emit to stdout
> - sh = logging.StreamHandler()
> + All log messages will be logged to files. The log level of the console handler
> + is configurable with `verbose`.
> +
> + Args:
> + verbose: If :data:`True`, log all messages to the console.
> + If :data:`False`, log to console with the :data:`logging.INFO` level.
> + output_dir: The directory where the log files will be located.
> + The names of the log files correspond to the name of the logger instance.
> + """
> + self.setLevel(1)
> +
> + sh = StreamHandler()
> sh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
> - sh.setLevel(logging.INFO) # console handler default level
> + if not verbose:
> + sh.setLevel(logging.INFO)
> + self.addHandler(sh)
>
> - if SETTINGS.verbose is True:
> - sh.setLevel(logging.DEBUG)
> + self._add_file_handlers(Path(output_dir, self.name))
>
> - self._logger.addHandler(sh)
> - self.sh = sh
> + def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None:
> + """Set the DTS execution stage and optionally log to files.
>
> - # prepare the output folder
> - if not os.path.exists(SETTINGS.output_dir):
> - os.mkdir(SETTINGS.output_dir)
> + Set the DTS execution stage of the DTSLog class and optionally add
> + file handlers to the instance if the log file name is provided.
>
> - logging_path_prefix = os.path.join(SETTINGS.output_dir, node)
> + The file handlers log all messages. One is a regular human-readable log file and
> + the other one is a machine-readable log file with extra debug information.
>
> - fh = logging.FileHandler(f"{logging_path_prefix}.log")
> - fh.setFormatter(
> - logging.Formatter(
> - fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
> - datefmt=date_fmt,
> - )
> - )
> + Args:
> + stage: The DTS stage to set.
> + log_file_path: An optional path of the log file to use. This should be a full path
> + (either relative or absolute) without suffix (which will be appended).
> + """
> + self._remove_extra_file_handlers()
> +
> + if DTSLogger._stage != stage:
> + self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.")
> + DTSLogger._stage = stage
> +
> + if log_file_path:
> + self._extra_file_handlers.extend(self._add_file_handlers(log_file_path))
> +
> + def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]:
> + """Add file handlers to the DTS root logger.
> +
> + Add two type of file handlers:
> +
> + * A regular file handler with suffix ".log",
> + * A machine-readable file handler with suffix ".verbose.log".
> + This format provides extensive information for debugging and detailed analysis.
> +
> + Args:
> + log_file_path: The full path to the log file without suffix.
> +
> + Returns:
> + The newly created file handlers.
>
> - self._logger.addHandler(fh)
> - self.fh = fh
> + """
> + fh = FileHandler(f"{log_file_path}.log")
> + fh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
> + self.addHandler(fh)
>
> - # This outputs EVERYTHING, intended for post-mortem debugging
> - # Also optimized for processing via AWK (awk -F '|' ...)
> - verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log")
> + verbose_fh = FileHandler(f"{log_file_path}.verbose.log")
> verbose_fh.setFormatter(
> logging.Formatter(
> - fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
> + "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
> "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s",
> datefmt=date_fmt,
> )
> )
> + self.addHandler(verbose_fh)
>
> - self._logger.addHandler(verbose_fh)
> - self.verbose_fh = verbose_fh
> -
> - super(DTSLOG, self).__init__(self._logger, dict(node=self.node))
> -
> - def logger_exit(self) -> None:
> - """Remove the stream handler and the logfile handler."""
> - for handler in (self.sh, self.fh, self.verbose_fh):
> - handler.flush()
> - self._logger.removeHandler(handler)
> -
> -
> -class _LoggerDictType(TypedDict):
> - logger: DTSLOG
> - name: str
> - node: str
> -
> + return [fh, verbose_fh]
>
> -# List for saving all loggers in use
> -_Loggers: list[_LoggerDictType] = []
> + def _remove_extra_file_handlers(self) -> None:
> + """Remove any extra file handlers that have been added to the logger."""
> + if self._extra_file_handlers:
> + for extra_file_handler in self._extra_file_handlers:
> + self.removeHandler(extra_file_handler)
>
> + self._extra_file_handlers = []
>
> -def getLogger(name: str, node: str = "suite") -> DTSLOG:
> - """Get DTS logger adapter identified by name and node.
>
> - An existing logger will be returned if one with the exact name and node already exists.
> - A new one will be created and stored otherwise.
> +def get_dts_logger(name: str = None) -> DTSLogger:
> + """Return a DTS logger instance identified by `name`.
>
> Args:
> - name: The name of the logger.
> - node: An additional identifier for the logger.
> + name: If :data:`None`, return the DTS root logger.
> + If specified, return a child of the DTS root logger.
>
> Returns:
> - A logger uniquely identified by both name and node.
> + The DTS root logger or a child logger identified by `name`.
> """
> - global _Loggers
> - # return saved logger
> - logger: _LoggerDictType
> - for logger in _Loggers:
> - if logger["name"] == name and logger["node"] == node:
> - return logger["logger"]
> -
> - # return new logger
> - dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node)
> - _Loggers.append({"logger": dts_logger, "name": name, "node": node})
> - return dts_logger
> + logging.setLoggerClass(DTSLogger)
> + if name:
> + name = f"{dts_root_logger_name}.{name}"
> + else:
> + name = dts_root_logger_name
> + logger = logging.getLogger(name)
> + logging.setLoggerClass(logging.Logger)
What's the benefit of setting the logger class back to logging.Logger
here? Is the idea basically that if someone wanted to use the logging
module we shouldn't implicitly make them use our DTSLogger?
> + return logger # type: ignore[return-value]
> diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
> index 51a01d6b5e..1910c81c3c 100644
> --- a/dts/framework/remote_session/__init__.py
> +++ b/dts/framework/remote_session/__init__.py
> @@ -15,7 +15,7 @@
> # pylama:ignore=W0611
>
> from framework.config import NodeConfiguration
> -from framework.logger import DTSLOG
> +from framework.logger import DTSLogger
>
> from .interactive_remote_session import InteractiveRemoteSession
> from .interactive_shell import InteractiveShell
> @@ -26,7 +26,7 @@
>
>
> def create_remote_session(
> - node_config: NodeConfiguration, name: str, logger: DTSLOG
> + node_config: NodeConfiguration, name: str, logger: DTSLogger
> ) -> RemoteSession:
> """Factory for non-interactive remote sessions.
>
> @@ -45,7 +45,7 @@ def create_remote_session(
>
>
> def create_interactive_session(
> - node_config: NodeConfiguration, logger: DTSLOG
> + node_config: NodeConfiguration, logger: DTSLogger
> ) -> InteractiveRemoteSession:
> """Factory for interactive remote sessions.
>
> diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py
> index 1cc82e3377..c50790db79 100644
> --- a/dts/framework/remote_session/interactive_remote_session.py
> +++ b/dts/framework/remote_session/interactive_remote_session.py
> @@ -16,7 +16,7 @@
>
> from framework.config import NodeConfiguration
> from framework.exception import SSHConnectionError
> -from framework.logger import DTSLOG
> +from framework.logger import DTSLogger
>
>
> class InteractiveRemoteSession:
> @@ -50,11 +50,11 @@ class InteractiveRemoteSession:
> username: str
> password: str
> session: SSHClient
> - _logger: DTSLOG
> + _logger: DTSLogger
> _node_config: NodeConfiguration
> _transport: Transport | None
>
> - def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None:
> + def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None:
> """Connect to the node during initialization.
>
> Args:
> diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py
> index b158f963b6..5cfe202e15 100644
> --- a/dts/framework/remote_session/interactive_shell.py
> +++ b/dts/framework/remote_session/interactive_shell.py
> @@ -20,7 +20,7 @@
>
> from paramiko import Channel, SSHClient, channel # type: ignore[import]
>
> -from framework.logger import DTSLOG
> +from framework.logger import DTSLogger
> from framework.settings import SETTINGS
>
>
> @@ -38,7 +38,7 @@ class InteractiveShell(ABC):
> _stdin: channel.ChannelStdinFile
> _stdout: channel.ChannelFile
> _ssh_channel: Channel
> - _logger: DTSLOG
> + _logger: DTSLogger
> _timeout: float
> _app_args: str
>
> @@ -61,7 +61,7 @@ class InteractiveShell(ABC):
> def __init__(
> self,
> interactive_session: SSHClient,
> - logger: DTSLOG,
> + logger: DTSLogger,
> get_privileged_command: Callable[[str], str] | None,
> app_args: str = "",
> timeout: float = SETTINGS.timeout,
> diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
> index 2059f9a981..a69dc99400 100644
> --- a/dts/framework/remote_session/remote_session.py
> +++ b/dts/framework/remote_session/remote_session.py
> @@ -9,14 +9,13 @@
> the structure of the result of a command execution.
> """
>
> -
> import dataclasses
> from abc import ABC, abstractmethod
> from pathlib import PurePath
>
> from framework.config import NodeConfiguration
> from framework.exception import RemoteCommandExecutionError
> -from framework.logger import DTSLOG
> +from framework.logger import DTSLogger
> from framework.settings import SETTINGS
>
>
> @@ -75,14 +74,14 @@ class RemoteSession(ABC):
> username: str
> password: str
> history: list[CommandResult]
> - _logger: DTSLOG
> + _logger: DTSLogger
> _node_config: NodeConfiguration
>
> def __init__(
> self,
> node_config: NodeConfiguration,
> session_name: str,
> - logger: DTSLOG,
> + logger: DTSLogger,
> ):
> """Connect to the node during initialization.
>
> @@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None:
> Args:
> force: Force the closure of the connection. This may not clean up all resources.
> """
> - self._logger.logger_exit()
> self._close(force)
>
> @abstractmethod
> diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> index f58b0adc13..035e3368ef 100644
> --- a/dts/framework/runner.py
> +++ b/dts/framework/runner.py
> @@ -19,9 +19,10 @@
>
> import importlib
> import inspect
> -import logging
> +import os
> import re
> import sys
> +from pathlib import Path
> from types import MethodType
> from typing import Iterable
>
> @@ -38,7 +39,7 @@
> SSHTimeoutError,
> TestCaseVerifyError,
> )
> -from .logger import DTSLOG, getLogger
> +from .logger import DTSLogger, DtsStage, get_dts_logger
> from .settings import SETTINGS
> from .test_result import (
> BuildTargetResult,
> @@ -73,7 +74,7 @@ class DTSRunner:
> """
>
> _configuration: Configuration
> - _logger: DTSLOG
> + _logger: DTSLogger
> _result: DTSResult
> _test_suite_class_prefix: str
> _test_suite_module_prefix: str
> @@ -83,7 +84,10 @@ class DTSRunner:
> def __init__(self):
> """Initialize the instance with configuration, logger, result and string constants."""
> self._configuration = load_config()
> - self._logger = getLogger("DTSRunner")
> + self._logger = get_dts_logger()
> + if not os.path.exists(SETTINGS.output_dir):
> + os.makedirs(SETTINGS.output_dir)
> + self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir)
> self._result = DTSResult(self._logger)
> self._test_suite_class_prefix = "Test"
> self._test_suite_module_prefix = "tests.TestSuite_"
> @@ -137,6 +141,7 @@ def run(self):
>
> # for all Execution sections
> for execution in self._configuration.executions:
> + self._logger.set_stage(DtsStage.execution)
This ends up getting set twice in short succession which of course
doesn't functionally cause a problem, but I don't exactly see the
point of setting it twice. We could either set it here or set it in
the _run_execution, but i think it makes more sense to set it in the
_run_execution method as that is the method where we are doing the
setup, here we are only initializing the nodes which is still in a
sense "pre execution."
> self._logger.info(
> f"Running execution with SUT '{execution.system_under_test_node.name}'."
> )
> @@ -164,6 +169,7 @@ def run(self):
>
> finally:
> try:
> + self._logger.set_stage(DtsStage.post_execution)
> for node in (sut_nodes | tg_nodes).values():
> node.close()
> self._result.update_teardown(Result.PASS)
> @@ -419,6 +425,7 @@ def _run_execution(
>
> finally:
> try:
> + self._logger.set_stage(DtsStage.execution)
> sut_node.tear_down_execution()
> execution_result.update_teardown(Result.PASS)
> except Exception as e:
> @@ -447,6 +454,7 @@ def _run_build_target(
> with the current build target.
> test_suites_with_cases: The test suites with test cases to run.
> """
> + self._logger.set_stage(DtsStage.build_target)
> self._logger.info(f"Running build target '{build_target.name}'.")
>
> try:
> @@ -463,6 +471,7 @@ def _run_build_target(
>
> finally:
> try:
> + self._logger.set_stage(DtsStage.build_target)
> sut_node.tear_down_build_target()
> build_target_result.update_teardown(Result.PASS)
> except Exception as e:
> @@ -535,6 +544,7 @@ def _run_test_suite(
> BlockingTestSuiteError: If a blocking test suite fails.
> """
> test_suite_name = test_suite_with_cases.test_suite_class.__name__
> + self._logger.set_stage(DtsStage.suite, Path(SETTINGS.output_dir, test_suite_name))
> test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
> try:
> self._logger.info(f"Starting test suite setup: {test_suite_name}")
> @@ -683,5 +693,4 @@ def _exit_dts(self) -> None:
> if self._logger:
> self._logger.info("DTS execution has ended.")
>
> - logging.shutdown()
> sys.exit(self._result.get_return_code())
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index eedb2d20ee..28f84fd793 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -42,7 +42,7 @@
> TestSuiteConfig,
> )
> from .exception import DTSError, ErrorSeverity
> -from .logger import DTSLOG
> +from .logger import DTSLogger
> from .settings import SETTINGS
> from .test_suite import TestSuite
>
> @@ -237,13 +237,13 @@ class DTSResult(BaseResult):
> """
>
> dpdk_version: str | None
> - _logger: DTSLOG
> + _logger: DTSLogger
> _errors: list[Exception]
> _return_code: ErrorSeverity
> _stats_result: Union["Statistics", None]
> _stats_filename: str
>
> - def __init__(self, logger: DTSLOG):
> + def __init__(self, logger: DTSLogger):
> """Extend the constructor with top-level specifics.
>
> Args:
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index f9fe88093e..365f80e21a 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -21,7 +21,7 @@
> from scapy.packet import Packet, Padding # type: ignore[import]
>
> from .exception import TestCaseVerifyError
> -from .logger import DTSLOG, getLogger
> +from .logger import DTSLogger, get_dts_logger
> from .testbed_model import Port, PortLink, SutNode, TGNode
> from .utils import get_packet_summaries
>
> @@ -61,7 +61,7 @@ class TestSuite(object):
> #: Whether the test suite is blocking. A failure of a blocking test suite
> #: will block the execution of all subsequent test suites in the current build target.
> is_blocking: ClassVar[bool] = False
> - _logger: DTSLOG
> + _logger: DTSLogger
> _port_links: list[PortLink]
> _sut_port_ingress: Port
> _sut_port_egress: Port
> @@ -88,7 +88,7 @@ def __init__(
> """
> self.sut_node = sut_node
> self.tg_node = tg_node
> - self._logger = getLogger(self.__class__.__name__)
> + self._logger = get_dts_logger(self.__class__.__name__)
> self._port_links = []
> self._process_links()
> self._sut_port_ingress, self._tg_port_egress = (
> diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
> index 1a55fadf78..74061f6262 100644
> --- a/dts/framework/testbed_model/node.py
> +++ b/dts/framework/testbed_model/node.py
> @@ -23,7 +23,7 @@
> NodeConfiguration,
> )
> from framework.exception import ConfigurationError
> -from framework.logger import DTSLOG, getLogger
> +from framework.logger import DTSLogger, get_dts_logger
> from framework.settings import SETTINGS
>
> from .cpu import (
> @@ -63,7 +63,7 @@ class Node(ABC):
> name: str
> lcores: list[LogicalCore]
> ports: list[Port]
> - _logger: DTSLOG
> + _logger: DTSLogger
> _other_sessions: list[OSSession]
> _execution_config: ExecutionConfiguration
> virtual_devices: list[VirtualDevice]
> @@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration):
> """
> self.config = node_config
> self.name = node_config.name
> - self._logger = getLogger(self.name)
> + self._logger = get_dts_logger(self.name)
> self.main_session = create_session(self.config, self.name, self._logger)
>
> self._logger.info(f"Connected to node: {self.name}")
> @@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession:
> connection = create_session(
> self.config,
> session_name,
> - getLogger(session_name, node=self.name),
> + get_dts_logger(session_name),
> )
> self._other_sessions.append(connection)
> return connection
> @@ -299,7 +299,6 @@ def close(self) -> None:
> self.main_session.close()
> for session in self._other_sessions:
> session.close()
> - self._logger.logger_exit()
>
> @staticmethod
> def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
> @@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
> return func
>
>
> -def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession:
> +def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession:
> """Factory for OS-aware sessions.
>
> Args:
> diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
> index ac6bb5e112..6983aa4a77 100644
> --- a/dts/framework/testbed_model/os_session.py
> +++ b/dts/framework/testbed_model/os_session.py
> @@ -21,7 +21,6 @@
> the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux
> and other commands for other OSs. It also translates the path to match the underlying OS.
> """
> -
> from abc import ABC, abstractmethod
> from collections.abc import Iterable
> from ipaddress import IPv4Interface, IPv6Interface
> @@ -29,7 +28,7 @@
> from typing import Type, TypeVar, Union
>
> from framework.config import Architecture, NodeConfiguration, NodeInfo
> -from framework.logger import DTSLOG
> +from framework.logger import DTSLogger
> from framework.remote_session import (
> CommandResult,
> InteractiveRemoteSession,
> @@ -62,7 +61,7 @@ class OSSession(ABC):
>
> _config: NodeConfiguration
> name: str
> - _logger: DTSLOG
> + _logger: DTSLogger
> remote_session: RemoteSession
> interactive_session: InteractiveRemoteSession
>
> @@ -70,7 +69,7 @@ def __init__(
> self,
> node_config: NodeConfiguration,
> name: str,
> - logger: DTSLOG,
> + logger: DTSLogger,
> ):
> """Initialize the OS-aware session.
>
> diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
> index c49fbff488..d86d7fb532 100644
> --- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py
> +++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
> @@ -13,7 +13,7 @@
> from scapy.packet import Packet # type: ignore[import]
>
> from framework.config import TrafficGeneratorConfig
> -from framework.logger import DTSLOG, getLogger
> +from framework.logger import DTSLogger, get_dts_logger
> from framework.testbed_model.node import Node
> from framework.testbed_model.port import Port
> from framework.utils import get_packet_summaries
> @@ -28,7 +28,7 @@ class TrafficGenerator(ABC):
>
> _config: TrafficGeneratorConfig
> _tg_node: Node
> - _logger: DTSLOG
> + _logger: DTSLogger
>
> def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
> """Initialize the traffic generator.
> @@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
> """
> self._config = config
> self._tg_node = tg_node
> - self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
> + self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
>
> def send_packet(self, packet: Packet, port: Port) -> None:
> """Send `packet` and block until it is fully sent.
> diff --git a/dts/main.py b/dts/main.py
> index 1ffe8ff81f..d30c164b95 100755
> --- a/dts/main.py
> +++ b/dts/main.py
> @@ -30,5 +30,4 @@ def main() -> None:
>
> # Main program begins here
> if __name__ == "__main__":
> - logging.raiseExceptions = True
> main()
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 6/7] dts: refactor logging configuration
2024-02-12 16:45 ` Jeremy Spewock
@ 2024-02-14 7:49 ` Juraj Linkeš
2024-02-14 16:51 ` Jeremy Spewock
0 siblings, 1 reply; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-14 7:49 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev
Hi Jeremy,
Just a reminder, please strip the parts you're not commenting on.
On Mon, Feb 12, 2024 at 5:45 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
> On Tue, Feb 6, 2024 at 9:57 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >
> > Remove unused parts of the code and add useful features:
> > 1. Add DTS execution stages such as execution and test suite to better
> > identify where in the DTS lifecycle we are when investigating logs,
> > 2. Logging to separate files in specific stages, which is mainly useful
> > for having test suite logs in additional separate files.
> > 3. Remove the dependence on the settings module which enhances the
> > usefulness of the logger module, as it can now be imported in more
> > modules.
> >
> > The execution stages and the files to log to are the same for all DTS
> > loggers. To achieve this, we have one DTS root logger which should be
> > used for handling stage switching and all other loggers are children of
> > this DTS root logger. The DTS root logger is the one where we change the
> > behavior of all loggers (the stage and which files to log to) and the
> > child loggers just log messages under a different name.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> > dts/framework/logger.py | 235 +++++++++++-------
> > dts/framework/remote_session/__init__.py | 6 +-
> > .../interactive_remote_session.py | 6 +-
> > .../remote_session/interactive_shell.py | 6 +-
> > .../remote_session/remote_session.py | 8 +-
> > dts/framework/runner.py | 19 +-
> > dts/framework/test_result.py | 6 +-
> > dts/framework/test_suite.py | 6 +-
> > dts/framework/testbed_model/node.py | 11 +-
> > dts/framework/testbed_model/os_session.py | 7 +-
> > .../traffic_generator/traffic_generator.py | 6 +-
> > dts/main.py | 1 -
> > 12 files changed, 183 insertions(+), 134 deletions(-)
> >
> > diff --git a/dts/framework/logger.py b/dts/framework/logger.py
> > index cfa6e8cd72..568edad82d 100644
> > --- a/dts/framework/logger.py
> > +++ b/dts/framework/logger.py
> > @@ -5,141 +5,186 @@
> >
> > """DTS logger module.
> >
> > -DTS framework and TestSuite logs are saved in different log files.
> > +The module provides several additional features:
> > +
> > + * The storage of DTS execution stages,
> > + * Logging to console, a human-readable log file and a machine-readable log file,
> > + * Optional log files for specific stages.
> > """
> >
> > import logging
> > -import os.path
> > -from typing import TypedDict
> > +from enum import auto
> > +from logging import FileHandler, StreamHandler
> > +from pathlib import Path
> > +from typing import ClassVar
> >
> > -from .settings import SETTINGS
> > +from .utils import StrEnum
> >
> > date_fmt = "%Y/%m/%d %H:%M:%S"
> > -stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
> > +stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s"
> > +dts_root_logger_name = "dts"
> > +
> > +
> > +class DtsStage(StrEnum):
> > + """The DTS execution stage."""
> >
> > + #:
> > + pre_execution = auto()
> > + #:
> > + execution = auto()
> > + #:
> > + build_target = auto()
> > + #:
> > + suite = auto()
> > + #:
> > + post_execution = auto()
> >
> > -class DTSLOG(logging.LoggerAdapter):
> > - """DTS logger adapter class for framework and testsuites.
> >
> > - The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment
> > - variable control the verbosity of output. If enabled, all messages will be emitted to the
> > - console.
> > +class DTSLogger(logging.Logger):
> > + """The DTS logger class.
> >
> > - The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment
> > - variable modify the directory where the logs will be stored.
> > + The class extends the :class:`~logging.Logger` class to add the DTS execution stage information
> > + to log records. The stage is common to all loggers, so it's stored in a class variable.
> >
> > - Attributes:
> > - node: The additional identifier. Currently unused.
> > - sh: The handler which emits logs to console.
> > - fh: The handler which emits logs to a file.
> > - verbose_fh: Just as fh, but logs with a different, more verbose, format.
> > + Any time we switch to a new stage, we have the ability to log to an additional log file along
> > + with a supplementary log file with machine-readable format. These two log files are used until
> > + a new stage switch occurs. This is useful mainly for logging per test suite.
> > """
> >
> > - _logger: logging.Logger
> > - node: str
> > - sh: logging.StreamHandler
> > - fh: logging.FileHandler
> > - verbose_fh: logging.FileHandler
> > + _stage: ClassVar[DtsStage] = DtsStage.pre_execution
> > + _extra_file_handlers: list[FileHandler] = []
> >
> > - def __init__(self, logger: logging.Logger, node: str = "suite"):
> > - """Extend the constructor with additional handlers.
> > + def __init__(self, *args, **kwargs):
> > + """Extend the constructor with extra file handlers."""
> > + self._extra_file_handlers = []
> > + super().__init__(*args, **kwargs)
> >
> > - One handler logs to the console, the other one to a file, with either a regular or verbose
> > - format.
> > + def makeRecord(self, *args, **kwargs):
>
> Is the return type annotation here skipped because of the `:meta private:`?
>
Basically. We're modifying a method defined elsewhere, but there's no
harm in adding the return type.
> > + """Generates a record with additional stage information.
> >
> > - Args:
> > - logger: The logger from which to create the logger adapter.
> > - node: An additional identifier. Currently unused.
> > + This is the default method for the :class:`~logging.Logger` class. We extend it
> > + to add stage information to the record.
> > +
> > + :meta private:
> > +
> > + Returns:
> > + record: The generated record with the stage information.
> > """
> > - self._logger = logger
> > - # 1 means log everything, this will be used by file handlers if their level
> > - # is not set
> > - self._logger.setLevel(1)
> > + record = super().makeRecord(*args, **kwargs)
> > + record.stage = DTSLogger._stage
> > + return record
> > +
> > + def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None:
> > + """Add logger handlers to the DTS root logger.
> > +
> > + This method should be called only on the DTS root logger.
> > + The log records from child loggers will propagate to these handlers.
> > +
> > + Three handlers are added:
> >
> > - self.node = node
> > + * A console handler,
> > + * A file handler,
> > + * A supplementary file handler with machine-readable logs
> > + containing more debug information.
> >
> > - # add handler to emit to stdout
> > - sh = logging.StreamHandler()
> > + All log messages will be logged to files. The log level of the console handler
> > + is configurable with `verbose`.
> > +
> > + Args:
> > + verbose: If :data:`True`, log all messages to the console.
> > + If :data:`False`, log to console with the :data:`logging.INFO` level.
> > + output_dir: The directory where the log files will be located.
> > + The names of the log files correspond to the name of the logger instance.
> > + """
> > + self.setLevel(1)
> > +
> > + sh = StreamHandler()
> > sh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
> > - sh.setLevel(logging.INFO) # console handler default level
> > + if not verbose:
> > + sh.setLevel(logging.INFO)
> > + self.addHandler(sh)
> >
> > - if SETTINGS.verbose is True:
> > - sh.setLevel(logging.DEBUG)
> > + self._add_file_handlers(Path(output_dir, self.name))
> >
> > - self._logger.addHandler(sh)
> > - self.sh = sh
> > + def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None:
> > + """Set the DTS execution stage and optionally log to files.
> >
> > - # prepare the output folder
> > - if not os.path.exists(SETTINGS.output_dir):
> > - os.mkdir(SETTINGS.output_dir)
> > + Set the DTS execution stage of the DTSLog class and optionally add
> > + file handlers to the instance if the log file name is provided.
> >
> > - logging_path_prefix = os.path.join(SETTINGS.output_dir, node)
> > + The file handlers log all messages. One is a regular human-readable log file and
> > + the other one is a machine-readable log file with extra debug information.
> >
> > - fh = logging.FileHandler(f"{logging_path_prefix}.log")
> > - fh.setFormatter(
> > - logging.Formatter(
> > - fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
> > - datefmt=date_fmt,
> > - )
> > - )
> > + Args:
> > + stage: The DTS stage to set.
> > + log_file_path: An optional path of the log file to use. This should be a full path
> > + (either relative or absolute) without suffix (which will be appended).
> > + """
> > + self._remove_extra_file_handlers()
> > +
> > + if DTSLogger._stage != stage:
> > + self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.")
> > + DTSLogger._stage = stage
> > +
> > + if log_file_path:
> > + self._extra_file_handlers.extend(self._add_file_handlers(log_file_path))
> > +
> > + def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]:
> > + """Add file handlers to the DTS root logger.
> > +
> > + Add two type of file handlers:
> > +
> > + * A regular file handler with suffix ".log",
> > + * A machine-readable file handler with suffix ".verbose.log".
> > + This format provides extensive information for debugging and detailed analysis.
> > +
> > + Args:
> > + log_file_path: The full path to the log file without suffix.
> > +
> > + Returns:
> > + The newly created file handlers.
> >
> > - self._logger.addHandler(fh)
> > - self.fh = fh
> > + """
> > + fh = FileHandler(f"{log_file_path}.log")
> > + fh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
> > + self.addHandler(fh)
> >
> > - # This outputs EVERYTHING, intended for post-mortem debugging
> > - # Also optimized for processing via AWK (awk -F '|' ...)
> > - verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log")
> > + verbose_fh = FileHandler(f"{log_file_path}.verbose.log")
> > verbose_fh.setFormatter(
> > logging.Formatter(
> > - fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
> > + "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
> > "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s",
> > datefmt=date_fmt,
> > )
> > )
> > + self.addHandler(verbose_fh)
> >
> > - self._logger.addHandler(verbose_fh)
> > - self.verbose_fh = verbose_fh
> > -
> > - super(DTSLOG, self).__init__(self._logger, dict(node=self.node))
> > -
> > - def logger_exit(self) -> None:
> > - """Remove the stream handler and the logfile handler."""
> > - for handler in (self.sh, self.fh, self.verbose_fh):
> > - handler.flush()
> > - self._logger.removeHandler(handler)
> > -
> > -
> > -class _LoggerDictType(TypedDict):
> > - logger: DTSLOG
> > - name: str
> > - node: str
> > -
> > + return [fh, verbose_fh]
> >
> > -# List for saving all loggers in use
> > -_Loggers: list[_LoggerDictType] = []
> > + def _remove_extra_file_handlers(self) -> None:
> > + """Remove any extra file handlers that have been added to the logger."""
> > + if self._extra_file_handlers:
> > + for extra_file_handler in self._extra_file_handlers:
> > + self.removeHandler(extra_file_handler)
> >
> > + self._extra_file_handlers = []
> >
> > -def getLogger(name: str, node: str = "suite") -> DTSLOG:
> > - """Get DTS logger adapter identified by name and node.
> >
> > - An existing logger will be returned if one with the exact name and node already exists.
> > - A new one will be created and stored otherwise.
> > +def get_dts_logger(name: str = None) -> DTSLogger:
> > + """Return a DTS logger instance identified by `name`.
> >
> > Args:
> > - name: The name of the logger.
> > - node: An additional identifier for the logger.
> > + name: If :data:`None`, return the DTS root logger.
> > + If specified, return a child of the DTS root logger.
> >
> > Returns:
> > - A logger uniquely identified by both name and node.
> > + The DTS root logger or a child logger identified by `name`.
> > """
> > - global _Loggers
> > - # return saved logger
> > - logger: _LoggerDictType
> > - for logger in _Loggers:
> > - if logger["name"] == name and logger["node"] == node:
> > - return logger["logger"]
> > -
> > - # return new logger
> > - dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node)
> > - _Loggers.append({"logger": dts_logger, "name": name, "node": node})
> > - return dts_logger
> > + logging.setLoggerClass(DTSLogger)
> > + if name:
> > + name = f"{dts_root_logger_name}.{name}"
> > + else:
> > + name = dts_root_logger_name
> > + logger = logging.getLogger(name)
> > + logging.setLoggerClass(logging.Logger)
>
> What's the benefit of setting the logger class back to logging.Logger
> here? Is the idea basically that if someone wanted to use the logging
> module we shouldn't implicitly make them use our DTSLogger?
>
Yes. We should actually set it to whatever was there before (it may
not be logging.Logger), so I'll change that.+
> > + return logger # type: ignore[return-value]
> > diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
> > index 51a01d6b5e..1910c81c3c 100644
> > --- a/dts/framework/remote_session/__init__.py
> > +++ b/dts/framework/remote_session/__init__.py
> > @@ -15,7 +15,7 @@
> > # pylama:ignore=W0611
> >
> > from framework.config import NodeConfiguration
> > -from framework.logger import DTSLOG
> > +from framework.logger import DTSLogger
> >
> > from .interactive_remote_session import InteractiveRemoteSession
> > from .interactive_shell import InteractiveShell
> > @@ -26,7 +26,7 @@
> >
> >
> > def create_remote_session(
> > - node_config: NodeConfiguration, name: str, logger: DTSLOG
> > + node_config: NodeConfiguration, name: str, logger: DTSLogger
> > ) -> RemoteSession:
> > """Factory for non-interactive remote sessions.
> >
> > @@ -45,7 +45,7 @@ def create_remote_session(
> >
> >
> > def create_interactive_session(
> > - node_config: NodeConfiguration, logger: DTSLOG
> > + node_config: NodeConfiguration, logger: DTSLogger
> > ) -> InteractiveRemoteSession:
> > """Factory for interactive remote sessions.
> >
> > diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py
> > index 1cc82e3377..c50790db79 100644
> > --- a/dts/framework/remote_session/interactive_remote_session.py
> > +++ b/dts/framework/remote_session/interactive_remote_session.py
> > @@ -16,7 +16,7 @@
> >
> > from framework.config import NodeConfiguration
> > from framework.exception import SSHConnectionError
> > -from framework.logger import DTSLOG
> > +from framework.logger import DTSLogger
> >
> >
> > class InteractiveRemoteSession:
> > @@ -50,11 +50,11 @@ class InteractiveRemoteSession:
> > username: str
> > password: str
> > session: SSHClient
> > - _logger: DTSLOG
> > + _logger: DTSLogger
> > _node_config: NodeConfiguration
> > _transport: Transport | None
> >
> > - def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None:
> > + def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None:
> > """Connect to the node during initialization.
> >
> > Args:
> > diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py
> > index b158f963b6..5cfe202e15 100644
> > --- a/dts/framework/remote_session/interactive_shell.py
> > +++ b/dts/framework/remote_session/interactive_shell.py
> > @@ -20,7 +20,7 @@
> >
> > from paramiko import Channel, SSHClient, channel # type: ignore[import]
> >
> > -from framework.logger import DTSLOG
> > +from framework.logger import DTSLogger
> > from framework.settings import SETTINGS
> >
> >
> > @@ -38,7 +38,7 @@ class InteractiveShell(ABC):
> > _stdin: channel.ChannelStdinFile
> > _stdout: channel.ChannelFile
> > _ssh_channel: Channel
> > - _logger: DTSLOG
> > + _logger: DTSLogger
> > _timeout: float
> > _app_args: str
> >
> > @@ -61,7 +61,7 @@ class InteractiveShell(ABC):
> > def __init__(
> > self,
> > interactive_session: SSHClient,
> > - logger: DTSLOG,
> > + logger: DTSLogger,
> > get_privileged_command: Callable[[str], str] | None,
> > app_args: str = "",
> > timeout: float = SETTINGS.timeout,
> > diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
> > index 2059f9a981..a69dc99400 100644
> > --- a/dts/framework/remote_session/remote_session.py
> > +++ b/dts/framework/remote_session/remote_session.py
> > @@ -9,14 +9,13 @@
> > the structure of the result of a command execution.
> > """
> >
> > -
> > import dataclasses
> > from abc import ABC, abstractmethod
> > from pathlib import PurePath
> >
> > from framework.config import NodeConfiguration
> > from framework.exception import RemoteCommandExecutionError
> > -from framework.logger import DTSLOG
> > +from framework.logger import DTSLogger
> > from framework.settings import SETTINGS
> >
> >
> > @@ -75,14 +74,14 @@ class RemoteSession(ABC):
> > username: str
> > password: str
> > history: list[CommandResult]
> > - _logger: DTSLOG
> > + _logger: DTSLogger
> > _node_config: NodeConfiguration
> >
> > def __init__(
> > self,
> > node_config: NodeConfiguration,
> > session_name: str,
> > - logger: DTSLOG,
> > + logger: DTSLogger,
> > ):
> > """Connect to the node during initialization.
> >
> > @@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None:
> > Args:
> > force: Force the closure of the connection. This may not clean up all resources.
> > """
> > - self._logger.logger_exit()
> > self._close(force)
> >
> > @abstractmethod
> > diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> > index f58b0adc13..035e3368ef 100644
> > --- a/dts/framework/runner.py
> > +++ b/dts/framework/runner.py
> > @@ -19,9 +19,10 @@
> >
> > import importlib
> > import inspect
> > -import logging
> > +import os
> > import re
> > import sys
> > +from pathlib import Path
> > from types import MethodType
> > from typing import Iterable
> >
> > @@ -38,7 +39,7 @@
> > SSHTimeoutError,
> > TestCaseVerifyError,
> > )
> > -from .logger import DTSLOG, getLogger
> > +from .logger import DTSLogger, DtsStage, get_dts_logger
> > from .settings import SETTINGS
> > from .test_result import (
> > BuildTargetResult,
> > @@ -73,7 +74,7 @@ class DTSRunner:
> > """
> >
> > _configuration: Configuration
> > - _logger: DTSLOG
> > + _logger: DTSLogger
> > _result: DTSResult
> > _test_suite_class_prefix: str
> > _test_suite_module_prefix: str
> > @@ -83,7 +84,10 @@ class DTSRunner:
> > def __init__(self):
> > """Initialize the instance with configuration, logger, result and string constants."""
> > self._configuration = load_config()
> > - self._logger = getLogger("DTSRunner")
> > + self._logger = get_dts_logger()
> > + if not os.path.exists(SETTINGS.output_dir):
> > + os.makedirs(SETTINGS.output_dir)
> > + self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir)
> > self._result = DTSResult(self._logger)
> > self._test_suite_class_prefix = "Test"
> > self._test_suite_module_prefix = "tests.TestSuite_"
> > @@ -137,6 +141,7 @@ def run(self):
> >
> > # for all Execution sections
> > for execution in self._configuration.executions:
> > + self._logger.set_stage(DtsStage.execution)
>
> This ends up getting set twice in short succession which of course
> doesn't functionally cause a problem, but I don't exactly see the
> point of setting it twice.
I'm not sure what you're referring to. The stage is only set once at
the start of each execution (and more generally, at the start of each
stage). It's also set in each finally block to properly mark the
cleanup of that stage. There are two finally blocks in the execution
stage where that could happen, but otherwise it should be set exactly
once.
> We could either set it here or set it in
> the _run_execution, but i think it makes more sense to set it in the
> _run_execution method as that is the method where we are doing the
> setup, here we are only initializing the nodes which is still in a
> sense "pre execution."
Init nodes was pre-execution before the patch. I've moved it to
execution because of two reasons:
1. The nodes are actually defined per-execution. It just made more
sense to move them to execution. Also, if an error happens while
connecting to a node, we don't want to abort the whole DTS run, just
the execution, as the next execution could be connecting to a
different node.
2. We're not just doing node init here, we're also discovering which
test cases to run. This is essential to do as soon as possible (before
anything else happens in the execution, such as connecting the nodes)
so that we can mark blocked test cases in case of an error. Test case
discovery is definitely part of each execution and putting node init
after that was a natural consequence. There's an argument for
discovering test cases of all executions before actually running any
of the executions. It's certainly doable, but the code doesn't look
that good (when not modifying the original execution config (which we
shouldn't do) - I've tried) and there's actually not much of a point
to do it this way, the result is almost the same. Where it makes a
difference is when there's an error in test suite configuration and
later executions - the user would have to wait for the previous
execution to fully run to discover the error).
>
>
> > self._logger.info(
> > f"Running execution with SUT '{execution.system_under_test_node.name}'."
> > )
> > @@ -164,6 +169,7 @@ def run(self):
> >
> > finally:
> > try:
> > + self._logger.set_stage(DtsStage.post_execution)
> > for node in (sut_nodes | tg_nodes).values():
> > node.close()
> > self._result.update_teardown(Result.PASS)
> > @@ -419,6 +425,7 @@ def _run_execution(
> >
> > finally:
> > try:
> > + self._logger.set_stage(DtsStage.execution)
> > sut_node.tear_down_execution()
> > execution_result.update_teardown(Result.PASS)
> > except Exception as e:
> > @@ -447,6 +454,7 @@ def _run_build_target(
> > with the current build target.
> > test_suites_with_cases: The test suites with test cases to run.
> > """
> > + self._logger.set_stage(DtsStage.build_target)
> > self._logger.info(f"Running build target '{build_target.name}'.")
> >
> > try:
> > @@ -463,6 +471,7 @@ def _run_build_target(
> >
> > finally:
> > try:
> > + self._logger.set_stage(DtsStage.build_target)
> > sut_node.tear_down_build_target()
> > build_target_result.update_teardown(Result.PASS)
> > except Exception as e:
> > @@ -535,6 +544,7 @@ def _run_test_suite(
> > BlockingTestSuiteError: If a blocking test suite fails.
> > """
> > test_suite_name = test_suite_with_cases.test_suite_class.__name__
> > + self._logger.set_stage(DtsStage.suite, Path(SETTINGS.output_dir, test_suite_name))
> > test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
> > try:
> > self._logger.info(f"Starting test suite setup: {test_suite_name}")
> > @@ -683,5 +693,4 @@ def _exit_dts(self) -> None:
> > if self._logger:
> > self._logger.info("DTS execution has ended.")
> >
> > - logging.shutdown()
> > sys.exit(self._result.get_return_code())
> > diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> > index eedb2d20ee..28f84fd793 100644
> > --- a/dts/framework/test_result.py
> > +++ b/dts/framework/test_result.py
> > @@ -42,7 +42,7 @@
> > TestSuiteConfig,
> > )
> > from .exception import DTSError, ErrorSeverity
> > -from .logger import DTSLOG
> > +from .logger import DTSLogger
> > from .settings import SETTINGS
> > from .test_suite import TestSuite
> >
> > @@ -237,13 +237,13 @@ class DTSResult(BaseResult):
> > """
> >
> > dpdk_version: str | None
> > - _logger: DTSLOG
> > + _logger: DTSLogger
> > _errors: list[Exception]
> > _return_code: ErrorSeverity
> > _stats_result: Union["Statistics", None]
> > _stats_filename: str
> >
> > - def __init__(self, logger: DTSLOG):
> > + def __init__(self, logger: DTSLogger):
> > """Extend the constructor with top-level specifics.
> >
> > Args:
> > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> > index f9fe88093e..365f80e21a 100644
> > --- a/dts/framework/test_suite.py
> > +++ b/dts/framework/test_suite.py
> > @@ -21,7 +21,7 @@
> > from scapy.packet import Packet, Padding # type: ignore[import]
> >
> > from .exception import TestCaseVerifyError
> > -from .logger import DTSLOG, getLogger
> > +from .logger import DTSLogger, get_dts_logger
> > from .testbed_model import Port, PortLink, SutNode, TGNode
> > from .utils import get_packet_summaries
> >
> > @@ -61,7 +61,7 @@ class TestSuite(object):
> > #: Whether the test suite is blocking. A failure of a blocking test suite
> > #: will block the execution of all subsequent test suites in the current build target.
> > is_blocking: ClassVar[bool] = False
> > - _logger: DTSLOG
> > + _logger: DTSLogger
> > _port_links: list[PortLink]
> > _sut_port_ingress: Port
> > _sut_port_egress: Port
> > @@ -88,7 +88,7 @@ def __init__(
> > """
> > self.sut_node = sut_node
> > self.tg_node = tg_node
> > - self._logger = getLogger(self.__class__.__name__)
> > + self._logger = get_dts_logger(self.__class__.__name__)
> > self._port_links = []
> > self._process_links()
> > self._sut_port_ingress, self._tg_port_egress = (
> > diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
> > index 1a55fadf78..74061f6262 100644
> > --- a/dts/framework/testbed_model/node.py
> > +++ b/dts/framework/testbed_model/node.py
> > @@ -23,7 +23,7 @@
> > NodeConfiguration,
> > )
> > from framework.exception import ConfigurationError
> > -from framework.logger import DTSLOG, getLogger
> > +from framework.logger import DTSLogger, get_dts_logger
> > from framework.settings import SETTINGS
> >
> > from .cpu import (
> > @@ -63,7 +63,7 @@ class Node(ABC):
> > name: str
> > lcores: list[LogicalCore]
> > ports: list[Port]
> > - _logger: DTSLOG
> > + _logger: DTSLogger
> > _other_sessions: list[OSSession]
> > _execution_config: ExecutionConfiguration
> > virtual_devices: list[VirtualDevice]
> > @@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration):
> > """
> > self.config = node_config
> > self.name = node_config.name
> > - self._logger = getLogger(self.name)
> > + self._logger = get_dts_logger(self.name)
> > self.main_session = create_session(self.config, self.name, self._logger)
> >
> > self._logger.info(f"Connected to node: {self.name}")
> > @@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession:
> > connection = create_session(
> > self.config,
> > session_name,
> > - getLogger(session_name, node=self.name),
> > + get_dts_logger(session_name),
> > )
> > self._other_sessions.append(connection)
> > return connection
> > @@ -299,7 +299,6 @@ def close(self) -> None:
> > self.main_session.close()
> > for session in self._other_sessions:
> > session.close()
> > - self._logger.logger_exit()
> >
> > @staticmethod
> > def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
> > @@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
> > return func
> >
> >
> > -def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession:
> > +def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession:
> > """Factory for OS-aware sessions.
> >
> > Args:
> > diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
> > index ac6bb5e112..6983aa4a77 100644
> > --- a/dts/framework/testbed_model/os_session.py
> > +++ b/dts/framework/testbed_model/os_session.py
> > @@ -21,7 +21,6 @@
> > the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux
> > and other commands for other OSs. It also translates the path to match the underlying OS.
> > """
> > -
> > from abc import ABC, abstractmethod
> > from collections.abc import Iterable
> > from ipaddress import IPv4Interface, IPv6Interface
> > @@ -29,7 +28,7 @@
> > from typing import Type, TypeVar, Union
> >
> > from framework.config import Architecture, NodeConfiguration, NodeInfo
> > -from framework.logger import DTSLOG
> > +from framework.logger import DTSLogger
> > from framework.remote_session import (
> > CommandResult,
> > InteractiveRemoteSession,
> > @@ -62,7 +61,7 @@ class OSSession(ABC):
> >
> > _config: NodeConfiguration
> > name: str
> > - _logger: DTSLOG
> > + _logger: DTSLogger
> > remote_session: RemoteSession
> > interactive_session: InteractiveRemoteSession
> >
> > @@ -70,7 +69,7 @@ def __init__(
> > self,
> > node_config: NodeConfiguration,
> > name: str,
> > - logger: DTSLOG,
> > + logger: DTSLogger,
> > ):
> > """Initialize the OS-aware session.
> >
> > diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
> > index c49fbff488..d86d7fb532 100644
> > --- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py
> > +++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
> > @@ -13,7 +13,7 @@
> > from scapy.packet import Packet # type: ignore[import]
> >
> > from framework.config import TrafficGeneratorConfig
> > -from framework.logger import DTSLOG, getLogger
> > +from framework.logger import DTSLogger, get_dts_logger
> > from framework.testbed_model.node import Node
> > from framework.testbed_model.port import Port
> > from framework.utils import get_packet_summaries
> > @@ -28,7 +28,7 @@ class TrafficGenerator(ABC):
> >
> > _config: TrafficGeneratorConfig
> > _tg_node: Node
> > - _logger: DTSLOG
> > + _logger: DTSLogger
> >
> > def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
> > """Initialize the traffic generator.
> > @@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
> > """
> > self._config = config
> > self._tg_node = tg_node
> > - self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
> > + self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
> >
> > def send_packet(self, packet: Packet, port: Port) -> None:
> > """Send `packet` and block until it is fully sent.
> > diff --git a/dts/main.py b/dts/main.py
> > index 1ffe8ff81f..d30c164b95 100755
> > --- a/dts/main.py
> > +++ b/dts/main.py
> > @@ -30,5 +30,4 @@ def main() -> None:
> >
> > # Main program begins here
> > if __name__ == "__main__":
> > - logging.raiseExceptions = True
> > main()
> > --
> > 2.34.1
> >
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 3/7] dts: filter test suites in executions
2024-02-12 16:44 ` Jeremy Spewock
@ 2024-02-14 9:55 ` Juraj Linkeš
0 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-14 9:55 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev
On Mon, Feb 12, 2024 at 5:44 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
> On Tue, Feb 6, 2024 at 9:57 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >
> > We're currently filtering which test cases to run after some setup
> > steps, such as DPDK build, have already been taken. This prohibits us to
> > mark the test suites and cases that were supposed to be run as blocked
> > when an earlier setup fails, as that information is not available at
> > that time.
> >
> > To remedy this, move the filtering to the beginning of each execution.
> > This is the first action taken in each execution and if we can't filter
> > the test cases, such as due to invalid inputs, we abort the whole
> > execution. No test suites nor cases will be marked as blocked as we
> > don't know which were supposed to be run.
> >
> > On top of that, the filtering takes place in the TestSuite class, which
> > should only concern itself with test suite and test case logic, not the
> > processing behind the scenes. The logic has been moved to DTSRunner
> > which should do all the processing needed to run test suites.
> >
> > The filtering itself introduces a few changes/assumptions which are more
> > sensible than before:
> > 1. Assumption: There is just one TestSuite child class in each test
> > suite module. This was an implicit assumption before as we couldn't
> > specify the TestSuite classes in the test run configuration, just the
> > modules. The name of the TestSuite child class starts with "Test" and
> > then corresponds to the name of the module with CamelCase naming.
> > 2. Unknown test cases specified both in the test run configuration and
> > the environment variable/command line argument are no longer silently
> > ignored. This is a quality of life improvement for users, as they
> > could easily be not aware of the silent ignoration.
> >
> > Also, a change in the code results in pycodestyle warning and error:
> > [E] E203 whitespace before ':'
> > [W] W503 line break before binary operator
> >
> > These two are not PEP8 compliant, so they're disabled.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> > dts/framework/config/__init__.py | 24 +-
> > dts/framework/config/conf_yaml_schema.json | 2 +-
> > dts/framework/runner.py | 426 +++++++++++++++------
> > dts/framework/settings.py | 3 +-
> > dts/framework/test_result.py | 34 ++
> > dts/framework/test_suite.py | 85 +---
> > dts/pyproject.toml | 3 +
> > dts/tests/TestSuite_smoke_tests.py | 2 +-
> > 8 files changed, 382 insertions(+), 197 deletions(-)
> >
> > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
> > index 62eded7f04..c6a93b3b89 100644
> > --- a/dts/framework/config/__init__.py
> > +++ b/dts/framework/config/__init__.py
> > @@ -36,7 +36,7 @@
> > import json
> > import os.path
> > import pathlib
> > -from dataclasses import dataclass
> > +from dataclasses import dataclass, fields
> > from enum import auto, unique
> > from typing import Union
> >
> > @@ -506,6 +506,28 @@ def from_dict(
> > vdevs=vdevs,
> > )
> >
> > + def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration":
> > + """Create a shallow copy with any of the fields modified.
> > +
> > + The only new data are those passed to this method.
> > + The rest are copied from the object's fields calling the method.
> > +
> > + Args:
> > + **kwargs: The names and types of keyword arguments are defined
> > + by the fields of the :class:`ExecutionConfiguration` class.
> > +
> > + Returns:
> > + The copied and modified execution configuration.
> > + """
> > + new_config = {}
> > + for field in fields(self):
> > + if field.name in kwargs:
> > + new_config[field.name] = kwargs[field.name]
> > + else:
> > + new_config[field.name] = getattr(self, field.name)
> > +
> > + return ExecutionConfiguration(**new_config)
> > +
> >
> > @dataclass(slots=True, frozen=True)
> > class Configuration:
> > diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
> > index 84e45fe3c2..051b079fe4 100644
> > --- a/dts/framework/config/conf_yaml_schema.json
> > +++ b/dts/framework/config/conf_yaml_schema.json
> > @@ -197,7 +197,7 @@
> > },
> > "cases": {
> > "type": "array",
> > - "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.",
> > + "description": "If specified, only this subset of test suite's test cases will be run.",
> > "items": {
> > "type": "string"
> > },
> > diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> > index 933685d638..3e95cf9e26 100644
> > --- a/dts/framework/runner.py
> > +++ b/dts/framework/runner.py
> > @@ -17,17 +17,27 @@
> > and the test case stage runs test cases individually.
> > """
> >
> > +import importlib
> > +import inspect
> > import logging
> > +import re
> > import sys
> > from types import MethodType
> > +from typing import Iterable
> >
> > from .config import (
> > BuildTargetConfiguration,
> > + Configuration,
> > ExecutionConfiguration,
> > TestSuiteConfig,
> > load_config,
> > )
> > -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
> > +from .exception import (
> > + BlockingTestSuiteError,
> > + ConfigurationError,
> > + SSHTimeoutError,
> > + TestCaseVerifyError,
> > +)
> > from .logger import DTSLOG, getLogger
> > from .settings import SETTINGS
> > from .test_result import (
> > @@ -37,8 +47,9 @@
> > Result,
> > TestCaseResult,
> > TestSuiteResult,
> > + TestSuiteWithCases,
> > )
> > -from .test_suite import TestSuite, get_test_suites
> > +from .test_suite import TestSuite
> > from .testbed_model import SutNode, TGNode
> >
> >
> > @@ -59,13 +70,23 @@ class DTSRunner:
> > given execution, the next execution begins.
> > """
> >
> > + _configuration: Configuration
> > _logger: DTSLOG
> > _result: DTSResult
> > + _test_suite_class_prefix: str
> > + _test_suite_module_prefix: str
> > + _func_test_case_regex: str
> > + _perf_test_case_regex: str
> >
> > def __init__(self):
> > - """Initialize the instance with logger and result."""
> > + """Initialize the instance with configuration, logger, result and string constants."""
> > + self._configuration = load_config()
> > self._logger = getLogger("DTSRunner")
> > self._result = DTSResult(self._logger)
> > + self._test_suite_class_prefix = "Test"
> > + self._test_suite_module_prefix = "tests.TestSuite_"
> > + self._func_test_case_regex = r"test_(?!perf_)"
> > + self._perf_test_case_regex = r"test_perf_"
> >
> > def run(self):
> > """Run all build targets in all executions from the test run configuration.
> > @@ -106,29 +127,28 @@ def run(self):
> > try:
> > # check the python version of the server that runs dts
> > self._check_dts_python_version()
> > + self._result.update_setup(Result.PASS)
> >
> > # for all Execution sections
> > - for execution in load_config().executions:
> > - sut_node = sut_nodes.get(execution.system_under_test_node.name)
> > - tg_node = tg_nodes.get(execution.traffic_generator_node.name)
> > -
> > + for execution in self._configuration.executions:
> > + self._logger.info(
> > + f"Running execution with SUT '{execution.system_under_test_node.name}'."
> > + )
> > + execution_result = self._result.add_execution(execution.system_under_test_node)
> > try:
> > - if not sut_node:
> > - sut_node = SutNode(execution.system_under_test_node)
> > - sut_nodes[sut_node.name] = sut_node
> > - if not tg_node:
> > - tg_node = TGNode(execution.traffic_generator_node)
> > - tg_nodes[tg_node.name] = tg_node
> > - self._result.update_setup(Result.PASS)
> > + test_suites_with_cases = self._get_test_suites_with_cases(
> > + execution.test_suites, execution.func, execution.perf
> > + )
> > except Exception as e:
> > - failed_node = execution.system_under_test_node.name
> > - if sut_node:
> > - failed_node = execution.traffic_generator_node.name
> > - self._logger.exception(f"The Creation of node {failed_node} failed.")
> > - self._result.update_setup(Result.FAIL, e)
> > + self._logger.exception(
> > + f"Invalid test suite configuration found: " f"{execution.test_suites}."
> > + )
> > + execution_result.update_setup(Result.FAIL, e)
> >
> > else:
> > - self._run_execution(sut_node, tg_node, execution)
> > + self._connect_nodes_and_run_execution(
> > + sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases
> > + )
> >
> > except Exception as e:
> > self._logger.exception("An unexpected error has occurred.")
> > @@ -163,11 +183,204 @@ def _check_dts_python_version(self) -> None:
> > )
> > self._logger.warning("Please use Python >= 3.10 instead.")
> >
> > + def _get_test_suites_with_cases(
> > + self,
> > + test_suite_configs: list[TestSuiteConfig],
> > + func: bool,
> > + perf: bool,
> > + ) -> list[TestSuiteWithCases]:
> > + """Test suites with test cases discovery.
> > +
> > + The test suites with test cases defined in the user configuration are discovered
> > + and stored for future use so that we don't import the modules twice and so that
> > + the list of test suites with test cases is available for recording right away.
> > +
> > + Args:
> > + test_suite_configs: Test suite configurations.
> > + func: Whether to include functional test cases in the final list.
> > + perf: Whether to include performance test cases in the final list.
> > +
> > + Returns:
> > + The discovered test suites, each with test cases.
> > + """
> > + test_suites_with_cases = []
> > +
> > + for test_suite_config in test_suite_configs:
> > + test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
> > + test_cases = []
> > + func_test_cases, perf_test_cases = self._filter_test_cases(
> > + test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases)
> > + )
> > + if func:
> > + test_cases.extend(func_test_cases)
> > + if perf:
> > + test_cases.extend(perf_test_cases)
> > +
> > + test_suites_with_cases.append(
> > + TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases)
> > + )
> > +
> > + return test_suites_with_cases
> > +
> > + def _get_test_suite_class(self, test_suite_name: str) -> type[TestSuite]:
> > + """Find the :class:`TestSuite` class with `test_suite_name` in the corresponding module.
> > +
> > + The method assumes that the :class:`TestSuite` class starts
> > + with `self._test_suite_class_prefix`,
> > + continuing with `test_suite_name` with CamelCase convention.
> > + It also assumes there's only one test suite in each module and the module name
> > + is `test_suite_name` prefixed with `self._test_suite_module_prefix`.
> > +
> > + The CamelCase convention is not tested, only lowercase strings are compared.
> > +
> > + Args:
> > + test_suite_name: The name of the test suite to find.
> > +
> > + Returns:
> > + The found test suite.
> > +
> > + Raises:
> > + ConfigurationError: If the corresponding module is not found or
> > + a valid :class:`TestSuite` is not found in the module.
> > + """
> > +
> > + def is_test_suite(object) -> bool:
> > + """Check whether `object` is a :class:`TestSuite`.
> > +
> > + The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself.
> > +
> > + Args:
> > + object: The object to be checked.
> > +
> > + Returns:
> > + :data:`True` if `object` is a subclass of `TestSuite`.
> > + """
> > + try:
> > + if issubclass(object, TestSuite) and object is not TestSuite:
> > + return True
> > + except TypeError:
> > + return False
> > + return False
> > +
> > + testsuite_module_path = f"{self._test_suite_module_prefix}{test_suite_name}"
> > + try:
> > + test_suite_module = importlib.import_module(testsuite_module_path)
> > + except ModuleNotFoundError as e:
> > + raise ConfigurationError(
> > + f"Test suite module '{testsuite_module_path}' not found."
> > + ) from e
> > +
> > + lowercase_suite_name = test_suite_name.replace("_", "").lower()
> > + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite):
> > + if (
> > + class_name.startswith(self._test_suite_class_prefix)
> > + and lowercase_suite_name == class_name[len(self._test_suite_class_prefix) :].lower()
> > + ):
>
> Would it be simpler to instead just make lowercase_suite_name =
> f"{self._test_suite_class_prefix}{test_suite_name.replace("_",
> "").lower()}" so that you can just directly compare class_name ==
> lowercase_suite_name? Both ways should have the exact same result of
> course so it isn't important, I was just curious.
>
I've looked at how the code looks and it is better. I also changed
some of the variable names (test_suite_name -> module_name and
lowercase_suite_name -> lowercase_suite_to_find), updated the
docstring and now I'm much happier with the result.
> > + return class_obj
> > + raise ConfigurationError(
> > + f"Couldn't find any valid test suites in {test_suite_module.__name__}."
> > + )
> > +
> > + def _filter_test_cases(
> > + self, test_suite_class: type[TestSuite], test_cases_to_run: set[str]
> > + ) -> tuple[list[MethodType], list[MethodType]]:
> > + """Filter `test_cases_to_run` from `test_suite_class`.
> > +
> > + There are two rounds of filtering if `test_cases_to_run` is not empty.
> > + The first filters `test_cases_to_run` from all methods of `test_suite_class`.
> > + Then the methods are separated into functional and performance test cases.
> > + If a method doesn't match neither the functional nor performance name prefix, it's an error.
>
> I think this is a double negative but could be either "if a method
> doesn't match either ... or ..." or "if a method matches neither ...
> nor ...". I have a small preference to the second of the two options
> though because the "neither" makes the negative more clear in my mind.
>
I'll change this, thanks for the grammar fix.
> > +
> > + Args:
> > + test_suite_class: The class of the test suite.
> > + test_cases_to_run: Test case names to filter from `test_suite_class`.
> > + If empty, return all matching test cases.
> > +
> > + Returns:
> > + A list of test case methods that should be executed.
> > +
> > + Raises:
> > + ConfigurationError: If a test case from `test_cases_to_run` is not found
> > + or it doesn't match either the functional nor performance name prefix.
> > + """
> > + func_test_cases = []
> > + perf_test_cases = []
> > + name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
> > + if test_cases_to_run:
> > + name_method_tuples = [
> > + (name, method) for name, method in name_method_tuples if name in test_cases_to_run
> > + ]
> > + if len(name_method_tuples) < len(test_cases_to_run):
> > + missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples}
> > + raise ConfigurationError(
> > + f"Test cases {missing_test_cases} not found among methods "
> > + f"of {test_suite_class.__name__}."
> > + )
> > +
> > + for test_case_name, test_case_method in name_method_tuples:
> > + if re.match(self._func_test_case_regex, test_case_name):
> > + func_test_cases.append(test_case_method)
> > + elif re.match(self._perf_test_case_regex, test_case_name):
> > + perf_test_cases.append(test_case_method)
> > + elif test_cases_to_run:
> > + raise ConfigurationError(
> > + f"Method '{test_case_name}' doesn't match neither "
> > + f"a functional nor a performance test case name."
>
> Same thing here with the double negative.
>
>
>
> > + )
> > +
> > + return func_test_cases, perf_test_cases
> > +
> > + def _connect_nodes_and_run_execution(
> > + self,
> > + sut_nodes: dict[str, SutNode],
> > + tg_nodes: dict[str, TGNode],
> > + execution: ExecutionConfiguration,
> > + execution_result: ExecutionResult,
> > + test_suites_with_cases: Iterable[TestSuiteWithCases],
> > + ) -> None:
> > + """Connect nodes, then continue to run the given execution.
> > +
> > + Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`.
> > + If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`,
> > + respectively.
> > + If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`.
> > +
> > + Args:
> > + sut_nodes: A dictionary storing connected/to be connected SUT nodes.
> > + tg_nodes: A dictionary storing connected/to be connected TG nodes.
> > + execution: An execution's test run configuration.
> > + execution_result: The execution's result.
> > + test_suites_with_cases: The test suites with test cases to run.
> > + """
> > + sut_node = sut_nodes.get(execution.system_under_test_node.name)
> > + tg_node = tg_nodes.get(execution.traffic_generator_node.name)
> > +
> > + try:
> > + if not sut_node:
> > + sut_node = SutNode(execution.system_under_test_node)
> > + sut_nodes[sut_node.name] = sut_node
> > + if not tg_node:
> > + tg_node = TGNode(execution.traffic_generator_node)
> > + tg_nodes[tg_node.name] = tg_node
> > + except Exception as e:
> > + failed_node = execution.system_under_test_node.name
> > + if sut_node:
> > + failed_node = execution.traffic_generator_node.name
> > + self._logger.exception(f"The Creation of node {failed_node} failed.")
> > + execution_result.update_setup(Result.FAIL, e)
> > +
> > + else:
> > + self._run_execution(
> > + sut_node, tg_node, execution, execution_result, test_suites_with_cases
> > + )
> > +
> > def _run_execution(
> > self,
> > sut_node: SutNode,
> > tg_node: TGNode,
> > execution: ExecutionConfiguration,
> > + execution_result: ExecutionResult,
> > + test_suites_with_cases: Iterable[TestSuiteWithCases],
> > ) -> None:
> > """Run the given execution.
> >
> > @@ -178,11 +391,11 @@ def _run_execution(
> > sut_node: The execution's SUT node.
> > tg_node: The execution's TG node.
> > execution: An execution's test run configuration.
> > + execution_result: The execution's result.
> > + test_suites_with_cases: The test suites with test cases to run.
> > """
> > self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
> > - execution_result = self._result.add_execution(sut_node.config)
> > execution_result.add_sut_info(sut_node.node_info)
> > -
> > try:
> > sut_node.set_up_execution(execution)
> > execution_result.update_setup(Result.PASS)
> > @@ -192,7 +405,10 @@ def _run_execution(
> >
> > else:
> > for build_target in execution.build_targets:
> > - self._run_build_target(sut_node, tg_node, build_target, execution, execution_result)
> > + build_target_result = execution_result.add_build_target(build_target)
> > + self._run_build_target(
> > + sut_node, tg_node, build_target, build_target_result, test_suites_with_cases
> > + )
> >
> > finally:
> > try:
> > @@ -207,8 +423,8 @@ def _run_build_target(
> > sut_node: SutNode,
> > tg_node: TGNode,
> > build_target: BuildTargetConfiguration,
> > - execution: ExecutionConfiguration,
> > - execution_result: ExecutionResult,
> > + build_target_result: BuildTargetResult,
> > + test_suites_with_cases: Iterable[TestSuiteWithCases],
> > ) -> None:
> > """Run the given build target.
> >
> > @@ -220,11 +436,11 @@ def _run_build_target(
> > sut_node: The execution's sut node.
> > tg_node: The execution's tg node.
> > build_target: A build target's test run configuration.
> > - execution: The build target's execution's test run configuration.
> > - execution_result: The execution level result object associated with the execution.
> > + build_target_result: The build target level result object associated
> > + with the current build target.
> > + test_suites_with_cases: The test suites with test cases to run.
> > """
> > self._logger.info(f"Running build target '{build_target.name}'.")
> > - build_target_result = execution_result.add_build_target(build_target)
> >
> > try:
> > sut_node.set_up_build_target(build_target)
> > @@ -236,7 +452,7 @@ def _run_build_target(
> > build_target_result.update_setup(Result.FAIL, e)
> >
> > else:
> > - self._run_test_suites(sut_node, tg_node, execution, build_target_result)
> > + self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases)
> >
> > finally:
> > try:
> > @@ -250,10 +466,10 @@ def _run_test_suites(
> > self,
> > sut_node: SutNode,
> > tg_node: TGNode,
> > - execution: ExecutionConfiguration,
> > build_target_result: BuildTargetResult,
> > + test_suites_with_cases: Iterable[TestSuiteWithCases],
> > ) -> None:
> > - """Run the execution's (possibly a subset of) test suites using the current build target.
> > + """Run `test_suites_with_cases` with the current build target.
> >
> > The method assumes the build target we're testing has already been built on the SUT node.
> > The current build target thus corresponds to the current DPDK build present on the SUT node.
> > @@ -264,22 +480,20 @@ def _run_test_suites(
> > Args:
> > sut_node: The execution's SUT node.
> > tg_node: The execution's TG node.
> > - execution: The execution's test run configuration associated
> > - with the current build target.
> > build_target_result: The build target level result object associated
> > with the current build target.
> > + test_suites_with_cases: The test suites with test cases to run.
> > """
> > end_build_target = False
> > - if not execution.skip_smoke_tests:
> > - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
> > - for test_suite_config in execution.test_suites:
> > + for test_suite_with_cases in test_suites_with_cases:
> > + test_suite_result = build_target_result.add_test_suite(
> > + test_suite_with_cases.test_suite_class.__name__
> > + )
> > try:
> > - self._run_test_suite_module(
> > - sut_node, tg_node, execution, build_target_result, test_suite_config
> > - )
> > + self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
> > except BlockingTestSuiteError as e:
> > self._logger.exception(
> > - f"An error occurred within {test_suite_config.test_suite}. "
> > + f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
> > "Skipping build target..."
> > )
> > self._result.add_error(e)
> > @@ -288,15 +502,14 @@ def _run_test_suites(
> > if end_build_target:
> > break
> >
> > - def _run_test_suite_module(
> > + def _run_test_suite(
> > self,
> > sut_node: SutNode,
> > tg_node: TGNode,
> > - execution: ExecutionConfiguration,
> > - build_target_result: BuildTargetResult,
> > - test_suite_config: TestSuiteConfig,
> > + test_suite_result: TestSuiteResult,
> > + test_suite_with_cases: TestSuiteWithCases,
> > ) -> None:
> > - """Set up, execute and tear down all test suites in a single test suite module.
> > + """Set up, execute and tear down `test_suite_with_cases`.
> >
> > The method assumes the build target we're testing has already been built on the SUT node.
> > The current build target thus corresponds to the current DPDK build present on the SUT node.
> > @@ -306,92 +519,79 @@ def _run_test_suite_module(
> >
> > Record the setup and the teardown and handle failures.
> >
> > - The test cases to execute are discovered when creating the :class:`TestSuite` object.
> > -
> > Args:
> > sut_node: The execution's SUT node.
> > tg_node: The execution's TG node.
> > - execution: The execution's test run configuration associated
> > - with the current build target.
> > - build_target_result: The build target level result object associated
> > - with the current build target.
> > - test_suite_config: Test suite test run configuration specifying the test suite module
> > - and possibly a subset of test cases of test suites in that module.
> > + test_suite_result: The test suite level result object associated
> > + with the current test suite.
> > + test_suite_with_cases: The test suite with test cases to run.
> >
> > Raises:
> > BlockingTestSuiteError: If a blocking test suite fails.
> > """
> > + test_suite_name = test_suite_with_cases.test_suite_class.__name__
> > + test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
> > try:
> > - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
> > - test_suite_classes = get_test_suites(full_suite_path)
> > - suites_str = ", ".join((x.__name__ for x in test_suite_classes))
> > - self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
> > + self._logger.info(f"Starting test suite setup: {test_suite_name}")
> > + test_suite.set_up_suite()
> > + test_suite_result.update_setup(Result.PASS)
> > + self._logger.info(f"Test suite setup successful: {test_suite_name}")
> > except Exception as e:
> > - self._logger.exception("An error occurred when searching for test suites.")
> > - self._result.update_setup(Result.ERROR, e)
> > + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
> > + test_suite_result.update_setup(Result.ERROR, e)
> >
> > else:
> > - for test_suite_class in test_suite_classes:
> > - test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases)
> > -
> > - test_suite_name = test_suite.__class__.__name__
> > - test_suite_result = build_target_result.add_test_suite(test_suite_name)
> > - try:
> > - self._logger.info(f"Starting test suite setup: {test_suite_name}")
> > - test_suite.set_up_suite()
> > - test_suite_result.update_setup(Result.PASS)
> > - self._logger.info(f"Test suite setup successful: {test_suite_name}")
> > - except Exception as e:
> > - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
> > - test_suite_result.update_setup(Result.ERROR, e)
> > -
> > - else:
> > - self._execute_test_suite(execution.func, test_suite, test_suite_result)
> > -
> > - finally:
> > - try:
> > - test_suite.tear_down_suite()
> > - sut_node.kill_cleanup_dpdk_apps()
> > - test_suite_result.update_teardown(Result.PASS)
> > - except Exception as e:
> > - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
> > - self._logger.warning(
> > - f"Test suite '{test_suite_name}' teardown failed, "
> > - f"the next test suite may be affected."
> > - )
> > - test_suite_result.update_setup(Result.ERROR, e)
> > - if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
> > - raise BlockingTestSuiteError(test_suite_name)
> > + self._execute_test_suite(
> > + test_suite,
> > + test_suite_with_cases.test_cases,
> > + test_suite_result,
> > + )
> > + finally:
> > + try:
> > + test_suite.tear_down_suite()
> > + sut_node.kill_cleanup_dpdk_apps()
> > + test_suite_result.update_teardown(Result.PASS)
> > + except Exception as e:
> > + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
> > + self._logger.warning(
> > + f"Test suite '{test_suite_name}' teardown failed, "
> > + "the next test suite may be affected."
> > + )
> > + test_suite_result.update_setup(Result.ERROR, e)
> > + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
> > + raise BlockingTestSuiteError(test_suite_name)
> >
> > def _execute_test_suite(
> > - self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
> > + self,
> > + test_suite: TestSuite,
> > + test_cases: Iterable[MethodType],
> > + test_suite_result: TestSuiteResult,
> > ) -> None:
> > - """Execute all discovered test cases in `test_suite`.
> > + """Execute all `test_cases` in `test_suite`.
> >
> > If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment
> > variable is set, in case of a test case failure, the test case will be executed again
> > until it passes or it fails that many times in addition of the first failure.
> >
> > Args:
> > - func: Whether to execute functional test cases.
> > test_suite: The test suite object.
> > + test_cases: The list of test case methods.
> > test_suite_result: The test suite level result object associated
> > with the current test suite.
> > """
> > - if func:
> > - for test_case_method in test_suite._get_functional_test_cases():
> > - test_case_name = test_case_method.__name__
> > - test_case_result = test_suite_result.add_test_case(test_case_name)
> > - all_attempts = SETTINGS.re_run + 1
> > - attempt_nr = 1
> > + for test_case_method in test_cases:
> > + test_case_name = test_case_method.__name__
> > + test_case_result = test_suite_result.add_test_case(test_case_name)
> > + all_attempts = SETTINGS.re_run + 1
> > + attempt_nr = 1
> > + self._run_test_case(test_suite, test_case_method, test_case_result)
> > + while not test_case_result and attempt_nr < all_attempts:
> > + attempt_nr += 1
> > + self._logger.info(
> > + f"Re-running FAILED test case '{test_case_name}'. "
> > + f"Attempt number {attempt_nr} out of {all_attempts}."
> > + )
> > self._run_test_case(test_suite, test_case_method, test_case_result)
> > - while not test_case_result and attempt_nr < all_attempts:
> > - attempt_nr += 1
> > - self._logger.info(
> > - f"Re-running FAILED test case '{test_case_name}'. "
> > - f"Attempt number {attempt_nr} out of {all_attempts}."
> > - )
> > - self._run_test_case(test_suite, test_case_method, test_case_result)
> >
> > def _run_test_case(
> > self,
> > @@ -399,7 +599,7 @@ def _run_test_case(
> > test_case_method: MethodType,
> > test_case_result: TestCaseResult,
> > ) -> None:
> > - """Setup, execute and teardown a test case in `test_suite`.
> > + """Setup, execute and teardown `test_case_method` from `test_suite`.
> >
> > Record the result of the setup and the teardown and handle failures.
> >
> > @@ -424,7 +624,7 @@ def _run_test_case(
> >
> > else:
> > # run test case if setup was successful
> > - self._execute_test_case(test_case_method, test_case_result)
> > + self._execute_test_case(test_suite, test_case_method, test_case_result)
> >
> > finally:
> > try:
> > @@ -440,11 +640,15 @@ def _run_test_case(
> > test_case_result.update(Result.ERROR)
> >
> > def _execute_test_case(
> > - self, test_case_method: MethodType, test_case_result: TestCaseResult
> > + self,
> > + test_suite: TestSuite,
> > + test_case_method: MethodType,
> > + test_case_result: TestCaseResult,
> > ) -> None:
> > - """Execute one test case, record the result and handle failures.
> > + """Execute `test_case_method` from `test_suite`, record the result and handle failures.
> >
> > Args:
> > + test_suite: The test suite object.
> > test_case_method: The test case method.
> > test_case_result: The test case level result object associated
> > with the current test case.
> > @@ -452,7 +656,7 @@ def _execute_test_case(
> > test_case_name = test_case_method.__name__
> > try:
> > self._logger.info(f"Starting test case execution: {test_case_name}")
> > - test_case_method()
> > + test_case_method(test_suite)
> > test_case_result.update(Result.PASS)
> > self._logger.info(f"Test case execution PASSED: {test_case_name}")
> >
> > diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> > index 609c8d0e62..2b8bfbe0ed 100644
> > --- a/dts/framework/settings.py
> > +++ b/dts/framework/settings.py
> > @@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser:
> > "--test-cases",
> > action=_env_arg("DTS_TESTCASES"),
> > default="",
> > - help="[DTS_TESTCASES] Comma-separated list of test cases to execute. "
> > - "Unknown test cases will be silently ignored.",
> > + help="[DTS_TESTCASES] Comma-separated list of test cases to execute.",
> > )
> >
> > parser.add_argument(
> > diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> > index 4467749a9d..075195fd5b 100644
> > --- a/dts/framework/test_result.py
> > +++ b/dts/framework/test_result.py
> > @@ -25,7 +25,9 @@
> >
> > import os.path
> > from collections.abc import MutableSequence
> > +from dataclasses import dataclass
> > from enum import Enum, auto
> > +from types import MethodType
> >
> > from .config import (
> > OS,
> > @@ -36,10 +38,42 @@
> > CPUType,
> > NodeConfiguration,
> > NodeInfo,
> > + TestSuiteConfig,
> > )
> > from .exception import DTSError, ErrorSeverity
> > from .logger import DTSLOG
> > from .settings import SETTINGS
> > +from .test_suite import TestSuite
> > +
> > +
> > +@dataclass(slots=True, frozen=True)
> > +class TestSuiteWithCases:
> > + """A test suite class with test case methods.
> > +
> > + An auxiliary class holding a test case class with test case methods. The intended use of this
> > + class is to hold a subset of test cases (which could be all test cases) because we don't have
> > + all the data to instantiate the class at the point of inspection. The knowledge of this subset
> > + is needed in case an error occurs before the class is instantiated and we need to record
> > + which test cases were blocked by the error.
> > +
> > + Attributes:
> > + test_suite_class: The test suite class.
> > + test_cases: The test case methods.
> > + """
> > +
> > + test_suite_class: type[TestSuite]
> > + test_cases: list[MethodType]
> > +
> > + def create_config(self) -> TestSuiteConfig:
> > + """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
> > +
> > + Returns:
> > + The :class:`TestSuiteConfig` representation.
> > + """
> > + return TestSuiteConfig(
> > + test_suite=self.test_suite_class.__name__,
> > + test_cases=[test_case.__name__ for test_case in self.test_cases],
> > + )
> >
> >
> > class Result(Enum):
> > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> > index b02fd36147..f9fe88093e 100644
> > --- a/dts/framework/test_suite.py
> > +++ b/dts/framework/test_suite.py
> > @@ -11,25 +11,17 @@
> > * Testbed (SUT, TG) configuration,
> > * Packet sending and verification,
> > * Test case verification.
> > -
> > -The module also defines a function, :func:`get_test_suites`,
> > -for gathering test suites from a Python module.
> > """
> >
> > -import importlib
> > -import inspect
> > -import re
> > from ipaddress import IPv4Interface, IPv6Interface, ip_interface
> > -from types import MethodType
> > -from typing import Any, ClassVar, Union
> > +from typing import ClassVar, Union
> >
> > from scapy.layers.inet import IP # type: ignore[import]
> > from scapy.layers.l2 import Ether # type: ignore[import]
> > from scapy.packet import Packet, Padding # type: ignore[import]
> >
> > -from .exception import ConfigurationError, TestCaseVerifyError
> > +from .exception import TestCaseVerifyError
> > from .logger import DTSLOG, getLogger
> > -from .settings import SETTINGS
> > from .testbed_model import Port, PortLink, SutNode, TGNode
> > from .utils import get_packet_summaries
> >
> > @@ -37,7 +29,6 @@
> > class TestSuite(object):
> > """The base class with building blocks needed by most test cases.
> >
> > - * Test case filtering and collection,
> > * Test suite setup/cleanup methods to override,
> > * Test case setup/cleanup methods to override,
> > * Test case verification,
> > @@ -71,7 +62,6 @@ class TestSuite(object):
> > #: will block the execution of all subsequent test suites in the current build target.
> > is_blocking: ClassVar[bool] = False
> > _logger: DTSLOG
> > - _test_cases_to_run: list[str]
> > _port_links: list[PortLink]
> > _sut_port_ingress: Port
> > _sut_port_egress: Port
> > @@ -86,24 +76,19 @@ def __init__(
> > self,
> > sut_node: SutNode,
> > tg_node: TGNode,
> > - test_cases: list[str],
> > ):
> > """Initialize the test suite testbed information and basic configuration.
> >
> > - Process what test cases to run, find links between ports and set up
> > - default IP addresses to be used when configuring them.
> > + Find links between ports and set up default IP addresses to be used when
> > + configuring them.
> >
> > Args:
> > sut_node: The SUT node where the test suite will run.
> > tg_node: The TG node where the test suite will run.
> > - test_cases: The list of test cases to execute.
> > - If empty, all test cases will be executed.
> > """
> > self.sut_node = sut_node
> > self.tg_node = tg_node
> > self._logger = getLogger(self.__class__.__name__)
> > - self._test_cases_to_run = test_cases
> > - self._test_cases_to_run.extend(SETTINGS.test_cases)
> > self._port_links = []
> > self._process_links()
> > self._sut_port_ingress, self._tg_port_egress = (
> > @@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
> > if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
> > return False
> > return True
> > -
> > - def _get_functional_test_cases(self) -> list[MethodType]:
> > - """Get all functional test cases defined in this TestSuite.
> > -
> > - Returns:
> > - The list of functional test cases of this TestSuite.
> > - """
> > - return self._get_test_cases(r"test_(?!perf_)")
> > -
> > - def _get_test_cases(self, test_case_regex: str) -> list[MethodType]:
> > - """Return a list of test cases matching test_case_regex.
> > -
> > - Returns:
> > - The list of test cases matching test_case_regex of this TestSuite.
> > - """
> > - self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.")
> > - filtered_test_cases = []
> > - for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod):
> > - if self._should_be_executed(test_case_name, test_case_regex):
> > - filtered_test_cases.append(test_case)
> > - cases_str = ", ".join((x.__name__ for x in filtered_test_cases))
> > - self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.")
> > - return filtered_test_cases
> > -
> > - def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool:
> > - """Check whether the test case should be scheduled to be executed."""
> > - match = bool(re.match(test_case_regex, test_case_name))
> > - if self._test_cases_to_run:
> > - return match and test_case_name in self._test_cases_to_run
> > -
> > - return match
> > -
> > -
> > -def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
> > - r"""Find all :class:`TestSuite`\s in a Python module.
> > -
> > - Args:
> > - testsuite_module_path: The path to the Python module.
> > -
> > - Returns:
> > - The list of :class:`TestSuite`\s found within the Python module.
> > -
> > - Raises:
> > - ConfigurationError: The test suite module was not found.
> > - """
> > -
> > - def is_test_suite(object: Any) -> bool:
> > - try:
> > - if issubclass(object, TestSuite) and object is not TestSuite:
> > - return True
> > - except TypeError:
> > - return False
> > - return False
> > -
> > - try:
> > - testcase_module = importlib.import_module(testsuite_module_path)
> > - except ModuleNotFoundError as e:
> > - raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e
> > - return [
> > - test_suite_class
> > - for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite)
> > - ]
> > diff --git a/dts/pyproject.toml b/dts/pyproject.toml
> > index 28bd970ae4..8eb92b4f11 100644
> > --- a/dts/pyproject.toml
> > +++ b/dts/pyproject.toml
> > @@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes"
> > format = "pylint"
> > max_line_length = 100
> >
> > +[tool.pylama.linter.pycodestyle]
> > +ignore = "E203,W503"
> > +
> > [tool.pylama.linter.pydocstyle]
> > convention = "google"
> >
> > diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
> > index 5e2bac14bd..7b2a0e97f8 100644
> > --- a/dts/tests/TestSuite_smoke_tests.py
> > +++ b/dts/tests/TestSuite_smoke_tests.py
> > @@ -21,7 +21,7 @@
> > from framework.utils import REGEX_FOR_PCI_ADDRESS
> >
> >
> > -class SmokeTests(TestSuite):
> > +class TestSmokeTests(TestSuite):
> > """DPDK and infrastructure smoke test suite.
> >
> > The test cases validate the most basic DPDK functionality needed for all other test suites.
> > --
> > 2.34.1
> >
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 6/7] dts: refactor logging configuration
2024-02-14 7:49 ` Juraj Linkeš
@ 2024-02-14 16:51 ` Jeremy Spewock
0 siblings, 0 replies; 44+ messages in thread
From: Jeremy Spewock @ 2024-02-14 16:51 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev
On Wed, Feb 14, 2024 at 2:49 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> Hi Jeremy,
>
> Just a reminder, please strip the parts you're not commenting on.
>
<snip>
> > > + def __init__(self, *args, **kwargs):
> > > + """Extend the constructor with extra file handlers."""
> > > + self._extra_file_handlers = []
> > > + super().__init__(*args, **kwargs)
> > >
> > > - One handler logs to the console, the other one to a file, with either a regular or verbose
> > > - format.
> > > + def makeRecord(self, *args, **kwargs):
> >
> > Is the return type annotation here skipped because of the `:meta private:`?
> >
>
> Basically. We're modifying a method defined elsewhere, but there's no
> harm in adding the return type.
>
<snip>
> > > + logging.setLoggerClass(DTSLogger)
> > > + if name:
> > > + name = f"{dts_root_logger_name}.{name}"
> > > + else:
> > > + name = dts_root_logger_name
> > > + logger = logging.getLogger(name)
> > > + logging.setLoggerClass(logging.Logger)
> >
> > What's the benefit of setting the logger class back to logging.Logger
> > here? Is the idea basically that if someone wanted to use the logging
> > module we shouldn't implicitly make them use our DTSLogger?
> >
>
> Yes. We should actually set it to whatever was there before (it may
> not be logging.Logger), so I'll change that.+
>
>
<snip>
> > > @@ -137,6 +141,7 @@ def run(self):
> > >
> > > # for all Execution sections
> > > for execution in self._configuration.executions:
> > > + self._logger.set_stage(DtsStage.execution)
> >
> > This ends up getting set twice in short succession which of course
> > doesn't functionally cause a problem, but I don't exactly see the
> > point of setting it twice.
>
> I'm not sure what you're referring to. The stage is only set once at
> the start of each execution (and more generally, at the start of each
> stage). It's also set in each finally block to properly mark the
> cleanup of that stage. There are two finally blocks in the execution
> stage where that could happen, but otherwise it should be set exactly
> once.
You're right, I misread where it was being set the second time in the
code when I was reading this and thought there was a way where it
could get set twice but it cannot. Apologies, it makes sense that you
weren't sure what I was referring to because what I was referring to
doesn't exist.
>
> > We could either set it here or set it in
> > the _run_execution, but i think it makes more sense to set it in the
> > _run_execution method as that is the method where we are doing the
> > setup, here we are only initializing the nodes which is still in a
> > sense "pre execution."
>
> Init nodes was pre-execution before the patch. I've moved it to
> execution because of two reasons:
> 1. The nodes are actually defined per-execution. It just made more
> sense to move them to execution. Also, if an error happens while
> connecting to a node, we don't want to abort the whole DTS run, just
> the execution, as the next execution could be connecting to a
> different node.
> 2. We're not just doing node init here, we're also discovering which
> test cases to run. This is essential to do as soon as possible (before
> anything else happens in the execution, such as connecting the nodes)
> so that we can mark blocked test cases in case of an error. Test case
> discovery is definitely part of each execution and putting node init
> after that was a natural consequence. There's an argument for
> discovering test cases of all executions before actually running any
> of the executions. It's certainly doable, but the code doesn't look
> that good (when not modifying the original execution config (which we
> shouldn't do) - I've tried) and there's actually not much of a point
> to do it this way, the result is almost the same. Where it makes a
> difference is when there's an error in test suite configuration and
> later executions - the user would have to wait for the previous
> execution to fully run to discover the error).
>
These are very good points. I was still thinking of these
initializations as part of the pre-execution but I agree with you that
the things we are doing are actually part of the execution. Thank you
for elaborating!
> >
> >
<snip>
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 0/7] test case blocking and logging
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
` (6 preceding siblings ...)
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
@ 2024-02-23 7:54 ` Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 1/7] dts: convert dts.py methods to class Juraj Linkeš
` (7 more replies)
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
8 siblings, 8 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
We currently don't record test case results that couldn't be executed
because of a previous failure, such as when a test suite setup failed,
resulting in no executed test cases.
In order to record the test cases that couldn't be executed, we must
know the lists of test suites and test cases ahead of the actual test
suite execution, as an error could occur before we even start executing
test suites.
In addition, the patch series contains two refactors and one
improvement.
The first refactor is closely related. The dts.py was renamed to
runner.py and given a clear purpose - running the test suites and all
other orchestration needed to run test suites. The logic for this was
not all in the original dts.py module and it was brought there. The
runner is also responsible for recording results, which is the blocked
test cases are recorded.
The other refactor, logging, is related to the first refactor. The
logging module was simplified while extending capabilities. Each test
suite logs into its own log file in addition to the main log file which
the runner must handle (as it knows when we start executing particular
test suites). The runner also handles the switching between execution
stages for the purposes of logging.
The one aforementioned improvement is in unifying how we specify test
cases in the conf.yaml file and in the environment variable/command line
argument.
v2:
Rebase and update of the whole patch. There are changes in all parts of
the code, mainly improving the design and logic.
Also added the last patch which improves test suite specification on the
cmdline.
v3:
Improved variables in _get_test_suite_class along with docstring.
Fixed smoke test suite not being added into the list of test suites to
be executed.
Juraj Linkeš (7):
dts: convert dts.py methods to class
dts: move test suite execution logic to DTSRunner
dts: filter test suites in executions
dts: reorganize test result
dts: block all test cases when earlier setup fails
dts: refactor logging configuration
dts: improve test suite and case filtering
doc/guides/tools/dts.rst | 14 +-
dts/framework/config/__init__.py | 36 +-
dts/framework/config/conf_yaml_schema.json | 2 +-
dts/framework/dts.py | 338 ---------
dts/framework/logger.py | 236 +++---
dts/framework/remote_session/__init__.py | 6 +-
.../interactive_remote_session.py | 6 +-
.../remote_session/interactive_shell.py | 6 +-
.../remote_session/remote_session.py | 8 +-
dts/framework/runner.py | 706 ++++++++++++++++++
dts/framework/settings.py | 188 +++--
dts/framework/test_result.py | 565 ++++++++------
dts/framework/test_suite.py | 239 +-----
dts/framework/testbed_model/node.py | 11 +-
dts/framework/testbed_model/os_session.py | 7 +-
.../traffic_generator/traffic_generator.py | 6 +-
dts/main.py | 9 +-
dts/pyproject.toml | 3 +
dts/tests/TestSuite_smoke_tests.py | 2 +-
19 files changed, 1360 insertions(+), 1028 deletions(-)
delete mode 100644 dts/framework/dts.py
create mode 100644 dts/framework/runner.py
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 1/7] dts: convert dts.py methods to class
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
@ 2024-02-23 7:54 ` Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš
` (6 subsequent siblings)
7 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
The dts.py module deviates from the rest of the code without a clear
reason. Converting it into a class and using better naming will improve
organization and code readability.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/dts.py | 338 ----------------------------------------
dts/framework/runner.py | 333 +++++++++++++++++++++++++++++++++++++++
dts/main.py | 6 +-
3 files changed, 337 insertions(+), 340 deletions(-)
delete mode 100644 dts/framework/dts.py
create mode 100644 dts/framework/runner.py
diff --git a/dts/framework/dts.py b/dts/framework/dts.py
deleted file mode 100644
index e16d4578a0..0000000000
--- a/dts/framework/dts.py
+++ /dev/null
@@ -1,338 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2019 Intel Corporation
-# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
-# Copyright(c) 2022-2023 University of New Hampshire
-
-r"""Test suite runner module.
-
-A DTS run is split into stages:
-
- #. Execution stage,
- #. Build target stage,
- #. Test suite stage,
- #. Test case stage.
-
-The module is responsible for running tests on testbeds defined in the test run configuration.
-Each setup or teardown of each stage is recorded in a :class:`~.test_result.DTSResult` or
-one of its subclasses. The test case results are also recorded.
-
-If an error occurs, the current stage is aborted, the error is recorded and the run continues in
-the next iteration of the same stage. The return code is the highest `severity` of all
-:class:`~.exception.DTSError`\s.
-
-Example:
- An error occurs in a build target setup. The current build target is aborted and the run
- continues with the next build target. If the errored build target was the last one in the given
- execution, the next execution begins.
-
-Attributes:
- dts_logger: The logger instance used in this module.
- result: The top level result used in the module.
-"""
-
-import sys
-
-from .config import (
- BuildTargetConfiguration,
- ExecutionConfiguration,
- TestSuiteConfig,
- load_config,
-)
-from .exception import BlockingTestSuiteError
-from .logger import DTSLOG, getLogger
-from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
-from .test_suite import get_test_suites
-from .testbed_model import SutNode, TGNode
-
-# dummy defaults to satisfy linters
-dts_logger: DTSLOG = None # type: ignore[assignment]
-result: DTSResult = DTSResult(dts_logger)
-
-
-def run_all() -> None:
- """Run all build targets in all executions from the test run configuration.
-
- Before running test suites, executions and build targets are first set up.
- The executions and build targets defined in the test run configuration are iterated over.
- The executions define which tests to run and where to run them and build targets define
- the DPDK build setup.
-
- The tests suites are set up for each execution/build target tuple and each scheduled
- test case within the test suite is set up, executed and torn down. After all test cases
- have been executed, the test suite is torn down and the next build target will be tested.
-
- All the nested steps look like this:
-
- #. Execution setup
-
- #. Build target setup
-
- #. Test suite setup
-
- #. Test case setup
- #. Test case logic
- #. Test case teardown
-
- #. Test suite teardown
-
- #. Build target teardown
-
- #. Execution teardown
-
- The test cases are filtered according to the specification in the test run configuration and
- the :option:`--test-cases` command line argument or
- the :envvar:`DTS_TESTCASES` environment variable.
- """
- global dts_logger
- global result
-
- # create a regular DTS logger and create a new result with it
- dts_logger = getLogger("DTSRunner")
- result = DTSResult(dts_logger)
-
- # check the python version of the server that run dts
- _check_dts_python_version()
-
- sut_nodes: dict[str, SutNode] = {}
- tg_nodes: dict[str, TGNode] = {}
- try:
- # for all Execution sections
- for execution in load_config().executions:
- sut_node = sut_nodes.get(execution.system_under_test_node.name)
- tg_node = tg_nodes.get(execution.traffic_generator_node.name)
-
- try:
- if not sut_node:
- sut_node = SutNode(execution.system_under_test_node)
- sut_nodes[sut_node.name] = sut_node
- if not tg_node:
- tg_node = TGNode(execution.traffic_generator_node)
- tg_nodes[tg_node.name] = tg_node
- result.update_setup(Result.PASS)
- except Exception as e:
- failed_node = execution.system_under_test_node.name
- if sut_node:
- failed_node = execution.traffic_generator_node.name
- dts_logger.exception(f"Creation of node {failed_node} failed.")
- result.update_setup(Result.FAIL, e)
-
- else:
- _run_execution(sut_node, tg_node, execution, result)
-
- except Exception as e:
- dts_logger.exception("An unexpected error has occurred.")
- result.add_error(e)
- raise
-
- finally:
- try:
- for node in (sut_nodes | tg_nodes).values():
- node.close()
- result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Final cleanup of nodes failed.")
- result.update_teardown(Result.ERROR, e)
-
- # we need to put the sys.exit call outside the finally clause to make sure
- # that unexpected exceptions will propagate
- # in that case, the error that should be reported is the uncaught exception as
- # that is a severe error originating from the framework
- # at that point, we'll only have partial results which could be impacted by the
- # error causing the uncaught exception, making them uninterpretable
- _exit_dts()
-
-
-def _check_dts_python_version() -> None:
- """Check the required Python version - v3.10."""
-
- def RED(text: str) -> str:
- return f"\u001B[31;1m{str(text)}\u001B[0m"
-
- if sys.version_info.major < 3 or (sys.version_info.major == 3 and sys.version_info.minor < 10):
- print(
- RED(
- (
- "WARNING: DTS execution node's python version is lower than"
- "python 3.10, is deprecated and will not work in future releases."
- )
- ),
- file=sys.stderr,
- )
- print(RED("Please use Python >= 3.10 instead"), file=sys.stderr)
-
-
-def _run_execution(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- result: DTSResult,
-) -> None:
- """Run the given execution.
-
- This involves running the execution setup as well as running all build targets
- in the given execution. After that, execution teardown is run.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- execution: An execution's test run configuration.
- result: The top level result object.
- """
- dts_logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
- execution_result = result.add_execution(sut_node.config)
- execution_result.add_sut_info(sut_node.node_info)
-
- try:
- sut_node.set_up_execution(execution)
- execution_result.update_setup(Result.PASS)
- except Exception as e:
- dts_logger.exception("Execution setup failed.")
- execution_result.update_setup(Result.FAIL, e)
-
- else:
- for build_target in execution.build_targets:
- _run_build_target(sut_node, tg_node, build_target, execution, execution_result)
-
- finally:
- try:
- sut_node.tear_down_execution()
- execution_result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Execution teardown failed.")
- execution_result.update_teardown(Result.FAIL, e)
-
-
-def _run_build_target(
- sut_node: SutNode,
- tg_node: TGNode,
- build_target: BuildTargetConfiguration,
- execution: ExecutionConfiguration,
- execution_result: ExecutionResult,
-) -> None:
- """Run the given build target.
-
- This involves running the build target setup as well as running all test suites
- in the given execution the build target is defined in.
- After that, build target teardown is run.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- build_target: A build target's test run configuration.
- execution: The build target's execution's test run configuration.
- execution_result: The execution level result object associated with the execution.
- """
- dts_logger.info(f"Running build target '{build_target.name}'.")
- build_target_result = execution_result.add_build_target(build_target)
-
- try:
- sut_node.set_up_build_target(build_target)
- result.dpdk_version = sut_node.dpdk_version
- build_target_result.add_build_target_info(sut_node.get_build_target_info())
- build_target_result.update_setup(Result.PASS)
- except Exception as e:
- dts_logger.exception("Build target setup failed.")
- build_target_result.update_setup(Result.FAIL, e)
-
- else:
- _run_all_suites(sut_node, tg_node, execution, build_target_result)
-
- finally:
- try:
- sut_node.tear_down_build_target()
- build_target_result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Build target teardown failed.")
- build_target_result.update_teardown(Result.FAIL, e)
-
-
-def _run_all_suites(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
-) -> None:
- """Run the execution's (possibly a subset) test suites using the current build target.
-
- The function assumes the build target we're testing has already been built on the SUT node.
- The current build target thus corresponds to the current DPDK build present on the SUT node.
-
- If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
- in the current build target won't be executed.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- execution: The execution's test run configuration associated with the current build target.
- build_target_result: The build target level result object associated
- with the current build target.
- """
- end_build_target = False
- if not execution.skip_smoke_tests:
- execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
- for test_suite_config in execution.test_suites:
- try:
- _run_single_suite(sut_node, tg_node, execution, build_target_result, test_suite_config)
- except BlockingTestSuiteError as e:
- dts_logger.exception(
- f"An error occurred within {test_suite_config.test_suite}. Skipping build target."
- )
- result.add_error(e)
- end_build_target = True
- # if a blocking test failed and we need to bail out of suite executions
- if end_build_target:
- break
-
-
-def _run_single_suite(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
- test_suite_config: TestSuiteConfig,
-) -> None:
- """Run all test suite in a single test suite module.
-
- The function assumes the build target we're testing has already been built on the SUT node.
- The current build target thus corresponds to the current DPDK build present on the SUT node.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- execution: The execution's test run configuration associated with the current build target.
- build_target_result: The build target level result object associated
- with the current build target.
- test_suite_config: Test suite test run configuration specifying the test suite module
- and possibly a subset of test cases of test suites in that module.
-
- Raises:
- BlockingTestSuiteError: If a blocking test suite fails.
- """
- try:
- full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
- test_suite_classes = get_test_suites(full_suite_path)
- suites_str = ", ".join((x.__name__ for x in test_suite_classes))
- dts_logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
- except Exception as e:
- dts_logger.exception("An error occurred when searching for test suites.")
- result.update_setup(Result.ERROR, e)
-
- else:
- for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(
- sut_node,
- tg_node,
- test_suite_config.test_cases,
- execution.func,
- build_target_result,
- )
- test_suite.run()
-
-
-def _exit_dts() -> None:
- """Process all errors and exit with the proper exit code."""
- result.process()
-
- if dts_logger:
- dts_logger.info("DTS execution has ended.")
- sys.exit(result.get_return_code())
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
new file mode 100644
index 0000000000..acc1c4d6db
--- /dev/null
+++ b/dts/framework/runner.py
@@ -0,0 +1,333 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2019 Intel Corporation
+# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
+# Copyright(c) 2022-2023 University of New Hampshire
+
+"""Test suite runner module.
+
+The module is responsible for running DTS in a series of stages:
+
+ #. Execution stage,
+ #. Build target stage,
+ #. Test suite stage,
+ #. Test case stage.
+
+The execution and build target stages set up the environment before running test suites.
+The test suite stage sets up steps common to all test cases
+and the test case stage runs test cases individually.
+"""
+
+import logging
+import sys
+
+from .config import (
+ BuildTargetConfiguration,
+ ExecutionConfiguration,
+ TestSuiteConfig,
+ load_config,
+)
+from .exception import BlockingTestSuiteError
+from .logger import DTSLOG, getLogger
+from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
+from .test_suite import get_test_suites
+from .testbed_model import SutNode, TGNode
+
+
+class DTSRunner:
+ r"""Test suite runner class.
+
+ The class is responsible for running tests on testbeds defined in the test run configuration.
+ Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult`
+ or one of its subclasses. The test case results are also recorded.
+
+ If an error occurs, the current stage is aborted, the error is recorded and the run continues in
+ the next iteration of the same stage. The return code is the highest `severity` of all
+ :class:`~.framework.exception.DTSError`\s.
+
+ Example:
+ An error occurs in a build target setup. The current build target is aborted and the run
+ continues with the next build target. If the errored build target was the last one in the
+ given execution, the next execution begins.
+ """
+
+ _logger: DTSLOG
+ _result: DTSResult
+
+ def __init__(self):
+ """Initialize the instance with logger and result."""
+ self._logger = getLogger("DTSRunner")
+ self._result = DTSResult(self._logger)
+
+ def run(self):
+ """Run all build targets in all executions from the test run configuration.
+
+ Before running test suites, executions and build targets are first set up.
+ The executions and build targets defined in the test run configuration are iterated over.
+ The executions define which tests to run and where to run them and build targets define
+ the DPDK build setup.
+
+ The tests suites are set up for each execution/build target tuple and each discovered
+ test case within the test suite is set up, executed and torn down. After all test cases
+ have been executed, the test suite is torn down and the next build target will be tested.
+
+ All the nested steps look like this:
+
+ #. Execution setup
+
+ #. Build target setup
+
+ #. Test suite setup
+
+ #. Test case setup
+ #. Test case logic
+ #. Test case teardown
+
+ #. Test suite teardown
+
+ #. Build target teardown
+
+ #. Execution teardown
+
+ The test cases are filtered according to the specification in the test run configuration and
+ the :option:`--test-cases` command line argument or
+ the :envvar:`DTS_TESTCASES` environment variable.
+ """
+ sut_nodes: dict[str, SutNode] = {}
+ tg_nodes: dict[str, TGNode] = {}
+ try:
+ # check the python version of the server that runs dts
+ self._check_dts_python_version()
+
+ # for all Execution sections
+ for execution in load_config().executions:
+ sut_node = sut_nodes.get(execution.system_under_test_node.name)
+ tg_node = tg_nodes.get(execution.traffic_generator_node.name)
+
+ try:
+ if not sut_node:
+ sut_node = SutNode(execution.system_under_test_node)
+ sut_nodes[sut_node.name] = sut_node
+ if not tg_node:
+ tg_node = TGNode(execution.traffic_generator_node)
+ tg_nodes[tg_node.name] = tg_node
+ self._result.update_setup(Result.PASS)
+ except Exception as e:
+ failed_node = execution.system_under_test_node.name
+ if sut_node:
+ failed_node = execution.traffic_generator_node.name
+ self._logger.exception(f"The Creation of node {failed_node} failed.")
+ self._result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_execution(sut_node, tg_node, execution)
+
+ except Exception as e:
+ self._logger.exception("An unexpected error has occurred.")
+ self._result.add_error(e)
+ raise
+
+ finally:
+ try:
+ for node in (sut_nodes | tg_nodes).values():
+ node.close()
+ self._result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("The final cleanup of nodes failed.")
+ self._result.update_teardown(Result.ERROR, e)
+
+ # we need to put the sys.exit call outside the finally clause to make sure
+ # that unexpected exceptions will propagate
+ # in that case, the error that should be reported is the uncaught exception as
+ # that is a severe error originating from the framework
+ # at that point, we'll only have partial results which could be impacted by the
+ # error causing the uncaught exception, making them uninterpretable
+ self._exit_dts()
+
+ def _check_dts_python_version(self) -> None:
+ """Check the required Python version - v3.10."""
+ if sys.version_info.major < 3 or (
+ sys.version_info.major == 3 and sys.version_info.minor < 10
+ ):
+ self._logger.warning(
+ "DTS execution node's python version is lower than Python 3.10, "
+ "is deprecated and will not work in future releases."
+ )
+ self._logger.warning("Please use Python >= 3.10 instead.")
+
+ def _run_execution(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ ) -> None:
+ """Run the given execution.
+
+ This involves running the execution setup as well as running all build targets
+ in the given execution. After that, execution teardown is run.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: An execution's test run configuration.
+ """
+ self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
+ execution_result = self._result.add_execution(sut_node.config)
+ execution_result.add_sut_info(sut_node.node_info)
+
+ try:
+ sut_node.set_up_execution(execution)
+ execution_result.update_setup(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Execution setup failed.")
+ execution_result.update_setup(Result.FAIL, e)
+
+ else:
+ for build_target in execution.build_targets:
+ self._run_build_target(sut_node, tg_node, build_target, execution, execution_result)
+
+ finally:
+ try:
+ sut_node.tear_down_execution()
+ execution_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Execution teardown failed.")
+ execution_result.update_teardown(Result.FAIL, e)
+
+ def _run_build_target(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ build_target: BuildTargetConfiguration,
+ execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ ) -> None:
+ """Run the given build target.
+
+ This involves running the build target setup as well as running all test suites
+ of the build target's execution.
+ After that, build target teardown is run.
+
+ Args:
+ sut_node: The execution's sut node.
+ tg_node: The execution's tg node.
+ build_target: A build target's test run configuration.
+ execution: The build target's execution's test run configuration.
+ execution_result: The execution level result object associated with the execution.
+ """
+ self._logger.info(f"Running build target '{build_target.name}'.")
+ build_target_result = execution_result.add_build_target(build_target)
+
+ try:
+ sut_node.set_up_build_target(build_target)
+ self._result.dpdk_version = sut_node.dpdk_version
+ build_target_result.add_build_target_info(sut_node.get_build_target_info())
+ build_target_result.update_setup(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Build target setup failed.")
+ build_target_result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_all_suites(sut_node, tg_node, execution, build_target_result)
+
+ finally:
+ try:
+ sut_node.tear_down_build_target()
+ build_target_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Build target teardown failed.")
+ build_target_result.update_teardown(Result.FAIL, e)
+
+ def _run_all_suites(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ build_target_result: BuildTargetResult,
+ ) -> None:
+ """Run the execution's (possibly a subset of) test suites using the current build target.
+
+ The method assumes the build target we're testing has already been built on the SUT node.
+ The current build target thus corresponds to the current DPDK build present on the SUT node.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: The execution's test run configuration associated
+ with the current build target.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ """
+ end_build_target = False
+ if not execution.skip_smoke_tests:
+ execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
+ for test_suite_config in execution.test_suites:
+ try:
+ self._run_single_suite(
+ sut_node, tg_node, execution, build_target_result, test_suite_config
+ )
+ except BlockingTestSuiteError as e:
+ self._logger.exception(
+ f"An error occurred within {test_suite_config.test_suite}. "
+ "Skipping build target..."
+ )
+ self._result.add_error(e)
+ end_build_target = True
+ # if a blocking test failed and we need to bail out of suite executions
+ if end_build_target:
+ break
+
+ def _run_single_suite(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ build_target_result: BuildTargetResult,
+ test_suite_config: TestSuiteConfig,
+ ) -> None:
+ """Run all test suites in a single test suite module.
+
+ The method assumes the build target we're testing has already been built on the SUT node.
+ The current build target thus corresponds to the current DPDK build present on the SUT node.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: The execution's test run configuration associated
+ with the current build target.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ test_suite_config: Test suite test run configuration specifying the test suite module
+ and possibly a subset of test cases of test suites in that module.
+
+ Raises:
+ BlockingTestSuiteError: If a blocking test suite fails.
+ """
+ try:
+ full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
+ test_suite_classes = get_test_suites(full_suite_path)
+ suites_str = ", ".join((x.__name__ for x in test_suite_classes))
+ self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
+ except Exception as e:
+ self._logger.exception("An error occurred when searching for test suites.")
+ self._result.update_setup(Result.ERROR, e)
+
+ else:
+ for test_suite_class in test_suite_classes:
+ test_suite = test_suite_class(
+ sut_node,
+ tg_node,
+ test_suite_config.test_cases,
+ execution.func,
+ build_target_result,
+ )
+ test_suite.run()
+
+ def _exit_dts(self) -> None:
+ """Process all errors and exit with the proper exit code."""
+ self._result.process()
+
+ if self._logger:
+ self._logger.info("DTS execution has ended.")
+
+ logging.shutdown()
+ sys.exit(self._result.get_return_code())
diff --git a/dts/main.py b/dts/main.py
index f703615d11..1ffe8ff81f 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -21,9 +21,11 @@ def main() -> None:
be modified before the settings module is imported anywhere else in the framework.
"""
settings.SETTINGS = settings.get_settings()
- from framework import dts
- dts.run_all()
+ from framework.runner import DTSRunner
+
+ dts = DTSRunner()
+ dts.run()
# Main program begins here
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 2/7] dts: move test suite execution logic to DTSRunner
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 1/7] dts: convert dts.py methods to class Juraj Linkeš
@ 2024-02-23 7:54 ` Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 3/7] dts: filter test suites in executions Juraj Linkeš
` (5 subsequent siblings)
7 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
Move the code responsible for running the test suite from the
TestSuite class to the DTSRunner class. This restructuring decision
was made to consolidate and unify the related logic into a single unit.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 175 ++++++++++++++++++++++++++++++++----
dts/framework/test_suite.py | 152 ++-----------------------------
2 files changed, 169 insertions(+), 158 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index acc1c4d6db..933685d638 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -19,6 +19,7 @@
import logging
import sys
+from types import MethodType
from .config import (
BuildTargetConfiguration,
@@ -26,10 +27,18 @@
TestSuiteConfig,
load_config,
)
-from .exception import BlockingTestSuiteError
+from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
from .logger import DTSLOG, getLogger
-from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
-from .test_suite import get_test_suites
+from .settings import SETTINGS
+from .test_result import (
+ BuildTargetResult,
+ DTSResult,
+ ExecutionResult,
+ Result,
+ TestCaseResult,
+ TestSuiteResult,
+)
+from .test_suite import TestSuite, get_test_suites
from .testbed_model import SutNode, TGNode
@@ -227,7 +236,7 @@ def _run_build_target(
build_target_result.update_setup(Result.FAIL, e)
else:
- self._run_all_suites(sut_node, tg_node, execution, build_target_result)
+ self._run_test_suites(sut_node, tg_node, execution, build_target_result)
finally:
try:
@@ -237,7 +246,7 @@ def _run_build_target(
self._logger.exception("Build target teardown failed.")
build_target_result.update_teardown(Result.FAIL, e)
- def _run_all_suites(
+ def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
@@ -249,6 +258,9 @@ def _run_all_suites(
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
+ in the current build target won't be executed.
+
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
@@ -262,7 +274,7 @@ def _run_all_suites(
execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
for test_suite_config in execution.test_suites:
try:
- self._run_single_suite(
+ self._run_test_suite_module(
sut_node, tg_node, execution, build_target_result, test_suite_config
)
except BlockingTestSuiteError as e:
@@ -276,7 +288,7 @@ def _run_all_suites(
if end_build_target:
break
- def _run_single_suite(
+ def _run_test_suite_module(
self,
sut_node: SutNode,
tg_node: TGNode,
@@ -284,11 +296,18 @@ def _run_single_suite(
build_target_result: BuildTargetResult,
test_suite_config: TestSuiteConfig,
) -> None:
- """Run all test suites in a single test suite module.
+ """Set up, execute and tear down all test suites in a single test suite module.
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ Test suite execution consists of running the discovered test cases.
+ A test case run consists of setup, execution and teardown of said test case.
+
+ Record the setup and the teardown and handle failures.
+
+ The test cases to execute are discovered when creating the :class:`TestSuite` object.
+
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
@@ -313,14 +332,140 @@ def _run_single_suite(
else:
for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(
- sut_node,
- tg_node,
- test_suite_config.test_cases,
- execution.func,
- build_target_result,
+ test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases)
+
+ test_suite_name = test_suite.__class__.__name__
+ test_suite_result = build_target_result.add_test_suite(test_suite_name)
+ try:
+ self._logger.info(f"Starting test suite setup: {test_suite_name}")
+ test_suite.set_up_suite()
+ test_suite_result.update_setup(Result.PASS)
+ self._logger.info(f"Test suite setup successful: {test_suite_name}")
+ except Exception as e:
+ self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
+ test_suite_result.update_setup(Result.ERROR, e)
+
+ else:
+ self._execute_test_suite(execution.func, test_suite, test_suite_result)
+
+ finally:
+ try:
+ test_suite.tear_down_suite()
+ sut_node.kill_cleanup_dpdk_apps()
+ test_suite_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
+ self._logger.warning(
+ f"Test suite '{test_suite_name}' teardown failed, "
+ f"the next test suite may be affected."
+ )
+ test_suite_result.update_setup(Result.ERROR, e)
+ if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
+ raise BlockingTestSuiteError(test_suite_name)
+
+ def _execute_test_suite(
+ self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
+ ) -> None:
+ """Execute all discovered test cases in `test_suite`.
+
+ If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment
+ variable is set, in case of a test case failure, the test case will be executed again
+ until it passes or it fails that many times in addition of the first failure.
+
+ Args:
+ func: Whether to execute functional test cases.
+ test_suite: The test suite object.
+ test_suite_result: The test suite level result object associated
+ with the current test suite.
+ """
+ if func:
+ for test_case_method in test_suite._get_functional_test_cases():
+ test_case_name = test_case_method.__name__
+ test_case_result = test_suite_result.add_test_case(test_case_name)
+ all_attempts = SETTINGS.re_run + 1
+ attempt_nr = 1
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+
+ def _run_test_case(
+ self,
+ test_suite: TestSuite,
+ test_case_method: MethodType,
+ test_case_result: TestCaseResult,
+ ) -> None:
+ """Setup, execute and teardown a test case in `test_suite`.
+
+ Record the result of the setup and the teardown and handle failures.
+
+ Args:
+ test_suite: The test suite object.
+ test_case_method: The test case method.
+ test_case_result: The test case level result object associated
+ with the current test case.
+ """
+ test_case_name = test_case_method.__name__
+
+ try:
+ # run set_up function for each case
+ test_suite.set_up_test_case()
+ test_case_result.update_setup(Result.PASS)
+ except SSHTimeoutError as e:
+ self._logger.exception(f"Test case setup FAILED: {test_case_name}")
+ test_case_result.update_setup(Result.FAIL, e)
+ except Exception as e:
+ self._logger.exception(f"Test case setup ERROR: {test_case_name}")
+ test_case_result.update_setup(Result.ERROR, e)
+
+ else:
+ # run test case if setup was successful
+ self._execute_test_case(test_case_method, test_case_result)
+
+ finally:
+ try:
+ test_suite.tear_down_test_case()
+ test_case_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test case teardown ERROR: {test_case_name}")
+ self._logger.warning(
+ f"Test case '{test_case_name}' teardown failed, "
+ f"the next test case may be affected."
)
- test_suite.run()
+ test_case_result.update_teardown(Result.ERROR, e)
+ test_case_result.update(Result.ERROR)
+
+ def _execute_test_case(
+ self, test_case_method: MethodType, test_case_result: TestCaseResult
+ ) -> None:
+ """Execute one test case, record the result and handle failures.
+
+ Args:
+ test_case_method: The test case method.
+ test_case_result: The test case level result object associated
+ with the current test case.
+ """
+ test_case_name = test_case_method.__name__
+ try:
+ self._logger.info(f"Starting test case execution: {test_case_name}")
+ test_case_method()
+ test_case_result.update(Result.PASS)
+ self._logger.info(f"Test case execution PASSED: {test_case_name}")
+
+ except TestCaseVerifyError as e:
+ self._logger.exception(f"Test case execution FAILED: {test_case_name}")
+ test_case_result.update(Result.FAIL, e)
+ except Exception as e:
+ self._logger.exception(f"Test case execution ERROR: {test_case_name}")
+ test_case_result.update(Result.ERROR, e)
+ except KeyboardInterrupt:
+ self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}")
+ test_case_result.update(Result.SKIP)
+ raise KeyboardInterrupt("Stop DTS")
def _exit_dts(self) -> None:
"""Process all errors and exit with the proper exit code."""
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index dfb391ffbd..b02fd36147 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -8,7 +8,6 @@
must be extended by subclasses which add test cases. The :class:`TestSuite` contains the basics
needed by subclasses:
- * Test suite and test case execution flow,
* Testbed (SUT, TG) configuration,
* Packet sending and verification,
* Test case verification.
@@ -28,27 +27,22 @@
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
-from .exception import (
- BlockingTestSuiteError,
- ConfigurationError,
- SSHTimeoutError,
- TestCaseVerifyError,
-)
+from .exception import ConfigurationError, TestCaseVerifyError
from .logger import DTSLOG, getLogger
from .settings import SETTINGS
-from .test_result import BuildTargetResult, Result, TestCaseResult, TestSuiteResult
from .testbed_model import Port, PortLink, SutNode, TGNode
from .utils import get_packet_summaries
class TestSuite(object):
- """The base class with methods for handling the basic flow of a test suite.
+ """The base class with building blocks needed by most test cases.
* Test case filtering and collection,
- * Test suite setup/cleanup,
- * Test setup/cleanup,
- * Test case execution,
- * Error handling and results storage.
+ * Test suite setup/cleanup methods to override,
+ * Test case setup/cleanup methods to override,
+ * Test case verification,
+ * Testbed configuration,
+ * Traffic sending and verification.
Test cases are implemented by subclasses. Test cases are all methods starting with ``test_``,
further divided into performance test cases (starting with ``test_perf_``)
@@ -60,10 +54,6 @@ class TestSuite(object):
The union of both lists will be used. Any unknown test cases from the latter lists
will be silently ignored.
- If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable
- is set, in case of a test case failure, the test case will be executed again until it passes
- or it fails that many times in addition of the first failure.
-
The methods named ``[set_up|tear_down]_[suite|test_case]`` should be overridden in subclasses
if the appropriate test suite/test case fixtures are needed.
@@ -82,8 +72,6 @@ class TestSuite(object):
is_blocking: ClassVar[bool] = False
_logger: DTSLOG
_test_cases_to_run: list[str]
- _func: bool
- _result: TestSuiteResult
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -99,30 +87,23 @@ def __init__(
sut_node: SutNode,
tg_node: TGNode,
test_cases: list[str],
- func: bool,
- build_target_result: BuildTargetResult,
):
"""Initialize the test suite testbed information and basic configuration.
- Process what test cases to run, create the associated
- :class:`~.test_result.TestSuiteResult`, find links between ports
- and set up default IP addresses to be used when configuring them.
+ Process what test cases to run, find links between ports and set up
+ default IP addresses to be used when configuring them.
Args:
sut_node: The SUT node where the test suite will run.
tg_node: The TG node where the test suite will run.
test_cases: The list of test cases to execute.
If empty, all test cases will be executed.
- func: Whether to run functional tests.
- build_target_result: The build target result this test suite is run in.
"""
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = getLogger(self.__class__.__name__)
self._test_cases_to_run = test_cases
self._test_cases_to_run.extend(SETTINGS.test_cases)
- self._func = func
- self._result = build_target_result.add_test_suite(self.__class__.__name__)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
@@ -384,62 +365,6 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
return False
return True
- def run(self) -> None:
- """Set up, execute and tear down the whole suite.
-
- Test suite execution consists of running all test cases scheduled to be executed.
- A test case run consists of setup, execution and teardown of said test case.
-
- Record the setup and the teardown and handle failures.
-
- The list of scheduled test cases is constructed when creating the :class:`TestSuite` object.
- """
- test_suite_name = self.__class__.__name__
-
- try:
- self._logger.info(f"Starting test suite setup: {test_suite_name}")
- self.set_up_suite()
- self._result.update_setup(Result.PASS)
- self._logger.info(f"Test suite setup successful: {test_suite_name}")
- except Exception as e:
- self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
- self._result.update_setup(Result.ERROR, e)
-
- else:
- self._execute_test_suite()
-
- finally:
- try:
- self.tear_down_suite()
- self.sut_node.kill_cleanup_dpdk_apps()
- self._result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
- self._logger.warning(
- f"Test suite '{test_suite_name}' teardown failed, "
- f"the next test suite may be affected."
- )
- self._result.update_setup(Result.ERROR, e)
- if len(self._result.get_errors()) > 0 and self.is_blocking:
- raise BlockingTestSuiteError(test_suite_name)
-
- def _execute_test_suite(self) -> None:
- """Execute all test cases scheduled to be executed in this suite."""
- if self._func:
- for test_case_method in self._get_functional_test_cases():
- test_case_name = test_case_method.__name__
- test_case_result = self._result.add_test_case(test_case_name)
- all_attempts = SETTINGS.re_run + 1
- attempt_nr = 1
- self._run_test_case(test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
- self._run_test_case(test_case_method, test_case_result)
-
def _get_functional_test_cases(self) -> list[MethodType]:
"""Get all functional test cases defined in this TestSuite.
@@ -471,65 +396,6 @@ def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool
return match
- def _run_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
- ) -> None:
- """Setup, execute and teardown a test case in this suite.
-
- Record the result of the setup and the teardown and handle failures.
- """
- test_case_name = test_case_method.__name__
-
- try:
- # run set_up function for each case
- self.set_up_test_case()
- test_case_result.update_setup(Result.PASS)
- except SSHTimeoutError as e:
- self._logger.exception(f"Test case setup FAILED: {test_case_name}")
- test_case_result.update_setup(Result.FAIL, e)
- except Exception as e:
- self._logger.exception(f"Test case setup ERROR: {test_case_name}")
- test_case_result.update_setup(Result.ERROR, e)
-
- else:
- # run test case if setup was successful
- self._execute_test_case(test_case_method, test_case_result)
-
- finally:
- try:
- self.tear_down_test_case()
- test_case_result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test case teardown ERROR: {test_case_name}")
- self._logger.warning(
- f"Test case '{test_case_name}' teardown failed, "
- f"the next test case may be affected."
- )
- test_case_result.update_teardown(Result.ERROR, e)
- test_case_result.update(Result.ERROR)
-
- def _execute_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
- ) -> None:
- """Execute one test case, record the result and handle failures."""
- test_case_name = test_case_method.__name__
- try:
- self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method()
- test_case_result.update(Result.PASS)
- self._logger.info(f"Test case execution PASSED: {test_case_name}")
-
- except TestCaseVerifyError as e:
- self._logger.exception(f"Test case execution FAILED: {test_case_name}")
- test_case_result.update(Result.FAIL, e)
- except Exception as e:
- self._logger.exception(f"Test case execution ERROR: {test_case_name}")
- test_case_result.update(Result.ERROR, e)
- except KeyboardInterrupt:
- self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}")
- test_case_result.update(Result.SKIP)
- raise KeyboardInterrupt("Stop DTS")
-
def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
r"""Find all :class:`TestSuite`\s in a Python module.
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 3/7] dts: filter test suites in executions
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 1/7] dts: convert dts.py methods to class Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš
@ 2024-02-23 7:54 ` Juraj Linkeš
2024-02-27 21:21 ` Jeremy Spewock
2024-02-23 7:54 ` [PATCH v3 4/7] dts: reorganize test result Juraj Linkeš
` (4 subsequent siblings)
7 siblings, 1 reply; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
We're currently filtering which test cases to run after some setup
steps, such as DPDK build, have already been taken. This prohibits us to
mark the test suites and cases that were supposed to be run as blocked
when an earlier setup fails, as that information is not available at
that time.
To remedy this, move the filtering to the beginning of each execution.
This is the first action taken in each execution and if we can't filter
the test cases, such as due to invalid inputs, we abort the whole
execution. No test suites nor cases will be marked as blocked as we
don't know which were supposed to be run.
On top of that, the filtering takes place in the TestSuite class, which
should only concern itself with test suite and test case logic, not the
processing behind the scenes. The logic has been moved to DTSRunner
which should do all the processing needed to run test suites.
The filtering itself introduces a few changes/assumptions which are more
sensible than before:
1. Assumption: There is just one TestSuite child class in each test
suite module. This was an implicit assumption before as we couldn't
specify the TestSuite classes in the test run configuration, just the
modules. The name of the TestSuite child class starts with "Test" and
then corresponds to the name of the module with CamelCase naming.
2. Unknown test cases specified both in the test run configuration and
the environment variable/command line argument are no longer silently
ignored. This is a quality of life improvement for users, as they
could easily be not aware of the silent ignoration.
Also, a change in the code results in pycodestyle warning and error:
[E] E203 whitespace before ':'
[W] W503 line break before binary operator
These two are not PEP8 compliant, so they're disabled.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/config/__init__.py | 24 +-
dts/framework/config/conf_yaml_schema.json | 2 +-
dts/framework/runner.py | 432 +++++++++++++++------
dts/framework/settings.py | 3 +-
dts/framework/test_result.py | 34 ++
dts/framework/test_suite.py | 85 +---
dts/pyproject.toml | 3 +
dts/tests/TestSuite_smoke_tests.py | 2 +-
8 files changed, 388 insertions(+), 197 deletions(-)
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 62eded7f04..c6a93b3b89 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -36,7 +36,7 @@
import json
import os.path
import pathlib
-from dataclasses import dataclass
+from dataclasses import dataclass, fields
from enum import auto, unique
from typing import Union
@@ -506,6 +506,28 @@ def from_dict(
vdevs=vdevs,
)
+ def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration":
+ """Create a shallow copy with any of the fields modified.
+
+ The only new data are those passed to this method.
+ The rest are copied from the object's fields calling the method.
+
+ Args:
+ **kwargs: The names and types of keyword arguments are defined
+ by the fields of the :class:`ExecutionConfiguration` class.
+
+ Returns:
+ The copied and modified execution configuration.
+ """
+ new_config = {}
+ for field in fields(self):
+ if field.name in kwargs:
+ new_config[field.name] = kwargs[field.name]
+ else:
+ new_config[field.name] = getattr(self, field.name)
+
+ return ExecutionConfiguration(**new_config)
+
@dataclass(slots=True, frozen=True)
class Configuration:
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 84e45fe3c2..051b079fe4 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -197,7 +197,7 @@
},
"cases": {
"type": "array",
- "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.",
+ "description": "If specified, only this subset of test suite's test cases will be run.",
"items": {
"type": "string"
},
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 933685d638..e8030365ac 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -17,17 +17,27 @@
and the test case stage runs test cases individually.
"""
+import importlib
+import inspect
import logging
+import re
import sys
from types import MethodType
+from typing import Iterable
from .config import (
BuildTargetConfiguration,
+ Configuration,
ExecutionConfiguration,
TestSuiteConfig,
load_config,
)
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .exception import (
+ BlockingTestSuiteError,
+ ConfigurationError,
+ SSHTimeoutError,
+ TestCaseVerifyError,
+)
from .logger import DTSLOG, getLogger
from .settings import SETTINGS
from .test_result import (
@@ -37,8 +47,9 @@
Result,
TestCaseResult,
TestSuiteResult,
+ TestSuiteWithCases,
)
-from .test_suite import TestSuite, get_test_suites
+from .test_suite import TestSuite
from .testbed_model import SutNode, TGNode
@@ -59,13 +70,23 @@ class DTSRunner:
given execution, the next execution begins.
"""
+ _configuration: Configuration
_logger: DTSLOG
_result: DTSResult
+ _test_suite_class_prefix: str
+ _test_suite_module_prefix: str
+ _func_test_case_regex: str
+ _perf_test_case_regex: str
def __init__(self):
- """Initialize the instance with logger and result."""
+ """Initialize the instance with configuration, logger, result and string constants."""
+ self._configuration = load_config()
self._logger = getLogger("DTSRunner")
self._result = DTSResult(self._logger)
+ self._test_suite_class_prefix = "Test"
+ self._test_suite_module_prefix = "tests.TestSuite_"
+ self._func_test_case_regex = r"test_(?!perf_)"
+ self._perf_test_case_regex = r"test_perf_"
def run(self):
"""Run all build targets in all executions from the test run configuration.
@@ -106,29 +127,32 @@ def run(self):
try:
# check the python version of the server that runs dts
self._check_dts_python_version()
+ self._result.update_setup(Result.PASS)
# for all Execution sections
- for execution in load_config().executions:
- sut_node = sut_nodes.get(execution.system_under_test_node.name)
- tg_node = tg_nodes.get(execution.traffic_generator_node.name)
-
+ for execution in self._configuration.executions:
+ self._logger.info(
+ f"Running execution with SUT '{execution.system_under_test_node.name}'."
+ )
+ execution_result = self._result.add_execution(execution.system_under_test_node)
+ # we don't want to modify the original config, so create a copy
+ execution_test_suites = list(execution.test_suites)
+ if not execution.skip_smoke_tests:
+ execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
try:
- if not sut_node:
- sut_node = SutNode(execution.system_under_test_node)
- sut_nodes[sut_node.name] = sut_node
- if not tg_node:
- tg_node = TGNode(execution.traffic_generator_node)
- tg_nodes[tg_node.name] = tg_node
- self._result.update_setup(Result.PASS)
+ test_suites_with_cases = self._get_test_suites_with_cases(
+ execution_test_suites, execution.func, execution.perf
+ )
except Exception as e:
- failed_node = execution.system_under_test_node.name
- if sut_node:
- failed_node = execution.traffic_generator_node.name
- self._logger.exception(f"The Creation of node {failed_node} failed.")
- self._result.update_setup(Result.FAIL, e)
+ self._logger.exception(
+ f"Invalid test suite configuration found: " f"{execution_test_suites}."
+ )
+ execution_result.update_setup(Result.FAIL, e)
else:
- self._run_execution(sut_node, tg_node, execution)
+ self._connect_nodes_and_run_execution(
+ sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases
+ )
except Exception as e:
self._logger.exception("An unexpected error has occurred.")
@@ -163,11 +187,206 @@ def _check_dts_python_version(self) -> None:
)
self._logger.warning("Please use Python >= 3.10 instead.")
+ def _get_test_suites_with_cases(
+ self,
+ test_suite_configs: list[TestSuiteConfig],
+ func: bool,
+ perf: bool,
+ ) -> list[TestSuiteWithCases]:
+ """Test suites with test cases discovery.
+
+ The test suites with test cases defined in the user configuration are discovered
+ and stored for future use so that we don't import the modules twice and so that
+ the list of test suites with test cases is available for recording right away.
+
+ Args:
+ test_suite_configs: Test suite configurations.
+ func: Whether to include functional test cases in the final list.
+ perf: Whether to include performance test cases in the final list.
+
+ Returns:
+ The discovered test suites, each with test cases.
+ """
+ test_suites_with_cases = []
+
+ for test_suite_config in test_suite_configs:
+ test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
+ test_cases = []
+ func_test_cases, perf_test_cases = self._filter_test_cases(
+ test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases)
+ )
+ if func:
+ test_cases.extend(func_test_cases)
+ if perf:
+ test_cases.extend(perf_test_cases)
+
+ test_suites_with_cases.append(
+ TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases)
+ )
+
+ return test_suites_with_cases
+
+ def _get_test_suite_class(self, module_name: str) -> type[TestSuite]:
+ """Find the :class:`TestSuite` class in `module_name`.
+
+ The full module name is `module_name` prefixed with `self._test_suite_module_prefix`.
+ The module name is a standard filename with words separated with underscores.
+ Search the `module_name` for a :class:`TestSuite` class which starts
+ with `self._test_suite_class_prefix`, continuing with CamelCase `module_name`.
+ The first matching class is returned.
+
+ The CamelCase convention is not tested, only lowercase strings are compared.
+
+ Args:
+ module_name: The module name without prefix where to search for the test suite.
+
+ Returns:
+ The found test suite class.
+
+ Raises:
+ ConfigurationError: If the corresponding module is not found or
+ a valid :class:`TestSuite` is not found in the module.
+ """
+
+ def is_test_suite(object) -> bool:
+ """Check whether `object` is a :class:`TestSuite`.
+
+ The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself.
+
+ Args:
+ object: The object to be checked.
+
+ Returns:
+ :data:`True` if `object` is a subclass of `TestSuite`.
+ """
+ try:
+ if issubclass(object, TestSuite) and object is not TestSuite:
+ return True
+ except TypeError:
+ return False
+ return False
+
+ testsuite_module_path = f"{self._test_suite_module_prefix}{module_name}"
+ try:
+ test_suite_module = importlib.import_module(testsuite_module_path)
+ except ModuleNotFoundError as e:
+ raise ConfigurationError(
+ f"Test suite module '{testsuite_module_path}' not found."
+ ) from e
+
+ lowercase_suite_to_find = (
+ f"{self._test_suite_class_prefix}{module_name.replace('_', '')}".lower()
+ )
+ for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite):
+ if (
+ class_name.startswith(self._test_suite_class_prefix)
+ and lowercase_suite_to_find == class_name.lower()
+ ):
+ return class_obj
+ raise ConfigurationError(
+ f"Couldn't find any valid test suites in {test_suite_module.__name__}."
+ )
+
+ def _filter_test_cases(
+ self, test_suite_class: type[TestSuite], test_cases_to_run: set[str]
+ ) -> tuple[list[MethodType], list[MethodType]]:
+ """Filter `test_cases_to_run` from `test_suite_class`.
+
+ There are two rounds of filtering if `test_cases_to_run` is not empty.
+ The first filters `test_cases_to_run` from all methods of `test_suite_class`.
+ Then the methods are separated into functional and performance test cases.
+ If a method matches neither the functional nor performance name prefix, it's an error.
+
+ Args:
+ test_suite_class: The class of the test suite.
+ test_cases_to_run: Test case names to filter from `test_suite_class`.
+ If empty, return all matching test cases.
+
+ Returns:
+ A list of test case methods that should be executed.
+
+ Raises:
+ ConfigurationError: If a test case from `test_cases_to_run` is not found
+ or it doesn't match either the functional nor performance name prefix.
+ """
+ func_test_cases = []
+ perf_test_cases = []
+ name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
+ if test_cases_to_run:
+ name_method_tuples = [
+ (name, method) for name, method in name_method_tuples if name in test_cases_to_run
+ ]
+ if len(name_method_tuples) < len(test_cases_to_run):
+ missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples}
+ raise ConfigurationError(
+ f"Test cases {missing_test_cases} not found among methods "
+ f"of {test_suite_class.__name__}."
+ )
+
+ for test_case_name, test_case_method in name_method_tuples:
+ if re.match(self._func_test_case_regex, test_case_name):
+ func_test_cases.append(test_case_method)
+ elif re.match(self._perf_test_case_regex, test_case_name):
+ perf_test_cases.append(test_case_method)
+ elif test_cases_to_run:
+ raise ConfigurationError(
+ f"Method '{test_case_name}' matches neither "
+ f"a functional nor a performance test case name."
+ )
+
+ return func_test_cases, perf_test_cases
+
+ def _connect_nodes_and_run_execution(
+ self,
+ sut_nodes: dict[str, SutNode],
+ tg_nodes: dict[str, TGNode],
+ execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
+ ) -> None:
+ """Connect nodes, then continue to run the given execution.
+
+ Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`.
+ If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`,
+ respectively.
+ If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`.
+
+ Args:
+ sut_nodes: A dictionary storing connected/to be connected SUT nodes.
+ tg_nodes: A dictionary storing connected/to be connected TG nodes.
+ execution: An execution's test run configuration.
+ execution_result: The execution's result.
+ test_suites_with_cases: The test suites with test cases to run.
+ """
+ sut_node = sut_nodes.get(execution.system_under_test_node.name)
+ tg_node = tg_nodes.get(execution.traffic_generator_node.name)
+
+ try:
+ if not sut_node:
+ sut_node = SutNode(execution.system_under_test_node)
+ sut_nodes[sut_node.name] = sut_node
+ if not tg_node:
+ tg_node = TGNode(execution.traffic_generator_node)
+ tg_nodes[tg_node.name] = tg_node
+ except Exception as e:
+ failed_node = execution.system_under_test_node.name
+ if sut_node:
+ failed_node = execution.traffic_generator_node.name
+ self._logger.exception(f"The Creation of node {failed_node} failed.")
+ execution_result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_execution(
+ sut_node, tg_node, execution, execution_result, test_suites_with_cases
+ )
+
def _run_execution(
self,
sut_node: SutNode,
tg_node: TGNode,
execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
"""Run the given execution.
@@ -178,11 +397,11 @@ def _run_execution(
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
execution: An execution's test run configuration.
+ execution_result: The execution's result.
+ test_suites_with_cases: The test suites with test cases to run.
"""
self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
- execution_result = self._result.add_execution(sut_node.config)
execution_result.add_sut_info(sut_node.node_info)
-
try:
sut_node.set_up_execution(execution)
execution_result.update_setup(Result.PASS)
@@ -192,7 +411,10 @@ def _run_execution(
else:
for build_target in execution.build_targets:
- self._run_build_target(sut_node, tg_node, build_target, execution, execution_result)
+ build_target_result = execution_result.add_build_target(build_target)
+ self._run_build_target(
+ sut_node, tg_node, build_target, build_target_result, test_suites_with_cases
+ )
finally:
try:
@@ -207,8 +429,8 @@ def _run_build_target(
sut_node: SutNode,
tg_node: TGNode,
build_target: BuildTargetConfiguration,
- execution: ExecutionConfiguration,
- execution_result: ExecutionResult,
+ build_target_result: BuildTargetResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
"""Run the given build target.
@@ -220,11 +442,11 @@ def _run_build_target(
sut_node: The execution's sut node.
tg_node: The execution's tg node.
build_target: A build target's test run configuration.
- execution: The build target's execution's test run configuration.
- execution_result: The execution level result object associated with the execution.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ test_suites_with_cases: The test suites with test cases to run.
"""
self._logger.info(f"Running build target '{build_target.name}'.")
- build_target_result = execution_result.add_build_target(build_target)
try:
sut_node.set_up_build_target(build_target)
@@ -236,7 +458,7 @@ def _run_build_target(
build_target_result.update_setup(Result.FAIL, e)
else:
- self._run_test_suites(sut_node, tg_node, execution, build_target_result)
+ self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases)
finally:
try:
@@ -250,10 +472,10 @@ def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
- execution: ExecutionConfiguration,
build_target_result: BuildTargetResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
- """Run the execution's (possibly a subset of) test suites using the current build target.
+ """Run `test_suites_with_cases` with the current build target.
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
@@ -264,22 +486,20 @@ def _run_test_suites(
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
- execution: The execution's test run configuration associated
- with the current build target.
build_target_result: The build target level result object associated
with the current build target.
+ test_suites_with_cases: The test suites with test cases to run.
"""
end_build_target = False
- if not execution.skip_smoke_tests:
- execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
- for test_suite_config in execution.test_suites:
+ for test_suite_with_cases in test_suites_with_cases:
+ test_suite_result = build_target_result.add_test_suite(
+ test_suite_with_cases.test_suite_class.__name__
+ )
try:
- self._run_test_suite_module(
- sut_node, tg_node, execution, build_target_result, test_suite_config
- )
+ self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
except BlockingTestSuiteError as e:
self._logger.exception(
- f"An error occurred within {test_suite_config.test_suite}. "
+ f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
"Skipping build target..."
)
self._result.add_error(e)
@@ -288,15 +508,14 @@ def _run_test_suites(
if end_build_target:
break
- def _run_test_suite_module(
+ def _run_test_suite(
self,
sut_node: SutNode,
tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
- test_suite_config: TestSuiteConfig,
+ test_suite_result: TestSuiteResult,
+ test_suite_with_cases: TestSuiteWithCases,
) -> None:
- """Set up, execute and tear down all test suites in a single test suite module.
+ """Set up, execute and tear down `test_suite_with_cases`.
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
@@ -306,92 +525,79 @@ def _run_test_suite_module(
Record the setup and the teardown and handle failures.
- The test cases to execute are discovered when creating the :class:`TestSuite` object.
-
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
- execution: The execution's test run configuration associated
- with the current build target.
- build_target_result: The build target level result object associated
- with the current build target.
- test_suite_config: Test suite test run configuration specifying the test suite module
- and possibly a subset of test cases of test suites in that module.
+ test_suite_result: The test suite level result object associated
+ with the current test suite.
+ test_suite_with_cases: The test suite with test cases to run.
Raises:
BlockingTestSuiteError: If a blocking test suite fails.
"""
+ test_suite_name = test_suite_with_cases.test_suite_class.__name__
+ test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
try:
- full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
- test_suite_classes = get_test_suites(full_suite_path)
- suites_str = ", ".join((x.__name__ for x in test_suite_classes))
- self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
+ self._logger.info(f"Starting test suite setup: {test_suite_name}")
+ test_suite.set_up_suite()
+ test_suite_result.update_setup(Result.PASS)
+ self._logger.info(f"Test suite setup successful: {test_suite_name}")
except Exception as e:
- self._logger.exception("An error occurred when searching for test suites.")
- self._result.update_setup(Result.ERROR, e)
+ self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
+ test_suite_result.update_setup(Result.ERROR, e)
else:
- for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases)
-
- test_suite_name = test_suite.__class__.__name__
- test_suite_result = build_target_result.add_test_suite(test_suite_name)
- try:
- self._logger.info(f"Starting test suite setup: {test_suite_name}")
- test_suite.set_up_suite()
- test_suite_result.update_setup(Result.PASS)
- self._logger.info(f"Test suite setup successful: {test_suite_name}")
- except Exception as e:
- self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
- test_suite_result.update_setup(Result.ERROR, e)
-
- else:
- self._execute_test_suite(execution.func, test_suite, test_suite_result)
-
- finally:
- try:
- test_suite.tear_down_suite()
- sut_node.kill_cleanup_dpdk_apps()
- test_suite_result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
- self._logger.warning(
- f"Test suite '{test_suite_name}' teardown failed, "
- f"the next test suite may be affected."
- )
- test_suite_result.update_setup(Result.ERROR, e)
- if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
- raise BlockingTestSuiteError(test_suite_name)
+ self._execute_test_suite(
+ test_suite,
+ test_suite_with_cases.test_cases,
+ test_suite_result,
+ )
+ finally:
+ try:
+ test_suite.tear_down_suite()
+ sut_node.kill_cleanup_dpdk_apps()
+ test_suite_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
+ self._logger.warning(
+ f"Test suite '{test_suite_name}' teardown failed, "
+ "the next test suite may be affected."
+ )
+ test_suite_result.update_setup(Result.ERROR, e)
+ if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
+ raise BlockingTestSuiteError(test_suite_name)
def _execute_test_suite(
- self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
+ self,
+ test_suite: TestSuite,
+ test_cases: Iterable[MethodType],
+ test_suite_result: TestSuiteResult,
) -> None:
- """Execute all discovered test cases in `test_suite`.
+ """Execute all `test_cases` in `test_suite`.
If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment
variable is set, in case of a test case failure, the test case will be executed again
until it passes or it fails that many times in addition of the first failure.
Args:
- func: Whether to execute functional test cases.
test_suite: The test suite object.
+ test_cases: The list of test case methods.
test_suite_result: The test suite level result object associated
with the current test suite.
"""
- if func:
- for test_case_method in test_suite._get_functional_test_cases():
- test_case_name = test_case_method.__name__
- test_case_result = test_suite_result.add_test_case(test_case_name)
- all_attempts = SETTINGS.re_run + 1
- attempt_nr = 1
+ for test_case_method in test_cases:
+ test_case_name = test_case_method.__name__
+ test_case_result = test_suite_result.add_test_case(test_case_name)
+ all_attempts = SETTINGS.re_run + 1
+ attempt_nr = 1
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
self._run_test_case(test_suite, test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
- self._run_test_case(test_suite, test_case_method, test_case_result)
def _run_test_case(
self,
@@ -399,7 +605,7 @@ def _run_test_case(
test_case_method: MethodType,
test_case_result: TestCaseResult,
) -> None:
- """Setup, execute and teardown a test case in `test_suite`.
+ """Setup, execute and teardown `test_case_method` from `test_suite`.
Record the result of the setup and the teardown and handle failures.
@@ -424,7 +630,7 @@ def _run_test_case(
else:
# run test case if setup was successful
- self._execute_test_case(test_case_method, test_case_result)
+ self._execute_test_case(test_suite, test_case_method, test_case_result)
finally:
try:
@@ -440,11 +646,15 @@ def _run_test_case(
test_case_result.update(Result.ERROR)
def _execute_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
+ self,
+ test_suite: TestSuite,
+ test_case_method: MethodType,
+ test_case_result: TestCaseResult,
) -> None:
- """Execute one test case, record the result and handle failures.
+ """Execute `test_case_method` from `test_suite`, record the result and handle failures.
Args:
+ test_suite: The test suite object.
test_case_method: The test case method.
test_case_result: The test case level result object associated
with the current test case.
@@ -452,7 +662,7 @@ def _execute_test_case(
test_case_name = test_case_method.__name__
try:
self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method()
+ test_case_method(test_suite)
test_case_result.update(Result.PASS)
self._logger.info(f"Test case execution PASSED: {test_case_name}")
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 609c8d0e62..2b8bfbe0ed 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser:
"--test-cases",
action=_env_arg("DTS_TESTCASES"),
default="",
- help="[DTS_TESTCASES] Comma-separated list of test cases to execute. "
- "Unknown test cases will be silently ignored.",
+ help="[DTS_TESTCASES] Comma-separated list of test cases to execute.",
)
parser.add_argument(
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 4467749a9d..075195fd5b 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -25,7 +25,9 @@
import os.path
from collections.abc import MutableSequence
+from dataclasses import dataclass
from enum import Enum, auto
+from types import MethodType
from .config import (
OS,
@@ -36,10 +38,42 @@
CPUType,
NodeConfiguration,
NodeInfo,
+ TestSuiteConfig,
)
from .exception import DTSError, ErrorSeverity
from .logger import DTSLOG
from .settings import SETTINGS
+from .test_suite import TestSuite
+
+
+@dataclass(slots=True, frozen=True)
+class TestSuiteWithCases:
+ """A test suite class with test case methods.
+
+ An auxiliary class holding a test case class with test case methods. The intended use of this
+ class is to hold a subset of test cases (which could be all test cases) because we don't have
+ all the data to instantiate the class at the point of inspection. The knowledge of this subset
+ is needed in case an error occurs before the class is instantiated and we need to record
+ which test cases were blocked by the error.
+
+ Attributes:
+ test_suite_class: The test suite class.
+ test_cases: The test case methods.
+ """
+
+ test_suite_class: type[TestSuite]
+ test_cases: list[MethodType]
+
+ def create_config(self) -> TestSuiteConfig:
+ """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
+
+ Returns:
+ The :class:`TestSuiteConfig` representation.
+ """
+ return TestSuiteConfig(
+ test_suite=self.test_suite_class.__name__,
+ test_cases=[test_case.__name__ for test_case in self.test_cases],
+ )
class Result(Enum):
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index b02fd36147..f9fe88093e 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -11,25 +11,17 @@
* Testbed (SUT, TG) configuration,
* Packet sending and verification,
* Test case verification.
-
-The module also defines a function, :func:`get_test_suites`,
-for gathering test suites from a Python module.
"""
-import importlib
-import inspect
-import re
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
-from types import MethodType
-from typing import Any, ClassVar, Union
+from typing import ClassVar, Union
from scapy.layers.inet import IP # type: ignore[import]
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
-from .exception import ConfigurationError, TestCaseVerifyError
+from .exception import TestCaseVerifyError
from .logger import DTSLOG, getLogger
-from .settings import SETTINGS
from .testbed_model import Port, PortLink, SutNode, TGNode
from .utils import get_packet_summaries
@@ -37,7 +29,6 @@
class TestSuite(object):
"""The base class with building blocks needed by most test cases.
- * Test case filtering and collection,
* Test suite setup/cleanup methods to override,
* Test case setup/cleanup methods to override,
* Test case verification,
@@ -71,7 +62,6 @@ class TestSuite(object):
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
_logger: DTSLOG
- _test_cases_to_run: list[str]
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -86,24 +76,19 @@ def __init__(
self,
sut_node: SutNode,
tg_node: TGNode,
- test_cases: list[str],
):
"""Initialize the test suite testbed information and basic configuration.
- Process what test cases to run, find links between ports and set up
- default IP addresses to be used when configuring them.
+ Find links between ports and set up default IP addresses to be used when
+ configuring them.
Args:
sut_node: The SUT node where the test suite will run.
tg_node: The TG node where the test suite will run.
- test_cases: The list of test cases to execute.
- If empty, all test cases will be executed.
"""
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = getLogger(self.__class__.__name__)
- self._test_cases_to_run = test_cases
- self._test_cases_to_run.extend(SETTINGS.test_cases)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
@@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
return False
return True
-
- def _get_functional_test_cases(self) -> list[MethodType]:
- """Get all functional test cases defined in this TestSuite.
-
- Returns:
- The list of functional test cases of this TestSuite.
- """
- return self._get_test_cases(r"test_(?!perf_)")
-
- def _get_test_cases(self, test_case_regex: str) -> list[MethodType]:
- """Return a list of test cases matching test_case_regex.
-
- Returns:
- The list of test cases matching test_case_regex of this TestSuite.
- """
- self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.")
- filtered_test_cases = []
- for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod):
- if self._should_be_executed(test_case_name, test_case_regex):
- filtered_test_cases.append(test_case)
- cases_str = ", ".join((x.__name__ for x in filtered_test_cases))
- self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.")
- return filtered_test_cases
-
- def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool:
- """Check whether the test case should be scheduled to be executed."""
- match = bool(re.match(test_case_regex, test_case_name))
- if self._test_cases_to_run:
- return match and test_case_name in self._test_cases_to_run
-
- return match
-
-
-def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
- r"""Find all :class:`TestSuite`\s in a Python module.
-
- Args:
- testsuite_module_path: The path to the Python module.
-
- Returns:
- The list of :class:`TestSuite`\s found within the Python module.
-
- Raises:
- ConfigurationError: The test suite module was not found.
- """
-
- def is_test_suite(object: Any) -> bool:
- try:
- if issubclass(object, TestSuite) and object is not TestSuite:
- return True
- except TypeError:
- return False
- return False
-
- try:
- testcase_module = importlib.import_module(testsuite_module_path)
- except ModuleNotFoundError as e:
- raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e
- return [
- test_suite_class
- for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite)
- ]
diff --git a/dts/pyproject.toml b/dts/pyproject.toml
index 28bd970ae4..8eb92b4f11 100644
--- a/dts/pyproject.toml
+++ b/dts/pyproject.toml
@@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes"
format = "pylint"
max_line_length = 100
+[tool.pylama.linter.pycodestyle]
+ignore = "E203,W503"
+
[tool.pylama.linter.pydocstyle]
convention = "google"
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index 5e2bac14bd..7b2a0e97f8 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -21,7 +21,7 @@
from framework.utils import REGEX_FOR_PCI_ADDRESS
-class SmokeTests(TestSuite):
+class TestSmokeTests(TestSuite):
"""DPDK and infrastructure smoke test suite.
The test cases validate the most basic DPDK functionality needed for all other test suites.
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 4/7] dts: reorganize test result
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
` (2 preceding siblings ...)
2024-02-23 7:54 ` [PATCH v3 3/7] dts: filter test suites in executions Juraj Linkeš
@ 2024-02-23 7:54 ` Juraj Linkeš
2024-02-23 7:55 ` [PATCH v3 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš
` (3 subsequent siblings)
7 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
The current order of Result classes in the test_suite.py module is
guided by the needs of type hints, which is not as intuitively readable
as ordering them by the occurrences in code. The order goes from the
topmost level to lowermost:
BaseResult
DTSResult
ExecutionResult
BuildTargetResult
TestSuiteResult
TestCaseResult
This is the same order as they're used in the runner module and they're
also used in the same order between themselves in the test_result
module.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/test_result.py | 411 ++++++++++++++++++-----------------
1 file changed, 206 insertions(+), 205 deletions(-)
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 075195fd5b..abdbafab10 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -28,6 +28,7 @@
from dataclasses import dataclass
from enum import Enum, auto
from types import MethodType
+from typing import Union
from .config import (
OS,
@@ -129,58 +130,6 @@ def __bool__(self) -> bool:
return bool(self.result)
-class Statistics(dict):
- """How many test cases ended in which result state along some other basic information.
-
- Subclassing :class:`dict` provides a convenient way to format the data.
-
- The data are stored in the following keys:
-
- * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases.
- * **DPDK VERSION** (:class:`str`) -- The tested DPDK version.
- """
-
- def __init__(self, dpdk_version: str | None):
- """Extend the constructor with keys in which the data are stored.
-
- Args:
- dpdk_version: The version of tested DPDK.
- """
- super(Statistics, self).__init__()
- for result in Result:
- self[result.name] = 0
- self["PASS RATE"] = 0.0
- self["DPDK VERSION"] = dpdk_version
-
- def __iadd__(self, other: Result) -> "Statistics":
- """Add a Result to the final count.
-
- Example:
- stats: Statistics = Statistics() # empty Statistics
- stats += Result.PASS # add a Result to `stats`
-
- Args:
- other: The Result to add to this statistics object.
-
- Returns:
- The modified statistics object.
- """
- self[other.name] += 1
- self["PASS RATE"] = (
- float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result)
- )
- return self
-
- def __str__(self) -> str:
- """Each line contains the formatted key = value pair."""
- stats_str = ""
- for key, value in self.items():
- stats_str += f"{key:<12} = {value}\n"
- # according to docs, we should use \n when writing to text files
- # on all platforms
- return stats_str
-
-
class BaseResult(object):
"""Common data and behavior of DTS results.
@@ -245,7 +194,7 @@ def get_errors(self) -> list[Exception]:
"""
return self._get_setup_teardown_errors() + self._get_inner_errors()
- def add_stats(self, statistics: Statistics) -> None:
+ def add_stats(self, statistics: "Statistics") -> None:
"""Collate stats from the whole result hierarchy.
Args:
@@ -255,91 +204,149 @@ def add_stats(self, statistics: Statistics) -> None:
inner_result.add_stats(statistics)
-class TestCaseResult(BaseResult, FixtureResult):
- r"""The test case specific result.
+class DTSResult(BaseResult):
+ """Stores environment information and test results from a DTS run.
- Stores the result of the actual test case. This is done by adding an extra superclass
- in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and
- the class is itself a record of the test case.
+ * Execution level information, such as testbed and the test suite list,
+ * Build target level information, such as compiler, target OS and cpu,
+ * Test suite and test case results,
+ * All errors that are caught and recorded during DTS execution.
+
+ The information is stored hierarchically. This is the first level of the hierarchy
+ and as such is where the data form the whole hierarchy is collated or processed.
+
+ The internal list stores the results of all executions.
Attributes:
- test_case_name: The test case name.
+ dpdk_version: The DPDK version to record.
"""
- test_case_name: str
+ dpdk_version: str | None
+ _logger: DTSLOG
+ _errors: list[Exception]
+ _return_code: ErrorSeverity
+ _stats_result: Union["Statistics", None]
+ _stats_filename: str
- def __init__(self, test_case_name: str):
- """Extend the constructor with `test_case_name`.
+ def __init__(self, logger: DTSLOG):
+ """Extend the constructor with top-level specifics.
Args:
- test_case_name: The test case's name.
+ logger: The logger instance the whole result will use.
"""
- super(TestCaseResult, self).__init__()
- self.test_case_name = test_case_name
+ super(DTSResult, self).__init__()
+ self.dpdk_version = None
+ self._logger = logger
+ self._errors = []
+ self._return_code = ErrorSeverity.NO_ERR
+ self._stats_result = None
+ self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
- def update(self, result: Result, error: Exception | None = None) -> None:
- """Update the test case result.
+ def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult":
+ """Add and return the inner result (execution).
- This updates the result of the test case itself and doesn't affect
- the results of the setup and teardown steps in any way.
+ Args:
+ sut_node: The SUT node's test run configuration.
+
+ Returns:
+ The execution's result.
+ """
+ execution_result = ExecutionResult(sut_node)
+ self._inner_results.append(execution_result)
+ return execution_result
+
+ def add_error(self, error: Exception) -> None:
+ """Record an error that occurred outside any execution.
Args:
- result: The result of the test case.
- error: The error that occurred in case of a failure.
+ error: The exception to record.
"""
- self.result = result
- self.error = error
+ self._errors.append(error)
- def _get_inner_errors(self) -> list[Exception]:
- if self.error:
- return [self.error]
- return []
+ def process(self) -> None:
+ """Process the data after a whole DTS run.
- def add_stats(self, statistics: Statistics) -> None:
- r"""Add the test case result to statistics.
+ The data is added to inner objects during runtime and this object is not updated
+ at that time. This requires us to process the inner data after it's all been gathered.
- The base method goes through the hierarchy recursively and this method is here to stop
- the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree.
+ The processing gathers all errors and the statistics of test case results.
+ """
+ self._errors += self.get_errors()
+ if self._errors and self._logger:
+ self._logger.debug("Summary of errors:")
+ for error in self._errors:
+ self._logger.debug(repr(error))
- Args:
- statistics: The :class:`Statistics` object where the stats will be added.
+ self._stats_result = Statistics(self.dpdk_version)
+ self.add_stats(self._stats_result)
+ with open(self._stats_filename, "w+") as stats_file:
+ stats_file.write(str(self._stats_result))
+
+ def get_return_code(self) -> int:
+ """Go through all stored Exceptions and return the final DTS error code.
+
+ Returns:
+ The highest error code found.
"""
- statistics += self.result
+ for error in self._errors:
+ error_return_code = ErrorSeverity.GENERIC_ERR
+ if isinstance(error, DTSError):
+ error_return_code = error.severity
- def __bool__(self) -> bool:
- """The test case passed only if setup, teardown and the test case itself passed."""
- return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
+ if error_return_code > self._return_code:
+ self._return_code = error_return_code
+ return int(self._return_code)
-class TestSuiteResult(BaseResult):
- """The test suite specific result.
- The internal list stores the results of all test cases in a given test suite.
+class ExecutionResult(BaseResult):
+ """The execution specific result.
+
+ The internal list stores the results of all build targets in a given execution.
Attributes:
- suite_name: The test suite name.
+ sut_node: The SUT node used in the execution.
+ sut_os_name: The operating system of the SUT node.
+ sut_os_version: The operating system version of the SUT node.
+ sut_kernel_version: The operating system kernel version of the SUT node.
"""
- suite_name: str
+ sut_node: NodeConfiguration
+ sut_os_name: str
+ sut_os_version: str
+ sut_kernel_version: str
- def __init__(self, suite_name: str):
- """Extend the constructor with `suite_name`.
+ def __init__(self, sut_node: NodeConfiguration):
+ """Extend the constructor with the `sut_node`'s config.
Args:
- suite_name: The test suite's name.
+ sut_node: The SUT node's test run configuration used in the execution.
"""
- super(TestSuiteResult, self).__init__()
- self.suite_name = suite_name
+ super(ExecutionResult, self).__init__()
+ self.sut_node = sut_node
- def add_test_case(self, test_case_name: str) -> TestCaseResult:
- """Add and return the inner result (test case).
+ def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult":
+ """Add and return the inner result (build target).
+
+ Args:
+ build_target: The build target's test run configuration.
Returns:
- The test case's result.
+ The build target's result.
"""
- test_case_result = TestCaseResult(test_case_name)
- self._inner_results.append(test_case_result)
- return test_case_result
+ build_target_result = BuildTargetResult(build_target)
+ self._inner_results.append(build_target_result)
+ return build_target_result
+
+ def add_sut_info(self, sut_info: NodeInfo) -> None:
+ """Add SUT information gathered at runtime.
+
+ Args:
+ sut_info: The additional SUT node information.
+ """
+ self.sut_os_name = sut_info.os_name
+ self.sut_os_version = sut_info.os_version
+ self.sut_kernel_version = sut_info.kernel_version
class BuildTargetResult(BaseResult):
@@ -386,7 +393,7 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def add_test_suite(self, test_suite_name: str) -> TestSuiteResult:
+ def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult":
"""Add and return the inner result (test suite).
Returns:
@@ -397,146 +404,140 @@ def add_test_suite(self, test_suite_name: str) -> TestSuiteResult:
return test_suite_result
-class ExecutionResult(BaseResult):
- """The execution specific result.
+class TestSuiteResult(BaseResult):
+ """The test suite specific result.
- The internal list stores the results of all build targets in a given execution.
+ The internal list stores the results of all test cases in a given test suite.
Attributes:
- sut_node: The SUT node used in the execution.
- sut_os_name: The operating system of the SUT node.
- sut_os_version: The operating system version of the SUT node.
- sut_kernel_version: The operating system kernel version of the SUT node.
+ suite_name: The test suite name.
"""
- sut_node: NodeConfiguration
- sut_os_name: str
- sut_os_version: str
- sut_kernel_version: str
+ suite_name: str
- def __init__(self, sut_node: NodeConfiguration):
- """Extend the constructor with the `sut_node`'s config.
+ def __init__(self, suite_name: str):
+ """Extend the constructor with `suite_name`.
Args:
- sut_node: The SUT node's test run configuration used in the execution.
+ suite_name: The test suite's name.
"""
- super(ExecutionResult, self).__init__()
- self.sut_node = sut_node
-
- def add_build_target(self, build_target: BuildTargetConfiguration) -> BuildTargetResult:
- """Add and return the inner result (build target).
+ super(TestSuiteResult, self).__init__()
+ self.suite_name = suite_name
- Args:
- build_target: The build target's test run configuration.
+ def add_test_case(self, test_case_name: str) -> "TestCaseResult":
+ """Add and return the inner result (test case).
Returns:
- The build target's result.
- """
- build_target_result = BuildTargetResult(build_target)
- self._inner_results.append(build_target_result)
- return build_target_result
-
- def add_sut_info(self, sut_info: NodeInfo) -> None:
- """Add SUT information gathered at runtime.
-
- Args:
- sut_info: The additional SUT node information.
+ The test case's result.
"""
- self.sut_os_name = sut_info.os_name
- self.sut_os_version = sut_info.os_version
- self.sut_kernel_version = sut_info.kernel_version
+ test_case_result = TestCaseResult(test_case_name)
+ self._inner_results.append(test_case_result)
+ return test_case_result
-class DTSResult(BaseResult):
- """Stores environment information and test results from a DTS run.
-
- * Execution level information, such as testbed and the test suite list,
- * Build target level information, such as compiler, target OS and cpu,
- * Test suite and test case results,
- * All errors that are caught and recorded during DTS execution.
-
- The information is stored hierarchically. This is the first level of the hierarchy
- and as such is where the data form the whole hierarchy is collated or processed.
+class TestCaseResult(BaseResult, FixtureResult):
+ r"""The test case specific result.
- The internal list stores the results of all executions.
+ Stores the result of the actual test case. This is done by adding an extra superclass
+ in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and
+ the class is itself a record of the test case.
Attributes:
- dpdk_version: The DPDK version to record.
+ test_case_name: The test case name.
"""
- dpdk_version: str | None
- _logger: DTSLOG
- _errors: list[Exception]
- _return_code: ErrorSeverity
- _stats_result: Statistics | None
- _stats_filename: str
+ test_case_name: str
- def __init__(self, logger: DTSLOG):
- """Extend the constructor with top-level specifics.
+ def __init__(self, test_case_name: str):
+ """Extend the constructor with `test_case_name`.
Args:
- logger: The logger instance the whole result will use.
+ test_case_name: The test case's name.
"""
- super(DTSResult, self).__init__()
- self.dpdk_version = None
- self._logger = logger
- self._errors = []
- self._return_code = ErrorSeverity.NO_ERR
- self._stats_result = None
- self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
+ super(TestCaseResult, self).__init__()
+ self.test_case_name = test_case_name
- def add_execution(self, sut_node: NodeConfiguration) -> ExecutionResult:
- """Add and return the inner result (execution).
+ def update(self, result: Result, error: Exception | None = None) -> None:
+ """Update the test case result.
- Args:
- sut_node: The SUT node's test run configuration.
+ This updates the result of the test case itself and doesn't affect
+ the results of the setup and teardown steps in any way.
- Returns:
- The execution's result.
+ Args:
+ result: The result of the test case.
+ error: The error that occurred in case of a failure.
"""
- execution_result = ExecutionResult(sut_node)
- self._inner_results.append(execution_result)
- return execution_result
+ self.result = result
+ self.error = error
- def add_error(self, error: Exception) -> None:
- """Record an error that occurred outside any execution.
+ def _get_inner_errors(self) -> list[Exception]:
+ if self.error:
+ return [self.error]
+ return []
+
+ def add_stats(self, statistics: "Statistics") -> None:
+ r"""Add the test case result to statistics.
+
+ The base method goes through the hierarchy recursively and this method is here to stop
+ the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree.
Args:
- error: The exception to record.
+ statistics: The :class:`Statistics` object where the stats will be added.
"""
- self._errors.append(error)
+ statistics += self.result
- def process(self) -> None:
- """Process the data after a whole DTS run.
+ def __bool__(self) -> bool:
+ """The test case passed only if setup, teardown and the test case itself passed."""
+ return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
- The data is added to inner objects during runtime and this object is not updated
- at that time. This requires us to process the inner data after it's all been gathered.
- The processing gathers all errors and the statistics of test case results.
+class Statistics(dict):
+ """How many test cases ended in which result state along some other basic information.
+
+ Subclassing :class:`dict` provides a convenient way to format the data.
+
+ The data are stored in the following keys:
+
+ * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases.
+ * **DPDK VERSION** (:class:`str`) -- The tested DPDK version.
+ """
+
+ def __init__(self, dpdk_version: str | None):
+ """Extend the constructor with keys in which the data are stored.
+
+ Args:
+ dpdk_version: The version of tested DPDK.
"""
- self._errors += self.get_errors()
- if self._errors and self._logger:
- self._logger.debug("Summary of errors:")
- for error in self._errors:
- self._logger.debug(repr(error))
+ super(Statistics, self).__init__()
+ for result in Result:
+ self[result.name] = 0
+ self["PASS RATE"] = 0.0
+ self["DPDK VERSION"] = dpdk_version
- self._stats_result = Statistics(self.dpdk_version)
- self.add_stats(self._stats_result)
- with open(self._stats_filename, "w+") as stats_file:
- stats_file.write(str(self._stats_result))
+ def __iadd__(self, other: Result) -> "Statistics":
+ """Add a Result to the final count.
- def get_return_code(self) -> int:
- """Go through all stored Exceptions and return the final DTS error code.
+ Example:
+ stats: Statistics = Statistics() # empty Statistics
+ stats += Result.PASS # add a Result to `stats`
+
+ Args:
+ other: The Result to add to this statistics object.
Returns:
- The highest error code found.
+ The modified statistics object.
"""
- for error in self._errors:
- error_return_code = ErrorSeverity.GENERIC_ERR
- if isinstance(error, DTSError):
- error_return_code = error.severity
-
- if error_return_code > self._return_code:
- self._return_code = error_return_code
+ self[other.name] += 1
+ self["PASS RATE"] = (
+ float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result)
+ )
+ return self
- return int(self._return_code)
+ def __str__(self) -> str:
+ """Each line contains the formatted key = value pair."""
+ stats_str = ""
+ for key, value in self.items():
+ stats_str += f"{key:<12} = {value}\n"
+ # according to docs, we should use \n when writing to text files
+ # on all platforms
+ return stats_str
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 5/7] dts: block all test cases when earlier setup fails
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
` (3 preceding siblings ...)
2024-02-23 7:54 ` [PATCH v3 4/7] dts: reorganize test result Juraj Linkeš
@ 2024-02-23 7:55 ` Juraj Linkeš
2024-02-23 7:55 ` [PATCH v3 6/7] dts: refactor logging configuration Juraj Linkeš
` (2 subsequent siblings)
7 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-23 7:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
In case of a failure before a test suite, the child results will be
recursively recorded as blocked, giving us a full report which was
missing previously.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 21 ++--
dts/framework/test_result.py | 186 +++++++++++++++++++++++++----------
2 files changed, 148 insertions(+), 59 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index e8030365ac..511f8be064 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -60,13 +60,15 @@ class DTSRunner:
Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult`
or one of its subclasses. The test case results are also recorded.
- If an error occurs, the current stage is aborted, the error is recorded and the run continues in
- the next iteration of the same stage. The return code is the highest `severity` of all
+ If an error occurs, the current stage is aborted, the error is recorded, everything in
+ the inner stages is marked as blocked and the run continues in the next iteration
+ of the same stage. The return code is the highest `severity` of all
:class:`~.framework.exception.DTSError`\s.
Example:
- An error occurs in a build target setup. The current build target is aborted and the run
- continues with the next build target. If the errored build target was the last one in the
+ An error occurs in a build target setup. The current build target is aborted,
+ all test suites and their test cases are marked as blocked and the run continues
+ with the next build target. If the errored build target was the last one in the
given execution, the next execution begins.
"""
@@ -100,6 +102,10 @@ def run(self):
test case within the test suite is set up, executed and torn down. After all test cases
have been executed, the test suite is torn down and the next build target will be tested.
+ In order to properly mark test suites and test cases as blocked in case of a failure,
+ we need to have discovered which test suites and test cases to run before any failures
+ happen. The discovery happens at the earliest point at the start of each execution.
+
All the nested steps look like this:
#. Execution setup
@@ -134,7 +140,7 @@ def run(self):
self._logger.info(
f"Running execution with SUT '{execution.system_under_test_node.name}'."
)
- execution_result = self._result.add_execution(execution.system_under_test_node)
+ execution_result = self._result.add_execution(execution)
# we don't want to modify the original config, so create a copy
execution_test_suites = list(execution.test_suites)
if not execution.skip_smoke_tests:
@@ -143,6 +149,7 @@ def run(self):
test_suites_with_cases = self._get_test_suites_with_cases(
execution_test_suites, execution.func, execution.perf
)
+ execution_result.test_suites_with_cases = test_suites_with_cases
except Exception as e:
self._logger.exception(
f"Invalid test suite configuration found: " f"{execution_test_suites}."
@@ -492,9 +499,7 @@ def _run_test_suites(
"""
end_build_target = False
for test_suite_with_cases in test_suites_with_cases:
- test_suite_result = build_target_result.add_test_suite(
- test_suite_with_cases.test_suite_class.__name__
- )
+ test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
try:
self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
except BlockingTestSuiteError as e:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index abdbafab10..eedb2d20ee 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -37,7 +37,7 @@
BuildTargetInfo,
Compiler,
CPUType,
- NodeConfiguration,
+ ExecutionConfiguration,
NodeInfo,
TestSuiteConfig,
)
@@ -88,6 +88,8 @@ class Result(Enum):
ERROR = auto()
#:
SKIP = auto()
+ #:
+ BLOCK = auto()
def __bool__(self) -> bool:
"""Only PASS is True."""
@@ -141,21 +143,26 @@ class BaseResult(object):
Attributes:
setup_result: The result of the setup of the particular stage.
teardown_result: The results of the teardown of the particular stage.
+ child_results: The results of the descendants in the results hierarchy.
"""
setup_result: FixtureResult
teardown_result: FixtureResult
- _inner_results: MutableSequence["BaseResult"]
+ child_results: MutableSequence["BaseResult"]
def __init__(self):
"""Initialize the constructor."""
self.setup_result = FixtureResult()
self.teardown_result = FixtureResult()
- self._inner_results = []
+ self.child_results = []
def update_setup(self, result: Result, error: Exception | None = None) -> None:
"""Store the setup result.
+ If the result is :attr:`~Result.BLOCK`, :attr:`~Result.ERROR` or :attr:`~Result.FAIL`,
+ then the corresponding child results in result hierarchy
+ are also marked with :attr:`~Result.BLOCK`.
+
Args:
result: The result of the setup.
error: The error that occurred in case of a failure.
@@ -163,6 +170,16 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
self.setup_result.result = result
self.setup_result.error = error
+ if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
+ self.update_teardown(Result.BLOCK)
+ self._block_result()
+
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed.
+
+ The blocking of child results should be done in overloaded methods.
+ """
+
def update_teardown(self, result: Result, error: Exception | None = None) -> None:
"""Store the teardown result.
@@ -181,10 +198,8 @@ def _get_setup_teardown_errors(self) -> list[Exception]:
errors.append(self.teardown_result.error)
return errors
- def _get_inner_errors(self) -> list[Exception]:
- return [
- error for inner_result in self._inner_results for error in inner_result.get_errors()
- ]
+ def _get_child_errors(self) -> list[Exception]:
+ return [error for child_result in self.child_results for error in child_result.get_errors()]
def get_errors(self) -> list[Exception]:
"""Compile errors from the whole result hierarchy.
@@ -192,7 +207,7 @@ def get_errors(self) -> list[Exception]:
Returns:
The errors from setup, teardown and all errors found in the whole result hierarchy.
"""
- return self._get_setup_teardown_errors() + self._get_inner_errors()
+ return self._get_setup_teardown_errors() + self._get_child_errors()
def add_stats(self, statistics: "Statistics") -> None:
"""Collate stats from the whole result hierarchy.
@@ -200,8 +215,8 @@ def add_stats(self, statistics: "Statistics") -> None:
Args:
statistics: The :class:`Statistics` object where the stats will be collated.
"""
- for inner_result in self._inner_results:
- inner_result.add_stats(statistics)
+ for child_result in self.child_results:
+ child_result.add_stats(statistics)
class DTSResult(BaseResult):
@@ -242,18 +257,18 @@ def __init__(self, logger: DTSLOG):
self._stats_result = None
self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
- def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult":
- """Add and return the inner result (execution).
+ def add_execution(self, execution: ExecutionConfiguration) -> "ExecutionResult":
+ """Add and return the child result (execution).
Args:
- sut_node: The SUT node's test run configuration.
+ execution: The execution's test run configuration.
Returns:
The execution's result.
"""
- execution_result = ExecutionResult(sut_node)
- self._inner_results.append(execution_result)
- return execution_result
+ result = ExecutionResult(execution)
+ self.child_results.append(result)
+ return result
def add_error(self, error: Exception) -> None:
"""Record an error that occurred outside any execution.
@@ -266,8 +281,8 @@ def add_error(self, error: Exception) -> None:
def process(self) -> None:
"""Process the data after a whole DTS run.
- The data is added to inner objects during runtime and this object is not updated
- at that time. This requires us to process the inner data after it's all been gathered.
+ The data is added to child objects during runtime and this object is not updated
+ at that time. This requires us to process the child data after it's all been gathered.
The processing gathers all errors and the statistics of test case results.
"""
@@ -305,28 +320,30 @@ class ExecutionResult(BaseResult):
The internal list stores the results of all build targets in a given execution.
Attributes:
- sut_node: The SUT node used in the execution.
sut_os_name: The operating system of the SUT node.
sut_os_version: The operating system version of the SUT node.
sut_kernel_version: The operating system kernel version of the SUT node.
"""
- sut_node: NodeConfiguration
sut_os_name: str
sut_os_version: str
sut_kernel_version: str
+ _config: ExecutionConfiguration
+ _parent_result: DTSResult
+ _test_suites_with_cases: list[TestSuiteWithCases]
- def __init__(self, sut_node: NodeConfiguration):
- """Extend the constructor with the `sut_node`'s config.
+ def __init__(self, execution: ExecutionConfiguration):
+ """Extend the constructor with the execution's config and DTSResult.
Args:
- sut_node: The SUT node's test run configuration used in the execution.
+ execution: The execution's test run configuration.
"""
super(ExecutionResult, self).__init__()
- self.sut_node = sut_node
+ self._config = execution
+ self._test_suites_with_cases = []
def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult":
- """Add and return the inner result (build target).
+ """Add and return the child result (build target).
Args:
build_target: The build target's test run configuration.
@@ -334,9 +351,34 @@ def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTarg
Returns:
The build target's result.
"""
- build_target_result = BuildTargetResult(build_target)
- self._inner_results.append(build_target_result)
- return build_target_result
+ result = BuildTargetResult(
+ self._test_suites_with_cases,
+ build_target,
+ )
+ self.child_results.append(result)
+ return result
+
+ @property
+ def test_suites_with_cases(self) -> list[TestSuiteWithCases]:
+ """The test suites with test cases to be executed in this execution.
+
+ The test suites can only be assigned once.
+
+ Returns:
+ The list of test suites with test cases. If an error occurs between
+ the initialization of :class:`ExecutionResult` and assigning test cases to the instance,
+ return an empty list, representing that we don't know what to execute.
+ """
+ return self._test_suites_with_cases
+
+ @test_suites_with_cases.setter
+ def test_suites_with_cases(self, test_suites_with_cases: list[TestSuiteWithCases]) -> None:
+ if self._test_suites_with_cases:
+ raise ValueError(
+ "Attempted to assign test suites to an execution result "
+ "which already has test suites."
+ )
+ self._test_suites_with_cases = test_suites_with_cases
def add_sut_info(self, sut_info: NodeInfo) -> None:
"""Add SUT information gathered at runtime.
@@ -348,6 +390,12 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
self.sut_os_version = sut_info.os_version
self.sut_kernel_version = sut_info.kernel_version
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ for build_target in self._config.build_targets:
+ child_result = self.add_build_target(build_target)
+ child_result.update_setup(Result.BLOCK)
+
class BuildTargetResult(BaseResult):
"""The build target specific result.
@@ -369,11 +417,17 @@ class BuildTargetResult(BaseResult):
compiler: Compiler
compiler_version: str | None
dpdk_version: str | None
+ _test_suites_with_cases: list[TestSuiteWithCases]
- def __init__(self, build_target: BuildTargetConfiguration):
- """Extend the constructor with the `build_target`'s build target config.
+ def __init__(
+ self,
+ test_suites_with_cases: list[TestSuiteWithCases],
+ build_target: BuildTargetConfiguration,
+ ):
+ """Extend the constructor with the build target's config and ExecutionResult.
Args:
+ test_suites_with_cases: The test suites with test cases to be run in this build target.
build_target: The build target's test run configuration.
"""
super(BuildTargetResult, self).__init__()
@@ -383,6 +437,23 @@ def __init__(self, build_target: BuildTargetConfiguration):
self.compiler = build_target.compiler
self.compiler_version = None
self.dpdk_version = None
+ self._test_suites_with_cases = test_suites_with_cases
+
+ def add_test_suite(
+ self,
+ test_suite_with_cases: TestSuiteWithCases,
+ ) -> "TestSuiteResult":
+ """Add and return the child result (test suite).
+
+ Args:
+ test_suite_with_cases: The test suite with test cases.
+
+ Returns:
+ The test suite's result.
+ """
+ result = TestSuiteResult(test_suite_with_cases)
+ self.child_results.append(result)
+ return result
def add_build_target_info(self, versions: BuildTargetInfo) -> None:
"""Add information about the build target gathered at runtime.
@@ -393,15 +464,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult":
- """Add and return the inner result (test suite).
-
- Returns:
- The test suite's result.
- """
- test_suite_result = TestSuiteResult(test_suite_name)
- self._inner_results.append(test_suite_result)
- return test_suite_result
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ for test_suite_with_cases in self._test_suites_with_cases:
+ child_result = self.add_test_suite(test_suite_with_cases)
+ child_result.update_setup(Result.BLOCK)
class TestSuiteResult(BaseResult):
@@ -410,29 +477,42 @@ class TestSuiteResult(BaseResult):
The internal list stores the results of all test cases in a given test suite.
Attributes:
- suite_name: The test suite name.
+ test_suite_name: The test suite name.
"""
- suite_name: str
+ test_suite_name: str
+ _test_suite_with_cases: TestSuiteWithCases
+ _parent_result: BuildTargetResult
+ _child_configs: list[str]
- def __init__(self, suite_name: str):
- """Extend the constructor with `suite_name`.
+ def __init__(self, test_suite_with_cases: TestSuiteWithCases):
+ """Extend the constructor with test suite's config and BuildTargetResult.
Args:
- suite_name: The test suite's name.
+ test_suite_with_cases: The test suite with test cases.
"""
super(TestSuiteResult, self).__init__()
- self.suite_name = suite_name
+ self.test_suite_name = test_suite_with_cases.test_suite_class.__name__
+ self._test_suite_with_cases = test_suite_with_cases
def add_test_case(self, test_case_name: str) -> "TestCaseResult":
- """Add and return the inner result (test case).
+ """Add and return the child result (test case).
+
+ Args:
+ test_case_name: The name of the test case.
Returns:
The test case's result.
"""
- test_case_result = TestCaseResult(test_case_name)
- self._inner_results.append(test_case_result)
- return test_case_result
+ result = TestCaseResult(test_case_name)
+ self.child_results.append(result)
+ return result
+
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ for test_case_method in self._test_suite_with_cases.test_cases:
+ child_result = self.add_test_case(test_case_method.__name__)
+ child_result.update_setup(Result.BLOCK)
class TestCaseResult(BaseResult, FixtureResult):
@@ -449,7 +529,7 @@ class TestCaseResult(BaseResult, FixtureResult):
test_case_name: str
def __init__(self, test_case_name: str):
- """Extend the constructor with `test_case_name`.
+ """Extend the constructor with test case's name and TestSuiteResult.
Args:
test_case_name: The test case's name.
@@ -470,7 +550,7 @@ def update(self, result: Result, error: Exception | None = None) -> None:
self.result = result
self.error = error
- def _get_inner_errors(self) -> list[Exception]:
+ def _get_child_errors(self) -> list[Exception]:
if self.error:
return [self.error]
return []
@@ -486,6 +566,10 @@ def add_stats(self, statistics: "Statistics") -> None:
"""
statistics += self.result
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ self.update(Result.BLOCK)
+
def __bool__(self) -> bool:
"""The test case passed only if setup, teardown and the test case itself passed."""
return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 6/7] dts: refactor logging configuration
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
` (4 preceding siblings ...)
2024-02-23 7:55 ` [PATCH v3 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš
@ 2024-02-23 7:55 ` Juraj Linkeš
2024-02-23 7:55 ` [PATCH v3 7/7] dts: improve test suite and case filtering Juraj Linkeš
2024-02-27 21:24 ` [PATCH v3 0/7] test case blocking and logging Jeremy Spewock
7 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-23 7:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
Remove unused parts of the code and add useful features:
1. Add DTS execution stages such as execution and test suite to better
identify where in the DTS lifecycle we are when investigating logs,
2. Logging to separate files in specific stages, which is mainly useful
for having test suite logs in additional separate files.
3. Remove the dependence on the settings module which enhances the
usefulness of the logger module, as it can now be imported in more
modules.
The execution stages and the files to log to are the same for all DTS
loggers. To achieve this, we have one DTS root logger which should be
used for handling stage switching and all other loggers are children of
this DTS root logger. The DTS root logger is the one where we change the
behavior of all loggers (the stage and which files to log to) and the
child loggers just log messages under a different name.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/logger.py | 236 +++++++++++-------
dts/framework/remote_session/__init__.py | 6 +-
.../interactive_remote_session.py | 6 +-
.../remote_session/interactive_shell.py | 6 +-
.../remote_session/remote_session.py | 8 +-
dts/framework/runner.py | 19 +-
dts/framework/test_result.py | 6 +-
dts/framework/test_suite.py | 6 +-
dts/framework/testbed_model/node.py | 11 +-
dts/framework/testbed_model/os_session.py | 7 +-
.../traffic_generator/traffic_generator.py | 6 +-
dts/main.py | 3 -
12 files changed, 184 insertions(+), 136 deletions(-)
diff --git a/dts/framework/logger.py b/dts/framework/logger.py
index cfa6e8cd72..db4a7698e0 100644
--- a/dts/framework/logger.py
+++ b/dts/framework/logger.py
@@ -5,141 +5,187 @@
"""DTS logger module.
-DTS framework and TestSuite logs are saved in different log files.
+The module provides several additional features:
+
+ * The storage of DTS execution stages,
+ * Logging to console, a human-readable log file and a machine-readable log file,
+ * Optional log files for specific stages.
"""
import logging
-import os.path
-from typing import TypedDict
+from enum import auto
+from logging import FileHandler, StreamHandler
+from pathlib import Path
+from typing import ClassVar
-from .settings import SETTINGS
+from .utils import StrEnum
date_fmt = "%Y/%m/%d %H:%M:%S"
-stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s"
+dts_root_logger_name = "dts"
+
+
+class DtsStage(StrEnum):
+ """The DTS execution stage."""
+ #:
+ pre_execution = auto()
+ #:
+ execution = auto()
+ #:
+ build_target = auto()
+ #:
+ suite = auto()
+ #:
+ post_execution = auto()
-class DTSLOG(logging.LoggerAdapter):
- """DTS logger adapter class for framework and testsuites.
- The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment
- variable control the verbosity of output. If enabled, all messages will be emitted to the
- console.
+class DTSLogger(logging.Logger):
+ """The DTS logger class.
- The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment
- variable modify the directory where the logs will be stored.
+ The class extends the :class:`~logging.Logger` class to add the DTS execution stage information
+ to log records. The stage is common to all loggers, so it's stored in a class variable.
- Attributes:
- node: The additional identifier. Currently unused.
- sh: The handler which emits logs to console.
- fh: The handler which emits logs to a file.
- verbose_fh: Just as fh, but logs with a different, more verbose, format.
+ Any time we switch to a new stage, we have the ability to log to an additional log file along
+ with a supplementary log file with machine-readable format. These two log files are used until
+ a new stage switch occurs. This is useful mainly for logging per test suite.
"""
- _logger: logging.Logger
- node: str
- sh: logging.StreamHandler
- fh: logging.FileHandler
- verbose_fh: logging.FileHandler
+ _stage: ClassVar[DtsStage] = DtsStage.pre_execution
+ _extra_file_handlers: list[FileHandler] = []
- def __init__(self, logger: logging.Logger, node: str = "suite"):
- """Extend the constructor with additional handlers.
+ def __init__(self, *args, **kwargs):
+ """Extend the constructor with extra file handlers."""
+ self._extra_file_handlers = []
+ super().__init__(*args, **kwargs)
- One handler logs to the console, the other one to a file, with either a regular or verbose
- format.
+ def makeRecord(self, *args, **kwargs) -> logging.LogRecord:
+ """Generates a record with additional stage information.
- Args:
- logger: The logger from which to create the logger adapter.
- node: An additional identifier. Currently unused.
+ This is the default method for the :class:`~logging.Logger` class. We extend it
+ to add stage information to the record.
+
+ :meta private:
+
+ Returns:
+ record: The generated record with the stage information.
"""
- self._logger = logger
- # 1 means log everything, this will be used by file handlers if their level
- # is not set
- self._logger.setLevel(1)
+ record = super().makeRecord(*args, **kwargs)
+ record.stage = DTSLogger._stage # type: ignore[attr-defined]
+ return record
+
+ def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None:
+ """Add logger handlers to the DTS root logger.
+
+ This method should be called only on the DTS root logger.
+ The log records from child loggers will propagate to these handlers.
+
+ Three handlers are added:
- self.node = node
+ * A console handler,
+ * A file handler,
+ * A supplementary file handler with machine-readable logs
+ containing more debug information.
- # add handler to emit to stdout
- sh = logging.StreamHandler()
+ All log messages will be logged to files. The log level of the console handler
+ is configurable with `verbose`.
+
+ Args:
+ verbose: If :data:`True`, log all messages to the console.
+ If :data:`False`, log to console with the :data:`logging.INFO` level.
+ output_dir: The directory where the log files will be located.
+ The names of the log files correspond to the name of the logger instance.
+ """
+ self.setLevel(1)
+
+ sh = StreamHandler()
sh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
- sh.setLevel(logging.INFO) # console handler default level
+ if not verbose:
+ sh.setLevel(logging.INFO)
+ self.addHandler(sh)
- if SETTINGS.verbose is True:
- sh.setLevel(logging.DEBUG)
+ self._add_file_handlers(Path(output_dir, self.name))
- self._logger.addHandler(sh)
- self.sh = sh
+ def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None:
+ """Set the DTS execution stage and optionally log to files.
- # prepare the output folder
- if not os.path.exists(SETTINGS.output_dir):
- os.mkdir(SETTINGS.output_dir)
+ Set the DTS execution stage of the DTSLog class and optionally add
+ file handlers to the instance if the log file name is provided.
- logging_path_prefix = os.path.join(SETTINGS.output_dir, node)
+ The file handlers log all messages. One is a regular human-readable log file and
+ the other one is a machine-readable log file with extra debug information.
- fh = logging.FileHandler(f"{logging_path_prefix}.log")
- fh.setFormatter(
- logging.Formatter(
- fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
- datefmt=date_fmt,
- )
- )
+ Args:
+ stage: The DTS stage to set.
+ log_file_path: An optional path of the log file to use. This should be a full path
+ (either relative or absolute) without suffix (which will be appended).
+ """
+ self._remove_extra_file_handlers()
+
+ if DTSLogger._stage != stage:
+ self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.")
+ DTSLogger._stage = stage
+
+ if log_file_path:
+ self._extra_file_handlers.extend(self._add_file_handlers(log_file_path))
+
+ def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]:
+ """Add file handlers to the DTS root logger.
+
+ Add two type of file handlers:
+
+ * A regular file handler with suffix ".log",
+ * A machine-readable file handler with suffix ".verbose.log".
+ This format provides extensive information for debugging and detailed analysis.
+
+ Args:
+ log_file_path: The full path to the log file without suffix.
+
+ Returns:
+ The newly created file handlers.
- self._logger.addHandler(fh)
- self.fh = fh
+ """
+ fh = FileHandler(f"{log_file_path}.log")
+ fh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
+ self.addHandler(fh)
- # This outputs EVERYTHING, intended for post-mortem debugging
- # Also optimized for processing via AWK (awk -F '|' ...)
- verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log")
+ verbose_fh = FileHandler(f"{log_file_path}.verbose.log")
verbose_fh.setFormatter(
logging.Formatter(
- fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
+ "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
"%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s",
datefmt=date_fmt,
)
)
+ self.addHandler(verbose_fh)
- self._logger.addHandler(verbose_fh)
- self.verbose_fh = verbose_fh
-
- super(DTSLOG, self).__init__(self._logger, dict(node=self.node))
-
- def logger_exit(self) -> None:
- """Remove the stream handler and the logfile handler."""
- for handler in (self.sh, self.fh, self.verbose_fh):
- handler.flush()
- self._logger.removeHandler(handler)
-
-
-class _LoggerDictType(TypedDict):
- logger: DTSLOG
- name: str
- node: str
-
+ return [fh, verbose_fh]
-# List for saving all loggers in use
-_Loggers: list[_LoggerDictType] = []
+ def _remove_extra_file_handlers(self) -> None:
+ """Remove any extra file handlers that have been added to the logger."""
+ if self._extra_file_handlers:
+ for extra_file_handler in self._extra_file_handlers:
+ self.removeHandler(extra_file_handler)
+ self._extra_file_handlers = []
-def getLogger(name: str, node: str = "suite") -> DTSLOG:
- """Get DTS logger adapter identified by name and node.
- An existing logger will be returned if one with the exact name and node already exists.
- A new one will be created and stored otherwise.
+def get_dts_logger(name: str = None) -> DTSLogger:
+ """Return a DTS logger instance identified by `name`.
Args:
- name: The name of the logger.
- node: An additional identifier for the logger.
+ name: If :data:`None`, return the DTS root logger.
+ If specified, return a child of the DTS root logger.
Returns:
- A logger uniquely identified by both name and node.
+ The DTS root logger or a child logger identified by `name`.
"""
- global _Loggers
- # return saved logger
- logger: _LoggerDictType
- for logger in _Loggers:
- if logger["name"] == name and logger["node"] == node:
- return logger["logger"]
-
- # return new logger
- dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node)
- _Loggers.append({"logger": dts_logger, "name": name, "node": node})
- return dts_logger
+ original_logger_class = logging.getLoggerClass()
+ logging.setLoggerClass(DTSLogger)
+ if name:
+ name = f"{dts_root_logger_name}.{name}"
+ else:
+ name = dts_root_logger_name
+ logger = logging.getLogger(name)
+ logging.setLoggerClass(original_logger_class)
+ return logger # type: ignore[return-value]
diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
index 51a01d6b5e..1910c81c3c 100644
--- a/dts/framework/remote_session/__init__.py
+++ b/dts/framework/remote_session/__init__.py
@@ -15,7 +15,7 @@
# pylama:ignore=W0611
from framework.config import NodeConfiguration
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from .interactive_remote_session import InteractiveRemoteSession
from .interactive_shell import InteractiveShell
@@ -26,7 +26,7 @@
def create_remote_session(
- node_config: NodeConfiguration, name: str, logger: DTSLOG
+ node_config: NodeConfiguration, name: str, logger: DTSLogger
) -> RemoteSession:
"""Factory for non-interactive remote sessions.
@@ -45,7 +45,7 @@ def create_remote_session(
def create_interactive_session(
- node_config: NodeConfiguration, logger: DTSLOG
+ node_config: NodeConfiguration, logger: DTSLogger
) -> InteractiveRemoteSession:
"""Factory for interactive remote sessions.
diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py
index 1cc82e3377..c50790db79 100644
--- a/dts/framework/remote_session/interactive_remote_session.py
+++ b/dts/framework/remote_session/interactive_remote_session.py
@@ -16,7 +16,7 @@
from framework.config import NodeConfiguration
from framework.exception import SSHConnectionError
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
class InteractiveRemoteSession:
@@ -50,11 +50,11 @@ class InteractiveRemoteSession:
username: str
password: str
session: SSHClient
- _logger: DTSLOG
+ _logger: DTSLogger
_node_config: NodeConfiguration
_transport: Transport | None
- def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None:
+ def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None:
"""Connect to the node during initialization.
Args:
diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py
index b158f963b6..5cfe202e15 100644
--- a/dts/framework/remote_session/interactive_shell.py
+++ b/dts/framework/remote_session/interactive_shell.py
@@ -20,7 +20,7 @@
from paramiko import Channel, SSHClient, channel # type: ignore[import]
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from framework.settings import SETTINGS
@@ -38,7 +38,7 @@ class InteractiveShell(ABC):
_stdin: channel.ChannelStdinFile
_stdout: channel.ChannelFile
_ssh_channel: Channel
- _logger: DTSLOG
+ _logger: DTSLogger
_timeout: float
_app_args: str
@@ -61,7 +61,7 @@ class InteractiveShell(ABC):
def __init__(
self,
interactive_session: SSHClient,
- logger: DTSLOG,
+ logger: DTSLogger,
get_privileged_command: Callable[[str], str] | None,
app_args: str = "",
timeout: float = SETTINGS.timeout,
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
index 2059f9a981..a69dc99400 100644
--- a/dts/framework/remote_session/remote_session.py
+++ b/dts/framework/remote_session/remote_session.py
@@ -9,14 +9,13 @@
the structure of the result of a command execution.
"""
-
import dataclasses
from abc import ABC, abstractmethod
from pathlib import PurePath
from framework.config import NodeConfiguration
from framework.exception import RemoteCommandExecutionError
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from framework.settings import SETTINGS
@@ -75,14 +74,14 @@ class RemoteSession(ABC):
username: str
password: str
history: list[CommandResult]
- _logger: DTSLOG
+ _logger: DTSLogger
_node_config: NodeConfiguration
def __init__(
self,
node_config: NodeConfiguration,
session_name: str,
- logger: DTSLOG,
+ logger: DTSLogger,
):
"""Connect to the node during initialization.
@@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None:
Args:
force: Force the closure of the connection. This may not clean up all resources.
"""
- self._logger.logger_exit()
self._close(force)
@abstractmethod
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 511f8be064..af927b11a9 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -19,9 +19,10 @@
import importlib
import inspect
-import logging
+import os
import re
import sys
+from pathlib import Path
from types import MethodType
from typing import Iterable
@@ -38,7 +39,7 @@
SSHTimeoutError,
TestCaseVerifyError,
)
-from .logger import DTSLOG, getLogger
+from .logger import DTSLogger, DtsStage, get_dts_logger
from .settings import SETTINGS
from .test_result import (
BuildTargetResult,
@@ -73,7 +74,7 @@ class DTSRunner:
"""
_configuration: Configuration
- _logger: DTSLOG
+ _logger: DTSLogger
_result: DTSResult
_test_suite_class_prefix: str
_test_suite_module_prefix: str
@@ -83,7 +84,10 @@ class DTSRunner:
def __init__(self):
"""Initialize the instance with configuration, logger, result and string constants."""
self._configuration = load_config()
- self._logger = getLogger("DTSRunner")
+ self._logger = get_dts_logger()
+ if not os.path.exists(SETTINGS.output_dir):
+ os.makedirs(SETTINGS.output_dir)
+ self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir)
self._result = DTSResult(self._logger)
self._test_suite_class_prefix = "Test"
self._test_suite_module_prefix = "tests.TestSuite_"
@@ -137,6 +141,7 @@ def run(self):
# for all Execution sections
for execution in self._configuration.executions:
+ self._logger.set_stage(DtsStage.execution)
self._logger.info(
f"Running execution with SUT '{execution.system_under_test_node.name}'."
)
@@ -168,6 +173,7 @@ def run(self):
finally:
try:
+ self._logger.set_stage(DtsStage.post_execution)
for node in (sut_nodes | tg_nodes).values():
node.close()
self._result.update_teardown(Result.PASS)
@@ -425,6 +431,7 @@ def _run_execution(
finally:
try:
+ self._logger.set_stage(DtsStage.execution)
sut_node.tear_down_execution()
execution_result.update_teardown(Result.PASS)
except Exception as e:
@@ -453,6 +460,7 @@ def _run_build_target(
with the current build target.
test_suites_with_cases: The test suites with test cases to run.
"""
+ self._logger.set_stage(DtsStage.build_target)
self._logger.info(f"Running build target '{build_target.name}'.")
try:
@@ -469,6 +477,7 @@ def _run_build_target(
finally:
try:
+ self._logger.set_stage(DtsStage.build_target)
sut_node.tear_down_build_target()
build_target_result.update_teardown(Result.PASS)
except Exception as e:
@@ -541,6 +550,7 @@ def _run_test_suite(
BlockingTestSuiteError: If a blocking test suite fails.
"""
test_suite_name = test_suite_with_cases.test_suite_class.__name__
+ self._logger.set_stage(DtsStage.suite, Path(SETTINGS.output_dir, test_suite_name))
test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
try:
self._logger.info(f"Starting test suite setup: {test_suite_name}")
@@ -689,5 +699,4 @@ def _exit_dts(self) -> None:
if self._logger:
self._logger.info("DTS execution has ended.")
- logging.shutdown()
sys.exit(self._result.get_return_code())
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index eedb2d20ee..28f84fd793 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -42,7 +42,7 @@
TestSuiteConfig,
)
from .exception import DTSError, ErrorSeverity
-from .logger import DTSLOG
+from .logger import DTSLogger
from .settings import SETTINGS
from .test_suite import TestSuite
@@ -237,13 +237,13 @@ class DTSResult(BaseResult):
"""
dpdk_version: str | None
- _logger: DTSLOG
+ _logger: DTSLogger
_errors: list[Exception]
_return_code: ErrorSeverity
_stats_result: Union["Statistics", None]
_stats_filename: str
- def __init__(self, logger: DTSLOG):
+ def __init__(self, logger: DTSLogger):
"""Extend the constructor with top-level specifics.
Args:
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index f9fe88093e..365f80e21a 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -21,7 +21,7 @@
from scapy.packet import Packet, Padding # type: ignore[import]
from .exception import TestCaseVerifyError
-from .logger import DTSLOG, getLogger
+from .logger import DTSLogger, get_dts_logger
from .testbed_model import Port, PortLink, SutNode, TGNode
from .utils import get_packet_summaries
@@ -61,7 +61,7 @@ class TestSuite(object):
#: Whether the test suite is blocking. A failure of a blocking test suite
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
- _logger: DTSLOG
+ _logger: DTSLogger
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -88,7 +88,7 @@ def __init__(
"""
self.sut_node = sut_node
self.tg_node = tg_node
- self._logger = getLogger(self.__class__.__name__)
+ self._logger = get_dts_logger(self.__class__.__name__)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
index 1a55fadf78..74061f6262 100644
--- a/dts/framework/testbed_model/node.py
+++ b/dts/framework/testbed_model/node.py
@@ -23,7 +23,7 @@
NodeConfiguration,
)
from framework.exception import ConfigurationError
-from framework.logger import DTSLOG, getLogger
+from framework.logger import DTSLogger, get_dts_logger
from framework.settings import SETTINGS
from .cpu import (
@@ -63,7 +63,7 @@ class Node(ABC):
name: str
lcores: list[LogicalCore]
ports: list[Port]
- _logger: DTSLOG
+ _logger: DTSLogger
_other_sessions: list[OSSession]
_execution_config: ExecutionConfiguration
virtual_devices: list[VirtualDevice]
@@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration):
"""
self.config = node_config
self.name = node_config.name
- self._logger = getLogger(self.name)
+ self._logger = get_dts_logger(self.name)
self.main_session = create_session(self.config, self.name, self._logger)
self._logger.info(f"Connected to node: {self.name}")
@@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession:
connection = create_session(
self.config,
session_name,
- getLogger(session_name, node=self.name),
+ get_dts_logger(session_name),
)
self._other_sessions.append(connection)
return connection
@@ -299,7 +299,6 @@ def close(self) -> None:
self.main_session.close()
for session in self._other_sessions:
session.close()
- self._logger.logger_exit()
@staticmethod
def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
@@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
return func
-def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession:
+def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession:
"""Factory for OS-aware sessions.
Args:
diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
index ac6bb5e112..6983aa4a77 100644
--- a/dts/framework/testbed_model/os_session.py
+++ b/dts/framework/testbed_model/os_session.py
@@ -21,7 +21,6 @@
the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux
and other commands for other OSs. It also translates the path to match the underlying OS.
"""
-
from abc import ABC, abstractmethod
from collections.abc import Iterable
from ipaddress import IPv4Interface, IPv6Interface
@@ -29,7 +28,7 @@
from typing import Type, TypeVar, Union
from framework.config import Architecture, NodeConfiguration, NodeInfo
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from framework.remote_session import (
CommandResult,
InteractiveRemoteSession,
@@ -62,7 +61,7 @@ class OSSession(ABC):
_config: NodeConfiguration
name: str
- _logger: DTSLOG
+ _logger: DTSLogger
remote_session: RemoteSession
interactive_session: InteractiveRemoteSession
@@ -70,7 +69,7 @@ def __init__(
self,
node_config: NodeConfiguration,
name: str,
- logger: DTSLOG,
+ logger: DTSLogger,
):
"""Initialize the OS-aware session.
diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
index c49fbff488..d86d7fb532 100644
--- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py
+++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
@@ -13,7 +13,7 @@
from scapy.packet import Packet # type: ignore[import]
from framework.config import TrafficGeneratorConfig
-from framework.logger import DTSLOG, getLogger
+from framework.logger import DTSLogger, get_dts_logger
from framework.testbed_model.node import Node
from framework.testbed_model.port import Port
from framework.utils import get_packet_summaries
@@ -28,7 +28,7 @@ class TrafficGenerator(ABC):
_config: TrafficGeneratorConfig
_tg_node: Node
- _logger: DTSLOG
+ _logger: DTSLogger
def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
"""Initialize the traffic generator.
@@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
"""
self._config = config
self._tg_node = tg_node
- self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
+ self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
def send_packet(self, packet: Packet, port: Port) -> None:
"""Send `packet` and block until it is fully sent.
diff --git a/dts/main.py b/dts/main.py
index 1ffe8ff81f..fa878cc16e 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -6,8 +6,6 @@
"""The DTS executable."""
-import logging
-
from framework import settings
@@ -30,5 +28,4 @@ def main() -> None:
# Main program begins here
if __name__ == "__main__":
- logging.raiseExceptions = True
main()
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 7/7] dts: improve test suite and case filtering
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
` (5 preceding siblings ...)
2024-02-23 7:55 ` [PATCH v3 6/7] dts: refactor logging configuration Juraj Linkeš
@ 2024-02-23 7:55 ` Juraj Linkeš
2024-02-27 21:24 ` [PATCH v3 0/7] test case blocking and logging Jeremy Spewock
7 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-23 7:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro
Cc: dev, Juraj Linkeš
The two places where we specify which test suite and test cases to run
are complimentary and not that intuitive to use. A unified way provides
a better user experience.
The syntax in test run configuration file has not changed, but the
environment variable and the command line arguments was changed to match
the config file syntax. This required changes in the settings module
which greatly simplified the parsing of the environment variables while
retaining the same functionality.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
doc/guides/tools/dts.rst | 14 ++-
dts/framework/config/__init__.py | 12 +-
dts/framework/runner.py | 18 +--
dts/framework/settings.py | 187 ++++++++++++++-----------------
dts/framework/test_suite.py | 2 +-
5 files changed, 114 insertions(+), 119 deletions(-)
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index f686ca487c..d1c3c2af7a 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,28 +215,30 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
.. code-block:: console
(dts-py3.10) $ ./main.py --help
- usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN]
+ usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
options:
-h, --help show this help message and exit
--config-file CONFIG_FILE
- [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
+ [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
--output-dir OUTPUT_DIR, --output OUTPUT_DIR
[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
-t TIMEOUT, --timeout TIMEOUT
[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
- -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None)
+ -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
--tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
--compile-timeout COMPILE_TIMEOUT
[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
- --test-cases TEST_CASES
- [DTS_TESTCASES] Comma-separated list of test cases to execute. Unknown test cases will be silently ignored. (default: )
+ --test-suite TEST_SUITE [TEST_CASES ...]
+ [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
+ variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
+ (default: [])
--re-run RE_RUN, --re_run RE_RUN
- [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs (default: 0)
+ [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index c6a93b3b89..4cb5c74059 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -35,9 +35,9 @@
import json
import os.path
-import pathlib
from dataclasses import dataclass, fields
from enum import auto, unique
+from pathlib import Path
from typing import Union
import warlock # type: ignore[import]
@@ -53,7 +53,6 @@
TrafficGeneratorConfigDict,
)
from framework.exception import ConfigurationError
-from framework.settings import SETTINGS
from framework.utils import StrEnum
@@ -571,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration":
return Configuration(executions=executions)
-def load_config() -> Configuration:
+def load_config(config_file_path: Path) -> Configuration:
"""Load DTS test run configuration from a file.
Load the YAML test run configuration file
@@ -581,13 +580,16 @@ def load_config() -> Configuration:
The YAML test run configuration file is specified in the :option:`--config-file` command line
argument or the :envvar:`DTS_CFG_FILE` environment variable.
+ Args:
+ config_file_path: The path to the YAML test run configuration file.
+
Returns:
The parsed test run configuration.
"""
- with open(SETTINGS.config_file_path, "r") as f:
+ with open(config_file_path, "r") as f:
config_data = yaml.safe_load(f)
- schema_path = os.path.join(pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json")
+ schema_path = os.path.join(Path(__file__).parent.resolve(), "conf_yaml_schema.json")
with open(schema_path, "r") as f:
schema = json.load(f)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index af927b11a9..cabff0a7b2 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -24,7 +24,7 @@
import sys
from pathlib import Path
from types import MethodType
-from typing import Iterable
+from typing import Iterable, Sequence
from .config import (
BuildTargetConfiguration,
@@ -83,7 +83,7 @@ class DTSRunner:
def __init__(self):
"""Initialize the instance with configuration, logger, result and string constants."""
- self._configuration = load_config()
+ self._configuration = load_config(SETTINGS.config_file_path)
self._logger = get_dts_logger()
if not os.path.exists(SETTINGS.output_dir):
os.makedirs(SETTINGS.output_dir)
@@ -129,7 +129,7 @@ def run(self):
#. Execution teardown
The test cases are filtered according to the specification in the test run configuration and
- the :option:`--test-cases` command line argument or
+ the :option:`--test-suite` command line argument or
the :envvar:`DTS_TESTCASES` environment variable.
"""
sut_nodes: dict[str, SutNode] = {}
@@ -147,7 +147,9 @@ def run(self):
)
execution_result = self._result.add_execution(execution)
# we don't want to modify the original config, so create a copy
- execution_test_suites = list(execution.test_suites)
+ execution_test_suites = list(
+ SETTINGS.test_suites if SETTINGS.test_suites else execution.test_suites
+ )
if not execution.skip_smoke_tests:
execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
try:
@@ -226,7 +228,7 @@ def _get_test_suites_with_cases(
test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
test_cases = []
func_test_cases, perf_test_cases = self._filter_test_cases(
- test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases)
+ test_suite_class, test_suite_config.test_cases
)
if func:
test_cases.extend(func_test_cases)
@@ -301,7 +303,7 @@ def is_test_suite(object) -> bool:
)
def _filter_test_cases(
- self, test_suite_class: type[TestSuite], test_cases_to_run: set[str]
+ self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
) -> tuple[list[MethodType], list[MethodType]]:
"""Filter `test_cases_to_run` from `test_suite_class`.
@@ -330,7 +332,9 @@ def _filter_test_cases(
(name, method) for name, method in name_method_tuples if name in test_cases_to_run
]
if len(name_method_tuples) < len(test_cases_to_run):
- missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples}
+ missing_test_cases = set(test_cases_to_run) - {
+ name for name, _ in name_method_tuples
+ }
raise ConfigurationError(
f"Test cases {missing_test_cases} not found among methods "
f"of {test_suite_class.__name__}."
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 2b8bfbe0ed..688e8679a7 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -48,10 +48,11 @@
The path to a DPDK tarball, git commit ID, tag ID or tree ID to test.
-.. option:: --test-cases
-.. envvar:: DTS_TESTCASES
+.. option:: --test-suite
+.. envvar:: DTS_TEST_SUITES
- A comma-separated list of test cases to execute. Unknown test cases will be silently ignored.
+ A test suite with test cases which may be specified multiple times.
+ In the environment variable, the suites are joined with a comma.
.. option:: --re-run, --re_run
.. envvar:: DTS_RERUN
@@ -71,83 +72,13 @@
import argparse
import os
-from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass, field
from pathlib import Path
-from typing import Any, TypeVar
+from typing import Any
+from .config import TestSuiteConfig
from .utils import DPDKGitTarball
-_T = TypeVar("_T")
-
-
-def _env_arg(env_var: str) -> Any:
- """A helper method augmenting the argparse Action with environment variables.
-
- If the supplied environment variable is defined, then the default value
- of the argument is modified. This satisfies the priority order of
- command line argument > environment variable > default value.
-
- Arguments with no values (flags) should be defined using the const keyword argument
- (True or False). When the argument is specified, it will be set to const, if not specified,
- the default will be stored (possibly modified by the corresponding environment variable).
-
- Other arguments work the same as default argparse arguments, that is using
- the default 'store' action.
-
- Returns:
- The modified argparse.Action.
- """
-
- class _EnvironmentArgument(argparse.Action):
- def __init__(
- self,
- option_strings: Sequence[str],
- dest: str,
- nargs: str | int | None = None,
- const: bool | None = None,
- default: Any = None,
- type: Callable[[str], _T | argparse.FileType | None] = None,
- choices: Iterable[_T] | None = None,
- required: bool = False,
- help: str | None = None,
- metavar: str | tuple[str, ...] | None = None,
- ) -> None:
- env_var_value = os.environ.get(env_var)
- default = env_var_value or default
- if const is not None:
- nargs = 0
- default = const if env_var_value else default
- type = None
- choices = None
- metavar = None
- super(_EnvironmentArgument, self).__init__(
- option_strings,
- dest,
- nargs=nargs,
- const=const,
- default=default,
- type=type,
- choices=choices,
- required=required,
- help=help,
- metavar=metavar,
- )
-
- def __call__(
- self,
- parser: argparse.ArgumentParser,
- namespace: argparse.Namespace,
- values: Any,
- option_string: str = None,
- ) -> None:
- if self.const is not None:
- setattr(namespace, self.dest, self.const)
- else:
- setattr(namespace, self.dest, values)
-
- return _EnvironmentArgument
-
@dataclass(slots=True)
class Settings:
@@ -171,7 +102,7 @@ class Settings:
#:
compile_timeout: float = 1200
#:
- test_cases: list[str] = field(default_factory=list)
+ test_suites: list[TestSuiteConfig] = field(default_factory=list)
#:
re_run: int = 0
@@ -180,6 +111,31 @@ class Settings:
def _get_parser() -> argparse.ArgumentParser:
+ """Create the argument parser for DTS.
+
+ Command line options take precedence over environment variables, which in turn take precedence
+ over default values.
+
+ Returns:
+ argparse.ArgumentParser: The configured argument parser with defined options.
+ """
+
+ def env_arg(env_var: str, default: Any) -> Any:
+ """A helper function augmenting the argparse with environment variables.
+
+ If the supplied environment variable is defined, then the default value
+ of the argument is modified. This satisfies the priority order of
+ command line argument > environment variable > default value.
+
+ Args:
+ env_var: Environment variable name.
+ default: Default value.
+
+ Returns:
+ Environment variable or default value.
+ """
+ return os.environ.get(env_var) or default
+
parser = argparse.ArgumentParser(
description="Run DPDK test suites. All options may be specified with the environment "
"variables provided in brackets. Command line arguments have higher priority.",
@@ -188,25 +144,23 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--config-file",
- action=_env_arg("DTS_CFG_FILE"),
- default=SETTINGS.config_file_path,
+ default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
type=Path,
- help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets.",
+ help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
+ "SUTs and targets.",
)
parser.add_argument(
"--output-dir",
"--output",
- action=_env_arg("DTS_OUTPUT_DIR"),
- default=SETTINGS.output_dir,
+ default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
)
parser.add_argument(
"-t",
"--timeout",
- action=_env_arg("DTS_TIMEOUT"),
- default=SETTINGS.timeout,
+ default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
type=float,
help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
)
@@ -214,9 +168,8 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"-v",
"--verbose",
- action=_env_arg("DTS_VERBOSE"),
- default=SETTINGS.verbose,
- const=True,
+ action="store_true",
+ default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
"to the console.",
)
@@ -224,8 +177,8 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"-s",
"--skip-setup",
- action=_env_arg("DTS_SKIP_SETUP"),
- const=True,
+ action="store_true",
+ default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
)
@@ -233,8 +186,7 @@ def _get_parser() -> argparse.ArgumentParser:
"--tarball",
"--snapshot",
"--git-ref",
- action=_env_arg("DTS_DPDK_TARBALL"),
- default=SETTINGS.dpdk_tarball_path,
+ default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
type=Path,
help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
"tag ID or tree ID to test. To test local changes, first commit them, "
@@ -243,36 +195,71 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--compile-timeout",
- action=_env_arg("DTS_COMPILE_TIMEOUT"),
- default=SETTINGS.compile_timeout,
+ default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
type=float,
help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
)
parser.add_argument(
- "--test-cases",
- action=_env_arg("DTS_TESTCASES"),
- default="",
- help="[DTS_TESTCASES] Comma-separated list of test cases to execute.",
+ "--test-suite",
+ action="append",
+ nargs="+",
+ metavar=("TEST_SUITE", "TEST_CASES"),
+ default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
+ help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
+ "The first parameter is the test suite name, and the rest are test case names, "
+ "which are optional. May be specified multiple times. To specify multiple test suites in "
+ "the environment variable, join the lists with a comma. "
+ "Examples: "
+ "--test-suite suite case case --test-suite suite case ... | "
+ "DTS_TEST_SUITES='suite case case, suite case, ...' | "
+ "--test-suite suite --test-suite suite case ... | "
+ "DTS_TEST_SUITES='suite, suite case, ...'",
)
parser.add_argument(
"--re-run",
"--re_run",
- action=_env_arg("DTS_RERUN"),
- default=SETTINGS.re_run,
+ default=env_arg("DTS_RERUN", SETTINGS.re_run),
type=int,
help="[DTS_RERUN] Re-run each test case the specified number of times "
- "if a test failure occurs",
+ "if a test failure occurs.",
)
return parser
+def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
+ """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
+
+ Args:
+ args: The arguments to process. The args is a string from an environment variable
+ or a list of from the user input containing tests suites with tests cases,
+ each of which is a list of [test_suite, test_case, test_case, ...].
+
+ Returns:
+ A list of test suite configurations to execute.
+ """
+ if isinstance(args, str):
+ # Environment variable in the form of "suite case case, suite case, suite, ..."
+ args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
+
+ test_suites_to_run = []
+ for suite_with_cases in args:
+ test_suites_to_run.append(
+ TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
+ )
+
+ return test_suites_to_run
+
+
def get_settings() -> Settings:
"""Create new settings with inputs from the user.
The inputs are taken from the command line and from environment variables.
+
+ Returns:
+ The new settings object.
"""
parsed_args = _get_parser().parse_args()
return Settings(
@@ -287,6 +274,6 @@ def get_settings() -> Settings:
else Path(parsed_args.tarball)
),
compile_timeout=parsed_args.compile_timeout,
- test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []),
+ test_suites=_process_test_suites(parsed_args.test_suite),
re_run=parsed_args.re_run,
)
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 365f80e21a..1957ea7328 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -40,7 +40,7 @@ class TestSuite(object):
and functional test cases (all other test cases).
By default, all test cases will be executed. A list of testcase names may be specified
- in the YAML test run configuration file and in the :option:`--test-cases` command line argument
+ in the YAML test run configuration file and in the :option:`--test-suite` command line argument
or in the :envvar:`DTS_TESTCASES` environment variable to filter which test cases to run.
The union of both lists will be used. Any unknown test cases from the latter lists
will be silently ignored.
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v3 3/7] dts: filter test suites in executions
2024-02-23 7:54 ` [PATCH v3 3/7] dts: filter test suites in executions Juraj Linkeš
@ 2024-02-27 21:21 ` Jeremy Spewock
2024-02-28 9:16 ` Juraj Linkeš
0 siblings, 1 reply; 44+ messages in thread
From: Jeremy Spewock @ 2024-02-27 21:21 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev
<snip>
> +
> + lowercase_suite_to_find = (
> + f"{self._test_suite_class_prefix}{module_name.replace('_', '')}".lower()
> + )
> + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite):
> + if (
> + class_name.startswith(self._test_suite_class_prefix)
Is this line required? When we check later if the
lowercase_suite_to_find is equal, we know that this variable we made
starts with the designated prefix because that's just how we made it.
It seems redundant because they could not be equal if the class name
didn't start with the prefix. Is the idea behind this a type of
optimization because this makes us ignore anything that doesn't have
the proper prefix before going further?
>
> + and lowercase_suite_to_find == class_name.lower()
> + ):
> + return class_obj
> + raise ConfigurationError(
> + f"Couldn't find any valid test suites in {test_suite_module.__name__}."
> + )
<snip>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v3 0/7] test case blocking and logging
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
` (6 preceding siblings ...)
2024-02-23 7:55 ` [PATCH v3 7/7] dts: improve test suite and case filtering Juraj Linkeš
@ 2024-02-27 21:24 ` Jeremy Spewock
7 siblings, 0 replies; 44+ messages in thread
From: Jeremy Spewock @ 2024-02-27 21:24 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev
I just had a very small comment which has nothing to do with
functionality and is really just about something being checked twice
which does no harm. Outside of that however, the whole series looked
good to me.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v3 3/7] dts: filter test suites in executions
2024-02-27 21:21 ` Jeremy Spewock
@ 2024-02-28 9:16 ` Juraj Linkeš
0 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-02-28 9:16 UTC (permalink / raw)
To: Jeremy Spewock
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev
On Tue, Feb 27, 2024 at 10:22 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
> <snip>
> > +
> > + lowercase_suite_to_find = (
> > + f"{self._test_suite_class_prefix}{module_name.replace('_', '')}".lower()
> > + )
> > + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite):
> > + if (
> > + class_name.startswith(self._test_suite_class_prefix)
>
> Is this line required? When we check later if the
> lowercase_suite_to_find is equal, we know that this variable we made
> starts with the designated prefix because that's just how we made it.
> It seems redundant because they could not be equal if the class name
> didn't start with the prefix. Is the idea behind this a type of
> optimization because this makes us ignore anything that doesn't have
> the proper prefix before going further?
>
There is a slight difference. The
class_name.startswith(self._test_suite_class_prefix) condition checks
uppercase characters as well. I'm not sure it's actually worth it.
Maybe we could change the name check to also check for the CamelCase
convention. I'll look into that.
> >
> > + and lowercase_suite_to_find == class_name.lower()
> > + ):
> > + return class_obj
> > + raise ConfigurationError(
> > + f"Couldn't find any valid test suites in {test_suite_module.__name__}."
> > + )
> <snip>
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 0/7] test case blocking and logging
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
` (7 preceding siblings ...)
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
@ 2024-03-01 10:55 ` Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 1/7] dts: convert dts.py methods to class Juraj Linkeš
` (8 more replies)
8 siblings, 9 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
We currently don't record test case results that couldn't be executed
because of a previous failure, such as when a test suite setup failed,
resulting in no executed test cases.
In order to record the test cases that couldn't be executed, we must
know the lists of test suites and test cases ahead of the actual test
suite execution, as an error could occur before we even start executing
test suites.
In addition, the patch series contains two refactors and one
improvement.
The first refactor is closely related. The dts.py was renamed to
runner.py and given a clear purpose - running the test suites and all
other orchestration needed to run test suites. The logic for this was
not all in the original dts.py module and it was brought there. The
runner is also responsible for recording results, which is the blocked
test cases are recorded.
The other refactor, logging, is related to the first refactor. The
logging module was simplified while extending capabilities. Each test
suite logs into its own log file in addition to the main log file which
the runner must handle (as it knows when we start executing particular
test suites). The runner also handles the switching between execution
stages for the purposes of logging.
The one aforementioned improvement is in unifying how we specify test
cases in the conf.yaml file and in the environment variable/command line
argument.
v2:
Rebase and update of the whole patch. There are changes in all parts of
the code, mainly improving the design and logic.
Also added the last patch which improves test suite specification on the
cmdline.
v3:
Improved variables in _get_test_suite_class along with docstring.
Fixed smoke test suite not being added into the list of test suites to
be executed.
v4:
Added the checking of CamelCase convention when discovering test cases.
Also added additional test stages. The stages were split into setup and
teardown.
Juraj Linkeš (7):
dts: convert dts.py methods to class
dts: move test suite execution logic to DTSRunner
dts: filter test suites in executions
dts: reorganize test result
dts: block all test cases when earlier setup fails
dts: refactor logging configuration
dts: improve test suite and case filtering
doc/guides/tools/dts.rst | 14 +-
dts/framework/config/__init__.py | 36 +-
dts/framework/config/conf_yaml_schema.json | 2 +-
dts/framework/dts.py | 338 ---------
dts/framework/logger.py | 246 +++---
dts/framework/remote_session/__init__.py | 6 +-
.../interactive_remote_session.py | 6 +-
.../remote_session/interactive_shell.py | 6 +-
.../remote_session/remote_session.py | 8 +-
dts/framework/runner.py | 711 ++++++++++++++++++
dts/framework/settings.py | 188 +++--
dts/framework/test_result.py | 565 ++++++++------
dts/framework/test_suite.py | 239 +-----
dts/framework/testbed_model/node.py | 11 +-
dts/framework/testbed_model/os_session.py | 7 +-
.../traffic_generator/traffic_generator.py | 6 +-
dts/main.py | 9 +-
dts/pyproject.toml | 3 +
dts/tests/TestSuite_os_udp.py | 2 +-
dts/tests/TestSuite_smoke_tests.py | 2 +-
20 files changed, 1375 insertions(+), 1030 deletions(-)
delete mode 100644 dts/framework/dts.py
create mode 100644 dts/framework/runner.py
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 1/7] dts: convert dts.py methods to class
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
@ 2024-03-01 10:55 ` Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš
` (7 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
The dts.py module deviates from the rest of the code without a clear
reason. Converting it into a class and using better naming will improve
organization and code readability.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/dts.py | 338 ----------------------------------------
dts/framework/runner.py | 333 +++++++++++++++++++++++++++++++++++++++
dts/main.py | 6 +-
3 files changed, 337 insertions(+), 340 deletions(-)
delete mode 100644 dts/framework/dts.py
create mode 100644 dts/framework/runner.py
diff --git a/dts/framework/dts.py b/dts/framework/dts.py
deleted file mode 100644
index e16d4578a0..0000000000
--- a/dts/framework/dts.py
+++ /dev/null
@@ -1,338 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2019 Intel Corporation
-# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
-# Copyright(c) 2022-2023 University of New Hampshire
-
-r"""Test suite runner module.
-
-A DTS run is split into stages:
-
- #. Execution stage,
- #. Build target stage,
- #. Test suite stage,
- #. Test case stage.
-
-The module is responsible for running tests on testbeds defined in the test run configuration.
-Each setup or teardown of each stage is recorded in a :class:`~.test_result.DTSResult` or
-one of its subclasses. The test case results are also recorded.
-
-If an error occurs, the current stage is aborted, the error is recorded and the run continues in
-the next iteration of the same stage. The return code is the highest `severity` of all
-:class:`~.exception.DTSError`\s.
-
-Example:
- An error occurs in a build target setup. The current build target is aborted and the run
- continues with the next build target. If the errored build target was the last one in the given
- execution, the next execution begins.
-
-Attributes:
- dts_logger: The logger instance used in this module.
- result: The top level result used in the module.
-"""
-
-import sys
-
-from .config import (
- BuildTargetConfiguration,
- ExecutionConfiguration,
- TestSuiteConfig,
- load_config,
-)
-from .exception import BlockingTestSuiteError
-from .logger import DTSLOG, getLogger
-from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
-from .test_suite import get_test_suites
-from .testbed_model import SutNode, TGNode
-
-# dummy defaults to satisfy linters
-dts_logger: DTSLOG = None # type: ignore[assignment]
-result: DTSResult = DTSResult(dts_logger)
-
-
-def run_all() -> None:
- """Run all build targets in all executions from the test run configuration.
-
- Before running test suites, executions and build targets are first set up.
- The executions and build targets defined in the test run configuration are iterated over.
- The executions define which tests to run and where to run them and build targets define
- the DPDK build setup.
-
- The tests suites are set up for each execution/build target tuple and each scheduled
- test case within the test suite is set up, executed and torn down. After all test cases
- have been executed, the test suite is torn down and the next build target will be tested.
-
- All the nested steps look like this:
-
- #. Execution setup
-
- #. Build target setup
-
- #. Test suite setup
-
- #. Test case setup
- #. Test case logic
- #. Test case teardown
-
- #. Test suite teardown
-
- #. Build target teardown
-
- #. Execution teardown
-
- The test cases are filtered according to the specification in the test run configuration and
- the :option:`--test-cases` command line argument or
- the :envvar:`DTS_TESTCASES` environment variable.
- """
- global dts_logger
- global result
-
- # create a regular DTS logger and create a new result with it
- dts_logger = getLogger("DTSRunner")
- result = DTSResult(dts_logger)
-
- # check the python version of the server that run dts
- _check_dts_python_version()
-
- sut_nodes: dict[str, SutNode] = {}
- tg_nodes: dict[str, TGNode] = {}
- try:
- # for all Execution sections
- for execution in load_config().executions:
- sut_node = sut_nodes.get(execution.system_under_test_node.name)
- tg_node = tg_nodes.get(execution.traffic_generator_node.name)
-
- try:
- if not sut_node:
- sut_node = SutNode(execution.system_under_test_node)
- sut_nodes[sut_node.name] = sut_node
- if not tg_node:
- tg_node = TGNode(execution.traffic_generator_node)
- tg_nodes[tg_node.name] = tg_node
- result.update_setup(Result.PASS)
- except Exception as e:
- failed_node = execution.system_under_test_node.name
- if sut_node:
- failed_node = execution.traffic_generator_node.name
- dts_logger.exception(f"Creation of node {failed_node} failed.")
- result.update_setup(Result.FAIL, e)
-
- else:
- _run_execution(sut_node, tg_node, execution, result)
-
- except Exception as e:
- dts_logger.exception("An unexpected error has occurred.")
- result.add_error(e)
- raise
-
- finally:
- try:
- for node in (sut_nodes | tg_nodes).values():
- node.close()
- result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Final cleanup of nodes failed.")
- result.update_teardown(Result.ERROR, e)
-
- # we need to put the sys.exit call outside the finally clause to make sure
- # that unexpected exceptions will propagate
- # in that case, the error that should be reported is the uncaught exception as
- # that is a severe error originating from the framework
- # at that point, we'll only have partial results which could be impacted by the
- # error causing the uncaught exception, making them uninterpretable
- _exit_dts()
-
-
-def _check_dts_python_version() -> None:
- """Check the required Python version - v3.10."""
-
- def RED(text: str) -> str:
- return f"\u001B[31;1m{str(text)}\u001B[0m"
-
- if sys.version_info.major < 3 or (sys.version_info.major == 3 and sys.version_info.minor < 10):
- print(
- RED(
- (
- "WARNING: DTS execution node's python version is lower than"
- "python 3.10, is deprecated and will not work in future releases."
- )
- ),
- file=sys.stderr,
- )
- print(RED("Please use Python >= 3.10 instead"), file=sys.stderr)
-
-
-def _run_execution(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- result: DTSResult,
-) -> None:
- """Run the given execution.
-
- This involves running the execution setup as well as running all build targets
- in the given execution. After that, execution teardown is run.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- execution: An execution's test run configuration.
- result: The top level result object.
- """
- dts_logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
- execution_result = result.add_execution(sut_node.config)
- execution_result.add_sut_info(sut_node.node_info)
-
- try:
- sut_node.set_up_execution(execution)
- execution_result.update_setup(Result.PASS)
- except Exception as e:
- dts_logger.exception("Execution setup failed.")
- execution_result.update_setup(Result.FAIL, e)
-
- else:
- for build_target in execution.build_targets:
- _run_build_target(sut_node, tg_node, build_target, execution, execution_result)
-
- finally:
- try:
- sut_node.tear_down_execution()
- execution_result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Execution teardown failed.")
- execution_result.update_teardown(Result.FAIL, e)
-
-
-def _run_build_target(
- sut_node: SutNode,
- tg_node: TGNode,
- build_target: BuildTargetConfiguration,
- execution: ExecutionConfiguration,
- execution_result: ExecutionResult,
-) -> None:
- """Run the given build target.
-
- This involves running the build target setup as well as running all test suites
- in the given execution the build target is defined in.
- After that, build target teardown is run.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- build_target: A build target's test run configuration.
- execution: The build target's execution's test run configuration.
- execution_result: The execution level result object associated with the execution.
- """
- dts_logger.info(f"Running build target '{build_target.name}'.")
- build_target_result = execution_result.add_build_target(build_target)
-
- try:
- sut_node.set_up_build_target(build_target)
- result.dpdk_version = sut_node.dpdk_version
- build_target_result.add_build_target_info(sut_node.get_build_target_info())
- build_target_result.update_setup(Result.PASS)
- except Exception as e:
- dts_logger.exception("Build target setup failed.")
- build_target_result.update_setup(Result.FAIL, e)
-
- else:
- _run_all_suites(sut_node, tg_node, execution, build_target_result)
-
- finally:
- try:
- sut_node.tear_down_build_target()
- build_target_result.update_teardown(Result.PASS)
- except Exception as e:
- dts_logger.exception("Build target teardown failed.")
- build_target_result.update_teardown(Result.FAIL, e)
-
-
-def _run_all_suites(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
-) -> None:
- """Run the execution's (possibly a subset) test suites using the current build target.
-
- The function assumes the build target we're testing has already been built on the SUT node.
- The current build target thus corresponds to the current DPDK build present on the SUT node.
-
- If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
- in the current build target won't be executed.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- execution: The execution's test run configuration associated with the current build target.
- build_target_result: The build target level result object associated
- with the current build target.
- """
- end_build_target = False
- if not execution.skip_smoke_tests:
- execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
- for test_suite_config in execution.test_suites:
- try:
- _run_single_suite(sut_node, tg_node, execution, build_target_result, test_suite_config)
- except BlockingTestSuiteError as e:
- dts_logger.exception(
- f"An error occurred within {test_suite_config.test_suite}. Skipping build target."
- )
- result.add_error(e)
- end_build_target = True
- # if a blocking test failed and we need to bail out of suite executions
- if end_build_target:
- break
-
-
-def _run_single_suite(
- sut_node: SutNode,
- tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
- test_suite_config: TestSuiteConfig,
-) -> None:
- """Run all test suite in a single test suite module.
-
- The function assumes the build target we're testing has already been built on the SUT node.
- The current build target thus corresponds to the current DPDK build present on the SUT node.
-
- Args:
- sut_node: The execution's SUT node.
- tg_node: The execution's TG node.
- execution: The execution's test run configuration associated with the current build target.
- build_target_result: The build target level result object associated
- with the current build target.
- test_suite_config: Test suite test run configuration specifying the test suite module
- and possibly a subset of test cases of test suites in that module.
-
- Raises:
- BlockingTestSuiteError: If a blocking test suite fails.
- """
- try:
- full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
- test_suite_classes = get_test_suites(full_suite_path)
- suites_str = ", ".join((x.__name__ for x in test_suite_classes))
- dts_logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
- except Exception as e:
- dts_logger.exception("An error occurred when searching for test suites.")
- result.update_setup(Result.ERROR, e)
-
- else:
- for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(
- sut_node,
- tg_node,
- test_suite_config.test_cases,
- execution.func,
- build_target_result,
- )
- test_suite.run()
-
-
-def _exit_dts() -> None:
- """Process all errors and exit with the proper exit code."""
- result.process()
-
- if dts_logger:
- dts_logger.info("DTS execution has ended.")
- sys.exit(result.get_return_code())
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
new file mode 100644
index 0000000000..acc1c4d6db
--- /dev/null
+++ b/dts/framework/runner.py
@@ -0,0 +1,333 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2019 Intel Corporation
+# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
+# Copyright(c) 2022-2023 University of New Hampshire
+
+"""Test suite runner module.
+
+The module is responsible for running DTS in a series of stages:
+
+ #. Execution stage,
+ #. Build target stage,
+ #. Test suite stage,
+ #. Test case stage.
+
+The execution and build target stages set up the environment before running test suites.
+The test suite stage sets up steps common to all test cases
+and the test case stage runs test cases individually.
+"""
+
+import logging
+import sys
+
+from .config import (
+ BuildTargetConfiguration,
+ ExecutionConfiguration,
+ TestSuiteConfig,
+ load_config,
+)
+from .exception import BlockingTestSuiteError
+from .logger import DTSLOG, getLogger
+from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
+from .test_suite import get_test_suites
+from .testbed_model import SutNode, TGNode
+
+
+class DTSRunner:
+ r"""Test suite runner class.
+
+ The class is responsible for running tests on testbeds defined in the test run configuration.
+ Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult`
+ or one of its subclasses. The test case results are also recorded.
+
+ If an error occurs, the current stage is aborted, the error is recorded and the run continues in
+ the next iteration of the same stage. The return code is the highest `severity` of all
+ :class:`~.framework.exception.DTSError`\s.
+
+ Example:
+ An error occurs in a build target setup. The current build target is aborted and the run
+ continues with the next build target. If the errored build target was the last one in the
+ given execution, the next execution begins.
+ """
+
+ _logger: DTSLOG
+ _result: DTSResult
+
+ def __init__(self):
+ """Initialize the instance with logger and result."""
+ self._logger = getLogger("DTSRunner")
+ self._result = DTSResult(self._logger)
+
+ def run(self):
+ """Run all build targets in all executions from the test run configuration.
+
+ Before running test suites, executions and build targets are first set up.
+ The executions and build targets defined in the test run configuration are iterated over.
+ The executions define which tests to run and where to run them and build targets define
+ the DPDK build setup.
+
+ The tests suites are set up for each execution/build target tuple and each discovered
+ test case within the test suite is set up, executed and torn down. After all test cases
+ have been executed, the test suite is torn down and the next build target will be tested.
+
+ All the nested steps look like this:
+
+ #. Execution setup
+
+ #. Build target setup
+
+ #. Test suite setup
+
+ #. Test case setup
+ #. Test case logic
+ #. Test case teardown
+
+ #. Test suite teardown
+
+ #. Build target teardown
+
+ #. Execution teardown
+
+ The test cases are filtered according to the specification in the test run configuration and
+ the :option:`--test-cases` command line argument or
+ the :envvar:`DTS_TESTCASES` environment variable.
+ """
+ sut_nodes: dict[str, SutNode] = {}
+ tg_nodes: dict[str, TGNode] = {}
+ try:
+ # check the python version of the server that runs dts
+ self._check_dts_python_version()
+
+ # for all Execution sections
+ for execution in load_config().executions:
+ sut_node = sut_nodes.get(execution.system_under_test_node.name)
+ tg_node = tg_nodes.get(execution.traffic_generator_node.name)
+
+ try:
+ if not sut_node:
+ sut_node = SutNode(execution.system_under_test_node)
+ sut_nodes[sut_node.name] = sut_node
+ if not tg_node:
+ tg_node = TGNode(execution.traffic_generator_node)
+ tg_nodes[tg_node.name] = tg_node
+ self._result.update_setup(Result.PASS)
+ except Exception as e:
+ failed_node = execution.system_under_test_node.name
+ if sut_node:
+ failed_node = execution.traffic_generator_node.name
+ self._logger.exception(f"The Creation of node {failed_node} failed.")
+ self._result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_execution(sut_node, tg_node, execution)
+
+ except Exception as e:
+ self._logger.exception("An unexpected error has occurred.")
+ self._result.add_error(e)
+ raise
+
+ finally:
+ try:
+ for node in (sut_nodes | tg_nodes).values():
+ node.close()
+ self._result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("The final cleanup of nodes failed.")
+ self._result.update_teardown(Result.ERROR, e)
+
+ # we need to put the sys.exit call outside the finally clause to make sure
+ # that unexpected exceptions will propagate
+ # in that case, the error that should be reported is the uncaught exception as
+ # that is a severe error originating from the framework
+ # at that point, we'll only have partial results which could be impacted by the
+ # error causing the uncaught exception, making them uninterpretable
+ self._exit_dts()
+
+ def _check_dts_python_version(self) -> None:
+ """Check the required Python version - v3.10."""
+ if sys.version_info.major < 3 or (
+ sys.version_info.major == 3 and sys.version_info.minor < 10
+ ):
+ self._logger.warning(
+ "DTS execution node's python version is lower than Python 3.10, "
+ "is deprecated and will not work in future releases."
+ )
+ self._logger.warning("Please use Python >= 3.10 instead.")
+
+ def _run_execution(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ ) -> None:
+ """Run the given execution.
+
+ This involves running the execution setup as well as running all build targets
+ in the given execution. After that, execution teardown is run.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: An execution's test run configuration.
+ """
+ self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
+ execution_result = self._result.add_execution(sut_node.config)
+ execution_result.add_sut_info(sut_node.node_info)
+
+ try:
+ sut_node.set_up_execution(execution)
+ execution_result.update_setup(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Execution setup failed.")
+ execution_result.update_setup(Result.FAIL, e)
+
+ else:
+ for build_target in execution.build_targets:
+ self._run_build_target(sut_node, tg_node, build_target, execution, execution_result)
+
+ finally:
+ try:
+ sut_node.tear_down_execution()
+ execution_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Execution teardown failed.")
+ execution_result.update_teardown(Result.FAIL, e)
+
+ def _run_build_target(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ build_target: BuildTargetConfiguration,
+ execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ ) -> None:
+ """Run the given build target.
+
+ This involves running the build target setup as well as running all test suites
+ of the build target's execution.
+ After that, build target teardown is run.
+
+ Args:
+ sut_node: The execution's sut node.
+ tg_node: The execution's tg node.
+ build_target: A build target's test run configuration.
+ execution: The build target's execution's test run configuration.
+ execution_result: The execution level result object associated with the execution.
+ """
+ self._logger.info(f"Running build target '{build_target.name}'.")
+ build_target_result = execution_result.add_build_target(build_target)
+
+ try:
+ sut_node.set_up_build_target(build_target)
+ self._result.dpdk_version = sut_node.dpdk_version
+ build_target_result.add_build_target_info(sut_node.get_build_target_info())
+ build_target_result.update_setup(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Build target setup failed.")
+ build_target_result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_all_suites(sut_node, tg_node, execution, build_target_result)
+
+ finally:
+ try:
+ sut_node.tear_down_build_target()
+ build_target_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception("Build target teardown failed.")
+ build_target_result.update_teardown(Result.FAIL, e)
+
+ def _run_all_suites(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ build_target_result: BuildTargetResult,
+ ) -> None:
+ """Run the execution's (possibly a subset of) test suites using the current build target.
+
+ The method assumes the build target we're testing has already been built on the SUT node.
+ The current build target thus corresponds to the current DPDK build present on the SUT node.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: The execution's test run configuration associated
+ with the current build target.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ """
+ end_build_target = False
+ if not execution.skip_smoke_tests:
+ execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
+ for test_suite_config in execution.test_suites:
+ try:
+ self._run_single_suite(
+ sut_node, tg_node, execution, build_target_result, test_suite_config
+ )
+ except BlockingTestSuiteError as e:
+ self._logger.exception(
+ f"An error occurred within {test_suite_config.test_suite}. "
+ "Skipping build target..."
+ )
+ self._result.add_error(e)
+ end_build_target = True
+ # if a blocking test failed and we need to bail out of suite executions
+ if end_build_target:
+ break
+
+ def _run_single_suite(
+ self,
+ sut_node: SutNode,
+ tg_node: TGNode,
+ execution: ExecutionConfiguration,
+ build_target_result: BuildTargetResult,
+ test_suite_config: TestSuiteConfig,
+ ) -> None:
+ """Run all test suites in a single test suite module.
+
+ The method assumes the build target we're testing has already been built on the SUT node.
+ The current build target thus corresponds to the current DPDK build present on the SUT node.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: The execution's test run configuration associated
+ with the current build target.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ test_suite_config: Test suite test run configuration specifying the test suite module
+ and possibly a subset of test cases of test suites in that module.
+
+ Raises:
+ BlockingTestSuiteError: If a blocking test suite fails.
+ """
+ try:
+ full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
+ test_suite_classes = get_test_suites(full_suite_path)
+ suites_str = ", ".join((x.__name__ for x in test_suite_classes))
+ self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
+ except Exception as e:
+ self._logger.exception("An error occurred when searching for test suites.")
+ self._result.update_setup(Result.ERROR, e)
+
+ else:
+ for test_suite_class in test_suite_classes:
+ test_suite = test_suite_class(
+ sut_node,
+ tg_node,
+ test_suite_config.test_cases,
+ execution.func,
+ build_target_result,
+ )
+ test_suite.run()
+
+ def _exit_dts(self) -> None:
+ """Process all errors and exit with the proper exit code."""
+ self._result.process()
+
+ if self._logger:
+ self._logger.info("DTS execution has ended.")
+
+ logging.shutdown()
+ sys.exit(self._result.get_return_code())
diff --git a/dts/main.py b/dts/main.py
index f703615d11..1ffe8ff81f 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -21,9 +21,11 @@ def main() -> None:
be modified before the settings module is imported anywhere else in the framework.
"""
settings.SETTINGS = settings.get_settings()
- from framework import dts
- dts.run_all()
+ from framework.runner import DTSRunner
+
+ dts = DTSRunner()
+ dts.run()
# Main program begins here
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 2/7] dts: move test suite execution logic to DTSRunner
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 1/7] dts: convert dts.py methods to class Juraj Linkeš
@ 2024-03-01 10:55 ` Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 3/7] dts: filter test suites in executions Juraj Linkeš
` (6 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
Move the code responsible for running the test suite from the
TestSuite class to the DTSRunner class. This restructuring decision
was made to consolidate and unify the related logic into a single unit.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 175 ++++++++++++++++++++++++++++++++----
dts/framework/test_suite.py | 152 ++-----------------------------
2 files changed, 169 insertions(+), 158 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index acc1c4d6db..933685d638 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -19,6 +19,7 @@
import logging
import sys
+from types import MethodType
from .config import (
BuildTargetConfiguration,
@@ -26,10 +27,18 @@
TestSuiteConfig,
load_config,
)
-from .exception import BlockingTestSuiteError
+from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
from .logger import DTSLOG, getLogger
-from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result
-from .test_suite import get_test_suites
+from .settings import SETTINGS
+from .test_result import (
+ BuildTargetResult,
+ DTSResult,
+ ExecutionResult,
+ Result,
+ TestCaseResult,
+ TestSuiteResult,
+)
+from .test_suite import TestSuite, get_test_suites
from .testbed_model import SutNode, TGNode
@@ -227,7 +236,7 @@ def _run_build_target(
build_target_result.update_setup(Result.FAIL, e)
else:
- self._run_all_suites(sut_node, tg_node, execution, build_target_result)
+ self._run_test_suites(sut_node, tg_node, execution, build_target_result)
finally:
try:
@@ -237,7 +246,7 @@ def _run_build_target(
self._logger.exception("Build target teardown failed.")
build_target_result.update_teardown(Result.FAIL, e)
- def _run_all_suites(
+ def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
@@ -249,6 +258,9 @@ def _run_all_suites(
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
+ in the current build target won't be executed.
+
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
@@ -262,7 +274,7 @@ def _run_all_suites(
execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
for test_suite_config in execution.test_suites:
try:
- self._run_single_suite(
+ self._run_test_suite_module(
sut_node, tg_node, execution, build_target_result, test_suite_config
)
except BlockingTestSuiteError as e:
@@ -276,7 +288,7 @@ def _run_all_suites(
if end_build_target:
break
- def _run_single_suite(
+ def _run_test_suite_module(
self,
sut_node: SutNode,
tg_node: TGNode,
@@ -284,11 +296,18 @@ def _run_single_suite(
build_target_result: BuildTargetResult,
test_suite_config: TestSuiteConfig,
) -> None:
- """Run all test suites in a single test suite module.
+ """Set up, execute and tear down all test suites in a single test suite module.
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
+ Test suite execution consists of running the discovered test cases.
+ A test case run consists of setup, execution and teardown of said test case.
+
+ Record the setup and the teardown and handle failures.
+
+ The test cases to execute are discovered when creating the :class:`TestSuite` object.
+
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
@@ -313,14 +332,140 @@ def _run_single_suite(
else:
for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(
- sut_node,
- tg_node,
- test_suite_config.test_cases,
- execution.func,
- build_target_result,
+ test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases)
+
+ test_suite_name = test_suite.__class__.__name__
+ test_suite_result = build_target_result.add_test_suite(test_suite_name)
+ try:
+ self._logger.info(f"Starting test suite setup: {test_suite_name}")
+ test_suite.set_up_suite()
+ test_suite_result.update_setup(Result.PASS)
+ self._logger.info(f"Test suite setup successful: {test_suite_name}")
+ except Exception as e:
+ self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
+ test_suite_result.update_setup(Result.ERROR, e)
+
+ else:
+ self._execute_test_suite(execution.func, test_suite, test_suite_result)
+
+ finally:
+ try:
+ test_suite.tear_down_suite()
+ sut_node.kill_cleanup_dpdk_apps()
+ test_suite_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
+ self._logger.warning(
+ f"Test suite '{test_suite_name}' teardown failed, "
+ f"the next test suite may be affected."
+ )
+ test_suite_result.update_setup(Result.ERROR, e)
+ if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
+ raise BlockingTestSuiteError(test_suite_name)
+
+ def _execute_test_suite(
+ self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
+ ) -> None:
+ """Execute all discovered test cases in `test_suite`.
+
+ If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment
+ variable is set, in case of a test case failure, the test case will be executed again
+ until it passes or it fails that many times in addition of the first failure.
+
+ Args:
+ func: Whether to execute functional test cases.
+ test_suite: The test suite object.
+ test_suite_result: The test suite level result object associated
+ with the current test suite.
+ """
+ if func:
+ for test_case_method in test_suite._get_functional_test_cases():
+ test_case_name = test_case_method.__name__
+ test_case_result = test_suite_result.add_test_case(test_case_name)
+ all_attempts = SETTINGS.re_run + 1
+ attempt_nr = 1
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+
+ def _run_test_case(
+ self,
+ test_suite: TestSuite,
+ test_case_method: MethodType,
+ test_case_result: TestCaseResult,
+ ) -> None:
+ """Setup, execute and teardown a test case in `test_suite`.
+
+ Record the result of the setup and the teardown and handle failures.
+
+ Args:
+ test_suite: The test suite object.
+ test_case_method: The test case method.
+ test_case_result: The test case level result object associated
+ with the current test case.
+ """
+ test_case_name = test_case_method.__name__
+
+ try:
+ # run set_up function for each case
+ test_suite.set_up_test_case()
+ test_case_result.update_setup(Result.PASS)
+ except SSHTimeoutError as e:
+ self._logger.exception(f"Test case setup FAILED: {test_case_name}")
+ test_case_result.update_setup(Result.FAIL, e)
+ except Exception as e:
+ self._logger.exception(f"Test case setup ERROR: {test_case_name}")
+ test_case_result.update_setup(Result.ERROR, e)
+
+ else:
+ # run test case if setup was successful
+ self._execute_test_case(test_case_method, test_case_result)
+
+ finally:
+ try:
+ test_suite.tear_down_test_case()
+ test_case_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test case teardown ERROR: {test_case_name}")
+ self._logger.warning(
+ f"Test case '{test_case_name}' teardown failed, "
+ f"the next test case may be affected."
)
- test_suite.run()
+ test_case_result.update_teardown(Result.ERROR, e)
+ test_case_result.update(Result.ERROR)
+
+ def _execute_test_case(
+ self, test_case_method: MethodType, test_case_result: TestCaseResult
+ ) -> None:
+ """Execute one test case, record the result and handle failures.
+
+ Args:
+ test_case_method: The test case method.
+ test_case_result: The test case level result object associated
+ with the current test case.
+ """
+ test_case_name = test_case_method.__name__
+ try:
+ self._logger.info(f"Starting test case execution: {test_case_name}")
+ test_case_method()
+ test_case_result.update(Result.PASS)
+ self._logger.info(f"Test case execution PASSED: {test_case_name}")
+
+ except TestCaseVerifyError as e:
+ self._logger.exception(f"Test case execution FAILED: {test_case_name}")
+ test_case_result.update(Result.FAIL, e)
+ except Exception as e:
+ self._logger.exception(f"Test case execution ERROR: {test_case_name}")
+ test_case_result.update(Result.ERROR, e)
+ except KeyboardInterrupt:
+ self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}")
+ test_case_result.update(Result.SKIP)
+ raise KeyboardInterrupt("Stop DTS")
def _exit_dts(self) -> None:
"""Process all errors and exit with the proper exit code."""
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index dfb391ffbd..b02fd36147 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -8,7 +8,6 @@
must be extended by subclasses which add test cases. The :class:`TestSuite` contains the basics
needed by subclasses:
- * Test suite and test case execution flow,
* Testbed (SUT, TG) configuration,
* Packet sending and verification,
* Test case verification.
@@ -28,27 +27,22 @@
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
-from .exception import (
- BlockingTestSuiteError,
- ConfigurationError,
- SSHTimeoutError,
- TestCaseVerifyError,
-)
+from .exception import ConfigurationError, TestCaseVerifyError
from .logger import DTSLOG, getLogger
from .settings import SETTINGS
-from .test_result import BuildTargetResult, Result, TestCaseResult, TestSuiteResult
from .testbed_model import Port, PortLink, SutNode, TGNode
from .utils import get_packet_summaries
class TestSuite(object):
- """The base class with methods for handling the basic flow of a test suite.
+ """The base class with building blocks needed by most test cases.
* Test case filtering and collection,
- * Test suite setup/cleanup,
- * Test setup/cleanup,
- * Test case execution,
- * Error handling and results storage.
+ * Test suite setup/cleanup methods to override,
+ * Test case setup/cleanup methods to override,
+ * Test case verification,
+ * Testbed configuration,
+ * Traffic sending and verification.
Test cases are implemented by subclasses. Test cases are all methods starting with ``test_``,
further divided into performance test cases (starting with ``test_perf_``)
@@ -60,10 +54,6 @@ class TestSuite(object):
The union of both lists will be used. Any unknown test cases from the latter lists
will be silently ignored.
- If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable
- is set, in case of a test case failure, the test case will be executed again until it passes
- or it fails that many times in addition of the first failure.
-
The methods named ``[set_up|tear_down]_[suite|test_case]`` should be overridden in subclasses
if the appropriate test suite/test case fixtures are needed.
@@ -82,8 +72,6 @@ class TestSuite(object):
is_blocking: ClassVar[bool] = False
_logger: DTSLOG
_test_cases_to_run: list[str]
- _func: bool
- _result: TestSuiteResult
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -99,30 +87,23 @@ def __init__(
sut_node: SutNode,
tg_node: TGNode,
test_cases: list[str],
- func: bool,
- build_target_result: BuildTargetResult,
):
"""Initialize the test suite testbed information and basic configuration.
- Process what test cases to run, create the associated
- :class:`~.test_result.TestSuiteResult`, find links between ports
- and set up default IP addresses to be used when configuring them.
+ Process what test cases to run, find links between ports and set up
+ default IP addresses to be used when configuring them.
Args:
sut_node: The SUT node where the test suite will run.
tg_node: The TG node where the test suite will run.
test_cases: The list of test cases to execute.
If empty, all test cases will be executed.
- func: Whether to run functional tests.
- build_target_result: The build target result this test suite is run in.
"""
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = getLogger(self.__class__.__name__)
self._test_cases_to_run = test_cases
self._test_cases_to_run.extend(SETTINGS.test_cases)
- self._func = func
- self._result = build_target_result.add_test_suite(self.__class__.__name__)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
@@ -384,62 +365,6 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
return False
return True
- def run(self) -> None:
- """Set up, execute and tear down the whole suite.
-
- Test suite execution consists of running all test cases scheduled to be executed.
- A test case run consists of setup, execution and teardown of said test case.
-
- Record the setup and the teardown and handle failures.
-
- The list of scheduled test cases is constructed when creating the :class:`TestSuite` object.
- """
- test_suite_name = self.__class__.__name__
-
- try:
- self._logger.info(f"Starting test suite setup: {test_suite_name}")
- self.set_up_suite()
- self._result.update_setup(Result.PASS)
- self._logger.info(f"Test suite setup successful: {test_suite_name}")
- except Exception as e:
- self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
- self._result.update_setup(Result.ERROR, e)
-
- else:
- self._execute_test_suite()
-
- finally:
- try:
- self.tear_down_suite()
- self.sut_node.kill_cleanup_dpdk_apps()
- self._result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
- self._logger.warning(
- f"Test suite '{test_suite_name}' teardown failed, "
- f"the next test suite may be affected."
- )
- self._result.update_setup(Result.ERROR, e)
- if len(self._result.get_errors()) > 0 and self.is_blocking:
- raise BlockingTestSuiteError(test_suite_name)
-
- def _execute_test_suite(self) -> None:
- """Execute all test cases scheduled to be executed in this suite."""
- if self._func:
- for test_case_method in self._get_functional_test_cases():
- test_case_name = test_case_method.__name__
- test_case_result = self._result.add_test_case(test_case_name)
- all_attempts = SETTINGS.re_run + 1
- attempt_nr = 1
- self._run_test_case(test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
- self._run_test_case(test_case_method, test_case_result)
-
def _get_functional_test_cases(self) -> list[MethodType]:
"""Get all functional test cases defined in this TestSuite.
@@ -471,65 +396,6 @@ def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool
return match
- def _run_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
- ) -> None:
- """Setup, execute and teardown a test case in this suite.
-
- Record the result of the setup and the teardown and handle failures.
- """
- test_case_name = test_case_method.__name__
-
- try:
- # run set_up function for each case
- self.set_up_test_case()
- test_case_result.update_setup(Result.PASS)
- except SSHTimeoutError as e:
- self._logger.exception(f"Test case setup FAILED: {test_case_name}")
- test_case_result.update_setup(Result.FAIL, e)
- except Exception as e:
- self._logger.exception(f"Test case setup ERROR: {test_case_name}")
- test_case_result.update_setup(Result.ERROR, e)
-
- else:
- # run test case if setup was successful
- self._execute_test_case(test_case_method, test_case_result)
-
- finally:
- try:
- self.tear_down_test_case()
- test_case_result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test case teardown ERROR: {test_case_name}")
- self._logger.warning(
- f"Test case '{test_case_name}' teardown failed, "
- f"the next test case may be affected."
- )
- test_case_result.update_teardown(Result.ERROR, e)
- test_case_result.update(Result.ERROR)
-
- def _execute_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
- ) -> None:
- """Execute one test case, record the result and handle failures."""
- test_case_name = test_case_method.__name__
- try:
- self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method()
- test_case_result.update(Result.PASS)
- self._logger.info(f"Test case execution PASSED: {test_case_name}")
-
- except TestCaseVerifyError as e:
- self._logger.exception(f"Test case execution FAILED: {test_case_name}")
- test_case_result.update(Result.FAIL, e)
- except Exception as e:
- self._logger.exception(f"Test case execution ERROR: {test_case_name}")
- test_case_result.update(Result.ERROR, e)
- except KeyboardInterrupt:
- self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}")
- test_case_result.update(Result.SKIP)
- raise KeyboardInterrupt("Stop DTS")
-
def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
r"""Find all :class:`TestSuite`\s in a Python module.
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 3/7] dts: filter test suites in executions
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 1/7] dts: convert dts.py methods to class Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš
@ 2024-03-01 10:55 ` Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 4/7] dts: reorganize test result Juraj Linkeš
` (5 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
We're currently filtering which test cases to run after some setup
steps, such as DPDK build, have already been taken. This prohibits us to
mark the test suites and cases that were supposed to be run as blocked
when an earlier setup fails, as that information is not available at
that time.
To remedy this, move the filtering to the beginning of each execution.
This is the first action taken in each execution and if we can't filter
the test cases, such as due to invalid inputs, we abort the whole
execution. No test suites nor cases will be marked as blocked as we
don't know which were supposed to be run.
On top of that, the filtering takes place in the TestSuite class, which
should only concern itself with test suite and test case logic, not the
processing behind the scenes. The logic has been moved to DTSRunner
which should do all the processing needed to run test suites.
The filtering itself introduces a few changes/assumptions which are more
sensible than before:
1. Assumption: There is just one TestSuite child class in each test
suite module. This was an implicit assumption before as we couldn't
specify the TestSuite classes in the test run configuration, just the
modules. The name of the TestSuite child class starts with "Test" and
then corresponds to the name of the module with CamelCase naming.
2. Unknown test cases specified both in the test run configuration and
the environment variable/command line argument are no longer silently
ignored. This is a quality of life improvement for users, as they
could easily be not aware of the silent ignoration.
Also, a change in the code results in pycodestyle warning and error:
[E] E203 whitespace before ':'
[W] W503 line break before binary operator
These two are not PEP8 compliant, so they're disabled.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/config/__init__.py | 24 +-
dts/framework/config/conf_yaml_schema.json | 2 +-
dts/framework/runner.py | 433 +++++++++++++++------
dts/framework/settings.py | 3 +-
dts/framework/test_result.py | 34 ++
dts/framework/test_suite.py | 85 +---
dts/pyproject.toml | 3 +
dts/tests/TestSuite_os_udp.py | 2 +-
dts/tests/TestSuite_smoke_tests.py | 2 +-
9 files changed, 390 insertions(+), 198 deletions(-)
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 62eded7f04..c6a93b3b89 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -36,7 +36,7 @@
import json
import os.path
import pathlib
-from dataclasses import dataclass
+from dataclasses import dataclass, fields
from enum import auto, unique
from typing import Union
@@ -506,6 +506,28 @@ def from_dict(
vdevs=vdevs,
)
+ def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration":
+ """Create a shallow copy with any of the fields modified.
+
+ The only new data are those passed to this method.
+ The rest are copied from the object's fields calling the method.
+
+ Args:
+ **kwargs: The names and types of keyword arguments are defined
+ by the fields of the :class:`ExecutionConfiguration` class.
+
+ Returns:
+ The copied and modified execution configuration.
+ """
+ new_config = {}
+ for field in fields(self):
+ if field.name in kwargs:
+ new_config[field.name] = kwargs[field.name]
+ else:
+ new_config[field.name] = getattr(self, field.name)
+
+ return ExecutionConfiguration(**new_config)
+
@dataclass(slots=True, frozen=True)
class Configuration:
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 84e45fe3c2..051b079fe4 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -197,7 +197,7 @@
},
"cases": {
"type": "array",
- "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.",
+ "description": "If specified, only this subset of test suite's test cases will be run.",
"items": {
"type": "string"
},
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 933685d638..5f6bcbbb86 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -17,17 +17,27 @@
and the test case stage runs test cases individually.
"""
+import importlib
+import inspect
import logging
+import re
import sys
from types import MethodType
+from typing import Iterable
from .config import (
BuildTargetConfiguration,
+ Configuration,
ExecutionConfiguration,
TestSuiteConfig,
load_config,
)
-from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError
+from .exception import (
+ BlockingTestSuiteError,
+ ConfigurationError,
+ SSHTimeoutError,
+ TestCaseVerifyError,
+)
from .logger import DTSLOG, getLogger
from .settings import SETTINGS
from .test_result import (
@@ -37,8 +47,9 @@
Result,
TestCaseResult,
TestSuiteResult,
+ TestSuiteWithCases,
)
-from .test_suite import TestSuite, get_test_suites
+from .test_suite import TestSuite
from .testbed_model import SutNode, TGNode
@@ -59,13 +70,23 @@ class DTSRunner:
given execution, the next execution begins.
"""
+ _configuration: Configuration
_logger: DTSLOG
_result: DTSResult
+ _test_suite_class_prefix: str
+ _test_suite_module_prefix: str
+ _func_test_case_regex: str
+ _perf_test_case_regex: str
def __init__(self):
- """Initialize the instance with logger and result."""
+ """Initialize the instance with configuration, logger, result and string constants."""
+ self._configuration = load_config()
self._logger = getLogger("DTSRunner")
self._result = DTSResult(self._logger)
+ self._test_suite_class_prefix = "Test"
+ self._test_suite_module_prefix = "tests.TestSuite_"
+ self._func_test_case_regex = r"test_(?!perf_)"
+ self._perf_test_case_regex = r"test_perf_"
def run(self):
"""Run all build targets in all executions from the test run configuration.
@@ -106,29 +127,32 @@ def run(self):
try:
# check the python version of the server that runs dts
self._check_dts_python_version()
+ self._result.update_setup(Result.PASS)
# for all Execution sections
- for execution in load_config().executions:
- sut_node = sut_nodes.get(execution.system_under_test_node.name)
- tg_node = tg_nodes.get(execution.traffic_generator_node.name)
-
+ for execution in self._configuration.executions:
+ self._logger.info(
+ f"Running execution with SUT '{execution.system_under_test_node.name}'."
+ )
+ execution_result = self._result.add_execution(execution.system_under_test_node)
+ # we don't want to modify the original config, so create a copy
+ execution_test_suites = list(execution.test_suites)
+ if not execution.skip_smoke_tests:
+ execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
try:
- if not sut_node:
- sut_node = SutNode(execution.system_under_test_node)
- sut_nodes[sut_node.name] = sut_node
- if not tg_node:
- tg_node = TGNode(execution.traffic_generator_node)
- tg_nodes[tg_node.name] = tg_node
- self._result.update_setup(Result.PASS)
+ test_suites_with_cases = self._get_test_suites_with_cases(
+ execution_test_suites, execution.func, execution.perf
+ )
except Exception as e:
- failed_node = execution.system_under_test_node.name
- if sut_node:
- failed_node = execution.traffic_generator_node.name
- self._logger.exception(f"The Creation of node {failed_node} failed.")
- self._result.update_setup(Result.FAIL, e)
+ self._logger.exception(
+ f"Invalid test suite configuration found: " f"{execution_test_suites}."
+ )
+ execution_result.update_setup(Result.FAIL, e)
else:
- self._run_execution(sut_node, tg_node, execution)
+ self._connect_nodes_and_run_execution(
+ sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases
+ )
except Exception as e:
self._logger.exception("An unexpected error has occurred.")
@@ -163,11 +187,207 @@ def _check_dts_python_version(self) -> None:
)
self._logger.warning("Please use Python >= 3.10 instead.")
+ def _get_test_suites_with_cases(
+ self,
+ test_suite_configs: list[TestSuiteConfig],
+ func: bool,
+ perf: bool,
+ ) -> list[TestSuiteWithCases]:
+ """Test suites with test cases discovery.
+
+ The test suites with test cases defined in the user configuration are discovered
+ and stored for future use so that we don't import the modules twice and so that
+ the list of test suites with test cases is available for recording right away.
+
+ Args:
+ test_suite_configs: Test suite configurations.
+ func: Whether to include functional test cases in the final list.
+ perf: Whether to include performance test cases in the final list.
+
+ Returns:
+ The discovered test suites, each with test cases.
+ """
+ test_suites_with_cases = []
+
+ for test_suite_config in test_suite_configs:
+ test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
+ test_cases = []
+ func_test_cases, perf_test_cases = self._filter_test_cases(
+ test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases)
+ )
+ if func:
+ test_cases.extend(func_test_cases)
+ if perf:
+ test_cases.extend(perf_test_cases)
+
+ test_suites_with_cases.append(
+ TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases)
+ )
+
+ return test_suites_with_cases
+
+ def _get_test_suite_class(self, module_name: str) -> type[TestSuite]:
+ """Find the :class:`TestSuite` class in `module_name`.
+
+ The full module name is `module_name` prefixed with `self._test_suite_module_prefix`.
+ The module name is a standard filename with words separated with underscores.
+ Search the `module_name` for a :class:`TestSuite` class which starts
+ with `self._test_suite_class_prefix`, continuing with CamelCase `module_name`.
+ The first matching class is returned.
+
+ The CamelCase convention applies to abbreviations, acronyms, initialisms and so on::
+
+ OS -> Os
+ TCP -> Tcp
+
+ Args:
+ module_name: The module name without prefix where to search for the test suite.
+
+ Returns:
+ The found test suite class.
+
+ Raises:
+ ConfigurationError: If the corresponding module is not found or
+ a valid :class:`TestSuite` is not found in the module.
+ """
+
+ def is_test_suite(object) -> bool:
+ """Check whether `object` is a :class:`TestSuite`.
+
+ The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself.
+
+ Args:
+ object: The object to be checked.
+
+ Returns:
+ :data:`True` if `object` is a subclass of `TestSuite`.
+ """
+ try:
+ if issubclass(object, TestSuite) and object is not TestSuite:
+ return True
+ except TypeError:
+ return False
+ return False
+
+ testsuite_module_path = f"{self._test_suite_module_prefix}{module_name}"
+ try:
+ test_suite_module = importlib.import_module(testsuite_module_path)
+ except ModuleNotFoundError as e:
+ raise ConfigurationError(
+ f"Test suite module '{testsuite_module_path}' not found."
+ ) from e
+
+ camel_case_suite_name = "".join(
+ [suite_word.capitalize() for suite_word in module_name.split("_")]
+ )
+ full_suite_name_to_find = f"{self._test_suite_class_prefix}{camel_case_suite_name}"
+ for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite):
+ if class_name == full_suite_name_to_find:
+ return class_obj
+ raise ConfigurationError(
+ f"Couldn't find any valid test suites in {test_suite_module.__name__}."
+ )
+
+ def _filter_test_cases(
+ self, test_suite_class: type[TestSuite], test_cases_to_run: set[str]
+ ) -> tuple[list[MethodType], list[MethodType]]:
+ """Filter `test_cases_to_run` from `test_suite_class`.
+
+ There are two rounds of filtering if `test_cases_to_run` is not empty.
+ The first filters `test_cases_to_run` from all methods of `test_suite_class`.
+ Then the methods are separated into functional and performance test cases.
+ If a method matches neither the functional nor performance name prefix, it's an error.
+
+ Args:
+ test_suite_class: The class of the test suite.
+ test_cases_to_run: Test case names to filter from `test_suite_class`.
+ If empty, return all matching test cases.
+
+ Returns:
+ A list of test case methods that should be executed.
+
+ Raises:
+ ConfigurationError: If a test case from `test_cases_to_run` is not found
+ or it doesn't match either the functional nor performance name prefix.
+ """
+ func_test_cases = []
+ perf_test_cases = []
+ name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction)
+ if test_cases_to_run:
+ name_method_tuples = [
+ (name, method) for name, method in name_method_tuples if name in test_cases_to_run
+ ]
+ if len(name_method_tuples) < len(test_cases_to_run):
+ missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples}
+ raise ConfigurationError(
+ f"Test cases {missing_test_cases} not found among methods "
+ f"of {test_suite_class.__name__}."
+ )
+
+ for test_case_name, test_case_method in name_method_tuples:
+ if re.match(self._func_test_case_regex, test_case_name):
+ func_test_cases.append(test_case_method)
+ elif re.match(self._perf_test_case_regex, test_case_name):
+ perf_test_cases.append(test_case_method)
+ elif test_cases_to_run:
+ raise ConfigurationError(
+ f"Method '{test_case_name}' matches neither "
+ f"a functional nor a performance test case name."
+ )
+
+ return func_test_cases, perf_test_cases
+
+ def _connect_nodes_and_run_execution(
+ self,
+ sut_nodes: dict[str, SutNode],
+ tg_nodes: dict[str, TGNode],
+ execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
+ ) -> None:
+ """Connect nodes, then continue to run the given execution.
+
+ Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`.
+ If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`,
+ respectively.
+ If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`.
+
+ Args:
+ sut_nodes: A dictionary storing connected/to be connected SUT nodes.
+ tg_nodes: A dictionary storing connected/to be connected TG nodes.
+ execution: An execution's test run configuration.
+ execution_result: The execution's result.
+ test_suites_with_cases: The test suites with test cases to run.
+ """
+ sut_node = sut_nodes.get(execution.system_under_test_node.name)
+ tg_node = tg_nodes.get(execution.traffic_generator_node.name)
+
+ try:
+ if not sut_node:
+ sut_node = SutNode(execution.system_under_test_node)
+ sut_nodes[sut_node.name] = sut_node
+ if not tg_node:
+ tg_node = TGNode(execution.traffic_generator_node)
+ tg_nodes[tg_node.name] = tg_node
+ except Exception as e:
+ failed_node = execution.system_under_test_node.name
+ if sut_node:
+ failed_node = execution.traffic_generator_node.name
+ self._logger.exception(f"The Creation of node {failed_node} failed.")
+ execution_result.update_setup(Result.FAIL, e)
+
+ else:
+ self._run_execution(
+ sut_node, tg_node, execution, execution_result, test_suites_with_cases
+ )
+
def _run_execution(
self,
sut_node: SutNode,
tg_node: TGNode,
execution: ExecutionConfiguration,
+ execution_result: ExecutionResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
"""Run the given execution.
@@ -178,11 +398,11 @@ def _run_execution(
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
execution: An execution's test run configuration.
+ execution_result: The execution's result.
+ test_suites_with_cases: The test suites with test cases to run.
"""
self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.")
- execution_result = self._result.add_execution(sut_node.config)
execution_result.add_sut_info(sut_node.node_info)
-
try:
sut_node.set_up_execution(execution)
execution_result.update_setup(Result.PASS)
@@ -192,7 +412,10 @@ def _run_execution(
else:
for build_target in execution.build_targets:
- self._run_build_target(sut_node, tg_node, build_target, execution, execution_result)
+ build_target_result = execution_result.add_build_target(build_target)
+ self._run_build_target(
+ sut_node, tg_node, build_target, build_target_result, test_suites_with_cases
+ )
finally:
try:
@@ -207,8 +430,8 @@ def _run_build_target(
sut_node: SutNode,
tg_node: TGNode,
build_target: BuildTargetConfiguration,
- execution: ExecutionConfiguration,
- execution_result: ExecutionResult,
+ build_target_result: BuildTargetResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
"""Run the given build target.
@@ -220,11 +443,11 @@ def _run_build_target(
sut_node: The execution's sut node.
tg_node: The execution's tg node.
build_target: A build target's test run configuration.
- execution: The build target's execution's test run configuration.
- execution_result: The execution level result object associated with the execution.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ test_suites_with_cases: The test suites with test cases to run.
"""
self._logger.info(f"Running build target '{build_target.name}'.")
- build_target_result = execution_result.add_build_target(build_target)
try:
sut_node.set_up_build_target(build_target)
@@ -236,7 +459,7 @@ def _run_build_target(
build_target_result.update_setup(Result.FAIL, e)
else:
- self._run_test_suites(sut_node, tg_node, execution, build_target_result)
+ self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases)
finally:
try:
@@ -250,10 +473,10 @@ def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
- execution: ExecutionConfiguration,
build_target_result: BuildTargetResult,
+ test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
- """Run the execution's (possibly a subset of) test suites using the current build target.
+ """Run `test_suites_with_cases` with the current build target.
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
@@ -264,22 +487,20 @@ def _run_test_suites(
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
- execution: The execution's test run configuration associated
- with the current build target.
build_target_result: The build target level result object associated
with the current build target.
+ test_suites_with_cases: The test suites with test cases to run.
"""
end_build_target = False
- if not execution.skip_smoke_tests:
- execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
- for test_suite_config in execution.test_suites:
+ for test_suite_with_cases in test_suites_with_cases:
+ test_suite_result = build_target_result.add_test_suite(
+ test_suite_with_cases.test_suite_class.__name__
+ )
try:
- self._run_test_suite_module(
- sut_node, tg_node, execution, build_target_result, test_suite_config
- )
+ self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
except BlockingTestSuiteError as e:
self._logger.exception(
- f"An error occurred within {test_suite_config.test_suite}. "
+ f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
"Skipping build target..."
)
self._result.add_error(e)
@@ -288,15 +509,14 @@ def _run_test_suites(
if end_build_target:
break
- def _run_test_suite_module(
+ def _run_test_suite(
self,
sut_node: SutNode,
tg_node: TGNode,
- execution: ExecutionConfiguration,
- build_target_result: BuildTargetResult,
- test_suite_config: TestSuiteConfig,
+ test_suite_result: TestSuiteResult,
+ test_suite_with_cases: TestSuiteWithCases,
) -> None:
- """Set up, execute and tear down all test suites in a single test suite module.
+ """Set up, execute and tear down `test_suite_with_cases`.
The method assumes the build target we're testing has already been built on the SUT node.
The current build target thus corresponds to the current DPDK build present on the SUT node.
@@ -306,92 +526,79 @@ def _run_test_suite_module(
Record the setup and the teardown and handle failures.
- The test cases to execute are discovered when creating the :class:`TestSuite` object.
-
Args:
sut_node: The execution's SUT node.
tg_node: The execution's TG node.
- execution: The execution's test run configuration associated
- with the current build target.
- build_target_result: The build target level result object associated
- with the current build target.
- test_suite_config: Test suite test run configuration specifying the test suite module
- and possibly a subset of test cases of test suites in that module.
+ test_suite_result: The test suite level result object associated
+ with the current test suite.
+ test_suite_with_cases: The test suite with test cases to run.
Raises:
BlockingTestSuiteError: If a blocking test suite fails.
"""
+ test_suite_name = test_suite_with_cases.test_suite_class.__name__
+ test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
try:
- full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
- test_suite_classes = get_test_suites(full_suite_path)
- suites_str = ", ".join((x.__name__ for x in test_suite_classes))
- self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.")
+ self._logger.info(f"Starting test suite setup: {test_suite_name}")
+ test_suite.set_up_suite()
+ test_suite_result.update_setup(Result.PASS)
+ self._logger.info(f"Test suite setup successful: {test_suite_name}")
except Exception as e:
- self._logger.exception("An error occurred when searching for test suites.")
- self._result.update_setup(Result.ERROR, e)
+ self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
+ test_suite_result.update_setup(Result.ERROR, e)
else:
- for test_suite_class in test_suite_classes:
- test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases)
-
- test_suite_name = test_suite.__class__.__name__
- test_suite_result = build_target_result.add_test_suite(test_suite_name)
- try:
- self._logger.info(f"Starting test suite setup: {test_suite_name}")
- test_suite.set_up_suite()
- test_suite_result.update_setup(Result.PASS)
- self._logger.info(f"Test suite setup successful: {test_suite_name}")
- except Exception as e:
- self._logger.exception(f"Test suite setup ERROR: {test_suite_name}")
- test_suite_result.update_setup(Result.ERROR, e)
-
- else:
- self._execute_test_suite(execution.func, test_suite, test_suite_result)
-
- finally:
- try:
- test_suite.tear_down_suite()
- sut_node.kill_cleanup_dpdk_apps()
- test_suite_result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
- self._logger.warning(
- f"Test suite '{test_suite_name}' teardown failed, "
- f"the next test suite may be affected."
- )
- test_suite_result.update_setup(Result.ERROR, e)
- if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
- raise BlockingTestSuiteError(test_suite_name)
+ self._execute_test_suite(
+ test_suite,
+ test_suite_with_cases.test_cases,
+ test_suite_result,
+ )
+ finally:
+ try:
+ test_suite.tear_down_suite()
+ sut_node.kill_cleanup_dpdk_apps()
+ test_suite_result.update_teardown(Result.PASS)
+ except Exception as e:
+ self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}")
+ self._logger.warning(
+ f"Test suite '{test_suite_name}' teardown failed, "
+ "the next test suite may be affected."
+ )
+ test_suite_result.update_setup(Result.ERROR, e)
+ if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking:
+ raise BlockingTestSuiteError(test_suite_name)
def _execute_test_suite(
- self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult
+ self,
+ test_suite: TestSuite,
+ test_cases: Iterable[MethodType],
+ test_suite_result: TestSuiteResult,
) -> None:
- """Execute all discovered test cases in `test_suite`.
+ """Execute all `test_cases` in `test_suite`.
If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment
variable is set, in case of a test case failure, the test case will be executed again
until it passes or it fails that many times in addition of the first failure.
Args:
- func: Whether to execute functional test cases.
test_suite: The test suite object.
+ test_cases: The list of test case methods.
test_suite_result: The test suite level result object associated
with the current test suite.
"""
- if func:
- for test_case_method in test_suite._get_functional_test_cases():
- test_case_name = test_case_method.__name__
- test_case_result = test_suite_result.add_test_case(test_case_name)
- all_attempts = SETTINGS.re_run + 1
- attempt_nr = 1
+ for test_case_method in test_cases:
+ test_case_name = test_case_method.__name__
+ test_case_result = test_suite_result.add_test_case(test_case_name)
+ all_attempts = SETTINGS.re_run + 1
+ attempt_nr = 1
+ self._run_test_case(test_suite, test_case_method, test_case_result)
+ while not test_case_result and attempt_nr < all_attempts:
+ attempt_nr += 1
+ self._logger.info(
+ f"Re-running FAILED test case '{test_case_name}'. "
+ f"Attempt number {attempt_nr} out of {all_attempts}."
+ )
self._run_test_case(test_suite, test_case_method, test_case_result)
- while not test_case_result and attempt_nr < all_attempts:
- attempt_nr += 1
- self._logger.info(
- f"Re-running FAILED test case '{test_case_name}'. "
- f"Attempt number {attempt_nr} out of {all_attempts}."
- )
- self._run_test_case(test_suite, test_case_method, test_case_result)
def _run_test_case(
self,
@@ -399,7 +606,7 @@ def _run_test_case(
test_case_method: MethodType,
test_case_result: TestCaseResult,
) -> None:
- """Setup, execute and teardown a test case in `test_suite`.
+ """Setup, execute and teardown `test_case_method` from `test_suite`.
Record the result of the setup and the teardown and handle failures.
@@ -424,7 +631,7 @@ def _run_test_case(
else:
# run test case if setup was successful
- self._execute_test_case(test_case_method, test_case_result)
+ self._execute_test_case(test_suite, test_case_method, test_case_result)
finally:
try:
@@ -440,11 +647,15 @@ def _run_test_case(
test_case_result.update(Result.ERROR)
def _execute_test_case(
- self, test_case_method: MethodType, test_case_result: TestCaseResult
+ self,
+ test_suite: TestSuite,
+ test_case_method: MethodType,
+ test_case_result: TestCaseResult,
) -> None:
- """Execute one test case, record the result and handle failures.
+ """Execute `test_case_method` from `test_suite`, record the result and handle failures.
Args:
+ test_suite: The test suite object.
test_case_method: The test case method.
test_case_result: The test case level result object associated
with the current test case.
@@ -452,7 +663,7 @@ def _execute_test_case(
test_case_name = test_case_method.__name__
try:
self._logger.info(f"Starting test case execution: {test_case_name}")
- test_case_method()
+ test_case_method(test_suite)
test_case_result.update(Result.PASS)
self._logger.info(f"Test case execution PASSED: {test_case_name}")
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 609c8d0e62..2b8bfbe0ed 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser:
"--test-cases",
action=_env_arg("DTS_TESTCASES"),
default="",
- help="[DTS_TESTCASES] Comma-separated list of test cases to execute. "
- "Unknown test cases will be silently ignored.",
+ help="[DTS_TESTCASES] Comma-separated list of test cases to execute.",
)
parser.add_argument(
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 4467749a9d..075195fd5b 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -25,7 +25,9 @@
import os.path
from collections.abc import MutableSequence
+from dataclasses import dataclass
from enum import Enum, auto
+from types import MethodType
from .config import (
OS,
@@ -36,10 +38,42 @@
CPUType,
NodeConfiguration,
NodeInfo,
+ TestSuiteConfig,
)
from .exception import DTSError, ErrorSeverity
from .logger import DTSLOG
from .settings import SETTINGS
+from .test_suite import TestSuite
+
+
+@dataclass(slots=True, frozen=True)
+class TestSuiteWithCases:
+ """A test suite class with test case methods.
+
+ An auxiliary class holding a test case class with test case methods. The intended use of this
+ class is to hold a subset of test cases (which could be all test cases) because we don't have
+ all the data to instantiate the class at the point of inspection. The knowledge of this subset
+ is needed in case an error occurs before the class is instantiated and we need to record
+ which test cases were blocked by the error.
+
+ Attributes:
+ test_suite_class: The test suite class.
+ test_cases: The test case methods.
+ """
+
+ test_suite_class: type[TestSuite]
+ test_cases: list[MethodType]
+
+ def create_config(self) -> TestSuiteConfig:
+ """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
+
+ Returns:
+ The :class:`TestSuiteConfig` representation.
+ """
+ return TestSuiteConfig(
+ test_suite=self.test_suite_class.__name__,
+ test_cases=[test_case.__name__ for test_case in self.test_cases],
+ )
class Result(Enum):
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index b02fd36147..f9fe88093e 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -11,25 +11,17 @@
* Testbed (SUT, TG) configuration,
* Packet sending and verification,
* Test case verification.
-
-The module also defines a function, :func:`get_test_suites`,
-for gathering test suites from a Python module.
"""
-import importlib
-import inspect
-import re
from ipaddress import IPv4Interface, IPv6Interface, ip_interface
-from types import MethodType
-from typing import Any, ClassVar, Union
+from typing import ClassVar, Union
from scapy.layers.inet import IP # type: ignore[import]
from scapy.layers.l2 import Ether # type: ignore[import]
from scapy.packet import Packet, Padding # type: ignore[import]
-from .exception import ConfigurationError, TestCaseVerifyError
+from .exception import TestCaseVerifyError
from .logger import DTSLOG, getLogger
-from .settings import SETTINGS
from .testbed_model import Port, PortLink, SutNode, TGNode
from .utils import get_packet_summaries
@@ -37,7 +29,6 @@
class TestSuite(object):
"""The base class with building blocks needed by most test cases.
- * Test case filtering and collection,
* Test suite setup/cleanup methods to override,
* Test case setup/cleanup methods to override,
* Test case verification,
@@ -71,7 +62,6 @@ class TestSuite(object):
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
_logger: DTSLOG
- _test_cases_to_run: list[str]
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -86,24 +76,19 @@ def __init__(
self,
sut_node: SutNode,
tg_node: TGNode,
- test_cases: list[str],
):
"""Initialize the test suite testbed information and basic configuration.
- Process what test cases to run, find links between ports and set up
- default IP addresses to be used when configuring them.
+ Find links between ports and set up default IP addresses to be used when
+ configuring them.
Args:
sut_node: The SUT node where the test suite will run.
tg_node: The TG node where the test suite will run.
- test_cases: The list of test cases to execute.
- If empty, all test cases will be executed.
"""
self.sut_node = sut_node
self.tg_node = tg_node
self._logger = getLogger(self.__class__.__name__)
- self._test_cases_to_run = test_cases
- self._test_cases_to_run.extend(SETTINGS.test_cases)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
@@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool:
if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst:
return False
return True
-
- def _get_functional_test_cases(self) -> list[MethodType]:
- """Get all functional test cases defined in this TestSuite.
-
- Returns:
- The list of functional test cases of this TestSuite.
- """
- return self._get_test_cases(r"test_(?!perf_)")
-
- def _get_test_cases(self, test_case_regex: str) -> list[MethodType]:
- """Return a list of test cases matching test_case_regex.
-
- Returns:
- The list of test cases matching test_case_regex of this TestSuite.
- """
- self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.")
- filtered_test_cases = []
- for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod):
- if self._should_be_executed(test_case_name, test_case_regex):
- filtered_test_cases.append(test_case)
- cases_str = ", ".join((x.__name__ for x in filtered_test_cases))
- self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.")
- return filtered_test_cases
-
- def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool:
- """Check whether the test case should be scheduled to be executed."""
- match = bool(re.match(test_case_regex, test_case_name))
- if self._test_cases_to_run:
- return match and test_case_name in self._test_cases_to_run
-
- return match
-
-
-def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
- r"""Find all :class:`TestSuite`\s in a Python module.
-
- Args:
- testsuite_module_path: The path to the Python module.
-
- Returns:
- The list of :class:`TestSuite`\s found within the Python module.
-
- Raises:
- ConfigurationError: The test suite module was not found.
- """
-
- def is_test_suite(object: Any) -> bool:
- try:
- if issubclass(object, TestSuite) and object is not TestSuite:
- return True
- except TypeError:
- return False
- return False
-
- try:
- testcase_module = importlib.import_module(testsuite_module_path)
- except ModuleNotFoundError as e:
- raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e
- return [
- test_suite_class
- for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite)
- ]
diff --git a/dts/pyproject.toml b/dts/pyproject.toml
index 28bd970ae4..8eb92b4f11 100644
--- a/dts/pyproject.toml
+++ b/dts/pyproject.toml
@@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes"
format = "pylint"
max_line_length = 100
+[tool.pylama.linter.pycodestyle]
+ignore = "E203,W503"
+
[tool.pylama.linter.pydocstyle]
convention = "google"
diff --git a/dts/tests/TestSuite_os_udp.py b/dts/tests/TestSuite_os_udp.py
index 2cf29d37bb..b4784dd95e 100644
--- a/dts/tests/TestSuite_os_udp.py
+++ b/dts/tests/TestSuite_os_udp.py
@@ -13,7 +13,7 @@
from framework.test_suite import TestSuite
-class TestOSUdp(TestSuite):
+class TestOsUdp(TestSuite):
"""IPv4 UDP OS routing test suite."""
def set_up_suite(self) -> None:
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index 5e2bac14bd..7b2a0e97f8 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -21,7 +21,7 @@
from framework.utils import REGEX_FOR_PCI_ADDRESS
-class SmokeTests(TestSuite):
+class TestSmokeTests(TestSuite):
"""DPDK and infrastructure smoke test suite.
The test cases validate the most basic DPDK functionality needed for all other test suites.
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 4/7] dts: reorganize test result
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
` (2 preceding siblings ...)
2024-03-01 10:55 ` [PATCH v4 3/7] dts: filter test suites in executions Juraj Linkeš
@ 2024-03-01 10:55 ` Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš
` (4 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
The current order of Result classes in the test_suite.py module is
guided by the needs of type hints, which is not as intuitively readable
as ordering them by the occurrences in code. The order goes from the
topmost level to lowermost:
BaseResult
DTSResult
ExecutionResult
BuildTargetResult
TestSuiteResult
TestCaseResult
This is the same order as they're used in the runner module and they're
also used in the same order between themselves in the test_result
module.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/test_result.py | 411 ++++++++++++++++++-----------------
1 file changed, 206 insertions(+), 205 deletions(-)
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 075195fd5b..abdbafab10 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -28,6 +28,7 @@
from dataclasses import dataclass
from enum import Enum, auto
from types import MethodType
+from typing import Union
from .config import (
OS,
@@ -129,58 +130,6 @@ def __bool__(self) -> bool:
return bool(self.result)
-class Statistics(dict):
- """How many test cases ended in which result state along some other basic information.
-
- Subclassing :class:`dict` provides a convenient way to format the data.
-
- The data are stored in the following keys:
-
- * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases.
- * **DPDK VERSION** (:class:`str`) -- The tested DPDK version.
- """
-
- def __init__(self, dpdk_version: str | None):
- """Extend the constructor with keys in which the data are stored.
-
- Args:
- dpdk_version: The version of tested DPDK.
- """
- super(Statistics, self).__init__()
- for result in Result:
- self[result.name] = 0
- self["PASS RATE"] = 0.0
- self["DPDK VERSION"] = dpdk_version
-
- def __iadd__(self, other: Result) -> "Statistics":
- """Add a Result to the final count.
-
- Example:
- stats: Statistics = Statistics() # empty Statistics
- stats += Result.PASS # add a Result to `stats`
-
- Args:
- other: The Result to add to this statistics object.
-
- Returns:
- The modified statistics object.
- """
- self[other.name] += 1
- self["PASS RATE"] = (
- float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result)
- )
- return self
-
- def __str__(self) -> str:
- """Each line contains the formatted key = value pair."""
- stats_str = ""
- for key, value in self.items():
- stats_str += f"{key:<12} = {value}\n"
- # according to docs, we should use \n when writing to text files
- # on all platforms
- return stats_str
-
-
class BaseResult(object):
"""Common data and behavior of DTS results.
@@ -245,7 +194,7 @@ def get_errors(self) -> list[Exception]:
"""
return self._get_setup_teardown_errors() + self._get_inner_errors()
- def add_stats(self, statistics: Statistics) -> None:
+ def add_stats(self, statistics: "Statistics") -> None:
"""Collate stats from the whole result hierarchy.
Args:
@@ -255,91 +204,149 @@ def add_stats(self, statistics: Statistics) -> None:
inner_result.add_stats(statistics)
-class TestCaseResult(BaseResult, FixtureResult):
- r"""The test case specific result.
+class DTSResult(BaseResult):
+ """Stores environment information and test results from a DTS run.
- Stores the result of the actual test case. This is done by adding an extra superclass
- in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and
- the class is itself a record of the test case.
+ * Execution level information, such as testbed and the test suite list,
+ * Build target level information, such as compiler, target OS and cpu,
+ * Test suite and test case results,
+ * All errors that are caught and recorded during DTS execution.
+
+ The information is stored hierarchically. This is the first level of the hierarchy
+ and as such is where the data form the whole hierarchy is collated or processed.
+
+ The internal list stores the results of all executions.
Attributes:
- test_case_name: The test case name.
+ dpdk_version: The DPDK version to record.
"""
- test_case_name: str
+ dpdk_version: str | None
+ _logger: DTSLOG
+ _errors: list[Exception]
+ _return_code: ErrorSeverity
+ _stats_result: Union["Statistics", None]
+ _stats_filename: str
- def __init__(self, test_case_name: str):
- """Extend the constructor with `test_case_name`.
+ def __init__(self, logger: DTSLOG):
+ """Extend the constructor with top-level specifics.
Args:
- test_case_name: The test case's name.
+ logger: The logger instance the whole result will use.
"""
- super(TestCaseResult, self).__init__()
- self.test_case_name = test_case_name
+ super(DTSResult, self).__init__()
+ self.dpdk_version = None
+ self._logger = logger
+ self._errors = []
+ self._return_code = ErrorSeverity.NO_ERR
+ self._stats_result = None
+ self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
- def update(self, result: Result, error: Exception | None = None) -> None:
- """Update the test case result.
+ def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult":
+ """Add and return the inner result (execution).
- This updates the result of the test case itself and doesn't affect
- the results of the setup and teardown steps in any way.
+ Args:
+ sut_node: The SUT node's test run configuration.
+
+ Returns:
+ The execution's result.
+ """
+ execution_result = ExecutionResult(sut_node)
+ self._inner_results.append(execution_result)
+ return execution_result
+
+ def add_error(self, error: Exception) -> None:
+ """Record an error that occurred outside any execution.
Args:
- result: The result of the test case.
- error: The error that occurred in case of a failure.
+ error: The exception to record.
"""
- self.result = result
- self.error = error
+ self._errors.append(error)
- def _get_inner_errors(self) -> list[Exception]:
- if self.error:
- return [self.error]
- return []
+ def process(self) -> None:
+ """Process the data after a whole DTS run.
- def add_stats(self, statistics: Statistics) -> None:
- r"""Add the test case result to statistics.
+ The data is added to inner objects during runtime and this object is not updated
+ at that time. This requires us to process the inner data after it's all been gathered.
- The base method goes through the hierarchy recursively and this method is here to stop
- the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree.
+ The processing gathers all errors and the statistics of test case results.
+ """
+ self._errors += self.get_errors()
+ if self._errors and self._logger:
+ self._logger.debug("Summary of errors:")
+ for error in self._errors:
+ self._logger.debug(repr(error))
- Args:
- statistics: The :class:`Statistics` object where the stats will be added.
+ self._stats_result = Statistics(self.dpdk_version)
+ self.add_stats(self._stats_result)
+ with open(self._stats_filename, "w+") as stats_file:
+ stats_file.write(str(self._stats_result))
+
+ def get_return_code(self) -> int:
+ """Go through all stored Exceptions and return the final DTS error code.
+
+ Returns:
+ The highest error code found.
"""
- statistics += self.result
+ for error in self._errors:
+ error_return_code = ErrorSeverity.GENERIC_ERR
+ if isinstance(error, DTSError):
+ error_return_code = error.severity
- def __bool__(self) -> bool:
- """The test case passed only if setup, teardown and the test case itself passed."""
- return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
+ if error_return_code > self._return_code:
+ self._return_code = error_return_code
+ return int(self._return_code)
-class TestSuiteResult(BaseResult):
- """The test suite specific result.
- The internal list stores the results of all test cases in a given test suite.
+class ExecutionResult(BaseResult):
+ """The execution specific result.
+
+ The internal list stores the results of all build targets in a given execution.
Attributes:
- suite_name: The test suite name.
+ sut_node: The SUT node used in the execution.
+ sut_os_name: The operating system of the SUT node.
+ sut_os_version: The operating system version of the SUT node.
+ sut_kernel_version: The operating system kernel version of the SUT node.
"""
- suite_name: str
+ sut_node: NodeConfiguration
+ sut_os_name: str
+ sut_os_version: str
+ sut_kernel_version: str
- def __init__(self, suite_name: str):
- """Extend the constructor with `suite_name`.
+ def __init__(self, sut_node: NodeConfiguration):
+ """Extend the constructor with the `sut_node`'s config.
Args:
- suite_name: The test suite's name.
+ sut_node: The SUT node's test run configuration used in the execution.
"""
- super(TestSuiteResult, self).__init__()
- self.suite_name = suite_name
+ super(ExecutionResult, self).__init__()
+ self.sut_node = sut_node
- def add_test_case(self, test_case_name: str) -> TestCaseResult:
- """Add and return the inner result (test case).
+ def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult":
+ """Add and return the inner result (build target).
+
+ Args:
+ build_target: The build target's test run configuration.
Returns:
- The test case's result.
+ The build target's result.
"""
- test_case_result = TestCaseResult(test_case_name)
- self._inner_results.append(test_case_result)
- return test_case_result
+ build_target_result = BuildTargetResult(build_target)
+ self._inner_results.append(build_target_result)
+ return build_target_result
+
+ def add_sut_info(self, sut_info: NodeInfo) -> None:
+ """Add SUT information gathered at runtime.
+
+ Args:
+ sut_info: The additional SUT node information.
+ """
+ self.sut_os_name = sut_info.os_name
+ self.sut_os_version = sut_info.os_version
+ self.sut_kernel_version = sut_info.kernel_version
class BuildTargetResult(BaseResult):
@@ -386,7 +393,7 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def add_test_suite(self, test_suite_name: str) -> TestSuiteResult:
+ def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult":
"""Add and return the inner result (test suite).
Returns:
@@ -397,146 +404,140 @@ def add_test_suite(self, test_suite_name: str) -> TestSuiteResult:
return test_suite_result
-class ExecutionResult(BaseResult):
- """The execution specific result.
+class TestSuiteResult(BaseResult):
+ """The test suite specific result.
- The internal list stores the results of all build targets in a given execution.
+ The internal list stores the results of all test cases in a given test suite.
Attributes:
- sut_node: The SUT node used in the execution.
- sut_os_name: The operating system of the SUT node.
- sut_os_version: The operating system version of the SUT node.
- sut_kernel_version: The operating system kernel version of the SUT node.
+ suite_name: The test suite name.
"""
- sut_node: NodeConfiguration
- sut_os_name: str
- sut_os_version: str
- sut_kernel_version: str
+ suite_name: str
- def __init__(self, sut_node: NodeConfiguration):
- """Extend the constructor with the `sut_node`'s config.
+ def __init__(self, suite_name: str):
+ """Extend the constructor with `suite_name`.
Args:
- sut_node: The SUT node's test run configuration used in the execution.
+ suite_name: The test suite's name.
"""
- super(ExecutionResult, self).__init__()
- self.sut_node = sut_node
-
- def add_build_target(self, build_target: BuildTargetConfiguration) -> BuildTargetResult:
- """Add and return the inner result (build target).
+ super(TestSuiteResult, self).__init__()
+ self.suite_name = suite_name
- Args:
- build_target: The build target's test run configuration.
+ def add_test_case(self, test_case_name: str) -> "TestCaseResult":
+ """Add and return the inner result (test case).
Returns:
- The build target's result.
- """
- build_target_result = BuildTargetResult(build_target)
- self._inner_results.append(build_target_result)
- return build_target_result
-
- def add_sut_info(self, sut_info: NodeInfo) -> None:
- """Add SUT information gathered at runtime.
-
- Args:
- sut_info: The additional SUT node information.
+ The test case's result.
"""
- self.sut_os_name = sut_info.os_name
- self.sut_os_version = sut_info.os_version
- self.sut_kernel_version = sut_info.kernel_version
+ test_case_result = TestCaseResult(test_case_name)
+ self._inner_results.append(test_case_result)
+ return test_case_result
-class DTSResult(BaseResult):
- """Stores environment information and test results from a DTS run.
-
- * Execution level information, such as testbed and the test suite list,
- * Build target level information, such as compiler, target OS and cpu,
- * Test suite and test case results,
- * All errors that are caught and recorded during DTS execution.
-
- The information is stored hierarchically. This is the first level of the hierarchy
- and as such is where the data form the whole hierarchy is collated or processed.
+class TestCaseResult(BaseResult, FixtureResult):
+ r"""The test case specific result.
- The internal list stores the results of all executions.
+ Stores the result of the actual test case. This is done by adding an extra superclass
+ in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and
+ the class is itself a record of the test case.
Attributes:
- dpdk_version: The DPDK version to record.
+ test_case_name: The test case name.
"""
- dpdk_version: str | None
- _logger: DTSLOG
- _errors: list[Exception]
- _return_code: ErrorSeverity
- _stats_result: Statistics | None
- _stats_filename: str
+ test_case_name: str
- def __init__(self, logger: DTSLOG):
- """Extend the constructor with top-level specifics.
+ def __init__(self, test_case_name: str):
+ """Extend the constructor with `test_case_name`.
Args:
- logger: The logger instance the whole result will use.
+ test_case_name: The test case's name.
"""
- super(DTSResult, self).__init__()
- self.dpdk_version = None
- self._logger = logger
- self._errors = []
- self._return_code = ErrorSeverity.NO_ERR
- self._stats_result = None
- self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
+ super(TestCaseResult, self).__init__()
+ self.test_case_name = test_case_name
- def add_execution(self, sut_node: NodeConfiguration) -> ExecutionResult:
- """Add and return the inner result (execution).
+ def update(self, result: Result, error: Exception | None = None) -> None:
+ """Update the test case result.
- Args:
- sut_node: The SUT node's test run configuration.
+ This updates the result of the test case itself and doesn't affect
+ the results of the setup and teardown steps in any way.
- Returns:
- The execution's result.
+ Args:
+ result: The result of the test case.
+ error: The error that occurred in case of a failure.
"""
- execution_result = ExecutionResult(sut_node)
- self._inner_results.append(execution_result)
- return execution_result
+ self.result = result
+ self.error = error
- def add_error(self, error: Exception) -> None:
- """Record an error that occurred outside any execution.
+ def _get_inner_errors(self) -> list[Exception]:
+ if self.error:
+ return [self.error]
+ return []
+
+ def add_stats(self, statistics: "Statistics") -> None:
+ r"""Add the test case result to statistics.
+
+ The base method goes through the hierarchy recursively and this method is here to stop
+ the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree.
Args:
- error: The exception to record.
+ statistics: The :class:`Statistics` object where the stats will be added.
"""
- self._errors.append(error)
+ statistics += self.result
- def process(self) -> None:
- """Process the data after a whole DTS run.
+ def __bool__(self) -> bool:
+ """The test case passed only if setup, teardown and the test case itself passed."""
+ return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
- The data is added to inner objects during runtime and this object is not updated
- at that time. This requires us to process the inner data after it's all been gathered.
- The processing gathers all errors and the statistics of test case results.
+class Statistics(dict):
+ """How many test cases ended in which result state along some other basic information.
+
+ Subclassing :class:`dict` provides a convenient way to format the data.
+
+ The data are stored in the following keys:
+
+ * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases.
+ * **DPDK VERSION** (:class:`str`) -- The tested DPDK version.
+ """
+
+ def __init__(self, dpdk_version: str | None):
+ """Extend the constructor with keys in which the data are stored.
+
+ Args:
+ dpdk_version: The version of tested DPDK.
"""
- self._errors += self.get_errors()
- if self._errors and self._logger:
- self._logger.debug("Summary of errors:")
- for error in self._errors:
- self._logger.debug(repr(error))
+ super(Statistics, self).__init__()
+ for result in Result:
+ self[result.name] = 0
+ self["PASS RATE"] = 0.0
+ self["DPDK VERSION"] = dpdk_version
- self._stats_result = Statistics(self.dpdk_version)
- self.add_stats(self._stats_result)
- with open(self._stats_filename, "w+") as stats_file:
- stats_file.write(str(self._stats_result))
+ def __iadd__(self, other: Result) -> "Statistics":
+ """Add a Result to the final count.
- def get_return_code(self) -> int:
- """Go through all stored Exceptions and return the final DTS error code.
+ Example:
+ stats: Statistics = Statistics() # empty Statistics
+ stats += Result.PASS # add a Result to `stats`
+
+ Args:
+ other: The Result to add to this statistics object.
Returns:
- The highest error code found.
+ The modified statistics object.
"""
- for error in self._errors:
- error_return_code = ErrorSeverity.GENERIC_ERR
- if isinstance(error, DTSError):
- error_return_code = error.severity
-
- if error_return_code > self._return_code:
- self._return_code = error_return_code
+ self[other.name] += 1
+ self["PASS RATE"] = (
+ float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result)
+ )
+ return self
- return int(self._return_code)
+ def __str__(self) -> str:
+ """Each line contains the formatted key = value pair."""
+ stats_str = ""
+ for key, value in self.items():
+ stats_str += f"{key:<12} = {value}\n"
+ # according to docs, we should use \n when writing to text files
+ # on all platforms
+ return stats_str
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 5/7] dts: block all test cases when earlier setup fails
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
` (3 preceding siblings ...)
2024-03-01 10:55 ` [PATCH v4 4/7] dts: reorganize test result Juraj Linkeš
@ 2024-03-01 10:55 ` Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 6/7] dts: refactor logging configuration Juraj Linkeš
` (3 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
In case of a failure before a test suite, the child results will be
recursively recorded as blocked, giving us a full report which was
missing previously.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/runner.py | 21 ++--
dts/framework/test_result.py | 186 +++++++++++++++++++++++++----------
2 files changed, 148 insertions(+), 59 deletions(-)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 5f6bcbbb86..864015c350 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -60,13 +60,15 @@ class DTSRunner:
Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult`
or one of its subclasses. The test case results are also recorded.
- If an error occurs, the current stage is aborted, the error is recorded and the run continues in
- the next iteration of the same stage. The return code is the highest `severity` of all
+ If an error occurs, the current stage is aborted, the error is recorded, everything in
+ the inner stages is marked as blocked and the run continues in the next iteration
+ of the same stage. The return code is the highest `severity` of all
:class:`~.framework.exception.DTSError`\s.
Example:
- An error occurs in a build target setup. The current build target is aborted and the run
- continues with the next build target. If the errored build target was the last one in the
+ An error occurs in a build target setup. The current build target is aborted,
+ all test suites and their test cases are marked as blocked and the run continues
+ with the next build target. If the errored build target was the last one in the
given execution, the next execution begins.
"""
@@ -100,6 +102,10 @@ def run(self):
test case within the test suite is set up, executed and torn down. After all test cases
have been executed, the test suite is torn down and the next build target will be tested.
+ In order to properly mark test suites and test cases as blocked in case of a failure,
+ we need to have discovered which test suites and test cases to run before any failures
+ happen. The discovery happens at the earliest point at the start of each execution.
+
All the nested steps look like this:
#. Execution setup
@@ -134,7 +140,7 @@ def run(self):
self._logger.info(
f"Running execution with SUT '{execution.system_under_test_node.name}'."
)
- execution_result = self._result.add_execution(execution.system_under_test_node)
+ execution_result = self._result.add_execution(execution)
# we don't want to modify the original config, so create a copy
execution_test_suites = list(execution.test_suites)
if not execution.skip_smoke_tests:
@@ -143,6 +149,7 @@ def run(self):
test_suites_with_cases = self._get_test_suites_with_cases(
execution_test_suites, execution.func, execution.perf
)
+ execution_result.test_suites_with_cases = test_suites_with_cases
except Exception as e:
self._logger.exception(
f"Invalid test suite configuration found: " f"{execution_test_suites}."
@@ -493,9 +500,7 @@ def _run_test_suites(
"""
end_build_target = False
for test_suite_with_cases in test_suites_with_cases:
- test_suite_result = build_target_result.add_test_suite(
- test_suite_with_cases.test_suite_class.__name__
- )
+ test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
try:
self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
except BlockingTestSuiteError as e:
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index abdbafab10..eedb2d20ee 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -37,7 +37,7 @@
BuildTargetInfo,
Compiler,
CPUType,
- NodeConfiguration,
+ ExecutionConfiguration,
NodeInfo,
TestSuiteConfig,
)
@@ -88,6 +88,8 @@ class Result(Enum):
ERROR = auto()
#:
SKIP = auto()
+ #:
+ BLOCK = auto()
def __bool__(self) -> bool:
"""Only PASS is True."""
@@ -141,21 +143,26 @@ class BaseResult(object):
Attributes:
setup_result: The result of the setup of the particular stage.
teardown_result: The results of the teardown of the particular stage.
+ child_results: The results of the descendants in the results hierarchy.
"""
setup_result: FixtureResult
teardown_result: FixtureResult
- _inner_results: MutableSequence["BaseResult"]
+ child_results: MutableSequence["BaseResult"]
def __init__(self):
"""Initialize the constructor."""
self.setup_result = FixtureResult()
self.teardown_result = FixtureResult()
- self._inner_results = []
+ self.child_results = []
def update_setup(self, result: Result, error: Exception | None = None) -> None:
"""Store the setup result.
+ If the result is :attr:`~Result.BLOCK`, :attr:`~Result.ERROR` or :attr:`~Result.FAIL`,
+ then the corresponding child results in result hierarchy
+ are also marked with :attr:`~Result.BLOCK`.
+
Args:
result: The result of the setup.
error: The error that occurred in case of a failure.
@@ -163,6 +170,16 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None:
self.setup_result.result = result
self.setup_result.error = error
+ if result in [Result.BLOCK, Result.ERROR, Result.FAIL]:
+ self.update_teardown(Result.BLOCK)
+ self._block_result()
+
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed.
+
+ The blocking of child results should be done in overloaded methods.
+ """
+
def update_teardown(self, result: Result, error: Exception | None = None) -> None:
"""Store the teardown result.
@@ -181,10 +198,8 @@ def _get_setup_teardown_errors(self) -> list[Exception]:
errors.append(self.teardown_result.error)
return errors
- def _get_inner_errors(self) -> list[Exception]:
- return [
- error for inner_result in self._inner_results for error in inner_result.get_errors()
- ]
+ def _get_child_errors(self) -> list[Exception]:
+ return [error for child_result in self.child_results for error in child_result.get_errors()]
def get_errors(self) -> list[Exception]:
"""Compile errors from the whole result hierarchy.
@@ -192,7 +207,7 @@ def get_errors(self) -> list[Exception]:
Returns:
The errors from setup, teardown and all errors found in the whole result hierarchy.
"""
- return self._get_setup_teardown_errors() + self._get_inner_errors()
+ return self._get_setup_teardown_errors() + self._get_child_errors()
def add_stats(self, statistics: "Statistics") -> None:
"""Collate stats from the whole result hierarchy.
@@ -200,8 +215,8 @@ def add_stats(self, statistics: "Statistics") -> None:
Args:
statistics: The :class:`Statistics` object where the stats will be collated.
"""
- for inner_result in self._inner_results:
- inner_result.add_stats(statistics)
+ for child_result in self.child_results:
+ child_result.add_stats(statistics)
class DTSResult(BaseResult):
@@ -242,18 +257,18 @@ def __init__(self, logger: DTSLOG):
self._stats_result = None
self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt")
- def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult":
- """Add and return the inner result (execution).
+ def add_execution(self, execution: ExecutionConfiguration) -> "ExecutionResult":
+ """Add and return the child result (execution).
Args:
- sut_node: The SUT node's test run configuration.
+ execution: The execution's test run configuration.
Returns:
The execution's result.
"""
- execution_result = ExecutionResult(sut_node)
- self._inner_results.append(execution_result)
- return execution_result
+ result = ExecutionResult(execution)
+ self.child_results.append(result)
+ return result
def add_error(self, error: Exception) -> None:
"""Record an error that occurred outside any execution.
@@ -266,8 +281,8 @@ def add_error(self, error: Exception) -> None:
def process(self) -> None:
"""Process the data after a whole DTS run.
- The data is added to inner objects during runtime and this object is not updated
- at that time. This requires us to process the inner data after it's all been gathered.
+ The data is added to child objects during runtime and this object is not updated
+ at that time. This requires us to process the child data after it's all been gathered.
The processing gathers all errors and the statistics of test case results.
"""
@@ -305,28 +320,30 @@ class ExecutionResult(BaseResult):
The internal list stores the results of all build targets in a given execution.
Attributes:
- sut_node: The SUT node used in the execution.
sut_os_name: The operating system of the SUT node.
sut_os_version: The operating system version of the SUT node.
sut_kernel_version: The operating system kernel version of the SUT node.
"""
- sut_node: NodeConfiguration
sut_os_name: str
sut_os_version: str
sut_kernel_version: str
+ _config: ExecutionConfiguration
+ _parent_result: DTSResult
+ _test_suites_with_cases: list[TestSuiteWithCases]
- def __init__(self, sut_node: NodeConfiguration):
- """Extend the constructor with the `sut_node`'s config.
+ def __init__(self, execution: ExecutionConfiguration):
+ """Extend the constructor with the execution's config and DTSResult.
Args:
- sut_node: The SUT node's test run configuration used in the execution.
+ execution: The execution's test run configuration.
"""
super(ExecutionResult, self).__init__()
- self.sut_node = sut_node
+ self._config = execution
+ self._test_suites_with_cases = []
def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult":
- """Add and return the inner result (build target).
+ """Add and return the child result (build target).
Args:
build_target: The build target's test run configuration.
@@ -334,9 +351,34 @@ def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTarg
Returns:
The build target's result.
"""
- build_target_result = BuildTargetResult(build_target)
- self._inner_results.append(build_target_result)
- return build_target_result
+ result = BuildTargetResult(
+ self._test_suites_with_cases,
+ build_target,
+ )
+ self.child_results.append(result)
+ return result
+
+ @property
+ def test_suites_with_cases(self) -> list[TestSuiteWithCases]:
+ """The test suites with test cases to be executed in this execution.
+
+ The test suites can only be assigned once.
+
+ Returns:
+ The list of test suites with test cases. If an error occurs between
+ the initialization of :class:`ExecutionResult` and assigning test cases to the instance,
+ return an empty list, representing that we don't know what to execute.
+ """
+ return self._test_suites_with_cases
+
+ @test_suites_with_cases.setter
+ def test_suites_with_cases(self, test_suites_with_cases: list[TestSuiteWithCases]) -> None:
+ if self._test_suites_with_cases:
+ raise ValueError(
+ "Attempted to assign test suites to an execution result "
+ "which already has test suites."
+ )
+ self._test_suites_with_cases = test_suites_with_cases
def add_sut_info(self, sut_info: NodeInfo) -> None:
"""Add SUT information gathered at runtime.
@@ -348,6 +390,12 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
self.sut_os_version = sut_info.os_version
self.sut_kernel_version = sut_info.kernel_version
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ for build_target in self._config.build_targets:
+ child_result = self.add_build_target(build_target)
+ child_result.update_setup(Result.BLOCK)
+
class BuildTargetResult(BaseResult):
"""The build target specific result.
@@ -369,11 +417,17 @@ class BuildTargetResult(BaseResult):
compiler: Compiler
compiler_version: str | None
dpdk_version: str | None
+ _test_suites_with_cases: list[TestSuiteWithCases]
- def __init__(self, build_target: BuildTargetConfiguration):
- """Extend the constructor with the `build_target`'s build target config.
+ def __init__(
+ self,
+ test_suites_with_cases: list[TestSuiteWithCases],
+ build_target: BuildTargetConfiguration,
+ ):
+ """Extend the constructor with the build target's config and ExecutionResult.
Args:
+ test_suites_with_cases: The test suites with test cases to be run in this build target.
build_target: The build target's test run configuration.
"""
super(BuildTargetResult, self).__init__()
@@ -383,6 +437,23 @@ def __init__(self, build_target: BuildTargetConfiguration):
self.compiler = build_target.compiler
self.compiler_version = None
self.dpdk_version = None
+ self._test_suites_with_cases = test_suites_with_cases
+
+ def add_test_suite(
+ self,
+ test_suite_with_cases: TestSuiteWithCases,
+ ) -> "TestSuiteResult":
+ """Add and return the child result (test suite).
+
+ Args:
+ test_suite_with_cases: The test suite with test cases.
+
+ Returns:
+ The test suite's result.
+ """
+ result = TestSuiteResult(test_suite_with_cases)
+ self.child_results.append(result)
+ return result
def add_build_target_info(self, versions: BuildTargetInfo) -> None:
"""Add information about the build target gathered at runtime.
@@ -393,15 +464,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None:
self.compiler_version = versions.compiler_version
self.dpdk_version = versions.dpdk_version
- def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult":
- """Add and return the inner result (test suite).
-
- Returns:
- The test suite's result.
- """
- test_suite_result = TestSuiteResult(test_suite_name)
- self._inner_results.append(test_suite_result)
- return test_suite_result
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ for test_suite_with_cases in self._test_suites_with_cases:
+ child_result = self.add_test_suite(test_suite_with_cases)
+ child_result.update_setup(Result.BLOCK)
class TestSuiteResult(BaseResult):
@@ -410,29 +477,42 @@ class TestSuiteResult(BaseResult):
The internal list stores the results of all test cases in a given test suite.
Attributes:
- suite_name: The test suite name.
+ test_suite_name: The test suite name.
"""
- suite_name: str
+ test_suite_name: str
+ _test_suite_with_cases: TestSuiteWithCases
+ _parent_result: BuildTargetResult
+ _child_configs: list[str]
- def __init__(self, suite_name: str):
- """Extend the constructor with `suite_name`.
+ def __init__(self, test_suite_with_cases: TestSuiteWithCases):
+ """Extend the constructor with test suite's config and BuildTargetResult.
Args:
- suite_name: The test suite's name.
+ test_suite_with_cases: The test suite with test cases.
"""
super(TestSuiteResult, self).__init__()
- self.suite_name = suite_name
+ self.test_suite_name = test_suite_with_cases.test_suite_class.__name__
+ self._test_suite_with_cases = test_suite_with_cases
def add_test_case(self, test_case_name: str) -> "TestCaseResult":
- """Add and return the inner result (test case).
+ """Add and return the child result (test case).
+
+ Args:
+ test_case_name: The name of the test case.
Returns:
The test case's result.
"""
- test_case_result = TestCaseResult(test_case_name)
- self._inner_results.append(test_case_result)
- return test_case_result
+ result = TestCaseResult(test_case_name)
+ self.child_results.append(result)
+ return result
+
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ for test_case_method in self._test_suite_with_cases.test_cases:
+ child_result = self.add_test_case(test_case_method.__name__)
+ child_result.update_setup(Result.BLOCK)
class TestCaseResult(BaseResult, FixtureResult):
@@ -449,7 +529,7 @@ class TestCaseResult(BaseResult, FixtureResult):
test_case_name: str
def __init__(self, test_case_name: str):
- """Extend the constructor with `test_case_name`.
+ """Extend the constructor with test case's name and TestSuiteResult.
Args:
test_case_name: The test case's name.
@@ -470,7 +550,7 @@ def update(self, result: Result, error: Exception | None = None) -> None:
self.result = result
self.error = error
- def _get_inner_errors(self) -> list[Exception]:
+ def _get_child_errors(self) -> list[Exception]:
if self.error:
return [self.error]
return []
@@ -486,6 +566,10 @@ def add_stats(self, statistics: "Statistics") -> None:
"""
statistics += self.result
+ def _block_result(self) -> None:
+ r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
+ self.update(Result.BLOCK)
+
def __bool__(self) -> bool:
"""The test case passed only if setup, teardown and the test case itself passed."""
return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result)
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 6/7] dts: refactor logging configuration
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
` (4 preceding siblings ...)
2024-03-01 10:55 ` [PATCH v4 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš
@ 2024-03-01 10:55 ` Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 7/7] dts: improve test suite and case filtering Juraj Linkeš
` (2 subsequent siblings)
8 siblings, 0 replies; 44+ messages in thread
From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
Remove unused parts of the code and add useful features:
1. Add DTS execution stages such as execution and test suite to better
identify where in the DTS lifecycle we are when investigating logs,
2. Logging to separate files in specific stages, which is mainly useful
for having test suite logs in additional separate files.
3. Remove the dependence on the settings module which enhances the
usefulness of the logger module, as it can now be imported in more
modules.
The execution stages and the files to log to are the same for all DTS
loggers. To achieve this, we have one DTS root logger which should be
used for handling stage switching and all other loggers are children of
this DTS root logger. The DTS root logger is the one where we change the
behavior of all loggers (the stage and which files to log to) and the
child loggers just log messages under a different name.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/logger.py | 246 +++++++++++-------
dts/framework/remote_session/__init__.py | 6 +-
.../interactive_remote_session.py | 6 +-
.../remote_session/interactive_shell.py | 6 +-
.../remote_session/remote_session.py | 8 +-
dts/framework/runner.py | 23 +-
dts/framework/test_result.py | 6 +-
dts/framework/test_suite.py | 6 +-
dts/framework/testbed_model/node.py | 11 +-
dts/framework/testbed_model/os_session.py | 7 +-
.../traffic_generator/traffic_generator.py | 6 +-
dts/main.py | 3 -
12 files changed, 197 insertions(+), 137 deletions(-)
diff --git a/dts/framework/logger.py b/dts/framework/logger.py
index cfa6e8cd72..fc6c50c983 100644
--- a/dts/framework/logger.py
+++ b/dts/framework/logger.py
@@ -5,141 +5,195 @@
"""DTS logger module.
-DTS framework and TestSuite logs are saved in different log files.
+The module provides several additional features:
+
+ * The storage of DTS execution stages,
+ * Logging to console, a human-readable log file and a machine-readable log file,
+ * Optional log files for specific stages.
"""
import logging
-import os.path
-from typing import TypedDict
+from enum import auto
+from logging import FileHandler, StreamHandler
+from pathlib import Path
+from typing import ClassVar
-from .settings import SETTINGS
+from .utils import StrEnum
date_fmt = "%Y/%m/%d %H:%M:%S"
-stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s"
+dts_root_logger_name = "dts"
+
+
+class DtsStage(StrEnum):
+ """The DTS execution stage."""
+
+ #:
+ pre_execution = auto()
+ #:
+ execution_setup = auto()
+ #:
+ execution_teardown = auto()
+ #:
+ build_target_setup = auto()
+ #:
+ build_target_teardown = auto()
+ #:
+ test_suite_setup = auto()
+ #:
+ test_suite = auto()
+ #:
+ test_suite_teardown = auto()
+ #:
+ post_execution = auto()
+
+
+class DTSLogger(logging.Logger):
+ """The DTS logger class.
+
+ The class extends the :class:`~logging.Logger` class to add the DTS execution stage information
+ to log records. The stage is common to all loggers, so it's stored in a class variable.
+
+ Any time we switch to a new stage, we have the ability to log to an additional log file along
+ with a supplementary log file with machine-readable format. These two log files are used until
+ a new stage switch occurs. This is useful mainly for logging per test suite.
+ """
+ _stage: ClassVar[DtsStage] = DtsStage.pre_execution
+ _extra_file_handlers: list[FileHandler] = []
-class DTSLOG(logging.LoggerAdapter):
- """DTS logger adapter class for framework and testsuites.
+ def __init__(self, *args, **kwargs):
+ """Extend the constructor with extra file handlers."""
+ self._extra_file_handlers = []
+ super().__init__(*args, **kwargs)
- The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment
- variable control the verbosity of output. If enabled, all messages will be emitted to the
- console.
+ def makeRecord(self, *args, **kwargs) -> logging.LogRecord:
+ """Generates a record with additional stage information.
- The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment
- variable modify the directory where the logs will be stored.
+ This is the default method for the :class:`~logging.Logger` class. We extend it
+ to add stage information to the record.
- Attributes:
- node: The additional identifier. Currently unused.
- sh: The handler which emits logs to console.
- fh: The handler which emits logs to a file.
- verbose_fh: Just as fh, but logs with a different, more verbose, format.
- """
+ :meta private:
+
+ Returns:
+ record: The generated record with the stage information.
+ """
+ record = super().makeRecord(*args, **kwargs)
+ record.stage = DTSLogger._stage # type: ignore[attr-defined]
+ return record
+
+ def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None:
+ """Add logger handlers to the DTS root logger.
+
+ This method should be called only on the DTS root logger.
+ The log records from child loggers will propagate to these handlers.
- _logger: logging.Logger
- node: str
- sh: logging.StreamHandler
- fh: logging.FileHandler
- verbose_fh: logging.FileHandler
+ Three handlers are added:
- def __init__(self, logger: logging.Logger, node: str = "suite"):
- """Extend the constructor with additional handlers.
+ * A console handler,
+ * A file handler,
+ * A supplementary file handler with machine-readable logs
+ containing more debug information.
- One handler logs to the console, the other one to a file, with either a regular or verbose
- format.
+ All log messages will be logged to files. The log level of the console handler
+ is configurable with `verbose`.
Args:
- logger: The logger from which to create the logger adapter.
- node: An additional identifier. Currently unused.
+ verbose: If :data:`True`, log all messages to the console.
+ If :data:`False`, log to console with the :data:`logging.INFO` level.
+ output_dir: The directory where the log files will be located.
+ The names of the log files correspond to the name of the logger instance.
"""
- self._logger = logger
- # 1 means log everything, this will be used by file handlers if their level
- # is not set
- self._logger.setLevel(1)
+ self.setLevel(1)
- self.node = node
-
- # add handler to emit to stdout
- sh = logging.StreamHandler()
+ sh = StreamHandler()
sh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
- sh.setLevel(logging.INFO) # console handler default level
+ if not verbose:
+ sh.setLevel(logging.INFO)
+ self.addHandler(sh)
- if SETTINGS.verbose is True:
- sh.setLevel(logging.DEBUG)
+ self._add_file_handlers(Path(output_dir, self.name))
- self._logger.addHandler(sh)
- self.sh = sh
+ def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None:
+ """Set the DTS execution stage and optionally log to files.
- # prepare the output folder
- if not os.path.exists(SETTINGS.output_dir):
- os.mkdir(SETTINGS.output_dir)
+ Set the DTS execution stage of the DTSLog class and optionally add
+ file handlers to the instance if the log file name is provided.
- logging_path_prefix = os.path.join(SETTINGS.output_dir, node)
+ The file handlers log all messages. One is a regular human-readable log file and
+ the other one is a machine-readable log file with extra debug information.
- fh = logging.FileHandler(f"{logging_path_prefix}.log")
- fh.setFormatter(
- logging.Formatter(
- fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
- datefmt=date_fmt,
- )
- )
+ Args:
+ stage: The DTS stage to set.
+ log_file_path: An optional path of the log file to use. This should be a full path
+ (either relative or absolute) without suffix (which will be appended).
+ """
+ self._remove_extra_file_handlers()
- self._logger.addHandler(fh)
- self.fh = fh
+ if DTSLogger._stage != stage:
+ self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.")
+ DTSLogger._stage = stage
- # This outputs EVERYTHING, intended for post-mortem debugging
- # Also optimized for processing via AWK (awk -F '|' ...)
- verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log")
+ if log_file_path:
+ self._extra_file_handlers.extend(self._add_file_handlers(log_file_path))
+
+ def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]:
+ """Add file handlers to the DTS root logger.
+
+ Add two type of file handlers:
+
+ * A regular file handler with suffix ".log",
+ * A machine-readable file handler with suffix ".verbose.log".
+ This format provides extensive information for debugging and detailed analysis.
+
+ Args:
+ log_file_path: The full path to the log file without suffix.
+
+ Returns:
+ The newly created file handlers.
+
+ """
+ fh = FileHandler(f"{log_file_path}.log")
+ fh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
+ self.addHandler(fh)
+
+ verbose_fh = FileHandler(f"{log_file_path}.verbose.log")
verbose_fh.setFormatter(
logging.Formatter(
- fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
+ "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
"%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s",
datefmt=date_fmt,
)
)
+ self.addHandler(verbose_fh)
- self._logger.addHandler(verbose_fh)
- self.verbose_fh = verbose_fh
-
- super(DTSLOG, self).__init__(self._logger, dict(node=self.node))
-
- def logger_exit(self) -> None:
- """Remove the stream handler and the logfile handler."""
- for handler in (self.sh, self.fh, self.verbose_fh):
- handler.flush()
- self._logger.removeHandler(handler)
-
-
-class _LoggerDictType(TypedDict):
- logger: DTSLOG
- name: str
- node: str
-
+ return [fh, verbose_fh]
-# List for saving all loggers in use
-_Loggers: list[_LoggerDictType] = []
+ def _remove_extra_file_handlers(self) -> None:
+ """Remove any extra file handlers that have been added to the logger."""
+ if self._extra_file_handlers:
+ for extra_file_handler in self._extra_file_handlers:
+ self.removeHandler(extra_file_handler)
+ self._extra_file_handlers = []
-def getLogger(name: str, node: str = "suite") -> DTSLOG:
- """Get DTS logger adapter identified by name and node.
- An existing logger will be returned if one with the exact name and node already exists.
- A new one will be created and stored otherwise.
+def get_dts_logger(name: str = None) -> DTSLogger:
+ """Return a DTS logger instance identified by `name`.
Args:
- name: The name of the logger.
- node: An additional identifier for the logger.
+ name: If :data:`None`, return the DTS root logger.
+ If specified, return a child of the DTS root logger.
Returns:
- A logger uniquely identified by both name and node.
+ The DTS root logger or a child logger identified by `name`.
"""
- global _Loggers
- # return saved logger
- logger: _LoggerDictType
- for logger in _Loggers:
- if logger["name"] == name and logger["node"] == node:
- return logger["logger"]
-
- # return new logger
- dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node)
- _Loggers.append({"logger": dts_logger, "name": name, "node": node})
- return dts_logger
+ original_logger_class = logging.getLoggerClass()
+ logging.setLoggerClass(DTSLogger)
+ if name:
+ name = f"{dts_root_logger_name}.{name}"
+ else:
+ name = dts_root_logger_name
+ logger = logging.getLogger(name)
+ logging.setLoggerClass(original_logger_class)
+ return logger # type: ignore[return-value]
diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
index 51a01d6b5e..1910c81c3c 100644
--- a/dts/framework/remote_session/__init__.py
+++ b/dts/framework/remote_session/__init__.py
@@ -15,7 +15,7 @@
# pylama:ignore=W0611
from framework.config import NodeConfiguration
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from .interactive_remote_session import InteractiveRemoteSession
from .interactive_shell import InteractiveShell
@@ -26,7 +26,7 @@
def create_remote_session(
- node_config: NodeConfiguration, name: str, logger: DTSLOG
+ node_config: NodeConfiguration, name: str, logger: DTSLogger
) -> RemoteSession:
"""Factory for non-interactive remote sessions.
@@ -45,7 +45,7 @@ def create_remote_session(
def create_interactive_session(
- node_config: NodeConfiguration, logger: DTSLOG
+ node_config: NodeConfiguration, logger: DTSLogger
) -> InteractiveRemoteSession:
"""Factory for interactive remote sessions.
diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py
index 1cc82e3377..c50790db79 100644
--- a/dts/framework/remote_session/interactive_remote_session.py
+++ b/dts/framework/remote_session/interactive_remote_session.py
@@ -16,7 +16,7 @@
from framework.config import NodeConfiguration
from framework.exception import SSHConnectionError
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
class InteractiveRemoteSession:
@@ -50,11 +50,11 @@ class InteractiveRemoteSession:
username: str
password: str
session: SSHClient
- _logger: DTSLOG
+ _logger: DTSLogger
_node_config: NodeConfiguration
_transport: Transport | None
- def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None:
+ def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None:
"""Connect to the node during initialization.
Args:
diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py
index b158f963b6..5cfe202e15 100644
--- a/dts/framework/remote_session/interactive_shell.py
+++ b/dts/framework/remote_session/interactive_shell.py
@@ -20,7 +20,7 @@
from paramiko import Channel, SSHClient, channel # type: ignore[import]
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from framework.settings import SETTINGS
@@ -38,7 +38,7 @@ class InteractiveShell(ABC):
_stdin: channel.ChannelStdinFile
_stdout: channel.ChannelFile
_ssh_channel: Channel
- _logger: DTSLOG
+ _logger: DTSLogger
_timeout: float
_app_args: str
@@ -61,7 +61,7 @@ class InteractiveShell(ABC):
def __init__(
self,
interactive_session: SSHClient,
- logger: DTSLOG,
+ logger: DTSLogger,
get_privileged_command: Callable[[str], str] | None,
app_args: str = "",
timeout: float = SETTINGS.timeout,
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
index 2059f9a981..a69dc99400 100644
--- a/dts/framework/remote_session/remote_session.py
+++ b/dts/framework/remote_session/remote_session.py
@@ -9,14 +9,13 @@
the structure of the result of a command execution.
"""
-
import dataclasses
from abc import ABC, abstractmethod
from pathlib import PurePath
from framework.config import NodeConfiguration
from framework.exception import RemoteCommandExecutionError
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from framework.settings import SETTINGS
@@ -75,14 +74,14 @@ class RemoteSession(ABC):
username: str
password: str
history: list[CommandResult]
- _logger: DTSLOG
+ _logger: DTSLogger
_node_config: NodeConfiguration
def __init__(
self,
node_config: NodeConfiguration,
session_name: str,
- logger: DTSLOG,
+ logger: DTSLogger,
):
"""Connect to the node during initialization.
@@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None:
Args:
force: Force the closure of the connection. This may not clean up all resources.
"""
- self._logger.logger_exit()
self._close(force)
@abstractmethod
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 864015c350..dfee8ebd7c 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -19,9 +19,10 @@
import importlib
import inspect
-import logging
+import os
import re
import sys
+from pathlib import Path
from types import MethodType
from typing import Iterable
@@ -38,7 +39,7 @@
SSHTimeoutError,
TestCaseVerifyError,
)
-from .logger import DTSLOG, getLogger
+from .logger import DTSLogger, DtsStage, get_dts_logger
from .settings import SETTINGS
from .test_result import (
BuildTargetResult,
@@ -73,7 +74,7 @@ class DTSRunner:
"""
_configuration: Configuration
- _logger: DTSLOG
+ _logger: DTSLogger
_result: DTSResult
_test_suite_class_prefix: str
_test_suite_module_prefix: str
@@ -83,7 +84,10 @@ class DTSRunner:
def __init__(self):
"""Initialize the instance with configuration, logger, result and string constants."""
self._configuration = load_config()
- self._logger = getLogger("DTSRunner")
+ self._logger = get_dts_logger()
+ if not os.path.exists(SETTINGS.output_dir):
+ os.makedirs(SETTINGS.output_dir)
+ self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir)
self._result = DTSResult(self._logger)
self._test_suite_class_prefix = "Test"
self._test_suite_module_prefix = "tests.TestSuite_"
@@ -137,6 +141,7 @@ def run(self):
# for all Execution sections
for execution in self._configuration.executions:
+ self._logger.set_stage(DtsStage.execution_setup)
self._logger.info(
f"Running execution with SUT '{execution.system_under_test_node.name}'."
)
@@ -168,6 +173,7 @@ def run(self):
finally:
try:
+ self._logger.set_stage(DtsStage.post_execution)
for node in (sut_nodes | tg_nodes).values():
node.close()
self._result.update_teardown(Result.PASS)
@@ -426,6 +432,7 @@ def _run_execution(
finally:
try:
+ self._logger.set_stage(DtsStage.execution_teardown)
sut_node.tear_down_execution()
execution_result.update_teardown(Result.PASS)
except Exception as e:
@@ -454,6 +461,7 @@ def _run_build_target(
with the current build target.
test_suites_with_cases: The test suites with test cases to run.
"""
+ self._logger.set_stage(DtsStage.build_target_setup)
self._logger.info(f"Running build target '{build_target.name}'.")
try:
@@ -470,6 +478,7 @@ def _run_build_target(
finally:
try:
+ self._logger.set_stage(DtsStage.build_target_teardown)
sut_node.tear_down_build_target()
build_target_result.update_teardown(Result.PASS)
except Exception as e:
@@ -542,6 +551,9 @@ def _run_test_suite(
BlockingTestSuiteError: If a blocking test suite fails.
"""
test_suite_name = test_suite_with_cases.test_suite_class.__name__
+ self._logger.set_stage(
+ DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name)
+ )
test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node)
try:
self._logger.info(f"Starting test suite setup: {test_suite_name}")
@@ -560,6 +572,7 @@ def _run_test_suite(
)
finally:
try:
+ self._logger.set_stage(DtsStage.test_suite_teardown)
test_suite.tear_down_suite()
sut_node.kill_cleanup_dpdk_apps()
test_suite_result.update_teardown(Result.PASS)
@@ -591,6 +604,7 @@ def _execute_test_suite(
test_suite_result: The test suite level result object associated
with the current test suite.
"""
+ self._logger.set_stage(DtsStage.test_suite)
for test_case_method in test_cases:
test_case_name = test_case_method.__name__
test_case_result = test_suite_result.add_test_case(test_case_name)
@@ -690,5 +704,4 @@ def _exit_dts(self) -> None:
if self._logger:
self._logger.info("DTS execution has ended.")
- logging.shutdown()
sys.exit(self._result.get_return_code())
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index eedb2d20ee..28f84fd793 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -42,7 +42,7 @@
TestSuiteConfig,
)
from .exception import DTSError, ErrorSeverity
-from .logger import DTSLOG
+from .logger import DTSLogger
from .settings import SETTINGS
from .test_suite import TestSuite
@@ -237,13 +237,13 @@ class DTSResult(BaseResult):
"""
dpdk_version: str | None
- _logger: DTSLOG
+ _logger: DTSLogger
_errors: list[Exception]
_return_code: ErrorSeverity
_stats_result: Union["Statistics", None]
_stats_filename: str
- def __init__(self, logger: DTSLOG):
+ def __init__(self, logger: DTSLogger):
"""Extend the constructor with top-level specifics.
Args:
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index f9fe88093e..365f80e21a 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -21,7 +21,7 @@
from scapy.packet import Packet, Padding # type: ignore[import]
from .exception import TestCaseVerifyError
-from .logger import DTSLOG, getLogger
+from .logger import DTSLogger, get_dts_logger
from .testbed_model import Port, PortLink, SutNode, TGNode
from .utils import get_packet_summaries
@@ -61,7 +61,7 @@ class TestSuite(object):
#: Whether the test suite is blocking. A failure of a blocking test suite
#: will block the execution of all subsequent test suites in the current build target.
is_blocking: ClassVar[bool] = False
- _logger: DTSLOG
+ _logger: DTSLogger
_port_links: list[PortLink]
_sut_port_ingress: Port
_sut_port_egress: Port
@@ -88,7 +88,7 @@ def __init__(
"""
self.sut_node = sut_node
self.tg_node = tg_node
- self._logger = getLogger(self.__class__.__name__)
+ self._logger = get_dts_logger(self.__class__.__name__)
self._port_links = []
self._process_links()
self._sut_port_ingress, self._tg_port_egress = (
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
index 1a55fadf78..74061f6262 100644
--- a/dts/framework/testbed_model/node.py
+++ b/dts/framework/testbed_model/node.py
@@ -23,7 +23,7 @@
NodeConfiguration,
)
from framework.exception import ConfigurationError
-from framework.logger import DTSLOG, getLogger
+from framework.logger import DTSLogger, get_dts_logger
from framework.settings import SETTINGS
from .cpu import (
@@ -63,7 +63,7 @@ class Node(ABC):
name: str
lcores: list[LogicalCore]
ports: list[Port]
- _logger: DTSLOG
+ _logger: DTSLogger
_other_sessions: list[OSSession]
_execution_config: ExecutionConfiguration
virtual_devices: list[VirtualDevice]
@@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration):
"""
self.config = node_config
self.name = node_config.name
- self._logger = getLogger(self.name)
+ self._logger = get_dts_logger(self.name)
self.main_session = create_session(self.config, self.name, self._logger)
self._logger.info(f"Connected to node: {self.name}")
@@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession:
connection = create_session(
self.config,
session_name,
- getLogger(session_name, node=self.name),
+ get_dts_logger(session_name),
)
self._other_sessions.append(connection)
return connection
@@ -299,7 +299,6 @@ def close(self) -> None:
self.main_session.close()
for session in self._other_sessions:
session.close()
- self._logger.logger_exit()
@staticmethod
def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
@@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
return func
-def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession:
+def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession:
"""Factory for OS-aware sessions.
Args:
diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
index ac6bb5e112..6983aa4a77 100644
--- a/dts/framework/testbed_model/os_session.py
+++ b/dts/framework/testbed_model/os_session.py
@@ -21,7 +21,6 @@
the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux
and other commands for other OSs. It also translates the path to match the underlying OS.
"""
-
from abc import ABC, abstractmethod
from collections.abc import Iterable
from ipaddress import IPv4Interface, IPv6Interface
@@ -29,7 +28,7 @@
from typing import Type, TypeVar, Union
from framework.config import Architecture, NodeConfiguration, NodeInfo
-from framework.logger import DTSLOG
+from framework.logger import DTSLogger
from framework.remote_session import (
CommandResult,
InteractiveRemoteSession,
@@ -62,7 +61,7 @@ class OSSession(ABC):
_config: NodeConfiguration
name: str
- _logger: DTSLOG
+ _logger: DTSLogger
remote_session: RemoteSession
interactive_session: InteractiveRemoteSession
@@ -70,7 +69,7 @@ def __init__(
self,
node_config: NodeConfiguration,
name: str,
- logger: DTSLOG,
+ logger: DTSLogger,
):
"""Initialize the OS-aware session.
diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
index c49fbff488..d86d7fb532 100644
--- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py
+++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
@@ -13,7 +13,7 @@
from scapy.packet import Packet # type: ignore[import]
from framework.config import TrafficGeneratorConfig
-from framework.logger import DTSLOG, getLogger
+from framework.logger import DTSLogger, get_dts_logger
from framework.testbed_model.node import Node
from framework.testbed_model.port import Port
from framework.utils import get_packet_summaries
@@ -28,7 +28,7 @@ class TrafficGenerator(ABC):
_config: TrafficGeneratorConfig
_tg_node: Node
- _logger: DTSLOG
+ _logger: DTSLogger
def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
"""Initialize the traffic generator.
@@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig):
"""
self._config = config
self._tg_node = tg_node
- self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
+ self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}")
def send_packet(self, packet: Packet, port: Port) -> None:
"""Send `packet` and block until it is fully sent.
diff --git a/dts/main.py b/dts/main.py
index 1ffe8ff81f..fa878cc16e 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -6,8 +6,6 @@
"""The DTS executable."""
-import logging
-
from framework import settings
@@ -30,5 +28,4 @@ def main() -> None:
# Main program begins here
if __name__ == "__main__":
- logging.raiseExceptions = True
main()
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 7/7] dts: improve test suite and case filtering
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
` (5 preceding siblings ...)
2024-03-01 10:55 ` [PATCH v4 6/7] dts: refactor logging configuration Juraj Linkeš
@ 2024-03-01 10:55 ` Juraj Linkeš
2024-03-01 17:41 ` Jeremy Spewock
2024-03-01 16:11 ` [PATCH v4 0/7] test case blocking and logging Patrick Robb
2024-03-07 14:55 ` Thomas Monjalon
8 siblings, 1 reply; 44+ messages in thread
From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte
Cc: dev, Juraj Linkeš
The two places where we specify which test suite and test cases to run
are complimentary and not that intuitive to use. A unified way provides
a better user experience.
The syntax in test run configuration file has not changed, but the
environment variable and the command line arguments was changed to match
the config file syntax. This required changes in the settings module
which greatly simplified the parsing of the environment variables while
retaining the same functionality.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
doc/guides/tools/dts.rst | 14 ++-
dts/framework/config/__init__.py | 12 +-
dts/framework/runner.py | 18 +--
dts/framework/settings.py | 187 ++++++++++++++-----------------
dts/framework/test_suite.py | 2 +-
5 files changed, 114 insertions(+), 119 deletions(-)
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index f686ca487c..d1c3c2af7a 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,28 +215,30 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
.. code-block:: console
(dts-py3.10) $ ./main.py --help
- usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN]
+ usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
options:
-h, --help show this help message and exit
--config-file CONFIG_FILE
- [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
+ [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
--output-dir OUTPUT_DIR, --output OUTPUT_DIR
[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
-t TIMEOUT, --timeout TIMEOUT
[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
- -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None)
+ -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
--tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
--compile-timeout COMPILE_TIMEOUT
[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
- --test-cases TEST_CASES
- [DTS_TESTCASES] Comma-separated list of test cases to execute. Unknown test cases will be silently ignored. (default: )
+ --test-suite TEST_SUITE [TEST_CASES ...]
+ [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
+ variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
+ (default: [])
--re-run RE_RUN, --re_run RE_RUN
- [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs (default: 0)
+ [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index c6a93b3b89..4cb5c74059 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -35,9 +35,9 @@
import json
import os.path
-import pathlib
from dataclasses import dataclass, fields
from enum import auto, unique
+from pathlib import Path
from typing import Union
import warlock # type: ignore[import]
@@ -53,7 +53,6 @@
TrafficGeneratorConfigDict,
)
from framework.exception import ConfigurationError
-from framework.settings import SETTINGS
from framework.utils import StrEnum
@@ -571,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration":
return Configuration(executions=executions)
-def load_config() -> Configuration:
+def load_config(config_file_path: Path) -> Configuration:
"""Load DTS test run configuration from a file.
Load the YAML test run configuration file
@@ -581,13 +580,16 @@ def load_config() -> Configuration:
The YAML test run configuration file is specified in the :option:`--config-file` command line
argument or the :envvar:`DTS_CFG_FILE` environment variable.
+ Args:
+ config_file_path: The path to the YAML test run configuration file.
+
Returns:
The parsed test run configuration.
"""
- with open(SETTINGS.config_file_path, "r") as f:
+ with open(config_file_path, "r") as f:
config_data = yaml.safe_load(f)
- schema_path = os.path.join(pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json")
+ schema_path = os.path.join(Path(__file__).parent.resolve(), "conf_yaml_schema.json")
with open(schema_path, "r") as f:
schema = json.load(f)
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index dfee8ebd7c..db8e3ba96b 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -24,7 +24,7 @@
import sys
from pathlib import Path
from types import MethodType
-from typing import Iterable
+from typing import Iterable, Sequence
from .config import (
BuildTargetConfiguration,
@@ -83,7 +83,7 @@ class DTSRunner:
def __init__(self):
"""Initialize the instance with configuration, logger, result and string constants."""
- self._configuration = load_config()
+ self._configuration = load_config(SETTINGS.config_file_path)
self._logger = get_dts_logger()
if not os.path.exists(SETTINGS.output_dir):
os.makedirs(SETTINGS.output_dir)
@@ -129,7 +129,7 @@ def run(self):
#. Execution teardown
The test cases are filtered according to the specification in the test run configuration and
- the :option:`--test-cases` command line argument or
+ the :option:`--test-suite` command line argument or
the :envvar:`DTS_TESTCASES` environment variable.
"""
sut_nodes: dict[str, SutNode] = {}
@@ -147,7 +147,9 @@ def run(self):
)
execution_result = self._result.add_execution(execution)
# we don't want to modify the original config, so create a copy
- execution_test_suites = list(execution.test_suites)
+ execution_test_suites = list(
+ SETTINGS.test_suites if SETTINGS.test_suites else execution.test_suites
+ )
if not execution.skip_smoke_tests:
execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
try:
@@ -226,7 +228,7 @@ def _get_test_suites_with_cases(
test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
test_cases = []
func_test_cases, perf_test_cases = self._filter_test_cases(
- test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases)
+ test_suite_class, test_suite_config.test_cases
)
if func:
test_cases.extend(func_test_cases)
@@ -302,7 +304,7 @@ def is_test_suite(object) -> bool:
)
def _filter_test_cases(
- self, test_suite_class: type[TestSuite], test_cases_to_run: set[str]
+ self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
) -> tuple[list[MethodType], list[MethodType]]:
"""Filter `test_cases_to_run` from `test_suite_class`.
@@ -331,7 +333,9 @@ def _filter_test_cases(
(name, method) for name, method in name_method_tuples if name in test_cases_to_run
]
if len(name_method_tuples) < len(test_cases_to_run):
- missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples}
+ missing_test_cases = set(test_cases_to_run) - {
+ name for name, _ in name_method_tuples
+ }
raise ConfigurationError(
f"Test cases {missing_test_cases} not found among methods "
f"of {test_suite_class.__name__}."
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 2b8bfbe0ed..688e8679a7 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -48,10 +48,11 @@
The path to a DPDK tarball, git commit ID, tag ID or tree ID to test.
-.. option:: --test-cases
-.. envvar:: DTS_TESTCASES
+.. option:: --test-suite
+.. envvar:: DTS_TEST_SUITES
- A comma-separated list of test cases to execute. Unknown test cases will be silently ignored.
+ A test suite with test cases which may be specified multiple times.
+ In the environment variable, the suites are joined with a comma.
.. option:: --re-run, --re_run
.. envvar:: DTS_RERUN
@@ -71,83 +72,13 @@
import argparse
import os
-from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass, field
from pathlib import Path
-from typing import Any, TypeVar
+from typing import Any
+from .config import TestSuiteConfig
from .utils import DPDKGitTarball
-_T = TypeVar("_T")
-
-
-def _env_arg(env_var: str) -> Any:
- """A helper method augmenting the argparse Action with environment variables.
-
- If the supplied environment variable is defined, then the default value
- of the argument is modified. This satisfies the priority order of
- command line argument > environment variable > default value.
-
- Arguments with no values (flags) should be defined using the const keyword argument
- (True or False). When the argument is specified, it will be set to const, if not specified,
- the default will be stored (possibly modified by the corresponding environment variable).
-
- Other arguments work the same as default argparse arguments, that is using
- the default 'store' action.
-
- Returns:
- The modified argparse.Action.
- """
-
- class _EnvironmentArgument(argparse.Action):
- def __init__(
- self,
- option_strings: Sequence[str],
- dest: str,
- nargs: str | int | None = None,
- const: bool | None = None,
- default: Any = None,
- type: Callable[[str], _T | argparse.FileType | None] = None,
- choices: Iterable[_T] | None = None,
- required: bool = False,
- help: str | None = None,
- metavar: str | tuple[str, ...] | None = None,
- ) -> None:
- env_var_value = os.environ.get(env_var)
- default = env_var_value or default
- if const is not None:
- nargs = 0
- default = const if env_var_value else default
- type = None
- choices = None
- metavar = None
- super(_EnvironmentArgument, self).__init__(
- option_strings,
- dest,
- nargs=nargs,
- const=const,
- default=default,
- type=type,
- choices=choices,
- required=required,
- help=help,
- metavar=metavar,
- )
-
- def __call__(
- self,
- parser: argparse.ArgumentParser,
- namespace: argparse.Namespace,
- values: Any,
- option_string: str = None,
- ) -> None:
- if self.const is not None:
- setattr(namespace, self.dest, self.const)
- else:
- setattr(namespace, self.dest, values)
-
- return _EnvironmentArgument
-
@dataclass(slots=True)
class Settings:
@@ -171,7 +102,7 @@ class Settings:
#:
compile_timeout: float = 1200
#:
- test_cases: list[str] = field(default_factory=list)
+ test_suites: list[TestSuiteConfig] = field(default_factory=list)
#:
re_run: int = 0
@@ -180,6 +111,31 @@ class Settings:
def _get_parser() -> argparse.ArgumentParser:
+ """Create the argument parser for DTS.
+
+ Command line options take precedence over environment variables, which in turn take precedence
+ over default values.
+
+ Returns:
+ argparse.ArgumentParser: The configured argument parser with defined options.
+ """
+
+ def env_arg(env_var: str, default: Any) -> Any:
+ """A helper function augmenting the argparse with environment variables.
+
+ If the supplied environment variable is defined, then the default value
+ of the argument is modified. This satisfies the priority order of
+ command line argument > environment variable > default value.
+
+ Args:
+ env_var: Environment variable name.
+ default: Default value.
+
+ Returns:
+ Environment variable or default value.
+ """
+ return os.environ.get(env_var) or default
+
parser = argparse.ArgumentParser(
description="Run DPDK test suites. All options may be specified with the environment "
"variables provided in brackets. Command line arguments have higher priority.",
@@ -188,25 +144,23 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--config-file",
- action=_env_arg("DTS_CFG_FILE"),
- default=SETTINGS.config_file_path,
+ default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
type=Path,
- help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets.",
+ help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
+ "SUTs and targets.",
)
parser.add_argument(
"--output-dir",
"--output",
- action=_env_arg("DTS_OUTPUT_DIR"),
- default=SETTINGS.output_dir,
+ default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
)
parser.add_argument(
"-t",
"--timeout",
- action=_env_arg("DTS_TIMEOUT"),
- default=SETTINGS.timeout,
+ default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
type=float,
help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
)
@@ -214,9 +168,8 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"-v",
"--verbose",
- action=_env_arg("DTS_VERBOSE"),
- default=SETTINGS.verbose,
- const=True,
+ action="store_true",
+ default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
"to the console.",
)
@@ -224,8 +177,8 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"-s",
"--skip-setup",
- action=_env_arg("DTS_SKIP_SETUP"),
- const=True,
+ action="store_true",
+ default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
)
@@ -233,8 +186,7 @@ def _get_parser() -> argparse.ArgumentParser:
"--tarball",
"--snapshot",
"--git-ref",
- action=_env_arg("DTS_DPDK_TARBALL"),
- default=SETTINGS.dpdk_tarball_path,
+ default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
type=Path,
help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
"tag ID or tree ID to test. To test local changes, first commit them, "
@@ -243,36 +195,71 @@ def _get_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--compile-timeout",
- action=_env_arg("DTS_COMPILE_TIMEOUT"),
- default=SETTINGS.compile_timeout,
+ default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
type=float,
help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
)
parser.add_argument(
- "--test-cases",
- action=_env_arg("DTS_TESTCASES"),
- default="",
- help="[DTS_TESTCASES] Comma-separated list of test cases to execute.",
+ "--test-suite",
+ action="append",
+ nargs="+",
+ metavar=("TEST_SUITE", "TEST_CASES"),
+ default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
+ help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
+ "The first parameter is the test suite name, and the rest are test case names, "
+ "which are optional. May be specified multiple times. To specify multiple test suites in "
+ "the environment variable, join the lists with a comma. "
+ "Examples: "
+ "--test-suite suite case case --test-suite suite case ... | "
+ "DTS_TEST_SUITES='suite case case, suite case, ...' | "
+ "--test-suite suite --test-suite suite case ... | "
+ "DTS_TEST_SUITES='suite, suite case, ...'",
)
parser.add_argument(
"--re-run",
"--re_run",
- action=_env_arg("DTS_RERUN"),
- default=SETTINGS.re_run,
+ default=env_arg("DTS_RERUN", SETTINGS.re_run),
type=int,
help="[DTS_RERUN] Re-run each test case the specified number of times "
- "if a test failure occurs",
+ "if a test failure occurs.",
)
return parser
+def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
+ """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
+
+ Args:
+ args: The arguments to process. The args is a string from an environment variable
+ or a list of from the user input containing tests suites with tests cases,
+ each of which is a list of [test_suite, test_case, test_case, ...].
+
+ Returns:
+ A list of test suite configurations to execute.
+ """
+ if isinstance(args, str):
+ # Environment variable in the form of "suite case case, suite case, suite, ..."
+ args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
+
+ test_suites_to_run = []
+ for suite_with_cases in args:
+ test_suites_to_run.append(
+ TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
+ )
+
+ return test_suites_to_run
+
+
def get_settings() -> Settings:
"""Create new settings with inputs from the user.
The inputs are taken from the command line and from environment variables.
+
+ Returns:
+ The new settings object.
"""
parsed_args = _get_parser().parse_args()
return Settings(
@@ -287,6 +274,6 @@ def get_settings() -> Settings:
else Path(parsed_args.tarball)
),
compile_timeout=parsed_args.compile_timeout,
- test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []),
+ test_suites=_process_test_suites(parsed_args.test_suite),
re_run=parsed_args.re_run,
)
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 365f80e21a..1957ea7328 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -40,7 +40,7 @@ class TestSuite(object):
and functional test cases (all other test cases).
By default, all test cases will be executed. A list of testcase names may be specified
- in the YAML test run configuration file and in the :option:`--test-cases` command line argument
+ in the YAML test run configuration file and in the :option:`--test-suite` command line argument
or in the :envvar:`DTS_TESTCASES` environment variable to filter which test cases to run.
The union of both lists will be used. Any unknown test cases from the latter lists
will be silently ignored.
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 0/7] test case blocking and logging
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
` (6 preceding siblings ...)
2024-03-01 10:55 ` [PATCH v4 7/7] dts: improve test suite and case filtering Juraj Linkeš
@ 2024-03-01 16:11 ` Patrick Robb
2024-03-07 14:55 ` Thomas Monjalon
8 siblings, 0 replies; 44+ messages in thread
From: Patrick Robb @ 2024-03-01 16:11 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, jspewock, paul.szczepanek,
Luca.Vizzarro, npratte, dev
[-- Attachment #1: Type: text/plain, Size: 4111 bytes --]
The Community CI Testing Lab had an infra failure this morning and some
patches including yours were affected with false failures. The issue is now
resolved and we are rerunning the tests in question for all patches
submitted today.
On Fri, Mar 1, 2024 at 5:55 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> We currently don't record test case results that couldn't be executed
> because of a previous failure, such as when a test suite setup failed,
> resulting in no executed test cases.
>
> In order to record the test cases that couldn't be executed, we must
> know the lists of test suites and test cases ahead of the actual test
> suite execution, as an error could occur before we even start executing
> test suites.
>
> In addition, the patch series contains two refactors and one
> improvement.
>
> The first refactor is closely related. The dts.py was renamed to
> runner.py and given a clear purpose - running the test suites and all
> other orchestration needed to run test suites. The logic for this was
> not all in the original dts.py module and it was brought there. The
> runner is also responsible for recording results, which is the blocked
> test cases are recorded.
>
> The other refactor, logging, is related to the first refactor. The
> logging module was simplified while extending capabilities. Each test
> suite logs into its own log file in addition to the main log file which
> the runner must handle (as it knows when we start executing particular
> test suites). The runner also handles the switching between execution
> stages for the purposes of logging.
>
> The one aforementioned improvement is in unifying how we specify test
> cases in the conf.yaml file and in the environment variable/command line
> argument.
>
> v2:
> Rebase and update of the whole patch. There are changes in all parts of
> the code, mainly improving the design and logic.
> Also added the last patch which improves test suite specification on the
> cmdline.
>
> v3:
> Improved variables in _get_test_suite_class along with docstring.
> Fixed smoke test suite not being added into the list of test suites to
> be executed.
>
> v4:
> Added the checking of CamelCase convention when discovering test cases.
> Also added additional test stages. The stages were split into setup and
> teardown.
>
> Juraj Linkeš (7):
> dts: convert dts.py methods to class
> dts: move test suite execution logic to DTSRunner
> dts: filter test suites in executions
> dts: reorganize test result
> dts: block all test cases when earlier setup fails
> dts: refactor logging configuration
> dts: improve test suite and case filtering
>
> doc/guides/tools/dts.rst | 14 +-
> dts/framework/config/__init__.py | 36 +-
> dts/framework/config/conf_yaml_schema.json | 2 +-
> dts/framework/dts.py | 338 ---------
> dts/framework/logger.py | 246 +++---
> dts/framework/remote_session/__init__.py | 6 +-
> .../interactive_remote_session.py | 6 +-
> .../remote_session/interactive_shell.py | 6 +-
> .../remote_session/remote_session.py | 8 +-
> dts/framework/runner.py | 711 ++++++++++++++++++
> dts/framework/settings.py | 188 +++--
> dts/framework/test_result.py | 565 ++++++++------
> dts/framework/test_suite.py | 239 +-----
> dts/framework/testbed_model/node.py | 11 +-
> dts/framework/testbed_model/os_session.py | 7 +-
> .../traffic_generator/traffic_generator.py | 6 +-
> dts/main.py | 9 +-
> dts/pyproject.toml | 3 +
> dts/tests/TestSuite_os_udp.py | 2 +-
> dts/tests/TestSuite_smoke_tests.py | 2 +-
> 20 files changed, 1375 insertions(+), 1030 deletions(-)
> delete mode 100644 dts/framework/dts.py
> create mode 100644 dts/framework/runner.py
>
> --
> 2.34.1
>
>
[-- Attachment #2: Type: text/html, Size: 4804 bytes --]
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 7/7] dts: improve test suite and case filtering
2024-03-01 10:55 ` [PATCH v4 7/7] dts: improve test suite and case filtering Juraj Linkeš
@ 2024-03-01 17:41 ` Jeremy Spewock
2024-03-01 17:43 ` Jeremy Spewock
0 siblings, 1 reply; 44+ messages in thread
From: Jeremy Spewock @ 2024-03-01 17:41 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dev
For the whole series:
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
On Fri, Mar 1, 2024 at 5:55 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> The two places where we specify which test suite and test cases to run
> are complimentary and not that intuitive to use. A unified way provides
> a better user experience.
>
> The syntax in test run configuration file has not changed, but the
> environment variable and the command line arguments was changed to match
> the config file syntax. This required changes in the settings module
> which greatly simplified the parsing of the environment variables while
> retaining the same functionality.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
> doc/guides/tools/dts.rst | 14 ++-
> dts/framework/config/__init__.py | 12 +-
> dts/framework/runner.py | 18 +--
> dts/framework/settings.py | 187 ++++++++++++++-----------------
> dts/framework/test_suite.py | 2 +-
> 5 files changed, 114 insertions(+), 119 deletions(-)
>
> diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
> index f686ca487c..d1c3c2af7a 100644
> --- a/doc/guides/tools/dts.rst
> +++ b/doc/guides/tools/dts.rst
> @@ -215,28 +215,30 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
> .. code-block:: console
>
> (dts-py3.10) $ ./main.py --help
> - usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN]
> + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
>
> Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
>
> options:
> -h, --help show this help message and exit
> --config-file CONFIG_FILE
> - [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
> + [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
> --output-dir OUTPUT_DIR, --output OUTPUT_DIR
> [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
> -t TIMEOUT, --timeout TIMEOUT
> [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
> -v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
> - -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None)
> + -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
> --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
> [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
> --compile-timeout COMPILE_TIMEOUT
> [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
> - --test-cases TEST_CASES
> - [DTS_TESTCASES] Comma-separated list of test cases to execute. Unknown test cases will be silently ignored. (default: )
> + --test-suite TEST_SUITE [TEST_CASES ...]
> + [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
> + variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
> + (default: [])
> --re-run RE_RUN, --re_run RE_RUN
> - [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs (default: 0)
> + [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
>
>
> The brackets contain the names of environment variables that set the same thing.
> diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
> index c6a93b3b89..4cb5c74059 100644
> --- a/dts/framework/config/__init__.py
> +++ b/dts/framework/config/__init__.py
> @@ -35,9 +35,9 @@
>
> import json
> import os.path
> -import pathlib
> from dataclasses import dataclass, fields
> from enum import auto, unique
> +from pathlib import Path
> from typing import Union
>
> import warlock # type: ignore[import]
> @@ -53,7 +53,6 @@
> TrafficGeneratorConfigDict,
> )
> from framework.exception import ConfigurationError
> -from framework.settings import SETTINGS
> from framework.utils import StrEnum
>
>
> @@ -571,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration":
> return Configuration(executions=executions)
>
>
> -def load_config() -> Configuration:
> +def load_config(config_file_path: Path) -> Configuration:
> """Load DTS test run configuration from a file.
>
> Load the YAML test run configuration file
> @@ -581,13 +580,16 @@ def load_config() -> Configuration:
> The YAML test run configuration file is specified in the :option:`--config-file` command line
> argument or the :envvar:`DTS_CFG_FILE` environment variable.
>
> + Args:
> + config_file_path: The path to the YAML test run configuration file.
> +
> Returns:
> The parsed test run configuration.
> """
> - with open(SETTINGS.config_file_path, "r") as f:
> + with open(config_file_path, "r") as f:
> config_data = yaml.safe_load(f)
>
> - schema_path = os.path.join(pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json")
> + schema_path = os.path.join(Path(__file__).parent.resolve(), "conf_yaml_schema.json")
>
> with open(schema_path, "r") as f:
> schema = json.load(f)
> diff --git a/dts/framework/runner.py b/dts/framework/runner.py
> index dfee8ebd7c..db8e3ba96b 100644
> --- a/dts/framework/runner.py
> +++ b/dts/framework/runner.py
> @@ -24,7 +24,7 @@
> import sys
> from pathlib import Path
> from types import MethodType
> -from typing import Iterable
> +from typing import Iterable, Sequence
>
> from .config import (
> BuildTargetConfiguration,
> @@ -83,7 +83,7 @@ class DTSRunner:
>
> def __init__(self):
> """Initialize the instance with configuration, logger, result and string constants."""
> - self._configuration = load_config()
> + self._configuration = load_config(SETTINGS.config_file_path)
> self._logger = get_dts_logger()
> if not os.path.exists(SETTINGS.output_dir):
> os.makedirs(SETTINGS.output_dir)
> @@ -129,7 +129,7 @@ def run(self):
> #. Execution teardown
>
> The test cases are filtered according to the specification in the test run configuration and
> - the :option:`--test-cases` command line argument or
> + the :option:`--test-suite` command line argument or
> the :envvar:`DTS_TESTCASES` environment variable.
> """
> sut_nodes: dict[str, SutNode] = {}
> @@ -147,7 +147,9 @@ def run(self):
> )
> execution_result = self._result.add_execution(execution)
> # we don't want to modify the original config, so create a copy
> - execution_test_suites = list(execution.test_suites)
> + execution_test_suites = list(
> + SETTINGS.test_suites if SETTINGS.test_suites else execution.test_suites
> + )
> if not execution.skip_smoke_tests:
> execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")]
> try:
> @@ -226,7 +228,7 @@ def _get_test_suites_with_cases(
> test_suite_class = self._get_test_suite_class(test_suite_config.test_suite)
> test_cases = []
> func_test_cases, perf_test_cases = self._filter_test_cases(
> - test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases)
> + test_suite_class, test_suite_config.test_cases
> )
> if func:
> test_cases.extend(func_test_cases)
> @@ -302,7 +304,7 @@ def is_test_suite(object) -> bool:
> )
>
> def _filter_test_cases(
> - self, test_suite_class: type[TestSuite], test_cases_to_run: set[str]
> + self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
> ) -> tuple[list[MethodType], list[MethodType]]:
> """Filter `test_cases_to_run` from `test_suite_class`.
>
> @@ -331,7 +333,9 @@ def _filter_test_cases(
> (name, method) for name, method in name_method_tuples if name in test_cases_to_run
> ]
> if len(name_method_tuples) < len(test_cases_to_run):
> - missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples}
> + missing_test_cases = set(test_cases_to_run) - {
> + name for name, _ in name_method_tuples
> + }
> raise ConfigurationError(
> f"Test cases {missing_test_cases} not found among methods "
> f"of {test_suite_class.__name__}."
> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> index 2b8bfbe0ed..688e8679a7 100644
> --- a/dts/framework/settings.py
> +++ b/dts/framework/settings.py
> @@ -48,10 +48,11 @@
>
> The path to a DPDK tarball, git commit ID, tag ID or tree ID to test.
>
> -.. option:: --test-cases
> -.. envvar:: DTS_TESTCASES
> +.. option:: --test-suite
> +.. envvar:: DTS_TEST_SUITES
>
> - A comma-separated list of test cases to execute. Unknown test cases will be silently ignored.
> + A test suite with test cases which may be specified multiple times.
> + In the environment variable, the suites are joined with a comma.
>
> .. option:: --re-run, --re_run
> .. envvar:: DTS_RERUN
> @@ -71,83 +72,13 @@
>
> import argparse
> import os
> -from collections.abc import Callable, Iterable, Sequence
> from dataclasses import dataclass, field
> from pathlib import Path
> -from typing import Any, TypeVar
> +from typing import Any
>
> +from .config import TestSuiteConfig
> from .utils import DPDKGitTarball
>
> -_T = TypeVar("_T")
> -
> -
> -def _env_arg(env_var: str) -> Any:
> - """A helper method augmenting the argparse Action with environment variables.
> -
> - If the supplied environment variable is defined, then the default value
> - of the argument is modified. This satisfies the priority order of
> - command line argument > environment variable > default value.
> -
> - Arguments with no values (flags) should be defined using the const keyword argument
> - (True or False). When the argument is specified, it will be set to const, if not specified,
> - the default will be stored (possibly modified by the corresponding environment variable).
> -
> - Other arguments work the same as default argparse arguments, that is using
> - the default 'store' action.
> -
> - Returns:
> - The modified argparse.Action.
> - """
> -
> - class _EnvironmentArgument(argparse.Action):
> - def __init__(
> - self,
> - option_strings: Sequence[str],
> - dest: str,
> - nargs: str | int | None = None,
> - const: bool | None = None,
> - default: Any = None,
> - type: Callable[[str], _T | argparse.FileType | None] = None,
> - choices: Iterable[_T] | None = None,
> - required: bool = False,
> - help: str | None = None,
> - metavar: str | tuple[str, ...] | None = None,
> - ) -> None:
> - env_var_value = os.environ.get(env_var)
> - default = env_var_value or default
> - if const is not None:
> - nargs = 0
> - default = const if env_var_value else default
> - type = None
> - choices = None
> - metavar = None
> - super(_EnvironmentArgument, self).__init__(
> - option_strings,
> - dest,
> - nargs=nargs,
> - const=const,
> - default=default,
> - type=type,
> - choices=choices,
> - required=required,
> - help=help,
> - metavar=metavar,
> - )
> -
> - def __call__(
> - self,
> - parser: argparse.ArgumentParser,
> - namespace: argparse.Namespace,
> - values: Any,
> - option_string: str = None,
> - ) -> None:
> - if self.const is not None:
> - setattr(namespace, self.dest, self.const)
> - else:
> - setattr(namespace, self.dest, values)
> -
> - return _EnvironmentArgument
> -
>
> @dataclass(slots=True)
> class Settings:
> @@ -171,7 +102,7 @@ class Settings:
> #:
> compile_timeout: float = 1200
> #:
> - test_cases: list[str] = field(default_factory=list)
> + test_suites: list[TestSuiteConfig] = field(default_factory=list)
> #:
> re_run: int = 0
>
> @@ -180,6 +111,31 @@ class Settings:
>
>
> def _get_parser() -> argparse.ArgumentParser:
> + """Create the argument parser for DTS.
> +
> + Command line options take precedence over environment variables, which in turn take precedence
> + over default values.
> +
> + Returns:
> + argparse.ArgumentParser: The configured argument parser with defined options.
> + """
> +
> + def env_arg(env_var: str, default: Any) -> Any:
> + """A helper function augmenting the argparse with environment variables.
> +
> + If the supplied environment variable is defined, then the default value
> + of the argument is modified. This satisfies the priority order of
> + command line argument > environment variable > default value.
> +
> + Args:
> + env_var: Environment variable name.
> + default: Default value.
> +
> + Returns:
> + Environment variable or default value.
> + """
> + return os.environ.get(env_var) or default
> +
> parser = argparse.ArgumentParser(
> description="Run DPDK test suites. All options may be specified with the environment "
> "variables provided in brackets. Command line arguments have higher priority.",
> @@ -188,25 +144,23 @@ def _get_parser() -> argparse.ArgumentParser:
>
> parser.add_argument(
> "--config-file",
> - action=_env_arg("DTS_CFG_FILE"),
> - default=SETTINGS.config_file_path,
> + default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
> type=Path,
> - help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets.",
> + help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
> + "SUTs and targets.",
> )
>
> parser.add_argument(
> "--output-dir",
> "--output",
> - action=_env_arg("DTS_OUTPUT_DIR"),
> - default=SETTINGS.output_dir,
> + default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
> help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
> )
>
> parser.add_argument(
> "-t",
> "--timeout",
> - action=_env_arg("DTS_TIMEOUT"),
> - default=SETTINGS.timeout,
> + default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
> type=float,
> help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
> )
> @@ -214,9 +168,8 @@ def _get_parser() -> argparse.ArgumentParser:
> parser.add_argument(
> "-v",
> "--verbose",
> - action=_env_arg("DTS_VERBOSE"),
> - default=SETTINGS.verbose,
> - const=True,
> + action="store_true",
> + default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
> help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
> "to the console.",
> )
> @@ -224,8 +177,8 @@ def _get_parser() -> argparse.ArgumentParser:
> parser.add_argument(
> "-s",
> "--skip-setup",
> - action=_env_arg("DTS_SKIP_SETUP"),
> - const=True,
> + action="store_true",
> + default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
> help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
> )
>
> @@ -233,8 +186,7 @@ def _get_parser() -> argparse.ArgumentParser:
> "--tarball",
> "--snapshot",
> "--git-ref",
> - action=_env_arg("DTS_DPDK_TARBALL"),
> - default=SETTINGS.dpdk_tarball_path,
> + default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
> type=Path,
> help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
> "tag ID or tree ID to test. To test local changes, first commit them, "
> @@ -243,36 +195,71 @@ def _get_parser() -> argparse.ArgumentParser:
>
> parser.add_argument(
> "--compile-timeout",
> - action=_env_arg("DTS_COMPILE_TIMEOUT"),
> - default=SETTINGS.compile_timeout,
> + default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
> type=float,
> help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
> )
>
> parser.add_argument(
> - "--test-cases",
> - action=_env_arg("DTS_TESTCASES"),
> - default="",
> - help="[DTS_TESTCASES] Comma-separated list of test cases to execute.",
> + "--test-suite",
> + action="append",
> + nargs="+",
> + metavar=("TEST_SUITE", "TEST_CASES"),
> + default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
> + help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
> + "The first parameter is the test suite name, and the rest are test case names, "
> + "which are optional. May be specified multiple times. To specify multiple test suites in "
> + "the environment variable, join the lists with a comma. "
> + "Examples: "
> + "--test-suite suite case case --test-suite suite case ... | "
> + "DTS_TEST_SUITES='suite case case, suite case, ...' | "
> + "--test-suite suite --test-suite suite case ... | "
> + "DTS_TEST_SUITES='suite, suite case, ...'",
> )
>
> parser.add_argument(
> "--re-run",
> "--re_run",
> - action=_env_arg("DTS_RERUN"),
> - default=SETTINGS.re_run,
> + default=env_arg("DTS_RERUN", SETTINGS.re_run),
> type=int,
> help="[DTS_RERUN] Re-run each test case the specified number of times "
> - "if a test failure occurs",
> + "if a test failure occurs.",
> )
>
> return parser
>
>
> +def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
> + """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
> +
> + Args:
> + args: The arguments to process. The args is a string from an environment variable
> + or a list of from the user input containing tests suites with tests cases,
> + each of which is a list of [test_suite, test_case, test_case, ...].
> +
> + Returns:
> + A list of test suite configurations to execute.
> + """
> + if isinstance(args, str):
> + # Environment variable in the form of "suite case case, suite case, suite, ..."
> + args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
> +
> + test_suites_to_run = []
> + for suite_with_cases in args:
> + test_suites_to_run.append(
> + TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
> + )
> +
> + return test_suites_to_run
> +
> +
> def get_settings() -> Settings:
> """Create new settings with inputs from the user.
>
> The inputs are taken from the command line and from environment variables.
> +
> + Returns:
> + The new settings object.
> """
> parsed_args = _get_parser().parse_args()
> return Settings(
> @@ -287,6 +274,6 @@ def get_settings() -> Settings:
> else Path(parsed_args.tarball)
> ),
> compile_timeout=parsed_args.compile_timeout,
> - test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []),
> + test_suites=_process_test_suites(parsed_args.test_suite),
> re_run=parsed_args.re_run,
> )
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index 365f80e21a..1957ea7328 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -40,7 +40,7 @@ class TestSuite(object):
> and functional test cases (all other test cases).
>
> By default, all test cases will be executed. A list of testcase names may be specified
> - in the YAML test run configuration file and in the :option:`--test-cases` command line argument
> + in the YAML test run configuration file and in the :option:`--test-suite` command line argument
> or in the :envvar:`DTS_TESTCASES` environment variable to filter which test cases to run.
> The union of both lists will be used. Any unknown test cases from the latter lists
> will be silently ignored.
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 7/7] dts: improve test suite and case filtering
2024-03-01 17:41 ` Jeremy Spewock
@ 2024-03-01 17:43 ` Jeremy Spewock
2024-03-07 11:04 ` Thomas Monjalon
0 siblings, 1 reply; 44+ messages in thread
From: Jeremy Spewock @ 2024-03-01 17:43 UTC (permalink / raw)
To: Juraj Linkeš
Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dev
On Fri, Mar 1, 2024 at 12:41 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
> For the whole series:
>
> Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
>
Apologies, I forgot to remove the content of the patch when I added my tag.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 7/7] dts: improve test suite and case filtering
2024-03-01 17:43 ` Jeremy Spewock
@ 2024-03-07 11:04 ` Thomas Monjalon
0 siblings, 0 replies; 44+ messages in thread
From: Thomas Monjalon @ 2024-03-07 11:04 UTC (permalink / raw)
To: Jeremy Spewock
Cc: Juraj Linkeš,
dev, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro,
npratte
01/03/2024 18:43, Jeremy Spewock:
> On Fri, Mar 1, 2024 at 12:41 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
> >
> > For the whole series:
> >
> > Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
> >
> Apologies, I forgot to remove the content of the patch when I added my tag.
The best is to keep the commit message,
and put your tag below others so we collect review/ack/test tags.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 0/7] test case blocking and logging
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
` (7 preceding siblings ...)
2024-03-01 16:11 ` [PATCH v4 0/7] test case blocking and logging Patrick Robb
@ 2024-03-07 14:55 ` Thomas Monjalon
8 siblings, 0 replies; 44+ messages in thread
From: Thomas Monjalon @ 2024-03-07 14:55 UTC (permalink / raw)
To: Juraj Linkeš
Cc: Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
Luca.Vizzarro, npratte, dev
> Juraj Linkeš (7):
> dts: convert dts.py methods to class
> dts: move test suite execution logic to DTSRunner
> dts: filter test suites in executions
> dts: reorganize test result
> dts: block all test cases when earlier setup fails
> dts: refactor logging configuration
> dts: improve test suite and case filtering
Applied, thanks.
^ permalink raw reply [flat|nested] 44+ messages in thread
end of thread, other threads:[~2024-03-07 14:55 UTC | newest]
Thread overview: 44+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 1/5] dts: convert dts.py methods to class Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 2/5] dts: move test suite execution logic to DTSRunner Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 3/5] dts: process test suites at the beginning of run Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 4/5] dts: block all testcases when earlier setup fails Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 5/5] dts: refactor logging configuration Juraj Linkeš
2024-01-08 18:47 ` [RFC PATCH v1 0/5] test case blocking and logging Jeremy Spewock
2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 1/7] dts: convert dts.py methods to class Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 3/7] dts: filter test suites in executions Juraj Linkeš
2024-02-12 16:44 ` Jeremy Spewock
2024-02-14 9:55 ` Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 4/7] dts: reorganize test result Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš
2024-02-06 14:57 ` [PATCH v2 6/7] dts: refactor logging configuration Juraj Linkeš
2024-02-12 16:45 ` Jeremy Spewock
2024-02-14 7:49 ` Juraj Linkeš
2024-02-14 16:51 ` Jeremy Spewock
2024-02-06 14:57 ` [PATCH v2 7/7] dts: improve test suite and case filtering Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 1/7] dts: convert dts.py methods to class Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 3/7] dts: filter test suites in executions Juraj Linkeš
2024-02-27 21:21 ` Jeremy Spewock
2024-02-28 9:16 ` Juraj Linkeš
2024-02-23 7:54 ` [PATCH v3 4/7] dts: reorganize test result Juraj Linkeš
2024-02-23 7:55 ` [PATCH v3 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš
2024-02-23 7:55 ` [PATCH v3 6/7] dts: refactor logging configuration Juraj Linkeš
2024-02-23 7:55 ` [PATCH v3 7/7] dts: improve test suite and case filtering Juraj Linkeš
2024-02-27 21:24 ` [PATCH v3 0/7] test case blocking and logging Jeremy Spewock
2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 1/7] dts: convert dts.py methods to class Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 3/7] dts: filter test suites in executions Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 4/7] dts: reorganize test result Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 6/7] dts: refactor logging configuration Juraj Linkeš
2024-03-01 10:55 ` [PATCH v4 7/7] dts: improve test suite and case filtering Juraj Linkeš
2024-03-01 17:41 ` Jeremy Spewock
2024-03-01 17:43 ` Jeremy Spewock
2024-03-07 11:04 ` Thomas Monjalon
2024-03-01 16:11 ` [PATCH v4 0/7] test case blocking and logging Patrick Robb
2024-03-07 14:55 ` Thomas Monjalon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).