* [RFC PATCH v1 0/5] test case blocking and logging
@ 2023-12-20 10:33 Juraj Linkeš
2023-12-20 10:33 ` [RFC PATCH v1 1/5] dts: convert dts.py methods to class Juraj Linkeš
` (8 more replies)
0 siblings, 9 replies; 44+ messages in thread
From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw)
To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek,
yoan.picchi, Luca.Vizzarro
Cc: dev, Juraj Linkeš
We currently don't store test cases that couldn't be executed because of
a previous failure, such as when a test suite setup failed, resulting in
no executed test cases.
In order to record the test cases that couldn't be executed, we must
know the lists of test suites and test cases ahead of the actual test
suite execution, as an error could occur before we even start executing
test suites.
In addition, the patch series contains two refactors.
The first refactor is closely related. The dts.py was renamed to
runner.py and given a clear purpose - running the test suites and all
other orchestration needed to run test suites. The logic for this was
not all in the original dts.py module and it was brought there. The
runner is also responsible for recording results, which is the blocked
test cases are recorded.
The other refactor, logging, is related to the first refactor. The
logging module was simplified while extending capabilities. Each test
suite logs into its own log file in addition to the main log file which
the runner must handle (as it knows when we start executing particular
test suites). The runner also handles the switching between execution
stages for the purposes of logging.
Juraj Linkeš (5):
dts: convert dts.py methods to class
dts: move test suite execution logic to DTSRunner
dts: process test suites at the beginning of run
dts: block all testcases when earlier setup fails
dts: refactor logging configuration
dts/framework/config/__init__.py | 8 +-
dts/framework/dts.py | 228 --------
dts/framework/logger.py | 162 +++---
dts/framework/remote_session/__init__.py | 4 +-
dts/framework/remote_session/os_session.py | 6 +-
.../remote_session/remote/__init__.py | 7 +-
.../remote/interactive_remote_session.py | 7 +-
.../remote/interactive_shell.py | 7 +-
.../remote_session/remote/remote_session.py | 8 +-
.../remote_session/remote/ssh_session.py | 5 +-
dts/framework/runner.py | 499 ++++++++++++++++++
dts/framework/test_result.py | 365 +++++++------
dts/framework/test_suite.py | 193 +------
dts/framework/testbed_model/node.py | 10 +-
dts/framework/testbed_model/scapy.py | 6 +-
.../testbed_model/traffic_generator.py | 5 +-
dts/main.py | 12 +-
dts/tests/TestSuite_smoke_tests.py | 2 +-
18 files changed, 830 insertions(+), 704 deletions(-)
delete mode 100644 dts/framework/dts.py
create mode 100644 dts/framework/runner.py
--
2.34.1
^ permalink raw reply [flat|nested] 44+ messages in thread* [RFC PATCH v1 1/5] dts: convert dts.py methods to class 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš @ 2023-12-20 10:33 ` Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 2/5] dts: move test suite execution logic to DTSRunner Juraj Linkeš ` (7 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, yoan.picchi, Luca.Vizzarro Cc: dev, Juraj Linkeš The dts.py module deviates from the rest of the code without a clear reason. Converting it into a class and using better naming will improve organization and code readability. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/config/__init__.py | 3 - dts/framework/dts.py | 228 ----------------------------- dts/framework/runner.py | 243 +++++++++++++++++++++++++++++++ dts/main.py | 6 +- 4 files changed, 247 insertions(+), 233 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index 9b32cf0532..497847afb9 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -314,6 +314,3 @@ def load_config() -> Configuration: config: dict[str, Any] = warlock.model_factory(schema, name="_Config")(config_data) config_obj: Configuration = Configuration.from_dict(dict(config)) return config_obj - - -CONFIGURATION = load_config() diff --git a/dts/framework/dts.py b/dts/framework/dts.py deleted file mode 100644 index 25d6942d81..0000000000 --- a/dts/framework/dts.py +++ /dev/null @@ -1,228 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2010-2019 Intel Corporation -# Copyright(c) 2022-2023 PANTHEON.tech s.r.o. -# Copyright(c) 2022-2023 University of New Hampshire - -import sys - -from .config import ( - CONFIGURATION, - BuildTargetConfiguration, - ExecutionConfiguration, - TestSuiteConfig, -) -from .exception import BlockingTestSuiteError -from .logger import DTSLOG, getLogger -from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result -from .test_suite import get_test_suites -from .testbed_model import SutNode, TGNode -from .utils import check_dts_python_version - -dts_logger: DTSLOG = getLogger("DTSRunner") -result: DTSResult = DTSResult(dts_logger) - - -def run_all() -> None: - """ - The main process of DTS. Runs all build targets in all executions from the main - config file. - """ - global dts_logger - global result - - # check the python version of the server that run dts - check_dts_python_version() - - sut_nodes: dict[str, SutNode] = {} - tg_nodes: dict[str, TGNode] = {} - try: - # for all Execution sections - for execution in CONFIGURATION.executions: - sut_node = sut_nodes.get(execution.system_under_test_node.name) - tg_node = tg_nodes.get(execution.traffic_generator_node.name) - - try: - if not sut_node: - sut_node = SutNode(execution.system_under_test_node) - sut_nodes[sut_node.name] = sut_node - if not tg_node: - tg_node = TGNode(execution.traffic_generator_node) - tg_nodes[tg_node.name] = tg_node - result.update_setup(Result.PASS) - except Exception as e: - failed_node = execution.system_under_test_node.name - if sut_node: - failed_node = execution.traffic_generator_node.name - dts_logger.exception(f"Creation of node {failed_node} failed.") - result.update_setup(Result.FAIL, e) - - else: - _run_execution(sut_node, tg_node, execution, result) - - except Exception as e: - dts_logger.exception("An unexpected error has occurred.") - result.add_error(e) - raise - - finally: - try: - for node in (sut_nodes | tg_nodes).values(): - node.close() - result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Final cleanup of nodes failed.") - result.update_teardown(Result.ERROR, e) - - # we need to put the sys.exit call outside the finally clause to make sure - # that unexpected exceptions will propagate - # in that case, the error that should be reported is the uncaught exception as - # that is a severe error originating from the framework - # at that point, we'll only have partial results which could be impacted by the - # error causing the uncaught exception, making them uninterpretable - _exit_dts() - - -def _run_execution( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - result: DTSResult, -) -> None: - """ - Run the given execution. This involves running the execution setup as well as - running all build targets in the given execution. - """ - dts_logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") - execution_result = result.add_execution(sut_node.config) - execution_result.add_sut_info(sut_node.node_info) - - try: - sut_node.set_up_execution(execution) - execution_result.update_setup(Result.PASS) - except Exception as e: - dts_logger.exception("Execution setup failed.") - execution_result.update_setup(Result.FAIL, e) - - else: - for build_target in execution.build_targets: - _run_build_target(sut_node, tg_node, build_target, execution, execution_result) - - finally: - try: - sut_node.tear_down_execution() - execution_result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Execution teardown failed.") - execution_result.update_teardown(Result.FAIL, e) - - -def _run_build_target( - sut_node: SutNode, - tg_node: TGNode, - build_target: BuildTargetConfiguration, - execution: ExecutionConfiguration, - execution_result: ExecutionResult, -) -> None: - """ - Run the given build target. - """ - dts_logger.info(f"Running build target '{build_target.name}'.") - build_target_result = execution_result.add_build_target(build_target) - - try: - sut_node.set_up_build_target(build_target) - result.dpdk_version = sut_node.dpdk_version - build_target_result.add_build_target_info(sut_node.get_build_target_info()) - build_target_result.update_setup(Result.PASS) - except Exception as e: - dts_logger.exception("Build target setup failed.") - build_target_result.update_setup(Result.FAIL, e) - - else: - _run_all_suites(sut_node, tg_node, execution, build_target_result) - - finally: - try: - sut_node.tear_down_build_target() - build_target_result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Build target teardown failed.") - build_target_result.update_teardown(Result.FAIL, e) - - -def _run_all_suites( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, -) -> None: - """ - Use the given build_target to run execution's test suites - with possibly only a subset of test cases. - If no subset is specified, run all test cases. - """ - end_build_target = False - if not execution.skip_smoke_tests: - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] - for test_suite_config in execution.test_suites: - try: - _run_single_suite(sut_node, tg_node, execution, build_target_result, test_suite_config) - except BlockingTestSuiteError as e: - dts_logger.exception( - f"An error occurred within {test_suite_config.test_suite}. Skipping build target." - ) - result.add_error(e) - end_build_target = True - # if a blocking test failed and we need to bail out of suite executions - if end_build_target: - break - - -def _run_single_suite( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, - test_suite_config: TestSuiteConfig, -) -> None: - """Runs a single test suite. - - Args: - sut_node: Node to run tests on. - execution: Execution the test case belongs to. - build_target_result: Build target configuration test case is run on - test_suite_config: Test suite configuration - - Raises: - BlockingTestSuiteError: If a test suite that was marked as blocking fails. - """ - try: - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" - test_suite_classes = get_test_suites(full_suite_path) - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) - dts_logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") - except Exception as e: - dts_logger.exception("An error occurred when searching for test suites.") - result.update_setup(Result.ERROR, e) - - else: - for test_suite_class in test_suite_classes: - test_suite = test_suite_class( - sut_node, - tg_node, - test_suite_config.test_cases, - execution.func, - build_target_result, - ) - test_suite.run() - - -def _exit_dts() -> None: - """ - Process all errors and exit with the proper exit code. - """ - result.process() - - if dts_logger: - dts_logger.info("DTS execution has ended.") - sys.exit(result.get_return_code()) diff --git a/dts/framework/runner.py b/dts/framework/runner.py new file mode 100644 index 0000000000..5b077c5805 --- /dev/null +++ b/dts/framework/runner.py @@ -0,0 +1,243 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2019 Intel Corporation +# Copyright(c) 2022-2023 PANTHEON.tech s.r.o. +# Copyright(c) 2022-2023 University of New Hampshire + +import logging +import sys + +from .config import ( + BuildTargetConfiguration, + Configuration, + ExecutionConfiguration, + TestSuiteConfig, +) +from .exception import BlockingTestSuiteError +from .logger import DTSLOG, getLogger +from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result +from .test_suite import get_test_suites +from .testbed_model import SutNode, TGNode +from .utils import check_dts_python_version + + +class DTSRunner: + _logger: DTSLOG + _result: DTSResult + _configuration: Configuration + + def __init__(self, configuration: Configuration): + self._logger = getLogger("DTSRunner") + self._result = DTSResult(self._logger) + self._configuration = configuration + + def run(self): + """ + The main process of DTS. Runs all build targets in all executions from the main + config file. + """ + # check the python version of the server that run dts + check_dts_python_version() + sut_nodes: dict[str, SutNode] = {} + tg_nodes: dict[str, TGNode] = {} + try: + # for all Execution sections + for execution in self._configuration.executions: + sut_node = sut_nodes.get(execution.system_under_test_node.name) + tg_node = tg_nodes.get(execution.traffic_generator_node.name) + + try: + if not sut_node: + sut_node = SutNode(execution.system_under_test_node) + sut_nodes[sut_node.name] = sut_node + if not tg_node: + tg_node = TGNode(execution.traffic_generator_node) + tg_nodes[tg_node.name] = tg_node + self._result.update_setup(Result.PASS) + except Exception as e: + failed_node = execution.system_under_test_node.name + if sut_node: + failed_node = execution.traffic_generator_node.name + self._logger.exception( + f"The Creation of node {failed_node} failed." + ) + self._result.update_setup(Result.FAIL, e) + + else: + self._run_execution(sut_node, tg_node, execution) + + except Exception as e: + self._logger.exception("An unexpected error has occurred.") + self._result.add_error(e) + raise + + finally: + try: + for node in (sut_nodes | tg_nodes).values(): + node.close() + self._result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("The final cleanup of nodes failed.") + self._result.update_teardown(Result.ERROR, e) + + # we need to put the sys.exit call outside the finally clause to make sure + # that unexpected exceptions will propagate + # in that case, the error that should be reported is the uncaught exception as + # that is a severe error originating from the framework + # at that point, we'll only have partial results which could be impacted by the + # error causing the uncaught exception, making them uninterpretable + self._exit_dts() + + def _run_execution( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + ) -> None: + """ + Run the given execution. This involves running the execution setup as well as + running all build targets in the given execution. + """ + self._logger.info( + f"Running execution with SUT '{execution.system_under_test_node.name}'." + ) + execution_result = self._result.add_execution(sut_node.config) + execution_result.add_sut_info(sut_node.node_info) + + try: + sut_node.set_up_execution(execution) + execution_result.update_setup(Result.PASS) + except Exception as e: + self._logger.exception("Execution setup failed.") + execution_result.update_setup(Result.FAIL, e) + + else: + for build_target in execution.build_targets: + self._run_build_target( + sut_node, tg_node, build_target, execution, execution_result + ) + + finally: + try: + sut_node.tear_down_execution() + execution_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("Execution teardown failed.") + execution_result.update_teardown(Result.FAIL, e) + + def _run_build_target( + self, + sut_node: SutNode, + tg_node: TGNode, + build_target: BuildTargetConfiguration, + execution: ExecutionConfiguration, + execution_result: ExecutionResult, + ) -> None: + """ + Run the given build target. + """ + self._logger.info(f"Running build target '{build_target.name}'.") + build_target_result = execution_result.add_build_target(build_target) + + try: + sut_node.set_up_build_target(build_target) + self._result.dpdk_version = sut_node.dpdk_version + build_target_result.add_build_target_info(sut_node.get_build_target_info()) + build_target_result.update_setup(Result.PASS) + except Exception as e: + self._logger.exception("Build target setup failed.") + build_target_result.update_setup(Result.FAIL, e) + + else: + self._run_all_suites(sut_node, tg_node, execution, build_target_result) + + finally: + try: + sut_node.tear_down_build_target() + build_target_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("Build target teardown failed.") + build_target_result.update_teardown(Result.FAIL, e) + + def _run_all_suites( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + build_target_result: BuildTargetResult, + ) -> None: + """ + Use the given build_target to run execution's test suites + with possibly only a subset of test cases. + If no subset is specified, run all test cases. + """ + end_build_target = False + if not execution.skip_smoke_tests: + execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] + for test_suite_config in execution.test_suites: + try: + self._run_single_suite( + sut_node, tg_node, execution, build_target_result, test_suite_config + ) + except BlockingTestSuiteError as e: + self._logger.exception( + f"An error occurred within {test_suite_config.test_suite}. " + "Skipping build target..." + ) + self._result.add_error(e) + end_build_target = True + # if a blocking test failed and we need to bail out of suite executions + if end_build_target: + break + + def _run_single_suite( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + build_target_result: BuildTargetResult, + test_suite_config: TestSuiteConfig, + ) -> None: + """Runs a single test suite. + + Args: + sut_node: Node to run tests on. + execution: Execution the test case belongs to. + build_target_result: Build target configuration test case is run on + test_suite_config: Test suite configuration + + Raises: + BlockingTestSuiteError: If a test suite that was marked as blocking fails. + """ + try: + full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" + test_suite_classes = get_test_suites(full_suite_path) + suites_str = ", ".join((x.__name__ for x in test_suite_classes)) + self._logger.debug( + f"Found test suites '{suites_str}' in '{full_suite_path}'." + ) + except Exception as e: + self._logger.exception("An error occurred when searching for test suites.") + self._result.update_setup(Result.ERROR, e) + + else: + for test_suite_class in test_suite_classes: + test_suite = test_suite_class( + sut_node, + tg_node, + test_suite_config.test_cases, + execution.func, + build_target_result, + ) + test_suite.run() + + def _exit_dts(self) -> None: + """ + Process all errors and exit with the proper exit code. + """ + self._result.process() + + if self._logger: + self._logger.info("DTS execution has ended.") + + logging.shutdown() + sys.exit(self._result.get_return_code()) diff --git a/dts/main.py b/dts/main.py index 43311fa847..879ce5cb89 100755 --- a/dts/main.py +++ b/dts/main.py @@ -10,11 +10,13 @@ import logging -from framework import dts +from framework.config import load_config +from framework.runner import DTSRunner def main() -> None: - dts.run_all() + dts = DTSRunner(configuration=load_config()) + dts.run() # Main program begins here -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [RFC PATCH v1 2/5] dts: move test suite execution logic to DTSRunner 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 1/5] dts: convert dts.py methods to class Juraj Linkeš @ 2023-12-20 10:33 ` Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 3/5] dts: process test suites at the beginning of run Juraj Linkeš ` (6 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, yoan.picchi, Luca.Vizzarro Cc: dev, Juraj Linkeš Move the code responsible for running the test suite from the TestSuite class to the DTSRunner class. This restructuring decision was made to consolidate and unify the related logic into a single unit. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/runner.py | 156 +++++++++++++++++++++++++++++++++--- dts/framework/test_suite.py | 134 +------------------------------ 2 files changed, 147 insertions(+), 143 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 5b077c5805..5e145a8066 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -5,6 +5,7 @@ import logging import sys +from types import MethodType from .config import ( BuildTargetConfiguration, @@ -12,10 +13,18 @@ ExecutionConfiguration, TestSuiteConfig, ) -from .exception import BlockingTestSuiteError +from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError from .logger import DTSLOG, getLogger -from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result -from .test_suite import get_test_suites +from .settings import SETTINGS +from .test_result import ( + BuildTargetResult, + DTSResult, + ExecutionResult, + Result, + TestCaseResult, + TestSuiteResult, +) +from .test_suite import TestSuite, get_test_suites from .testbed_model import SutNode, TGNode from .utils import check_dts_python_version @@ -148,7 +157,7 @@ def _run_build_target( build_target_result.update_setup(Result.FAIL, e) else: - self._run_all_suites(sut_node, tg_node, execution, build_target_result) + self._run_test_suites(sut_node, tg_node, execution, build_target_result) finally: try: @@ -158,7 +167,7 @@ def _run_build_target( self._logger.exception("Build target teardown failed.") build_target_result.update_teardown(Result.FAIL, e) - def _run_all_suites( + def _run_test_suites( self, sut_node: SutNode, tg_node: TGNode, @@ -175,7 +184,7 @@ def _run_all_suites( execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] for test_suite_config in execution.test_suites: try: - self._run_single_suite( + self._run_test_suite( sut_node, tg_node, execution, build_target_result, test_suite_config ) except BlockingTestSuiteError as e: @@ -189,7 +198,7 @@ def _run_all_suites( if end_build_target: break - def _run_single_suite( + def _run_test_suite( self, sut_node: SutNode, tg_node: TGNode, @@ -198,6 +207,9 @@ def _run_single_suite( test_suite_config: TestSuiteConfig, ) -> None: """Runs a single test suite. + Setup, execute and teardown the whole suite. + Suite execution consists of running all test cases scheduled to be executed. + A test cast run consists of setup, execution and teardown of said test case. Args: sut_node: Node to run tests on. @@ -222,13 +234,131 @@ def _run_single_suite( else: for test_suite_class in test_suite_classes: test_suite = test_suite_class( - sut_node, - tg_node, - test_suite_config.test_cases, - execution.func, - build_target_result, + sut_node, tg_node, test_suite_config.test_cases + ) + + test_suite_name = test_suite.__class__.__name__ + test_suite_result = build_target_result.add_test_suite(test_suite_name) + try: + self._logger.info(f"Starting test suite setup: {test_suite_name}") + test_suite.set_up_suite() + test_suite_result.update_setup(Result.PASS) + self._logger.info(f"Test suite setup successful: {test_suite_name}") + except Exception as e: + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") + test_suite_result.update_setup(Result.ERROR, e) + + else: + self._execute_test_suite( + execution.func, test_suite, test_suite_result + ) + + finally: + try: + test_suite.tear_down_suite() + sut_node.kill_cleanup_dpdk_apps() + test_suite_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception( + f"Test suite teardown ERROR: {test_suite_name}" + ) + self._logger.warning( + f"Test suite '{test_suite_name}' teardown failed, " + f"the next test suite may be affected." + ) + test_suite_result.update_setup(Result.ERROR, e) + if ( + len(test_suite_result.get_errors()) > 0 + and test_suite.is_blocking + ): + raise BlockingTestSuiteError(test_suite_name) + + def _execute_test_suite( + self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult + ) -> None: + """ + Execute all test cases scheduled to be executed in this suite. + """ + if func: + for test_case_method in test_suite._get_functional_test_cases(): + test_case_name = test_case_method.__name__ + test_case_result = test_suite_result.add_test_case(test_case_name) + all_attempts = SETTINGS.re_run + 1 + attempt_nr = 1 + self._run_test_case(test_suite, test_case_method, test_case_result) + while not test_case_result and attempt_nr < all_attempts: + attempt_nr += 1 + self._logger.info( + f"Re-running FAILED test case '{test_case_name}'. " + f"Attempt number {attempt_nr} out of {all_attempts}." + ) + self._run_test_case(test_suite, test_case_method, test_case_result) + + def _run_test_case( + self, + test_suite: TestSuite, + test_case_method: MethodType, + test_case_result: TestCaseResult, + ) -> None: + """ + Setup, execute and teardown a test case in this suite. + Exceptions are caught and recorded in logs and results. + """ + test_case_name = test_case_method.__name__ + + try: + # run set_up function for each case + test_suite.set_up_test_case() + test_case_result.update_setup(Result.PASS) + except SSHTimeoutError as e: + self._logger.exception(f"Test case setup FAILED: {test_case_name}") + test_case_result.update_setup(Result.FAIL, e) + except Exception as e: + self._logger.exception(f"Test case setup ERROR: {test_case_name}") + test_case_result.update_setup(Result.ERROR, e) + + else: + # run test case if setup was successful + self._execute_test_case(test_case_method, test_case_result) + + finally: + try: + test_suite.tear_down_test_case() + test_case_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test case teardown ERROR: {test_case_name}") + self._logger.warning( + f"Test case '{test_case_name}' teardown failed, " + f"the next test case may be affected." ) - test_suite.run() + test_case_result.update_teardown(Result.ERROR, e) + test_case_result.update(Result.ERROR) + + def _execute_test_case( + self, test_case_method: MethodType, test_case_result: TestCaseResult + ) -> None: + """ + Execute one test case and handle failures. + """ + test_case_name = test_case_method.__name__ + try: + self._logger.info(f"Starting test case execution: {test_case_name}") + test_case_method() + test_case_result.update(Result.PASS) + self._logger.info(f"Test case execution PASSED: {test_case_name}") + + except TestCaseVerifyError as e: + self._logger.exception(f"Test case execution FAILED: {test_case_name}") + test_case_result.update(Result.FAIL, e) + except Exception as e: + self._logger.exception(f"Test case execution ERROR: {test_case_name}") + test_case_result.update(Result.ERROR, e) + except KeyboardInterrupt: + self._logger.error( + f"Test case execution INTERRUPTED by user: {test_case_name}" + ) + test_case_result.update(Result.SKIP) + raise KeyboardInterrupt("Stop DTS") def _exit_dts(self) -> None: """ diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index 4a7907ec33..e96305deb0 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -17,15 +17,9 @@ from scapy.layers.l2 import Ether # type: ignore[import] from scapy.packet import Packet, Padding # type: ignore[import] -from .exception import ( - BlockingTestSuiteError, - ConfigurationError, - SSHTimeoutError, - TestCaseVerifyError, -) +from .exception import ConfigurationError, TestCaseVerifyError from .logger import DTSLOG, getLogger from .settings import SETTINGS -from .test_result import BuildTargetResult, Result, TestCaseResult, TestSuiteResult from .testbed_model import SutNode, TGNode from .testbed_model.hw.port import Port, PortLink from .utils import get_packet_summaries @@ -50,11 +44,10 @@ class TestSuite(object): """ sut_node: SutNode + tg_node: TGNode is_blocking = False _logger: DTSLOG _test_cases_to_run: list[str] - _func: bool - _result: TestSuiteResult _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -69,17 +62,13 @@ def __init__( self, sut_node: SutNode, tg_node: TGNode, - test_cases: list[str], - func: bool, - build_target_result: BuildTargetResult, + test_cases_to_run: list[str], ): self.sut_node = sut_node self.tg_node = tg_node self._logger = getLogger(self.__class__.__name__) - self._test_cases_to_run = test_cases + self._test_cases_to_run = test_cases_to_run self._test_cases_to_run.extend(SETTINGS.test_cases) - self._func = func - self._result = build_target_result.add_test_suite(self.__class__.__name__) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( @@ -280,60 +269,6 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: return False return True - def run(self) -> None: - """ - Setup, execute and teardown the whole suite. - Suite execution consists of running all test cases scheduled to be executed. - A test cast run consists of setup, execution and teardown of said test case. - """ - test_suite_name = self.__class__.__name__ - - try: - self._logger.info(f"Starting test suite setup: {test_suite_name}") - self.set_up_suite() - self._result.update_setup(Result.PASS) - self._logger.info(f"Test suite setup successful: {test_suite_name}") - except Exception as e: - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") - self._result.update_setup(Result.ERROR, e) - - else: - self._execute_test_suite() - - finally: - try: - self.tear_down_suite() - self.sut_node.kill_cleanup_dpdk_apps() - self._result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") - self._logger.warning( - f"Test suite '{test_suite_name}' teardown failed, " - f"the next test suite may be affected." - ) - self._result.update_setup(Result.ERROR, e) - if len(self._result.get_errors()) > 0 and self.is_blocking: - raise BlockingTestSuiteError(test_suite_name) - - def _execute_test_suite(self) -> None: - """ - Execute all test cases scheduled to be executed in this suite. - """ - if self._func: - for test_case_method in self._get_functional_test_cases(): - test_case_name = test_case_method.__name__ - test_case_result = self._result.add_test_case(test_case_name) - all_attempts = SETTINGS.re_run + 1 - attempt_nr = 1 - self._run_test_case(test_case_method, test_case_result) - while not test_case_result and attempt_nr < all_attempts: - attempt_nr += 1 - self._logger.info( - f"Re-running FAILED test case '{test_case_name}'. " - f"Attempt number {attempt_nr} out of {all_attempts}." - ) - self._run_test_case(test_case_method, test_case_result) - def _get_functional_test_cases(self) -> list[MethodType]: """ Get all functional test cases. @@ -363,67 +298,6 @@ def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool return match - def _run_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult - ) -> None: - """ - Setup, execute and teardown a test case in this suite. - Exceptions are caught and recorded in logs and results. - """ - test_case_name = test_case_method.__name__ - - try: - # run set_up function for each case - self.set_up_test_case() - test_case_result.update_setup(Result.PASS) - except SSHTimeoutError as e: - self._logger.exception(f"Test case setup FAILED: {test_case_name}") - test_case_result.update_setup(Result.FAIL, e) - except Exception as e: - self._logger.exception(f"Test case setup ERROR: {test_case_name}") - test_case_result.update_setup(Result.ERROR, e) - - else: - # run test case if setup was successful - self._execute_test_case(test_case_method, test_case_result) - - finally: - try: - self.tear_down_test_case() - test_case_result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test case teardown ERROR: {test_case_name}") - self._logger.warning( - f"Test case '{test_case_name}' teardown failed, " - f"the next test case may be affected." - ) - test_case_result.update_teardown(Result.ERROR, e) - test_case_result.update(Result.ERROR) - - def _execute_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult - ) -> None: - """ - Execute one test case and handle failures. - """ - test_case_name = test_case_method.__name__ - try: - self._logger.info(f"Starting test case execution: {test_case_name}") - test_case_method() - test_case_result.update(Result.PASS) - self._logger.info(f"Test case execution PASSED: {test_case_name}") - - except TestCaseVerifyError as e: - self._logger.exception(f"Test case execution FAILED: {test_case_name}") - test_case_result.update(Result.FAIL, e) - except Exception as e: - self._logger.exception(f"Test case execution ERROR: {test_case_name}") - test_case_result.update(Result.ERROR, e) - except KeyboardInterrupt: - self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}") - test_case_result.update(Result.SKIP) - raise KeyboardInterrupt("Stop DTS") - def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: def is_test_suite(object) -> bool: -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [RFC PATCH v1 3/5] dts: process test suites at the beginning of run 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 1/5] dts: convert dts.py methods to class Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 2/5] dts: move test suite execution logic to DTSRunner Juraj Linkeš @ 2023-12-20 10:33 ` Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 4/5] dts: block all testcases when earlier setup fails Juraj Linkeš ` (5 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, yoan.picchi, Luca.Vizzarro Cc: dev, Juraj Linkeš We initialize test suite/case objects at the start of the program and store them with custom execution config (add test case names) in Execution. This change helps identify errors with test suites earlier, and we have access to the right data when programs crash earlier. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/config/__init__.py | 5 +- dts/framework/runner.py | 309 ++++++++++++++++++++--------- dts/framework/test_suite.py | 59 +----- dts/tests/TestSuite_smoke_tests.py | 2 +- 4 files changed, 217 insertions(+), 158 deletions(-) diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index 497847afb9..d65ac625f8 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -238,7 +238,6 @@ class ExecutionConfiguration: system_under_test_node: SutNodeConfiguration traffic_generator_node: TGNodeConfiguration vdevs: list[str] - skip_smoke_tests: bool @staticmethod def from_dict( @@ -247,9 +246,10 @@ def from_dict( build_targets: list[BuildTargetConfiguration] = list( map(BuildTargetConfiguration.from_dict, d["build_targets"]) ) + if not d.get("skip_smoke_tests", False): + d["test_suites"].insert(0, "smoke_tests") test_suites: list[TestSuiteConfig] = list(map(TestSuiteConfig.from_dict, d["test_suites"])) sut_name = d["system_under_test_node"]["node_name"] - skip_smoke_tests = d.get("skip_smoke_tests", False) assert sut_name in node_map, f"Unknown SUT {sut_name} in execution {d}" system_under_test_node = node_map[sut_name] assert isinstance( @@ -270,7 +270,6 @@ def from_dict( build_targets=build_targets, perf=d["perf"], func=d["func"], - skip_smoke_tests=skip_smoke_tests, test_suites=test_suites, system_under_test_node=system_under_test_node, traffic_generator_node=traffic_generator_node, diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 5e145a8066..acc3342f0c 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -3,9 +3,14 @@ # Copyright(c) 2022-2023 PANTHEON.tech s.r.o. # Copyright(c) 2022-2023 University of New Hampshire +import importlib +import inspect import logging +import re import sys -from types import MethodType +from copy import deepcopy +from dataclasses import dataclass +from types import MethodType, ModuleType from .config import ( BuildTargetConfiguration, @@ -13,7 +18,12 @@ ExecutionConfiguration, TestSuiteConfig, ) -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError +from .exception import ( + BlockingTestSuiteError, + ConfigurationError, + SSHTimeoutError, + TestCaseVerifyError, +) from .logger import DTSLOG, getLogger from .settings import SETTINGS from .test_result import ( @@ -24,25 +34,55 @@ TestCaseResult, TestSuiteResult, ) -from .test_suite import TestSuite, get_test_suites +from .test_suite import TestSuite from .testbed_model import SutNode, TGNode from .utils import check_dts_python_version +@dataclass +class TestSuiteSetup: + test_suite: type[TestSuite] + test_cases: list[MethodType] + + def processed_config(self) -> TestSuiteConfig: + return TestSuiteConfig( + test_suite=self.test_suite.__name__, + test_cases=[test_case.__name__ for test_case in self.test_cases], + ) + + +@dataclass +class Execution: + config: ExecutionConfiguration + test_suite_setups: list[TestSuiteSetup] + + def processed_config(self) -> ExecutionConfiguration: + """ + Creating copy of execution config witch add test-case names. + """ + modified_execution_config = deepcopy(self.config) + modified_execution_config.test_suites[:] = [ + test_suite.processed_config() for test_suite in self.test_suite_setups + ] + return modified_execution_config + + class DTSRunner: _logger: DTSLOG _result: DTSResult - _configuration: Configuration + _executions: list[Execution] def __init__(self, configuration: Configuration): self._logger = getLogger("DTSRunner") self._result = DTSResult(self._logger) - self._configuration = configuration + self._executions = create_executions(configuration.executions) def run(self): """ The main process of DTS. Runs all build targets in all executions from the main config file. + Suite execution consists of running all test cases scheduled to be executed. + A test case run consists of setup, execution and teardown of said test case. """ # check the python version of the server that run dts check_dts_python_version() @@ -50,22 +90,22 @@ def run(self): tg_nodes: dict[str, TGNode] = {} try: # for all Execution sections - for execution in self._configuration.executions: - sut_node = sut_nodes.get(execution.system_under_test_node.name) - tg_node = tg_nodes.get(execution.traffic_generator_node.name) + for execution in self._executions: + sut_node = sut_nodes.get(execution.config.system_under_test_node.name) + tg_node = tg_nodes.get(execution.config.traffic_generator_node.name) try: if not sut_node: - sut_node = SutNode(execution.system_under_test_node) + sut_node = SutNode(execution.config.system_under_test_node) sut_nodes[sut_node.name] = sut_node if not tg_node: - tg_node = TGNode(execution.traffic_generator_node) + tg_node = TGNode(execution.config.traffic_generator_node) tg_nodes[tg_node.name] = tg_node self._result.update_setup(Result.PASS) except Exception as e: - failed_node = execution.system_under_test_node.name + failed_node = execution.config.system_under_test_node.name if sut_node: - failed_node = execution.traffic_generator_node.name + failed_node = execution.config.traffic_generator_node.name self._logger.exception( f"The Creation of node {failed_node} failed." ) @@ -100,29 +140,34 @@ def _run_execution( self, sut_node: SutNode, tg_node: TGNode, - execution: ExecutionConfiguration, + execution: Execution, ) -> None: """ Run the given execution. This involves running the execution setup as well as running all build targets in the given execution. """ self._logger.info( - f"Running execution with SUT '{execution.system_under_test_node.name}'." + "Running execution with SUT " + f"'{execution.config.system_under_test_node.name}'." ) execution_result = self._result.add_execution(sut_node.config) execution_result.add_sut_info(sut_node.node_info) try: - sut_node.set_up_execution(execution) + sut_node.set_up_execution(execution.config) execution_result.update_setup(Result.PASS) except Exception as e: self._logger.exception("Execution setup failed.") execution_result.update_setup(Result.FAIL, e) else: - for build_target in execution.build_targets: + for build_target in execution.config.build_targets: self._run_build_target( - sut_node, tg_node, build_target, execution, execution_result + sut_node, + tg_node, + build_target, + execution, + execution_result, ) finally: @@ -138,7 +183,7 @@ def _run_build_target( sut_node: SutNode, tg_node: TGNode, build_target: BuildTargetConfiguration, - execution: ExecutionConfiguration, + execution: Execution, execution_result: ExecutionResult, ) -> None: """ @@ -171,7 +216,7 @@ def _run_test_suites( self, sut_node: SutNode, tg_node: TGNode, - execution: ExecutionConfiguration, + execution: Execution, build_target_result: BuildTargetResult, ) -> None: """ @@ -180,16 +225,18 @@ def _run_test_suites( If no subset is specified, run all test cases. """ end_build_target = False - if not execution.skip_smoke_tests: - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] - for test_suite_config in execution.test_suites: + for test_suite_setup in execution.test_suite_setups: try: self._run_test_suite( - sut_node, tg_node, execution, build_target_result, test_suite_config + sut_node, + tg_node, + test_suite_setup, + build_target_result, ) except BlockingTestSuiteError as e: self._logger.exception( - f"An error occurred within {test_suite_config.test_suite}. " + "An error occurred within " + f"{test_suite_setup.test_suite.__name__}. " "Skipping build target..." ) self._result.add_error(e) @@ -202,14 +249,10 @@ def _run_test_suite( self, sut_node: SutNode, tg_node: TGNode, - execution: ExecutionConfiguration, + test_suite_setup: TestSuiteSetup, build_target_result: BuildTargetResult, - test_suite_config: TestSuiteConfig, ) -> None: """Runs a single test suite. - Setup, execute and teardown the whole suite. - Suite execution consists of running all test cases scheduled to be executed. - A test cast run consists of setup, execution and teardown of said test case. Args: sut_node: Node to run tests on. @@ -220,84 +263,67 @@ def _run_test_suite( Raises: BlockingTestSuiteError: If a test suite that was marked as blocking fails. """ + test_suite = test_suite_setup.test_suite(sut_node, tg_node) + test_suite_name = test_suite_setup.test_suite.__name__ + test_suite_result = build_target_result.add_test_suite(test_suite_name) try: - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" - test_suite_classes = get_test_suites(full_suite_path) - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) - self._logger.debug( - f"Found test suites '{suites_str}' in '{full_suite_path}'." - ) + self._logger.info(f"Starting test suite setup: {test_suite_name}") + test_suite.set_up_suite() + test_suite_result.update_setup(Result.PASS) + self._logger.info(f"Test suite setup successful: {test_suite_name}") except Exception as e: - self._logger.exception("An error occurred when searching for test suites.") - self._result.update_setup(Result.ERROR, e) + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") + test_suite_result.update_setup(Result.ERROR, e) else: - for test_suite_class in test_suite_classes: - test_suite = test_suite_class( - sut_node, tg_node, test_suite_config.test_cases - ) - - test_suite_name = test_suite.__class__.__name__ - test_suite_result = build_target_result.add_test_suite(test_suite_name) - try: - self._logger.info(f"Starting test suite setup: {test_suite_name}") - test_suite.set_up_suite() - test_suite_result.update_setup(Result.PASS) - self._logger.info(f"Test suite setup successful: {test_suite_name}") - except Exception as e: - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") - test_suite_result.update_setup(Result.ERROR, e) - - else: - self._execute_test_suite( - execution.func, test_suite, test_suite_result - ) + self._execute_test_suite( + test_suite, + test_suite_setup.test_cases, + test_suite_result, + ) - finally: - try: - test_suite.tear_down_suite() - sut_node.kill_cleanup_dpdk_apps() - test_suite_result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception( - f"Test suite teardown ERROR: {test_suite_name}" - ) - self._logger.warning( - f"Test suite '{test_suite_name}' teardown failed, " - f"the next test suite may be affected." - ) - test_suite_result.update_setup(Result.ERROR, e) - if ( - len(test_suite_result.get_errors()) > 0 - and test_suite.is_blocking - ): - raise BlockingTestSuiteError(test_suite_name) + finally: + try: + test_suite.tear_down_suite() + sut_node.kill_cleanup_dpdk_apps() + test_suite_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") + self._logger.warning( + f"Test suite '{test_suite_name}' teardown failed, " + "the next test suite may be affected." + ) + test_suite_result.update_setup(Result.ERROR, e) + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: + raise BlockingTestSuiteError(test_suite_name) def _execute_test_suite( - self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult + self, + test_suite: TestSuite, + test_cases: list[MethodType], + test_suite_result: TestSuiteResult, ) -> None: """ Execute all test cases scheduled to be executed in this suite. """ - if func: - for test_case_method in test_suite._get_functional_test_cases(): - test_case_name = test_case_method.__name__ - test_case_result = test_suite_result.add_test_case(test_case_name) - all_attempts = SETTINGS.re_run + 1 - attempt_nr = 1 - self._run_test_case(test_suite, test_case_method, test_case_result) - while not test_case_result and attempt_nr < all_attempts: - attempt_nr += 1 - self._logger.info( - f"Re-running FAILED test case '{test_case_name}'. " - f"Attempt number {attempt_nr} out of {all_attempts}." - ) - self._run_test_case(test_suite, test_case_method, test_case_result) + for test_case_method in test_cases: + test_case_name = test_case_method.__name__ + test_case_result = test_suite_result.add_test_case(test_case_name) + all_attempts = SETTINGS.re_run + 1 + attempt_nr = 1 + self._run_test_case(test_case_method, test_suite, test_case_result) + while not test_case_result and attempt_nr < all_attempts: + attempt_nr += 1 + self._logger.info( + f"Re-running FAILED test case '{test_case_name}'. " + f"Attempt number {attempt_nr} out of {all_attempts}." + ) + self._run_test_case(test_case_method, test_suite, test_case_result) def _run_test_case( self, - test_suite: TestSuite, test_case_method: MethodType, + test_suite: TestSuite, test_case_result: TestCaseResult, ) -> None: """ @@ -305,7 +331,6 @@ def _run_test_case( Exceptions are caught and recorded in logs and results. """ test_case_name = test_case_method.__name__ - try: # run set_up function for each case test_suite.set_up_test_case() @@ -319,7 +344,7 @@ def _run_test_case( else: # run test case if setup was successful - self._execute_test_case(test_case_method, test_case_result) + self._execute_test_case(test_case_method, test_suite, test_case_result) finally: try: @@ -335,7 +360,10 @@ def _run_test_case( test_case_result.update(Result.ERROR) def _execute_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult + self, + test_case_method: MethodType, + test_suite: TestSuite, + test_case_result: TestCaseResult, ) -> None: """ Execute one test case and handle failures. @@ -343,7 +371,7 @@ def _execute_test_case( test_case_name = test_case_method.__name__ try: self._logger.info(f"Starting test case execution: {test_case_name}") - test_case_method() + test_case_method(test_suite) test_case_result.update(Result.PASS) self._logger.info(f"Test case execution PASSED: {test_case_name}") @@ -371,3 +399,92 @@ def _exit_dts(self) -> None: logging.shutdown() sys.exit(self._result.get_return_code()) + + +def create_executions( + execution_configs: list[ExecutionConfiguration], +) -> list[Execution]: + executions: list[Execution] = [] + for execution_config in execution_configs: + test_suite_setups: list[TestSuiteSetup] = [] + + for test_suite_config in execution_config.test_suites: + testsuite_module_path = f"tests.TestSuite_{test_suite_config.test_suite}" + try: + suite_module = importlib.import_module(testsuite_module_path) + except ModuleNotFoundError as e: + raise ConfigurationError( + f"Test suite '{testsuite_module_path}' not found." + ) from e + + test_suite = _get_suite_class(suite_module, test_suite_config.test_suite) + + test_cases_to_run = test_suite_config.test_cases + test_cases_to_run.extend(SETTINGS.test_cases) + + test_cases = [] + if execution_config.func: + # add functional test cases + test_cases.extend( + _get_test_cases(test_suite, r"test_(?!perf_)", test_cases_to_run) + ) + + if execution_config.perf: + # add performance test cases + test_cases.extend( + _get_test_cases(test_suite, r"test_perf_", test_cases_to_run) + ) + + test_suite_setups.append( + TestSuiteSetup(test_suite=test_suite, test_cases=test_cases) + ) + + executions.append( + Execution( + config=execution_config, + test_suite_setups=test_suite_setups, + ) + ) + + return executions + + +def _get_suite_class(suite_module: ModuleType, suite_name: str) -> type[TestSuite]: + def is_test_suite(object) -> bool: + try: + if issubclass(object, TestSuite) and object is not TestSuite: + return True + except TypeError: + return False + return False + + suite_name_regex = suite_name.replace("_", "").lower() + for class_name, suite_class in inspect.getmembers(suite_module, is_test_suite): + if not class_name.startswith("Test"): + continue + + if suite_name_regex == class_name[4:].lower(): + return suite_class + raise ConfigurationError( + f"Cannot find valid test suite in {suite_module.__name__}." + ) + + +def _get_test_cases( + suite_class: type[TestSuite], test_case_regex: str, test_cases_to_run: list[str] +) -> list[MethodType]: + def should_be_executed(test_case_name: str) -> bool: + match = bool(re.match(test_case_regex, test_case_name)) + if test_cases_to_run: + return match and test_case_name in test_cases_to_run + + return match + + test_cases = [] + for test_case_name, test_case_method in inspect.getmembers( + suite_class, inspect.isfunction + ): + if should_be_executed(test_case_name): + test_cases.append(test_case_method) + + return test_cases diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index e96305deb0..e73206993d 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -6,20 +6,15 @@ Base class for creating DTS test cases. """ -import importlib -import inspect -import re from ipaddress import IPv4Interface, IPv6Interface, ip_interface -from types import MethodType from typing import Union from scapy.layers.inet import IP # type: ignore[import] from scapy.layers.l2 import Ether # type: ignore[import] from scapy.packet import Packet, Padding # type: ignore[import] -from .exception import ConfigurationError, TestCaseVerifyError +from .exception import TestCaseVerifyError from .logger import DTSLOG, getLogger -from .settings import SETTINGS from .testbed_model import SutNode, TGNode from .testbed_model.hw.port import Port, PortLink from .utils import get_packet_summaries @@ -47,7 +42,6 @@ class TestSuite(object): tg_node: TGNode is_blocking = False _logger: DTSLOG - _test_cases_to_run: list[str] _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -62,13 +56,10 @@ def __init__( self, sut_node: SutNode, tg_node: TGNode, - test_cases_to_run: list[str], ): self.sut_node = sut_node self.tg_node = tg_node self._logger = getLogger(self.__class__.__name__) - self._test_cases_to_run = test_cases_to_run - self._test_cases_to_run.extend(SETTINGS.test_cases) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( @@ -268,51 +259,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst: return False return True - - def _get_functional_test_cases(self) -> list[MethodType]: - """ - Get all functional test cases. - """ - return self._get_test_cases(r"test_(?!perf_)") - - def _get_test_cases(self, test_case_regex: str) -> list[MethodType]: - """ - Return a list of test cases matching test_case_regex. - """ - self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.") - filtered_test_cases = [] - for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod): - if self._should_be_executed(test_case_name, test_case_regex): - filtered_test_cases.append(test_case) - cases_str = ", ".join((x.__name__ for x in filtered_test_cases)) - self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.") - return filtered_test_cases - - def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool: - """ - Check whether the test case should be executed. - """ - match = bool(re.match(test_case_regex, test_case_name)) - if self._test_cases_to_run: - return match and test_case_name in self._test_cases_to_run - - return match - - -def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: - def is_test_suite(object) -> bool: - try: - if issubclass(object, TestSuite) and object is not TestSuite: - return True - except TypeError: - return False - return False - - try: - testcase_module = importlib.import_module(testsuite_module_path) - except ModuleNotFoundError as e: - raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e - return [ - test_suite_class - for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite) - ] diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py index 8958f58dac..aa4bae5b17 100644 --- a/dts/tests/TestSuite_smoke_tests.py +++ b/dts/tests/TestSuite_smoke_tests.py @@ -10,7 +10,7 @@ from framework.utils import REGEX_FOR_PCI_ADDRESS -class SmokeTests(TestSuite): +class TestSmokeTests(TestSuite): is_blocking = True # dicts in this list are expected to have two keys: # "pci_address" and "current_driver" -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [RFC PATCH v1 4/5] dts: block all testcases when earlier setup fails 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš ` (2 preceding siblings ...) 2023-12-20 10:33 ` [RFC PATCH v1 3/5] dts: process test suites at the beginning of run Juraj Linkeš @ 2023-12-20 10:33 ` Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 5/5] dts: refactor logging configuration Juraj Linkeš ` (4 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, yoan.picchi, Luca.Vizzarro Cc: dev, Juraj Linkeš In case of a failure during execution, build target or suite setup the test case results will be recorded as blocked. We also unify methods add_<result> to add_child_result to be more consistently. Now we store the corresponding config in each result with child configs and parent result. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/runner.py | 12 +- dts/framework/test_result.py | 361 ++++++++++++++++++++--------------- 2 files changed, 216 insertions(+), 157 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index acc3342f0c..28570d4a1c 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -74,7 +74,7 @@ class DTSRunner: def __init__(self, configuration: Configuration): self._logger = getLogger("DTSRunner") - self._result = DTSResult(self._logger) + self._result = DTSResult(configuration, self._logger) self._executions = create_executions(configuration.executions) def run(self): @@ -150,7 +150,7 @@ def _run_execution( "Running execution with SUT " f"'{execution.config.system_under_test_node.name}'." ) - execution_result = self._result.add_execution(sut_node.config) + execution_result = self._result.add_child_result(execution.processed_config()) execution_result.add_sut_info(sut_node.node_info) try: @@ -190,7 +190,7 @@ def _run_build_target( Run the given build target. """ self._logger.info(f"Running build target '{build_target.name}'.") - build_target_result = execution_result.add_build_target(build_target) + build_target_result = execution_result.add_child_result(build_target) try: sut_node.set_up_build_target(build_target) @@ -265,7 +265,9 @@ def _run_test_suite( """ test_suite = test_suite_setup.test_suite(sut_node, tg_node) test_suite_name = test_suite_setup.test_suite.__name__ - test_suite_result = build_target_result.add_test_suite(test_suite_name) + test_suite_result = build_target_result.add_child_result( + test_suite_setup.processed_config() + ) try: self._logger.info(f"Starting test suite setup: {test_suite_name}") test_suite.set_up_suite() @@ -308,7 +310,7 @@ def _execute_test_suite( """ for test_case_method in test_cases: test_case_name = test_case_method.__name__ - test_case_result = test_suite_result.add_test_case(test_case_name) + test_case_result = test_suite_result.add_child_result(test_case_name) all_attempts = SETTINGS.re_run + 1 attempt_nr = 1 self._run_test_case(test_case_method, test_suite, test_case_result) diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index 4c2e7e2418..dba2c55d36 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -9,6 +9,7 @@ import os.path from collections.abc import MutableSequence from enum import Enum, auto +from typing import Any, Union from .config import ( OS, @@ -16,9 +17,11 @@ BuildTargetConfiguration, BuildTargetInfo, Compiler, + Configuration, CPUType, - NodeConfiguration, + ExecutionConfiguration, NodeInfo, + TestSuiteConfig, ) from .exception import DTSError, ErrorSeverity from .logger import DTSLOG @@ -35,6 +38,7 @@ class Result(Enum): FAIL = auto() ERROR = auto() SKIP = auto() + BLOCK = auto() def __bool__(self) -> bool: return self is self.PASS @@ -63,42 +67,6 @@ def __bool__(self) -> bool: return bool(self.result) -class Statistics(dict): - """ - A helper class used to store the number of test cases by its result - along a few other basic information. - Using a dict provides a convenient way to format the data. - """ - - def __init__(self, dpdk_version: str | None): - super(Statistics, self).__init__() - for result in Result: - self[result.name] = 0 - self["PASS RATE"] = 0.0 - self["DPDK VERSION"] = dpdk_version - - def __iadd__(self, other: Result) -> "Statistics": - """ - Add a Result to the final count. - """ - self[other.name] += 1 - self["PASS RATE"] = ( - float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result) - ) - return self - - def __str__(self) -> str: - """ - Provide a string representation of the data. - """ - stats_str = "" - for key, value in self.items(): - stats_str += f"{key:<12} = {value}\n" - # according to docs, we should use \n when writing to text files - # on all platforms - return stats_str - - class BaseResult(object): """ The Base class for all results. Stores the results of @@ -109,6 +77,12 @@ class BaseResult(object): setup_result: FixtureResult teardown_result: FixtureResult _inner_results: MutableSequence["BaseResult"] + _child_configs: Union[ + list[ExecutionConfiguration], + list[BuildTargetConfiguration], + list[TestSuiteConfig], + list[str], + ] def __init__(self): self.setup_result = FixtureResult() @@ -119,6 +93,23 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None: self.setup_result.result = result self.setup_result.error = error + if result in [Result.BLOCK, Result.ERROR, Result.FAIL]: + for child_config in self._child_configs: + child_result = self.add_child_result(child_config) + child_result.block() + + def add_child_result(self, config: Any) -> "BaseResult": + """ + Adding corresponding result for each classes. + """ + + def block(self): + """ + Mark the result as block on corresponding classes. + """ + self.update_setup(Result.BLOCK) + self.update_teardown(Result.BLOCK) + def update_teardown(self, result: Result, error: Exception | None = None) -> None: self.teardown_result.result = result self.teardown_result.error = error @@ -139,119 +130,11 @@ def _get_inner_errors(self) -> list[Exception]: def get_errors(self) -> list[Exception]: return self._get_setup_teardown_errors() + self._get_inner_errors() - def add_stats(self, statistics: Statistics) -> None: + def add_stats(self, statistics: "Statistics") -> None: for inner_result in self._inner_results: inner_result.add_stats(statistics) -class TestCaseResult(BaseResult, FixtureResult): - """ - The test case specific result. - Stores the result of the actual test case. - Also stores the test case name. - """ - - test_case_name: str - - def __init__(self, test_case_name: str): - super(TestCaseResult, self).__init__() - self.test_case_name = test_case_name - - def update(self, result: Result, error: Exception | None = None) -> None: - self.result = result - self.error = error - - def _get_inner_errors(self) -> list[Exception]: - if self.error: - return [self.error] - return [] - - def add_stats(self, statistics: Statistics) -> None: - statistics += self.result - - def __bool__(self) -> bool: - return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) - - -class TestSuiteResult(BaseResult): - """ - The test suite specific result. - The _inner_results list stores results of test cases in a given test suite. - Also stores the test suite name. - """ - - suite_name: str - - def __init__(self, suite_name: str): - super(TestSuiteResult, self).__init__() - self.suite_name = suite_name - - def add_test_case(self, test_case_name: str) -> TestCaseResult: - test_case_result = TestCaseResult(test_case_name) - self._inner_results.append(test_case_result) - return test_case_result - - -class BuildTargetResult(BaseResult): - """ - The build target specific result. - The _inner_results list stores results of test suites in a given build target. - Also stores build target specifics, such as compiler used to build DPDK. - """ - - arch: Architecture - os: OS - cpu: CPUType - compiler: Compiler - compiler_version: str | None - dpdk_version: str | None - - def __init__(self, build_target: BuildTargetConfiguration): - super(BuildTargetResult, self).__init__() - self.arch = build_target.arch - self.os = build_target.os - self.cpu = build_target.cpu - self.compiler = build_target.compiler - self.compiler_version = None - self.dpdk_version = None - - def add_build_target_info(self, versions: BuildTargetInfo) -> None: - self.compiler_version = versions.compiler_version - self.dpdk_version = versions.dpdk_version - - def add_test_suite(self, test_suite_name: str) -> TestSuiteResult: - test_suite_result = TestSuiteResult(test_suite_name) - self._inner_results.append(test_suite_result) - return test_suite_result - - -class ExecutionResult(BaseResult): - """ - The execution specific result. - The _inner_results list stores results of build targets in a given execution. - Also stores the SUT node configuration. - """ - - sut_node: NodeConfiguration - sut_os_name: str - sut_os_version: str - sut_kernel_version: str - - def __init__(self, sut_node: NodeConfiguration): - super(ExecutionResult, self).__init__() - self.sut_node = sut_node - - def add_build_target(self, build_target: BuildTargetConfiguration) -> BuildTargetResult: - build_target_result = BuildTargetResult(build_target) - self._inner_results.append(build_target_result) - return build_target_result - - def add_sut_info(self, sut_info: NodeInfo): - self.sut_os_name = sut_info.os_name - self.sut_os_version = sut_info.os_version - self.sut_kernel_version = sut_info.kernel_version - - class DTSResult(BaseResult): """ Stores environment information and test results from a DTS run, which are: @@ -269,25 +152,27 @@ class DTSResult(BaseResult): """ dpdk_version: str | None + _child_configs: list[ExecutionConfiguration] _logger: DTSLOG _errors: list[Exception] _return_code: ErrorSeverity - _stats_result: Statistics | None + _stats_result: Union["Statistics", None] _stats_filename: str - def __init__(self, logger: DTSLOG): + def __init__(self, configuration: Configuration, logger: DTSLOG): super(DTSResult, self).__init__() self.dpdk_version = None + self._child_configs = configuration.executions self._logger = logger self._errors = [] self._return_code = ErrorSeverity.NO_ERR self._stats_result = None self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") - def add_execution(self, sut_node: NodeConfiguration) -> ExecutionResult: - execution_result = ExecutionResult(sut_node) - self._inner_results.append(execution_result) - return execution_result + def add_child_result(self, config: ExecutionConfiguration) -> "ExecutionResult": + result = ExecutionResult(config, self) + self._inner_results.append(result) + return result def add_error(self, error) -> None: self._errors.append(error) @@ -325,3 +210,175 @@ def get_return_code(self) -> int: self._return_code = error_return_code return int(self._return_code) + + +class ExecutionResult(BaseResult): + """ + The execution specific result. + The _inner_results list stores results of build targets in a given execution. + Also stores the SUT node configuration. + """ + + sut_os_name: str + sut_os_version: str + sut_kernel_version: str + _config: ExecutionConfiguration + _parent_result: DTSResult + _child_configs: list[BuildTargetConfiguration] + + def __init__(self, config: ExecutionConfiguration, parent_result: DTSResult): + super(ExecutionResult, self).__init__() + self._config = config + self._parent_result = parent_result + self._child_configs = config.build_targets + + def add_sut_info(self, sut_info: NodeInfo): + self.sut_os_name = sut_info.os_name + self.sut_os_version = sut_info.os_version + self.sut_kernel_version = sut_info.kernel_version + + def add_child_result(self, config: BuildTargetConfiguration) -> "BuildTargetResult": + result = BuildTargetResult(config, self) + self._inner_results.append(result) + return result + + +class BuildTargetResult(BaseResult): + """ + The build target specific result. + The _inner_results list stores results of test suites in a given build target. + Also stores build target specifics, such as compiler used to build DPDK. + """ + + arch: Architecture + os: OS + cpu: CPUType + compiler: Compiler + compiler_version: str | None + dpdk_version: str | None + _config: BuildTargetConfiguration + _parent_result: ExecutionResult + _child_configs: list[TestSuiteConfig] + + def __init__( + self, config: BuildTargetConfiguration, parent_result: ExecutionResult + ): + super(BuildTargetResult, self).__init__() + self.arch = config.arch + self.os = config.os + self.cpu = config.cpu + self.compiler = config.compiler + self.compiler_version = None + self.dpdk_version = None + self._config = config + self._parent_result = parent_result + self._child_configs = parent_result._config.test_suites + + def add_build_target_info(self, versions: BuildTargetInfo) -> None: + self.compiler_version = versions.compiler_version + self.dpdk_version = versions.dpdk_version + + def add_child_result( + self, + config: TestSuiteConfig, + ) -> "TestSuiteResult": + result = TestSuiteResult(config, self) + self._inner_results.append(result) + return result + + +class TestSuiteResult(BaseResult): + """ + The test suite specific result. + The _inner_results list stores results of test cases in a given test suite. + Also stores the test suite name. + """ + + _config: TestSuiteConfig + _parent_result: BuildTargetResult + _child_configs: list[str] + + def __init__(self, config: TestSuiteConfig, parent_result: BuildTargetResult): + super(TestSuiteResult, self).__init__() + self._config = config + self._parent_result = parent_result + self._child_configs = config.test_cases + + def add_child_result(self, config: str) -> "TestCaseResult": + result = TestCaseResult(config, self) + self._inner_results.append(result) + return result + + +class TestCaseResult(BaseResult, FixtureResult): + """ + The test case specific result. + Stores the result of the actual test case. + Also stores the test case name. + """ + + _config: str + _parent_result: TestSuiteResult + + def __init__(self, config: str, parent_result: TestSuiteResult): + super(TestCaseResult, self).__init__() + self._config = config + self._parent_result = parent_result + + def block(self): + self.update(Result.BLOCK) + + def update(self, result: Result, error: Exception | None = None) -> None: + self.result = result + self.error = error + + def _get_inner_errors(self) -> list[Exception]: + if self.error: + return [self.error] + return [] + + def add_stats(self, statistics: "Statistics") -> None: + statistics += self.result + + def __bool__(self) -> bool: + return ( + bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) + ) + + +class Statistics(dict): + """ + A helper class used to store the number of test cases by its result + along a few other basic information. + Using a dict provides a convenient way to format the data. + """ + + def __init__(self, dpdk_version: str | None): + super(Statistics, self).__init__() + for result in Result: + self[result.name] = 0 + self["PASS RATE"] = 0.0 + self["DPDK VERSION"] = dpdk_version + + def __iadd__(self, other: Result) -> "Statistics": + """ + Add a Result to the final count. + """ + self[other.name] += 1 + self["PASS RATE"] = ( + float(self[Result.PASS.name]) + * 100 + / sum(self[result.name] for result in Result) + ) + return self + + def __str__(self) -> str: + """ + Provide a string representation of the data. + """ + stats_str = "" + for key, value in self.items(): + stats_str += f"{key:<12} = {value}\n" + # according to docs, we should use \n when writing to text files + # on all platforms + return stats_str -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [RFC PATCH v1 5/5] dts: refactor logging configuration 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš ` (3 preceding siblings ...) 2023-12-20 10:33 ` [RFC PATCH v1 4/5] dts: block all testcases when earlier setup fails Juraj Linkeš @ 2023-12-20 10:33 ` Juraj Linkeš 2024-01-08 18:47 ` [RFC PATCH v1 0/5] test case blocking and logging Jeremy Spewock ` (3 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2023-12-20 10:33 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, yoan.picchi, Luca.Vizzarro Cc: dev, Juraj Linkeš Refactor logging for improved configuration and flexibility, investigating unclear arguments and exploring alternatives for logging test suites into separate files. In addition, efforts were made to ensure that the modules remained independent from the logger module, enabling potential use by other consumers. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/logger.py | 162 ++++++++---------- dts/framework/remote_session/__init__.py | 4 +- dts/framework/remote_session/os_session.py | 6 +- .../remote_session/remote/__init__.py | 7 +- .../remote/interactive_remote_session.py | 7 +- .../remote/interactive_shell.py | 7 +- .../remote_session/remote/remote_session.py | 8 +- .../remote_session/remote/ssh_session.py | 5 +- dts/framework/runner.py | 13 +- dts/framework/test_result.py | 6 +- dts/framework/test_suite.py | 6 +- dts/framework/testbed_model/node.py | 10 +- dts/framework/testbed_model/scapy.py | 6 +- .../testbed_model/traffic_generator.py | 5 +- dts/main.py | 6 +- 15 files changed, 124 insertions(+), 134 deletions(-) diff --git a/dts/framework/logger.py b/dts/framework/logger.py index bb2991e994..43c49c2d03 100644 --- a/dts/framework/logger.py +++ b/dts/framework/logger.py @@ -10,108 +10,98 @@ import logging import os.path -from typing import TypedDict - -from .settings import SETTINGS +from enum import Enum +from logging import FileHandler, StreamHandler +from pathlib import Path date_fmt = "%Y/%m/%d %H:%M:%S" -stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" +stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s" -class LoggerDictType(TypedDict): - logger: "DTSLOG" - name: str - node: str +def init_logger(verbose: bool, output_dir: str): + logging.raiseExceptions = False + DTSLog._output_dir = output_dir + logging.setLoggerClass(DTSLog) -# List for saving all using loggers -Loggers: list[LoggerDictType] = [] + root_logger = logging.getLogger() + root_logger.setLevel(1) + sh = StreamHandler() + sh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) + sh.setLevel(logging.INFO) + if verbose: + sh.setLevel(logging.DEBUG) + root_logger.addHandler(sh) -class DTSLOG(logging.LoggerAdapter): - """ - DTS log class for framework and testsuite. - """ + if not os.path.exists(output_dir): + os.mkdir(output_dir) - _logger: logging.Logger - node: str - sh: logging.StreamHandler - fh: logging.FileHandler - verbose_fh: logging.FileHandler + add_file_handlers(Path(output_dir, "dts")) - def __init__(self, logger: logging.Logger, node: str = "suite"): - self._logger = logger - # 1 means log everything, this will be used by file handlers if their level - # is not set - self._logger.setLevel(1) - self.node = node +def add_file_handlers(log_file_path: Path) -> list[FileHandler]: + root_logger = logging.getLogger() - # add handler to emit to stdout - sh = logging.StreamHandler() - sh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) - sh.setLevel(logging.INFO) # console handler default level + fh = FileHandler(f"{log_file_path}.log") + fh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) + root_logger.addHandler(fh) - if SETTINGS.verbose is True: - sh.setLevel(logging.DEBUG) + verbose_fh = FileHandler(f"{log_file_path}.verbose.log") + verbose_fh.setFormatter( + logging.Formatter( + "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" + "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s", + datefmt=date_fmt, + ) + ) + root_logger.addHandler(verbose_fh) - self._logger.addHandler(sh) - self.sh = sh + return [fh, verbose_fh] - # prepare the output folder - if not os.path.exists(SETTINGS.output_dir): - os.mkdir(SETTINGS.output_dir) - logging_path_prefix = os.path.join(SETTINGS.output_dir, node) +class DtsStage(Enum): + pre_execution = "pre-execution" + execution = "execution" + build_target = "build-target" + suite = "suite" + post_execution = "post-execution" - fh = logging.FileHandler(f"{logging_path_prefix}.log") - fh.setFormatter( - logging.Formatter( - fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s", - datefmt=date_fmt, - ) - ) + def __str__(self) -> str: + return self.value - self._logger.addHandler(fh) - self.fh = fh - - # This outputs EVERYTHING, intended for post-mortem debugging - # Also optimized for processing via AWK (awk -F '|' ...) - verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log") - verbose_fh.setFormatter( - logging.Formatter( - fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" - "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s", - datefmt=date_fmt, - ) - ) - self._logger.addHandler(verbose_fh) - self.verbose_fh = verbose_fh - - super(DTSLOG, self).__init__(self._logger, dict(node=self.node)) - - def logger_exit(self) -> None: - """ - Remove stream handler and logfile handler. - """ - for handler in (self.sh, self.fh, self.verbose_fh): - handler.flush() - self._logger.removeHandler(handler) - - -def getLogger(name: str, node: str = "suite") -> DTSLOG: - """ - Get logger handler and if there's no handler for specified Node will create one. - """ - global Loggers - # return saved logger - logger: LoggerDictType - for logger in Loggers: - if logger["name"] == name and logger["node"] == node: - return logger["logger"] - - # return new logger - dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node) - Loggers.append({"logger": dts_logger, "name": name, "node": node}) - return dts_logger +class DTSLog(logging.Logger): + _stage: DtsStage = DtsStage.pre_execution + _extra_file_handlers: list[FileHandler] = [] + _output_dir: None | str = None + + def makeRecord(self, *args, **kwargs): + record = super().makeRecord(*args, **kwargs) + record.stage = DTSLog._stage + return record + + def set_stage(self, stage: DtsStage, log_file_name: str | None = None): + self._remove_extra_file_handlers() + + if DTSLog._stage != stage: + self.info(f"Moving from stage '{DTSLog._stage}' to stage '{stage}'.") + DTSLog._stage = stage + + if log_file_name: + if DTSLog._output_dir: + DTSLog._extra_file_handlers.extend( + add_file_handlers(Path(DTSLog._output_dir, log_file_name)) + ) + else: + self.warning( + f"Cannot log '{DTSLog._stage}' stage in separate file, " + "output dir is not defined." + ) + + def _remove_extra_file_handlers(self) -> None: + if DTSLog._extra_file_handlers: + for extra_file_handler in DTSLog._extra_file_handlers: + self.root.removeHandler(extra_file_handler) + + DTSLog._extra_file_handlers = [] diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py index 6124417bd7..a4ec2f40ae 100644 --- a/dts/framework/remote_session/__init__.py +++ b/dts/framework/remote_session/__init__.py @@ -11,10 +11,10 @@ """ # pylama:ignore=W0611 +import logging from framework.config import OS, NodeConfiguration from framework.exception import ConfigurationError -from framework.logger import DTSLOG from .linux_session import LinuxSession from .os_session import InteractiveShellType, OSSession @@ -30,7 +30,7 @@ ) -def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession: +def create_session(node_config: NodeConfiguration, name: str, logger: logging.Logger) -> OSSession: match node_config.os: case OS.linux: return LinuxSession(node_config, name, logger) diff --git a/dts/framework/remote_session/os_session.py b/dts/framework/remote_session/os_session.py index 8a709eac1c..2524a4e669 100644 --- a/dts/framework/remote_session/os_session.py +++ b/dts/framework/remote_session/os_session.py @@ -2,6 +2,7 @@ # Copyright(c) 2023 PANTHEON.tech s.r.o. # Copyright(c) 2023 University of New Hampshire +import logging from abc import ABC, abstractmethod from collections.abc import Iterable from ipaddress import IPv4Interface, IPv6Interface @@ -9,7 +10,6 @@ from typing import Type, TypeVar, Union from framework.config import Architecture, NodeConfiguration, NodeInfo -from framework.logger import DTSLOG from framework.remote_session.remote import InteractiveShell from framework.settings import SETTINGS from framework.testbed_model import LogicalCore @@ -36,7 +36,7 @@ class OSSession(ABC): _config: NodeConfiguration name: str - _logger: DTSLOG + _logger: logging.Logger remote_session: RemoteSession interactive_session: InteractiveRemoteSession @@ -44,7 +44,7 @@ def __init__( self, node_config: NodeConfiguration, name: str, - logger: DTSLOG, + logger: logging.Logger, ): self._config = node_config self.name = name diff --git a/dts/framework/remote_session/remote/__init__.py b/dts/framework/remote_session/remote/__init__.py index 06403691a5..4a22155153 100644 --- a/dts/framework/remote_session/remote/__init__.py +++ b/dts/framework/remote_session/remote/__init__.py @@ -4,8 +4,9 @@ # pylama:ignore=W0611 +import logging + from framework.config import NodeConfiguration -from framework.logger import DTSLOG from .interactive_remote_session import InteractiveRemoteSession from .interactive_shell import InteractiveShell @@ -16,12 +17,12 @@ def create_remote_session( - node_config: NodeConfiguration, name: str, logger: DTSLOG + node_config: NodeConfiguration, name: str, logger: logging.Logger ) -> RemoteSession: return SSHSession(node_config, name, logger) def create_interactive_session( - node_config: NodeConfiguration, logger: DTSLOG + node_config: NodeConfiguration, logger: logging.Logger ) -> InteractiveRemoteSession: return InteractiveRemoteSession(node_config, logger) diff --git a/dts/framework/remote_session/remote/interactive_remote_session.py b/dts/framework/remote_session/remote/interactive_remote_session.py index 098ded1bb0..bf0996a747 100644 --- a/dts/framework/remote_session/remote/interactive_remote_session.py +++ b/dts/framework/remote_session/remote/interactive_remote_session.py @@ -2,7 +2,7 @@ # Copyright(c) 2023 University of New Hampshire """Handler for an SSH session dedicated to interactive shells.""" - +import logging import socket import traceback @@ -16,7 +16,6 @@ from framework.config import NodeConfiguration from framework.exception import SSHConnectionError -from framework.logger import DTSLOG class InteractiveRemoteSession: @@ -54,11 +53,11 @@ class InteractiveRemoteSession: username: str password: str session: SSHClient - _logger: DTSLOG + _logger: logging.Logger _node_config: NodeConfiguration _transport: Transport | None - def __init__(self, node_config: NodeConfiguration, _logger: DTSLOG) -> None: + def __init__(self, node_config: NodeConfiguration, _logger: logging.Logger) -> None: self._node_config = node_config self._logger = _logger self.hostname = node_config.hostname diff --git a/dts/framework/remote_session/remote/interactive_shell.py b/dts/framework/remote_session/remote/interactive_shell.py index 4db19fb9b3..b6074838c2 100644 --- a/dts/framework/remote_session/remote/interactive_shell.py +++ b/dts/framework/remote_session/remote/interactive_shell.py @@ -11,14 +11,13 @@ elevated privileges to start it is expected that the method for gaining those privileges is provided when initializing the class. """ - +import logging from abc import ABC from pathlib import PurePath from typing import Callable from paramiko import Channel, SSHClient, channel # type: ignore[import] -from framework.logger import DTSLOG from framework.settings import SETTINGS @@ -58,7 +57,7 @@ class InteractiveShell(ABC): _stdin: channel.ChannelStdinFile _stdout: channel.ChannelFile _ssh_channel: Channel - _logger: DTSLOG + _logger: logging.Logger _timeout: float _app_args: str _default_prompt: str = "" @@ -69,7 +68,7 @@ class InteractiveShell(ABC): def __init__( self, interactive_session: SSHClient, - logger: DTSLOG, + logger: logging.Logger, get_privileged_command: Callable[[str], str] | None, app_args: str = "", timeout: float = SETTINGS.timeout, diff --git a/dts/framework/remote_session/remote/remote_session.py b/dts/framework/remote_session/remote/remote_session.py index 719f7d1ef7..da78e5c921 100644 --- a/dts/framework/remote_session/remote/remote_session.py +++ b/dts/framework/remote_session/remote/remote_session.py @@ -2,14 +2,13 @@ # Copyright(c) 2010-2014 Intel Corporation # Copyright(c) 2022-2023 PANTHEON.tech s.r.o. # Copyright(c) 2022-2023 University of New Hampshire - import dataclasses +import logging from abc import ABC, abstractmethod from pathlib import PurePath from framework.config import NodeConfiguration from framework.exception import RemoteCommandExecutionError -from framework.logger import DTSLOG from framework.settings import SETTINGS @@ -50,14 +49,14 @@ class RemoteSession(ABC): username: str password: str history: list[CommandResult] - _logger: DTSLOG + _logger: logging.Logger _node_config: NodeConfiguration def __init__( self, node_config: NodeConfiguration, session_name: str, - logger: DTSLOG, + logger: logging.Logger, ): self._node_config = node_config @@ -120,7 +119,6 @@ def close(self, force: bool = False) -> None: """ Close the remote session and free all used resources. """ - self._logger.logger_exit() self._close(force) @abstractmethod diff --git a/dts/framework/remote_session/remote/ssh_session.py b/dts/framework/remote_session/remote/ssh_session.py index 1a7ee649ab..42441c4587 100644 --- a/dts/framework/remote_session/remote/ssh_session.py +++ b/dts/framework/remote_session/remote/ssh_session.py @@ -1,6 +1,6 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2023 PANTHEON.tech s.r.o. - +import logging import socket import traceback from pathlib import PurePath @@ -20,7 +20,6 @@ from framework.config import NodeConfiguration from framework.exception import SSHConnectionError, SSHSessionDeadError, SSHTimeoutError -from framework.logger import DTSLOG from .remote_session import CommandResult, RemoteSession @@ -49,7 +48,7 @@ def __init__( self, node_config: NodeConfiguration, session_name: str, - logger: DTSLOG, + logger: logging.Logger, ): super(SSHSession, self).__init__(node_config, session_name, logger) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 28570d4a1c..5c06e4ca1a 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -11,6 +11,7 @@ from copy import deepcopy from dataclasses import dataclass from types import MethodType, ModuleType +from typing import cast from .config import ( BuildTargetConfiguration, @@ -24,7 +25,7 @@ SSHTimeoutError, TestCaseVerifyError, ) -from .logger import DTSLOG, getLogger +from .logger import DTSLog, DtsStage from .settings import SETTINGS from .test_result import ( BuildTargetResult, @@ -68,12 +69,12 @@ def processed_config(self) -> ExecutionConfiguration: class DTSRunner: - _logger: DTSLOG + _logger: DTSLog _result: DTSResult _executions: list[Execution] def __init__(self, configuration: Configuration): - self._logger = getLogger("DTSRunner") + self._logger = cast(DTSLog, logging.getLogger("DTSRunner")) self._result = DTSResult(configuration, self._logger) self._executions = create_executions(configuration.executions) @@ -146,6 +147,7 @@ def _run_execution( Run the given execution. This involves running the execution setup as well as running all build targets in the given execution. """ + self._logger.set_stage(DtsStage.execution) self._logger.info( "Running execution with SUT " f"'{execution.config.system_under_test_node.name}'." @@ -175,6 +177,7 @@ def _run_execution( sut_node.tear_down_execution() execution_result.update_teardown(Result.PASS) except Exception as e: + self._logger.set_stage(DtsStage.execution) self._logger.exception("Execution teardown failed.") execution_result.update_teardown(Result.FAIL, e) @@ -189,6 +192,7 @@ def _run_build_target( """ Run the given build target. """ + self._logger.set_stage(DtsStage.build_target) self._logger.info(f"Running build target '{build_target.name}'.") build_target_result = execution_result.add_child_result(build_target) @@ -209,6 +213,7 @@ def _run_build_target( sut_node.tear_down_build_target() build_target_result.update_teardown(Result.PASS) except Exception as e: + self._logger.set_stage(DtsStage.build_target) self._logger.exception("Build target teardown failed.") build_target_result.update_teardown(Result.FAIL, e) @@ -265,6 +270,7 @@ def _run_test_suite( """ test_suite = test_suite_setup.test_suite(sut_node, tg_node) test_suite_name = test_suite_setup.test_suite.__name__ + self._logger.set_stage(DtsStage.suite, test_suite_name) test_suite_result = build_target_result.add_child_result( test_suite_setup.processed_config() ) @@ -397,6 +403,7 @@ def _exit_dts(self) -> None: self._result.process() if self._logger: + self._logger.set_stage(DtsStage.post_execution) self._logger.info("DTS execution has ended.") logging.shutdown() diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index dba2c55d36..221e75205e 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -6,6 +6,7 @@ Generic result container and reporters """ +import logging import os.path from collections.abc import MutableSequence from enum import Enum, auto @@ -24,7 +25,6 @@ TestSuiteConfig, ) from .exception import DTSError, ErrorSeverity -from .logger import DTSLOG from .settings import SETTINGS @@ -153,13 +153,13 @@ class DTSResult(BaseResult): dpdk_version: str | None _child_configs: list[ExecutionConfiguration] - _logger: DTSLOG + _logger: logging.Logger _errors: list[Exception] _return_code: ErrorSeverity _stats_result: Union["Statistics", None] _stats_filename: str - def __init__(self, configuration: Configuration, logger: DTSLOG): + def __init__(self, configuration: Configuration, logger: logging.Logger): super(DTSResult, self).__init__() self.dpdk_version = None self._child_configs = configuration.executions diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index e73206993d..9c9a8c1e08 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -6,6 +6,7 @@ Base class for creating DTS test cases. """ +import logging from ipaddress import IPv4Interface, IPv6Interface, ip_interface from typing import Union @@ -14,7 +15,6 @@ from scapy.packet import Packet, Padding # type: ignore[import] from .exception import TestCaseVerifyError -from .logger import DTSLOG, getLogger from .testbed_model import SutNode, TGNode from .testbed_model.hw.port import Port, PortLink from .utils import get_packet_summaries @@ -41,7 +41,7 @@ class TestSuite(object): sut_node: SutNode tg_node: TGNode is_blocking = False - _logger: DTSLOG + _logger: logging.Logger _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -59,7 +59,7 @@ def __init__( ): self.sut_node = sut_node self.tg_node = tg_node - self._logger = getLogger(self.__class__.__name__) + self._logger = logging.getLogger(self.__class__.__name__) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py index ef700d8114..a98c58df4f 100644 --- a/dts/framework/testbed_model/node.py +++ b/dts/framework/testbed_model/node.py @@ -6,7 +6,7 @@ """ A node is a generic host that DTS connects to and manages. """ - +import logging from abc import ABC from ipaddress import IPv4Interface, IPv6Interface from typing import Any, Callable, Type, Union @@ -16,7 +16,6 @@ ExecutionConfiguration, NodeConfiguration, ) -from framework.logger import DTSLOG, getLogger from framework.remote_session import InteractiveShellType, OSSession, create_session from framework.settings import SETTINGS @@ -43,7 +42,7 @@ class Node(ABC): name: str lcores: list[LogicalCore] ports: list[Port] - _logger: DTSLOG + _logger: logging.Logger _other_sessions: list[OSSession] _execution_config: ExecutionConfiguration virtual_devices: list[VirtualDevice] @@ -51,7 +50,7 @@ class Node(ABC): def __init__(self, node_config: NodeConfiguration): self.config = node_config self.name = node_config.name - self._logger = getLogger(self.name) + self._logger = logging.getLogger(self.name) self.main_session = create_session(self.config, self.name, self._logger) self._logger.info(f"Connected to node: {self.name}") @@ -137,7 +136,7 @@ def create_session(self, name: str) -> OSSession: connection = create_session( self.config, session_name, - getLogger(session_name, node=self.name), + logging.getLogger(session_name), ) self._other_sessions.append(connection) return connection @@ -237,7 +236,6 @@ def close(self) -> None: self.main_session.close() for session in self._other_sessions: session.close() - self._logger.logger_exit() @staticmethod def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: diff --git a/dts/framework/testbed_model/scapy.py b/dts/framework/testbed_model/scapy.py index 9083e92b3d..61058cd38a 100644 --- a/dts/framework/testbed_model/scapy.py +++ b/dts/framework/testbed_model/scapy.py @@ -13,6 +13,7 @@ """ import inspect +import logging import marshal import time import types @@ -24,7 +25,6 @@ from scapy.packet import Packet # type: ignore[import] from framework.config import OS, ScapyTrafficGeneratorConfig -from framework.logger import DTSLOG, getLogger from framework.remote_session import PythonShell from framework.settings import SETTINGS @@ -190,12 +190,12 @@ class ScapyTrafficGenerator(CapturingTrafficGenerator): rpc_server_proxy: xmlrpc.client.ServerProxy _config: ScapyTrafficGeneratorConfig _tg_node: TGNode - _logger: DTSLOG + _logger: logging.Logger def __init__(self, tg_node: TGNode, config: ScapyTrafficGeneratorConfig): self._config = config self._tg_node = tg_node - self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}") + self._logger = logging.getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}") assert ( self._tg_node.config.os == OS.linux diff --git a/dts/framework/testbed_model/traffic_generator.py b/dts/framework/testbed_model/traffic_generator.py index 28c35d3ce4..6b0838958a 100644 --- a/dts/framework/testbed_model/traffic_generator.py +++ b/dts/framework/testbed_model/traffic_generator.py @@ -7,12 +7,11 @@ These traffic generators can't capture received traffic, only count the number of received packets. """ - +import logging from abc import ABC, abstractmethod from scapy.packet import Packet # type: ignore[import] -from framework.logger import DTSLOG from framework.utils import get_packet_summaries from .hw.port import Port @@ -24,7 +23,7 @@ class TrafficGenerator(ABC): Defines the few basic methods that each traffic generator must implement. """ - _logger: DTSLOG + _logger: logging.Logger def send_packet(self, packet: Packet, port: Port) -> None: """Send a packet and block until it is fully sent. diff --git a/dts/main.py b/dts/main.py index 879ce5cb89..f2828148f0 100755 --- a/dts/main.py +++ b/dts/main.py @@ -8,18 +8,18 @@ A test framework for testing DPDK. """ -import logging - from framework.config import load_config +from framework.logger import init_logger from framework.runner import DTSRunner +from framework.settings import SETTINGS def main() -> None: + init_logger(SETTINGS.verbose, SETTINGS.output_dir) dts = DTSRunner(configuration=load_config()) dts.run() # Main program begins here if __name__ == "__main__": - logging.raiseExceptions = True main() -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [RFC PATCH v1 0/5] test case blocking and logging 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš ` (4 preceding siblings ...) 2023-12-20 10:33 ` [RFC PATCH v1 5/5] dts: refactor logging configuration Juraj Linkeš @ 2024-01-08 18:47 ` Jeremy Spewock 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš ` (2 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Jeremy Spewock @ 2024-01-08 18:47 UTC (permalink / raw) To: Juraj Linkeš Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, yoan.picchi, Luca.Vizzarro, dev [-- Attachment #1: Type: text/plain, Size: 85 bytes --] Definitely worth-while changes. I looked them over and it all looks good so far. +1 [-- Attachment #2: Type: text/html, Size: 323 bytes --] ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 0/7] test case blocking and logging 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš ` (5 preceding siblings ...) 2024-01-08 18:47 ` [RFC PATCH v1 0/5] test case blocking and logging Jeremy Spewock @ 2024-02-06 14:57 ` Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 1/7] dts: convert dts.py methods to class Juraj Linkeš ` (6 more replies) 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš 8 siblings, 7 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš We currently don't record test case results that couldn't be executed because of a previous failure, such as when a test suite setup failed, resulting in no executed test cases. In order to record the test cases that couldn't be executed, we must know the lists of test suites and test cases ahead of the actual test suite execution, as an error could occur before we even start executing test suites. In addition, the patch series contains two refactors and one improvement. The first refactor is closely related. The dts.py was renamed to runner.py and given a clear purpose - running the test suites and all other orchestration needed to run test suites. The logic for this was not all in the original dts.py module and it was brought there. The runner is also responsible for recording results, which is the blocked test cases are recorded. The other refactor, logging, is related to the first refactor. The logging module was simplified while extending capabilities. Each test suite logs into its own log file in addition to the main log file which the runner must handle (as it knows when we start executing particular test suites). The runner also handles the switching between execution stages for the purposes of logging. The one aforementioned improvement is in unifying how we specify test cases in the conf.yaml file and in the environment variable/command line argument. v2: Rebase and update of the whole patch. There are changes in all parts of the code, mainly improving the design and logic. Also added the last patch which improves test suite specification on the cmdline. Juraj Linkeš (7): dts: convert dts.py methods to class dts: move test suite execution logic to DTSRunner dts: filter test suites in executions dts: reorganize test result dts: block all test cases when earlier setup fails dts: refactor logging configuration dts: improve test suite and case filtering doc/guides/tools/dts.rst | 14 +- dts/framework/config/__init__.py | 36 +- dts/framework/config/conf_yaml_schema.json | 2 +- dts/framework/dts.py | 338 --------- dts/framework/logger.py | 235 +++--- dts/framework/remote_session/__init__.py | 6 +- .../interactive_remote_session.py | 6 +- .../remote_session/interactive_shell.py | 6 +- .../remote_session/remote_session.py | 8 +- dts/framework/runner.py | 701 ++++++++++++++++++ dts/framework/settings.py | 188 +++-- dts/framework/test_result.py | 565 ++++++++------ dts/framework/test_suite.py | 239 +----- dts/framework/testbed_model/node.py | 11 +- dts/framework/testbed_model/os_session.py | 7 +- .../traffic_generator/traffic_generator.py | 6 +- dts/main.py | 9 +- dts/pyproject.toml | 3 + dts/tests/TestSuite_smoke_tests.py | 2 +- 19 files changed, 1354 insertions(+), 1028 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 1/7] dts: convert dts.py methods to class 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš @ 2024-02-06 14:57 ` Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš ` (5 subsequent siblings) 6 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš The dts.py module deviates from the rest of the code without a clear reason. Converting it into a class and using better naming will improve organization and code readability. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/dts.py | 338 ---------------------------------------- dts/framework/runner.py | 333 +++++++++++++++++++++++++++++++++++++++ dts/main.py | 6 +- 3 files changed, 337 insertions(+), 340 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py diff --git a/dts/framework/dts.py b/dts/framework/dts.py deleted file mode 100644 index e16d4578a0..0000000000 --- a/dts/framework/dts.py +++ /dev/null @@ -1,338 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2010-2019 Intel Corporation -# Copyright(c) 2022-2023 PANTHEON.tech s.r.o. -# Copyright(c) 2022-2023 University of New Hampshire - -r"""Test suite runner module. - -A DTS run is split into stages: - - #. Execution stage, - #. Build target stage, - #. Test suite stage, - #. Test case stage. - -The module is responsible for running tests on testbeds defined in the test run configuration. -Each setup or teardown of each stage is recorded in a :class:`~.test_result.DTSResult` or -one of its subclasses. The test case results are also recorded. - -If an error occurs, the current stage is aborted, the error is recorded and the run continues in -the next iteration of the same stage. The return code is the highest `severity` of all -:class:`~.exception.DTSError`\s. - -Example: - An error occurs in a build target setup. The current build target is aborted and the run - continues with the next build target. If the errored build target was the last one in the given - execution, the next execution begins. - -Attributes: - dts_logger: The logger instance used in this module. - result: The top level result used in the module. -""" - -import sys - -from .config import ( - BuildTargetConfiguration, - ExecutionConfiguration, - TestSuiteConfig, - load_config, -) -from .exception import BlockingTestSuiteError -from .logger import DTSLOG, getLogger -from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result -from .test_suite import get_test_suites -from .testbed_model import SutNode, TGNode - -# dummy defaults to satisfy linters -dts_logger: DTSLOG = None # type: ignore[assignment] -result: DTSResult = DTSResult(dts_logger) - - -def run_all() -> None: - """Run all build targets in all executions from the test run configuration. - - Before running test suites, executions and build targets are first set up. - The executions and build targets defined in the test run configuration are iterated over. - The executions define which tests to run and where to run them and build targets define - the DPDK build setup. - - The tests suites are set up for each execution/build target tuple and each scheduled - test case within the test suite is set up, executed and torn down. After all test cases - have been executed, the test suite is torn down and the next build target will be tested. - - All the nested steps look like this: - - #. Execution setup - - #. Build target setup - - #. Test suite setup - - #. Test case setup - #. Test case logic - #. Test case teardown - - #. Test suite teardown - - #. Build target teardown - - #. Execution teardown - - The test cases are filtered according to the specification in the test run configuration and - the :option:`--test-cases` command line argument or - the :envvar:`DTS_TESTCASES` environment variable. - """ - global dts_logger - global result - - # create a regular DTS logger and create a new result with it - dts_logger = getLogger("DTSRunner") - result = DTSResult(dts_logger) - - # check the python version of the server that run dts - _check_dts_python_version() - - sut_nodes: dict[str, SutNode] = {} - tg_nodes: dict[str, TGNode] = {} - try: - # for all Execution sections - for execution in load_config().executions: - sut_node = sut_nodes.get(execution.system_under_test_node.name) - tg_node = tg_nodes.get(execution.traffic_generator_node.name) - - try: - if not sut_node: - sut_node = SutNode(execution.system_under_test_node) - sut_nodes[sut_node.name] = sut_node - if not tg_node: - tg_node = TGNode(execution.traffic_generator_node) - tg_nodes[tg_node.name] = tg_node - result.update_setup(Result.PASS) - except Exception as e: - failed_node = execution.system_under_test_node.name - if sut_node: - failed_node = execution.traffic_generator_node.name - dts_logger.exception(f"Creation of node {failed_node} failed.") - result.update_setup(Result.FAIL, e) - - else: - _run_execution(sut_node, tg_node, execution, result) - - except Exception as e: - dts_logger.exception("An unexpected error has occurred.") - result.add_error(e) - raise - - finally: - try: - for node in (sut_nodes | tg_nodes).values(): - node.close() - result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Final cleanup of nodes failed.") - result.update_teardown(Result.ERROR, e) - - # we need to put the sys.exit call outside the finally clause to make sure - # that unexpected exceptions will propagate - # in that case, the error that should be reported is the uncaught exception as - # that is a severe error originating from the framework - # at that point, we'll only have partial results which could be impacted by the - # error causing the uncaught exception, making them uninterpretable - _exit_dts() - - -def _check_dts_python_version() -> None: - """Check the required Python version - v3.10.""" - - def RED(text: str) -> str: - return f"\u001B[31;1m{str(text)}\u001B[0m" - - if sys.version_info.major < 3 or (sys.version_info.major == 3 and sys.version_info.minor < 10): - print( - RED( - ( - "WARNING: DTS execution node's python version is lower than" - "python 3.10, is deprecated and will not work in future releases." - ) - ), - file=sys.stderr, - ) - print(RED("Please use Python >= 3.10 instead"), file=sys.stderr) - - -def _run_execution( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - result: DTSResult, -) -> None: - """Run the given execution. - - This involves running the execution setup as well as running all build targets - in the given execution. After that, execution teardown is run. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - execution: An execution's test run configuration. - result: The top level result object. - """ - dts_logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") - execution_result = result.add_execution(sut_node.config) - execution_result.add_sut_info(sut_node.node_info) - - try: - sut_node.set_up_execution(execution) - execution_result.update_setup(Result.PASS) - except Exception as e: - dts_logger.exception("Execution setup failed.") - execution_result.update_setup(Result.FAIL, e) - - else: - for build_target in execution.build_targets: - _run_build_target(sut_node, tg_node, build_target, execution, execution_result) - - finally: - try: - sut_node.tear_down_execution() - execution_result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Execution teardown failed.") - execution_result.update_teardown(Result.FAIL, e) - - -def _run_build_target( - sut_node: SutNode, - tg_node: TGNode, - build_target: BuildTargetConfiguration, - execution: ExecutionConfiguration, - execution_result: ExecutionResult, -) -> None: - """Run the given build target. - - This involves running the build target setup as well as running all test suites - in the given execution the build target is defined in. - After that, build target teardown is run. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - build_target: A build target's test run configuration. - execution: The build target's execution's test run configuration. - execution_result: The execution level result object associated with the execution. - """ - dts_logger.info(f"Running build target '{build_target.name}'.") - build_target_result = execution_result.add_build_target(build_target) - - try: - sut_node.set_up_build_target(build_target) - result.dpdk_version = sut_node.dpdk_version - build_target_result.add_build_target_info(sut_node.get_build_target_info()) - build_target_result.update_setup(Result.PASS) - except Exception as e: - dts_logger.exception("Build target setup failed.") - build_target_result.update_setup(Result.FAIL, e) - - else: - _run_all_suites(sut_node, tg_node, execution, build_target_result) - - finally: - try: - sut_node.tear_down_build_target() - build_target_result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Build target teardown failed.") - build_target_result.update_teardown(Result.FAIL, e) - - -def _run_all_suites( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, -) -> None: - """Run the execution's (possibly a subset) test suites using the current build target. - - The function assumes the build target we're testing has already been built on the SUT node. - The current build target thus corresponds to the current DPDK build present on the SUT node. - - If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites - in the current build target won't be executed. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - execution: The execution's test run configuration associated with the current build target. - build_target_result: The build target level result object associated - with the current build target. - """ - end_build_target = False - if not execution.skip_smoke_tests: - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] - for test_suite_config in execution.test_suites: - try: - _run_single_suite(sut_node, tg_node, execution, build_target_result, test_suite_config) - except BlockingTestSuiteError as e: - dts_logger.exception( - f"An error occurred within {test_suite_config.test_suite}. Skipping build target." - ) - result.add_error(e) - end_build_target = True - # if a blocking test failed and we need to bail out of suite executions - if end_build_target: - break - - -def _run_single_suite( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, - test_suite_config: TestSuiteConfig, -) -> None: - """Run all test suite in a single test suite module. - - The function assumes the build target we're testing has already been built on the SUT node. - The current build target thus corresponds to the current DPDK build present on the SUT node. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - execution: The execution's test run configuration associated with the current build target. - build_target_result: The build target level result object associated - with the current build target. - test_suite_config: Test suite test run configuration specifying the test suite module - and possibly a subset of test cases of test suites in that module. - - Raises: - BlockingTestSuiteError: If a blocking test suite fails. - """ - try: - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" - test_suite_classes = get_test_suites(full_suite_path) - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) - dts_logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") - except Exception as e: - dts_logger.exception("An error occurred when searching for test suites.") - result.update_setup(Result.ERROR, e) - - else: - for test_suite_class in test_suite_classes: - test_suite = test_suite_class( - sut_node, - tg_node, - test_suite_config.test_cases, - execution.func, - build_target_result, - ) - test_suite.run() - - -def _exit_dts() -> None: - """Process all errors and exit with the proper exit code.""" - result.process() - - if dts_logger: - dts_logger.info("DTS execution has ended.") - sys.exit(result.get_return_code()) diff --git a/dts/framework/runner.py b/dts/framework/runner.py new file mode 100644 index 0000000000..acc1c4d6db --- /dev/null +++ b/dts/framework/runner.py @@ -0,0 +1,333 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2019 Intel Corporation +# Copyright(c) 2022-2023 PANTHEON.tech s.r.o. +# Copyright(c) 2022-2023 University of New Hampshire + +"""Test suite runner module. + +The module is responsible for running DTS in a series of stages: + + #. Execution stage, + #. Build target stage, + #. Test suite stage, + #. Test case stage. + +The execution and build target stages set up the environment before running test suites. +The test suite stage sets up steps common to all test cases +and the test case stage runs test cases individually. +""" + +import logging +import sys + +from .config import ( + BuildTargetConfiguration, + ExecutionConfiguration, + TestSuiteConfig, + load_config, +) +from .exception import BlockingTestSuiteError +from .logger import DTSLOG, getLogger +from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result +from .test_suite import get_test_suites +from .testbed_model import SutNode, TGNode + + +class DTSRunner: + r"""Test suite runner class. + + The class is responsible for running tests on testbeds defined in the test run configuration. + Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult` + or one of its subclasses. The test case results are also recorded. + + If an error occurs, the current stage is aborted, the error is recorded and the run continues in + the next iteration of the same stage. The return code is the highest `severity` of all + :class:`~.framework.exception.DTSError`\s. + + Example: + An error occurs in a build target setup. The current build target is aborted and the run + continues with the next build target. If the errored build target was the last one in the + given execution, the next execution begins. + """ + + _logger: DTSLOG + _result: DTSResult + + def __init__(self): + """Initialize the instance with logger and result.""" + self._logger = getLogger("DTSRunner") + self._result = DTSResult(self._logger) + + def run(self): + """Run all build targets in all executions from the test run configuration. + + Before running test suites, executions and build targets are first set up. + The executions and build targets defined in the test run configuration are iterated over. + The executions define which tests to run and where to run them and build targets define + the DPDK build setup. + + The tests suites are set up for each execution/build target tuple and each discovered + test case within the test suite is set up, executed and torn down. After all test cases + have been executed, the test suite is torn down and the next build target will be tested. + + All the nested steps look like this: + + #. Execution setup + + #. Build target setup + + #. Test suite setup + + #. Test case setup + #. Test case logic + #. Test case teardown + + #. Test suite teardown + + #. Build target teardown + + #. Execution teardown + + The test cases are filtered according to the specification in the test run configuration and + the :option:`--test-cases` command line argument or + the :envvar:`DTS_TESTCASES` environment variable. + """ + sut_nodes: dict[str, SutNode] = {} + tg_nodes: dict[str, TGNode] = {} + try: + # check the python version of the server that runs dts + self._check_dts_python_version() + + # for all Execution sections + for execution in load_config().executions: + sut_node = sut_nodes.get(execution.system_under_test_node.name) + tg_node = tg_nodes.get(execution.traffic_generator_node.name) + + try: + if not sut_node: + sut_node = SutNode(execution.system_under_test_node) + sut_nodes[sut_node.name] = sut_node + if not tg_node: + tg_node = TGNode(execution.traffic_generator_node) + tg_nodes[tg_node.name] = tg_node + self._result.update_setup(Result.PASS) + except Exception as e: + failed_node = execution.system_under_test_node.name + if sut_node: + failed_node = execution.traffic_generator_node.name + self._logger.exception(f"The Creation of node {failed_node} failed.") + self._result.update_setup(Result.FAIL, e) + + else: + self._run_execution(sut_node, tg_node, execution) + + except Exception as e: + self._logger.exception("An unexpected error has occurred.") + self._result.add_error(e) + raise + + finally: + try: + for node in (sut_nodes | tg_nodes).values(): + node.close() + self._result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("The final cleanup of nodes failed.") + self._result.update_teardown(Result.ERROR, e) + + # we need to put the sys.exit call outside the finally clause to make sure + # that unexpected exceptions will propagate + # in that case, the error that should be reported is the uncaught exception as + # that is a severe error originating from the framework + # at that point, we'll only have partial results which could be impacted by the + # error causing the uncaught exception, making them uninterpretable + self._exit_dts() + + def _check_dts_python_version(self) -> None: + """Check the required Python version - v3.10.""" + if sys.version_info.major < 3 or ( + sys.version_info.major == 3 and sys.version_info.minor < 10 + ): + self._logger.warning( + "DTS execution node's python version is lower than Python 3.10, " + "is deprecated and will not work in future releases." + ) + self._logger.warning("Please use Python >= 3.10 instead.") + + def _run_execution( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + ) -> None: + """Run the given execution. + + This involves running the execution setup as well as running all build targets + in the given execution. After that, execution teardown is run. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: An execution's test run configuration. + """ + self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") + execution_result = self._result.add_execution(sut_node.config) + execution_result.add_sut_info(sut_node.node_info) + + try: + sut_node.set_up_execution(execution) + execution_result.update_setup(Result.PASS) + except Exception as e: + self._logger.exception("Execution setup failed.") + execution_result.update_setup(Result.FAIL, e) + + else: + for build_target in execution.build_targets: + self._run_build_target(sut_node, tg_node, build_target, execution, execution_result) + + finally: + try: + sut_node.tear_down_execution() + execution_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("Execution teardown failed.") + execution_result.update_teardown(Result.FAIL, e) + + def _run_build_target( + self, + sut_node: SutNode, + tg_node: TGNode, + build_target: BuildTargetConfiguration, + execution: ExecutionConfiguration, + execution_result: ExecutionResult, + ) -> None: + """Run the given build target. + + This involves running the build target setup as well as running all test suites + of the build target's execution. + After that, build target teardown is run. + + Args: + sut_node: The execution's sut node. + tg_node: The execution's tg node. + build_target: A build target's test run configuration. + execution: The build target's execution's test run configuration. + execution_result: The execution level result object associated with the execution. + """ + self._logger.info(f"Running build target '{build_target.name}'.") + build_target_result = execution_result.add_build_target(build_target) + + try: + sut_node.set_up_build_target(build_target) + self._result.dpdk_version = sut_node.dpdk_version + build_target_result.add_build_target_info(sut_node.get_build_target_info()) + build_target_result.update_setup(Result.PASS) + except Exception as e: + self._logger.exception("Build target setup failed.") + build_target_result.update_setup(Result.FAIL, e) + + else: + self._run_all_suites(sut_node, tg_node, execution, build_target_result) + + finally: + try: + sut_node.tear_down_build_target() + build_target_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("Build target teardown failed.") + build_target_result.update_teardown(Result.FAIL, e) + + def _run_all_suites( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + build_target_result: BuildTargetResult, + ) -> None: + """Run the execution's (possibly a subset of) test suites using the current build target. + + The method assumes the build target we're testing has already been built on the SUT node. + The current build target thus corresponds to the current DPDK build present on the SUT node. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: The execution's test run configuration associated + with the current build target. + build_target_result: The build target level result object associated + with the current build target. + """ + end_build_target = False + if not execution.skip_smoke_tests: + execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] + for test_suite_config in execution.test_suites: + try: + self._run_single_suite( + sut_node, tg_node, execution, build_target_result, test_suite_config + ) + except BlockingTestSuiteError as e: + self._logger.exception( + f"An error occurred within {test_suite_config.test_suite}. " + "Skipping build target..." + ) + self._result.add_error(e) + end_build_target = True + # if a blocking test failed and we need to bail out of suite executions + if end_build_target: + break + + def _run_single_suite( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + build_target_result: BuildTargetResult, + test_suite_config: TestSuiteConfig, + ) -> None: + """Run all test suites in a single test suite module. + + The method assumes the build target we're testing has already been built on the SUT node. + The current build target thus corresponds to the current DPDK build present on the SUT node. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: The execution's test run configuration associated + with the current build target. + build_target_result: The build target level result object associated + with the current build target. + test_suite_config: Test suite test run configuration specifying the test suite module + and possibly a subset of test cases of test suites in that module. + + Raises: + BlockingTestSuiteError: If a blocking test suite fails. + """ + try: + full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" + test_suite_classes = get_test_suites(full_suite_path) + suites_str = ", ".join((x.__name__ for x in test_suite_classes)) + self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") + except Exception as e: + self._logger.exception("An error occurred when searching for test suites.") + self._result.update_setup(Result.ERROR, e) + + else: + for test_suite_class in test_suite_classes: + test_suite = test_suite_class( + sut_node, + tg_node, + test_suite_config.test_cases, + execution.func, + build_target_result, + ) + test_suite.run() + + def _exit_dts(self) -> None: + """Process all errors and exit with the proper exit code.""" + self._result.process() + + if self._logger: + self._logger.info("DTS execution has ended.") + + logging.shutdown() + sys.exit(self._result.get_return_code()) diff --git a/dts/main.py b/dts/main.py index f703615d11..1ffe8ff81f 100755 --- a/dts/main.py +++ b/dts/main.py @@ -21,9 +21,11 @@ def main() -> None: be modified before the settings module is imported anywhere else in the framework. """ settings.SETTINGS = settings.get_settings() - from framework import dts - dts.run_all() + from framework.runner import DTSRunner + + dts = DTSRunner() + dts.run() # Main program begins here -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 2/7] dts: move test suite execution logic to DTSRunner 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 1/7] dts: convert dts.py methods to class Juraj Linkeš @ 2024-02-06 14:57 ` Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 3/7] dts: filter test suites in executions Juraj Linkeš ` (4 subsequent siblings) 6 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš Move the code responsible for running the test suite from the TestSuite class to the DTSRunner class. This restructuring decision was made to consolidate and unify the related logic into a single unit. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/runner.py | 175 ++++++++++++++++++++++++++++++++---- dts/framework/test_suite.py | 152 ++----------------------------- 2 files changed, 169 insertions(+), 158 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index acc1c4d6db..933685d638 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -19,6 +19,7 @@ import logging import sys +from types import MethodType from .config import ( BuildTargetConfiguration, @@ -26,10 +27,18 @@ TestSuiteConfig, load_config, ) -from .exception import BlockingTestSuiteError +from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError from .logger import DTSLOG, getLogger -from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result -from .test_suite import get_test_suites +from .settings import SETTINGS +from .test_result import ( + BuildTargetResult, + DTSResult, + ExecutionResult, + Result, + TestCaseResult, + TestSuiteResult, +) +from .test_suite import TestSuite, get_test_suites from .testbed_model import SutNode, TGNode @@ -227,7 +236,7 @@ def _run_build_target( build_target_result.update_setup(Result.FAIL, e) else: - self._run_all_suites(sut_node, tg_node, execution, build_target_result) + self._run_test_suites(sut_node, tg_node, execution, build_target_result) finally: try: @@ -237,7 +246,7 @@ def _run_build_target( self._logger.exception("Build target teardown failed.") build_target_result.update_teardown(Result.FAIL, e) - def _run_all_suites( + def _run_test_suites( self, sut_node: SutNode, tg_node: TGNode, @@ -249,6 +258,9 @@ def _run_all_suites( The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. + If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites + in the current build target won't be executed. + Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. @@ -262,7 +274,7 @@ def _run_all_suites( execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] for test_suite_config in execution.test_suites: try: - self._run_single_suite( + self._run_test_suite_module( sut_node, tg_node, execution, build_target_result, test_suite_config ) except BlockingTestSuiteError as e: @@ -276,7 +288,7 @@ def _run_all_suites( if end_build_target: break - def _run_single_suite( + def _run_test_suite_module( self, sut_node: SutNode, tg_node: TGNode, @@ -284,11 +296,18 @@ def _run_single_suite( build_target_result: BuildTargetResult, test_suite_config: TestSuiteConfig, ) -> None: - """Run all test suites in a single test suite module. + """Set up, execute and tear down all test suites in a single test suite module. The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. + Test suite execution consists of running the discovered test cases. + A test case run consists of setup, execution and teardown of said test case. + + Record the setup and the teardown and handle failures. + + The test cases to execute are discovered when creating the :class:`TestSuite` object. + Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. @@ -313,14 +332,140 @@ def _run_single_suite( else: for test_suite_class in test_suite_classes: - test_suite = test_suite_class( - sut_node, - tg_node, - test_suite_config.test_cases, - execution.func, - build_target_result, + test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases) + + test_suite_name = test_suite.__class__.__name__ + test_suite_result = build_target_result.add_test_suite(test_suite_name) + try: + self._logger.info(f"Starting test suite setup: {test_suite_name}") + test_suite.set_up_suite() + test_suite_result.update_setup(Result.PASS) + self._logger.info(f"Test suite setup successful: {test_suite_name}") + except Exception as e: + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") + test_suite_result.update_setup(Result.ERROR, e) + + else: + self._execute_test_suite(execution.func, test_suite, test_suite_result) + + finally: + try: + test_suite.tear_down_suite() + sut_node.kill_cleanup_dpdk_apps() + test_suite_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") + self._logger.warning( + f"Test suite '{test_suite_name}' teardown failed, " + f"the next test suite may be affected." + ) + test_suite_result.update_setup(Result.ERROR, e) + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: + raise BlockingTestSuiteError(test_suite_name) + + def _execute_test_suite( + self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult + ) -> None: + """Execute all discovered test cases in `test_suite`. + + If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment + variable is set, in case of a test case failure, the test case will be executed again + until it passes or it fails that many times in addition of the first failure. + + Args: + func: Whether to execute functional test cases. + test_suite: The test suite object. + test_suite_result: The test suite level result object associated + with the current test suite. + """ + if func: + for test_case_method in test_suite._get_functional_test_cases(): + test_case_name = test_case_method.__name__ + test_case_result = test_suite_result.add_test_case(test_case_name) + all_attempts = SETTINGS.re_run + 1 + attempt_nr = 1 + self._run_test_case(test_suite, test_case_method, test_case_result) + while not test_case_result and attempt_nr < all_attempts: + attempt_nr += 1 + self._logger.info( + f"Re-running FAILED test case '{test_case_name}'. " + f"Attempt number {attempt_nr} out of {all_attempts}." + ) + self._run_test_case(test_suite, test_case_method, test_case_result) + + def _run_test_case( + self, + test_suite: TestSuite, + test_case_method: MethodType, + test_case_result: TestCaseResult, + ) -> None: + """Setup, execute and teardown a test case in `test_suite`. + + Record the result of the setup and the teardown and handle failures. + + Args: + test_suite: The test suite object. + test_case_method: The test case method. + test_case_result: The test case level result object associated + with the current test case. + """ + test_case_name = test_case_method.__name__ + + try: + # run set_up function for each case + test_suite.set_up_test_case() + test_case_result.update_setup(Result.PASS) + except SSHTimeoutError as e: + self._logger.exception(f"Test case setup FAILED: {test_case_name}") + test_case_result.update_setup(Result.FAIL, e) + except Exception as e: + self._logger.exception(f"Test case setup ERROR: {test_case_name}") + test_case_result.update_setup(Result.ERROR, e) + + else: + # run test case if setup was successful + self._execute_test_case(test_case_method, test_case_result) + + finally: + try: + test_suite.tear_down_test_case() + test_case_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test case teardown ERROR: {test_case_name}") + self._logger.warning( + f"Test case '{test_case_name}' teardown failed, " + f"the next test case may be affected." ) - test_suite.run() + test_case_result.update_teardown(Result.ERROR, e) + test_case_result.update(Result.ERROR) + + def _execute_test_case( + self, test_case_method: MethodType, test_case_result: TestCaseResult + ) -> None: + """Execute one test case, record the result and handle failures. + + Args: + test_case_method: The test case method. + test_case_result: The test case level result object associated + with the current test case. + """ + test_case_name = test_case_method.__name__ + try: + self._logger.info(f"Starting test case execution: {test_case_name}") + test_case_method() + test_case_result.update(Result.PASS) + self._logger.info(f"Test case execution PASSED: {test_case_name}") + + except TestCaseVerifyError as e: + self._logger.exception(f"Test case execution FAILED: {test_case_name}") + test_case_result.update(Result.FAIL, e) + except Exception as e: + self._logger.exception(f"Test case execution ERROR: {test_case_name}") + test_case_result.update(Result.ERROR, e) + except KeyboardInterrupt: + self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}") + test_case_result.update(Result.SKIP) + raise KeyboardInterrupt("Stop DTS") def _exit_dts(self) -> None: """Process all errors and exit with the proper exit code.""" diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index dfb391ffbd..b02fd36147 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -8,7 +8,6 @@ must be extended by subclasses which add test cases. The :class:`TestSuite` contains the basics needed by subclasses: - * Test suite and test case execution flow, * Testbed (SUT, TG) configuration, * Packet sending and verification, * Test case verification. @@ -28,27 +27,22 @@ from scapy.layers.l2 import Ether # type: ignore[import] from scapy.packet import Packet, Padding # type: ignore[import] -from .exception import ( - BlockingTestSuiteError, - ConfigurationError, - SSHTimeoutError, - TestCaseVerifyError, -) +from .exception import ConfigurationError, TestCaseVerifyError from .logger import DTSLOG, getLogger from .settings import SETTINGS -from .test_result import BuildTargetResult, Result, TestCaseResult, TestSuiteResult from .testbed_model import Port, PortLink, SutNode, TGNode from .utils import get_packet_summaries class TestSuite(object): - """The base class with methods for handling the basic flow of a test suite. + """The base class with building blocks needed by most test cases. * Test case filtering and collection, - * Test suite setup/cleanup, - * Test setup/cleanup, - * Test case execution, - * Error handling and results storage. + * Test suite setup/cleanup methods to override, + * Test case setup/cleanup methods to override, + * Test case verification, + * Testbed configuration, + * Traffic sending and verification. Test cases are implemented by subclasses. Test cases are all methods starting with ``test_``, further divided into performance test cases (starting with ``test_perf_``) @@ -60,10 +54,6 @@ class TestSuite(object): The union of both lists will be used. Any unknown test cases from the latter lists will be silently ignored. - If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable - is set, in case of a test case failure, the test case will be executed again until it passes - or it fails that many times in addition of the first failure. - The methods named ``[set_up|tear_down]_[suite|test_case]`` should be overridden in subclasses if the appropriate test suite/test case fixtures are needed. @@ -82,8 +72,6 @@ class TestSuite(object): is_blocking: ClassVar[bool] = False _logger: DTSLOG _test_cases_to_run: list[str] - _func: bool - _result: TestSuiteResult _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -99,30 +87,23 @@ def __init__( sut_node: SutNode, tg_node: TGNode, test_cases: list[str], - func: bool, - build_target_result: BuildTargetResult, ): """Initialize the test suite testbed information and basic configuration. - Process what test cases to run, create the associated - :class:`~.test_result.TestSuiteResult`, find links between ports - and set up default IP addresses to be used when configuring them. + Process what test cases to run, find links between ports and set up + default IP addresses to be used when configuring them. Args: sut_node: The SUT node where the test suite will run. tg_node: The TG node where the test suite will run. test_cases: The list of test cases to execute. If empty, all test cases will be executed. - func: Whether to run functional tests. - build_target_result: The build target result this test suite is run in. """ self.sut_node = sut_node self.tg_node = tg_node self._logger = getLogger(self.__class__.__name__) self._test_cases_to_run = test_cases self._test_cases_to_run.extend(SETTINGS.test_cases) - self._func = func - self._result = build_target_result.add_test_suite(self.__class__.__name__) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( @@ -384,62 +365,6 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: return False return True - def run(self) -> None: - """Set up, execute and tear down the whole suite. - - Test suite execution consists of running all test cases scheduled to be executed. - A test case run consists of setup, execution and teardown of said test case. - - Record the setup and the teardown and handle failures. - - The list of scheduled test cases is constructed when creating the :class:`TestSuite` object. - """ - test_suite_name = self.__class__.__name__ - - try: - self._logger.info(f"Starting test suite setup: {test_suite_name}") - self.set_up_suite() - self._result.update_setup(Result.PASS) - self._logger.info(f"Test suite setup successful: {test_suite_name}") - except Exception as e: - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") - self._result.update_setup(Result.ERROR, e) - - else: - self._execute_test_suite() - - finally: - try: - self.tear_down_suite() - self.sut_node.kill_cleanup_dpdk_apps() - self._result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") - self._logger.warning( - f"Test suite '{test_suite_name}' teardown failed, " - f"the next test suite may be affected." - ) - self._result.update_setup(Result.ERROR, e) - if len(self._result.get_errors()) > 0 and self.is_blocking: - raise BlockingTestSuiteError(test_suite_name) - - def _execute_test_suite(self) -> None: - """Execute all test cases scheduled to be executed in this suite.""" - if self._func: - for test_case_method in self._get_functional_test_cases(): - test_case_name = test_case_method.__name__ - test_case_result = self._result.add_test_case(test_case_name) - all_attempts = SETTINGS.re_run + 1 - attempt_nr = 1 - self._run_test_case(test_case_method, test_case_result) - while not test_case_result and attempt_nr < all_attempts: - attempt_nr += 1 - self._logger.info( - f"Re-running FAILED test case '{test_case_name}'. " - f"Attempt number {attempt_nr} out of {all_attempts}." - ) - self._run_test_case(test_case_method, test_case_result) - def _get_functional_test_cases(self) -> list[MethodType]: """Get all functional test cases defined in this TestSuite. @@ -471,65 +396,6 @@ def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool return match - def _run_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult - ) -> None: - """Setup, execute and teardown a test case in this suite. - - Record the result of the setup and the teardown and handle failures. - """ - test_case_name = test_case_method.__name__ - - try: - # run set_up function for each case - self.set_up_test_case() - test_case_result.update_setup(Result.PASS) - except SSHTimeoutError as e: - self._logger.exception(f"Test case setup FAILED: {test_case_name}") - test_case_result.update_setup(Result.FAIL, e) - except Exception as e: - self._logger.exception(f"Test case setup ERROR: {test_case_name}") - test_case_result.update_setup(Result.ERROR, e) - - else: - # run test case if setup was successful - self._execute_test_case(test_case_method, test_case_result) - - finally: - try: - self.tear_down_test_case() - test_case_result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test case teardown ERROR: {test_case_name}") - self._logger.warning( - f"Test case '{test_case_name}' teardown failed, " - f"the next test case may be affected." - ) - test_case_result.update_teardown(Result.ERROR, e) - test_case_result.update(Result.ERROR) - - def _execute_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult - ) -> None: - """Execute one test case, record the result and handle failures.""" - test_case_name = test_case_method.__name__ - try: - self._logger.info(f"Starting test case execution: {test_case_name}") - test_case_method() - test_case_result.update(Result.PASS) - self._logger.info(f"Test case execution PASSED: {test_case_name}") - - except TestCaseVerifyError as e: - self._logger.exception(f"Test case execution FAILED: {test_case_name}") - test_case_result.update(Result.FAIL, e) - except Exception as e: - self._logger.exception(f"Test case execution ERROR: {test_case_name}") - test_case_result.update(Result.ERROR, e) - except KeyboardInterrupt: - self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}") - test_case_result.update(Result.SKIP) - raise KeyboardInterrupt("Stop DTS") - def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: r"""Find all :class:`TestSuite`\s in a Python module. -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 3/7] dts: filter test suites in executions 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 1/7] dts: convert dts.py methods to class Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš @ 2024-02-06 14:57 ` Juraj Linkeš 2024-02-12 16:44 ` Jeremy Spewock 2024-02-06 14:57 ` [PATCH v2 4/7] dts: reorganize test result Juraj Linkeš ` (3 subsequent siblings) 6 siblings, 1 reply; 44+ messages in thread From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš We're currently filtering which test cases to run after some setup steps, such as DPDK build, have already been taken. This prohibits us to mark the test suites and cases that were supposed to be run as blocked when an earlier setup fails, as that information is not available at that time. To remedy this, move the filtering to the beginning of each execution. This is the first action taken in each execution and if we can't filter the test cases, such as due to invalid inputs, we abort the whole execution. No test suites nor cases will be marked as blocked as we don't know which were supposed to be run. On top of that, the filtering takes place in the TestSuite class, which should only concern itself with test suite and test case logic, not the processing behind the scenes. The logic has been moved to DTSRunner which should do all the processing needed to run test suites. The filtering itself introduces a few changes/assumptions which are more sensible than before: 1. Assumption: There is just one TestSuite child class in each test suite module. This was an implicit assumption before as we couldn't specify the TestSuite classes in the test run configuration, just the modules. The name of the TestSuite child class starts with "Test" and then corresponds to the name of the module with CamelCase naming. 2. Unknown test cases specified both in the test run configuration and the environment variable/command line argument are no longer silently ignored. This is a quality of life improvement for users, as they could easily be not aware of the silent ignoration. Also, a change in the code results in pycodestyle warning and error: [E] E203 whitespace before ':' [W] W503 line break before binary operator These two are not PEP8 compliant, so they're disabled. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/config/__init__.py | 24 +- dts/framework/config/conf_yaml_schema.json | 2 +- dts/framework/runner.py | 426 +++++++++++++++------ dts/framework/settings.py | 3 +- dts/framework/test_result.py | 34 ++ dts/framework/test_suite.py | 85 +--- dts/pyproject.toml | 3 + dts/tests/TestSuite_smoke_tests.py | 2 +- 8 files changed, 382 insertions(+), 197 deletions(-) diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index 62eded7f04..c6a93b3b89 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -36,7 +36,7 @@ import json import os.path import pathlib -from dataclasses import dataclass +from dataclasses import dataclass, fields from enum import auto, unique from typing import Union @@ -506,6 +506,28 @@ def from_dict( vdevs=vdevs, ) + def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration": + """Create a shallow copy with any of the fields modified. + + The only new data are those passed to this method. + The rest are copied from the object's fields calling the method. + + Args: + **kwargs: The names and types of keyword arguments are defined + by the fields of the :class:`ExecutionConfiguration` class. + + Returns: + The copied and modified execution configuration. + """ + new_config = {} + for field in fields(self): + if field.name in kwargs: + new_config[field.name] = kwargs[field.name] + else: + new_config[field.name] = getattr(self, field.name) + + return ExecutionConfiguration(**new_config) + @dataclass(slots=True, frozen=True) class Configuration: diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json index 84e45fe3c2..051b079fe4 100644 --- a/dts/framework/config/conf_yaml_schema.json +++ b/dts/framework/config/conf_yaml_schema.json @@ -197,7 +197,7 @@ }, "cases": { "type": "array", - "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.", + "description": "If specified, only this subset of test suite's test cases will be run.", "items": { "type": "string" }, diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 933685d638..3e95cf9e26 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -17,17 +17,27 @@ and the test case stage runs test cases individually. """ +import importlib +import inspect import logging +import re import sys from types import MethodType +from typing import Iterable from .config import ( BuildTargetConfiguration, + Configuration, ExecutionConfiguration, TestSuiteConfig, load_config, ) -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError +from .exception import ( + BlockingTestSuiteError, + ConfigurationError, + SSHTimeoutError, + TestCaseVerifyError, +) from .logger import DTSLOG, getLogger from .settings import SETTINGS from .test_result import ( @@ -37,8 +47,9 @@ Result, TestCaseResult, TestSuiteResult, + TestSuiteWithCases, ) -from .test_suite import TestSuite, get_test_suites +from .test_suite import TestSuite from .testbed_model import SutNode, TGNode @@ -59,13 +70,23 @@ class DTSRunner: given execution, the next execution begins. """ + _configuration: Configuration _logger: DTSLOG _result: DTSResult + _test_suite_class_prefix: str + _test_suite_module_prefix: str + _func_test_case_regex: str + _perf_test_case_regex: str def __init__(self): - """Initialize the instance with logger and result.""" + """Initialize the instance with configuration, logger, result and string constants.""" + self._configuration = load_config() self._logger = getLogger("DTSRunner") self._result = DTSResult(self._logger) + self._test_suite_class_prefix = "Test" + self._test_suite_module_prefix = "tests.TestSuite_" + self._func_test_case_regex = r"test_(?!perf_)" + self._perf_test_case_regex = r"test_perf_" def run(self): """Run all build targets in all executions from the test run configuration. @@ -106,29 +127,28 @@ def run(self): try: # check the python version of the server that runs dts self._check_dts_python_version() + self._result.update_setup(Result.PASS) # for all Execution sections - for execution in load_config().executions: - sut_node = sut_nodes.get(execution.system_under_test_node.name) - tg_node = tg_nodes.get(execution.traffic_generator_node.name) - + for execution in self._configuration.executions: + self._logger.info( + f"Running execution with SUT '{execution.system_under_test_node.name}'." + ) + execution_result = self._result.add_execution(execution.system_under_test_node) try: - if not sut_node: - sut_node = SutNode(execution.system_under_test_node) - sut_nodes[sut_node.name] = sut_node - if not tg_node: - tg_node = TGNode(execution.traffic_generator_node) - tg_nodes[tg_node.name] = tg_node - self._result.update_setup(Result.PASS) + test_suites_with_cases = self._get_test_suites_with_cases( + execution.test_suites, execution.func, execution.perf + ) except Exception as e: - failed_node = execution.system_under_test_node.name - if sut_node: - failed_node = execution.traffic_generator_node.name - self._logger.exception(f"The Creation of node {failed_node} failed.") - self._result.update_setup(Result.FAIL, e) + self._logger.exception( + f"Invalid test suite configuration found: " f"{execution.test_suites}." + ) + execution_result.update_setup(Result.FAIL, e) else: - self._run_execution(sut_node, tg_node, execution) + self._connect_nodes_and_run_execution( + sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases + ) except Exception as e: self._logger.exception("An unexpected error has occurred.") @@ -163,11 +183,204 @@ def _check_dts_python_version(self) -> None: ) self._logger.warning("Please use Python >= 3.10 instead.") + def _get_test_suites_with_cases( + self, + test_suite_configs: list[TestSuiteConfig], + func: bool, + perf: bool, + ) -> list[TestSuiteWithCases]: + """Test suites with test cases discovery. + + The test suites with test cases defined in the user configuration are discovered + and stored for future use so that we don't import the modules twice and so that + the list of test suites with test cases is available for recording right away. + + Args: + test_suite_configs: Test suite configurations. + func: Whether to include functional test cases in the final list. + perf: Whether to include performance test cases in the final list. + + Returns: + The discovered test suites, each with test cases. + """ + test_suites_with_cases = [] + + for test_suite_config in test_suite_configs: + test_suite_class = self._get_test_suite_class(test_suite_config.test_suite) + test_cases = [] + func_test_cases, perf_test_cases = self._filter_test_cases( + test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases) + ) + if func: + test_cases.extend(func_test_cases) + if perf: + test_cases.extend(perf_test_cases) + + test_suites_with_cases.append( + TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases) + ) + + return test_suites_with_cases + + def _get_test_suite_class(self, test_suite_name: str) -> type[TestSuite]: + """Find the :class:`TestSuite` class with `test_suite_name` in the corresponding module. + + The method assumes that the :class:`TestSuite` class starts + with `self._test_suite_class_prefix`, + continuing with `test_suite_name` with CamelCase convention. + It also assumes there's only one test suite in each module and the module name + is `test_suite_name` prefixed with `self._test_suite_module_prefix`. + + The CamelCase convention is not tested, only lowercase strings are compared. + + Args: + test_suite_name: The name of the test suite to find. + + Returns: + The found test suite. + + Raises: + ConfigurationError: If the corresponding module is not found or + a valid :class:`TestSuite` is not found in the module. + """ + + def is_test_suite(object) -> bool: + """Check whether `object` is a :class:`TestSuite`. + + The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself. + + Args: + object: The object to be checked. + + Returns: + :data:`True` if `object` is a subclass of `TestSuite`. + """ + try: + if issubclass(object, TestSuite) and object is not TestSuite: + return True + except TypeError: + return False + return False + + testsuite_module_path = f"{self._test_suite_module_prefix}{test_suite_name}" + try: + test_suite_module = importlib.import_module(testsuite_module_path) + except ModuleNotFoundError as e: + raise ConfigurationError( + f"Test suite module '{testsuite_module_path}' not found." + ) from e + + lowercase_suite_name = test_suite_name.replace("_", "").lower() + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite): + if ( + class_name.startswith(self._test_suite_class_prefix) + and lowercase_suite_name == class_name[len(self._test_suite_class_prefix) :].lower() + ): + return class_obj + raise ConfigurationError( + f"Couldn't find any valid test suites in {test_suite_module.__name__}." + ) + + def _filter_test_cases( + self, test_suite_class: type[TestSuite], test_cases_to_run: set[str] + ) -> tuple[list[MethodType], list[MethodType]]: + """Filter `test_cases_to_run` from `test_suite_class`. + + There are two rounds of filtering if `test_cases_to_run` is not empty. + The first filters `test_cases_to_run` from all methods of `test_suite_class`. + Then the methods are separated into functional and performance test cases. + If a method doesn't match neither the functional nor performance name prefix, it's an error. + + Args: + test_suite_class: The class of the test suite. + test_cases_to_run: Test case names to filter from `test_suite_class`. + If empty, return all matching test cases. + + Returns: + A list of test case methods that should be executed. + + Raises: + ConfigurationError: If a test case from `test_cases_to_run` is not found + or it doesn't match either the functional nor performance name prefix. + """ + func_test_cases = [] + perf_test_cases = [] + name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction) + if test_cases_to_run: + name_method_tuples = [ + (name, method) for name, method in name_method_tuples if name in test_cases_to_run + ] + if len(name_method_tuples) < len(test_cases_to_run): + missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples} + raise ConfigurationError( + f"Test cases {missing_test_cases} not found among methods " + f"of {test_suite_class.__name__}." + ) + + for test_case_name, test_case_method in name_method_tuples: + if re.match(self._func_test_case_regex, test_case_name): + func_test_cases.append(test_case_method) + elif re.match(self._perf_test_case_regex, test_case_name): + perf_test_cases.append(test_case_method) + elif test_cases_to_run: + raise ConfigurationError( + f"Method '{test_case_name}' doesn't match neither " + f"a functional nor a performance test case name." + ) + + return func_test_cases, perf_test_cases + + def _connect_nodes_and_run_execution( + self, + sut_nodes: dict[str, SutNode], + tg_nodes: dict[str, TGNode], + execution: ExecutionConfiguration, + execution_result: ExecutionResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], + ) -> None: + """Connect nodes, then continue to run the given execution. + + Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`. + If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`, + respectively. + If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`. + + Args: + sut_nodes: A dictionary storing connected/to be connected SUT nodes. + tg_nodes: A dictionary storing connected/to be connected TG nodes. + execution: An execution's test run configuration. + execution_result: The execution's result. + test_suites_with_cases: The test suites with test cases to run. + """ + sut_node = sut_nodes.get(execution.system_under_test_node.name) + tg_node = tg_nodes.get(execution.traffic_generator_node.name) + + try: + if not sut_node: + sut_node = SutNode(execution.system_under_test_node) + sut_nodes[sut_node.name] = sut_node + if not tg_node: + tg_node = TGNode(execution.traffic_generator_node) + tg_nodes[tg_node.name] = tg_node + except Exception as e: + failed_node = execution.system_under_test_node.name + if sut_node: + failed_node = execution.traffic_generator_node.name + self._logger.exception(f"The Creation of node {failed_node} failed.") + execution_result.update_setup(Result.FAIL, e) + + else: + self._run_execution( + sut_node, tg_node, execution, execution_result, test_suites_with_cases + ) + def _run_execution( self, sut_node: SutNode, tg_node: TGNode, execution: ExecutionConfiguration, + execution_result: ExecutionResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], ) -> None: """Run the given execution. @@ -178,11 +391,11 @@ def _run_execution( sut_node: The execution's SUT node. tg_node: The execution's TG node. execution: An execution's test run configuration. + execution_result: The execution's result. + test_suites_with_cases: The test suites with test cases to run. """ self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") - execution_result = self._result.add_execution(sut_node.config) execution_result.add_sut_info(sut_node.node_info) - try: sut_node.set_up_execution(execution) execution_result.update_setup(Result.PASS) @@ -192,7 +405,10 @@ def _run_execution( else: for build_target in execution.build_targets: - self._run_build_target(sut_node, tg_node, build_target, execution, execution_result) + build_target_result = execution_result.add_build_target(build_target) + self._run_build_target( + sut_node, tg_node, build_target, build_target_result, test_suites_with_cases + ) finally: try: @@ -207,8 +423,8 @@ def _run_build_target( sut_node: SutNode, tg_node: TGNode, build_target: BuildTargetConfiguration, - execution: ExecutionConfiguration, - execution_result: ExecutionResult, + build_target_result: BuildTargetResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], ) -> None: """Run the given build target. @@ -220,11 +436,11 @@ def _run_build_target( sut_node: The execution's sut node. tg_node: The execution's tg node. build_target: A build target's test run configuration. - execution: The build target's execution's test run configuration. - execution_result: The execution level result object associated with the execution. + build_target_result: The build target level result object associated + with the current build target. + test_suites_with_cases: The test suites with test cases to run. """ self._logger.info(f"Running build target '{build_target.name}'.") - build_target_result = execution_result.add_build_target(build_target) try: sut_node.set_up_build_target(build_target) @@ -236,7 +452,7 @@ def _run_build_target( build_target_result.update_setup(Result.FAIL, e) else: - self._run_test_suites(sut_node, tg_node, execution, build_target_result) + self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases) finally: try: @@ -250,10 +466,10 @@ def _run_test_suites( self, sut_node: SutNode, tg_node: TGNode, - execution: ExecutionConfiguration, build_target_result: BuildTargetResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], ) -> None: - """Run the execution's (possibly a subset of) test suites using the current build target. + """Run `test_suites_with_cases` with the current build target. The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. @@ -264,22 +480,20 @@ def _run_test_suites( Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. - execution: The execution's test run configuration associated - with the current build target. build_target_result: The build target level result object associated with the current build target. + test_suites_with_cases: The test suites with test cases to run. """ end_build_target = False - if not execution.skip_smoke_tests: - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] - for test_suite_config in execution.test_suites: + for test_suite_with_cases in test_suites_with_cases: + test_suite_result = build_target_result.add_test_suite( + test_suite_with_cases.test_suite_class.__name__ + ) try: - self._run_test_suite_module( - sut_node, tg_node, execution, build_target_result, test_suite_config - ) + self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases) except BlockingTestSuiteError as e: self._logger.exception( - f"An error occurred within {test_suite_config.test_suite}. " + f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. " "Skipping build target..." ) self._result.add_error(e) @@ -288,15 +502,14 @@ def _run_test_suites( if end_build_target: break - def _run_test_suite_module( + def _run_test_suite( self, sut_node: SutNode, tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, - test_suite_config: TestSuiteConfig, + test_suite_result: TestSuiteResult, + test_suite_with_cases: TestSuiteWithCases, ) -> None: - """Set up, execute and tear down all test suites in a single test suite module. + """Set up, execute and tear down `test_suite_with_cases`. The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. @@ -306,92 +519,79 @@ def _run_test_suite_module( Record the setup and the teardown and handle failures. - The test cases to execute are discovered when creating the :class:`TestSuite` object. - Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. - execution: The execution's test run configuration associated - with the current build target. - build_target_result: The build target level result object associated - with the current build target. - test_suite_config: Test suite test run configuration specifying the test suite module - and possibly a subset of test cases of test suites in that module. + test_suite_result: The test suite level result object associated + with the current test suite. + test_suite_with_cases: The test suite with test cases to run. Raises: BlockingTestSuiteError: If a blocking test suite fails. """ + test_suite_name = test_suite_with_cases.test_suite_class.__name__ + test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) try: - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" - test_suite_classes = get_test_suites(full_suite_path) - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) - self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") + self._logger.info(f"Starting test suite setup: {test_suite_name}") + test_suite.set_up_suite() + test_suite_result.update_setup(Result.PASS) + self._logger.info(f"Test suite setup successful: {test_suite_name}") except Exception as e: - self._logger.exception("An error occurred when searching for test suites.") - self._result.update_setup(Result.ERROR, e) + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") + test_suite_result.update_setup(Result.ERROR, e) else: - for test_suite_class in test_suite_classes: - test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases) - - test_suite_name = test_suite.__class__.__name__ - test_suite_result = build_target_result.add_test_suite(test_suite_name) - try: - self._logger.info(f"Starting test suite setup: {test_suite_name}") - test_suite.set_up_suite() - test_suite_result.update_setup(Result.PASS) - self._logger.info(f"Test suite setup successful: {test_suite_name}") - except Exception as e: - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") - test_suite_result.update_setup(Result.ERROR, e) - - else: - self._execute_test_suite(execution.func, test_suite, test_suite_result) - - finally: - try: - test_suite.tear_down_suite() - sut_node.kill_cleanup_dpdk_apps() - test_suite_result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") - self._logger.warning( - f"Test suite '{test_suite_name}' teardown failed, " - f"the next test suite may be affected." - ) - test_suite_result.update_setup(Result.ERROR, e) - if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: - raise BlockingTestSuiteError(test_suite_name) + self._execute_test_suite( + test_suite, + test_suite_with_cases.test_cases, + test_suite_result, + ) + finally: + try: + test_suite.tear_down_suite() + sut_node.kill_cleanup_dpdk_apps() + test_suite_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") + self._logger.warning( + f"Test suite '{test_suite_name}' teardown failed, " + "the next test suite may be affected." + ) + test_suite_result.update_setup(Result.ERROR, e) + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: + raise BlockingTestSuiteError(test_suite_name) def _execute_test_suite( - self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult + self, + test_suite: TestSuite, + test_cases: Iterable[MethodType], + test_suite_result: TestSuiteResult, ) -> None: - """Execute all discovered test cases in `test_suite`. + """Execute all `test_cases` in `test_suite`. If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable is set, in case of a test case failure, the test case will be executed again until it passes or it fails that many times in addition of the first failure. Args: - func: Whether to execute functional test cases. test_suite: The test suite object. + test_cases: The list of test case methods. test_suite_result: The test suite level result object associated with the current test suite. """ - if func: - for test_case_method in test_suite._get_functional_test_cases(): - test_case_name = test_case_method.__name__ - test_case_result = test_suite_result.add_test_case(test_case_name) - all_attempts = SETTINGS.re_run + 1 - attempt_nr = 1 + for test_case_method in test_cases: + test_case_name = test_case_method.__name__ + test_case_result = test_suite_result.add_test_case(test_case_name) + all_attempts = SETTINGS.re_run + 1 + attempt_nr = 1 + self._run_test_case(test_suite, test_case_method, test_case_result) + while not test_case_result and attempt_nr < all_attempts: + attempt_nr += 1 + self._logger.info( + f"Re-running FAILED test case '{test_case_name}'. " + f"Attempt number {attempt_nr} out of {all_attempts}." + ) self._run_test_case(test_suite, test_case_method, test_case_result) - while not test_case_result and attempt_nr < all_attempts: - attempt_nr += 1 - self._logger.info( - f"Re-running FAILED test case '{test_case_name}'. " - f"Attempt number {attempt_nr} out of {all_attempts}." - ) - self._run_test_case(test_suite, test_case_method, test_case_result) def _run_test_case( self, @@ -399,7 +599,7 @@ def _run_test_case( test_case_method: MethodType, test_case_result: TestCaseResult, ) -> None: - """Setup, execute and teardown a test case in `test_suite`. + """Setup, execute and teardown `test_case_method` from `test_suite`. Record the result of the setup and the teardown and handle failures. @@ -424,7 +624,7 @@ def _run_test_case( else: # run test case if setup was successful - self._execute_test_case(test_case_method, test_case_result) + self._execute_test_case(test_suite, test_case_method, test_case_result) finally: try: @@ -440,11 +640,15 @@ def _run_test_case( test_case_result.update(Result.ERROR) def _execute_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult + self, + test_suite: TestSuite, + test_case_method: MethodType, + test_case_result: TestCaseResult, ) -> None: - """Execute one test case, record the result and handle failures. + """Execute `test_case_method` from `test_suite`, record the result and handle failures. Args: + test_suite: The test suite object. test_case_method: The test case method. test_case_result: The test case level result object associated with the current test case. @@ -452,7 +656,7 @@ def _execute_test_case( test_case_name = test_case_method.__name__ try: self._logger.info(f"Starting test case execution: {test_case_name}") - test_case_method() + test_case_method(test_suite) test_case_result.update(Result.PASS) self._logger.info(f"Test case execution PASSED: {test_case_name}") diff --git a/dts/framework/settings.py b/dts/framework/settings.py index 609c8d0e62..2b8bfbe0ed 100644 --- a/dts/framework/settings.py +++ b/dts/framework/settings.py @@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser: "--test-cases", action=_env_arg("DTS_TESTCASES"), default="", - help="[DTS_TESTCASES] Comma-separated list of test cases to execute. " - "Unknown test cases will be silently ignored.", + help="[DTS_TESTCASES] Comma-separated list of test cases to execute.", ) parser.add_argument( diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index 4467749a9d..075195fd5b 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -25,7 +25,9 @@ import os.path from collections.abc import MutableSequence +from dataclasses import dataclass from enum import Enum, auto +from types import MethodType from .config import ( OS, @@ -36,10 +38,42 @@ CPUType, NodeConfiguration, NodeInfo, + TestSuiteConfig, ) from .exception import DTSError, ErrorSeverity from .logger import DTSLOG from .settings import SETTINGS +from .test_suite import TestSuite + + +@dataclass(slots=True, frozen=True) +class TestSuiteWithCases: + """A test suite class with test case methods. + + An auxiliary class holding a test case class with test case methods. The intended use of this + class is to hold a subset of test cases (which could be all test cases) because we don't have + all the data to instantiate the class at the point of inspection. The knowledge of this subset + is needed in case an error occurs before the class is instantiated and we need to record + which test cases were blocked by the error. + + Attributes: + test_suite_class: The test suite class. + test_cases: The test case methods. + """ + + test_suite_class: type[TestSuite] + test_cases: list[MethodType] + + def create_config(self) -> TestSuiteConfig: + """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases. + + Returns: + The :class:`TestSuiteConfig` representation. + """ + return TestSuiteConfig( + test_suite=self.test_suite_class.__name__, + test_cases=[test_case.__name__ for test_case in self.test_cases], + ) class Result(Enum): diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index b02fd36147..f9fe88093e 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -11,25 +11,17 @@ * Testbed (SUT, TG) configuration, * Packet sending and verification, * Test case verification. - -The module also defines a function, :func:`get_test_suites`, -for gathering test suites from a Python module. """ -import importlib -import inspect -import re from ipaddress import IPv4Interface, IPv6Interface, ip_interface -from types import MethodType -from typing import Any, ClassVar, Union +from typing import ClassVar, Union from scapy.layers.inet import IP # type: ignore[import] from scapy.layers.l2 import Ether # type: ignore[import] from scapy.packet import Packet, Padding # type: ignore[import] -from .exception import ConfigurationError, TestCaseVerifyError +from .exception import TestCaseVerifyError from .logger import DTSLOG, getLogger -from .settings import SETTINGS from .testbed_model import Port, PortLink, SutNode, TGNode from .utils import get_packet_summaries @@ -37,7 +29,6 @@ class TestSuite(object): """The base class with building blocks needed by most test cases. - * Test case filtering and collection, * Test suite setup/cleanup methods to override, * Test case setup/cleanup methods to override, * Test case verification, @@ -71,7 +62,6 @@ class TestSuite(object): #: will block the execution of all subsequent test suites in the current build target. is_blocking: ClassVar[bool] = False _logger: DTSLOG - _test_cases_to_run: list[str] _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -86,24 +76,19 @@ def __init__( self, sut_node: SutNode, tg_node: TGNode, - test_cases: list[str], ): """Initialize the test suite testbed information and basic configuration. - Process what test cases to run, find links between ports and set up - default IP addresses to be used when configuring them. + Find links between ports and set up default IP addresses to be used when + configuring them. Args: sut_node: The SUT node where the test suite will run. tg_node: The TG node where the test suite will run. - test_cases: The list of test cases to execute. - If empty, all test cases will be executed. """ self.sut_node = sut_node self.tg_node = tg_node self._logger = getLogger(self.__class__.__name__) - self._test_cases_to_run = test_cases - self._test_cases_to_run.extend(SETTINGS.test_cases) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( @@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst: return False return True - - def _get_functional_test_cases(self) -> list[MethodType]: - """Get all functional test cases defined in this TestSuite. - - Returns: - The list of functional test cases of this TestSuite. - """ - return self._get_test_cases(r"test_(?!perf_)") - - def _get_test_cases(self, test_case_regex: str) -> list[MethodType]: - """Return a list of test cases matching test_case_regex. - - Returns: - The list of test cases matching test_case_regex of this TestSuite. - """ - self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.") - filtered_test_cases = [] - for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod): - if self._should_be_executed(test_case_name, test_case_regex): - filtered_test_cases.append(test_case) - cases_str = ", ".join((x.__name__ for x in filtered_test_cases)) - self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.") - return filtered_test_cases - - def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool: - """Check whether the test case should be scheduled to be executed.""" - match = bool(re.match(test_case_regex, test_case_name)) - if self._test_cases_to_run: - return match and test_case_name in self._test_cases_to_run - - return match - - -def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: - r"""Find all :class:`TestSuite`\s in a Python module. - - Args: - testsuite_module_path: The path to the Python module. - - Returns: - The list of :class:`TestSuite`\s found within the Python module. - - Raises: - ConfigurationError: The test suite module was not found. - """ - - def is_test_suite(object: Any) -> bool: - try: - if issubclass(object, TestSuite) and object is not TestSuite: - return True - except TypeError: - return False - return False - - try: - testcase_module = importlib.import_module(testsuite_module_path) - except ModuleNotFoundError as e: - raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e - return [ - test_suite_class - for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite) - ] diff --git a/dts/pyproject.toml b/dts/pyproject.toml index 28bd970ae4..8eb92b4f11 100644 --- a/dts/pyproject.toml +++ b/dts/pyproject.toml @@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes" format = "pylint" max_line_length = 100 +[tool.pylama.linter.pycodestyle] +ignore = "E203,W503" + [tool.pylama.linter.pydocstyle] convention = "google" diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py index 5e2bac14bd..7b2a0e97f8 100644 --- a/dts/tests/TestSuite_smoke_tests.py +++ b/dts/tests/TestSuite_smoke_tests.py @@ -21,7 +21,7 @@ from framework.utils import REGEX_FOR_PCI_ADDRESS -class SmokeTests(TestSuite): +class TestSmokeTests(TestSuite): """DPDK and infrastructure smoke test suite. The test cases validate the most basic DPDK functionality needed for all other test suites. -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 3/7] dts: filter test suites in executions 2024-02-06 14:57 ` [PATCH v2 3/7] dts: filter test suites in executions Juraj Linkeš @ 2024-02-12 16:44 ` Jeremy Spewock 2024-02-14 9:55 ` Juraj Linkeš 0 siblings, 1 reply; 44+ messages in thread From: Jeremy Spewock @ 2024-02-12 16:44 UTC (permalink / raw) To: Juraj Linkeš Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev On Tue, Feb 6, 2024 at 9:57 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote: > > We're currently filtering which test cases to run after some setup > steps, such as DPDK build, have already been taken. This prohibits us to > mark the test suites and cases that were supposed to be run as blocked > when an earlier setup fails, as that information is not available at > that time. > > To remedy this, move the filtering to the beginning of each execution. > This is the first action taken in each execution and if we can't filter > the test cases, such as due to invalid inputs, we abort the whole > execution. No test suites nor cases will be marked as blocked as we > don't know which were supposed to be run. > > On top of that, the filtering takes place in the TestSuite class, which > should only concern itself with test suite and test case logic, not the > processing behind the scenes. The logic has been moved to DTSRunner > which should do all the processing needed to run test suites. > > The filtering itself introduces a few changes/assumptions which are more > sensible than before: > 1. Assumption: There is just one TestSuite child class in each test > suite module. This was an implicit assumption before as we couldn't > specify the TestSuite classes in the test run configuration, just the > modules. The name of the TestSuite child class starts with "Test" and > then corresponds to the name of the module with CamelCase naming. > 2. Unknown test cases specified both in the test run configuration and > the environment variable/command line argument are no longer silently > ignored. This is a quality of life improvement for users, as they > could easily be not aware of the silent ignoration. > > Also, a change in the code results in pycodestyle warning and error: > [E] E203 whitespace before ':' > [W] W503 line break before binary operator > > These two are not PEP8 compliant, so they're disabled. > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> > --- > dts/framework/config/__init__.py | 24 +- > dts/framework/config/conf_yaml_schema.json | 2 +- > dts/framework/runner.py | 426 +++++++++++++++------ > dts/framework/settings.py | 3 +- > dts/framework/test_result.py | 34 ++ > dts/framework/test_suite.py | 85 +--- > dts/pyproject.toml | 3 + > dts/tests/TestSuite_smoke_tests.py | 2 +- > 8 files changed, 382 insertions(+), 197 deletions(-) > > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py > index 62eded7f04..c6a93b3b89 100644 > --- a/dts/framework/config/__init__.py > +++ b/dts/framework/config/__init__.py > @@ -36,7 +36,7 @@ > import json > import os.path > import pathlib > -from dataclasses import dataclass > +from dataclasses import dataclass, fields > from enum import auto, unique > from typing import Union > > @@ -506,6 +506,28 @@ def from_dict( > vdevs=vdevs, > ) > > + def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration": > + """Create a shallow copy with any of the fields modified. > + > + The only new data are those passed to this method. > + The rest are copied from the object's fields calling the method. > + > + Args: > + **kwargs: The names and types of keyword arguments are defined > + by the fields of the :class:`ExecutionConfiguration` class. > + > + Returns: > + The copied and modified execution configuration. > + """ > + new_config = {} > + for field in fields(self): > + if field.name in kwargs: > + new_config[field.name] = kwargs[field.name] > + else: > + new_config[field.name] = getattr(self, field.name) > + > + return ExecutionConfiguration(**new_config) > + > > @dataclass(slots=True, frozen=True) > class Configuration: > diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json > index 84e45fe3c2..051b079fe4 100644 > --- a/dts/framework/config/conf_yaml_schema.json > +++ b/dts/framework/config/conf_yaml_schema.json > @@ -197,7 +197,7 @@ > }, > "cases": { > "type": "array", > - "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.", > + "description": "If specified, only this subset of test suite's test cases will be run.", > "items": { > "type": "string" > }, > diff --git a/dts/framework/runner.py b/dts/framework/runner.py > index 933685d638..3e95cf9e26 100644 > --- a/dts/framework/runner.py > +++ b/dts/framework/runner.py > @@ -17,17 +17,27 @@ > and the test case stage runs test cases individually. > """ > > +import importlib > +import inspect > import logging > +import re > import sys > from types import MethodType > +from typing import Iterable > > from .config import ( > BuildTargetConfiguration, > + Configuration, > ExecutionConfiguration, > TestSuiteConfig, > load_config, > ) > -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError > +from .exception import ( > + BlockingTestSuiteError, > + ConfigurationError, > + SSHTimeoutError, > + TestCaseVerifyError, > +) > from .logger import DTSLOG, getLogger > from .settings import SETTINGS > from .test_result import ( > @@ -37,8 +47,9 @@ > Result, > TestCaseResult, > TestSuiteResult, > + TestSuiteWithCases, > ) > -from .test_suite import TestSuite, get_test_suites > +from .test_suite import TestSuite > from .testbed_model import SutNode, TGNode > > > @@ -59,13 +70,23 @@ class DTSRunner: > given execution, the next execution begins. > """ > > + _configuration: Configuration > _logger: DTSLOG > _result: DTSResult > + _test_suite_class_prefix: str > + _test_suite_module_prefix: str > + _func_test_case_regex: str > + _perf_test_case_regex: str > > def __init__(self): > - """Initialize the instance with logger and result.""" > + """Initialize the instance with configuration, logger, result and string constants.""" > + self._configuration = load_config() > self._logger = getLogger("DTSRunner") > self._result = DTSResult(self._logger) > + self._test_suite_class_prefix = "Test" > + self._test_suite_module_prefix = "tests.TestSuite_" > + self._func_test_case_regex = r"test_(?!perf_)" > + self._perf_test_case_regex = r"test_perf_" > > def run(self): > """Run all build targets in all executions from the test run configuration. > @@ -106,29 +127,28 @@ def run(self): > try: > # check the python version of the server that runs dts > self._check_dts_python_version() > + self._result.update_setup(Result.PASS) > > # for all Execution sections > - for execution in load_config().executions: > - sut_node = sut_nodes.get(execution.system_under_test_node.name) > - tg_node = tg_nodes.get(execution.traffic_generator_node.name) > - > + for execution in self._configuration.executions: > + self._logger.info( > + f"Running execution with SUT '{execution.system_under_test_node.name}'." > + ) > + execution_result = self._result.add_execution(execution.system_under_test_node) > try: > - if not sut_node: > - sut_node = SutNode(execution.system_under_test_node) > - sut_nodes[sut_node.name] = sut_node > - if not tg_node: > - tg_node = TGNode(execution.traffic_generator_node) > - tg_nodes[tg_node.name] = tg_node > - self._result.update_setup(Result.PASS) > + test_suites_with_cases = self._get_test_suites_with_cases( > + execution.test_suites, execution.func, execution.perf > + ) > except Exception as e: > - failed_node = execution.system_under_test_node.name > - if sut_node: > - failed_node = execution.traffic_generator_node.name > - self._logger.exception(f"The Creation of node {failed_node} failed.") > - self._result.update_setup(Result.FAIL, e) > + self._logger.exception( > + f"Invalid test suite configuration found: " f"{execution.test_suites}." > + ) > + execution_result.update_setup(Result.FAIL, e) > > else: > - self._run_execution(sut_node, tg_node, execution) > + self._connect_nodes_and_run_execution( > + sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases > + ) > > except Exception as e: > self._logger.exception("An unexpected error has occurred.") > @@ -163,11 +183,204 @@ def _check_dts_python_version(self) -> None: > ) > self._logger.warning("Please use Python >= 3.10 instead.") > > + def _get_test_suites_with_cases( > + self, > + test_suite_configs: list[TestSuiteConfig], > + func: bool, > + perf: bool, > + ) -> list[TestSuiteWithCases]: > + """Test suites with test cases discovery. > + > + The test suites with test cases defined in the user configuration are discovered > + and stored for future use so that we don't import the modules twice and so that > + the list of test suites with test cases is available for recording right away. > + > + Args: > + test_suite_configs: Test suite configurations. > + func: Whether to include functional test cases in the final list. > + perf: Whether to include performance test cases in the final list. > + > + Returns: > + The discovered test suites, each with test cases. > + """ > + test_suites_with_cases = [] > + > + for test_suite_config in test_suite_configs: > + test_suite_class = self._get_test_suite_class(test_suite_config.test_suite) > + test_cases = [] > + func_test_cases, perf_test_cases = self._filter_test_cases( > + test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases) > + ) > + if func: > + test_cases.extend(func_test_cases) > + if perf: > + test_cases.extend(perf_test_cases) > + > + test_suites_with_cases.append( > + TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases) > + ) > + > + return test_suites_with_cases > + > + def _get_test_suite_class(self, test_suite_name: str) -> type[TestSuite]: > + """Find the :class:`TestSuite` class with `test_suite_name` in the corresponding module. > + > + The method assumes that the :class:`TestSuite` class starts > + with `self._test_suite_class_prefix`, > + continuing with `test_suite_name` with CamelCase convention. > + It also assumes there's only one test suite in each module and the module name > + is `test_suite_name` prefixed with `self._test_suite_module_prefix`. > + > + The CamelCase convention is not tested, only lowercase strings are compared. > + > + Args: > + test_suite_name: The name of the test suite to find. > + > + Returns: > + The found test suite. > + > + Raises: > + ConfigurationError: If the corresponding module is not found or > + a valid :class:`TestSuite` is not found in the module. > + """ > + > + def is_test_suite(object) -> bool: > + """Check whether `object` is a :class:`TestSuite`. > + > + The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself. > + > + Args: > + object: The object to be checked. > + > + Returns: > + :data:`True` if `object` is a subclass of `TestSuite`. > + """ > + try: > + if issubclass(object, TestSuite) and object is not TestSuite: > + return True > + except TypeError: > + return False > + return False > + > + testsuite_module_path = f"{self._test_suite_module_prefix}{test_suite_name}" > + try: > + test_suite_module = importlib.import_module(testsuite_module_path) > + except ModuleNotFoundError as e: > + raise ConfigurationError( > + f"Test suite module '{testsuite_module_path}' not found." > + ) from e > + > + lowercase_suite_name = test_suite_name.replace("_", "").lower() > + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite): > + if ( > + class_name.startswith(self._test_suite_class_prefix) > + and lowercase_suite_name == class_name[len(self._test_suite_class_prefix) :].lower() > + ): Would it be simpler to instead just make lowercase_suite_name = f"{self._test_suite_class_prefix}{test_suite_name.replace("_", "").lower()}" so that you can just directly compare class_name == lowercase_suite_name? Both ways should have the exact same result of course so it isn't important, I was just curious. > + return class_obj > + raise ConfigurationError( > + f"Couldn't find any valid test suites in {test_suite_module.__name__}." > + ) > + > + def _filter_test_cases( > + self, test_suite_class: type[TestSuite], test_cases_to_run: set[str] > + ) -> tuple[list[MethodType], list[MethodType]]: > + """Filter `test_cases_to_run` from `test_suite_class`. > + > + There are two rounds of filtering if `test_cases_to_run` is not empty. > + The first filters `test_cases_to_run` from all methods of `test_suite_class`. > + Then the methods are separated into functional and performance test cases. > + If a method doesn't match neither the functional nor performance name prefix, it's an error. I think this is a double negative but could be either "if a method doesn't match either ... or ..." or "if a method matches neither ... nor ...". I have a small preference to the second of the two options though because the "neither" makes the negative more clear in my mind. > + > + Args: > + test_suite_class: The class of the test suite. > + test_cases_to_run: Test case names to filter from `test_suite_class`. > + If empty, return all matching test cases. > + > + Returns: > + A list of test case methods that should be executed. > + > + Raises: > + ConfigurationError: If a test case from `test_cases_to_run` is not found > + or it doesn't match either the functional nor performance name prefix. > + """ > + func_test_cases = [] > + perf_test_cases = [] > + name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction) > + if test_cases_to_run: > + name_method_tuples = [ > + (name, method) for name, method in name_method_tuples if name in test_cases_to_run > + ] > + if len(name_method_tuples) < len(test_cases_to_run): > + missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples} > + raise ConfigurationError( > + f"Test cases {missing_test_cases} not found among methods " > + f"of {test_suite_class.__name__}." > + ) > + > + for test_case_name, test_case_method in name_method_tuples: > + if re.match(self._func_test_case_regex, test_case_name): > + func_test_cases.append(test_case_method) > + elif re.match(self._perf_test_case_regex, test_case_name): > + perf_test_cases.append(test_case_method) > + elif test_cases_to_run: > + raise ConfigurationError( > + f"Method '{test_case_name}' doesn't match neither " > + f"a functional nor a performance test case name." Same thing here with the double negative. > + ) > + > + return func_test_cases, perf_test_cases > + > + def _connect_nodes_and_run_execution( > + self, > + sut_nodes: dict[str, SutNode], > + tg_nodes: dict[str, TGNode], > + execution: ExecutionConfiguration, > + execution_result: ExecutionResult, > + test_suites_with_cases: Iterable[TestSuiteWithCases], > + ) -> None: > + """Connect nodes, then continue to run the given execution. > + > + Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`. > + If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`, > + respectively. > + If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`. > + > + Args: > + sut_nodes: A dictionary storing connected/to be connected SUT nodes. > + tg_nodes: A dictionary storing connected/to be connected TG nodes. > + execution: An execution's test run configuration. > + execution_result: The execution's result. > + test_suites_with_cases: The test suites with test cases to run. > + """ > + sut_node = sut_nodes.get(execution.system_under_test_node.name) > + tg_node = tg_nodes.get(execution.traffic_generator_node.name) > + > + try: > + if not sut_node: > + sut_node = SutNode(execution.system_under_test_node) > + sut_nodes[sut_node.name] = sut_node > + if not tg_node: > + tg_node = TGNode(execution.traffic_generator_node) > + tg_nodes[tg_node.name] = tg_node > + except Exception as e: > + failed_node = execution.system_under_test_node.name > + if sut_node: > + failed_node = execution.traffic_generator_node.name > + self._logger.exception(f"The Creation of node {failed_node} failed.") > + execution_result.update_setup(Result.FAIL, e) > + > + else: > + self._run_execution( > + sut_node, tg_node, execution, execution_result, test_suites_with_cases > + ) > + > def _run_execution( > self, > sut_node: SutNode, > tg_node: TGNode, > execution: ExecutionConfiguration, > + execution_result: ExecutionResult, > + test_suites_with_cases: Iterable[TestSuiteWithCases], > ) -> None: > """Run the given execution. > > @@ -178,11 +391,11 @@ def _run_execution( > sut_node: The execution's SUT node. > tg_node: The execution's TG node. > execution: An execution's test run configuration. > + execution_result: The execution's result. > + test_suites_with_cases: The test suites with test cases to run. > """ > self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") > - execution_result = self._result.add_execution(sut_node.config) > execution_result.add_sut_info(sut_node.node_info) > - > try: > sut_node.set_up_execution(execution) > execution_result.update_setup(Result.PASS) > @@ -192,7 +405,10 @@ def _run_execution( > > else: > for build_target in execution.build_targets: > - self._run_build_target(sut_node, tg_node, build_target, execution, execution_result) > + build_target_result = execution_result.add_build_target(build_target) > + self._run_build_target( > + sut_node, tg_node, build_target, build_target_result, test_suites_with_cases > + ) > > finally: > try: > @@ -207,8 +423,8 @@ def _run_build_target( > sut_node: SutNode, > tg_node: TGNode, > build_target: BuildTargetConfiguration, > - execution: ExecutionConfiguration, > - execution_result: ExecutionResult, > + build_target_result: BuildTargetResult, > + test_suites_with_cases: Iterable[TestSuiteWithCases], > ) -> None: > """Run the given build target. > > @@ -220,11 +436,11 @@ def _run_build_target( > sut_node: The execution's sut node. > tg_node: The execution's tg node. > build_target: A build target's test run configuration. > - execution: The build target's execution's test run configuration. > - execution_result: The execution level result object associated with the execution. > + build_target_result: The build target level result object associated > + with the current build target. > + test_suites_with_cases: The test suites with test cases to run. > """ > self._logger.info(f"Running build target '{build_target.name}'.") > - build_target_result = execution_result.add_build_target(build_target) > > try: > sut_node.set_up_build_target(build_target) > @@ -236,7 +452,7 @@ def _run_build_target( > build_target_result.update_setup(Result.FAIL, e) > > else: > - self._run_test_suites(sut_node, tg_node, execution, build_target_result) > + self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases) > > finally: > try: > @@ -250,10 +466,10 @@ def _run_test_suites( > self, > sut_node: SutNode, > tg_node: TGNode, > - execution: ExecutionConfiguration, > build_target_result: BuildTargetResult, > + test_suites_with_cases: Iterable[TestSuiteWithCases], > ) -> None: > - """Run the execution's (possibly a subset of) test suites using the current build target. > + """Run `test_suites_with_cases` with the current build target. > > The method assumes the build target we're testing has already been built on the SUT node. > The current build target thus corresponds to the current DPDK build present on the SUT node. > @@ -264,22 +480,20 @@ def _run_test_suites( > Args: > sut_node: The execution's SUT node. > tg_node: The execution's TG node. > - execution: The execution's test run configuration associated > - with the current build target. > build_target_result: The build target level result object associated > with the current build target. > + test_suites_with_cases: The test suites with test cases to run. > """ > end_build_target = False > - if not execution.skip_smoke_tests: > - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] > - for test_suite_config in execution.test_suites: > + for test_suite_with_cases in test_suites_with_cases: > + test_suite_result = build_target_result.add_test_suite( > + test_suite_with_cases.test_suite_class.__name__ > + ) > try: > - self._run_test_suite_module( > - sut_node, tg_node, execution, build_target_result, test_suite_config > - ) > + self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases) > except BlockingTestSuiteError as e: > self._logger.exception( > - f"An error occurred within {test_suite_config.test_suite}. " > + f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. " > "Skipping build target..." > ) > self._result.add_error(e) > @@ -288,15 +502,14 @@ def _run_test_suites( > if end_build_target: > break > > - def _run_test_suite_module( > + def _run_test_suite( > self, > sut_node: SutNode, > tg_node: TGNode, > - execution: ExecutionConfiguration, > - build_target_result: BuildTargetResult, > - test_suite_config: TestSuiteConfig, > + test_suite_result: TestSuiteResult, > + test_suite_with_cases: TestSuiteWithCases, > ) -> None: > - """Set up, execute and tear down all test suites in a single test suite module. > + """Set up, execute and tear down `test_suite_with_cases`. > > The method assumes the build target we're testing has already been built on the SUT node. > The current build target thus corresponds to the current DPDK build present on the SUT node. > @@ -306,92 +519,79 @@ def _run_test_suite_module( > > Record the setup and the teardown and handle failures. > > - The test cases to execute are discovered when creating the :class:`TestSuite` object. > - > Args: > sut_node: The execution's SUT node. > tg_node: The execution's TG node. > - execution: The execution's test run configuration associated > - with the current build target. > - build_target_result: The build target level result object associated > - with the current build target. > - test_suite_config: Test suite test run configuration specifying the test suite module > - and possibly a subset of test cases of test suites in that module. > + test_suite_result: The test suite level result object associated > + with the current test suite. > + test_suite_with_cases: The test suite with test cases to run. > > Raises: > BlockingTestSuiteError: If a blocking test suite fails. > """ > + test_suite_name = test_suite_with_cases.test_suite_class.__name__ > + test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) > try: > - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" > - test_suite_classes = get_test_suites(full_suite_path) > - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) > - self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") > + self._logger.info(f"Starting test suite setup: {test_suite_name}") > + test_suite.set_up_suite() > + test_suite_result.update_setup(Result.PASS) > + self._logger.info(f"Test suite setup successful: {test_suite_name}") > except Exception as e: > - self._logger.exception("An error occurred when searching for test suites.") > - self._result.update_setup(Result.ERROR, e) > + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") > + test_suite_result.update_setup(Result.ERROR, e) > > else: > - for test_suite_class in test_suite_classes: > - test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases) > - > - test_suite_name = test_suite.__class__.__name__ > - test_suite_result = build_target_result.add_test_suite(test_suite_name) > - try: > - self._logger.info(f"Starting test suite setup: {test_suite_name}") > - test_suite.set_up_suite() > - test_suite_result.update_setup(Result.PASS) > - self._logger.info(f"Test suite setup successful: {test_suite_name}") > - except Exception as e: > - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") > - test_suite_result.update_setup(Result.ERROR, e) > - > - else: > - self._execute_test_suite(execution.func, test_suite, test_suite_result) > - > - finally: > - try: > - test_suite.tear_down_suite() > - sut_node.kill_cleanup_dpdk_apps() > - test_suite_result.update_teardown(Result.PASS) > - except Exception as e: > - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") > - self._logger.warning( > - f"Test suite '{test_suite_name}' teardown failed, " > - f"the next test suite may be affected." > - ) > - test_suite_result.update_setup(Result.ERROR, e) > - if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: > - raise BlockingTestSuiteError(test_suite_name) > + self._execute_test_suite( > + test_suite, > + test_suite_with_cases.test_cases, > + test_suite_result, > + ) > + finally: > + try: > + test_suite.tear_down_suite() > + sut_node.kill_cleanup_dpdk_apps() > + test_suite_result.update_teardown(Result.PASS) > + except Exception as e: > + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") > + self._logger.warning( > + f"Test suite '{test_suite_name}' teardown failed, " > + "the next test suite may be affected." > + ) > + test_suite_result.update_setup(Result.ERROR, e) > + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: > + raise BlockingTestSuiteError(test_suite_name) > > def _execute_test_suite( > - self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult > + self, > + test_suite: TestSuite, > + test_cases: Iterable[MethodType], > + test_suite_result: TestSuiteResult, > ) -> None: > - """Execute all discovered test cases in `test_suite`. > + """Execute all `test_cases` in `test_suite`. > > If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment > variable is set, in case of a test case failure, the test case will be executed again > until it passes or it fails that many times in addition of the first failure. > > Args: > - func: Whether to execute functional test cases. > test_suite: The test suite object. > + test_cases: The list of test case methods. > test_suite_result: The test suite level result object associated > with the current test suite. > """ > - if func: > - for test_case_method in test_suite._get_functional_test_cases(): > - test_case_name = test_case_method.__name__ > - test_case_result = test_suite_result.add_test_case(test_case_name) > - all_attempts = SETTINGS.re_run + 1 > - attempt_nr = 1 > + for test_case_method in test_cases: > + test_case_name = test_case_method.__name__ > + test_case_result = test_suite_result.add_test_case(test_case_name) > + all_attempts = SETTINGS.re_run + 1 > + attempt_nr = 1 > + self._run_test_case(test_suite, test_case_method, test_case_result) > + while not test_case_result and attempt_nr < all_attempts: > + attempt_nr += 1 > + self._logger.info( > + f"Re-running FAILED test case '{test_case_name}'. " > + f"Attempt number {attempt_nr} out of {all_attempts}." > + ) > self._run_test_case(test_suite, test_case_method, test_case_result) > - while not test_case_result and attempt_nr < all_attempts: > - attempt_nr += 1 > - self._logger.info( > - f"Re-running FAILED test case '{test_case_name}'. " > - f"Attempt number {attempt_nr} out of {all_attempts}." > - ) > - self._run_test_case(test_suite, test_case_method, test_case_result) > > def _run_test_case( > self, > @@ -399,7 +599,7 @@ def _run_test_case( > test_case_method: MethodType, > test_case_result: TestCaseResult, > ) -> None: > - """Setup, execute and teardown a test case in `test_suite`. > + """Setup, execute and teardown `test_case_method` from `test_suite`. > > Record the result of the setup and the teardown and handle failures. > > @@ -424,7 +624,7 @@ def _run_test_case( > > else: > # run test case if setup was successful > - self._execute_test_case(test_case_method, test_case_result) > + self._execute_test_case(test_suite, test_case_method, test_case_result) > > finally: > try: > @@ -440,11 +640,15 @@ def _run_test_case( > test_case_result.update(Result.ERROR) > > def _execute_test_case( > - self, test_case_method: MethodType, test_case_result: TestCaseResult > + self, > + test_suite: TestSuite, > + test_case_method: MethodType, > + test_case_result: TestCaseResult, > ) -> None: > - """Execute one test case, record the result and handle failures. > + """Execute `test_case_method` from `test_suite`, record the result and handle failures. > > Args: > + test_suite: The test suite object. > test_case_method: The test case method. > test_case_result: The test case level result object associated > with the current test case. > @@ -452,7 +656,7 @@ def _execute_test_case( > test_case_name = test_case_method.__name__ > try: > self._logger.info(f"Starting test case execution: {test_case_name}") > - test_case_method() > + test_case_method(test_suite) > test_case_result.update(Result.PASS) > self._logger.info(f"Test case execution PASSED: {test_case_name}") > > diff --git a/dts/framework/settings.py b/dts/framework/settings.py > index 609c8d0e62..2b8bfbe0ed 100644 > --- a/dts/framework/settings.py > +++ b/dts/framework/settings.py > @@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser: > "--test-cases", > action=_env_arg("DTS_TESTCASES"), > default="", > - help="[DTS_TESTCASES] Comma-separated list of test cases to execute. " > - "Unknown test cases will be silently ignored.", > + help="[DTS_TESTCASES] Comma-separated list of test cases to execute.", > ) > > parser.add_argument( > diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py > index 4467749a9d..075195fd5b 100644 > --- a/dts/framework/test_result.py > +++ b/dts/framework/test_result.py > @@ -25,7 +25,9 @@ > > import os.path > from collections.abc import MutableSequence > +from dataclasses import dataclass > from enum import Enum, auto > +from types import MethodType > > from .config import ( > OS, > @@ -36,10 +38,42 @@ > CPUType, > NodeConfiguration, > NodeInfo, > + TestSuiteConfig, > ) > from .exception import DTSError, ErrorSeverity > from .logger import DTSLOG > from .settings import SETTINGS > +from .test_suite import TestSuite > + > + > +@dataclass(slots=True, frozen=True) > +class TestSuiteWithCases: > + """A test suite class with test case methods. > + > + An auxiliary class holding a test case class with test case methods. The intended use of this > + class is to hold a subset of test cases (which could be all test cases) because we don't have > + all the data to instantiate the class at the point of inspection. The knowledge of this subset > + is needed in case an error occurs before the class is instantiated and we need to record > + which test cases were blocked by the error. > + > + Attributes: > + test_suite_class: The test suite class. > + test_cases: The test case methods. > + """ > + > + test_suite_class: type[TestSuite] > + test_cases: list[MethodType] > + > + def create_config(self) -> TestSuiteConfig: > + """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases. > + > + Returns: > + The :class:`TestSuiteConfig` representation. > + """ > + return TestSuiteConfig( > + test_suite=self.test_suite_class.__name__, > + test_cases=[test_case.__name__ for test_case in self.test_cases], > + ) > > > class Result(Enum): > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py > index b02fd36147..f9fe88093e 100644 > --- a/dts/framework/test_suite.py > +++ b/dts/framework/test_suite.py > @@ -11,25 +11,17 @@ > * Testbed (SUT, TG) configuration, > * Packet sending and verification, > * Test case verification. > - > -The module also defines a function, :func:`get_test_suites`, > -for gathering test suites from a Python module. > """ > > -import importlib > -import inspect > -import re > from ipaddress import IPv4Interface, IPv6Interface, ip_interface > -from types import MethodType > -from typing import Any, ClassVar, Union > +from typing import ClassVar, Union > > from scapy.layers.inet import IP # type: ignore[import] > from scapy.layers.l2 import Ether # type: ignore[import] > from scapy.packet import Packet, Padding # type: ignore[import] > > -from .exception import ConfigurationError, TestCaseVerifyError > +from .exception import TestCaseVerifyError > from .logger import DTSLOG, getLogger > -from .settings import SETTINGS > from .testbed_model import Port, PortLink, SutNode, TGNode > from .utils import get_packet_summaries > > @@ -37,7 +29,6 @@ > class TestSuite(object): > """The base class with building blocks needed by most test cases. > > - * Test case filtering and collection, > * Test suite setup/cleanup methods to override, > * Test case setup/cleanup methods to override, > * Test case verification, > @@ -71,7 +62,6 @@ class TestSuite(object): > #: will block the execution of all subsequent test suites in the current build target. > is_blocking: ClassVar[bool] = False > _logger: DTSLOG > - _test_cases_to_run: list[str] > _port_links: list[PortLink] > _sut_port_ingress: Port > _sut_port_egress: Port > @@ -86,24 +76,19 @@ def __init__( > self, > sut_node: SutNode, > tg_node: TGNode, > - test_cases: list[str], > ): > """Initialize the test suite testbed information and basic configuration. > > - Process what test cases to run, find links between ports and set up > - default IP addresses to be used when configuring them. > + Find links between ports and set up default IP addresses to be used when > + configuring them. > > Args: > sut_node: The SUT node where the test suite will run. > tg_node: The TG node where the test suite will run. > - test_cases: The list of test cases to execute. > - If empty, all test cases will be executed. > """ > self.sut_node = sut_node > self.tg_node = tg_node > self._logger = getLogger(self.__class__.__name__) > - self._test_cases_to_run = test_cases > - self._test_cases_to_run.extend(SETTINGS.test_cases) > self._port_links = [] > self._process_links() > self._sut_port_ingress, self._tg_port_egress = ( > @@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: > if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst: > return False > return True > - > - def _get_functional_test_cases(self) -> list[MethodType]: > - """Get all functional test cases defined in this TestSuite. > - > - Returns: > - The list of functional test cases of this TestSuite. > - """ > - return self._get_test_cases(r"test_(?!perf_)") > - > - def _get_test_cases(self, test_case_regex: str) -> list[MethodType]: > - """Return a list of test cases matching test_case_regex. > - > - Returns: > - The list of test cases matching test_case_regex of this TestSuite. > - """ > - self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.") > - filtered_test_cases = [] > - for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod): > - if self._should_be_executed(test_case_name, test_case_regex): > - filtered_test_cases.append(test_case) > - cases_str = ", ".join((x.__name__ for x in filtered_test_cases)) > - self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.") > - return filtered_test_cases > - > - def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool: > - """Check whether the test case should be scheduled to be executed.""" > - match = bool(re.match(test_case_regex, test_case_name)) > - if self._test_cases_to_run: > - return match and test_case_name in self._test_cases_to_run > - > - return match > - > - > -def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: > - r"""Find all :class:`TestSuite`\s in a Python module. > - > - Args: > - testsuite_module_path: The path to the Python module. > - > - Returns: > - The list of :class:`TestSuite`\s found within the Python module. > - > - Raises: > - ConfigurationError: The test suite module was not found. > - """ > - > - def is_test_suite(object: Any) -> bool: > - try: > - if issubclass(object, TestSuite) and object is not TestSuite: > - return True > - except TypeError: > - return False > - return False > - > - try: > - testcase_module = importlib.import_module(testsuite_module_path) > - except ModuleNotFoundError as e: > - raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e > - return [ > - test_suite_class > - for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite) > - ] > diff --git a/dts/pyproject.toml b/dts/pyproject.toml > index 28bd970ae4..8eb92b4f11 100644 > --- a/dts/pyproject.toml > +++ b/dts/pyproject.toml > @@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes" > format = "pylint" > max_line_length = 100 > > +[tool.pylama.linter.pycodestyle] > +ignore = "E203,W503" > + > [tool.pylama.linter.pydocstyle] > convention = "google" > > diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py > index 5e2bac14bd..7b2a0e97f8 100644 > --- a/dts/tests/TestSuite_smoke_tests.py > +++ b/dts/tests/TestSuite_smoke_tests.py > @@ -21,7 +21,7 @@ > from framework.utils import REGEX_FOR_PCI_ADDRESS > > > -class SmokeTests(TestSuite): > +class TestSmokeTests(TestSuite): > """DPDK and infrastructure smoke test suite. > > The test cases validate the most basic DPDK functionality needed for all other test suites. > -- > 2.34.1 > ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 3/7] dts: filter test suites in executions 2024-02-12 16:44 ` Jeremy Spewock @ 2024-02-14 9:55 ` Juraj Linkeš 0 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-14 9:55 UTC (permalink / raw) To: Jeremy Spewock Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev On Mon, Feb 12, 2024 at 5:44 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote: > > On Tue, Feb 6, 2024 at 9:57 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote: > > > > We're currently filtering which test cases to run after some setup > > steps, such as DPDK build, have already been taken. This prohibits us to > > mark the test suites and cases that were supposed to be run as blocked > > when an earlier setup fails, as that information is not available at > > that time. > > > > To remedy this, move the filtering to the beginning of each execution. > > This is the first action taken in each execution and if we can't filter > > the test cases, such as due to invalid inputs, we abort the whole > > execution. No test suites nor cases will be marked as blocked as we > > don't know which were supposed to be run. > > > > On top of that, the filtering takes place in the TestSuite class, which > > should only concern itself with test suite and test case logic, not the > > processing behind the scenes. The logic has been moved to DTSRunner > > which should do all the processing needed to run test suites. > > > > The filtering itself introduces a few changes/assumptions which are more > > sensible than before: > > 1. Assumption: There is just one TestSuite child class in each test > > suite module. This was an implicit assumption before as we couldn't > > specify the TestSuite classes in the test run configuration, just the > > modules. The name of the TestSuite child class starts with "Test" and > > then corresponds to the name of the module with CamelCase naming. > > 2. Unknown test cases specified both in the test run configuration and > > the environment variable/command line argument are no longer silently > > ignored. This is a quality of life improvement for users, as they > > could easily be not aware of the silent ignoration. > > > > Also, a change in the code results in pycodestyle warning and error: > > [E] E203 whitespace before ':' > > [W] W503 line break before binary operator > > > > These two are not PEP8 compliant, so they're disabled. > > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> > > --- > > dts/framework/config/__init__.py | 24 +- > > dts/framework/config/conf_yaml_schema.json | 2 +- > > dts/framework/runner.py | 426 +++++++++++++++------ > > dts/framework/settings.py | 3 +- > > dts/framework/test_result.py | 34 ++ > > dts/framework/test_suite.py | 85 +--- > > dts/pyproject.toml | 3 + > > dts/tests/TestSuite_smoke_tests.py | 2 +- > > 8 files changed, 382 insertions(+), 197 deletions(-) > > > > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py > > index 62eded7f04..c6a93b3b89 100644 > > --- a/dts/framework/config/__init__.py > > +++ b/dts/framework/config/__init__.py > > @@ -36,7 +36,7 @@ > > import json > > import os.path > > import pathlib > > -from dataclasses import dataclass > > +from dataclasses import dataclass, fields > > from enum import auto, unique > > from typing import Union > > > > @@ -506,6 +506,28 @@ def from_dict( > > vdevs=vdevs, > > ) > > > > + def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration": > > + """Create a shallow copy with any of the fields modified. > > + > > + The only new data are those passed to this method. > > + The rest are copied from the object's fields calling the method. > > + > > + Args: > > + **kwargs: The names and types of keyword arguments are defined > > + by the fields of the :class:`ExecutionConfiguration` class. > > + > > + Returns: > > + The copied and modified execution configuration. > > + """ > > + new_config = {} > > + for field in fields(self): > > + if field.name in kwargs: > > + new_config[field.name] = kwargs[field.name] > > + else: > > + new_config[field.name] = getattr(self, field.name) > > + > > + return ExecutionConfiguration(**new_config) > > + > > > > @dataclass(slots=True, frozen=True) > > class Configuration: > > diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json > > index 84e45fe3c2..051b079fe4 100644 > > --- a/dts/framework/config/conf_yaml_schema.json > > +++ b/dts/framework/config/conf_yaml_schema.json > > @@ -197,7 +197,7 @@ > > }, > > "cases": { > > "type": "array", > > - "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.", > > + "description": "If specified, only this subset of test suite's test cases will be run.", > > "items": { > > "type": "string" > > }, > > diff --git a/dts/framework/runner.py b/dts/framework/runner.py > > index 933685d638..3e95cf9e26 100644 > > --- a/dts/framework/runner.py > > +++ b/dts/framework/runner.py > > @@ -17,17 +17,27 @@ > > and the test case stage runs test cases individually. > > """ > > > > +import importlib > > +import inspect > > import logging > > +import re > > import sys > > from types import MethodType > > +from typing import Iterable > > > > from .config import ( > > BuildTargetConfiguration, > > + Configuration, > > ExecutionConfiguration, > > TestSuiteConfig, > > load_config, > > ) > > -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError > > +from .exception import ( > > + BlockingTestSuiteError, > > + ConfigurationError, > > + SSHTimeoutError, > > + TestCaseVerifyError, > > +) > > from .logger import DTSLOG, getLogger > > from .settings import SETTINGS > > from .test_result import ( > > @@ -37,8 +47,9 @@ > > Result, > > TestCaseResult, > > TestSuiteResult, > > + TestSuiteWithCases, > > ) > > -from .test_suite import TestSuite, get_test_suites > > +from .test_suite import TestSuite > > from .testbed_model import SutNode, TGNode > > > > > > @@ -59,13 +70,23 @@ class DTSRunner: > > given execution, the next execution begins. > > """ > > > > + _configuration: Configuration > > _logger: DTSLOG > > _result: DTSResult > > + _test_suite_class_prefix: str > > + _test_suite_module_prefix: str > > + _func_test_case_regex: str > > + _perf_test_case_regex: str > > > > def __init__(self): > > - """Initialize the instance with logger and result.""" > > + """Initialize the instance with configuration, logger, result and string constants.""" > > + self._configuration = load_config() > > self._logger = getLogger("DTSRunner") > > self._result = DTSResult(self._logger) > > + self._test_suite_class_prefix = "Test" > > + self._test_suite_module_prefix = "tests.TestSuite_" > > + self._func_test_case_regex = r"test_(?!perf_)" > > + self._perf_test_case_regex = r"test_perf_" > > > > def run(self): > > """Run all build targets in all executions from the test run configuration. > > @@ -106,29 +127,28 @@ def run(self): > > try: > > # check the python version of the server that runs dts > > self._check_dts_python_version() > > + self._result.update_setup(Result.PASS) > > > > # for all Execution sections > > - for execution in load_config().executions: > > - sut_node = sut_nodes.get(execution.system_under_test_node.name) > > - tg_node = tg_nodes.get(execution.traffic_generator_node.name) > > - > > + for execution in self._configuration.executions: > > + self._logger.info( > > + f"Running execution with SUT '{execution.system_under_test_node.name}'." > > + ) > > + execution_result = self._result.add_execution(execution.system_under_test_node) > > try: > > - if not sut_node: > > - sut_node = SutNode(execution.system_under_test_node) > > - sut_nodes[sut_node.name] = sut_node > > - if not tg_node: > > - tg_node = TGNode(execution.traffic_generator_node) > > - tg_nodes[tg_node.name] = tg_node > > - self._result.update_setup(Result.PASS) > > + test_suites_with_cases = self._get_test_suites_with_cases( > > + execution.test_suites, execution.func, execution.perf > > + ) > > except Exception as e: > > - failed_node = execution.system_under_test_node.name > > - if sut_node: > > - failed_node = execution.traffic_generator_node.name > > - self._logger.exception(f"The Creation of node {failed_node} failed.") > > - self._result.update_setup(Result.FAIL, e) > > + self._logger.exception( > > + f"Invalid test suite configuration found: " f"{execution.test_suites}." > > + ) > > + execution_result.update_setup(Result.FAIL, e) > > > > else: > > - self._run_execution(sut_node, tg_node, execution) > > + self._connect_nodes_and_run_execution( > > + sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases > > + ) > > > > except Exception as e: > > self._logger.exception("An unexpected error has occurred.") > > @@ -163,11 +183,204 @@ def _check_dts_python_version(self) -> None: > > ) > > self._logger.warning("Please use Python >= 3.10 instead.") > > > > + def _get_test_suites_with_cases( > > + self, > > + test_suite_configs: list[TestSuiteConfig], > > + func: bool, > > + perf: bool, > > + ) -> list[TestSuiteWithCases]: > > + """Test suites with test cases discovery. > > + > > + The test suites with test cases defined in the user configuration are discovered > > + and stored for future use so that we don't import the modules twice and so that > > + the list of test suites with test cases is available for recording right away. > > + > > + Args: > > + test_suite_configs: Test suite configurations. > > + func: Whether to include functional test cases in the final list. > > + perf: Whether to include performance test cases in the final list. > > + > > + Returns: > > + The discovered test suites, each with test cases. > > + """ > > + test_suites_with_cases = [] > > + > > + for test_suite_config in test_suite_configs: > > + test_suite_class = self._get_test_suite_class(test_suite_config.test_suite) > > + test_cases = [] > > + func_test_cases, perf_test_cases = self._filter_test_cases( > > + test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases) > > + ) > > + if func: > > + test_cases.extend(func_test_cases) > > + if perf: > > + test_cases.extend(perf_test_cases) > > + > > + test_suites_with_cases.append( > > + TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases) > > + ) > > + > > + return test_suites_with_cases > > + > > + def _get_test_suite_class(self, test_suite_name: str) -> type[TestSuite]: > > + """Find the :class:`TestSuite` class with `test_suite_name` in the corresponding module. > > + > > + The method assumes that the :class:`TestSuite` class starts > > + with `self._test_suite_class_prefix`, > > + continuing with `test_suite_name` with CamelCase convention. > > + It also assumes there's only one test suite in each module and the module name > > + is `test_suite_name` prefixed with `self._test_suite_module_prefix`. > > + > > + The CamelCase convention is not tested, only lowercase strings are compared. > > + > > + Args: > > + test_suite_name: The name of the test suite to find. > > + > > + Returns: > > + The found test suite. > > + > > + Raises: > > + ConfigurationError: If the corresponding module is not found or > > + a valid :class:`TestSuite` is not found in the module. > > + """ > > + > > + def is_test_suite(object) -> bool: > > + """Check whether `object` is a :class:`TestSuite`. > > + > > + The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself. > > + > > + Args: > > + object: The object to be checked. > > + > > + Returns: > > + :data:`True` if `object` is a subclass of `TestSuite`. > > + """ > > + try: > > + if issubclass(object, TestSuite) and object is not TestSuite: > > + return True > > + except TypeError: > > + return False > > + return False > > + > > + testsuite_module_path = f"{self._test_suite_module_prefix}{test_suite_name}" > > + try: > > + test_suite_module = importlib.import_module(testsuite_module_path) > > + except ModuleNotFoundError as e: > > + raise ConfigurationError( > > + f"Test suite module '{testsuite_module_path}' not found." > > + ) from e > > + > > + lowercase_suite_name = test_suite_name.replace("_", "").lower() > > + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite): > > + if ( > > + class_name.startswith(self._test_suite_class_prefix) > > + and lowercase_suite_name == class_name[len(self._test_suite_class_prefix) :].lower() > > + ): > > Would it be simpler to instead just make lowercase_suite_name = > f"{self._test_suite_class_prefix}{test_suite_name.replace("_", > "").lower()}" so that you can just directly compare class_name == > lowercase_suite_name? Both ways should have the exact same result of > course so it isn't important, I was just curious. > I've looked at how the code looks and it is better. I also changed some of the variable names (test_suite_name -> module_name and lowercase_suite_name -> lowercase_suite_to_find), updated the docstring and now I'm much happier with the result. > > + return class_obj > > + raise ConfigurationError( > > + f"Couldn't find any valid test suites in {test_suite_module.__name__}." > > + ) > > + > > + def _filter_test_cases( > > + self, test_suite_class: type[TestSuite], test_cases_to_run: set[str] > > + ) -> tuple[list[MethodType], list[MethodType]]: > > + """Filter `test_cases_to_run` from `test_suite_class`. > > + > > + There are two rounds of filtering if `test_cases_to_run` is not empty. > > + The first filters `test_cases_to_run` from all methods of `test_suite_class`. > > + Then the methods are separated into functional and performance test cases. > > + If a method doesn't match neither the functional nor performance name prefix, it's an error. > > I think this is a double negative but could be either "if a method > doesn't match either ... or ..." or "if a method matches neither ... > nor ...". I have a small preference to the second of the two options > though because the "neither" makes the negative more clear in my mind. > I'll change this, thanks for the grammar fix. > > + > > + Args: > > + test_suite_class: The class of the test suite. > > + test_cases_to_run: Test case names to filter from `test_suite_class`. > > + If empty, return all matching test cases. > > + > > + Returns: > > + A list of test case methods that should be executed. > > + > > + Raises: > > + ConfigurationError: If a test case from `test_cases_to_run` is not found > > + or it doesn't match either the functional nor performance name prefix. > > + """ > > + func_test_cases = [] > > + perf_test_cases = [] > > + name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction) > > + if test_cases_to_run: > > + name_method_tuples = [ > > + (name, method) for name, method in name_method_tuples if name in test_cases_to_run > > + ] > > + if len(name_method_tuples) < len(test_cases_to_run): > > + missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples} > > + raise ConfigurationError( > > + f"Test cases {missing_test_cases} not found among methods " > > + f"of {test_suite_class.__name__}." > > + ) > > + > > + for test_case_name, test_case_method in name_method_tuples: > > + if re.match(self._func_test_case_regex, test_case_name): > > + func_test_cases.append(test_case_method) > > + elif re.match(self._perf_test_case_regex, test_case_name): > > + perf_test_cases.append(test_case_method) > > + elif test_cases_to_run: > > + raise ConfigurationError( > > + f"Method '{test_case_name}' doesn't match neither " > > + f"a functional nor a performance test case name." > > Same thing here with the double negative. > > > > > + ) > > + > > + return func_test_cases, perf_test_cases > > + > > + def _connect_nodes_and_run_execution( > > + self, > > + sut_nodes: dict[str, SutNode], > > + tg_nodes: dict[str, TGNode], > > + execution: ExecutionConfiguration, > > + execution_result: ExecutionResult, > > + test_suites_with_cases: Iterable[TestSuiteWithCases], > > + ) -> None: > > + """Connect nodes, then continue to run the given execution. > > + > > + Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`. > > + If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`, > > + respectively. > > + If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`. > > + > > + Args: > > + sut_nodes: A dictionary storing connected/to be connected SUT nodes. > > + tg_nodes: A dictionary storing connected/to be connected TG nodes. > > + execution: An execution's test run configuration. > > + execution_result: The execution's result. > > + test_suites_with_cases: The test suites with test cases to run. > > + """ > > + sut_node = sut_nodes.get(execution.system_under_test_node.name) > > + tg_node = tg_nodes.get(execution.traffic_generator_node.name) > > + > > + try: > > + if not sut_node: > > + sut_node = SutNode(execution.system_under_test_node) > > + sut_nodes[sut_node.name] = sut_node > > + if not tg_node: > > + tg_node = TGNode(execution.traffic_generator_node) > > + tg_nodes[tg_node.name] = tg_node > > + except Exception as e: > > + failed_node = execution.system_under_test_node.name > > + if sut_node: > > + failed_node = execution.traffic_generator_node.name > > + self._logger.exception(f"The Creation of node {failed_node} failed.") > > + execution_result.update_setup(Result.FAIL, e) > > + > > + else: > > + self._run_execution( > > + sut_node, tg_node, execution, execution_result, test_suites_with_cases > > + ) > > + > > def _run_execution( > > self, > > sut_node: SutNode, > > tg_node: TGNode, > > execution: ExecutionConfiguration, > > + execution_result: ExecutionResult, > > + test_suites_with_cases: Iterable[TestSuiteWithCases], > > ) -> None: > > """Run the given execution. > > > > @@ -178,11 +391,11 @@ def _run_execution( > > sut_node: The execution's SUT node. > > tg_node: The execution's TG node. > > execution: An execution's test run configuration. > > + execution_result: The execution's result. > > + test_suites_with_cases: The test suites with test cases to run. > > """ > > self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") > > - execution_result = self._result.add_execution(sut_node.config) > > execution_result.add_sut_info(sut_node.node_info) > > - > > try: > > sut_node.set_up_execution(execution) > > execution_result.update_setup(Result.PASS) > > @@ -192,7 +405,10 @@ def _run_execution( > > > > else: > > for build_target in execution.build_targets: > > - self._run_build_target(sut_node, tg_node, build_target, execution, execution_result) > > + build_target_result = execution_result.add_build_target(build_target) > > + self._run_build_target( > > + sut_node, tg_node, build_target, build_target_result, test_suites_with_cases > > + ) > > > > finally: > > try: > > @@ -207,8 +423,8 @@ def _run_build_target( > > sut_node: SutNode, > > tg_node: TGNode, > > build_target: BuildTargetConfiguration, > > - execution: ExecutionConfiguration, > > - execution_result: ExecutionResult, > > + build_target_result: BuildTargetResult, > > + test_suites_with_cases: Iterable[TestSuiteWithCases], > > ) -> None: > > """Run the given build target. > > > > @@ -220,11 +436,11 @@ def _run_build_target( > > sut_node: The execution's sut node. > > tg_node: The execution's tg node. > > build_target: A build target's test run configuration. > > - execution: The build target's execution's test run configuration. > > - execution_result: The execution level result object associated with the execution. > > + build_target_result: The build target level result object associated > > + with the current build target. > > + test_suites_with_cases: The test suites with test cases to run. > > """ > > self._logger.info(f"Running build target '{build_target.name}'.") > > - build_target_result = execution_result.add_build_target(build_target) > > > > try: > > sut_node.set_up_build_target(build_target) > > @@ -236,7 +452,7 @@ def _run_build_target( > > build_target_result.update_setup(Result.FAIL, e) > > > > else: > > - self._run_test_suites(sut_node, tg_node, execution, build_target_result) > > + self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases) > > > > finally: > > try: > > @@ -250,10 +466,10 @@ def _run_test_suites( > > self, > > sut_node: SutNode, > > tg_node: TGNode, > > - execution: ExecutionConfiguration, > > build_target_result: BuildTargetResult, > > + test_suites_with_cases: Iterable[TestSuiteWithCases], > > ) -> None: > > - """Run the execution's (possibly a subset of) test suites using the current build target. > > + """Run `test_suites_with_cases` with the current build target. > > > > The method assumes the build target we're testing has already been built on the SUT node. > > The current build target thus corresponds to the current DPDK build present on the SUT node. > > @@ -264,22 +480,20 @@ def _run_test_suites( > > Args: > > sut_node: The execution's SUT node. > > tg_node: The execution's TG node. > > - execution: The execution's test run configuration associated > > - with the current build target. > > build_target_result: The build target level result object associated > > with the current build target. > > + test_suites_with_cases: The test suites with test cases to run. > > """ > > end_build_target = False > > - if not execution.skip_smoke_tests: > > - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] > > - for test_suite_config in execution.test_suites: > > + for test_suite_with_cases in test_suites_with_cases: > > + test_suite_result = build_target_result.add_test_suite( > > + test_suite_with_cases.test_suite_class.__name__ > > + ) > > try: > > - self._run_test_suite_module( > > - sut_node, tg_node, execution, build_target_result, test_suite_config > > - ) > > + self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases) > > except BlockingTestSuiteError as e: > > self._logger.exception( > > - f"An error occurred within {test_suite_config.test_suite}. " > > + f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. " > > "Skipping build target..." > > ) > > self._result.add_error(e) > > @@ -288,15 +502,14 @@ def _run_test_suites( > > if end_build_target: > > break > > > > - def _run_test_suite_module( > > + def _run_test_suite( > > self, > > sut_node: SutNode, > > tg_node: TGNode, > > - execution: ExecutionConfiguration, > > - build_target_result: BuildTargetResult, > > - test_suite_config: TestSuiteConfig, > > + test_suite_result: TestSuiteResult, > > + test_suite_with_cases: TestSuiteWithCases, > > ) -> None: > > - """Set up, execute and tear down all test suites in a single test suite module. > > + """Set up, execute and tear down `test_suite_with_cases`. > > > > The method assumes the build target we're testing has already been built on the SUT node. > > The current build target thus corresponds to the current DPDK build present on the SUT node. > > @@ -306,92 +519,79 @@ def _run_test_suite_module( > > > > Record the setup and the teardown and handle failures. > > > > - The test cases to execute are discovered when creating the :class:`TestSuite` object. > > - > > Args: > > sut_node: The execution's SUT node. > > tg_node: The execution's TG node. > > - execution: The execution's test run configuration associated > > - with the current build target. > > - build_target_result: The build target level result object associated > > - with the current build target. > > - test_suite_config: Test suite test run configuration specifying the test suite module > > - and possibly a subset of test cases of test suites in that module. > > + test_suite_result: The test suite level result object associated > > + with the current test suite. > > + test_suite_with_cases: The test suite with test cases to run. > > > > Raises: > > BlockingTestSuiteError: If a blocking test suite fails. > > """ > > + test_suite_name = test_suite_with_cases.test_suite_class.__name__ > > + test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) > > try: > > - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" > > - test_suite_classes = get_test_suites(full_suite_path) > > - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) > > - self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") > > + self._logger.info(f"Starting test suite setup: {test_suite_name}") > > + test_suite.set_up_suite() > > + test_suite_result.update_setup(Result.PASS) > > + self._logger.info(f"Test suite setup successful: {test_suite_name}") > > except Exception as e: > > - self._logger.exception("An error occurred when searching for test suites.") > > - self._result.update_setup(Result.ERROR, e) > > + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") > > + test_suite_result.update_setup(Result.ERROR, e) > > > > else: > > - for test_suite_class in test_suite_classes: > > - test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases) > > - > > - test_suite_name = test_suite.__class__.__name__ > > - test_suite_result = build_target_result.add_test_suite(test_suite_name) > > - try: > > - self._logger.info(f"Starting test suite setup: {test_suite_name}") > > - test_suite.set_up_suite() > > - test_suite_result.update_setup(Result.PASS) > > - self._logger.info(f"Test suite setup successful: {test_suite_name}") > > - except Exception as e: > > - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") > > - test_suite_result.update_setup(Result.ERROR, e) > > - > > - else: > > - self._execute_test_suite(execution.func, test_suite, test_suite_result) > > - > > - finally: > > - try: > > - test_suite.tear_down_suite() > > - sut_node.kill_cleanup_dpdk_apps() > > - test_suite_result.update_teardown(Result.PASS) > > - except Exception as e: > > - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") > > - self._logger.warning( > > - f"Test suite '{test_suite_name}' teardown failed, " > > - f"the next test suite may be affected." > > - ) > > - test_suite_result.update_setup(Result.ERROR, e) > > - if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: > > - raise BlockingTestSuiteError(test_suite_name) > > + self._execute_test_suite( > > + test_suite, > > + test_suite_with_cases.test_cases, > > + test_suite_result, > > + ) > > + finally: > > + try: > > + test_suite.tear_down_suite() > > + sut_node.kill_cleanup_dpdk_apps() > > + test_suite_result.update_teardown(Result.PASS) > > + except Exception as e: > > + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") > > + self._logger.warning( > > + f"Test suite '{test_suite_name}' teardown failed, " > > + "the next test suite may be affected." > > + ) > > + test_suite_result.update_setup(Result.ERROR, e) > > + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: > > + raise BlockingTestSuiteError(test_suite_name) > > > > def _execute_test_suite( > > - self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult > > + self, > > + test_suite: TestSuite, > > + test_cases: Iterable[MethodType], > > + test_suite_result: TestSuiteResult, > > ) -> None: > > - """Execute all discovered test cases in `test_suite`. > > + """Execute all `test_cases` in `test_suite`. > > > > If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment > > variable is set, in case of a test case failure, the test case will be executed again > > until it passes or it fails that many times in addition of the first failure. > > > > Args: > > - func: Whether to execute functional test cases. > > test_suite: The test suite object. > > + test_cases: The list of test case methods. > > test_suite_result: The test suite level result object associated > > with the current test suite. > > """ > > - if func: > > - for test_case_method in test_suite._get_functional_test_cases(): > > - test_case_name = test_case_method.__name__ > > - test_case_result = test_suite_result.add_test_case(test_case_name) > > - all_attempts = SETTINGS.re_run + 1 > > - attempt_nr = 1 > > + for test_case_method in test_cases: > > + test_case_name = test_case_method.__name__ > > + test_case_result = test_suite_result.add_test_case(test_case_name) > > + all_attempts = SETTINGS.re_run + 1 > > + attempt_nr = 1 > > + self._run_test_case(test_suite, test_case_method, test_case_result) > > + while not test_case_result and attempt_nr < all_attempts: > > + attempt_nr += 1 > > + self._logger.info( > > + f"Re-running FAILED test case '{test_case_name}'. " > > + f"Attempt number {attempt_nr} out of {all_attempts}." > > + ) > > self._run_test_case(test_suite, test_case_method, test_case_result) > > - while not test_case_result and attempt_nr < all_attempts: > > - attempt_nr += 1 > > - self._logger.info( > > - f"Re-running FAILED test case '{test_case_name}'. " > > - f"Attempt number {attempt_nr} out of {all_attempts}." > > - ) > > - self._run_test_case(test_suite, test_case_method, test_case_result) > > > > def _run_test_case( > > self, > > @@ -399,7 +599,7 @@ def _run_test_case( > > test_case_method: MethodType, > > test_case_result: TestCaseResult, > > ) -> None: > > - """Setup, execute and teardown a test case in `test_suite`. > > + """Setup, execute and teardown `test_case_method` from `test_suite`. > > > > Record the result of the setup and the teardown and handle failures. > > > > @@ -424,7 +624,7 @@ def _run_test_case( > > > > else: > > # run test case if setup was successful > > - self._execute_test_case(test_case_method, test_case_result) > > + self._execute_test_case(test_suite, test_case_method, test_case_result) > > > > finally: > > try: > > @@ -440,11 +640,15 @@ def _run_test_case( > > test_case_result.update(Result.ERROR) > > > > def _execute_test_case( > > - self, test_case_method: MethodType, test_case_result: TestCaseResult > > + self, > > + test_suite: TestSuite, > > + test_case_method: MethodType, > > + test_case_result: TestCaseResult, > > ) -> None: > > - """Execute one test case, record the result and handle failures. > > + """Execute `test_case_method` from `test_suite`, record the result and handle failures. > > > > Args: > > + test_suite: The test suite object. > > test_case_method: The test case method. > > test_case_result: The test case level result object associated > > with the current test case. > > @@ -452,7 +656,7 @@ def _execute_test_case( > > test_case_name = test_case_method.__name__ > > try: > > self._logger.info(f"Starting test case execution: {test_case_name}") > > - test_case_method() > > + test_case_method(test_suite) > > test_case_result.update(Result.PASS) > > self._logger.info(f"Test case execution PASSED: {test_case_name}") > > > > diff --git a/dts/framework/settings.py b/dts/framework/settings.py > > index 609c8d0e62..2b8bfbe0ed 100644 > > --- a/dts/framework/settings.py > > +++ b/dts/framework/settings.py > > @@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser: > > "--test-cases", > > action=_env_arg("DTS_TESTCASES"), > > default="", > > - help="[DTS_TESTCASES] Comma-separated list of test cases to execute. " > > - "Unknown test cases will be silently ignored.", > > + help="[DTS_TESTCASES] Comma-separated list of test cases to execute.", > > ) > > > > parser.add_argument( > > diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py > > index 4467749a9d..075195fd5b 100644 > > --- a/dts/framework/test_result.py > > +++ b/dts/framework/test_result.py > > @@ -25,7 +25,9 @@ > > > > import os.path > > from collections.abc import MutableSequence > > +from dataclasses import dataclass > > from enum import Enum, auto > > +from types import MethodType > > > > from .config import ( > > OS, > > @@ -36,10 +38,42 @@ > > CPUType, > > NodeConfiguration, > > NodeInfo, > > + TestSuiteConfig, > > ) > > from .exception import DTSError, ErrorSeverity > > from .logger import DTSLOG > > from .settings import SETTINGS > > +from .test_suite import TestSuite > > + > > + > > +@dataclass(slots=True, frozen=True) > > +class TestSuiteWithCases: > > + """A test suite class with test case methods. > > + > > + An auxiliary class holding a test case class with test case methods. The intended use of this > > + class is to hold a subset of test cases (which could be all test cases) because we don't have > > + all the data to instantiate the class at the point of inspection. The knowledge of this subset > > + is needed in case an error occurs before the class is instantiated and we need to record > > + which test cases were blocked by the error. > > + > > + Attributes: > > + test_suite_class: The test suite class. > > + test_cases: The test case methods. > > + """ > > + > > + test_suite_class: type[TestSuite] > > + test_cases: list[MethodType] > > + > > + def create_config(self) -> TestSuiteConfig: > > + """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases. > > + > > + Returns: > > + The :class:`TestSuiteConfig` representation. > > + """ > > + return TestSuiteConfig( > > + test_suite=self.test_suite_class.__name__, > > + test_cases=[test_case.__name__ for test_case in self.test_cases], > > + ) > > > > > > class Result(Enum): > > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py > > index b02fd36147..f9fe88093e 100644 > > --- a/dts/framework/test_suite.py > > +++ b/dts/framework/test_suite.py > > @@ -11,25 +11,17 @@ > > * Testbed (SUT, TG) configuration, > > * Packet sending and verification, > > * Test case verification. > > - > > -The module also defines a function, :func:`get_test_suites`, > > -for gathering test suites from a Python module. > > """ > > > > -import importlib > > -import inspect > > -import re > > from ipaddress import IPv4Interface, IPv6Interface, ip_interface > > -from types import MethodType > > -from typing import Any, ClassVar, Union > > +from typing import ClassVar, Union > > > > from scapy.layers.inet import IP # type: ignore[import] > > from scapy.layers.l2 import Ether # type: ignore[import] > > from scapy.packet import Packet, Padding # type: ignore[import] > > > > -from .exception import ConfigurationError, TestCaseVerifyError > > +from .exception import TestCaseVerifyError > > from .logger import DTSLOG, getLogger > > -from .settings import SETTINGS > > from .testbed_model import Port, PortLink, SutNode, TGNode > > from .utils import get_packet_summaries > > > > @@ -37,7 +29,6 @@ > > class TestSuite(object): > > """The base class with building blocks needed by most test cases. > > > > - * Test case filtering and collection, > > * Test suite setup/cleanup methods to override, > > * Test case setup/cleanup methods to override, > > * Test case verification, > > @@ -71,7 +62,6 @@ class TestSuite(object): > > #: will block the execution of all subsequent test suites in the current build target. > > is_blocking: ClassVar[bool] = False > > _logger: DTSLOG > > - _test_cases_to_run: list[str] > > _port_links: list[PortLink] > > _sut_port_ingress: Port > > _sut_port_egress: Port > > @@ -86,24 +76,19 @@ def __init__( > > self, > > sut_node: SutNode, > > tg_node: TGNode, > > - test_cases: list[str], > > ): > > """Initialize the test suite testbed information and basic configuration. > > > > - Process what test cases to run, find links between ports and set up > > - default IP addresses to be used when configuring them. > > + Find links between ports and set up default IP addresses to be used when > > + configuring them. > > > > Args: > > sut_node: The SUT node where the test suite will run. > > tg_node: The TG node where the test suite will run. > > - test_cases: The list of test cases to execute. > > - If empty, all test cases will be executed. > > """ > > self.sut_node = sut_node > > self.tg_node = tg_node > > self._logger = getLogger(self.__class__.__name__) > > - self._test_cases_to_run = test_cases > > - self._test_cases_to_run.extend(SETTINGS.test_cases) > > self._port_links = [] > > self._process_links() > > self._sut_port_ingress, self._tg_port_egress = ( > > @@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: > > if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst: > > return False > > return True > > - > > - def _get_functional_test_cases(self) -> list[MethodType]: > > - """Get all functional test cases defined in this TestSuite. > > - > > - Returns: > > - The list of functional test cases of this TestSuite. > > - """ > > - return self._get_test_cases(r"test_(?!perf_)") > > - > > - def _get_test_cases(self, test_case_regex: str) -> list[MethodType]: > > - """Return a list of test cases matching test_case_regex. > > - > > - Returns: > > - The list of test cases matching test_case_regex of this TestSuite. > > - """ > > - self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.") > > - filtered_test_cases = [] > > - for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod): > > - if self._should_be_executed(test_case_name, test_case_regex): > > - filtered_test_cases.append(test_case) > > - cases_str = ", ".join((x.__name__ for x in filtered_test_cases)) > > - self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.") > > - return filtered_test_cases > > - > > - def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool: > > - """Check whether the test case should be scheduled to be executed.""" > > - match = bool(re.match(test_case_regex, test_case_name)) > > - if self._test_cases_to_run: > > - return match and test_case_name in self._test_cases_to_run > > - > > - return match > > - > > - > > -def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: > > - r"""Find all :class:`TestSuite`\s in a Python module. > > - > > - Args: > > - testsuite_module_path: The path to the Python module. > > - > > - Returns: > > - The list of :class:`TestSuite`\s found within the Python module. > > - > > - Raises: > > - ConfigurationError: The test suite module was not found. > > - """ > > - > > - def is_test_suite(object: Any) -> bool: > > - try: > > - if issubclass(object, TestSuite) and object is not TestSuite: > > - return True > > - except TypeError: > > - return False > > - return False > > - > > - try: > > - testcase_module = importlib.import_module(testsuite_module_path) > > - except ModuleNotFoundError as e: > > - raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e > > - return [ > > - test_suite_class > > - for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite) > > - ] > > diff --git a/dts/pyproject.toml b/dts/pyproject.toml > > index 28bd970ae4..8eb92b4f11 100644 > > --- a/dts/pyproject.toml > > +++ b/dts/pyproject.toml > > @@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes" > > format = "pylint" > > max_line_length = 100 > > > > +[tool.pylama.linter.pycodestyle] > > +ignore = "E203,W503" > > + > > [tool.pylama.linter.pydocstyle] > > convention = "google" > > > > diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py > > index 5e2bac14bd..7b2a0e97f8 100644 > > --- a/dts/tests/TestSuite_smoke_tests.py > > +++ b/dts/tests/TestSuite_smoke_tests.py > > @@ -21,7 +21,7 @@ > > from framework.utils import REGEX_FOR_PCI_ADDRESS > > > > > > -class SmokeTests(TestSuite): > > +class TestSmokeTests(TestSuite): > > """DPDK and infrastructure smoke test suite. > > > > The test cases validate the most basic DPDK functionality needed for all other test suites. > > -- > > 2.34.1 > > ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 4/7] dts: reorganize test result 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš ` (2 preceding siblings ...) 2024-02-06 14:57 ` [PATCH v2 3/7] dts: filter test suites in executions Juraj Linkeš @ 2024-02-06 14:57 ` Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš ` (2 subsequent siblings) 6 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš The current order of Result classes in the test_suite.py module is guided by the needs of type hints, which is not as intuitively readable as ordering them by the occurrences in code. The order goes from the topmost level to lowermost: BaseResult DTSResult ExecutionResult BuildTargetResult TestSuiteResult TestCaseResult This is the same order as they're used in the runner module and they're also used in the same order between themselves in the test_result module. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/test_result.py | 411 ++++++++++++++++++----------------- 1 file changed, 206 insertions(+), 205 deletions(-) diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index 075195fd5b..abdbafab10 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -28,6 +28,7 @@ from dataclasses import dataclass from enum import Enum, auto from types import MethodType +from typing import Union from .config import ( OS, @@ -129,58 +130,6 @@ def __bool__(self) -> bool: return bool(self.result) -class Statistics(dict): - """How many test cases ended in which result state along some other basic information. - - Subclassing :class:`dict` provides a convenient way to format the data. - - The data are stored in the following keys: - - * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases. - * **DPDK VERSION** (:class:`str`) -- The tested DPDK version. - """ - - def __init__(self, dpdk_version: str | None): - """Extend the constructor with keys in which the data are stored. - - Args: - dpdk_version: The version of tested DPDK. - """ - super(Statistics, self).__init__() - for result in Result: - self[result.name] = 0 - self["PASS RATE"] = 0.0 - self["DPDK VERSION"] = dpdk_version - - def __iadd__(self, other: Result) -> "Statistics": - """Add a Result to the final count. - - Example: - stats: Statistics = Statistics() # empty Statistics - stats += Result.PASS # add a Result to `stats` - - Args: - other: The Result to add to this statistics object. - - Returns: - The modified statistics object. - """ - self[other.name] += 1 - self["PASS RATE"] = ( - float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result) - ) - return self - - def __str__(self) -> str: - """Each line contains the formatted key = value pair.""" - stats_str = "" - for key, value in self.items(): - stats_str += f"{key:<12} = {value}\n" - # according to docs, we should use \n when writing to text files - # on all platforms - return stats_str - - class BaseResult(object): """Common data and behavior of DTS results. @@ -245,7 +194,7 @@ def get_errors(self) -> list[Exception]: """ return self._get_setup_teardown_errors() + self._get_inner_errors() - def add_stats(self, statistics: Statistics) -> None: + def add_stats(self, statistics: "Statistics") -> None: """Collate stats from the whole result hierarchy. Args: @@ -255,91 +204,149 @@ def add_stats(self, statistics: Statistics) -> None: inner_result.add_stats(statistics) -class TestCaseResult(BaseResult, FixtureResult): - r"""The test case specific result. +class DTSResult(BaseResult): + """Stores environment information and test results from a DTS run. - Stores the result of the actual test case. This is done by adding an extra superclass - in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and - the class is itself a record of the test case. + * Execution level information, such as testbed and the test suite list, + * Build target level information, such as compiler, target OS and cpu, + * Test suite and test case results, + * All errors that are caught and recorded during DTS execution. + + The information is stored hierarchically. This is the first level of the hierarchy + and as such is where the data form the whole hierarchy is collated or processed. + + The internal list stores the results of all executions. Attributes: - test_case_name: The test case name. + dpdk_version: The DPDK version to record. """ - test_case_name: str + dpdk_version: str | None + _logger: DTSLOG + _errors: list[Exception] + _return_code: ErrorSeverity + _stats_result: Union["Statistics", None] + _stats_filename: str - def __init__(self, test_case_name: str): - """Extend the constructor with `test_case_name`. + def __init__(self, logger: DTSLOG): + """Extend the constructor with top-level specifics. Args: - test_case_name: The test case's name. + logger: The logger instance the whole result will use. """ - super(TestCaseResult, self).__init__() - self.test_case_name = test_case_name + super(DTSResult, self).__init__() + self.dpdk_version = None + self._logger = logger + self._errors = [] + self._return_code = ErrorSeverity.NO_ERR + self._stats_result = None + self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") - def update(self, result: Result, error: Exception | None = None) -> None: - """Update the test case result. + def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult": + """Add and return the inner result (execution). - This updates the result of the test case itself and doesn't affect - the results of the setup and teardown steps in any way. + Args: + sut_node: The SUT node's test run configuration. + + Returns: + The execution's result. + """ + execution_result = ExecutionResult(sut_node) + self._inner_results.append(execution_result) + return execution_result + + def add_error(self, error: Exception) -> None: + """Record an error that occurred outside any execution. Args: - result: The result of the test case. - error: The error that occurred in case of a failure. + error: The exception to record. """ - self.result = result - self.error = error + self._errors.append(error) - def _get_inner_errors(self) -> list[Exception]: - if self.error: - return [self.error] - return [] + def process(self) -> None: + """Process the data after a whole DTS run. - def add_stats(self, statistics: Statistics) -> None: - r"""Add the test case result to statistics. + The data is added to inner objects during runtime and this object is not updated + at that time. This requires us to process the inner data after it's all been gathered. - The base method goes through the hierarchy recursively and this method is here to stop - the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree. + The processing gathers all errors and the statistics of test case results. + """ + self._errors += self.get_errors() + if self._errors and self._logger: + self._logger.debug("Summary of errors:") + for error in self._errors: + self._logger.debug(repr(error)) - Args: - statistics: The :class:`Statistics` object where the stats will be added. + self._stats_result = Statistics(self.dpdk_version) + self.add_stats(self._stats_result) + with open(self._stats_filename, "w+") as stats_file: + stats_file.write(str(self._stats_result)) + + def get_return_code(self) -> int: + """Go through all stored Exceptions and return the final DTS error code. + + Returns: + The highest error code found. """ - statistics += self.result + for error in self._errors: + error_return_code = ErrorSeverity.GENERIC_ERR + if isinstance(error, DTSError): + error_return_code = error.severity - def __bool__(self) -> bool: - """The test case passed only if setup, teardown and the test case itself passed.""" - return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) + if error_return_code > self._return_code: + self._return_code = error_return_code + return int(self._return_code) -class TestSuiteResult(BaseResult): - """The test suite specific result. - The internal list stores the results of all test cases in a given test suite. +class ExecutionResult(BaseResult): + """The execution specific result. + + The internal list stores the results of all build targets in a given execution. Attributes: - suite_name: The test suite name. + sut_node: The SUT node used in the execution. + sut_os_name: The operating system of the SUT node. + sut_os_version: The operating system version of the SUT node. + sut_kernel_version: The operating system kernel version of the SUT node. """ - suite_name: str + sut_node: NodeConfiguration + sut_os_name: str + sut_os_version: str + sut_kernel_version: str - def __init__(self, suite_name: str): - """Extend the constructor with `suite_name`. + def __init__(self, sut_node: NodeConfiguration): + """Extend the constructor with the `sut_node`'s config. Args: - suite_name: The test suite's name. + sut_node: The SUT node's test run configuration used in the execution. """ - super(TestSuiteResult, self).__init__() - self.suite_name = suite_name + super(ExecutionResult, self).__init__() + self.sut_node = sut_node - def add_test_case(self, test_case_name: str) -> TestCaseResult: - """Add and return the inner result (test case). + def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult": + """Add and return the inner result (build target). + + Args: + build_target: The build target's test run configuration. Returns: - The test case's result. + The build target's result. """ - test_case_result = TestCaseResult(test_case_name) - self._inner_results.append(test_case_result) - return test_case_result + build_target_result = BuildTargetResult(build_target) + self._inner_results.append(build_target_result) + return build_target_result + + def add_sut_info(self, sut_info: NodeInfo) -> None: + """Add SUT information gathered at runtime. + + Args: + sut_info: The additional SUT node information. + """ + self.sut_os_name = sut_info.os_name + self.sut_os_version = sut_info.os_version + self.sut_kernel_version = sut_info.kernel_version class BuildTargetResult(BaseResult): @@ -386,7 +393,7 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None: self.compiler_version = versions.compiler_version self.dpdk_version = versions.dpdk_version - def add_test_suite(self, test_suite_name: str) -> TestSuiteResult: + def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult": """Add and return the inner result (test suite). Returns: @@ -397,146 +404,140 @@ def add_test_suite(self, test_suite_name: str) -> TestSuiteResult: return test_suite_result -class ExecutionResult(BaseResult): - """The execution specific result. +class TestSuiteResult(BaseResult): + """The test suite specific result. - The internal list stores the results of all build targets in a given execution. + The internal list stores the results of all test cases in a given test suite. Attributes: - sut_node: The SUT node used in the execution. - sut_os_name: The operating system of the SUT node. - sut_os_version: The operating system version of the SUT node. - sut_kernel_version: The operating system kernel version of the SUT node. + suite_name: The test suite name. """ - sut_node: NodeConfiguration - sut_os_name: str - sut_os_version: str - sut_kernel_version: str + suite_name: str - def __init__(self, sut_node: NodeConfiguration): - """Extend the constructor with the `sut_node`'s config. + def __init__(self, suite_name: str): + """Extend the constructor with `suite_name`. Args: - sut_node: The SUT node's test run configuration used in the execution. + suite_name: The test suite's name. """ - super(ExecutionResult, self).__init__() - self.sut_node = sut_node - - def add_build_target(self, build_target: BuildTargetConfiguration) -> BuildTargetResult: - """Add and return the inner result (build target). + super(TestSuiteResult, self).__init__() + self.suite_name = suite_name - Args: - build_target: The build target's test run configuration. + def add_test_case(self, test_case_name: str) -> "TestCaseResult": + """Add and return the inner result (test case). Returns: - The build target's result. - """ - build_target_result = BuildTargetResult(build_target) - self._inner_results.append(build_target_result) - return build_target_result - - def add_sut_info(self, sut_info: NodeInfo) -> None: - """Add SUT information gathered at runtime. - - Args: - sut_info: The additional SUT node information. + The test case's result. """ - self.sut_os_name = sut_info.os_name - self.sut_os_version = sut_info.os_version - self.sut_kernel_version = sut_info.kernel_version + test_case_result = TestCaseResult(test_case_name) + self._inner_results.append(test_case_result) + return test_case_result -class DTSResult(BaseResult): - """Stores environment information and test results from a DTS run. - - * Execution level information, such as testbed and the test suite list, - * Build target level information, such as compiler, target OS and cpu, - * Test suite and test case results, - * All errors that are caught and recorded during DTS execution. - - The information is stored hierarchically. This is the first level of the hierarchy - and as such is where the data form the whole hierarchy is collated or processed. +class TestCaseResult(BaseResult, FixtureResult): + r"""The test case specific result. - The internal list stores the results of all executions. + Stores the result of the actual test case. This is done by adding an extra superclass + in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and + the class is itself a record of the test case. Attributes: - dpdk_version: The DPDK version to record. + test_case_name: The test case name. """ - dpdk_version: str | None - _logger: DTSLOG - _errors: list[Exception] - _return_code: ErrorSeverity - _stats_result: Statistics | None - _stats_filename: str + test_case_name: str - def __init__(self, logger: DTSLOG): - """Extend the constructor with top-level specifics. + def __init__(self, test_case_name: str): + """Extend the constructor with `test_case_name`. Args: - logger: The logger instance the whole result will use. + test_case_name: The test case's name. """ - super(DTSResult, self).__init__() - self.dpdk_version = None - self._logger = logger - self._errors = [] - self._return_code = ErrorSeverity.NO_ERR - self._stats_result = None - self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") + super(TestCaseResult, self).__init__() + self.test_case_name = test_case_name - def add_execution(self, sut_node: NodeConfiguration) -> ExecutionResult: - """Add and return the inner result (execution). + def update(self, result: Result, error: Exception | None = None) -> None: + """Update the test case result. - Args: - sut_node: The SUT node's test run configuration. + This updates the result of the test case itself and doesn't affect + the results of the setup and teardown steps in any way. - Returns: - The execution's result. + Args: + result: The result of the test case. + error: The error that occurred in case of a failure. """ - execution_result = ExecutionResult(sut_node) - self._inner_results.append(execution_result) - return execution_result + self.result = result + self.error = error - def add_error(self, error: Exception) -> None: - """Record an error that occurred outside any execution. + def _get_inner_errors(self) -> list[Exception]: + if self.error: + return [self.error] + return [] + + def add_stats(self, statistics: "Statistics") -> None: + r"""Add the test case result to statistics. + + The base method goes through the hierarchy recursively and this method is here to stop + the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree. Args: - error: The exception to record. + statistics: The :class:`Statistics` object where the stats will be added. """ - self._errors.append(error) + statistics += self.result - def process(self) -> None: - """Process the data after a whole DTS run. + def __bool__(self) -> bool: + """The test case passed only if setup, teardown and the test case itself passed.""" + return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) - The data is added to inner objects during runtime and this object is not updated - at that time. This requires us to process the inner data after it's all been gathered. - The processing gathers all errors and the statistics of test case results. +class Statistics(dict): + """How many test cases ended in which result state along some other basic information. + + Subclassing :class:`dict` provides a convenient way to format the data. + + The data are stored in the following keys: + + * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases. + * **DPDK VERSION** (:class:`str`) -- The tested DPDK version. + """ + + def __init__(self, dpdk_version: str | None): + """Extend the constructor with keys in which the data are stored. + + Args: + dpdk_version: The version of tested DPDK. """ - self._errors += self.get_errors() - if self._errors and self._logger: - self._logger.debug("Summary of errors:") - for error in self._errors: - self._logger.debug(repr(error)) + super(Statistics, self).__init__() + for result in Result: + self[result.name] = 0 + self["PASS RATE"] = 0.0 + self["DPDK VERSION"] = dpdk_version - self._stats_result = Statistics(self.dpdk_version) - self.add_stats(self._stats_result) - with open(self._stats_filename, "w+") as stats_file: - stats_file.write(str(self._stats_result)) + def __iadd__(self, other: Result) -> "Statistics": + """Add a Result to the final count. - def get_return_code(self) -> int: - """Go through all stored Exceptions and return the final DTS error code. + Example: + stats: Statistics = Statistics() # empty Statistics + stats += Result.PASS # add a Result to `stats` + + Args: + other: The Result to add to this statistics object. Returns: - The highest error code found. + The modified statistics object. """ - for error in self._errors: - error_return_code = ErrorSeverity.GENERIC_ERR - if isinstance(error, DTSError): - error_return_code = error.severity - - if error_return_code > self._return_code: - self._return_code = error_return_code + self[other.name] += 1 + self["PASS RATE"] = ( + float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result) + ) + return self - return int(self._return_code) + def __str__(self) -> str: + """Each line contains the formatted key = value pair.""" + stats_str = "" + for key, value in self.items(): + stats_str += f"{key:<12} = {value}\n" + # according to docs, we should use \n when writing to text files + # on all platforms + return stats_str -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 5/7] dts: block all test cases when earlier setup fails 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš ` (3 preceding siblings ...) 2024-02-06 14:57 ` [PATCH v2 4/7] dts: reorganize test result Juraj Linkeš @ 2024-02-06 14:57 ` Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 6/7] dts: refactor logging configuration Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 7/7] dts: improve test suite and case filtering Juraj Linkeš 6 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš In case of a failure before a test suite, the child results will be recursively recorded as blocked, giving us a full report which was missing previously. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/runner.py | 21 ++-- dts/framework/test_result.py | 186 +++++++++++++++++++++++++---------- 2 files changed, 148 insertions(+), 59 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 3e95cf9e26..f58b0adc13 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -60,13 +60,15 @@ class DTSRunner: Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult` or one of its subclasses. The test case results are also recorded. - If an error occurs, the current stage is aborted, the error is recorded and the run continues in - the next iteration of the same stage. The return code is the highest `severity` of all + If an error occurs, the current stage is aborted, the error is recorded, everything in + the inner stages is marked as blocked and the run continues in the next iteration + of the same stage. The return code is the highest `severity` of all :class:`~.framework.exception.DTSError`\s. Example: - An error occurs in a build target setup. The current build target is aborted and the run - continues with the next build target. If the errored build target was the last one in the + An error occurs in a build target setup. The current build target is aborted, + all test suites and their test cases are marked as blocked and the run continues + with the next build target. If the errored build target was the last one in the given execution, the next execution begins. """ @@ -100,6 +102,10 @@ def run(self): test case within the test suite is set up, executed and torn down. After all test cases have been executed, the test suite is torn down and the next build target will be tested. + In order to properly mark test suites and test cases as blocked in case of a failure, + we need to have discovered which test suites and test cases to run before any failures + happen. The discovery happens at the earliest point at the start of each execution. + All the nested steps look like this: #. Execution setup @@ -134,11 +140,12 @@ def run(self): self._logger.info( f"Running execution with SUT '{execution.system_under_test_node.name}'." ) - execution_result = self._result.add_execution(execution.system_under_test_node) + execution_result = self._result.add_execution(execution) try: test_suites_with_cases = self._get_test_suites_with_cases( execution.test_suites, execution.func, execution.perf ) + execution_result.test_suites_with_cases = test_suites_with_cases except Exception as e: self._logger.exception( f"Invalid test suite configuration found: " f"{execution.test_suites}." @@ -486,9 +493,7 @@ def _run_test_suites( """ end_build_target = False for test_suite_with_cases in test_suites_with_cases: - test_suite_result = build_target_result.add_test_suite( - test_suite_with_cases.test_suite_class.__name__ - ) + test_suite_result = build_target_result.add_test_suite(test_suite_with_cases) try: self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases) except BlockingTestSuiteError as e: diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index abdbafab10..eedb2d20ee 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -37,7 +37,7 @@ BuildTargetInfo, Compiler, CPUType, - NodeConfiguration, + ExecutionConfiguration, NodeInfo, TestSuiteConfig, ) @@ -88,6 +88,8 @@ class Result(Enum): ERROR = auto() #: SKIP = auto() + #: + BLOCK = auto() def __bool__(self) -> bool: """Only PASS is True.""" @@ -141,21 +143,26 @@ class BaseResult(object): Attributes: setup_result: The result of the setup of the particular stage. teardown_result: The results of the teardown of the particular stage. + child_results: The results of the descendants in the results hierarchy. """ setup_result: FixtureResult teardown_result: FixtureResult - _inner_results: MutableSequence["BaseResult"] + child_results: MutableSequence["BaseResult"] def __init__(self): """Initialize the constructor.""" self.setup_result = FixtureResult() self.teardown_result = FixtureResult() - self._inner_results = [] + self.child_results = [] def update_setup(self, result: Result, error: Exception | None = None) -> None: """Store the setup result. + If the result is :attr:`~Result.BLOCK`, :attr:`~Result.ERROR` or :attr:`~Result.FAIL`, + then the corresponding child results in result hierarchy + are also marked with :attr:`~Result.BLOCK`. + Args: result: The result of the setup. error: The error that occurred in case of a failure. @@ -163,6 +170,16 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None: self.setup_result.result = result self.setup_result.error = error + if result in [Result.BLOCK, Result.ERROR, Result.FAIL]: + self.update_teardown(Result.BLOCK) + self._block_result() + + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed. + + The blocking of child results should be done in overloaded methods. + """ + def update_teardown(self, result: Result, error: Exception | None = None) -> None: """Store the teardown result. @@ -181,10 +198,8 @@ def _get_setup_teardown_errors(self) -> list[Exception]: errors.append(self.teardown_result.error) return errors - def _get_inner_errors(self) -> list[Exception]: - return [ - error for inner_result in self._inner_results for error in inner_result.get_errors() - ] + def _get_child_errors(self) -> list[Exception]: + return [error for child_result in self.child_results for error in child_result.get_errors()] def get_errors(self) -> list[Exception]: """Compile errors from the whole result hierarchy. @@ -192,7 +207,7 @@ def get_errors(self) -> list[Exception]: Returns: The errors from setup, teardown and all errors found in the whole result hierarchy. """ - return self._get_setup_teardown_errors() + self._get_inner_errors() + return self._get_setup_teardown_errors() + self._get_child_errors() def add_stats(self, statistics: "Statistics") -> None: """Collate stats from the whole result hierarchy. @@ -200,8 +215,8 @@ def add_stats(self, statistics: "Statistics") -> None: Args: statistics: The :class:`Statistics` object where the stats will be collated. """ - for inner_result in self._inner_results: - inner_result.add_stats(statistics) + for child_result in self.child_results: + child_result.add_stats(statistics) class DTSResult(BaseResult): @@ -242,18 +257,18 @@ def __init__(self, logger: DTSLOG): self._stats_result = None self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") - def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult": - """Add and return the inner result (execution). + def add_execution(self, execution: ExecutionConfiguration) -> "ExecutionResult": + """Add and return the child result (execution). Args: - sut_node: The SUT node's test run configuration. + execution: The execution's test run configuration. Returns: The execution's result. """ - execution_result = ExecutionResult(sut_node) - self._inner_results.append(execution_result) - return execution_result + result = ExecutionResult(execution) + self.child_results.append(result) + return result def add_error(self, error: Exception) -> None: """Record an error that occurred outside any execution. @@ -266,8 +281,8 @@ def add_error(self, error: Exception) -> None: def process(self) -> None: """Process the data after a whole DTS run. - The data is added to inner objects during runtime and this object is not updated - at that time. This requires us to process the inner data after it's all been gathered. + The data is added to child objects during runtime and this object is not updated + at that time. This requires us to process the child data after it's all been gathered. The processing gathers all errors and the statistics of test case results. """ @@ -305,28 +320,30 @@ class ExecutionResult(BaseResult): The internal list stores the results of all build targets in a given execution. Attributes: - sut_node: The SUT node used in the execution. sut_os_name: The operating system of the SUT node. sut_os_version: The operating system version of the SUT node. sut_kernel_version: The operating system kernel version of the SUT node. """ - sut_node: NodeConfiguration sut_os_name: str sut_os_version: str sut_kernel_version: str + _config: ExecutionConfiguration + _parent_result: DTSResult + _test_suites_with_cases: list[TestSuiteWithCases] - def __init__(self, sut_node: NodeConfiguration): - """Extend the constructor with the `sut_node`'s config. + def __init__(self, execution: ExecutionConfiguration): + """Extend the constructor with the execution's config and DTSResult. Args: - sut_node: The SUT node's test run configuration used in the execution. + execution: The execution's test run configuration. """ super(ExecutionResult, self).__init__() - self.sut_node = sut_node + self._config = execution + self._test_suites_with_cases = [] def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult": - """Add and return the inner result (build target). + """Add and return the child result (build target). Args: build_target: The build target's test run configuration. @@ -334,9 +351,34 @@ def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTarg Returns: The build target's result. """ - build_target_result = BuildTargetResult(build_target) - self._inner_results.append(build_target_result) - return build_target_result + result = BuildTargetResult( + self._test_suites_with_cases, + build_target, + ) + self.child_results.append(result) + return result + + @property + def test_suites_with_cases(self) -> list[TestSuiteWithCases]: + """The test suites with test cases to be executed in this execution. + + The test suites can only be assigned once. + + Returns: + The list of test suites with test cases. If an error occurs between + the initialization of :class:`ExecutionResult` and assigning test cases to the instance, + return an empty list, representing that we don't know what to execute. + """ + return self._test_suites_with_cases + + @test_suites_with_cases.setter + def test_suites_with_cases(self, test_suites_with_cases: list[TestSuiteWithCases]) -> None: + if self._test_suites_with_cases: + raise ValueError( + "Attempted to assign test suites to an execution result " + "which already has test suites." + ) + self._test_suites_with_cases = test_suites_with_cases def add_sut_info(self, sut_info: NodeInfo) -> None: """Add SUT information gathered at runtime. @@ -348,6 +390,12 @@ def add_sut_info(self, sut_info: NodeInfo) -> None: self.sut_os_version = sut_info.os_version self.sut_kernel_version = sut_info.kernel_version + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + for build_target in self._config.build_targets: + child_result = self.add_build_target(build_target) + child_result.update_setup(Result.BLOCK) + class BuildTargetResult(BaseResult): """The build target specific result. @@ -369,11 +417,17 @@ class BuildTargetResult(BaseResult): compiler: Compiler compiler_version: str | None dpdk_version: str | None + _test_suites_with_cases: list[TestSuiteWithCases] - def __init__(self, build_target: BuildTargetConfiguration): - """Extend the constructor with the `build_target`'s build target config. + def __init__( + self, + test_suites_with_cases: list[TestSuiteWithCases], + build_target: BuildTargetConfiguration, + ): + """Extend the constructor with the build target's config and ExecutionResult. Args: + test_suites_with_cases: The test suites with test cases to be run in this build target. build_target: The build target's test run configuration. """ super(BuildTargetResult, self).__init__() @@ -383,6 +437,23 @@ def __init__(self, build_target: BuildTargetConfiguration): self.compiler = build_target.compiler self.compiler_version = None self.dpdk_version = None + self._test_suites_with_cases = test_suites_with_cases + + def add_test_suite( + self, + test_suite_with_cases: TestSuiteWithCases, + ) -> "TestSuiteResult": + """Add and return the child result (test suite). + + Args: + test_suite_with_cases: The test suite with test cases. + + Returns: + The test suite's result. + """ + result = TestSuiteResult(test_suite_with_cases) + self.child_results.append(result) + return result def add_build_target_info(self, versions: BuildTargetInfo) -> None: """Add information about the build target gathered at runtime. @@ -393,15 +464,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None: self.compiler_version = versions.compiler_version self.dpdk_version = versions.dpdk_version - def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult": - """Add and return the inner result (test suite). - - Returns: - The test suite's result. - """ - test_suite_result = TestSuiteResult(test_suite_name) - self._inner_results.append(test_suite_result) - return test_suite_result + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + for test_suite_with_cases in self._test_suites_with_cases: + child_result = self.add_test_suite(test_suite_with_cases) + child_result.update_setup(Result.BLOCK) class TestSuiteResult(BaseResult): @@ -410,29 +477,42 @@ class TestSuiteResult(BaseResult): The internal list stores the results of all test cases in a given test suite. Attributes: - suite_name: The test suite name. + test_suite_name: The test suite name. """ - suite_name: str + test_suite_name: str + _test_suite_with_cases: TestSuiteWithCases + _parent_result: BuildTargetResult + _child_configs: list[str] - def __init__(self, suite_name: str): - """Extend the constructor with `suite_name`. + def __init__(self, test_suite_with_cases: TestSuiteWithCases): + """Extend the constructor with test suite's config and BuildTargetResult. Args: - suite_name: The test suite's name. + test_suite_with_cases: The test suite with test cases. """ super(TestSuiteResult, self).__init__() - self.suite_name = suite_name + self.test_suite_name = test_suite_with_cases.test_suite_class.__name__ + self._test_suite_with_cases = test_suite_with_cases def add_test_case(self, test_case_name: str) -> "TestCaseResult": - """Add and return the inner result (test case). + """Add and return the child result (test case). + + Args: + test_case_name: The name of the test case. Returns: The test case's result. """ - test_case_result = TestCaseResult(test_case_name) - self._inner_results.append(test_case_result) - return test_case_result + result = TestCaseResult(test_case_name) + self.child_results.append(result) + return result + + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + for test_case_method in self._test_suite_with_cases.test_cases: + child_result = self.add_test_case(test_case_method.__name__) + child_result.update_setup(Result.BLOCK) class TestCaseResult(BaseResult, FixtureResult): @@ -449,7 +529,7 @@ class TestCaseResult(BaseResult, FixtureResult): test_case_name: str def __init__(self, test_case_name: str): - """Extend the constructor with `test_case_name`. + """Extend the constructor with test case's name and TestSuiteResult. Args: test_case_name: The test case's name. @@ -470,7 +550,7 @@ def update(self, result: Result, error: Exception | None = None) -> None: self.result = result self.error = error - def _get_inner_errors(self) -> list[Exception]: + def _get_child_errors(self) -> list[Exception]: if self.error: return [self.error] return [] @@ -486,6 +566,10 @@ def add_stats(self, statistics: "Statistics") -> None: """ statistics += self.result + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + self.update(Result.BLOCK) + def __bool__(self) -> bool: """The test case passed only if setup, teardown and the test case itself passed.""" return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 6/7] dts: refactor logging configuration 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš ` (4 preceding siblings ...) 2024-02-06 14:57 ` [PATCH v2 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš @ 2024-02-06 14:57 ` Juraj Linkeš 2024-02-12 16:45 ` Jeremy Spewock 2024-02-06 14:57 ` [PATCH v2 7/7] dts: improve test suite and case filtering Juraj Linkeš 6 siblings, 1 reply; 44+ messages in thread From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš Remove unused parts of the code and add useful features: 1. Add DTS execution stages such as execution and test suite to better identify where in the DTS lifecycle we are when investigating logs, 2. Logging to separate files in specific stages, which is mainly useful for having test suite logs in additional separate files. 3. Remove the dependence on the settings module which enhances the usefulness of the logger module, as it can now be imported in more modules. The execution stages and the files to log to are the same for all DTS loggers. To achieve this, we have one DTS root logger which should be used for handling stage switching and all other loggers are children of this DTS root logger. The DTS root logger is the one where we change the behavior of all loggers (the stage and which files to log to) and the child loggers just log messages under a different name. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/logger.py | 235 +++++++++++------- dts/framework/remote_session/__init__.py | 6 +- .../interactive_remote_session.py | 6 +- .../remote_session/interactive_shell.py | 6 +- .../remote_session/remote_session.py | 8 +- dts/framework/runner.py | 19 +- dts/framework/test_result.py | 6 +- dts/framework/test_suite.py | 6 +- dts/framework/testbed_model/node.py | 11 +- dts/framework/testbed_model/os_session.py | 7 +- .../traffic_generator/traffic_generator.py | 6 +- dts/main.py | 1 - 12 files changed, 183 insertions(+), 134 deletions(-) diff --git a/dts/framework/logger.py b/dts/framework/logger.py index cfa6e8cd72..568edad82d 100644 --- a/dts/framework/logger.py +++ b/dts/framework/logger.py @@ -5,141 +5,186 @@ """DTS logger module. -DTS framework and TestSuite logs are saved in different log files. +The module provides several additional features: + + * The storage of DTS execution stages, + * Logging to console, a human-readable log file and a machine-readable log file, + * Optional log files for specific stages. """ import logging -import os.path -from typing import TypedDict +from enum import auto +from logging import FileHandler, StreamHandler +from pathlib import Path +from typing import ClassVar -from .settings import SETTINGS +from .utils import StrEnum date_fmt = "%Y/%m/%d %H:%M:%S" -stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" +stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s" +dts_root_logger_name = "dts" + + +class DtsStage(StrEnum): + """The DTS execution stage.""" + #: + pre_execution = auto() + #: + execution = auto() + #: + build_target = auto() + #: + suite = auto() + #: + post_execution = auto() -class DTSLOG(logging.LoggerAdapter): - """DTS logger adapter class for framework and testsuites. - The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment - variable control the verbosity of output. If enabled, all messages will be emitted to the - console. +class DTSLogger(logging.Logger): + """The DTS logger class. - The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment - variable modify the directory where the logs will be stored. + The class extends the :class:`~logging.Logger` class to add the DTS execution stage information + to log records. The stage is common to all loggers, so it's stored in a class variable. - Attributes: - node: The additional identifier. Currently unused. - sh: The handler which emits logs to console. - fh: The handler which emits logs to a file. - verbose_fh: Just as fh, but logs with a different, more verbose, format. + Any time we switch to a new stage, we have the ability to log to an additional log file along + with a supplementary log file with machine-readable format. These two log files are used until + a new stage switch occurs. This is useful mainly for logging per test suite. """ - _logger: logging.Logger - node: str - sh: logging.StreamHandler - fh: logging.FileHandler - verbose_fh: logging.FileHandler + _stage: ClassVar[DtsStage] = DtsStage.pre_execution + _extra_file_handlers: list[FileHandler] = [] - def __init__(self, logger: logging.Logger, node: str = "suite"): - """Extend the constructor with additional handlers. + def __init__(self, *args, **kwargs): + """Extend the constructor with extra file handlers.""" + self._extra_file_handlers = [] + super().__init__(*args, **kwargs) - One handler logs to the console, the other one to a file, with either a regular or verbose - format. + def makeRecord(self, *args, **kwargs): + """Generates a record with additional stage information. - Args: - logger: The logger from which to create the logger adapter. - node: An additional identifier. Currently unused. + This is the default method for the :class:`~logging.Logger` class. We extend it + to add stage information to the record. + + :meta private: + + Returns: + record: The generated record with the stage information. """ - self._logger = logger - # 1 means log everything, this will be used by file handlers if their level - # is not set - self._logger.setLevel(1) + record = super().makeRecord(*args, **kwargs) + record.stage = DTSLogger._stage + return record + + def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None: + """Add logger handlers to the DTS root logger. + + This method should be called only on the DTS root logger. + The log records from child loggers will propagate to these handlers. + + Three handlers are added: - self.node = node + * A console handler, + * A file handler, + * A supplementary file handler with machine-readable logs + containing more debug information. - # add handler to emit to stdout - sh = logging.StreamHandler() + All log messages will be logged to files. The log level of the console handler + is configurable with `verbose`. + + Args: + verbose: If :data:`True`, log all messages to the console. + If :data:`False`, log to console with the :data:`logging.INFO` level. + output_dir: The directory where the log files will be located. + The names of the log files correspond to the name of the logger instance. + """ + self.setLevel(1) + + sh = StreamHandler() sh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) - sh.setLevel(logging.INFO) # console handler default level + if not verbose: + sh.setLevel(logging.INFO) + self.addHandler(sh) - if SETTINGS.verbose is True: - sh.setLevel(logging.DEBUG) + self._add_file_handlers(Path(output_dir, self.name)) - self._logger.addHandler(sh) - self.sh = sh + def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None: + """Set the DTS execution stage and optionally log to files. - # prepare the output folder - if not os.path.exists(SETTINGS.output_dir): - os.mkdir(SETTINGS.output_dir) + Set the DTS execution stage of the DTSLog class and optionally add + file handlers to the instance if the log file name is provided. - logging_path_prefix = os.path.join(SETTINGS.output_dir, node) + The file handlers log all messages. One is a regular human-readable log file and + the other one is a machine-readable log file with extra debug information. - fh = logging.FileHandler(f"{logging_path_prefix}.log") - fh.setFormatter( - logging.Formatter( - fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s", - datefmt=date_fmt, - ) - ) + Args: + stage: The DTS stage to set. + log_file_path: An optional path of the log file to use. This should be a full path + (either relative or absolute) without suffix (which will be appended). + """ + self._remove_extra_file_handlers() + + if DTSLogger._stage != stage: + self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.") + DTSLogger._stage = stage + + if log_file_path: + self._extra_file_handlers.extend(self._add_file_handlers(log_file_path)) + + def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]: + """Add file handlers to the DTS root logger. + + Add two type of file handlers: + + * A regular file handler with suffix ".log", + * A machine-readable file handler with suffix ".verbose.log". + This format provides extensive information for debugging and detailed analysis. + + Args: + log_file_path: The full path to the log file without suffix. + + Returns: + The newly created file handlers. - self._logger.addHandler(fh) - self.fh = fh + """ + fh = FileHandler(f"{log_file_path}.log") + fh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) + self.addHandler(fh) - # This outputs EVERYTHING, intended for post-mortem debugging - # Also optimized for processing via AWK (awk -F '|' ...) - verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log") + verbose_fh = FileHandler(f"{log_file_path}.verbose.log") verbose_fh.setFormatter( logging.Formatter( - fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" + "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s", datefmt=date_fmt, ) ) + self.addHandler(verbose_fh) - self._logger.addHandler(verbose_fh) - self.verbose_fh = verbose_fh - - super(DTSLOG, self).__init__(self._logger, dict(node=self.node)) - - def logger_exit(self) -> None: - """Remove the stream handler and the logfile handler.""" - for handler in (self.sh, self.fh, self.verbose_fh): - handler.flush() - self._logger.removeHandler(handler) - - -class _LoggerDictType(TypedDict): - logger: DTSLOG - name: str - node: str - + return [fh, verbose_fh] -# List for saving all loggers in use -_Loggers: list[_LoggerDictType] = [] + def _remove_extra_file_handlers(self) -> None: + """Remove any extra file handlers that have been added to the logger.""" + if self._extra_file_handlers: + for extra_file_handler in self._extra_file_handlers: + self.removeHandler(extra_file_handler) + self._extra_file_handlers = [] -def getLogger(name: str, node: str = "suite") -> DTSLOG: - """Get DTS logger adapter identified by name and node. - An existing logger will be returned if one with the exact name and node already exists. - A new one will be created and stored otherwise. +def get_dts_logger(name: str = None) -> DTSLogger: + """Return a DTS logger instance identified by `name`. Args: - name: The name of the logger. - node: An additional identifier for the logger. + name: If :data:`None`, return the DTS root logger. + If specified, return a child of the DTS root logger. Returns: - A logger uniquely identified by both name and node. + The DTS root logger or a child logger identified by `name`. """ - global _Loggers - # return saved logger - logger: _LoggerDictType - for logger in _Loggers: - if logger["name"] == name and logger["node"] == node: - return logger["logger"] - - # return new logger - dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node) - _Loggers.append({"logger": dts_logger, "name": name, "node": node}) - return dts_logger + logging.setLoggerClass(DTSLogger) + if name: + name = f"{dts_root_logger_name}.{name}" + else: + name = dts_root_logger_name + logger = logging.getLogger(name) + logging.setLoggerClass(logging.Logger) + return logger # type: ignore[return-value] diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py index 51a01d6b5e..1910c81c3c 100644 --- a/dts/framework/remote_session/__init__.py +++ b/dts/framework/remote_session/__init__.py @@ -15,7 +15,7 @@ # pylama:ignore=W0611 from framework.config import NodeConfiguration -from framework.logger import DTSLOG +from framework.logger import DTSLogger from .interactive_remote_session import InteractiveRemoteSession from .interactive_shell import InteractiveShell @@ -26,7 +26,7 @@ def create_remote_session( - node_config: NodeConfiguration, name: str, logger: DTSLOG + node_config: NodeConfiguration, name: str, logger: DTSLogger ) -> RemoteSession: """Factory for non-interactive remote sessions. @@ -45,7 +45,7 @@ def create_remote_session( def create_interactive_session( - node_config: NodeConfiguration, logger: DTSLOG + node_config: NodeConfiguration, logger: DTSLogger ) -> InteractiveRemoteSession: """Factory for interactive remote sessions. diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py index 1cc82e3377..c50790db79 100644 --- a/dts/framework/remote_session/interactive_remote_session.py +++ b/dts/framework/remote_session/interactive_remote_session.py @@ -16,7 +16,7 @@ from framework.config import NodeConfiguration from framework.exception import SSHConnectionError -from framework.logger import DTSLOG +from framework.logger import DTSLogger class InteractiveRemoteSession: @@ -50,11 +50,11 @@ class InteractiveRemoteSession: username: str password: str session: SSHClient - _logger: DTSLOG + _logger: DTSLogger _node_config: NodeConfiguration _transport: Transport | None - def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None: + def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None: """Connect to the node during initialization. Args: diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py index b158f963b6..5cfe202e15 100644 --- a/dts/framework/remote_session/interactive_shell.py +++ b/dts/framework/remote_session/interactive_shell.py @@ -20,7 +20,7 @@ from paramiko import Channel, SSHClient, channel # type: ignore[import] -from framework.logger import DTSLOG +from framework.logger import DTSLogger from framework.settings import SETTINGS @@ -38,7 +38,7 @@ class InteractiveShell(ABC): _stdin: channel.ChannelStdinFile _stdout: channel.ChannelFile _ssh_channel: Channel - _logger: DTSLOG + _logger: DTSLogger _timeout: float _app_args: str @@ -61,7 +61,7 @@ class InteractiveShell(ABC): def __init__( self, interactive_session: SSHClient, - logger: DTSLOG, + logger: DTSLogger, get_privileged_command: Callable[[str], str] | None, app_args: str = "", timeout: float = SETTINGS.timeout, diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py index 2059f9a981..a69dc99400 100644 --- a/dts/framework/remote_session/remote_session.py +++ b/dts/framework/remote_session/remote_session.py @@ -9,14 +9,13 @@ the structure of the result of a command execution. """ - import dataclasses from abc import ABC, abstractmethod from pathlib import PurePath from framework.config import NodeConfiguration from framework.exception import RemoteCommandExecutionError -from framework.logger import DTSLOG +from framework.logger import DTSLogger from framework.settings import SETTINGS @@ -75,14 +74,14 @@ class RemoteSession(ABC): username: str password: str history: list[CommandResult] - _logger: DTSLOG + _logger: DTSLogger _node_config: NodeConfiguration def __init__( self, node_config: NodeConfiguration, session_name: str, - logger: DTSLOG, + logger: DTSLogger, ): """Connect to the node during initialization. @@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None: Args: force: Force the closure of the connection. This may not clean up all resources. """ - self._logger.logger_exit() self._close(force) @abstractmethod diff --git a/dts/framework/runner.py b/dts/framework/runner.py index f58b0adc13..035e3368ef 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -19,9 +19,10 @@ import importlib import inspect -import logging +import os import re import sys +from pathlib import Path from types import MethodType from typing import Iterable @@ -38,7 +39,7 @@ SSHTimeoutError, TestCaseVerifyError, ) -from .logger import DTSLOG, getLogger +from .logger import DTSLogger, DtsStage, get_dts_logger from .settings import SETTINGS from .test_result import ( BuildTargetResult, @@ -73,7 +74,7 @@ class DTSRunner: """ _configuration: Configuration - _logger: DTSLOG + _logger: DTSLogger _result: DTSResult _test_suite_class_prefix: str _test_suite_module_prefix: str @@ -83,7 +84,10 @@ class DTSRunner: def __init__(self): """Initialize the instance with configuration, logger, result and string constants.""" self._configuration = load_config() - self._logger = getLogger("DTSRunner") + self._logger = get_dts_logger() + if not os.path.exists(SETTINGS.output_dir): + os.makedirs(SETTINGS.output_dir) + self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir) self._result = DTSResult(self._logger) self._test_suite_class_prefix = "Test" self._test_suite_module_prefix = "tests.TestSuite_" @@ -137,6 +141,7 @@ def run(self): # for all Execution sections for execution in self._configuration.executions: + self._logger.set_stage(DtsStage.execution) self._logger.info( f"Running execution with SUT '{execution.system_under_test_node.name}'." ) @@ -164,6 +169,7 @@ def run(self): finally: try: + self._logger.set_stage(DtsStage.post_execution) for node in (sut_nodes | tg_nodes).values(): node.close() self._result.update_teardown(Result.PASS) @@ -419,6 +425,7 @@ def _run_execution( finally: try: + self._logger.set_stage(DtsStage.execution) sut_node.tear_down_execution() execution_result.update_teardown(Result.PASS) except Exception as e: @@ -447,6 +454,7 @@ def _run_build_target( with the current build target. test_suites_with_cases: The test suites with test cases to run. """ + self._logger.set_stage(DtsStage.build_target) self._logger.info(f"Running build target '{build_target.name}'.") try: @@ -463,6 +471,7 @@ def _run_build_target( finally: try: + self._logger.set_stage(DtsStage.build_target) sut_node.tear_down_build_target() build_target_result.update_teardown(Result.PASS) except Exception as e: @@ -535,6 +544,7 @@ def _run_test_suite( BlockingTestSuiteError: If a blocking test suite fails. """ test_suite_name = test_suite_with_cases.test_suite_class.__name__ + self._logger.set_stage(DtsStage.suite, Path(SETTINGS.output_dir, test_suite_name)) test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) try: self._logger.info(f"Starting test suite setup: {test_suite_name}") @@ -683,5 +693,4 @@ def _exit_dts(self) -> None: if self._logger: self._logger.info("DTS execution has ended.") - logging.shutdown() sys.exit(self._result.get_return_code()) diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index eedb2d20ee..28f84fd793 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -42,7 +42,7 @@ TestSuiteConfig, ) from .exception import DTSError, ErrorSeverity -from .logger import DTSLOG +from .logger import DTSLogger from .settings import SETTINGS from .test_suite import TestSuite @@ -237,13 +237,13 @@ class DTSResult(BaseResult): """ dpdk_version: str | None - _logger: DTSLOG + _logger: DTSLogger _errors: list[Exception] _return_code: ErrorSeverity _stats_result: Union["Statistics", None] _stats_filename: str - def __init__(self, logger: DTSLOG): + def __init__(self, logger: DTSLogger): """Extend the constructor with top-level specifics. Args: diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index f9fe88093e..365f80e21a 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -21,7 +21,7 @@ from scapy.packet import Packet, Padding # type: ignore[import] from .exception import TestCaseVerifyError -from .logger import DTSLOG, getLogger +from .logger import DTSLogger, get_dts_logger from .testbed_model import Port, PortLink, SutNode, TGNode from .utils import get_packet_summaries @@ -61,7 +61,7 @@ class TestSuite(object): #: Whether the test suite is blocking. A failure of a blocking test suite #: will block the execution of all subsequent test suites in the current build target. is_blocking: ClassVar[bool] = False - _logger: DTSLOG + _logger: DTSLogger _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -88,7 +88,7 @@ def __init__( """ self.sut_node = sut_node self.tg_node = tg_node - self._logger = getLogger(self.__class__.__name__) + self._logger = get_dts_logger(self.__class__.__name__) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py index 1a55fadf78..74061f6262 100644 --- a/dts/framework/testbed_model/node.py +++ b/dts/framework/testbed_model/node.py @@ -23,7 +23,7 @@ NodeConfiguration, ) from framework.exception import ConfigurationError -from framework.logger import DTSLOG, getLogger +from framework.logger import DTSLogger, get_dts_logger from framework.settings import SETTINGS from .cpu import ( @@ -63,7 +63,7 @@ class Node(ABC): name: str lcores: list[LogicalCore] ports: list[Port] - _logger: DTSLOG + _logger: DTSLogger _other_sessions: list[OSSession] _execution_config: ExecutionConfiguration virtual_devices: list[VirtualDevice] @@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration): """ self.config = node_config self.name = node_config.name - self._logger = getLogger(self.name) + self._logger = get_dts_logger(self.name) self.main_session = create_session(self.config, self.name, self._logger) self._logger.info(f"Connected to node: {self.name}") @@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession: connection = create_session( self.config, session_name, - getLogger(session_name, node=self.name), + get_dts_logger(session_name), ) self._other_sessions.append(connection) return connection @@ -299,7 +299,6 @@ def close(self) -> None: self.main_session.close() for session in self._other_sessions: session.close() - self._logger.logger_exit() @staticmethod def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: @@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: return func -def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession: +def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession: """Factory for OS-aware sessions. Args: diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py index ac6bb5e112..6983aa4a77 100644 --- a/dts/framework/testbed_model/os_session.py +++ b/dts/framework/testbed_model/os_session.py @@ -21,7 +21,6 @@ the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux and other commands for other OSs. It also translates the path to match the underlying OS. """ - from abc import ABC, abstractmethod from collections.abc import Iterable from ipaddress import IPv4Interface, IPv6Interface @@ -29,7 +28,7 @@ from typing import Type, TypeVar, Union from framework.config import Architecture, NodeConfiguration, NodeInfo -from framework.logger import DTSLOG +from framework.logger import DTSLogger from framework.remote_session import ( CommandResult, InteractiveRemoteSession, @@ -62,7 +61,7 @@ class OSSession(ABC): _config: NodeConfiguration name: str - _logger: DTSLOG + _logger: DTSLogger remote_session: RemoteSession interactive_session: InteractiveRemoteSession @@ -70,7 +69,7 @@ def __init__( self, node_config: NodeConfiguration, name: str, - logger: DTSLOG, + logger: DTSLogger, ): """Initialize the OS-aware session. diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py index c49fbff488..d86d7fb532 100644 --- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py +++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py @@ -13,7 +13,7 @@ from scapy.packet import Packet # type: ignore[import] from framework.config import TrafficGeneratorConfig -from framework.logger import DTSLOG, getLogger +from framework.logger import DTSLogger, get_dts_logger from framework.testbed_model.node import Node from framework.testbed_model.port import Port from framework.utils import get_packet_summaries @@ -28,7 +28,7 @@ class TrafficGenerator(ABC): _config: TrafficGeneratorConfig _tg_node: Node - _logger: DTSLOG + _logger: DTSLogger def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): """Initialize the traffic generator. @@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): """ self._config = config self._tg_node = tg_node - self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}") + self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}") def send_packet(self, packet: Packet, port: Port) -> None: """Send `packet` and block until it is fully sent. diff --git a/dts/main.py b/dts/main.py index 1ffe8ff81f..d30c164b95 100755 --- a/dts/main.py +++ b/dts/main.py @@ -30,5 +30,4 @@ def main() -> None: # Main program begins here if __name__ == "__main__": - logging.raiseExceptions = True main() -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 6/7] dts: refactor logging configuration 2024-02-06 14:57 ` [PATCH v2 6/7] dts: refactor logging configuration Juraj Linkeš @ 2024-02-12 16:45 ` Jeremy Spewock 2024-02-14 7:49 ` Juraj Linkeš 0 siblings, 1 reply; 44+ messages in thread From: Jeremy Spewock @ 2024-02-12 16:45 UTC (permalink / raw) To: Juraj Linkeš Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev On Tue, Feb 6, 2024 at 9:57 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote: > > Remove unused parts of the code and add useful features: > 1. Add DTS execution stages such as execution and test suite to better > identify where in the DTS lifecycle we are when investigating logs, > 2. Logging to separate files in specific stages, which is mainly useful > for having test suite logs in additional separate files. > 3. Remove the dependence on the settings module which enhances the > usefulness of the logger module, as it can now be imported in more > modules. > > The execution stages and the files to log to are the same for all DTS > loggers. To achieve this, we have one DTS root logger which should be > used for handling stage switching and all other loggers are children of > this DTS root logger. The DTS root logger is the one where we change the > behavior of all loggers (the stage and which files to log to) and the > child loggers just log messages under a different name. > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> > --- > dts/framework/logger.py | 235 +++++++++++------- > dts/framework/remote_session/__init__.py | 6 +- > .../interactive_remote_session.py | 6 +- > .../remote_session/interactive_shell.py | 6 +- > .../remote_session/remote_session.py | 8 +- > dts/framework/runner.py | 19 +- > dts/framework/test_result.py | 6 +- > dts/framework/test_suite.py | 6 +- > dts/framework/testbed_model/node.py | 11 +- > dts/framework/testbed_model/os_session.py | 7 +- > .../traffic_generator/traffic_generator.py | 6 +- > dts/main.py | 1 - > 12 files changed, 183 insertions(+), 134 deletions(-) > > diff --git a/dts/framework/logger.py b/dts/framework/logger.py > index cfa6e8cd72..568edad82d 100644 > --- a/dts/framework/logger.py > +++ b/dts/framework/logger.py > @@ -5,141 +5,186 @@ > > """DTS logger module. > > -DTS framework and TestSuite logs are saved in different log files. > +The module provides several additional features: > + > + * The storage of DTS execution stages, > + * Logging to console, a human-readable log file and a machine-readable log file, > + * Optional log files for specific stages. > """ > > import logging > -import os.path > -from typing import TypedDict > +from enum import auto > +from logging import FileHandler, StreamHandler > +from pathlib import Path > +from typing import ClassVar > > -from .settings import SETTINGS > +from .utils import StrEnum > > date_fmt = "%Y/%m/%d %H:%M:%S" > -stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" > +stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s" > +dts_root_logger_name = "dts" > + > + > +class DtsStage(StrEnum): > + """The DTS execution stage.""" > > + #: > + pre_execution = auto() > + #: > + execution = auto() > + #: > + build_target = auto() > + #: > + suite = auto() > + #: > + post_execution = auto() > > -class DTSLOG(logging.LoggerAdapter): > - """DTS logger adapter class for framework and testsuites. > > - The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment > - variable control the verbosity of output. If enabled, all messages will be emitted to the > - console. > +class DTSLogger(logging.Logger): > + """The DTS logger class. > > - The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment > - variable modify the directory where the logs will be stored. > + The class extends the :class:`~logging.Logger` class to add the DTS execution stage information > + to log records. The stage is common to all loggers, so it's stored in a class variable. > > - Attributes: > - node: The additional identifier. Currently unused. > - sh: The handler which emits logs to console. > - fh: The handler which emits logs to a file. > - verbose_fh: Just as fh, but logs with a different, more verbose, format. > + Any time we switch to a new stage, we have the ability to log to an additional log file along > + with a supplementary log file with machine-readable format. These two log files are used until > + a new stage switch occurs. This is useful mainly for logging per test suite. > """ > > - _logger: logging.Logger > - node: str > - sh: logging.StreamHandler > - fh: logging.FileHandler > - verbose_fh: logging.FileHandler > + _stage: ClassVar[DtsStage] = DtsStage.pre_execution > + _extra_file_handlers: list[FileHandler] = [] > > - def __init__(self, logger: logging.Logger, node: str = "suite"): > - """Extend the constructor with additional handlers. > + def __init__(self, *args, **kwargs): > + """Extend the constructor with extra file handlers.""" > + self._extra_file_handlers = [] > + super().__init__(*args, **kwargs) > > - One handler logs to the console, the other one to a file, with either a regular or verbose > - format. > + def makeRecord(self, *args, **kwargs): Is the return type annotation here skipped because of the `:meta private:`? > + """Generates a record with additional stage information. > > - Args: > - logger: The logger from which to create the logger adapter. > - node: An additional identifier. Currently unused. > + This is the default method for the :class:`~logging.Logger` class. We extend it > + to add stage information to the record. > + > + :meta private: > + > + Returns: > + record: The generated record with the stage information. > """ > - self._logger = logger > - # 1 means log everything, this will be used by file handlers if their level > - # is not set > - self._logger.setLevel(1) > + record = super().makeRecord(*args, **kwargs) > + record.stage = DTSLogger._stage > + return record > + > + def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None: > + """Add logger handlers to the DTS root logger. > + > + This method should be called only on the DTS root logger. > + The log records from child loggers will propagate to these handlers. > + > + Three handlers are added: > > - self.node = node > + * A console handler, > + * A file handler, > + * A supplementary file handler with machine-readable logs > + containing more debug information. > > - # add handler to emit to stdout > - sh = logging.StreamHandler() > + All log messages will be logged to files. The log level of the console handler > + is configurable with `verbose`. > + > + Args: > + verbose: If :data:`True`, log all messages to the console. > + If :data:`False`, log to console with the :data:`logging.INFO` level. > + output_dir: The directory where the log files will be located. > + The names of the log files correspond to the name of the logger instance. > + """ > + self.setLevel(1) > + > + sh = StreamHandler() > sh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) > - sh.setLevel(logging.INFO) # console handler default level > + if not verbose: > + sh.setLevel(logging.INFO) > + self.addHandler(sh) > > - if SETTINGS.verbose is True: > - sh.setLevel(logging.DEBUG) > + self._add_file_handlers(Path(output_dir, self.name)) > > - self._logger.addHandler(sh) > - self.sh = sh > + def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None: > + """Set the DTS execution stage and optionally log to files. > > - # prepare the output folder > - if not os.path.exists(SETTINGS.output_dir): > - os.mkdir(SETTINGS.output_dir) > + Set the DTS execution stage of the DTSLog class and optionally add > + file handlers to the instance if the log file name is provided. > > - logging_path_prefix = os.path.join(SETTINGS.output_dir, node) > + The file handlers log all messages. One is a regular human-readable log file and > + the other one is a machine-readable log file with extra debug information. > > - fh = logging.FileHandler(f"{logging_path_prefix}.log") > - fh.setFormatter( > - logging.Formatter( > - fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s", > - datefmt=date_fmt, > - ) > - ) > + Args: > + stage: The DTS stage to set. > + log_file_path: An optional path of the log file to use. This should be a full path > + (either relative or absolute) without suffix (which will be appended). > + """ > + self._remove_extra_file_handlers() > + > + if DTSLogger._stage != stage: > + self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.") > + DTSLogger._stage = stage > + > + if log_file_path: > + self._extra_file_handlers.extend(self._add_file_handlers(log_file_path)) > + > + def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]: > + """Add file handlers to the DTS root logger. > + > + Add two type of file handlers: > + > + * A regular file handler with suffix ".log", > + * A machine-readable file handler with suffix ".verbose.log". > + This format provides extensive information for debugging and detailed analysis. > + > + Args: > + log_file_path: The full path to the log file without suffix. > + > + Returns: > + The newly created file handlers. > > - self._logger.addHandler(fh) > - self.fh = fh > + """ > + fh = FileHandler(f"{log_file_path}.log") > + fh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) > + self.addHandler(fh) > > - # This outputs EVERYTHING, intended for post-mortem debugging > - # Also optimized for processing via AWK (awk -F '|' ...) > - verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log") > + verbose_fh = FileHandler(f"{log_file_path}.verbose.log") > verbose_fh.setFormatter( > logging.Formatter( > - fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" > + "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" > "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s", > datefmt=date_fmt, > ) > ) > + self.addHandler(verbose_fh) > > - self._logger.addHandler(verbose_fh) > - self.verbose_fh = verbose_fh > - > - super(DTSLOG, self).__init__(self._logger, dict(node=self.node)) > - > - def logger_exit(self) -> None: > - """Remove the stream handler and the logfile handler.""" > - for handler in (self.sh, self.fh, self.verbose_fh): > - handler.flush() > - self._logger.removeHandler(handler) > - > - > -class _LoggerDictType(TypedDict): > - logger: DTSLOG > - name: str > - node: str > - > + return [fh, verbose_fh] > > -# List for saving all loggers in use > -_Loggers: list[_LoggerDictType] = [] > + def _remove_extra_file_handlers(self) -> None: > + """Remove any extra file handlers that have been added to the logger.""" > + if self._extra_file_handlers: > + for extra_file_handler in self._extra_file_handlers: > + self.removeHandler(extra_file_handler) > > + self._extra_file_handlers = [] > > -def getLogger(name: str, node: str = "suite") -> DTSLOG: > - """Get DTS logger adapter identified by name and node. > > - An existing logger will be returned if one with the exact name and node already exists. > - A new one will be created and stored otherwise. > +def get_dts_logger(name: str = None) -> DTSLogger: > + """Return a DTS logger instance identified by `name`. > > Args: > - name: The name of the logger. > - node: An additional identifier for the logger. > + name: If :data:`None`, return the DTS root logger. > + If specified, return a child of the DTS root logger. > > Returns: > - A logger uniquely identified by both name and node. > + The DTS root logger or a child logger identified by `name`. > """ > - global _Loggers > - # return saved logger > - logger: _LoggerDictType > - for logger in _Loggers: > - if logger["name"] == name and logger["node"] == node: > - return logger["logger"] > - > - # return new logger > - dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node) > - _Loggers.append({"logger": dts_logger, "name": name, "node": node}) > - return dts_logger > + logging.setLoggerClass(DTSLogger) > + if name: > + name = f"{dts_root_logger_name}.{name}" > + else: > + name = dts_root_logger_name > + logger = logging.getLogger(name) > + logging.setLoggerClass(logging.Logger) What's the benefit of setting the logger class back to logging.Logger here? Is the idea basically that if someone wanted to use the logging module we shouldn't implicitly make them use our DTSLogger? > + return logger # type: ignore[return-value] > diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py > index 51a01d6b5e..1910c81c3c 100644 > --- a/dts/framework/remote_session/__init__.py > +++ b/dts/framework/remote_session/__init__.py > @@ -15,7 +15,7 @@ > # pylama:ignore=W0611 > > from framework.config import NodeConfiguration > -from framework.logger import DTSLOG > +from framework.logger import DTSLogger > > from .interactive_remote_session import InteractiveRemoteSession > from .interactive_shell import InteractiveShell > @@ -26,7 +26,7 @@ > > > def create_remote_session( > - node_config: NodeConfiguration, name: str, logger: DTSLOG > + node_config: NodeConfiguration, name: str, logger: DTSLogger > ) -> RemoteSession: > """Factory for non-interactive remote sessions. > > @@ -45,7 +45,7 @@ def create_remote_session( > > > def create_interactive_session( > - node_config: NodeConfiguration, logger: DTSLOG > + node_config: NodeConfiguration, logger: DTSLogger > ) -> InteractiveRemoteSession: > """Factory for interactive remote sessions. > > diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py > index 1cc82e3377..c50790db79 100644 > --- a/dts/framework/remote_session/interactive_remote_session.py > +++ b/dts/framework/remote_session/interactive_remote_session.py > @@ -16,7 +16,7 @@ > > from framework.config import NodeConfiguration > from framework.exception import SSHConnectionError > -from framework.logger import DTSLOG > +from framework.logger import DTSLogger > > > class InteractiveRemoteSession: > @@ -50,11 +50,11 @@ class InteractiveRemoteSession: > username: str > password: str > session: SSHClient > - _logger: DTSLOG > + _logger: DTSLogger > _node_config: NodeConfiguration > _transport: Transport | None > > - def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None: > + def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None: > """Connect to the node during initialization. > > Args: > diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py > index b158f963b6..5cfe202e15 100644 > --- a/dts/framework/remote_session/interactive_shell.py > +++ b/dts/framework/remote_session/interactive_shell.py > @@ -20,7 +20,7 @@ > > from paramiko import Channel, SSHClient, channel # type: ignore[import] > > -from framework.logger import DTSLOG > +from framework.logger import DTSLogger > from framework.settings import SETTINGS > > > @@ -38,7 +38,7 @@ class InteractiveShell(ABC): > _stdin: channel.ChannelStdinFile > _stdout: channel.ChannelFile > _ssh_channel: Channel > - _logger: DTSLOG > + _logger: DTSLogger > _timeout: float > _app_args: str > > @@ -61,7 +61,7 @@ class InteractiveShell(ABC): > def __init__( > self, > interactive_session: SSHClient, > - logger: DTSLOG, > + logger: DTSLogger, > get_privileged_command: Callable[[str], str] | None, > app_args: str = "", > timeout: float = SETTINGS.timeout, > diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py > index 2059f9a981..a69dc99400 100644 > --- a/dts/framework/remote_session/remote_session.py > +++ b/dts/framework/remote_session/remote_session.py > @@ -9,14 +9,13 @@ > the structure of the result of a command execution. > """ > > - > import dataclasses > from abc import ABC, abstractmethod > from pathlib import PurePath > > from framework.config import NodeConfiguration > from framework.exception import RemoteCommandExecutionError > -from framework.logger import DTSLOG > +from framework.logger import DTSLogger > from framework.settings import SETTINGS > > > @@ -75,14 +74,14 @@ class RemoteSession(ABC): > username: str > password: str > history: list[CommandResult] > - _logger: DTSLOG > + _logger: DTSLogger > _node_config: NodeConfiguration > > def __init__( > self, > node_config: NodeConfiguration, > session_name: str, > - logger: DTSLOG, > + logger: DTSLogger, > ): > """Connect to the node during initialization. > > @@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None: > Args: > force: Force the closure of the connection. This may not clean up all resources. > """ > - self._logger.logger_exit() > self._close(force) > > @abstractmethod > diff --git a/dts/framework/runner.py b/dts/framework/runner.py > index f58b0adc13..035e3368ef 100644 > --- a/dts/framework/runner.py > +++ b/dts/framework/runner.py > @@ -19,9 +19,10 @@ > > import importlib > import inspect > -import logging > +import os > import re > import sys > +from pathlib import Path > from types import MethodType > from typing import Iterable > > @@ -38,7 +39,7 @@ > SSHTimeoutError, > TestCaseVerifyError, > ) > -from .logger import DTSLOG, getLogger > +from .logger import DTSLogger, DtsStage, get_dts_logger > from .settings import SETTINGS > from .test_result import ( > BuildTargetResult, > @@ -73,7 +74,7 @@ class DTSRunner: > """ > > _configuration: Configuration > - _logger: DTSLOG > + _logger: DTSLogger > _result: DTSResult > _test_suite_class_prefix: str > _test_suite_module_prefix: str > @@ -83,7 +84,10 @@ class DTSRunner: > def __init__(self): > """Initialize the instance with configuration, logger, result and string constants.""" > self._configuration = load_config() > - self._logger = getLogger("DTSRunner") > + self._logger = get_dts_logger() > + if not os.path.exists(SETTINGS.output_dir): > + os.makedirs(SETTINGS.output_dir) > + self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir) > self._result = DTSResult(self._logger) > self._test_suite_class_prefix = "Test" > self._test_suite_module_prefix = "tests.TestSuite_" > @@ -137,6 +141,7 @@ def run(self): > > # for all Execution sections > for execution in self._configuration.executions: > + self._logger.set_stage(DtsStage.execution) This ends up getting set twice in short succession which of course doesn't functionally cause a problem, but I don't exactly see the point of setting it twice. We could either set it here or set it in the _run_execution, but i think it makes more sense to set it in the _run_execution method as that is the method where we are doing the setup, here we are only initializing the nodes which is still in a sense "pre execution." > self._logger.info( > f"Running execution with SUT '{execution.system_under_test_node.name}'." > ) > @@ -164,6 +169,7 @@ def run(self): > > finally: > try: > + self._logger.set_stage(DtsStage.post_execution) > for node in (sut_nodes | tg_nodes).values(): > node.close() > self._result.update_teardown(Result.PASS) > @@ -419,6 +425,7 @@ def _run_execution( > > finally: > try: > + self._logger.set_stage(DtsStage.execution) > sut_node.tear_down_execution() > execution_result.update_teardown(Result.PASS) > except Exception as e: > @@ -447,6 +454,7 @@ def _run_build_target( > with the current build target. > test_suites_with_cases: The test suites with test cases to run. > """ > + self._logger.set_stage(DtsStage.build_target) > self._logger.info(f"Running build target '{build_target.name}'.") > > try: > @@ -463,6 +471,7 @@ def _run_build_target( > > finally: > try: > + self._logger.set_stage(DtsStage.build_target) > sut_node.tear_down_build_target() > build_target_result.update_teardown(Result.PASS) > except Exception as e: > @@ -535,6 +544,7 @@ def _run_test_suite( > BlockingTestSuiteError: If a blocking test suite fails. > """ > test_suite_name = test_suite_with_cases.test_suite_class.__name__ > + self._logger.set_stage(DtsStage.suite, Path(SETTINGS.output_dir, test_suite_name)) > test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) > try: > self._logger.info(f"Starting test suite setup: {test_suite_name}") > @@ -683,5 +693,4 @@ def _exit_dts(self) -> None: > if self._logger: > self._logger.info("DTS execution has ended.") > > - logging.shutdown() > sys.exit(self._result.get_return_code()) > diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py > index eedb2d20ee..28f84fd793 100644 > --- a/dts/framework/test_result.py > +++ b/dts/framework/test_result.py > @@ -42,7 +42,7 @@ > TestSuiteConfig, > ) > from .exception import DTSError, ErrorSeverity > -from .logger import DTSLOG > +from .logger import DTSLogger > from .settings import SETTINGS > from .test_suite import TestSuite > > @@ -237,13 +237,13 @@ class DTSResult(BaseResult): > """ > > dpdk_version: str | None > - _logger: DTSLOG > + _logger: DTSLogger > _errors: list[Exception] > _return_code: ErrorSeverity > _stats_result: Union["Statistics", None] > _stats_filename: str > > - def __init__(self, logger: DTSLOG): > + def __init__(self, logger: DTSLogger): > """Extend the constructor with top-level specifics. > > Args: > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py > index f9fe88093e..365f80e21a 100644 > --- a/dts/framework/test_suite.py > +++ b/dts/framework/test_suite.py > @@ -21,7 +21,7 @@ > from scapy.packet import Packet, Padding # type: ignore[import] > > from .exception import TestCaseVerifyError > -from .logger import DTSLOG, getLogger > +from .logger import DTSLogger, get_dts_logger > from .testbed_model import Port, PortLink, SutNode, TGNode > from .utils import get_packet_summaries > > @@ -61,7 +61,7 @@ class TestSuite(object): > #: Whether the test suite is blocking. A failure of a blocking test suite > #: will block the execution of all subsequent test suites in the current build target. > is_blocking: ClassVar[bool] = False > - _logger: DTSLOG > + _logger: DTSLogger > _port_links: list[PortLink] > _sut_port_ingress: Port > _sut_port_egress: Port > @@ -88,7 +88,7 @@ def __init__( > """ > self.sut_node = sut_node > self.tg_node = tg_node > - self._logger = getLogger(self.__class__.__name__) > + self._logger = get_dts_logger(self.__class__.__name__) > self._port_links = [] > self._process_links() > self._sut_port_ingress, self._tg_port_egress = ( > diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py > index 1a55fadf78..74061f6262 100644 > --- a/dts/framework/testbed_model/node.py > +++ b/dts/framework/testbed_model/node.py > @@ -23,7 +23,7 @@ > NodeConfiguration, > ) > from framework.exception import ConfigurationError > -from framework.logger import DTSLOG, getLogger > +from framework.logger import DTSLogger, get_dts_logger > from framework.settings import SETTINGS > > from .cpu import ( > @@ -63,7 +63,7 @@ class Node(ABC): > name: str > lcores: list[LogicalCore] > ports: list[Port] > - _logger: DTSLOG > + _logger: DTSLogger > _other_sessions: list[OSSession] > _execution_config: ExecutionConfiguration > virtual_devices: list[VirtualDevice] > @@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration): > """ > self.config = node_config > self.name = node_config.name > - self._logger = getLogger(self.name) > + self._logger = get_dts_logger(self.name) > self.main_session = create_session(self.config, self.name, self._logger) > > self._logger.info(f"Connected to node: {self.name}") > @@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession: > connection = create_session( > self.config, > session_name, > - getLogger(session_name, node=self.name), > + get_dts_logger(session_name), > ) > self._other_sessions.append(connection) > return connection > @@ -299,7 +299,6 @@ def close(self) -> None: > self.main_session.close() > for session in self._other_sessions: > session.close() > - self._logger.logger_exit() > > @staticmethod > def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: > @@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: > return func > > > -def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession: > +def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession: > """Factory for OS-aware sessions. > > Args: > diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py > index ac6bb5e112..6983aa4a77 100644 > --- a/dts/framework/testbed_model/os_session.py > +++ b/dts/framework/testbed_model/os_session.py > @@ -21,7 +21,6 @@ > the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux > and other commands for other OSs. It also translates the path to match the underlying OS. > """ > - > from abc import ABC, abstractmethod > from collections.abc import Iterable > from ipaddress import IPv4Interface, IPv6Interface > @@ -29,7 +28,7 @@ > from typing import Type, TypeVar, Union > > from framework.config import Architecture, NodeConfiguration, NodeInfo > -from framework.logger import DTSLOG > +from framework.logger import DTSLogger > from framework.remote_session import ( > CommandResult, > InteractiveRemoteSession, > @@ -62,7 +61,7 @@ class OSSession(ABC): > > _config: NodeConfiguration > name: str > - _logger: DTSLOG > + _logger: DTSLogger > remote_session: RemoteSession > interactive_session: InteractiveRemoteSession > > @@ -70,7 +69,7 @@ def __init__( > self, > node_config: NodeConfiguration, > name: str, > - logger: DTSLOG, > + logger: DTSLogger, > ): > """Initialize the OS-aware session. > > diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py > index c49fbff488..d86d7fb532 100644 > --- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py > +++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py > @@ -13,7 +13,7 @@ > from scapy.packet import Packet # type: ignore[import] > > from framework.config import TrafficGeneratorConfig > -from framework.logger import DTSLOG, getLogger > +from framework.logger import DTSLogger, get_dts_logger > from framework.testbed_model.node import Node > from framework.testbed_model.port import Port > from framework.utils import get_packet_summaries > @@ -28,7 +28,7 @@ class TrafficGenerator(ABC): > > _config: TrafficGeneratorConfig > _tg_node: Node > - _logger: DTSLOG > + _logger: DTSLogger > > def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): > """Initialize the traffic generator. > @@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): > """ > self._config = config > self._tg_node = tg_node > - self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}") > + self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}") > > def send_packet(self, packet: Packet, port: Port) -> None: > """Send `packet` and block until it is fully sent. > diff --git a/dts/main.py b/dts/main.py > index 1ffe8ff81f..d30c164b95 100755 > --- a/dts/main.py > +++ b/dts/main.py > @@ -30,5 +30,4 @@ def main() -> None: > > # Main program begins here > if __name__ == "__main__": > - logging.raiseExceptions = True > main() > -- > 2.34.1 > ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 6/7] dts: refactor logging configuration 2024-02-12 16:45 ` Jeremy Spewock @ 2024-02-14 7:49 ` Juraj Linkeš 2024-02-14 16:51 ` Jeremy Spewock 0 siblings, 1 reply; 44+ messages in thread From: Juraj Linkeš @ 2024-02-14 7:49 UTC (permalink / raw) To: Jeremy Spewock Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev Hi Jeremy, Just a reminder, please strip the parts you're not commenting on. On Mon, Feb 12, 2024 at 5:45 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote: > > On Tue, Feb 6, 2024 at 9:57 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote: > > > > Remove unused parts of the code and add useful features: > > 1. Add DTS execution stages such as execution and test suite to better > > identify where in the DTS lifecycle we are when investigating logs, > > 2. Logging to separate files in specific stages, which is mainly useful > > for having test suite logs in additional separate files. > > 3. Remove the dependence on the settings module which enhances the > > usefulness of the logger module, as it can now be imported in more > > modules. > > > > The execution stages and the files to log to are the same for all DTS > > loggers. To achieve this, we have one DTS root logger which should be > > used for handling stage switching and all other loggers are children of > > this DTS root logger. The DTS root logger is the one where we change the > > behavior of all loggers (the stage and which files to log to) and the > > child loggers just log messages under a different name. > > > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> > > --- > > dts/framework/logger.py | 235 +++++++++++------- > > dts/framework/remote_session/__init__.py | 6 +- > > .../interactive_remote_session.py | 6 +- > > .../remote_session/interactive_shell.py | 6 +- > > .../remote_session/remote_session.py | 8 +- > > dts/framework/runner.py | 19 +- > > dts/framework/test_result.py | 6 +- > > dts/framework/test_suite.py | 6 +- > > dts/framework/testbed_model/node.py | 11 +- > > dts/framework/testbed_model/os_session.py | 7 +- > > .../traffic_generator/traffic_generator.py | 6 +- > > dts/main.py | 1 - > > 12 files changed, 183 insertions(+), 134 deletions(-) > > > > diff --git a/dts/framework/logger.py b/dts/framework/logger.py > > index cfa6e8cd72..568edad82d 100644 > > --- a/dts/framework/logger.py > > +++ b/dts/framework/logger.py > > @@ -5,141 +5,186 @@ > > > > """DTS logger module. > > > > -DTS framework and TestSuite logs are saved in different log files. > > +The module provides several additional features: > > + > > + * The storage of DTS execution stages, > > + * Logging to console, a human-readable log file and a machine-readable log file, > > + * Optional log files for specific stages. > > """ > > > > import logging > > -import os.path > > -from typing import TypedDict > > +from enum import auto > > +from logging import FileHandler, StreamHandler > > +from pathlib import Path > > +from typing import ClassVar > > > > -from .settings import SETTINGS > > +from .utils import StrEnum > > > > date_fmt = "%Y/%m/%d %H:%M:%S" > > -stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" > > +stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s" > > +dts_root_logger_name = "dts" > > + > > + > > +class DtsStage(StrEnum): > > + """The DTS execution stage.""" > > > > + #: > > + pre_execution = auto() > > + #: > > + execution = auto() > > + #: > > + build_target = auto() > > + #: > > + suite = auto() > > + #: > > + post_execution = auto() > > > > -class DTSLOG(logging.LoggerAdapter): > > - """DTS logger adapter class for framework and testsuites. > > > > - The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment > > - variable control the verbosity of output. If enabled, all messages will be emitted to the > > - console. > > +class DTSLogger(logging.Logger): > > + """The DTS logger class. > > > > - The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment > > - variable modify the directory where the logs will be stored. > > + The class extends the :class:`~logging.Logger` class to add the DTS execution stage information > > + to log records. The stage is common to all loggers, so it's stored in a class variable. > > > > - Attributes: > > - node: The additional identifier. Currently unused. > > - sh: The handler which emits logs to console. > > - fh: The handler which emits logs to a file. > > - verbose_fh: Just as fh, but logs with a different, more verbose, format. > > + Any time we switch to a new stage, we have the ability to log to an additional log file along > > + with a supplementary log file with machine-readable format. These two log files are used until > > + a new stage switch occurs. This is useful mainly for logging per test suite. > > """ > > > > - _logger: logging.Logger > > - node: str > > - sh: logging.StreamHandler > > - fh: logging.FileHandler > > - verbose_fh: logging.FileHandler > > + _stage: ClassVar[DtsStage] = DtsStage.pre_execution > > + _extra_file_handlers: list[FileHandler] = [] > > > > - def __init__(self, logger: logging.Logger, node: str = "suite"): > > - """Extend the constructor with additional handlers. > > + def __init__(self, *args, **kwargs): > > + """Extend the constructor with extra file handlers.""" > > + self._extra_file_handlers = [] > > + super().__init__(*args, **kwargs) > > > > - One handler logs to the console, the other one to a file, with either a regular or verbose > > - format. > > + def makeRecord(self, *args, **kwargs): > > Is the return type annotation here skipped because of the `:meta private:`? > Basically. We're modifying a method defined elsewhere, but there's no harm in adding the return type. > > + """Generates a record with additional stage information. > > > > - Args: > > - logger: The logger from which to create the logger adapter. > > - node: An additional identifier. Currently unused. > > + This is the default method for the :class:`~logging.Logger` class. We extend it > > + to add stage information to the record. > > + > > + :meta private: > > + > > + Returns: > > + record: The generated record with the stage information. > > """ > > - self._logger = logger > > - # 1 means log everything, this will be used by file handlers if their level > > - # is not set > > - self._logger.setLevel(1) > > + record = super().makeRecord(*args, **kwargs) > > + record.stage = DTSLogger._stage > > + return record > > + > > + def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None: > > + """Add logger handlers to the DTS root logger. > > + > > + This method should be called only on the DTS root logger. > > + The log records from child loggers will propagate to these handlers. > > + > > + Three handlers are added: > > > > - self.node = node > > + * A console handler, > > + * A file handler, > > + * A supplementary file handler with machine-readable logs > > + containing more debug information. > > > > - # add handler to emit to stdout > > - sh = logging.StreamHandler() > > + All log messages will be logged to files. The log level of the console handler > > + is configurable with `verbose`. > > + > > + Args: > > + verbose: If :data:`True`, log all messages to the console. > > + If :data:`False`, log to console with the :data:`logging.INFO` level. > > + output_dir: The directory where the log files will be located. > > + The names of the log files correspond to the name of the logger instance. > > + """ > > + self.setLevel(1) > > + > > + sh = StreamHandler() > > sh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) > > - sh.setLevel(logging.INFO) # console handler default level > > + if not verbose: > > + sh.setLevel(logging.INFO) > > + self.addHandler(sh) > > > > - if SETTINGS.verbose is True: > > - sh.setLevel(logging.DEBUG) > > + self._add_file_handlers(Path(output_dir, self.name)) > > > > - self._logger.addHandler(sh) > > - self.sh = sh > > + def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None: > > + """Set the DTS execution stage and optionally log to files. > > > > - # prepare the output folder > > - if not os.path.exists(SETTINGS.output_dir): > > - os.mkdir(SETTINGS.output_dir) > > + Set the DTS execution stage of the DTSLog class and optionally add > > + file handlers to the instance if the log file name is provided. > > > > - logging_path_prefix = os.path.join(SETTINGS.output_dir, node) > > + The file handlers log all messages. One is a regular human-readable log file and > > + the other one is a machine-readable log file with extra debug information. > > > > - fh = logging.FileHandler(f"{logging_path_prefix}.log") > > - fh.setFormatter( > > - logging.Formatter( > > - fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s", > > - datefmt=date_fmt, > > - ) > > - ) > > + Args: > > + stage: The DTS stage to set. > > + log_file_path: An optional path of the log file to use. This should be a full path > > + (either relative or absolute) without suffix (which will be appended). > > + """ > > + self._remove_extra_file_handlers() > > + > > + if DTSLogger._stage != stage: > > + self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.") > > + DTSLogger._stage = stage > > + > > + if log_file_path: > > + self._extra_file_handlers.extend(self._add_file_handlers(log_file_path)) > > + > > + def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]: > > + """Add file handlers to the DTS root logger. > > + > > + Add two type of file handlers: > > + > > + * A regular file handler with suffix ".log", > > + * A machine-readable file handler with suffix ".verbose.log". > > + This format provides extensive information for debugging and detailed analysis. > > + > > + Args: > > + log_file_path: The full path to the log file without suffix. > > + > > + Returns: > > + The newly created file handlers. > > > > - self._logger.addHandler(fh) > > - self.fh = fh > > + """ > > + fh = FileHandler(f"{log_file_path}.log") > > + fh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) > > + self.addHandler(fh) > > > > - # This outputs EVERYTHING, intended for post-mortem debugging > > - # Also optimized for processing via AWK (awk -F '|' ...) > > - verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log") > > + verbose_fh = FileHandler(f"{log_file_path}.verbose.log") > > verbose_fh.setFormatter( > > logging.Formatter( > > - fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" > > + "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" > > "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s", > > datefmt=date_fmt, > > ) > > ) > > + self.addHandler(verbose_fh) > > > > - self._logger.addHandler(verbose_fh) > > - self.verbose_fh = verbose_fh > > - > > - super(DTSLOG, self).__init__(self._logger, dict(node=self.node)) > > - > > - def logger_exit(self) -> None: > > - """Remove the stream handler and the logfile handler.""" > > - for handler in (self.sh, self.fh, self.verbose_fh): > > - handler.flush() > > - self._logger.removeHandler(handler) > > - > > - > > -class _LoggerDictType(TypedDict): > > - logger: DTSLOG > > - name: str > > - node: str > > - > > + return [fh, verbose_fh] > > > > -# List for saving all loggers in use > > -_Loggers: list[_LoggerDictType] = [] > > + def _remove_extra_file_handlers(self) -> None: > > + """Remove any extra file handlers that have been added to the logger.""" > > + if self._extra_file_handlers: > > + for extra_file_handler in self._extra_file_handlers: > > + self.removeHandler(extra_file_handler) > > > > + self._extra_file_handlers = [] > > > > -def getLogger(name: str, node: str = "suite") -> DTSLOG: > > - """Get DTS logger adapter identified by name and node. > > > > - An existing logger will be returned if one with the exact name and node already exists. > > - A new one will be created and stored otherwise. > > +def get_dts_logger(name: str = None) -> DTSLogger: > > + """Return a DTS logger instance identified by `name`. > > > > Args: > > - name: The name of the logger. > > - node: An additional identifier for the logger. > > + name: If :data:`None`, return the DTS root logger. > > + If specified, return a child of the DTS root logger. > > > > Returns: > > - A logger uniquely identified by both name and node. > > + The DTS root logger or a child logger identified by `name`. > > """ > > - global _Loggers > > - # return saved logger > > - logger: _LoggerDictType > > - for logger in _Loggers: > > - if logger["name"] == name and logger["node"] == node: > > - return logger["logger"] > > - > > - # return new logger > > - dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node) > > - _Loggers.append({"logger": dts_logger, "name": name, "node": node}) > > - return dts_logger > > + logging.setLoggerClass(DTSLogger) > > + if name: > > + name = f"{dts_root_logger_name}.{name}" > > + else: > > + name = dts_root_logger_name > > + logger = logging.getLogger(name) > > + logging.setLoggerClass(logging.Logger) > > What's the benefit of setting the logger class back to logging.Logger > here? Is the idea basically that if someone wanted to use the logging > module we shouldn't implicitly make them use our DTSLogger? > Yes. We should actually set it to whatever was there before (it may not be logging.Logger), so I'll change that.+ > > + return logger # type: ignore[return-value] > > diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py > > index 51a01d6b5e..1910c81c3c 100644 > > --- a/dts/framework/remote_session/__init__.py > > +++ b/dts/framework/remote_session/__init__.py > > @@ -15,7 +15,7 @@ > > # pylama:ignore=W0611 > > > > from framework.config import NodeConfiguration > > -from framework.logger import DTSLOG > > +from framework.logger import DTSLogger > > > > from .interactive_remote_session import InteractiveRemoteSession > > from .interactive_shell import InteractiveShell > > @@ -26,7 +26,7 @@ > > > > > > def create_remote_session( > > - node_config: NodeConfiguration, name: str, logger: DTSLOG > > + node_config: NodeConfiguration, name: str, logger: DTSLogger > > ) -> RemoteSession: > > """Factory for non-interactive remote sessions. > > > > @@ -45,7 +45,7 @@ def create_remote_session( > > > > > > def create_interactive_session( > > - node_config: NodeConfiguration, logger: DTSLOG > > + node_config: NodeConfiguration, logger: DTSLogger > > ) -> InteractiveRemoteSession: > > """Factory for interactive remote sessions. > > > > diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py > > index 1cc82e3377..c50790db79 100644 > > --- a/dts/framework/remote_session/interactive_remote_session.py > > +++ b/dts/framework/remote_session/interactive_remote_session.py > > @@ -16,7 +16,7 @@ > > > > from framework.config import NodeConfiguration > > from framework.exception import SSHConnectionError > > -from framework.logger import DTSLOG > > +from framework.logger import DTSLogger > > > > > > class InteractiveRemoteSession: > > @@ -50,11 +50,11 @@ class InteractiveRemoteSession: > > username: str > > password: str > > session: SSHClient > > - _logger: DTSLOG > > + _logger: DTSLogger > > _node_config: NodeConfiguration > > _transport: Transport | None > > > > - def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None: > > + def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None: > > """Connect to the node during initialization. > > > > Args: > > diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py > > index b158f963b6..5cfe202e15 100644 > > --- a/dts/framework/remote_session/interactive_shell.py > > +++ b/dts/framework/remote_session/interactive_shell.py > > @@ -20,7 +20,7 @@ > > > > from paramiko import Channel, SSHClient, channel # type: ignore[import] > > > > -from framework.logger import DTSLOG > > +from framework.logger import DTSLogger > > from framework.settings import SETTINGS > > > > > > @@ -38,7 +38,7 @@ class InteractiveShell(ABC): > > _stdin: channel.ChannelStdinFile > > _stdout: channel.ChannelFile > > _ssh_channel: Channel > > - _logger: DTSLOG > > + _logger: DTSLogger > > _timeout: float > > _app_args: str > > > > @@ -61,7 +61,7 @@ class InteractiveShell(ABC): > > def __init__( > > self, > > interactive_session: SSHClient, > > - logger: DTSLOG, > > + logger: DTSLogger, > > get_privileged_command: Callable[[str], str] | None, > > app_args: str = "", > > timeout: float = SETTINGS.timeout, > > diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py > > index 2059f9a981..a69dc99400 100644 > > --- a/dts/framework/remote_session/remote_session.py > > +++ b/dts/framework/remote_session/remote_session.py > > @@ -9,14 +9,13 @@ > > the structure of the result of a command execution. > > """ > > > > - > > import dataclasses > > from abc import ABC, abstractmethod > > from pathlib import PurePath > > > > from framework.config import NodeConfiguration > > from framework.exception import RemoteCommandExecutionError > > -from framework.logger import DTSLOG > > +from framework.logger import DTSLogger > > from framework.settings import SETTINGS > > > > > > @@ -75,14 +74,14 @@ class RemoteSession(ABC): > > username: str > > password: str > > history: list[CommandResult] > > - _logger: DTSLOG > > + _logger: DTSLogger > > _node_config: NodeConfiguration > > > > def __init__( > > self, > > node_config: NodeConfiguration, > > session_name: str, > > - logger: DTSLOG, > > + logger: DTSLogger, > > ): > > """Connect to the node during initialization. > > > > @@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None: > > Args: > > force: Force the closure of the connection. This may not clean up all resources. > > """ > > - self._logger.logger_exit() > > self._close(force) > > > > @abstractmethod > > diff --git a/dts/framework/runner.py b/dts/framework/runner.py > > index f58b0adc13..035e3368ef 100644 > > --- a/dts/framework/runner.py > > +++ b/dts/framework/runner.py > > @@ -19,9 +19,10 @@ > > > > import importlib > > import inspect > > -import logging > > +import os > > import re > > import sys > > +from pathlib import Path > > from types import MethodType > > from typing import Iterable > > > > @@ -38,7 +39,7 @@ > > SSHTimeoutError, > > TestCaseVerifyError, > > ) > > -from .logger import DTSLOG, getLogger > > +from .logger import DTSLogger, DtsStage, get_dts_logger > > from .settings import SETTINGS > > from .test_result import ( > > BuildTargetResult, > > @@ -73,7 +74,7 @@ class DTSRunner: > > """ > > > > _configuration: Configuration > > - _logger: DTSLOG > > + _logger: DTSLogger > > _result: DTSResult > > _test_suite_class_prefix: str > > _test_suite_module_prefix: str > > @@ -83,7 +84,10 @@ class DTSRunner: > > def __init__(self): > > """Initialize the instance with configuration, logger, result and string constants.""" > > self._configuration = load_config() > > - self._logger = getLogger("DTSRunner") > > + self._logger = get_dts_logger() > > + if not os.path.exists(SETTINGS.output_dir): > > + os.makedirs(SETTINGS.output_dir) > > + self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir) > > self._result = DTSResult(self._logger) > > self._test_suite_class_prefix = "Test" > > self._test_suite_module_prefix = "tests.TestSuite_" > > @@ -137,6 +141,7 @@ def run(self): > > > > # for all Execution sections > > for execution in self._configuration.executions: > > + self._logger.set_stage(DtsStage.execution) > > This ends up getting set twice in short succession which of course > doesn't functionally cause a problem, but I don't exactly see the > point of setting it twice. I'm not sure what you're referring to. The stage is only set once at the start of each execution (and more generally, at the start of each stage). It's also set in each finally block to properly mark the cleanup of that stage. There are two finally blocks in the execution stage where that could happen, but otherwise it should be set exactly once. > We could either set it here or set it in > the _run_execution, but i think it makes more sense to set it in the > _run_execution method as that is the method where we are doing the > setup, here we are only initializing the nodes which is still in a > sense "pre execution." Init nodes was pre-execution before the patch. I've moved it to execution because of two reasons: 1. The nodes are actually defined per-execution. It just made more sense to move them to execution. Also, if an error happens while connecting to a node, we don't want to abort the whole DTS run, just the execution, as the next execution could be connecting to a different node. 2. We're not just doing node init here, we're also discovering which test cases to run. This is essential to do as soon as possible (before anything else happens in the execution, such as connecting the nodes) so that we can mark blocked test cases in case of an error. Test case discovery is definitely part of each execution and putting node init after that was a natural consequence. There's an argument for discovering test cases of all executions before actually running any of the executions. It's certainly doable, but the code doesn't look that good (when not modifying the original execution config (which we shouldn't do) - I've tried) and there's actually not much of a point to do it this way, the result is almost the same. Where it makes a difference is when there's an error in test suite configuration and later executions - the user would have to wait for the previous execution to fully run to discover the error). > > > > self._logger.info( > > f"Running execution with SUT '{execution.system_under_test_node.name}'." > > ) > > @@ -164,6 +169,7 @@ def run(self): > > > > finally: > > try: > > + self._logger.set_stage(DtsStage.post_execution) > > for node in (sut_nodes | tg_nodes).values(): > > node.close() > > self._result.update_teardown(Result.PASS) > > @@ -419,6 +425,7 @@ def _run_execution( > > > > finally: > > try: > > + self._logger.set_stage(DtsStage.execution) > > sut_node.tear_down_execution() > > execution_result.update_teardown(Result.PASS) > > except Exception as e: > > @@ -447,6 +454,7 @@ def _run_build_target( > > with the current build target. > > test_suites_with_cases: The test suites with test cases to run. > > """ > > + self._logger.set_stage(DtsStage.build_target) > > self._logger.info(f"Running build target '{build_target.name}'.") > > > > try: > > @@ -463,6 +471,7 @@ def _run_build_target( > > > > finally: > > try: > > + self._logger.set_stage(DtsStage.build_target) > > sut_node.tear_down_build_target() > > build_target_result.update_teardown(Result.PASS) > > except Exception as e: > > @@ -535,6 +544,7 @@ def _run_test_suite( > > BlockingTestSuiteError: If a blocking test suite fails. > > """ > > test_suite_name = test_suite_with_cases.test_suite_class.__name__ > > + self._logger.set_stage(DtsStage.suite, Path(SETTINGS.output_dir, test_suite_name)) > > test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) > > try: > > self._logger.info(f"Starting test suite setup: {test_suite_name}") > > @@ -683,5 +693,4 @@ def _exit_dts(self) -> None: > > if self._logger: > > self._logger.info("DTS execution has ended.") > > > > - logging.shutdown() > > sys.exit(self._result.get_return_code()) > > diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py > > index eedb2d20ee..28f84fd793 100644 > > --- a/dts/framework/test_result.py > > +++ b/dts/framework/test_result.py > > @@ -42,7 +42,7 @@ > > TestSuiteConfig, > > ) > > from .exception import DTSError, ErrorSeverity > > -from .logger import DTSLOG > > +from .logger import DTSLogger > > from .settings import SETTINGS > > from .test_suite import TestSuite > > > > @@ -237,13 +237,13 @@ class DTSResult(BaseResult): > > """ > > > > dpdk_version: str | None > > - _logger: DTSLOG > > + _logger: DTSLogger > > _errors: list[Exception] > > _return_code: ErrorSeverity > > _stats_result: Union["Statistics", None] > > _stats_filename: str > > > > - def __init__(self, logger: DTSLOG): > > + def __init__(self, logger: DTSLogger): > > """Extend the constructor with top-level specifics. > > > > Args: > > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py > > index f9fe88093e..365f80e21a 100644 > > --- a/dts/framework/test_suite.py > > +++ b/dts/framework/test_suite.py > > @@ -21,7 +21,7 @@ > > from scapy.packet import Packet, Padding # type: ignore[import] > > > > from .exception import TestCaseVerifyError > > -from .logger import DTSLOG, getLogger > > +from .logger import DTSLogger, get_dts_logger > > from .testbed_model import Port, PortLink, SutNode, TGNode > > from .utils import get_packet_summaries > > > > @@ -61,7 +61,7 @@ class TestSuite(object): > > #: Whether the test suite is blocking. A failure of a blocking test suite > > #: will block the execution of all subsequent test suites in the current build target. > > is_blocking: ClassVar[bool] = False > > - _logger: DTSLOG > > + _logger: DTSLogger > > _port_links: list[PortLink] > > _sut_port_ingress: Port > > _sut_port_egress: Port > > @@ -88,7 +88,7 @@ def __init__( > > """ > > self.sut_node = sut_node > > self.tg_node = tg_node > > - self._logger = getLogger(self.__class__.__name__) > > + self._logger = get_dts_logger(self.__class__.__name__) > > self._port_links = [] > > self._process_links() > > self._sut_port_ingress, self._tg_port_egress = ( > > diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py > > index 1a55fadf78..74061f6262 100644 > > --- a/dts/framework/testbed_model/node.py > > +++ b/dts/framework/testbed_model/node.py > > @@ -23,7 +23,7 @@ > > NodeConfiguration, > > ) > > from framework.exception import ConfigurationError > > -from framework.logger import DTSLOG, getLogger > > +from framework.logger import DTSLogger, get_dts_logger > > from framework.settings import SETTINGS > > > > from .cpu import ( > > @@ -63,7 +63,7 @@ class Node(ABC): > > name: str > > lcores: list[LogicalCore] > > ports: list[Port] > > - _logger: DTSLOG > > + _logger: DTSLogger > > _other_sessions: list[OSSession] > > _execution_config: ExecutionConfiguration > > virtual_devices: list[VirtualDevice] > > @@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration): > > """ > > self.config = node_config > > self.name = node_config.name > > - self._logger = getLogger(self.name) > > + self._logger = get_dts_logger(self.name) > > self.main_session = create_session(self.config, self.name, self._logger) > > > > self._logger.info(f"Connected to node: {self.name}") > > @@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession: > > connection = create_session( > > self.config, > > session_name, > > - getLogger(session_name, node=self.name), > > + get_dts_logger(session_name), > > ) > > self._other_sessions.append(connection) > > return connection > > @@ -299,7 +299,6 @@ def close(self) -> None: > > self.main_session.close() > > for session in self._other_sessions: > > session.close() > > - self._logger.logger_exit() > > > > @staticmethod > > def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: > > @@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: > > return func > > > > > > -def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession: > > +def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession: > > """Factory for OS-aware sessions. > > > > Args: > > diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py > > index ac6bb5e112..6983aa4a77 100644 > > --- a/dts/framework/testbed_model/os_session.py > > +++ b/dts/framework/testbed_model/os_session.py > > @@ -21,7 +21,6 @@ > > the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux > > and other commands for other OSs. It also translates the path to match the underlying OS. > > """ > > - > > from abc import ABC, abstractmethod > > from collections.abc import Iterable > > from ipaddress import IPv4Interface, IPv6Interface > > @@ -29,7 +28,7 @@ > > from typing import Type, TypeVar, Union > > > > from framework.config import Architecture, NodeConfiguration, NodeInfo > > -from framework.logger import DTSLOG > > +from framework.logger import DTSLogger > > from framework.remote_session import ( > > CommandResult, > > InteractiveRemoteSession, > > @@ -62,7 +61,7 @@ class OSSession(ABC): > > > > _config: NodeConfiguration > > name: str > > - _logger: DTSLOG > > + _logger: DTSLogger > > remote_session: RemoteSession > > interactive_session: InteractiveRemoteSession > > > > @@ -70,7 +69,7 @@ def __init__( > > self, > > node_config: NodeConfiguration, > > name: str, > > - logger: DTSLOG, > > + logger: DTSLogger, > > ): > > """Initialize the OS-aware session. > > > > diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py > > index c49fbff488..d86d7fb532 100644 > > --- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py > > +++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py > > @@ -13,7 +13,7 @@ > > from scapy.packet import Packet # type: ignore[import] > > > > from framework.config import TrafficGeneratorConfig > > -from framework.logger import DTSLOG, getLogger > > +from framework.logger import DTSLogger, get_dts_logger > > from framework.testbed_model.node import Node > > from framework.testbed_model.port import Port > > from framework.utils import get_packet_summaries > > @@ -28,7 +28,7 @@ class TrafficGenerator(ABC): > > > > _config: TrafficGeneratorConfig > > _tg_node: Node > > - _logger: DTSLOG > > + _logger: DTSLogger > > > > def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): > > """Initialize the traffic generator. > > @@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): > > """ > > self._config = config > > self._tg_node = tg_node > > - self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}") > > + self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}") > > > > def send_packet(self, packet: Packet, port: Port) -> None: > > """Send `packet` and block until it is fully sent. > > diff --git a/dts/main.py b/dts/main.py > > index 1ffe8ff81f..d30c164b95 100755 > > --- a/dts/main.py > > +++ b/dts/main.py > > @@ -30,5 +30,4 @@ def main() -> None: > > > > # Main program begins here > > if __name__ == "__main__": > > - logging.raiseExceptions = True > > main() > > -- > > 2.34.1 > > ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 6/7] dts: refactor logging configuration 2024-02-14 7:49 ` Juraj Linkeš @ 2024-02-14 16:51 ` Jeremy Spewock 0 siblings, 0 replies; 44+ messages in thread From: Jeremy Spewock @ 2024-02-14 16:51 UTC (permalink / raw) To: Juraj Linkeš Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev On Wed, Feb 14, 2024 at 2:49 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote: > > Hi Jeremy, > > Just a reminder, please strip the parts you're not commenting on. > <snip> > > > + def __init__(self, *args, **kwargs): > > > + """Extend the constructor with extra file handlers.""" > > > + self._extra_file_handlers = [] > > > + super().__init__(*args, **kwargs) > > > > > > - One handler logs to the console, the other one to a file, with either a regular or verbose > > > - format. > > > + def makeRecord(self, *args, **kwargs): > > > > Is the return type annotation here skipped because of the `:meta private:`? > > > > Basically. We're modifying a method defined elsewhere, but there's no > harm in adding the return type. > <snip> > > > + logging.setLoggerClass(DTSLogger) > > > + if name: > > > + name = f"{dts_root_logger_name}.{name}" > > > + else: > > > + name = dts_root_logger_name > > > + logger = logging.getLogger(name) > > > + logging.setLoggerClass(logging.Logger) > > > > What's the benefit of setting the logger class back to logging.Logger > > here? Is the idea basically that if someone wanted to use the logging > > module we shouldn't implicitly make them use our DTSLogger? > > > > Yes. We should actually set it to whatever was there before (it may > not be logging.Logger), so I'll change that.+ > > <snip> > > > @@ -137,6 +141,7 @@ def run(self): > > > > > > # for all Execution sections > > > for execution in self._configuration.executions: > > > + self._logger.set_stage(DtsStage.execution) > > > > This ends up getting set twice in short succession which of course > > doesn't functionally cause a problem, but I don't exactly see the > > point of setting it twice. > > I'm not sure what you're referring to. The stage is only set once at > the start of each execution (and more generally, at the start of each > stage). It's also set in each finally block to properly mark the > cleanup of that stage. There are two finally blocks in the execution > stage where that could happen, but otherwise it should be set exactly > once. You're right, I misread where it was being set the second time in the code when I was reading this and thought there was a way where it could get set twice but it cannot. Apologies, it makes sense that you weren't sure what I was referring to because what I was referring to doesn't exist. > > > We could either set it here or set it in > > the _run_execution, but i think it makes more sense to set it in the > > _run_execution method as that is the method where we are doing the > > setup, here we are only initializing the nodes which is still in a > > sense "pre execution." > > Init nodes was pre-execution before the patch. I've moved it to > execution because of two reasons: > 1. The nodes are actually defined per-execution. It just made more > sense to move them to execution. Also, if an error happens while > connecting to a node, we don't want to abort the whole DTS run, just > the execution, as the next execution could be connecting to a > different node. > 2. We're not just doing node init here, we're also discovering which > test cases to run. This is essential to do as soon as possible (before > anything else happens in the execution, such as connecting the nodes) > so that we can mark blocked test cases in case of an error. Test case > discovery is definitely part of each execution and putting node init > after that was a natural consequence. There's an argument for > discovering test cases of all executions before actually running any > of the executions. It's certainly doable, but the code doesn't look > that good (when not modifying the original execution config (which we > shouldn't do) - I've tried) and there's actually not much of a point > to do it this way, the result is almost the same. Where it makes a > difference is when there's an error in test suite configuration and > later executions - the user would have to wait for the previous > execution to fully run to discover the error). > These are very good points. I was still thinking of these initializations as part of the pre-execution but I agree with you that the things we are doing are actually part of the execution. Thank you for elaborating! > > > > <snip> ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 7/7] dts: improve test suite and case filtering 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš ` (5 preceding siblings ...) 2024-02-06 14:57 ` [PATCH v2 6/7] dts: refactor logging configuration Juraj Linkeš @ 2024-02-06 14:57 ` Juraj Linkeš 6 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-06 14:57 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš The two places where we specify which test suite and test cases to run are complimentary and not that intuitive to use. A unified way provides a better user experience. The syntax in test run configuration file has not changed, but the environment variable and the command line arguments was changed to match the config file syntax. This required changes in the settings module which greatly simplified the parsing of the environment variables while retaining the same functionality. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- doc/guides/tools/dts.rst | 14 ++- dts/framework/config/__init__.py | 12 +- dts/framework/runner.py | 21 ++-- dts/framework/settings.py | 187 ++++++++++++++----------------- dts/framework/test_suite.py | 2 +- dts/main.py | 2 - 6 files changed, 116 insertions(+), 122 deletions(-) diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst index f686ca487c..d1c3c2af7a 100644 --- a/doc/guides/tools/dts.rst +++ b/doc/guides/tools/dts.rst @@ -215,28 +215,30 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet .. code-block:: console (dts-py3.10) $ ./main.py --help - usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN] + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN] Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority. options: -h, --help show this help message and exit --config-file CONFIG_FILE - [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml) + [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml) --output-dir OUTPUT_DIR, --output OUTPUT_DIR [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output) -t TIMEOUT, --timeout TIMEOUT [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15) -v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False) - -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None) + -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False) --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz) --compile-timeout COMPILE_TIMEOUT [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200) - --test-cases TEST_CASES - [DTS_TESTCASES] Comma-separated list of test cases to execute. Unknown test cases will be silently ignored. (default: ) + --test-suite TEST_SUITE [TEST_CASES ...] + [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment + variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...' + (default: []) --re-run RE_RUN, --re_run RE_RUN - [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs (default: 0) + [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0) The brackets contain the names of environment variables that set the same thing. diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index c6a93b3b89..4cb5c74059 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -35,9 +35,9 @@ import json import os.path -import pathlib from dataclasses import dataclass, fields from enum import auto, unique +from pathlib import Path from typing import Union import warlock # type: ignore[import] @@ -53,7 +53,6 @@ TrafficGeneratorConfigDict, ) from framework.exception import ConfigurationError -from framework.settings import SETTINGS from framework.utils import StrEnum @@ -571,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration": return Configuration(executions=executions) -def load_config() -> Configuration: +def load_config(config_file_path: Path) -> Configuration: """Load DTS test run configuration from a file. Load the YAML test run configuration file @@ -581,13 +580,16 @@ def load_config() -> Configuration: The YAML test run configuration file is specified in the :option:`--config-file` command line argument or the :envvar:`DTS_CFG_FILE` environment variable. + Args: + config_file_path: The path to the YAML test run configuration file. + Returns: The parsed test run configuration. """ - with open(SETTINGS.config_file_path, "r") as f: + with open(config_file_path, "r") as f: config_data = yaml.safe_load(f) - schema_path = os.path.join(pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json") + schema_path = os.path.join(Path(__file__).parent.resolve(), "conf_yaml_schema.json") with open(schema_path, "r") as f: schema = json.load(f) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 035e3368ef..32c0698cd2 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -24,7 +24,7 @@ import sys from pathlib import Path from types import MethodType -from typing import Iterable +from typing import Iterable, Sequence from .config import ( BuildTargetConfiguration, @@ -83,7 +83,7 @@ class DTSRunner: def __init__(self): """Initialize the instance with configuration, logger, result and string constants.""" - self._configuration = load_config() + self._configuration = load_config(SETTINGS.config_file_path) self._logger = get_dts_logger() if not os.path.exists(SETTINGS.output_dir): os.makedirs(SETTINGS.output_dir) @@ -129,7 +129,7 @@ def run(self): #. Execution teardown The test cases are filtered according to the specification in the test run configuration and - the :option:`--test-cases` command line argument or + the :option:`--test-suite` command line argument or the :envvar:`DTS_TESTCASES` environment variable. """ sut_nodes: dict[str, SutNode] = {} @@ -146,14 +146,17 @@ def run(self): f"Running execution with SUT '{execution.system_under_test_node.name}'." ) execution_result = self._result.add_execution(execution) + test_suite_configs = ( + SETTINGS.test_suites if SETTINGS.test_suites else execution.test_suites + ) try: test_suites_with_cases = self._get_test_suites_with_cases( - execution.test_suites, execution.func, execution.perf + test_suite_configs, execution.func, execution.perf ) execution_result.test_suites_with_cases = test_suites_with_cases except Exception as e: self._logger.exception( - f"Invalid test suite configuration found: " f"{execution.test_suites}." + f"Invalid test suite configuration found: " f"{test_suite_configs}." ) execution_result.update_setup(Result.FAIL, e) @@ -222,7 +225,7 @@ def _get_test_suites_with_cases( test_suite_class = self._get_test_suite_class(test_suite_config.test_suite) test_cases = [] func_test_cases, perf_test_cases = self._filter_test_cases( - test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases) + test_suite_class, test_suite_config.test_cases ) if func: test_cases.extend(func_test_cases) @@ -295,7 +298,7 @@ def is_test_suite(object) -> bool: ) def _filter_test_cases( - self, test_suite_class: type[TestSuite], test_cases_to_run: set[str] + self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str] ) -> tuple[list[MethodType], list[MethodType]]: """Filter `test_cases_to_run` from `test_suite_class`. @@ -324,7 +327,9 @@ def _filter_test_cases( (name, method) for name, method in name_method_tuples if name in test_cases_to_run ] if len(name_method_tuples) < len(test_cases_to_run): - missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples} + missing_test_cases = set(test_cases_to_run) - { + name for name, _ in name_method_tuples + } raise ConfigurationError( f"Test cases {missing_test_cases} not found among methods " f"of {test_suite_class.__name__}." diff --git a/dts/framework/settings.py b/dts/framework/settings.py index 2b8bfbe0ed..688e8679a7 100644 --- a/dts/framework/settings.py +++ b/dts/framework/settings.py @@ -48,10 +48,11 @@ The path to a DPDK tarball, git commit ID, tag ID or tree ID to test. -.. option:: --test-cases -.. envvar:: DTS_TESTCASES +.. option:: --test-suite +.. envvar:: DTS_TEST_SUITES - A comma-separated list of test cases to execute. Unknown test cases will be silently ignored. + A test suite with test cases which may be specified multiple times. + In the environment variable, the suites are joined with a comma. .. option:: --re-run, --re_run .. envvar:: DTS_RERUN @@ -71,83 +72,13 @@ import argparse import os -from collections.abc import Callable, Iterable, Sequence from dataclasses import dataclass, field from pathlib import Path -from typing import Any, TypeVar +from typing import Any +from .config import TestSuiteConfig from .utils import DPDKGitTarball -_T = TypeVar("_T") - - -def _env_arg(env_var: str) -> Any: - """A helper method augmenting the argparse Action with environment variables. - - If the supplied environment variable is defined, then the default value - of the argument is modified. This satisfies the priority order of - command line argument > environment variable > default value. - - Arguments with no values (flags) should be defined using the const keyword argument - (True or False). When the argument is specified, it will be set to const, if not specified, - the default will be stored (possibly modified by the corresponding environment variable). - - Other arguments work the same as default argparse arguments, that is using - the default 'store' action. - - Returns: - The modified argparse.Action. - """ - - class _EnvironmentArgument(argparse.Action): - def __init__( - self, - option_strings: Sequence[str], - dest: str, - nargs: str | int | None = None, - const: bool | None = None, - default: Any = None, - type: Callable[[str], _T | argparse.FileType | None] = None, - choices: Iterable[_T] | None = None, - required: bool = False, - help: str | None = None, - metavar: str | tuple[str, ...] | None = None, - ) -> None: - env_var_value = os.environ.get(env_var) - default = env_var_value or default - if const is not None: - nargs = 0 - default = const if env_var_value else default - type = None - choices = None - metavar = None - super(_EnvironmentArgument, self).__init__( - option_strings, - dest, - nargs=nargs, - const=const, - default=default, - type=type, - choices=choices, - required=required, - help=help, - metavar=metavar, - ) - - def __call__( - self, - parser: argparse.ArgumentParser, - namespace: argparse.Namespace, - values: Any, - option_string: str = None, - ) -> None: - if self.const is not None: - setattr(namespace, self.dest, self.const) - else: - setattr(namespace, self.dest, values) - - return _EnvironmentArgument - @dataclass(slots=True) class Settings: @@ -171,7 +102,7 @@ class Settings: #: compile_timeout: float = 1200 #: - test_cases: list[str] = field(default_factory=list) + test_suites: list[TestSuiteConfig] = field(default_factory=list) #: re_run: int = 0 @@ -180,6 +111,31 @@ class Settings: def _get_parser() -> argparse.ArgumentParser: + """Create the argument parser for DTS. + + Command line options take precedence over environment variables, which in turn take precedence + over default values. + + Returns: + argparse.ArgumentParser: The configured argument parser with defined options. + """ + + def env_arg(env_var: str, default: Any) -> Any: + """A helper function augmenting the argparse with environment variables. + + If the supplied environment variable is defined, then the default value + of the argument is modified. This satisfies the priority order of + command line argument > environment variable > default value. + + Args: + env_var: Environment variable name. + default: Default value. + + Returns: + Environment variable or default value. + """ + return os.environ.get(env_var) or default + parser = argparse.ArgumentParser( description="Run DPDK test suites. All options may be specified with the environment " "variables provided in brackets. Command line arguments have higher priority.", @@ -188,25 +144,23 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "--config-file", - action=_env_arg("DTS_CFG_FILE"), - default=SETTINGS.config_file_path, + default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path), type=Path, - help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets.", + help="[DTS_CFG_FILE] The configuration file that describes the test cases, " + "SUTs and targets.", ) parser.add_argument( "--output-dir", "--output", - action=_env_arg("DTS_OUTPUT_DIR"), - default=SETTINGS.output_dir, + default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir), help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.", ) parser.add_argument( "-t", "--timeout", - action=_env_arg("DTS_TIMEOUT"), - default=SETTINGS.timeout, + default=env_arg("DTS_TIMEOUT", SETTINGS.timeout), type=float, help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.", ) @@ -214,9 +168,8 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "-v", "--verbose", - action=_env_arg("DTS_VERBOSE"), - default=SETTINGS.verbose, - const=True, + action="store_true", + default=env_arg("DTS_VERBOSE", SETTINGS.verbose), help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages " "to the console.", ) @@ -224,8 +177,8 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "-s", "--skip-setup", - action=_env_arg("DTS_SKIP_SETUP"), - const=True, + action="store_true", + default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup), help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.", ) @@ -233,8 +186,7 @@ def _get_parser() -> argparse.ArgumentParser: "--tarball", "--snapshot", "--git-ref", - action=_env_arg("DTS_DPDK_TARBALL"), - default=SETTINGS.dpdk_tarball_path, + default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path), type=Path, help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, " "tag ID or tree ID to test. To test local changes, first commit them, " @@ -243,36 +195,71 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "--compile-timeout", - action=_env_arg("DTS_COMPILE_TIMEOUT"), - default=SETTINGS.compile_timeout, + default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout), type=float, help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.", ) parser.add_argument( - "--test-cases", - action=_env_arg("DTS_TESTCASES"), - default="", - help="[DTS_TESTCASES] Comma-separated list of test cases to execute.", + "--test-suite", + action="append", + nargs="+", + metavar=("TEST_SUITE", "TEST_CASES"), + default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites), + help="[DTS_TEST_SUITES] A list containing a test suite with test cases. " + "The first parameter is the test suite name, and the rest are test case names, " + "which are optional. May be specified multiple times. To specify multiple test suites in " + "the environment variable, join the lists with a comma. " + "Examples: " + "--test-suite suite case case --test-suite suite case ... | " + "DTS_TEST_SUITES='suite case case, suite case, ...' | " + "--test-suite suite --test-suite suite case ... | " + "DTS_TEST_SUITES='suite, suite case, ...'", ) parser.add_argument( "--re-run", "--re_run", - action=_env_arg("DTS_RERUN"), - default=SETTINGS.re_run, + default=env_arg("DTS_RERUN", SETTINGS.re_run), type=int, help="[DTS_RERUN] Re-run each test case the specified number of times " - "if a test failure occurs", + "if a test failure occurs.", ) return parser +def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]: + """Process the given argument to a list of :class:`TestSuiteConfig` to execute. + + Args: + args: The arguments to process. The args is a string from an environment variable + or a list of from the user input containing tests suites with tests cases, + each of which is a list of [test_suite, test_case, test_case, ...]. + + Returns: + A list of test suite configurations to execute. + """ + if isinstance(args, str): + # Environment variable in the form of "suite case case, suite case, suite, ..." + args = [suite_with_cases.split() for suite_with_cases in args.split(",")] + + test_suites_to_run = [] + for suite_with_cases in args: + test_suites_to_run.append( + TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:]) + ) + + return test_suites_to_run + + def get_settings() -> Settings: """Create new settings with inputs from the user. The inputs are taken from the command line and from environment variables. + + Returns: + The new settings object. """ parsed_args = _get_parser().parse_args() return Settings( @@ -287,6 +274,6 @@ def get_settings() -> Settings: else Path(parsed_args.tarball) ), compile_timeout=parsed_args.compile_timeout, - test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []), + test_suites=_process_test_suites(parsed_args.test_suite), re_run=parsed_args.re_run, ) diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index 365f80e21a..1957ea7328 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -40,7 +40,7 @@ class TestSuite(object): and functional test cases (all other test cases). By default, all test cases will be executed. A list of testcase names may be specified - in the YAML test run configuration file and in the :option:`--test-cases` command line argument + in the YAML test run configuration file and in the :option:`--test-suite` command line argument or in the :envvar:`DTS_TESTCASES` environment variable to filter which test cases to run. The union of both lists will be used. Any unknown test cases from the latter lists will be silently ignored. diff --git a/dts/main.py b/dts/main.py index d30c164b95..fa878cc16e 100755 --- a/dts/main.py +++ b/dts/main.py @@ -6,8 +6,6 @@ """The DTS executable.""" -import logging - from framework import settings -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 0/7] test case blocking and logging 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš ` (6 preceding siblings ...) 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš @ 2024-02-23 7:54 ` Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 1/7] dts: convert dts.py methods to class Juraj Linkeš ` (7 more replies) 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš 8 siblings, 8 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš We currently don't record test case results that couldn't be executed because of a previous failure, such as when a test suite setup failed, resulting in no executed test cases. In order to record the test cases that couldn't be executed, we must know the lists of test suites and test cases ahead of the actual test suite execution, as an error could occur before we even start executing test suites. In addition, the patch series contains two refactors and one improvement. The first refactor is closely related. The dts.py was renamed to runner.py and given a clear purpose - running the test suites and all other orchestration needed to run test suites. The logic for this was not all in the original dts.py module and it was brought there. The runner is also responsible for recording results, which is the blocked test cases are recorded. The other refactor, logging, is related to the first refactor. The logging module was simplified while extending capabilities. Each test suite logs into its own log file in addition to the main log file which the runner must handle (as it knows when we start executing particular test suites). The runner also handles the switching between execution stages for the purposes of logging. The one aforementioned improvement is in unifying how we specify test cases in the conf.yaml file and in the environment variable/command line argument. v2: Rebase and update of the whole patch. There are changes in all parts of the code, mainly improving the design and logic. Also added the last patch which improves test suite specification on the cmdline. v3: Improved variables in _get_test_suite_class along with docstring. Fixed smoke test suite not being added into the list of test suites to be executed. Juraj Linkeš (7): dts: convert dts.py methods to class dts: move test suite execution logic to DTSRunner dts: filter test suites in executions dts: reorganize test result dts: block all test cases when earlier setup fails dts: refactor logging configuration dts: improve test suite and case filtering doc/guides/tools/dts.rst | 14 +- dts/framework/config/__init__.py | 36 +- dts/framework/config/conf_yaml_schema.json | 2 +- dts/framework/dts.py | 338 --------- dts/framework/logger.py | 236 +++--- dts/framework/remote_session/__init__.py | 6 +- .../interactive_remote_session.py | 6 +- .../remote_session/interactive_shell.py | 6 +- .../remote_session/remote_session.py | 8 +- dts/framework/runner.py | 706 ++++++++++++++++++ dts/framework/settings.py | 188 +++-- dts/framework/test_result.py | 565 ++++++++------ dts/framework/test_suite.py | 239 +----- dts/framework/testbed_model/node.py | 11 +- dts/framework/testbed_model/os_session.py | 7 +- .../traffic_generator/traffic_generator.py | 6 +- dts/main.py | 9 +- dts/pyproject.toml | 3 + dts/tests/TestSuite_smoke_tests.py | 2 +- 19 files changed, 1360 insertions(+), 1028 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 1/7] dts: convert dts.py methods to class 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš @ 2024-02-23 7:54 ` Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš ` (6 subsequent siblings) 7 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš The dts.py module deviates from the rest of the code without a clear reason. Converting it into a class and using better naming will improve organization and code readability. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/dts.py | 338 ---------------------------------------- dts/framework/runner.py | 333 +++++++++++++++++++++++++++++++++++++++ dts/main.py | 6 +- 3 files changed, 337 insertions(+), 340 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py diff --git a/dts/framework/dts.py b/dts/framework/dts.py deleted file mode 100644 index e16d4578a0..0000000000 --- a/dts/framework/dts.py +++ /dev/null @@ -1,338 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2010-2019 Intel Corporation -# Copyright(c) 2022-2023 PANTHEON.tech s.r.o. -# Copyright(c) 2022-2023 University of New Hampshire - -r"""Test suite runner module. - -A DTS run is split into stages: - - #. Execution stage, - #. Build target stage, - #. Test suite stage, - #. Test case stage. - -The module is responsible for running tests on testbeds defined in the test run configuration. -Each setup or teardown of each stage is recorded in a :class:`~.test_result.DTSResult` or -one of its subclasses. The test case results are also recorded. - -If an error occurs, the current stage is aborted, the error is recorded and the run continues in -the next iteration of the same stage. The return code is the highest `severity` of all -:class:`~.exception.DTSError`\s. - -Example: - An error occurs in a build target setup. The current build target is aborted and the run - continues with the next build target. If the errored build target was the last one in the given - execution, the next execution begins. - -Attributes: - dts_logger: The logger instance used in this module. - result: The top level result used in the module. -""" - -import sys - -from .config import ( - BuildTargetConfiguration, - ExecutionConfiguration, - TestSuiteConfig, - load_config, -) -from .exception import BlockingTestSuiteError -from .logger import DTSLOG, getLogger -from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result -from .test_suite import get_test_suites -from .testbed_model import SutNode, TGNode - -# dummy defaults to satisfy linters -dts_logger: DTSLOG = None # type: ignore[assignment] -result: DTSResult = DTSResult(dts_logger) - - -def run_all() -> None: - """Run all build targets in all executions from the test run configuration. - - Before running test suites, executions and build targets are first set up. - The executions and build targets defined in the test run configuration are iterated over. - The executions define which tests to run and where to run them and build targets define - the DPDK build setup. - - The tests suites are set up for each execution/build target tuple and each scheduled - test case within the test suite is set up, executed and torn down. After all test cases - have been executed, the test suite is torn down and the next build target will be tested. - - All the nested steps look like this: - - #. Execution setup - - #. Build target setup - - #. Test suite setup - - #. Test case setup - #. Test case logic - #. Test case teardown - - #. Test suite teardown - - #. Build target teardown - - #. Execution teardown - - The test cases are filtered according to the specification in the test run configuration and - the :option:`--test-cases` command line argument or - the :envvar:`DTS_TESTCASES` environment variable. - """ - global dts_logger - global result - - # create a regular DTS logger and create a new result with it - dts_logger = getLogger("DTSRunner") - result = DTSResult(dts_logger) - - # check the python version of the server that run dts - _check_dts_python_version() - - sut_nodes: dict[str, SutNode] = {} - tg_nodes: dict[str, TGNode] = {} - try: - # for all Execution sections - for execution in load_config().executions: - sut_node = sut_nodes.get(execution.system_under_test_node.name) - tg_node = tg_nodes.get(execution.traffic_generator_node.name) - - try: - if not sut_node: - sut_node = SutNode(execution.system_under_test_node) - sut_nodes[sut_node.name] = sut_node - if not tg_node: - tg_node = TGNode(execution.traffic_generator_node) - tg_nodes[tg_node.name] = tg_node - result.update_setup(Result.PASS) - except Exception as e: - failed_node = execution.system_under_test_node.name - if sut_node: - failed_node = execution.traffic_generator_node.name - dts_logger.exception(f"Creation of node {failed_node} failed.") - result.update_setup(Result.FAIL, e) - - else: - _run_execution(sut_node, tg_node, execution, result) - - except Exception as e: - dts_logger.exception("An unexpected error has occurred.") - result.add_error(e) - raise - - finally: - try: - for node in (sut_nodes | tg_nodes).values(): - node.close() - result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Final cleanup of nodes failed.") - result.update_teardown(Result.ERROR, e) - - # we need to put the sys.exit call outside the finally clause to make sure - # that unexpected exceptions will propagate - # in that case, the error that should be reported is the uncaught exception as - # that is a severe error originating from the framework - # at that point, we'll only have partial results which could be impacted by the - # error causing the uncaught exception, making them uninterpretable - _exit_dts() - - -def _check_dts_python_version() -> None: - """Check the required Python version - v3.10.""" - - def RED(text: str) -> str: - return f"\u001B[31;1m{str(text)}\u001B[0m" - - if sys.version_info.major < 3 or (sys.version_info.major == 3 and sys.version_info.minor < 10): - print( - RED( - ( - "WARNING: DTS execution node's python version is lower than" - "python 3.10, is deprecated and will not work in future releases." - ) - ), - file=sys.stderr, - ) - print(RED("Please use Python >= 3.10 instead"), file=sys.stderr) - - -def _run_execution( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - result: DTSResult, -) -> None: - """Run the given execution. - - This involves running the execution setup as well as running all build targets - in the given execution. After that, execution teardown is run. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - execution: An execution's test run configuration. - result: The top level result object. - """ - dts_logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") - execution_result = result.add_execution(sut_node.config) - execution_result.add_sut_info(sut_node.node_info) - - try: - sut_node.set_up_execution(execution) - execution_result.update_setup(Result.PASS) - except Exception as e: - dts_logger.exception("Execution setup failed.") - execution_result.update_setup(Result.FAIL, e) - - else: - for build_target in execution.build_targets: - _run_build_target(sut_node, tg_node, build_target, execution, execution_result) - - finally: - try: - sut_node.tear_down_execution() - execution_result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Execution teardown failed.") - execution_result.update_teardown(Result.FAIL, e) - - -def _run_build_target( - sut_node: SutNode, - tg_node: TGNode, - build_target: BuildTargetConfiguration, - execution: ExecutionConfiguration, - execution_result: ExecutionResult, -) -> None: - """Run the given build target. - - This involves running the build target setup as well as running all test suites - in the given execution the build target is defined in. - After that, build target teardown is run. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - build_target: A build target's test run configuration. - execution: The build target's execution's test run configuration. - execution_result: The execution level result object associated with the execution. - """ - dts_logger.info(f"Running build target '{build_target.name}'.") - build_target_result = execution_result.add_build_target(build_target) - - try: - sut_node.set_up_build_target(build_target) - result.dpdk_version = sut_node.dpdk_version - build_target_result.add_build_target_info(sut_node.get_build_target_info()) - build_target_result.update_setup(Result.PASS) - except Exception as e: - dts_logger.exception("Build target setup failed.") - build_target_result.update_setup(Result.FAIL, e) - - else: - _run_all_suites(sut_node, tg_node, execution, build_target_result) - - finally: - try: - sut_node.tear_down_build_target() - build_target_result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Build target teardown failed.") - build_target_result.update_teardown(Result.FAIL, e) - - -def _run_all_suites( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, -) -> None: - """Run the execution's (possibly a subset) test suites using the current build target. - - The function assumes the build target we're testing has already been built on the SUT node. - The current build target thus corresponds to the current DPDK build present on the SUT node. - - If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites - in the current build target won't be executed. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - execution: The execution's test run configuration associated with the current build target. - build_target_result: The build target level result object associated - with the current build target. - """ - end_build_target = False - if not execution.skip_smoke_tests: - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] - for test_suite_config in execution.test_suites: - try: - _run_single_suite(sut_node, tg_node, execution, build_target_result, test_suite_config) - except BlockingTestSuiteError as e: - dts_logger.exception( - f"An error occurred within {test_suite_config.test_suite}. Skipping build target." - ) - result.add_error(e) - end_build_target = True - # if a blocking test failed and we need to bail out of suite executions - if end_build_target: - break - - -def _run_single_suite( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, - test_suite_config: TestSuiteConfig, -) -> None: - """Run all test suite in a single test suite module. - - The function assumes the build target we're testing has already been built on the SUT node. - The current build target thus corresponds to the current DPDK build present on the SUT node. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - execution: The execution's test run configuration associated with the current build target. - build_target_result: The build target level result object associated - with the current build target. - test_suite_config: Test suite test run configuration specifying the test suite module - and possibly a subset of test cases of test suites in that module. - - Raises: - BlockingTestSuiteError: If a blocking test suite fails. - """ - try: - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" - test_suite_classes = get_test_suites(full_suite_path) - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) - dts_logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") - except Exception as e: - dts_logger.exception("An error occurred when searching for test suites.") - result.update_setup(Result.ERROR, e) - - else: - for test_suite_class in test_suite_classes: - test_suite = test_suite_class( - sut_node, - tg_node, - test_suite_config.test_cases, - execution.func, - build_target_result, - ) - test_suite.run() - - -def _exit_dts() -> None: - """Process all errors and exit with the proper exit code.""" - result.process() - - if dts_logger: - dts_logger.info("DTS execution has ended.") - sys.exit(result.get_return_code()) diff --git a/dts/framework/runner.py b/dts/framework/runner.py new file mode 100644 index 0000000000..acc1c4d6db --- /dev/null +++ b/dts/framework/runner.py @@ -0,0 +1,333 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2019 Intel Corporation +# Copyright(c) 2022-2023 PANTHEON.tech s.r.o. +# Copyright(c) 2022-2023 University of New Hampshire + +"""Test suite runner module. + +The module is responsible for running DTS in a series of stages: + + #. Execution stage, + #. Build target stage, + #. Test suite stage, + #. Test case stage. + +The execution and build target stages set up the environment before running test suites. +The test suite stage sets up steps common to all test cases +and the test case stage runs test cases individually. +""" + +import logging +import sys + +from .config import ( + BuildTargetConfiguration, + ExecutionConfiguration, + TestSuiteConfig, + load_config, +) +from .exception import BlockingTestSuiteError +from .logger import DTSLOG, getLogger +from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result +from .test_suite import get_test_suites +from .testbed_model import SutNode, TGNode + + +class DTSRunner: + r"""Test suite runner class. + + The class is responsible for running tests on testbeds defined in the test run configuration. + Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult` + or one of its subclasses. The test case results are also recorded. + + If an error occurs, the current stage is aborted, the error is recorded and the run continues in + the next iteration of the same stage. The return code is the highest `severity` of all + :class:`~.framework.exception.DTSError`\s. + + Example: + An error occurs in a build target setup. The current build target is aborted and the run + continues with the next build target. If the errored build target was the last one in the + given execution, the next execution begins. + """ + + _logger: DTSLOG + _result: DTSResult + + def __init__(self): + """Initialize the instance with logger and result.""" + self._logger = getLogger("DTSRunner") + self._result = DTSResult(self._logger) + + def run(self): + """Run all build targets in all executions from the test run configuration. + + Before running test suites, executions and build targets are first set up. + The executions and build targets defined in the test run configuration are iterated over. + The executions define which tests to run and where to run them and build targets define + the DPDK build setup. + + The tests suites are set up for each execution/build target tuple and each discovered + test case within the test suite is set up, executed and torn down. After all test cases + have been executed, the test suite is torn down and the next build target will be tested. + + All the nested steps look like this: + + #. Execution setup + + #. Build target setup + + #. Test suite setup + + #. Test case setup + #. Test case logic + #. Test case teardown + + #. Test suite teardown + + #. Build target teardown + + #. Execution teardown + + The test cases are filtered according to the specification in the test run configuration and + the :option:`--test-cases` command line argument or + the :envvar:`DTS_TESTCASES` environment variable. + """ + sut_nodes: dict[str, SutNode] = {} + tg_nodes: dict[str, TGNode] = {} + try: + # check the python version of the server that runs dts + self._check_dts_python_version() + + # for all Execution sections + for execution in load_config().executions: + sut_node = sut_nodes.get(execution.system_under_test_node.name) + tg_node = tg_nodes.get(execution.traffic_generator_node.name) + + try: + if not sut_node: + sut_node = SutNode(execution.system_under_test_node) + sut_nodes[sut_node.name] = sut_node + if not tg_node: + tg_node = TGNode(execution.traffic_generator_node) + tg_nodes[tg_node.name] = tg_node + self._result.update_setup(Result.PASS) + except Exception as e: + failed_node = execution.system_under_test_node.name + if sut_node: + failed_node = execution.traffic_generator_node.name + self._logger.exception(f"The Creation of node {failed_node} failed.") + self._result.update_setup(Result.FAIL, e) + + else: + self._run_execution(sut_node, tg_node, execution) + + except Exception as e: + self._logger.exception("An unexpected error has occurred.") + self._result.add_error(e) + raise + + finally: + try: + for node in (sut_nodes | tg_nodes).values(): + node.close() + self._result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("The final cleanup of nodes failed.") + self._result.update_teardown(Result.ERROR, e) + + # we need to put the sys.exit call outside the finally clause to make sure + # that unexpected exceptions will propagate + # in that case, the error that should be reported is the uncaught exception as + # that is a severe error originating from the framework + # at that point, we'll only have partial results which could be impacted by the + # error causing the uncaught exception, making them uninterpretable + self._exit_dts() + + def _check_dts_python_version(self) -> None: + """Check the required Python version - v3.10.""" + if sys.version_info.major < 3 or ( + sys.version_info.major == 3 and sys.version_info.minor < 10 + ): + self._logger.warning( + "DTS execution node's python version is lower than Python 3.10, " + "is deprecated and will not work in future releases." + ) + self._logger.warning("Please use Python >= 3.10 instead.") + + def _run_execution( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + ) -> None: + """Run the given execution. + + This involves running the execution setup as well as running all build targets + in the given execution. After that, execution teardown is run. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: An execution's test run configuration. + """ + self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") + execution_result = self._result.add_execution(sut_node.config) + execution_result.add_sut_info(sut_node.node_info) + + try: + sut_node.set_up_execution(execution) + execution_result.update_setup(Result.PASS) + except Exception as e: + self._logger.exception("Execution setup failed.") + execution_result.update_setup(Result.FAIL, e) + + else: + for build_target in execution.build_targets: + self._run_build_target(sut_node, tg_node, build_target, execution, execution_result) + + finally: + try: + sut_node.tear_down_execution() + execution_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("Execution teardown failed.") + execution_result.update_teardown(Result.FAIL, e) + + def _run_build_target( + self, + sut_node: SutNode, + tg_node: TGNode, + build_target: BuildTargetConfiguration, + execution: ExecutionConfiguration, + execution_result: ExecutionResult, + ) -> None: + """Run the given build target. + + This involves running the build target setup as well as running all test suites + of the build target's execution. + After that, build target teardown is run. + + Args: + sut_node: The execution's sut node. + tg_node: The execution's tg node. + build_target: A build target's test run configuration. + execution: The build target's execution's test run configuration. + execution_result: The execution level result object associated with the execution. + """ + self._logger.info(f"Running build target '{build_target.name}'.") + build_target_result = execution_result.add_build_target(build_target) + + try: + sut_node.set_up_build_target(build_target) + self._result.dpdk_version = sut_node.dpdk_version + build_target_result.add_build_target_info(sut_node.get_build_target_info()) + build_target_result.update_setup(Result.PASS) + except Exception as e: + self._logger.exception("Build target setup failed.") + build_target_result.update_setup(Result.FAIL, e) + + else: + self._run_all_suites(sut_node, tg_node, execution, build_target_result) + + finally: + try: + sut_node.tear_down_build_target() + build_target_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("Build target teardown failed.") + build_target_result.update_teardown(Result.FAIL, e) + + def _run_all_suites( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + build_target_result: BuildTargetResult, + ) -> None: + """Run the execution's (possibly a subset of) test suites using the current build target. + + The method assumes the build target we're testing has already been built on the SUT node. + The current build target thus corresponds to the current DPDK build present on the SUT node. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: The execution's test run configuration associated + with the current build target. + build_target_result: The build target level result object associated + with the current build target. + """ + end_build_target = False + if not execution.skip_smoke_tests: + execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] + for test_suite_config in execution.test_suites: + try: + self._run_single_suite( + sut_node, tg_node, execution, build_target_result, test_suite_config + ) + except BlockingTestSuiteError as e: + self._logger.exception( + f"An error occurred within {test_suite_config.test_suite}. " + "Skipping build target..." + ) + self._result.add_error(e) + end_build_target = True + # if a blocking test failed and we need to bail out of suite executions + if end_build_target: + break + + def _run_single_suite( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + build_target_result: BuildTargetResult, + test_suite_config: TestSuiteConfig, + ) -> None: + """Run all test suites in a single test suite module. + + The method assumes the build target we're testing has already been built on the SUT node. + The current build target thus corresponds to the current DPDK build present on the SUT node. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: The execution's test run configuration associated + with the current build target. + build_target_result: The build target level result object associated + with the current build target. + test_suite_config: Test suite test run configuration specifying the test suite module + and possibly a subset of test cases of test suites in that module. + + Raises: + BlockingTestSuiteError: If a blocking test suite fails. + """ + try: + full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" + test_suite_classes = get_test_suites(full_suite_path) + suites_str = ", ".join((x.__name__ for x in test_suite_classes)) + self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") + except Exception as e: + self._logger.exception("An error occurred when searching for test suites.") + self._result.update_setup(Result.ERROR, e) + + else: + for test_suite_class in test_suite_classes: + test_suite = test_suite_class( + sut_node, + tg_node, + test_suite_config.test_cases, + execution.func, + build_target_result, + ) + test_suite.run() + + def _exit_dts(self) -> None: + """Process all errors and exit with the proper exit code.""" + self._result.process() + + if self._logger: + self._logger.info("DTS execution has ended.") + + logging.shutdown() + sys.exit(self._result.get_return_code()) diff --git a/dts/main.py b/dts/main.py index f703615d11..1ffe8ff81f 100755 --- a/dts/main.py +++ b/dts/main.py @@ -21,9 +21,11 @@ def main() -> None: be modified before the settings module is imported anywhere else in the framework. """ settings.SETTINGS = settings.get_settings() - from framework import dts - dts.run_all() + from framework.runner import DTSRunner + + dts = DTSRunner() + dts.run() # Main program begins here -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 2/7] dts: move test suite execution logic to DTSRunner 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 1/7] dts: convert dts.py methods to class Juraj Linkeš @ 2024-02-23 7:54 ` Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 3/7] dts: filter test suites in executions Juraj Linkeš ` (5 subsequent siblings) 7 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš Move the code responsible for running the test suite from the TestSuite class to the DTSRunner class. This restructuring decision was made to consolidate and unify the related logic into a single unit. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/runner.py | 175 ++++++++++++++++++++++++++++++++---- dts/framework/test_suite.py | 152 ++----------------------------- 2 files changed, 169 insertions(+), 158 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index acc1c4d6db..933685d638 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -19,6 +19,7 @@ import logging import sys +from types import MethodType from .config import ( BuildTargetConfiguration, @@ -26,10 +27,18 @@ TestSuiteConfig, load_config, ) -from .exception import BlockingTestSuiteError +from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError from .logger import DTSLOG, getLogger -from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result -from .test_suite import get_test_suites +from .settings import SETTINGS +from .test_result import ( + BuildTargetResult, + DTSResult, + ExecutionResult, + Result, + TestCaseResult, + TestSuiteResult, +) +from .test_suite import TestSuite, get_test_suites from .testbed_model import SutNode, TGNode @@ -227,7 +236,7 @@ def _run_build_target( build_target_result.update_setup(Result.FAIL, e) else: - self._run_all_suites(sut_node, tg_node, execution, build_target_result) + self._run_test_suites(sut_node, tg_node, execution, build_target_result) finally: try: @@ -237,7 +246,7 @@ def _run_build_target( self._logger.exception("Build target teardown failed.") build_target_result.update_teardown(Result.FAIL, e) - def _run_all_suites( + def _run_test_suites( self, sut_node: SutNode, tg_node: TGNode, @@ -249,6 +258,9 @@ def _run_all_suites( The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. + If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites + in the current build target won't be executed. + Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. @@ -262,7 +274,7 @@ def _run_all_suites( execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] for test_suite_config in execution.test_suites: try: - self._run_single_suite( + self._run_test_suite_module( sut_node, tg_node, execution, build_target_result, test_suite_config ) except BlockingTestSuiteError as e: @@ -276,7 +288,7 @@ def _run_all_suites( if end_build_target: break - def _run_single_suite( + def _run_test_suite_module( self, sut_node: SutNode, tg_node: TGNode, @@ -284,11 +296,18 @@ def _run_single_suite( build_target_result: BuildTargetResult, test_suite_config: TestSuiteConfig, ) -> None: - """Run all test suites in a single test suite module. + """Set up, execute and tear down all test suites in a single test suite module. The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. + Test suite execution consists of running the discovered test cases. + A test case run consists of setup, execution and teardown of said test case. + + Record the setup and the teardown and handle failures. + + The test cases to execute are discovered when creating the :class:`TestSuite` object. + Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. @@ -313,14 +332,140 @@ def _run_single_suite( else: for test_suite_class in test_suite_classes: - test_suite = test_suite_class( - sut_node, - tg_node, - test_suite_config.test_cases, - execution.func, - build_target_result, + test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases) + + test_suite_name = test_suite.__class__.__name__ + test_suite_result = build_target_result.add_test_suite(test_suite_name) + try: + self._logger.info(f"Starting test suite setup: {test_suite_name}") + test_suite.set_up_suite() + test_suite_result.update_setup(Result.PASS) + self._logger.info(f"Test suite setup successful: {test_suite_name}") + except Exception as e: + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") + test_suite_result.update_setup(Result.ERROR, e) + + else: + self._execute_test_suite(execution.func, test_suite, test_suite_result) + + finally: + try: + test_suite.tear_down_suite() + sut_node.kill_cleanup_dpdk_apps() + test_suite_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") + self._logger.warning( + f"Test suite '{test_suite_name}' teardown failed, " + f"the next test suite may be affected." + ) + test_suite_result.update_setup(Result.ERROR, e) + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: + raise BlockingTestSuiteError(test_suite_name) + + def _execute_test_suite( + self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult + ) -> None: + """Execute all discovered test cases in `test_suite`. + + If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment + variable is set, in case of a test case failure, the test case will be executed again + until it passes or it fails that many times in addition of the first failure. + + Args: + func: Whether to execute functional test cases. + test_suite: The test suite object. + test_suite_result: The test suite level result object associated + with the current test suite. + """ + if func: + for test_case_method in test_suite._get_functional_test_cases(): + test_case_name = test_case_method.__name__ + test_case_result = test_suite_result.add_test_case(test_case_name) + all_attempts = SETTINGS.re_run + 1 + attempt_nr = 1 + self._run_test_case(test_suite, test_case_method, test_case_result) + while not test_case_result and attempt_nr < all_attempts: + attempt_nr += 1 + self._logger.info( + f"Re-running FAILED test case '{test_case_name}'. " + f"Attempt number {attempt_nr} out of {all_attempts}." + ) + self._run_test_case(test_suite, test_case_method, test_case_result) + + def _run_test_case( + self, + test_suite: TestSuite, + test_case_method: MethodType, + test_case_result: TestCaseResult, + ) -> None: + """Setup, execute and teardown a test case in `test_suite`. + + Record the result of the setup and the teardown and handle failures. + + Args: + test_suite: The test suite object. + test_case_method: The test case method. + test_case_result: The test case level result object associated + with the current test case. + """ + test_case_name = test_case_method.__name__ + + try: + # run set_up function for each case + test_suite.set_up_test_case() + test_case_result.update_setup(Result.PASS) + except SSHTimeoutError as e: + self._logger.exception(f"Test case setup FAILED: {test_case_name}") + test_case_result.update_setup(Result.FAIL, e) + except Exception as e: + self._logger.exception(f"Test case setup ERROR: {test_case_name}") + test_case_result.update_setup(Result.ERROR, e) + + else: + # run test case if setup was successful + self._execute_test_case(test_case_method, test_case_result) + + finally: + try: + test_suite.tear_down_test_case() + test_case_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test case teardown ERROR: {test_case_name}") + self._logger.warning( + f"Test case '{test_case_name}' teardown failed, " + f"the next test case may be affected." ) - test_suite.run() + test_case_result.update_teardown(Result.ERROR, e) + test_case_result.update(Result.ERROR) + + def _execute_test_case( + self, test_case_method: MethodType, test_case_result: TestCaseResult + ) -> None: + """Execute one test case, record the result and handle failures. + + Args: + test_case_method: The test case method. + test_case_result: The test case level result object associated + with the current test case. + """ + test_case_name = test_case_method.__name__ + try: + self._logger.info(f"Starting test case execution: {test_case_name}") + test_case_method() + test_case_result.update(Result.PASS) + self._logger.info(f"Test case execution PASSED: {test_case_name}") + + except TestCaseVerifyError as e: + self._logger.exception(f"Test case execution FAILED: {test_case_name}") + test_case_result.update(Result.FAIL, e) + except Exception as e: + self._logger.exception(f"Test case execution ERROR: {test_case_name}") + test_case_result.update(Result.ERROR, e) + except KeyboardInterrupt: + self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}") + test_case_result.update(Result.SKIP) + raise KeyboardInterrupt("Stop DTS") def _exit_dts(self) -> None: """Process all errors and exit with the proper exit code.""" diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index dfb391ffbd..b02fd36147 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -8,7 +8,6 @@ must be extended by subclasses which add test cases. The :class:`TestSuite` contains the basics needed by subclasses: - * Test suite and test case execution flow, * Testbed (SUT, TG) configuration, * Packet sending and verification, * Test case verification. @@ -28,27 +27,22 @@ from scapy.layers.l2 import Ether # type: ignore[import] from scapy.packet import Packet, Padding # type: ignore[import] -from .exception import ( - BlockingTestSuiteError, - ConfigurationError, - SSHTimeoutError, - TestCaseVerifyError, -) +from .exception import ConfigurationError, TestCaseVerifyError from .logger import DTSLOG, getLogger from .settings import SETTINGS -from .test_result import BuildTargetResult, Result, TestCaseResult, TestSuiteResult from .testbed_model import Port, PortLink, SutNode, TGNode from .utils import get_packet_summaries class TestSuite(object): - """The base class with methods for handling the basic flow of a test suite. + """The base class with building blocks needed by most test cases. * Test case filtering and collection, - * Test suite setup/cleanup, - * Test setup/cleanup, - * Test case execution, - * Error handling and results storage. + * Test suite setup/cleanup methods to override, + * Test case setup/cleanup methods to override, + * Test case verification, + * Testbed configuration, + * Traffic sending and verification. Test cases are implemented by subclasses. Test cases are all methods starting with ``test_``, further divided into performance test cases (starting with ``test_perf_``) @@ -60,10 +54,6 @@ class TestSuite(object): The union of both lists will be used. Any unknown test cases from the latter lists will be silently ignored. - If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable - is set, in case of a test case failure, the test case will be executed again until it passes - or it fails that many times in addition of the first failure. - The methods named ``[set_up|tear_down]_[suite|test_case]`` should be overridden in subclasses if the appropriate test suite/test case fixtures are needed. @@ -82,8 +72,6 @@ class TestSuite(object): is_blocking: ClassVar[bool] = False _logger: DTSLOG _test_cases_to_run: list[str] - _func: bool - _result: TestSuiteResult _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -99,30 +87,23 @@ def __init__( sut_node: SutNode, tg_node: TGNode, test_cases: list[str], - func: bool, - build_target_result: BuildTargetResult, ): """Initialize the test suite testbed information and basic configuration. - Process what test cases to run, create the associated - :class:`~.test_result.TestSuiteResult`, find links between ports - and set up default IP addresses to be used when configuring them. + Process what test cases to run, find links between ports and set up + default IP addresses to be used when configuring them. Args: sut_node: The SUT node where the test suite will run. tg_node: The TG node where the test suite will run. test_cases: The list of test cases to execute. If empty, all test cases will be executed. - func: Whether to run functional tests. - build_target_result: The build target result this test suite is run in. """ self.sut_node = sut_node self.tg_node = tg_node self._logger = getLogger(self.__class__.__name__) self._test_cases_to_run = test_cases self._test_cases_to_run.extend(SETTINGS.test_cases) - self._func = func - self._result = build_target_result.add_test_suite(self.__class__.__name__) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( @@ -384,62 +365,6 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: return False return True - def run(self) -> None: - """Set up, execute and tear down the whole suite. - - Test suite execution consists of running all test cases scheduled to be executed. - A test case run consists of setup, execution and teardown of said test case. - - Record the setup and the teardown and handle failures. - - The list of scheduled test cases is constructed when creating the :class:`TestSuite` object. - """ - test_suite_name = self.__class__.__name__ - - try: - self._logger.info(f"Starting test suite setup: {test_suite_name}") - self.set_up_suite() - self._result.update_setup(Result.PASS) - self._logger.info(f"Test suite setup successful: {test_suite_name}") - except Exception as e: - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") - self._result.update_setup(Result.ERROR, e) - - else: - self._execute_test_suite() - - finally: - try: - self.tear_down_suite() - self.sut_node.kill_cleanup_dpdk_apps() - self._result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") - self._logger.warning( - f"Test suite '{test_suite_name}' teardown failed, " - f"the next test suite may be affected." - ) - self._result.update_setup(Result.ERROR, e) - if len(self._result.get_errors()) > 0 and self.is_blocking: - raise BlockingTestSuiteError(test_suite_name) - - def _execute_test_suite(self) -> None: - """Execute all test cases scheduled to be executed in this suite.""" - if self._func: - for test_case_method in self._get_functional_test_cases(): - test_case_name = test_case_method.__name__ - test_case_result = self._result.add_test_case(test_case_name) - all_attempts = SETTINGS.re_run + 1 - attempt_nr = 1 - self._run_test_case(test_case_method, test_case_result) - while not test_case_result and attempt_nr < all_attempts: - attempt_nr += 1 - self._logger.info( - f"Re-running FAILED test case '{test_case_name}'. " - f"Attempt number {attempt_nr} out of {all_attempts}." - ) - self._run_test_case(test_case_method, test_case_result) - def _get_functional_test_cases(self) -> list[MethodType]: """Get all functional test cases defined in this TestSuite. @@ -471,65 +396,6 @@ def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool return match - def _run_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult - ) -> None: - """Setup, execute and teardown a test case in this suite. - - Record the result of the setup and the teardown and handle failures. - """ - test_case_name = test_case_method.__name__ - - try: - # run set_up function for each case - self.set_up_test_case() - test_case_result.update_setup(Result.PASS) - except SSHTimeoutError as e: - self._logger.exception(f"Test case setup FAILED: {test_case_name}") - test_case_result.update_setup(Result.FAIL, e) - except Exception as e: - self._logger.exception(f"Test case setup ERROR: {test_case_name}") - test_case_result.update_setup(Result.ERROR, e) - - else: - # run test case if setup was successful - self._execute_test_case(test_case_method, test_case_result) - - finally: - try: - self.tear_down_test_case() - test_case_result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test case teardown ERROR: {test_case_name}") - self._logger.warning( - f"Test case '{test_case_name}' teardown failed, " - f"the next test case may be affected." - ) - test_case_result.update_teardown(Result.ERROR, e) - test_case_result.update(Result.ERROR) - - def _execute_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult - ) -> None: - """Execute one test case, record the result and handle failures.""" - test_case_name = test_case_method.__name__ - try: - self._logger.info(f"Starting test case execution: {test_case_name}") - test_case_method() - test_case_result.update(Result.PASS) - self._logger.info(f"Test case execution PASSED: {test_case_name}") - - except TestCaseVerifyError as e: - self._logger.exception(f"Test case execution FAILED: {test_case_name}") - test_case_result.update(Result.FAIL, e) - except Exception as e: - self._logger.exception(f"Test case execution ERROR: {test_case_name}") - test_case_result.update(Result.ERROR, e) - except KeyboardInterrupt: - self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}") - test_case_result.update(Result.SKIP) - raise KeyboardInterrupt("Stop DTS") - def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: r"""Find all :class:`TestSuite`\s in a Python module. -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 3/7] dts: filter test suites in executions 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 1/7] dts: convert dts.py methods to class Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš @ 2024-02-23 7:54 ` Juraj Linkeš 2024-02-27 21:21 ` Jeremy Spewock 2024-02-23 7:54 ` [PATCH v3 4/7] dts: reorganize test result Juraj Linkeš ` (4 subsequent siblings) 7 siblings, 1 reply; 44+ messages in thread From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš We're currently filtering which test cases to run after some setup steps, such as DPDK build, have already been taken. This prohibits us to mark the test suites and cases that were supposed to be run as blocked when an earlier setup fails, as that information is not available at that time. To remedy this, move the filtering to the beginning of each execution. This is the first action taken in each execution and if we can't filter the test cases, such as due to invalid inputs, we abort the whole execution. No test suites nor cases will be marked as blocked as we don't know which were supposed to be run. On top of that, the filtering takes place in the TestSuite class, which should only concern itself with test suite and test case logic, not the processing behind the scenes. The logic has been moved to DTSRunner which should do all the processing needed to run test suites. The filtering itself introduces a few changes/assumptions which are more sensible than before: 1. Assumption: There is just one TestSuite child class in each test suite module. This was an implicit assumption before as we couldn't specify the TestSuite classes in the test run configuration, just the modules. The name of the TestSuite child class starts with "Test" and then corresponds to the name of the module with CamelCase naming. 2. Unknown test cases specified both in the test run configuration and the environment variable/command line argument are no longer silently ignored. This is a quality of life improvement for users, as they could easily be not aware of the silent ignoration. Also, a change in the code results in pycodestyle warning and error: [E] E203 whitespace before ':' [W] W503 line break before binary operator These two are not PEP8 compliant, so they're disabled. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/config/__init__.py | 24 +- dts/framework/config/conf_yaml_schema.json | 2 +- dts/framework/runner.py | 432 +++++++++++++++------ dts/framework/settings.py | 3 +- dts/framework/test_result.py | 34 ++ dts/framework/test_suite.py | 85 +--- dts/pyproject.toml | 3 + dts/tests/TestSuite_smoke_tests.py | 2 +- 8 files changed, 388 insertions(+), 197 deletions(-) diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index 62eded7f04..c6a93b3b89 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -36,7 +36,7 @@ import json import os.path import pathlib -from dataclasses import dataclass +from dataclasses import dataclass, fields from enum import auto, unique from typing import Union @@ -506,6 +506,28 @@ def from_dict( vdevs=vdevs, ) + def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration": + """Create a shallow copy with any of the fields modified. + + The only new data are those passed to this method. + The rest are copied from the object's fields calling the method. + + Args: + **kwargs: The names and types of keyword arguments are defined + by the fields of the :class:`ExecutionConfiguration` class. + + Returns: + The copied and modified execution configuration. + """ + new_config = {} + for field in fields(self): + if field.name in kwargs: + new_config[field.name] = kwargs[field.name] + else: + new_config[field.name] = getattr(self, field.name) + + return ExecutionConfiguration(**new_config) + @dataclass(slots=True, frozen=True) class Configuration: diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json index 84e45fe3c2..051b079fe4 100644 --- a/dts/framework/config/conf_yaml_schema.json +++ b/dts/framework/config/conf_yaml_schema.json @@ -197,7 +197,7 @@ }, "cases": { "type": "array", - "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.", + "description": "If specified, only this subset of test suite's test cases will be run.", "items": { "type": "string" }, diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 933685d638..e8030365ac 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -17,17 +17,27 @@ and the test case stage runs test cases individually. """ +import importlib +import inspect import logging +import re import sys from types import MethodType +from typing import Iterable from .config import ( BuildTargetConfiguration, + Configuration, ExecutionConfiguration, TestSuiteConfig, load_config, ) -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError +from .exception import ( + BlockingTestSuiteError, + ConfigurationError, + SSHTimeoutError, + TestCaseVerifyError, +) from .logger import DTSLOG, getLogger from .settings import SETTINGS from .test_result import ( @@ -37,8 +47,9 @@ Result, TestCaseResult, TestSuiteResult, + TestSuiteWithCases, ) -from .test_suite import TestSuite, get_test_suites +from .test_suite import TestSuite from .testbed_model import SutNode, TGNode @@ -59,13 +70,23 @@ class DTSRunner: given execution, the next execution begins. """ + _configuration: Configuration _logger: DTSLOG _result: DTSResult + _test_suite_class_prefix: str + _test_suite_module_prefix: str + _func_test_case_regex: str + _perf_test_case_regex: str def __init__(self): - """Initialize the instance with logger and result.""" + """Initialize the instance with configuration, logger, result and string constants.""" + self._configuration = load_config() self._logger = getLogger("DTSRunner") self._result = DTSResult(self._logger) + self._test_suite_class_prefix = "Test" + self._test_suite_module_prefix = "tests.TestSuite_" + self._func_test_case_regex = r"test_(?!perf_)" + self._perf_test_case_regex = r"test_perf_" def run(self): """Run all build targets in all executions from the test run configuration. @@ -106,29 +127,32 @@ def run(self): try: # check the python version of the server that runs dts self._check_dts_python_version() + self._result.update_setup(Result.PASS) # for all Execution sections - for execution in load_config().executions: - sut_node = sut_nodes.get(execution.system_under_test_node.name) - tg_node = tg_nodes.get(execution.traffic_generator_node.name) - + for execution in self._configuration.executions: + self._logger.info( + f"Running execution with SUT '{execution.system_under_test_node.name}'." + ) + execution_result = self._result.add_execution(execution.system_under_test_node) + # we don't want to modify the original config, so create a copy + execution_test_suites = list(execution.test_suites) + if not execution.skip_smoke_tests: + execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] try: - if not sut_node: - sut_node = SutNode(execution.system_under_test_node) - sut_nodes[sut_node.name] = sut_node - if not tg_node: - tg_node = TGNode(execution.traffic_generator_node) - tg_nodes[tg_node.name] = tg_node - self._result.update_setup(Result.PASS) + test_suites_with_cases = self._get_test_suites_with_cases( + execution_test_suites, execution.func, execution.perf + ) except Exception as e: - failed_node = execution.system_under_test_node.name - if sut_node: - failed_node = execution.traffic_generator_node.name - self._logger.exception(f"The Creation of node {failed_node} failed.") - self._result.update_setup(Result.FAIL, e) + self._logger.exception( + f"Invalid test suite configuration found: " f"{execution_test_suites}." + ) + execution_result.update_setup(Result.FAIL, e) else: - self._run_execution(sut_node, tg_node, execution) + self._connect_nodes_and_run_execution( + sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases + ) except Exception as e: self._logger.exception("An unexpected error has occurred.") @@ -163,11 +187,206 @@ def _check_dts_python_version(self) -> None: ) self._logger.warning("Please use Python >= 3.10 instead.") + def _get_test_suites_with_cases( + self, + test_suite_configs: list[TestSuiteConfig], + func: bool, + perf: bool, + ) -> list[TestSuiteWithCases]: + """Test suites with test cases discovery. + + The test suites with test cases defined in the user configuration are discovered + and stored for future use so that we don't import the modules twice and so that + the list of test suites with test cases is available for recording right away. + + Args: + test_suite_configs: Test suite configurations. + func: Whether to include functional test cases in the final list. + perf: Whether to include performance test cases in the final list. + + Returns: + The discovered test suites, each with test cases. + """ + test_suites_with_cases = [] + + for test_suite_config in test_suite_configs: + test_suite_class = self._get_test_suite_class(test_suite_config.test_suite) + test_cases = [] + func_test_cases, perf_test_cases = self._filter_test_cases( + test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases) + ) + if func: + test_cases.extend(func_test_cases) + if perf: + test_cases.extend(perf_test_cases) + + test_suites_with_cases.append( + TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases) + ) + + return test_suites_with_cases + + def _get_test_suite_class(self, module_name: str) -> type[TestSuite]: + """Find the :class:`TestSuite` class in `module_name`. + + The full module name is `module_name` prefixed with `self._test_suite_module_prefix`. + The module name is a standard filename with words separated with underscores. + Search the `module_name` for a :class:`TestSuite` class which starts + with `self._test_suite_class_prefix`, continuing with CamelCase `module_name`. + The first matching class is returned. + + The CamelCase convention is not tested, only lowercase strings are compared. + + Args: + module_name: The module name without prefix where to search for the test suite. + + Returns: + The found test suite class. + + Raises: + ConfigurationError: If the corresponding module is not found or + a valid :class:`TestSuite` is not found in the module. + """ + + def is_test_suite(object) -> bool: + """Check whether `object` is a :class:`TestSuite`. + + The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself. + + Args: + object: The object to be checked. + + Returns: + :data:`True` if `object` is a subclass of `TestSuite`. + """ + try: + if issubclass(object, TestSuite) and object is not TestSuite: + return True + except TypeError: + return False + return False + + testsuite_module_path = f"{self._test_suite_module_prefix}{module_name}" + try: + test_suite_module = importlib.import_module(testsuite_module_path) + except ModuleNotFoundError as e: + raise ConfigurationError( + f"Test suite module '{testsuite_module_path}' not found." + ) from e + + lowercase_suite_to_find = ( + f"{self._test_suite_class_prefix}{module_name.replace('_', '')}".lower() + ) + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite): + if ( + class_name.startswith(self._test_suite_class_prefix) + and lowercase_suite_to_find == class_name.lower() + ): + return class_obj + raise ConfigurationError( + f"Couldn't find any valid test suites in {test_suite_module.__name__}." + ) + + def _filter_test_cases( + self, test_suite_class: type[TestSuite], test_cases_to_run: set[str] + ) -> tuple[list[MethodType], list[MethodType]]: + """Filter `test_cases_to_run` from `test_suite_class`. + + There are two rounds of filtering if `test_cases_to_run` is not empty. + The first filters `test_cases_to_run` from all methods of `test_suite_class`. + Then the methods are separated into functional and performance test cases. + If a method matches neither the functional nor performance name prefix, it's an error. + + Args: + test_suite_class: The class of the test suite. + test_cases_to_run: Test case names to filter from `test_suite_class`. + If empty, return all matching test cases. + + Returns: + A list of test case methods that should be executed. + + Raises: + ConfigurationError: If a test case from `test_cases_to_run` is not found + or it doesn't match either the functional nor performance name prefix. + """ + func_test_cases = [] + perf_test_cases = [] + name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction) + if test_cases_to_run: + name_method_tuples = [ + (name, method) for name, method in name_method_tuples if name in test_cases_to_run + ] + if len(name_method_tuples) < len(test_cases_to_run): + missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples} + raise ConfigurationError( + f"Test cases {missing_test_cases} not found among methods " + f"of {test_suite_class.__name__}." + ) + + for test_case_name, test_case_method in name_method_tuples: + if re.match(self._func_test_case_regex, test_case_name): + func_test_cases.append(test_case_method) + elif re.match(self._perf_test_case_regex, test_case_name): + perf_test_cases.append(test_case_method) + elif test_cases_to_run: + raise ConfigurationError( + f"Method '{test_case_name}' matches neither " + f"a functional nor a performance test case name." + ) + + return func_test_cases, perf_test_cases + + def _connect_nodes_and_run_execution( + self, + sut_nodes: dict[str, SutNode], + tg_nodes: dict[str, TGNode], + execution: ExecutionConfiguration, + execution_result: ExecutionResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], + ) -> None: + """Connect nodes, then continue to run the given execution. + + Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`. + If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`, + respectively. + If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`. + + Args: + sut_nodes: A dictionary storing connected/to be connected SUT nodes. + tg_nodes: A dictionary storing connected/to be connected TG nodes. + execution: An execution's test run configuration. + execution_result: The execution's result. + test_suites_with_cases: The test suites with test cases to run. + """ + sut_node = sut_nodes.get(execution.system_under_test_node.name) + tg_node = tg_nodes.get(execution.traffic_generator_node.name) + + try: + if not sut_node: + sut_node = SutNode(execution.system_under_test_node) + sut_nodes[sut_node.name] = sut_node + if not tg_node: + tg_node = TGNode(execution.traffic_generator_node) + tg_nodes[tg_node.name] = tg_node + except Exception as e: + failed_node = execution.system_under_test_node.name + if sut_node: + failed_node = execution.traffic_generator_node.name + self._logger.exception(f"The Creation of node {failed_node} failed.") + execution_result.update_setup(Result.FAIL, e) + + else: + self._run_execution( + sut_node, tg_node, execution, execution_result, test_suites_with_cases + ) + def _run_execution( self, sut_node: SutNode, tg_node: TGNode, execution: ExecutionConfiguration, + execution_result: ExecutionResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], ) -> None: """Run the given execution. @@ -178,11 +397,11 @@ def _run_execution( sut_node: The execution's SUT node. tg_node: The execution's TG node. execution: An execution's test run configuration. + execution_result: The execution's result. + test_suites_with_cases: The test suites with test cases to run. """ self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") - execution_result = self._result.add_execution(sut_node.config) execution_result.add_sut_info(sut_node.node_info) - try: sut_node.set_up_execution(execution) execution_result.update_setup(Result.PASS) @@ -192,7 +411,10 @@ def _run_execution( else: for build_target in execution.build_targets: - self._run_build_target(sut_node, tg_node, build_target, execution, execution_result) + build_target_result = execution_result.add_build_target(build_target) + self._run_build_target( + sut_node, tg_node, build_target, build_target_result, test_suites_with_cases + ) finally: try: @@ -207,8 +429,8 @@ def _run_build_target( sut_node: SutNode, tg_node: TGNode, build_target: BuildTargetConfiguration, - execution: ExecutionConfiguration, - execution_result: ExecutionResult, + build_target_result: BuildTargetResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], ) -> None: """Run the given build target. @@ -220,11 +442,11 @@ def _run_build_target( sut_node: The execution's sut node. tg_node: The execution's tg node. build_target: A build target's test run configuration. - execution: The build target's execution's test run configuration. - execution_result: The execution level result object associated with the execution. + build_target_result: The build target level result object associated + with the current build target. + test_suites_with_cases: The test suites with test cases to run. """ self._logger.info(f"Running build target '{build_target.name}'.") - build_target_result = execution_result.add_build_target(build_target) try: sut_node.set_up_build_target(build_target) @@ -236,7 +458,7 @@ def _run_build_target( build_target_result.update_setup(Result.FAIL, e) else: - self._run_test_suites(sut_node, tg_node, execution, build_target_result) + self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases) finally: try: @@ -250,10 +472,10 @@ def _run_test_suites( self, sut_node: SutNode, tg_node: TGNode, - execution: ExecutionConfiguration, build_target_result: BuildTargetResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], ) -> None: - """Run the execution's (possibly a subset of) test suites using the current build target. + """Run `test_suites_with_cases` with the current build target. The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. @@ -264,22 +486,20 @@ def _run_test_suites( Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. - execution: The execution's test run configuration associated - with the current build target. build_target_result: The build target level result object associated with the current build target. + test_suites_with_cases: The test suites with test cases to run. """ end_build_target = False - if not execution.skip_smoke_tests: - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] - for test_suite_config in execution.test_suites: + for test_suite_with_cases in test_suites_with_cases: + test_suite_result = build_target_result.add_test_suite( + test_suite_with_cases.test_suite_class.__name__ + ) try: - self._run_test_suite_module( - sut_node, tg_node, execution, build_target_result, test_suite_config - ) + self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases) except BlockingTestSuiteError as e: self._logger.exception( - f"An error occurred within {test_suite_config.test_suite}. " + f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. " "Skipping build target..." ) self._result.add_error(e) @@ -288,15 +508,14 @@ def _run_test_suites( if end_build_target: break - def _run_test_suite_module( + def _run_test_suite( self, sut_node: SutNode, tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, - test_suite_config: TestSuiteConfig, + test_suite_result: TestSuiteResult, + test_suite_with_cases: TestSuiteWithCases, ) -> None: - """Set up, execute and tear down all test suites in a single test suite module. + """Set up, execute and tear down `test_suite_with_cases`. The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. @@ -306,92 +525,79 @@ def _run_test_suite_module( Record the setup and the teardown and handle failures. - The test cases to execute are discovered when creating the :class:`TestSuite` object. - Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. - execution: The execution's test run configuration associated - with the current build target. - build_target_result: The build target level result object associated - with the current build target. - test_suite_config: Test suite test run configuration specifying the test suite module - and possibly a subset of test cases of test suites in that module. + test_suite_result: The test suite level result object associated + with the current test suite. + test_suite_with_cases: The test suite with test cases to run. Raises: BlockingTestSuiteError: If a blocking test suite fails. """ + test_suite_name = test_suite_with_cases.test_suite_class.__name__ + test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) try: - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" - test_suite_classes = get_test_suites(full_suite_path) - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) - self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") + self._logger.info(f"Starting test suite setup: {test_suite_name}") + test_suite.set_up_suite() + test_suite_result.update_setup(Result.PASS) + self._logger.info(f"Test suite setup successful: {test_suite_name}") except Exception as e: - self._logger.exception("An error occurred when searching for test suites.") - self._result.update_setup(Result.ERROR, e) + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") + test_suite_result.update_setup(Result.ERROR, e) else: - for test_suite_class in test_suite_classes: - test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases) - - test_suite_name = test_suite.__class__.__name__ - test_suite_result = build_target_result.add_test_suite(test_suite_name) - try: - self._logger.info(f"Starting test suite setup: {test_suite_name}") - test_suite.set_up_suite() - test_suite_result.update_setup(Result.PASS) - self._logger.info(f"Test suite setup successful: {test_suite_name}") - except Exception as e: - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") - test_suite_result.update_setup(Result.ERROR, e) - - else: - self._execute_test_suite(execution.func, test_suite, test_suite_result) - - finally: - try: - test_suite.tear_down_suite() - sut_node.kill_cleanup_dpdk_apps() - test_suite_result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") - self._logger.warning( - f"Test suite '{test_suite_name}' teardown failed, " - f"the next test suite may be affected." - ) - test_suite_result.update_setup(Result.ERROR, e) - if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: - raise BlockingTestSuiteError(test_suite_name) + self._execute_test_suite( + test_suite, + test_suite_with_cases.test_cases, + test_suite_result, + ) + finally: + try: + test_suite.tear_down_suite() + sut_node.kill_cleanup_dpdk_apps() + test_suite_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") + self._logger.warning( + f"Test suite '{test_suite_name}' teardown failed, " + "the next test suite may be affected." + ) + test_suite_result.update_setup(Result.ERROR, e) + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: + raise BlockingTestSuiteError(test_suite_name) def _execute_test_suite( - self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult + self, + test_suite: TestSuite, + test_cases: Iterable[MethodType], + test_suite_result: TestSuiteResult, ) -> None: - """Execute all discovered test cases in `test_suite`. + """Execute all `test_cases` in `test_suite`. If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable is set, in case of a test case failure, the test case will be executed again until it passes or it fails that many times in addition of the first failure. Args: - func: Whether to execute functional test cases. test_suite: The test suite object. + test_cases: The list of test case methods. test_suite_result: The test suite level result object associated with the current test suite. """ - if func: - for test_case_method in test_suite._get_functional_test_cases(): - test_case_name = test_case_method.__name__ - test_case_result = test_suite_result.add_test_case(test_case_name) - all_attempts = SETTINGS.re_run + 1 - attempt_nr = 1 + for test_case_method in test_cases: + test_case_name = test_case_method.__name__ + test_case_result = test_suite_result.add_test_case(test_case_name) + all_attempts = SETTINGS.re_run + 1 + attempt_nr = 1 + self._run_test_case(test_suite, test_case_method, test_case_result) + while not test_case_result and attempt_nr < all_attempts: + attempt_nr += 1 + self._logger.info( + f"Re-running FAILED test case '{test_case_name}'. " + f"Attempt number {attempt_nr} out of {all_attempts}." + ) self._run_test_case(test_suite, test_case_method, test_case_result) - while not test_case_result and attempt_nr < all_attempts: - attempt_nr += 1 - self._logger.info( - f"Re-running FAILED test case '{test_case_name}'. " - f"Attempt number {attempt_nr} out of {all_attempts}." - ) - self._run_test_case(test_suite, test_case_method, test_case_result) def _run_test_case( self, @@ -399,7 +605,7 @@ def _run_test_case( test_case_method: MethodType, test_case_result: TestCaseResult, ) -> None: - """Setup, execute and teardown a test case in `test_suite`. + """Setup, execute and teardown `test_case_method` from `test_suite`. Record the result of the setup and the teardown and handle failures. @@ -424,7 +630,7 @@ def _run_test_case( else: # run test case if setup was successful - self._execute_test_case(test_case_method, test_case_result) + self._execute_test_case(test_suite, test_case_method, test_case_result) finally: try: @@ -440,11 +646,15 @@ def _run_test_case( test_case_result.update(Result.ERROR) def _execute_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult + self, + test_suite: TestSuite, + test_case_method: MethodType, + test_case_result: TestCaseResult, ) -> None: - """Execute one test case, record the result and handle failures. + """Execute `test_case_method` from `test_suite`, record the result and handle failures. Args: + test_suite: The test suite object. test_case_method: The test case method. test_case_result: The test case level result object associated with the current test case. @@ -452,7 +662,7 @@ def _execute_test_case( test_case_name = test_case_method.__name__ try: self._logger.info(f"Starting test case execution: {test_case_name}") - test_case_method() + test_case_method(test_suite) test_case_result.update(Result.PASS) self._logger.info(f"Test case execution PASSED: {test_case_name}") diff --git a/dts/framework/settings.py b/dts/framework/settings.py index 609c8d0e62..2b8bfbe0ed 100644 --- a/dts/framework/settings.py +++ b/dts/framework/settings.py @@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser: "--test-cases", action=_env_arg("DTS_TESTCASES"), default="", - help="[DTS_TESTCASES] Comma-separated list of test cases to execute. " - "Unknown test cases will be silently ignored.", + help="[DTS_TESTCASES] Comma-separated list of test cases to execute.", ) parser.add_argument( diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index 4467749a9d..075195fd5b 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -25,7 +25,9 @@ import os.path from collections.abc import MutableSequence +from dataclasses import dataclass from enum import Enum, auto +from types import MethodType from .config import ( OS, @@ -36,10 +38,42 @@ CPUType, NodeConfiguration, NodeInfo, + TestSuiteConfig, ) from .exception import DTSError, ErrorSeverity from .logger import DTSLOG from .settings import SETTINGS +from .test_suite import TestSuite + + +@dataclass(slots=True, frozen=True) +class TestSuiteWithCases: + """A test suite class with test case methods. + + An auxiliary class holding a test case class with test case methods. The intended use of this + class is to hold a subset of test cases (which could be all test cases) because we don't have + all the data to instantiate the class at the point of inspection. The knowledge of this subset + is needed in case an error occurs before the class is instantiated and we need to record + which test cases were blocked by the error. + + Attributes: + test_suite_class: The test suite class. + test_cases: The test case methods. + """ + + test_suite_class: type[TestSuite] + test_cases: list[MethodType] + + def create_config(self) -> TestSuiteConfig: + """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases. + + Returns: + The :class:`TestSuiteConfig` representation. + """ + return TestSuiteConfig( + test_suite=self.test_suite_class.__name__, + test_cases=[test_case.__name__ for test_case in self.test_cases], + ) class Result(Enum): diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index b02fd36147..f9fe88093e 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -11,25 +11,17 @@ * Testbed (SUT, TG) configuration, * Packet sending and verification, * Test case verification. - -The module also defines a function, :func:`get_test_suites`, -for gathering test suites from a Python module. """ -import importlib -import inspect -import re from ipaddress import IPv4Interface, IPv6Interface, ip_interface -from types import MethodType -from typing import Any, ClassVar, Union +from typing import ClassVar, Union from scapy.layers.inet import IP # type: ignore[import] from scapy.layers.l2 import Ether # type: ignore[import] from scapy.packet import Packet, Padding # type: ignore[import] -from .exception import ConfigurationError, TestCaseVerifyError +from .exception import TestCaseVerifyError from .logger import DTSLOG, getLogger -from .settings import SETTINGS from .testbed_model import Port, PortLink, SutNode, TGNode from .utils import get_packet_summaries @@ -37,7 +29,6 @@ class TestSuite(object): """The base class with building blocks needed by most test cases. - * Test case filtering and collection, * Test suite setup/cleanup methods to override, * Test case setup/cleanup methods to override, * Test case verification, @@ -71,7 +62,6 @@ class TestSuite(object): #: will block the execution of all subsequent test suites in the current build target. is_blocking: ClassVar[bool] = False _logger: DTSLOG - _test_cases_to_run: list[str] _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -86,24 +76,19 @@ def __init__( self, sut_node: SutNode, tg_node: TGNode, - test_cases: list[str], ): """Initialize the test suite testbed information and basic configuration. - Process what test cases to run, find links between ports and set up - default IP addresses to be used when configuring them. + Find links between ports and set up default IP addresses to be used when + configuring them. Args: sut_node: The SUT node where the test suite will run. tg_node: The TG node where the test suite will run. - test_cases: The list of test cases to execute. - If empty, all test cases will be executed. """ self.sut_node = sut_node self.tg_node = tg_node self._logger = getLogger(self.__class__.__name__) - self._test_cases_to_run = test_cases - self._test_cases_to_run.extend(SETTINGS.test_cases) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( @@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst: return False return True - - def _get_functional_test_cases(self) -> list[MethodType]: - """Get all functional test cases defined in this TestSuite. - - Returns: - The list of functional test cases of this TestSuite. - """ - return self._get_test_cases(r"test_(?!perf_)") - - def _get_test_cases(self, test_case_regex: str) -> list[MethodType]: - """Return a list of test cases matching test_case_regex. - - Returns: - The list of test cases matching test_case_regex of this TestSuite. - """ - self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.") - filtered_test_cases = [] - for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod): - if self._should_be_executed(test_case_name, test_case_regex): - filtered_test_cases.append(test_case) - cases_str = ", ".join((x.__name__ for x in filtered_test_cases)) - self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.") - return filtered_test_cases - - def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool: - """Check whether the test case should be scheduled to be executed.""" - match = bool(re.match(test_case_regex, test_case_name)) - if self._test_cases_to_run: - return match and test_case_name in self._test_cases_to_run - - return match - - -def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: - r"""Find all :class:`TestSuite`\s in a Python module. - - Args: - testsuite_module_path: The path to the Python module. - - Returns: - The list of :class:`TestSuite`\s found within the Python module. - - Raises: - ConfigurationError: The test suite module was not found. - """ - - def is_test_suite(object: Any) -> bool: - try: - if issubclass(object, TestSuite) and object is not TestSuite: - return True - except TypeError: - return False - return False - - try: - testcase_module = importlib.import_module(testsuite_module_path) - except ModuleNotFoundError as e: - raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e - return [ - test_suite_class - for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite) - ] diff --git a/dts/pyproject.toml b/dts/pyproject.toml index 28bd970ae4..8eb92b4f11 100644 --- a/dts/pyproject.toml +++ b/dts/pyproject.toml @@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes" format = "pylint" max_line_length = 100 +[tool.pylama.linter.pycodestyle] +ignore = "E203,W503" + [tool.pylama.linter.pydocstyle] convention = "google" diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py index 5e2bac14bd..7b2a0e97f8 100644 --- a/dts/tests/TestSuite_smoke_tests.py +++ b/dts/tests/TestSuite_smoke_tests.py @@ -21,7 +21,7 @@ from framework.utils import REGEX_FOR_PCI_ADDRESS -class SmokeTests(TestSuite): +class TestSmokeTests(TestSuite): """DPDK and infrastructure smoke test suite. The test cases validate the most basic DPDK functionality needed for all other test suites. -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v3 3/7] dts: filter test suites in executions 2024-02-23 7:54 ` [PATCH v3 3/7] dts: filter test suites in executions Juraj Linkeš @ 2024-02-27 21:21 ` Jeremy Spewock 2024-02-28 9:16 ` Juraj Linkeš 0 siblings, 1 reply; 44+ messages in thread From: Jeremy Spewock @ 2024-02-27 21:21 UTC (permalink / raw) To: Juraj Linkeš Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev <snip> > + > + lowercase_suite_to_find = ( > + f"{self._test_suite_class_prefix}{module_name.replace('_', '')}".lower() > + ) > + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite): > + if ( > + class_name.startswith(self._test_suite_class_prefix) Is this line required? When we check later if the lowercase_suite_to_find is equal, we know that this variable we made starts with the designated prefix because that's just how we made it. It seems redundant because they could not be equal if the class name didn't start with the prefix. Is the idea behind this a type of optimization because this makes us ignore anything that doesn't have the proper prefix before going further? > > + and lowercase_suite_to_find == class_name.lower() > + ): > + return class_obj > + raise ConfigurationError( > + f"Couldn't find any valid test suites in {test_suite_module.__name__}." > + ) <snip> ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v3 3/7] dts: filter test suites in executions 2024-02-27 21:21 ` Jeremy Spewock @ 2024-02-28 9:16 ` Juraj Linkeš 0 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-28 9:16 UTC (permalink / raw) To: Jeremy Spewock Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev On Tue, Feb 27, 2024 at 10:22 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote: > > <snip> > > + > > + lowercase_suite_to_find = ( > > + f"{self._test_suite_class_prefix}{module_name.replace('_', '')}".lower() > > + ) > > + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite): > > + if ( > > + class_name.startswith(self._test_suite_class_prefix) > > Is this line required? When we check later if the > lowercase_suite_to_find is equal, we know that this variable we made > starts with the designated prefix because that's just how we made it. > It seems redundant because they could not be equal if the class name > didn't start with the prefix. Is the idea behind this a type of > optimization because this makes us ignore anything that doesn't have > the proper prefix before going further? > There is a slight difference. The class_name.startswith(self._test_suite_class_prefix) condition checks uppercase characters as well. I'm not sure it's actually worth it. Maybe we could change the name check to also check for the CamelCase convention. I'll look into that. > > > > + and lowercase_suite_to_find == class_name.lower() > > + ): > > + return class_obj > > + raise ConfigurationError( > > + f"Couldn't find any valid test suites in {test_suite_module.__name__}." > > + ) > <snip> ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 4/7] dts: reorganize test result 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš ` (2 preceding siblings ...) 2024-02-23 7:54 ` [PATCH v3 3/7] dts: filter test suites in executions Juraj Linkeš @ 2024-02-23 7:54 ` Juraj Linkeš 2024-02-23 7:55 ` [PATCH v3 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš ` (3 subsequent siblings) 7 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-23 7:54 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš The current order of Result classes in the test_suite.py module is guided by the needs of type hints, which is not as intuitively readable as ordering them by the occurrences in code. The order goes from the topmost level to lowermost: BaseResult DTSResult ExecutionResult BuildTargetResult TestSuiteResult TestCaseResult This is the same order as they're used in the runner module and they're also used in the same order between themselves in the test_result module. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/test_result.py | 411 ++++++++++++++++++----------------- 1 file changed, 206 insertions(+), 205 deletions(-) diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index 075195fd5b..abdbafab10 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -28,6 +28,7 @@ from dataclasses import dataclass from enum import Enum, auto from types import MethodType +from typing import Union from .config import ( OS, @@ -129,58 +130,6 @@ def __bool__(self) -> bool: return bool(self.result) -class Statistics(dict): - """How many test cases ended in which result state along some other basic information. - - Subclassing :class:`dict` provides a convenient way to format the data. - - The data are stored in the following keys: - - * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases. - * **DPDK VERSION** (:class:`str`) -- The tested DPDK version. - """ - - def __init__(self, dpdk_version: str | None): - """Extend the constructor with keys in which the data are stored. - - Args: - dpdk_version: The version of tested DPDK. - """ - super(Statistics, self).__init__() - for result in Result: - self[result.name] = 0 - self["PASS RATE"] = 0.0 - self["DPDK VERSION"] = dpdk_version - - def __iadd__(self, other: Result) -> "Statistics": - """Add a Result to the final count. - - Example: - stats: Statistics = Statistics() # empty Statistics - stats += Result.PASS # add a Result to `stats` - - Args: - other: The Result to add to this statistics object. - - Returns: - The modified statistics object. - """ - self[other.name] += 1 - self["PASS RATE"] = ( - float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result) - ) - return self - - def __str__(self) -> str: - """Each line contains the formatted key = value pair.""" - stats_str = "" - for key, value in self.items(): - stats_str += f"{key:<12} = {value}\n" - # according to docs, we should use \n when writing to text files - # on all platforms - return stats_str - - class BaseResult(object): """Common data and behavior of DTS results. @@ -245,7 +194,7 @@ def get_errors(self) -> list[Exception]: """ return self._get_setup_teardown_errors() + self._get_inner_errors() - def add_stats(self, statistics: Statistics) -> None: + def add_stats(self, statistics: "Statistics") -> None: """Collate stats from the whole result hierarchy. Args: @@ -255,91 +204,149 @@ def add_stats(self, statistics: Statistics) -> None: inner_result.add_stats(statistics) -class TestCaseResult(BaseResult, FixtureResult): - r"""The test case specific result. +class DTSResult(BaseResult): + """Stores environment information and test results from a DTS run. - Stores the result of the actual test case. This is done by adding an extra superclass - in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and - the class is itself a record of the test case. + * Execution level information, such as testbed and the test suite list, + * Build target level information, such as compiler, target OS and cpu, + * Test suite and test case results, + * All errors that are caught and recorded during DTS execution. + + The information is stored hierarchically. This is the first level of the hierarchy + and as such is where the data form the whole hierarchy is collated or processed. + + The internal list stores the results of all executions. Attributes: - test_case_name: The test case name. + dpdk_version: The DPDK version to record. """ - test_case_name: str + dpdk_version: str | None + _logger: DTSLOG + _errors: list[Exception] + _return_code: ErrorSeverity + _stats_result: Union["Statistics", None] + _stats_filename: str - def __init__(self, test_case_name: str): - """Extend the constructor with `test_case_name`. + def __init__(self, logger: DTSLOG): + """Extend the constructor with top-level specifics. Args: - test_case_name: The test case's name. + logger: The logger instance the whole result will use. """ - super(TestCaseResult, self).__init__() - self.test_case_name = test_case_name + super(DTSResult, self).__init__() + self.dpdk_version = None + self._logger = logger + self._errors = [] + self._return_code = ErrorSeverity.NO_ERR + self._stats_result = None + self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") - def update(self, result: Result, error: Exception | None = None) -> None: - """Update the test case result. + def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult": + """Add and return the inner result (execution). - This updates the result of the test case itself and doesn't affect - the results of the setup and teardown steps in any way. + Args: + sut_node: The SUT node's test run configuration. + + Returns: + The execution's result. + """ + execution_result = ExecutionResult(sut_node) + self._inner_results.append(execution_result) + return execution_result + + def add_error(self, error: Exception) -> None: + """Record an error that occurred outside any execution. Args: - result: The result of the test case. - error: The error that occurred in case of a failure. + error: The exception to record. """ - self.result = result - self.error = error + self._errors.append(error) - def _get_inner_errors(self) -> list[Exception]: - if self.error: - return [self.error] - return [] + def process(self) -> None: + """Process the data after a whole DTS run. - def add_stats(self, statistics: Statistics) -> None: - r"""Add the test case result to statistics. + The data is added to inner objects during runtime and this object is not updated + at that time. This requires us to process the inner data after it's all been gathered. - The base method goes through the hierarchy recursively and this method is here to stop - the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree. + The processing gathers all errors and the statistics of test case results. + """ + self._errors += self.get_errors() + if self._errors and self._logger: + self._logger.debug("Summary of errors:") + for error in self._errors: + self._logger.debug(repr(error)) - Args: - statistics: The :class:`Statistics` object where the stats will be added. + self._stats_result = Statistics(self.dpdk_version) + self.add_stats(self._stats_result) + with open(self._stats_filename, "w+") as stats_file: + stats_file.write(str(self._stats_result)) + + def get_return_code(self) -> int: + """Go through all stored Exceptions and return the final DTS error code. + + Returns: + The highest error code found. """ - statistics += self.result + for error in self._errors: + error_return_code = ErrorSeverity.GENERIC_ERR + if isinstance(error, DTSError): + error_return_code = error.severity - def __bool__(self) -> bool: - """The test case passed only if setup, teardown and the test case itself passed.""" - return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) + if error_return_code > self._return_code: + self._return_code = error_return_code + return int(self._return_code) -class TestSuiteResult(BaseResult): - """The test suite specific result. - The internal list stores the results of all test cases in a given test suite. +class ExecutionResult(BaseResult): + """The execution specific result. + + The internal list stores the results of all build targets in a given execution. Attributes: - suite_name: The test suite name. + sut_node: The SUT node used in the execution. + sut_os_name: The operating system of the SUT node. + sut_os_version: The operating system version of the SUT node. + sut_kernel_version: The operating system kernel version of the SUT node. """ - suite_name: str + sut_node: NodeConfiguration + sut_os_name: str + sut_os_version: str + sut_kernel_version: str - def __init__(self, suite_name: str): - """Extend the constructor with `suite_name`. + def __init__(self, sut_node: NodeConfiguration): + """Extend the constructor with the `sut_node`'s config. Args: - suite_name: The test suite's name. + sut_node: The SUT node's test run configuration used in the execution. """ - super(TestSuiteResult, self).__init__() - self.suite_name = suite_name + super(ExecutionResult, self).__init__() + self.sut_node = sut_node - def add_test_case(self, test_case_name: str) -> TestCaseResult: - """Add and return the inner result (test case). + def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult": + """Add and return the inner result (build target). + + Args: + build_target: The build target's test run configuration. Returns: - The test case's result. + The build target's result. """ - test_case_result = TestCaseResult(test_case_name) - self._inner_results.append(test_case_result) - return test_case_result + build_target_result = BuildTargetResult(build_target) + self._inner_results.append(build_target_result) + return build_target_result + + def add_sut_info(self, sut_info: NodeInfo) -> None: + """Add SUT information gathered at runtime. + + Args: + sut_info: The additional SUT node information. + """ + self.sut_os_name = sut_info.os_name + self.sut_os_version = sut_info.os_version + self.sut_kernel_version = sut_info.kernel_version class BuildTargetResult(BaseResult): @@ -386,7 +393,7 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None: self.compiler_version = versions.compiler_version self.dpdk_version = versions.dpdk_version - def add_test_suite(self, test_suite_name: str) -> TestSuiteResult: + def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult": """Add and return the inner result (test suite). Returns: @@ -397,146 +404,140 @@ def add_test_suite(self, test_suite_name: str) -> TestSuiteResult: return test_suite_result -class ExecutionResult(BaseResult): - """The execution specific result. +class TestSuiteResult(BaseResult): + """The test suite specific result. - The internal list stores the results of all build targets in a given execution. + The internal list stores the results of all test cases in a given test suite. Attributes: - sut_node: The SUT node used in the execution. - sut_os_name: The operating system of the SUT node. - sut_os_version: The operating system version of the SUT node. - sut_kernel_version: The operating system kernel version of the SUT node. + suite_name: The test suite name. """ - sut_node: NodeConfiguration - sut_os_name: str - sut_os_version: str - sut_kernel_version: str + suite_name: str - def __init__(self, sut_node: NodeConfiguration): - """Extend the constructor with the `sut_node`'s config. + def __init__(self, suite_name: str): + """Extend the constructor with `suite_name`. Args: - sut_node: The SUT node's test run configuration used in the execution. + suite_name: The test suite's name. """ - super(ExecutionResult, self).__init__() - self.sut_node = sut_node - - def add_build_target(self, build_target: BuildTargetConfiguration) -> BuildTargetResult: - """Add and return the inner result (build target). + super(TestSuiteResult, self).__init__() + self.suite_name = suite_name - Args: - build_target: The build target's test run configuration. + def add_test_case(self, test_case_name: str) -> "TestCaseResult": + """Add and return the inner result (test case). Returns: - The build target's result. - """ - build_target_result = BuildTargetResult(build_target) - self._inner_results.append(build_target_result) - return build_target_result - - def add_sut_info(self, sut_info: NodeInfo) -> None: - """Add SUT information gathered at runtime. - - Args: - sut_info: The additional SUT node information. + The test case's result. """ - self.sut_os_name = sut_info.os_name - self.sut_os_version = sut_info.os_version - self.sut_kernel_version = sut_info.kernel_version + test_case_result = TestCaseResult(test_case_name) + self._inner_results.append(test_case_result) + return test_case_result -class DTSResult(BaseResult): - """Stores environment information and test results from a DTS run. - - * Execution level information, such as testbed and the test suite list, - * Build target level information, such as compiler, target OS and cpu, - * Test suite and test case results, - * All errors that are caught and recorded during DTS execution. - - The information is stored hierarchically. This is the first level of the hierarchy - and as such is where the data form the whole hierarchy is collated or processed. +class TestCaseResult(BaseResult, FixtureResult): + r"""The test case specific result. - The internal list stores the results of all executions. + Stores the result of the actual test case. This is done by adding an extra superclass + in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and + the class is itself a record of the test case. Attributes: - dpdk_version: The DPDK version to record. + test_case_name: The test case name. """ - dpdk_version: str | None - _logger: DTSLOG - _errors: list[Exception] - _return_code: ErrorSeverity - _stats_result: Statistics | None - _stats_filename: str + test_case_name: str - def __init__(self, logger: DTSLOG): - """Extend the constructor with top-level specifics. + def __init__(self, test_case_name: str): + """Extend the constructor with `test_case_name`. Args: - logger: The logger instance the whole result will use. + test_case_name: The test case's name. """ - super(DTSResult, self).__init__() - self.dpdk_version = None - self._logger = logger - self._errors = [] - self._return_code = ErrorSeverity.NO_ERR - self._stats_result = None - self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") + super(TestCaseResult, self).__init__() + self.test_case_name = test_case_name - def add_execution(self, sut_node: NodeConfiguration) -> ExecutionResult: - """Add and return the inner result (execution). + def update(self, result: Result, error: Exception | None = None) -> None: + """Update the test case result. - Args: - sut_node: The SUT node's test run configuration. + This updates the result of the test case itself and doesn't affect + the results of the setup and teardown steps in any way. - Returns: - The execution's result. + Args: + result: The result of the test case. + error: The error that occurred in case of a failure. """ - execution_result = ExecutionResult(sut_node) - self._inner_results.append(execution_result) - return execution_result + self.result = result + self.error = error - def add_error(self, error: Exception) -> None: - """Record an error that occurred outside any execution. + def _get_inner_errors(self) -> list[Exception]: + if self.error: + return [self.error] + return [] + + def add_stats(self, statistics: "Statistics") -> None: + r"""Add the test case result to statistics. + + The base method goes through the hierarchy recursively and this method is here to stop + the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree. Args: - error: The exception to record. + statistics: The :class:`Statistics` object where the stats will be added. """ - self._errors.append(error) + statistics += self.result - def process(self) -> None: - """Process the data after a whole DTS run. + def __bool__(self) -> bool: + """The test case passed only if setup, teardown and the test case itself passed.""" + return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) - The data is added to inner objects during runtime and this object is not updated - at that time. This requires us to process the inner data after it's all been gathered. - The processing gathers all errors and the statistics of test case results. +class Statistics(dict): + """How many test cases ended in which result state along some other basic information. + + Subclassing :class:`dict` provides a convenient way to format the data. + + The data are stored in the following keys: + + * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases. + * **DPDK VERSION** (:class:`str`) -- The tested DPDK version. + """ + + def __init__(self, dpdk_version: str | None): + """Extend the constructor with keys in which the data are stored. + + Args: + dpdk_version: The version of tested DPDK. """ - self._errors += self.get_errors() - if self._errors and self._logger: - self._logger.debug("Summary of errors:") - for error in self._errors: - self._logger.debug(repr(error)) + super(Statistics, self).__init__() + for result in Result: + self[result.name] = 0 + self["PASS RATE"] = 0.0 + self["DPDK VERSION"] = dpdk_version - self._stats_result = Statistics(self.dpdk_version) - self.add_stats(self._stats_result) - with open(self._stats_filename, "w+") as stats_file: - stats_file.write(str(self._stats_result)) + def __iadd__(self, other: Result) -> "Statistics": + """Add a Result to the final count. - def get_return_code(self) -> int: - """Go through all stored Exceptions and return the final DTS error code. + Example: + stats: Statistics = Statistics() # empty Statistics + stats += Result.PASS # add a Result to `stats` + + Args: + other: The Result to add to this statistics object. Returns: - The highest error code found. + The modified statistics object. """ - for error in self._errors: - error_return_code = ErrorSeverity.GENERIC_ERR - if isinstance(error, DTSError): - error_return_code = error.severity - - if error_return_code > self._return_code: - self._return_code = error_return_code + self[other.name] += 1 + self["PASS RATE"] = ( + float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result) + ) + return self - return int(self._return_code) + def __str__(self) -> str: + """Each line contains the formatted key = value pair.""" + stats_str = "" + for key, value in self.items(): + stats_str += f"{key:<12} = {value}\n" + # according to docs, we should use \n when writing to text files + # on all platforms + return stats_str -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 5/7] dts: block all test cases when earlier setup fails 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš ` (3 preceding siblings ...) 2024-02-23 7:54 ` [PATCH v3 4/7] dts: reorganize test result Juraj Linkeš @ 2024-02-23 7:55 ` Juraj Linkeš 2024-02-23 7:55 ` [PATCH v3 6/7] dts: refactor logging configuration Juraj Linkeš ` (2 subsequent siblings) 7 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-23 7:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš In case of a failure before a test suite, the child results will be recursively recorded as blocked, giving us a full report which was missing previously. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/runner.py | 21 ++-- dts/framework/test_result.py | 186 +++++++++++++++++++++++++---------- 2 files changed, 148 insertions(+), 59 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index e8030365ac..511f8be064 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -60,13 +60,15 @@ class DTSRunner: Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult` or one of its subclasses. The test case results are also recorded. - If an error occurs, the current stage is aborted, the error is recorded and the run continues in - the next iteration of the same stage. The return code is the highest `severity` of all + If an error occurs, the current stage is aborted, the error is recorded, everything in + the inner stages is marked as blocked and the run continues in the next iteration + of the same stage. The return code is the highest `severity` of all :class:`~.framework.exception.DTSError`\s. Example: - An error occurs in a build target setup. The current build target is aborted and the run - continues with the next build target. If the errored build target was the last one in the + An error occurs in a build target setup. The current build target is aborted, + all test suites and their test cases are marked as blocked and the run continues + with the next build target. If the errored build target was the last one in the given execution, the next execution begins. """ @@ -100,6 +102,10 @@ def run(self): test case within the test suite is set up, executed and torn down. After all test cases have been executed, the test suite is torn down and the next build target will be tested. + In order to properly mark test suites and test cases as blocked in case of a failure, + we need to have discovered which test suites and test cases to run before any failures + happen. The discovery happens at the earliest point at the start of each execution. + All the nested steps look like this: #. Execution setup @@ -134,7 +140,7 @@ def run(self): self._logger.info( f"Running execution with SUT '{execution.system_under_test_node.name}'." ) - execution_result = self._result.add_execution(execution.system_under_test_node) + execution_result = self._result.add_execution(execution) # we don't want to modify the original config, so create a copy execution_test_suites = list(execution.test_suites) if not execution.skip_smoke_tests: @@ -143,6 +149,7 @@ def run(self): test_suites_with_cases = self._get_test_suites_with_cases( execution_test_suites, execution.func, execution.perf ) + execution_result.test_suites_with_cases = test_suites_with_cases except Exception as e: self._logger.exception( f"Invalid test suite configuration found: " f"{execution_test_suites}." @@ -492,9 +499,7 @@ def _run_test_suites( """ end_build_target = False for test_suite_with_cases in test_suites_with_cases: - test_suite_result = build_target_result.add_test_suite( - test_suite_with_cases.test_suite_class.__name__ - ) + test_suite_result = build_target_result.add_test_suite(test_suite_with_cases) try: self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases) except BlockingTestSuiteError as e: diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index abdbafab10..eedb2d20ee 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -37,7 +37,7 @@ BuildTargetInfo, Compiler, CPUType, - NodeConfiguration, + ExecutionConfiguration, NodeInfo, TestSuiteConfig, ) @@ -88,6 +88,8 @@ class Result(Enum): ERROR = auto() #: SKIP = auto() + #: + BLOCK = auto() def __bool__(self) -> bool: """Only PASS is True.""" @@ -141,21 +143,26 @@ class BaseResult(object): Attributes: setup_result: The result of the setup of the particular stage. teardown_result: The results of the teardown of the particular stage. + child_results: The results of the descendants in the results hierarchy. """ setup_result: FixtureResult teardown_result: FixtureResult - _inner_results: MutableSequence["BaseResult"] + child_results: MutableSequence["BaseResult"] def __init__(self): """Initialize the constructor.""" self.setup_result = FixtureResult() self.teardown_result = FixtureResult() - self._inner_results = [] + self.child_results = [] def update_setup(self, result: Result, error: Exception | None = None) -> None: """Store the setup result. + If the result is :attr:`~Result.BLOCK`, :attr:`~Result.ERROR` or :attr:`~Result.FAIL`, + then the corresponding child results in result hierarchy + are also marked with :attr:`~Result.BLOCK`. + Args: result: The result of the setup. error: The error that occurred in case of a failure. @@ -163,6 +170,16 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None: self.setup_result.result = result self.setup_result.error = error + if result in [Result.BLOCK, Result.ERROR, Result.FAIL]: + self.update_teardown(Result.BLOCK) + self._block_result() + + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed. + + The blocking of child results should be done in overloaded methods. + """ + def update_teardown(self, result: Result, error: Exception | None = None) -> None: """Store the teardown result. @@ -181,10 +198,8 @@ def _get_setup_teardown_errors(self) -> list[Exception]: errors.append(self.teardown_result.error) return errors - def _get_inner_errors(self) -> list[Exception]: - return [ - error for inner_result in self._inner_results for error in inner_result.get_errors() - ] + def _get_child_errors(self) -> list[Exception]: + return [error for child_result in self.child_results for error in child_result.get_errors()] def get_errors(self) -> list[Exception]: """Compile errors from the whole result hierarchy. @@ -192,7 +207,7 @@ def get_errors(self) -> list[Exception]: Returns: The errors from setup, teardown and all errors found in the whole result hierarchy. """ - return self._get_setup_teardown_errors() + self._get_inner_errors() + return self._get_setup_teardown_errors() + self._get_child_errors() def add_stats(self, statistics: "Statistics") -> None: """Collate stats from the whole result hierarchy. @@ -200,8 +215,8 @@ def add_stats(self, statistics: "Statistics") -> None: Args: statistics: The :class:`Statistics` object where the stats will be collated. """ - for inner_result in self._inner_results: - inner_result.add_stats(statistics) + for child_result in self.child_results: + child_result.add_stats(statistics) class DTSResult(BaseResult): @@ -242,18 +257,18 @@ def __init__(self, logger: DTSLOG): self._stats_result = None self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") - def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult": - """Add and return the inner result (execution). + def add_execution(self, execution: ExecutionConfiguration) -> "ExecutionResult": + """Add and return the child result (execution). Args: - sut_node: The SUT node's test run configuration. + execution: The execution's test run configuration. Returns: The execution's result. """ - execution_result = ExecutionResult(sut_node) - self._inner_results.append(execution_result) - return execution_result + result = ExecutionResult(execution) + self.child_results.append(result) + return result def add_error(self, error: Exception) -> None: """Record an error that occurred outside any execution. @@ -266,8 +281,8 @@ def add_error(self, error: Exception) -> None: def process(self) -> None: """Process the data after a whole DTS run. - The data is added to inner objects during runtime and this object is not updated - at that time. This requires us to process the inner data after it's all been gathered. + The data is added to child objects during runtime and this object is not updated + at that time. This requires us to process the child data after it's all been gathered. The processing gathers all errors and the statistics of test case results. """ @@ -305,28 +320,30 @@ class ExecutionResult(BaseResult): The internal list stores the results of all build targets in a given execution. Attributes: - sut_node: The SUT node used in the execution. sut_os_name: The operating system of the SUT node. sut_os_version: The operating system version of the SUT node. sut_kernel_version: The operating system kernel version of the SUT node. """ - sut_node: NodeConfiguration sut_os_name: str sut_os_version: str sut_kernel_version: str + _config: ExecutionConfiguration + _parent_result: DTSResult + _test_suites_with_cases: list[TestSuiteWithCases] - def __init__(self, sut_node: NodeConfiguration): - """Extend the constructor with the `sut_node`'s config. + def __init__(self, execution: ExecutionConfiguration): + """Extend the constructor with the execution's config and DTSResult. Args: - sut_node: The SUT node's test run configuration used in the execution. + execution: The execution's test run configuration. """ super(ExecutionResult, self).__init__() - self.sut_node = sut_node + self._config = execution + self._test_suites_with_cases = [] def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult": - """Add and return the inner result (build target). + """Add and return the child result (build target). Args: build_target: The build target's test run configuration. @@ -334,9 +351,34 @@ def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTarg Returns: The build target's result. """ - build_target_result = BuildTargetResult(build_target) - self._inner_results.append(build_target_result) - return build_target_result + result = BuildTargetResult( + self._test_suites_with_cases, + build_target, + ) + self.child_results.append(result) + return result + + @property + def test_suites_with_cases(self) -> list[TestSuiteWithCases]: + """The test suites with test cases to be executed in this execution. + + The test suites can only be assigned once. + + Returns: + The list of test suites with test cases. If an error occurs between + the initialization of :class:`ExecutionResult` and assigning test cases to the instance, + return an empty list, representing that we don't know what to execute. + """ + return self._test_suites_with_cases + + @test_suites_with_cases.setter + def test_suites_with_cases(self, test_suites_with_cases: list[TestSuiteWithCases]) -> None: + if self._test_suites_with_cases: + raise ValueError( + "Attempted to assign test suites to an execution result " + "which already has test suites." + ) + self._test_suites_with_cases = test_suites_with_cases def add_sut_info(self, sut_info: NodeInfo) -> None: """Add SUT information gathered at runtime. @@ -348,6 +390,12 @@ def add_sut_info(self, sut_info: NodeInfo) -> None: self.sut_os_version = sut_info.os_version self.sut_kernel_version = sut_info.kernel_version + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + for build_target in self._config.build_targets: + child_result = self.add_build_target(build_target) + child_result.update_setup(Result.BLOCK) + class BuildTargetResult(BaseResult): """The build target specific result. @@ -369,11 +417,17 @@ class BuildTargetResult(BaseResult): compiler: Compiler compiler_version: str | None dpdk_version: str | None + _test_suites_with_cases: list[TestSuiteWithCases] - def __init__(self, build_target: BuildTargetConfiguration): - """Extend the constructor with the `build_target`'s build target config. + def __init__( + self, + test_suites_with_cases: list[TestSuiteWithCases], + build_target: BuildTargetConfiguration, + ): + """Extend the constructor with the build target's config and ExecutionResult. Args: + test_suites_with_cases: The test suites with test cases to be run in this build target. build_target: The build target's test run configuration. """ super(BuildTargetResult, self).__init__() @@ -383,6 +437,23 @@ def __init__(self, build_target: BuildTargetConfiguration): self.compiler = build_target.compiler self.compiler_version = None self.dpdk_version = None + self._test_suites_with_cases = test_suites_with_cases + + def add_test_suite( + self, + test_suite_with_cases: TestSuiteWithCases, + ) -> "TestSuiteResult": + """Add and return the child result (test suite). + + Args: + test_suite_with_cases: The test suite with test cases. + + Returns: + The test suite's result. + """ + result = TestSuiteResult(test_suite_with_cases) + self.child_results.append(result) + return result def add_build_target_info(self, versions: BuildTargetInfo) -> None: """Add information about the build target gathered at runtime. @@ -393,15 +464,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None: self.compiler_version = versions.compiler_version self.dpdk_version = versions.dpdk_version - def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult": - """Add and return the inner result (test suite). - - Returns: - The test suite's result. - """ - test_suite_result = TestSuiteResult(test_suite_name) - self._inner_results.append(test_suite_result) - return test_suite_result + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + for test_suite_with_cases in self._test_suites_with_cases: + child_result = self.add_test_suite(test_suite_with_cases) + child_result.update_setup(Result.BLOCK) class TestSuiteResult(BaseResult): @@ -410,29 +477,42 @@ class TestSuiteResult(BaseResult): The internal list stores the results of all test cases in a given test suite. Attributes: - suite_name: The test suite name. + test_suite_name: The test suite name. """ - suite_name: str + test_suite_name: str + _test_suite_with_cases: TestSuiteWithCases + _parent_result: BuildTargetResult + _child_configs: list[str] - def __init__(self, suite_name: str): - """Extend the constructor with `suite_name`. + def __init__(self, test_suite_with_cases: TestSuiteWithCases): + """Extend the constructor with test suite's config and BuildTargetResult. Args: - suite_name: The test suite's name. + test_suite_with_cases: The test suite with test cases. """ super(TestSuiteResult, self).__init__() - self.suite_name = suite_name + self.test_suite_name = test_suite_with_cases.test_suite_class.__name__ + self._test_suite_with_cases = test_suite_with_cases def add_test_case(self, test_case_name: str) -> "TestCaseResult": - """Add and return the inner result (test case). + """Add and return the child result (test case). + + Args: + test_case_name: The name of the test case. Returns: The test case's result. """ - test_case_result = TestCaseResult(test_case_name) - self._inner_results.append(test_case_result) - return test_case_result + result = TestCaseResult(test_case_name) + self.child_results.append(result) + return result + + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + for test_case_method in self._test_suite_with_cases.test_cases: + child_result = self.add_test_case(test_case_method.__name__) + child_result.update_setup(Result.BLOCK) class TestCaseResult(BaseResult, FixtureResult): @@ -449,7 +529,7 @@ class TestCaseResult(BaseResult, FixtureResult): test_case_name: str def __init__(self, test_case_name: str): - """Extend the constructor with `test_case_name`. + """Extend the constructor with test case's name and TestSuiteResult. Args: test_case_name: The test case's name. @@ -470,7 +550,7 @@ def update(self, result: Result, error: Exception | None = None) -> None: self.result = result self.error = error - def _get_inner_errors(self) -> list[Exception]: + def _get_child_errors(self) -> list[Exception]: if self.error: return [self.error] return [] @@ -486,6 +566,10 @@ def add_stats(self, statistics: "Statistics") -> None: """ statistics += self.result + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + self.update(Result.BLOCK) + def __bool__(self) -> bool: """The test case passed only if setup, teardown and the test case itself passed.""" return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 6/7] dts: refactor logging configuration 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš ` (4 preceding siblings ...) 2024-02-23 7:55 ` [PATCH v3 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš @ 2024-02-23 7:55 ` Juraj Linkeš 2024-02-23 7:55 ` [PATCH v3 7/7] dts: improve test suite and case filtering Juraj Linkeš 2024-02-27 21:24 ` [PATCH v3 0/7] test case blocking and logging Jeremy Spewock 7 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-23 7:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš Remove unused parts of the code and add useful features: 1. Add DTS execution stages such as execution and test suite to better identify where in the DTS lifecycle we are when investigating logs, 2. Logging to separate files in specific stages, which is mainly useful for having test suite logs in additional separate files. 3. Remove the dependence on the settings module which enhances the usefulness of the logger module, as it can now be imported in more modules. The execution stages and the files to log to are the same for all DTS loggers. To achieve this, we have one DTS root logger which should be used for handling stage switching and all other loggers are children of this DTS root logger. The DTS root logger is the one where we change the behavior of all loggers (the stage and which files to log to) and the child loggers just log messages under a different name. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/logger.py | 236 +++++++++++------- dts/framework/remote_session/__init__.py | 6 +- .../interactive_remote_session.py | 6 +- .../remote_session/interactive_shell.py | 6 +- .../remote_session/remote_session.py | 8 +- dts/framework/runner.py | 19 +- dts/framework/test_result.py | 6 +- dts/framework/test_suite.py | 6 +- dts/framework/testbed_model/node.py | 11 +- dts/framework/testbed_model/os_session.py | 7 +- .../traffic_generator/traffic_generator.py | 6 +- dts/main.py | 3 - 12 files changed, 184 insertions(+), 136 deletions(-) diff --git a/dts/framework/logger.py b/dts/framework/logger.py index cfa6e8cd72..db4a7698e0 100644 --- a/dts/framework/logger.py +++ b/dts/framework/logger.py @@ -5,141 +5,187 @@ """DTS logger module. -DTS framework and TestSuite logs are saved in different log files. +The module provides several additional features: + + * The storage of DTS execution stages, + * Logging to console, a human-readable log file and a machine-readable log file, + * Optional log files for specific stages. """ import logging -import os.path -from typing import TypedDict +from enum import auto +from logging import FileHandler, StreamHandler +from pathlib import Path +from typing import ClassVar -from .settings import SETTINGS +from .utils import StrEnum date_fmt = "%Y/%m/%d %H:%M:%S" -stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" +stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s" +dts_root_logger_name = "dts" + + +class DtsStage(StrEnum): + """The DTS execution stage.""" + #: + pre_execution = auto() + #: + execution = auto() + #: + build_target = auto() + #: + suite = auto() + #: + post_execution = auto() -class DTSLOG(logging.LoggerAdapter): - """DTS logger adapter class for framework and testsuites. - The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment - variable control the verbosity of output. If enabled, all messages will be emitted to the - console. +class DTSLogger(logging.Logger): + """The DTS logger class. - The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment - variable modify the directory where the logs will be stored. + The class extends the :class:`~logging.Logger` class to add the DTS execution stage information + to log records. The stage is common to all loggers, so it's stored in a class variable. - Attributes: - node: The additional identifier. Currently unused. - sh: The handler which emits logs to console. - fh: The handler which emits logs to a file. - verbose_fh: Just as fh, but logs with a different, more verbose, format. + Any time we switch to a new stage, we have the ability to log to an additional log file along + with a supplementary log file with machine-readable format. These two log files are used until + a new stage switch occurs. This is useful mainly for logging per test suite. """ - _logger: logging.Logger - node: str - sh: logging.StreamHandler - fh: logging.FileHandler - verbose_fh: logging.FileHandler + _stage: ClassVar[DtsStage] = DtsStage.pre_execution + _extra_file_handlers: list[FileHandler] = [] - def __init__(self, logger: logging.Logger, node: str = "suite"): - """Extend the constructor with additional handlers. + def __init__(self, *args, **kwargs): + """Extend the constructor with extra file handlers.""" + self._extra_file_handlers = [] + super().__init__(*args, **kwargs) - One handler logs to the console, the other one to a file, with either a regular or verbose - format. + def makeRecord(self, *args, **kwargs) -> logging.LogRecord: + """Generates a record with additional stage information. - Args: - logger: The logger from which to create the logger adapter. - node: An additional identifier. Currently unused. + This is the default method for the :class:`~logging.Logger` class. We extend it + to add stage information to the record. + + :meta private: + + Returns: + record: The generated record with the stage information. """ - self._logger = logger - # 1 means log everything, this will be used by file handlers if their level - # is not set - self._logger.setLevel(1) + record = super().makeRecord(*args, **kwargs) + record.stage = DTSLogger._stage # type: ignore[attr-defined] + return record + + def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None: + """Add logger handlers to the DTS root logger. + + This method should be called only on the DTS root logger. + The log records from child loggers will propagate to these handlers. + + Three handlers are added: - self.node = node + * A console handler, + * A file handler, + * A supplementary file handler with machine-readable logs + containing more debug information. - # add handler to emit to stdout - sh = logging.StreamHandler() + All log messages will be logged to files. The log level of the console handler + is configurable with `verbose`. + + Args: + verbose: If :data:`True`, log all messages to the console. + If :data:`False`, log to console with the :data:`logging.INFO` level. + output_dir: The directory where the log files will be located. + The names of the log files correspond to the name of the logger instance. + """ + self.setLevel(1) + + sh = StreamHandler() sh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) - sh.setLevel(logging.INFO) # console handler default level + if not verbose: + sh.setLevel(logging.INFO) + self.addHandler(sh) - if SETTINGS.verbose is True: - sh.setLevel(logging.DEBUG) + self._add_file_handlers(Path(output_dir, self.name)) - self._logger.addHandler(sh) - self.sh = sh + def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None: + """Set the DTS execution stage and optionally log to files. - # prepare the output folder - if not os.path.exists(SETTINGS.output_dir): - os.mkdir(SETTINGS.output_dir) + Set the DTS execution stage of the DTSLog class and optionally add + file handlers to the instance if the log file name is provided. - logging_path_prefix = os.path.join(SETTINGS.output_dir, node) + The file handlers log all messages. One is a regular human-readable log file and + the other one is a machine-readable log file with extra debug information. - fh = logging.FileHandler(f"{logging_path_prefix}.log") - fh.setFormatter( - logging.Formatter( - fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s", - datefmt=date_fmt, - ) - ) + Args: + stage: The DTS stage to set. + log_file_path: An optional path of the log file to use. This should be a full path + (either relative or absolute) without suffix (which will be appended). + """ + self._remove_extra_file_handlers() + + if DTSLogger._stage != stage: + self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.") + DTSLogger._stage = stage + + if log_file_path: + self._extra_file_handlers.extend(self._add_file_handlers(log_file_path)) + + def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]: + """Add file handlers to the DTS root logger. + + Add two type of file handlers: + + * A regular file handler with suffix ".log", + * A machine-readable file handler with suffix ".verbose.log". + This format provides extensive information for debugging and detailed analysis. + + Args: + log_file_path: The full path to the log file without suffix. + + Returns: + The newly created file handlers. - self._logger.addHandler(fh) - self.fh = fh + """ + fh = FileHandler(f"{log_file_path}.log") + fh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) + self.addHandler(fh) - # This outputs EVERYTHING, intended for post-mortem debugging - # Also optimized for processing via AWK (awk -F '|' ...) - verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log") + verbose_fh = FileHandler(f"{log_file_path}.verbose.log") verbose_fh.setFormatter( logging.Formatter( - fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" + "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s", datefmt=date_fmt, ) ) + self.addHandler(verbose_fh) - self._logger.addHandler(verbose_fh) - self.verbose_fh = verbose_fh - - super(DTSLOG, self).__init__(self._logger, dict(node=self.node)) - - def logger_exit(self) -> None: - """Remove the stream handler and the logfile handler.""" - for handler in (self.sh, self.fh, self.verbose_fh): - handler.flush() - self._logger.removeHandler(handler) - - -class _LoggerDictType(TypedDict): - logger: DTSLOG - name: str - node: str - + return [fh, verbose_fh] -# List for saving all loggers in use -_Loggers: list[_LoggerDictType] = [] + def _remove_extra_file_handlers(self) -> None: + """Remove any extra file handlers that have been added to the logger.""" + if self._extra_file_handlers: + for extra_file_handler in self._extra_file_handlers: + self.removeHandler(extra_file_handler) + self._extra_file_handlers = [] -def getLogger(name: str, node: str = "suite") -> DTSLOG: - """Get DTS logger adapter identified by name and node. - An existing logger will be returned if one with the exact name and node already exists. - A new one will be created and stored otherwise. +def get_dts_logger(name: str = None) -> DTSLogger: + """Return a DTS logger instance identified by `name`. Args: - name: The name of the logger. - node: An additional identifier for the logger. + name: If :data:`None`, return the DTS root logger. + If specified, return a child of the DTS root logger. Returns: - A logger uniquely identified by both name and node. + The DTS root logger or a child logger identified by `name`. """ - global _Loggers - # return saved logger - logger: _LoggerDictType - for logger in _Loggers: - if logger["name"] == name and logger["node"] == node: - return logger["logger"] - - # return new logger - dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node) - _Loggers.append({"logger": dts_logger, "name": name, "node": node}) - return dts_logger + original_logger_class = logging.getLoggerClass() + logging.setLoggerClass(DTSLogger) + if name: + name = f"{dts_root_logger_name}.{name}" + else: + name = dts_root_logger_name + logger = logging.getLogger(name) + logging.setLoggerClass(original_logger_class) + return logger # type: ignore[return-value] diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py index 51a01d6b5e..1910c81c3c 100644 --- a/dts/framework/remote_session/__init__.py +++ b/dts/framework/remote_session/__init__.py @@ -15,7 +15,7 @@ # pylama:ignore=W0611 from framework.config import NodeConfiguration -from framework.logger import DTSLOG +from framework.logger import DTSLogger from .interactive_remote_session import InteractiveRemoteSession from .interactive_shell import InteractiveShell @@ -26,7 +26,7 @@ def create_remote_session( - node_config: NodeConfiguration, name: str, logger: DTSLOG + node_config: NodeConfiguration, name: str, logger: DTSLogger ) -> RemoteSession: """Factory for non-interactive remote sessions. @@ -45,7 +45,7 @@ def create_remote_session( def create_interactive_session( - node_config: NodeConfiguration, logger: DTSLOG + node_config: NodeConfiguration, logger: DTSLogger ) -> InteractiveRemoteSession: """Factory for interactive remote sessions. diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py index 1cc82e3377..c50790db79 100644 --- a/dts/framework/remote_session/interactive_remote_session.py +++ b/dts/framework/remote_session/interactive_remote_session.py @@ -16,7 +16,7 @@ from framework.config import NodeConfiguration from framework.exception import SSHConnectionError -from framework.logger import DTSLOG +from framework.logger import DTSLogger class InteractiveRemoteSession: @@ -50,11 +50,11 @@ class InteractiveRemoteSession: username: str password: str session: SSHClient - _logger: DTSLOG + _logger: DTSLogger _node_config: NodeConfiguration _transport: Transport | None - def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None: + def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None: """Connect to the node during initialization. Args: diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py index b158f963b6..5cfe202e15 100644 --- a/dts/framework/remote_session/interactive_shell.py +++ b/dts/framework/remote_session/interactive_shell.py @@ -20,7 +20,7 @@ from paramiko import Channel, SSHClient, channel # type: ignore[import] -from framework.logger import DTSLOG +from framework.logger import DTSLogger from framework.settings import SETTINGS @@ -38,7 +38,7 @@ class InteractiveShell(ABC): _stdin: channel.ChannelStdinFile _stdout: channel.ChannelFile _ssh_channel: Channel - _logger: DTSLOG + _logger: DTSLogger _timeout: float _app_args: str @@ -61,7 +61,7 @@ class InteractiveShell(ABC): def __init__( self, interactive_session: SSHClient, - logger: DTSLOG, + logger: DTSLogger, get_privileged_command: Callable[[str], str] | None, app_args: str = "", timeout: float = SETTINGS.timeout, diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py index 2059f9a981..a69dc99400 100644 --- a/dts/framework/remote_session/remote_session.py +++ b/dts/framework/remote_session/remote_session.py @@ -9,14 +9,13 @@ the structure of the result of a command execution. """ - import dataclasses from abc import ABC, abstractmethod from pathlib import PurePath from framework.config import NodeConfiguration from framework.exception import RemoteCommandExecutionError -from framework.logger import DTSLOG +from framework.logger import DTSLogger from framework.settings import SETTINGS @@ -75,14 +74,14 @@ class RemoteSession(ABC): username: str password: str history: list[CommandResult] - _logger: DTSLOG + _logger: DTSLogger _node_config: NodeConfiguration def __init__( self, node_config: NodeConfiguration, session_name: str, - logger: DTSLOG, + logger: DTSLogger, ): """Connect to the node during initialization. @@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None: Args: force: Force the closure of the connection. This may not clean up all resources. """ - self._logger.logger_exit() self._close(force) @abstractmethod diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 511f8be064..af927b11a9 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -19,9 +19,10 @@ import importlib import inspect -import logging +import os import re import sys +from pathlib import Path from types import MethodType from typing import Iterable @@ -38,7 +39,7 @@ SSHTimeoutError, TestCaseVerifyError, ) -from .logger import DTSLOG, getLogger +from .logger import DTSLogger, DtsStage, get_dts_logger from .settings import SETTINGS from .test_result import ( BuildTargetResult, @@ -73,7 +74,7 @@ class DTSRunner: """ _configuration: Configuration - _logger: DTSLOG + _logger: DTSLogger _result: DTSResult _test_suite_class_prefix: str _test_suite_module_prefix: str @@ -83,7 +84,10 @@ class DTSRunner: def __init__(self): """Initialize the instance with configuration, logger, result and string constants.""" self._configuration = load_config() - self._logger = getLogger("DTSRunner") + self._logger = get_dts_logger() + if not os.path.exists(SETTINGS.output_dir): + os.makedirs(SETTINGS.output_dir) + self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir) self._result = DTSResult(self._logger) self._test_suite_class_prefix = "Test" self._test_suite_module_prefix = "tests.TestSuite_" @@ -137,6 +141,7 @@ def run(self): # for all Execution sections for execution in self._configuration.executions: + self._logger.set_stage(DtsStage.execution) self._logger.info( f"Running execution with SUT '{execution.system_under_test_node.name}'." ) @@ -168,6 +173,7 @@ def run(self): finally: try: + self._logger.set_stage(DtsStage.post_execution) for node in (sut_nodes | tg_nodes).values(): node.close() self._result.update_teardown(Result.PASS) @@ -425,6 +431,7 @@ def _run_execution( finally: try: + self._logger.set_stage(DtsStage.execution) sut_node.tear_down_execution() execution_result.update_teardown(Result.PASS) except Exception as e: @@ -453,6 +460,7 @@ def _run_build_target( with the current build target. test_suites_with_cases: The test suites with test cases to run. """ + self._logger.set_stage(DtsStage.build_target) self._logger.info(f"Running build target '{build_target.name}'.") try: @@ -469,6 +477,7 @@ def _run_build_target( finally: try: + self._logger.set_stage(DtsStage.build_target) sut_node.tear_down_build_target() build_target_result.update_teardown(Result.PASS) except Exception as e: @@ -541,6 +550,7 @@ def _run_test_suite( BlockingTestSuiteError: If a blocking test suite fails. """ test_suite_name = test_suite_with_cases.test_suite_class.__name__ + self._logger.set_stage(DtsStage.suite, Path(SETTINGS.output_dir, test_suite_name)) test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) try: self._logger.info(f"Starting test suite setup: {test_suite_name}") @@ -689,5 +699,4 @@ def _exit_dts(self) -> None: if self._logger: self._logger.info("DTS execution has ended.") - logging.shutdown() sys.exit(self._result.get_return_code()) diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index eedb2d20ee..28f84fd793 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -42,7 +42,7 @@ TestSuiteConfig, ) from .exception import DTSError, ErrorSeverity -from .logger import DTSLOG +from .logger import DTSLogger from .settings import SETTINGS from .test_suite import TestSuite @@ -237,13 +237,13 @@ class DTSResult(BaseResult): """ dpdk_version: str | None - _logger: DTSLOG + _logger: DTSLogger _errors: list[Exception] _return_code: ErrorSeverity _stats_result: Union["Statistics", None] _stats_filename: str - def __init__(self, logger: DTSLOG): + def __init__(self, logger: DTSLogger): """Extend the constructor with top-level specifics. Args: diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index f9fe88093e..365f80e21a 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -21,7 +21,7 @@ from scapy.packet import Packet, Padding # type: ignore[import] from .exception import TestCaseVerifyError -from .logger import DTSLOG, getLogger +from .logger import DTSLogger, get_dts_logger from .testbed_model import Port, PortLink, SutNode, TGNode from .utils import get_packet_summaries @@ -61,7 +61,7 @@ class TestSuite(object): #: Whether the test suite is blocking. A failure of a blocking test suite #: will block the execution of all subsequent test suites in the current build target. is_blocking: ClassVar[bool] = False - _logger: DTSLOG + _logger: DTSLogger _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -88,7 +88,7 @@ def __init__( """ self.sut_node = sut_node self.tg_node = tg_node - self._logger = getLogger(self.__class__.__name__) + self._logger = get_dts_logger(self.__class__.__name__) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py index 1a55fadf78..74061f6262 100644 --- a/dts/framework/testbed_model/node.py +++ b/dts/framework/testbed_model/node.py @@ -23,7 +23,7 @@ NodeConfiguration, ) from framework.exception import ConfigurationError -from framework.logger import DTSLOG, getLogger +from framework.logger import DTSLogger, get_dts_logger from framework.settings import SETTINGS from .cpu import ( @@ -63,7 +63,7 @@ class Node(ABC): name: str lcores: list[LogicalCore] ports: list[Port] - _logger: DTSLOG + _logger: DTSLogger _other_sessions: list[OSSession] _execution_config: ExecutionConfiguration virtual_devices: list[VirtualDevice] @@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration): """ self.config = node_config self.name = node_config.name - self._logger = getLogger(self.name) + self._logger = get_dts_logger(self.name) self.main_session = create_session(self.config, self.name, self._logger) self._logger.info(f"Connected to node: {self.name}") @@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession: connection = create_session( self.config, session_name, - getLogger(session_name, node=self.name), + get_dts_logger(session_name), ) self._other_sessions.append(connection) return connection @@ -299,7 +299,6 @@ def close(self) -> None: self.main_session.close() for session in self._other_sessions: session.close() - self._logger.logger_exit() @staticmethod def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: @@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: return func -def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession: +def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession: """Factory for OS-aware sessions. Args: diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py index ac6bb5e112..6983aa4a77 100644 --- a/dts/framework/testbed_model/os_session.py +++ b/dts/framework/testbed_model/os_session.py @@ -21,7 +21,6 @@ the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux and other commands for other OSs. It also translates the path to match the underlying OS. """ - from abc import ABC, abstractmethod from collections.abc import Iterable from ipaddress import IPv4Interface, IPv6Interface @@ -29,7 +28,7 @@ from typing import Type, TypeVar, Union from framework.config import Architecture, NodeConfiguration, NodeInfo -from framework.logger import DTSLOG +from framework.logger import DTSLogger from framework.remote_session import ( CommandResult, InteractiveRemoteSession, @@ -62,7 +61,7 @@ class OSSession(ABC): _config: NodeConfiguration name: str - _logger: DTSLOG + _logger: DTSLogger remote_session: RemoteSession interactive_session: InteractiveRemoteSession @@ -70,7 +69,7 @@ def __init__( self, node_config: NodeConfiguration, name: str, - logger: DTSLOG, + logger: DTSLogger, ): """Initialize the OS-aware session. diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py index c49fbff488..d86d7fb532 100644 --- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py +++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py @@ -13,7 +13,7 @@ from scapy.packet import Packet # type: ignore[import] from framework.config import TrafficGeneratorConfig -from framework.logger import DTSLOG, getLogger +from framework.logger import DTSLogger, get_dts_logger from framework.testbed_model.node import Node from framework.testbed_model.port import Port from framework.utils import get_packet_summaries @@ -28,7 +28,7 @@ class TrafficGenerator(ABC): _config: TrafficGeneratorConfig _tg_node: Node - _logger: DTSLOG + _logger: DTSLogger def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): """Initialize the traffic generator. @@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): """ self._config = config self._tg_node = tg_node - self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}") + self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}") def send_packet(self, packet: Packet, port: Port) -> None: """Send `packet` and block until it is fully sent. diff --git a/dts/main.py b/dts/main.py index 1ffe8ff81f..fa878cc16e 100755 --- a/dts/main.py +++ b/dts/main.py @@ -6,8 +6,6 @@ """The DTS executable.""" -import logging - from framework import settings @@ -30,5 +28,4 @@ def main() -> None: # Main program begins here if __name__ == "__main__": - logging.raiseExceptions = True main() -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v3 7/7] dts: improve test suite and case filtering 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš ` (5 preceding siblings ...) 2024-02-23 7:55 ` [PATCH v3 6/7] dts: refactor logging configuration Juraj Linkeš @ 2024-02-23 7:55 ` Juraj Linkeš 2024-02-27 21:24 ` [PATCH v3 0/7] test case blocking and logging Jeremy Spewock 7 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-02-23 7:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro Cc: dev, Juraj Linkeš The two places where we specify which test suite and test cases to run are complimentary and not that intuitive to use. A unified way provides a better user experience. The syntax in test run configuration file has not changed, but the environment variable and the command line arguments was changed to match the config file syntax. This required changes in the settings module which greatly simplified the parsing of the environment variables while retaining the same functionality. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- doc/guides/tools/dts.rst | 14 ++- dts/framework/config/__init__.py | 12 +- dts/framework/runner.py | 18 +-- dts/framework/settings.py | 187 ++++++++++++++----------------- dts/framework/test_suite.py | 2 +- 5 files changed, 114 insertions(+), 119 deletions(-) diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst index f686ca487c..d1c3c2af7a 100644 --- a/doc/guides/tools/dts.rst +++ b/doc/guides/tools/dts.rst @@ -215,28 +215,30 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet .. code-block:: console (dts-py3.10) $ ./main.py --help - usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN] + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN] Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority. options: -h, --help show this help message and exit --config-file CONFIG_FILE - [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml) + [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml) --output-dir OUTPUT_DIR, --output OUTPUT_DIR [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output) -t TIMEOUT, --timeout TIMEOUT [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15) -v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False) - -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None) + -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False) --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz) --compile-timeout COMPILE_TIMEOUT [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200) - --test-cases TEST_CASES - [DTS_TESTCASES] Comma-separated list of test cases to execute. Unknown test cases will be silently ignored. (default: ) + --test-suite TEST_SUITE [TEST_CASES ...] + [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment + variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...' + (default: []) --re-run RE_RUN, --re_run RE_RUN - [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs (default: 0) + [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0) The brackets contain the names of environment variables that set the same thing. diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index c6a93b3b89..4cb5c74059 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -35,9 +35,9 @@ import json import os.path -import pathlib from dataclasses import dataclass, fields from enum import auto, unique +from pathlib import Path from typing import Union import warlock # type: ignore[import] @@ -53,7 +53,6 @@ TrafficGeneratorConfigDict, ) from framework.exception import ConfigurationError -from framework.settings import SETTINGS from framework.utils import StrEnum @@ -571,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration": return Configuration(executions=executions) -def load_config() -> Configuration: +def load_config(config_file_path: Path) -> Configuration: """Load DTS test run configuration from a file. Load the YAML test run configuration file @@ -581,13 +580,16 @@ def load_config() -> Configuration: The YAML test run configuration file is specified in the :option:`--config-file` command line argument or the :envvar:`DTS_CFG_FILE` environment variable. + Args: + config_file_path: The path to the YAML test run configuration file. + Returns: The parsed test run configuration. """ - with open(SETTINGS.config_file_path, "r") as f: + with open(config_file_path, "r") as f: config_data = yaml.safe_load(f) - schema_path = os.path.join(pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json") + schema_path = os.path.join(Path(__file__).parent.resolve(), "conf_yaml_schema.json") with open(schema_path, "r") as f: schema = json.load(f) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index af927b11a9..cabff0a7b2 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -24,7 +24,7 @@ import sys from pathlib import Path from types import MethodType -from typing import Iterable +from typing import Iterable, Sequence from .config import ( BuildTargetConfiguration, @@ -83,7 +83,7 @@ class DTSRunner: def __init__(self): """Initialize the instance with configuration, logger, result and string constants.""" - self._configuration = load_config() + self._configuration = load_config(SETTINGS.config_file_path) self._logger = get_dts_logger() if not os.path.exists(SETTINGS.output_dir): os.makedirs(SETTINGS.output_dir) @@ -129,7 +129,7 @@ def run(self): #. Execution teardown The test cases are filtered according to the specification in the test run configuration and - the :option:`--test-cases` command line argument or + the :option:`--test-suite` command line argument or the :envvar:`DTS_TESTCASES` environment variable. """ sut_nodes: dict[str, SutNode] = {} @@ -147,7 +147,9 @@ def run(self): ) execution_result = self._result.add_execution(execution) # we don't want to modify the original config, so create a copy - execution_test_suites = list(execution.test_suites) + execution_test_suites = list( + SETTINGS.test_suites if SETTINGS.test_suites else execution.test_suites + ) if not execution.skip_smoke_tests: execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] try: @@ -226,7 +228,7 @@ def _get_test_suites_with_cases( test_suite_class = self._get_test_suite_class(test_suite_config.test_suite) test_cases = [] func_test_cases, perf_test_cases = self._filter_test_cases( - test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases) + test_suite_class, test_suite_config.test_cases ) if func: test_cases.extend(func_test_cases) @@ -301,7 +303,7 @@ def is_test_suite(object) -> bool: ) def _filter_test_cases( - self, test_suite_class: type[TestSuite], test_cases_to_run: set[str] + self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str] ) -> tuple[list[MethodType], list[MethodType]]: """Filter `test_cases_to_run` from `test_suite_class`. @@ -330,7 +332,9 @@ def _filter_test_cases( (name, method) for name, method in name_method_tuples if name in test_cases_to_run ] if len(name_method_tuples) < len(test_cases_to_run): - missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples} + missing_test_cases = set(test_cases_to_run) - { + name for name, _ in name_method_tuples + } raise ConfigurationError( f"Test cases {missing_test_cases} not found among methods " f"of {test_suite_class.__name__}." diff --git a/dts/framework/settings.py b/dts/framework/settings.py index 2b8bfbe0ed..688e8679a7 100644 --- a/dts/framework/settings.py +++ b/dts/framework/settings.py @@ -48,10 +48,11 @@ The path to a DPDK tarball, git commit ID, tag ID or tree ID to test. -.. option:: --test-cases -.. envvar:: DTS_TESTCASES +.. option:: --test-suite +.. envvar:: DTS_TEST_SUITES - A comma-separated list of test cases to execute. Unknown test cases will be silently ignored. + A test suite with test cases which may be specified multiple times. + In the environment variable, the suites are joined with a comma. .. option:: --re-run, --re_run .. envvar:: DTS_RERUN @@ -71,83 +72,13 @@ import argparse import os -from collections.abc import Callable, Iterable, Sequence from dataclasses import dataclass, field from pathlib import Path -from typing import Any, TypeVar +from typing import Any +from .config import TestSuiteConfig from .utils import DPDKGitTarball -_T = TypeVar("_T") - - -def _env_arg(env_var: str) -> Any: - """A helper method augmenting the argparse Action with environment variables. - - If the supplied environment variable is defined, then the default value - of the argument is modified. This satisfies the priority order of - command line argument > environment variable > default value. - - Arguments with no values (flags) should be defined using the const keyword argument - (True or False). When the argument is specified, it will be set to const, if not specified, - the default will be stored (possibly modified by the corresponding environment variable). - - Other arguments work the same as default argparse arguments, that is using - the default 'store' action. - - Returns: - The modified argparse.Action. - """ - - class _EnvironmentArgument(argparse.Action): - def __init__( - self, - option_strings: Sequence[str], - dest: str, - nargs: str | int | None = None, - const: bool | None = None, - default: Any = None, - type: Callable[[str], _T | argparse.FileType | None] = None, - choices: Iterable[_T] | None = None, - required: bool = False, - help: str | None = None, - metavar: str | tuple[str, ...] | None = None, - ) -> None: - env_var_value = os.environ.get(env_var) - default = env_var_value or default - if const is not None: - nargs = 0 - default = const if env_var_value else default - type = None - choices = None - metavar = None - super(_EnvironmentArgument, self).__init__( - option_strings, - dest, - nargs=nargs, - const=const, - default=default, - type=type, - choices=choices, - required=required, - help=help, - metavar=metavar, - ) - - def __call__( - self, - parser: argparse.ArgumentParser, - namespace: argparse.Namespace, - values: Any, - option_string: str = None, - ) -> None: - if self.const is not None: - setattr(namespace, self.dest, self.const) - else: - setattr(namespace, self.dest, values) - - return _EnvironmentArgument - @dataclass(slots=True) class Settings: @@ -171,7 +102,7 @@ class Settings: #: compile_timeout: float = 1200 #: - test_cases: list[str] = field(default_factory=list) + test_suites: list[TestSuiteConfig] = field(default_factory=list) #: re_run: int = 0 @@ -180,6 +111,31 @@ class Settings: def _get_parser() -> argparse.ArgumentParser: + """Create the argument parser for DTS. + + Command line options take precedence over environment variables, which in turn take precedence + over default values. + + Returns: + argparse.ArgumentParser: The configured argument parser with defined options. + """ + + def env_arg(env_var: str, default: Any) -> Any: + """A helper function augmenting the argparse with environment variables. + + If the supplied environment variable is defined, then the default value + of the argument is modified. This satisfies the priority order of + command line argument > environment variable > default value. + + Args: + env_var: Environment variable name. + default: Default value. + + Returns: + Environment variable or default value. + """ + return os.environ.get(env_var) or default + parser = argparse.ArgumentParser( description="Run DPDK test suites. All options may be specified with the environment " "variables provided in brackets. Command line arguments have higher priority.", @@ -188,25 +144,23 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "--config-file", - action=_env_arg("DTS_CFG_FILE"), - default=SETTINGS.config_file_path, + default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path), type=Path, - help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets.", + help="[DTS_CFG_FILE] The configuration file that describes the test cases, " + "SUTs and targets.", ) parser.add_argument( "--output-dir", "--output", - action=_env_arg("DTS_OUTPUT_DIR"), - default=SETTINGS.output_dir, + default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir), help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.", ) parser.add_argument( "-t", "--timeout", - action=_env_arg("DTS_TIMEOUT"), - default=SETTINGS.timeout, + default=env_arg("DTS_TIMEOUT", SETTINGS.timeout), type=float, help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.", ) @@ -214,9 +168,8 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "-v", "--verbose", - action=_env_arg("DTS_VERBOSE"), - default=SETTINGS.verbose, - const=True, + action="store_true", + default=env_arg("DTS_VERBOSE", SETTINGS.verbose), help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages " "to the console.", ) @@ -224,8 +177,8 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "-s", "--skip-setup", - action=_env_arg("DTS_SKIP_SETUP"), - const=True, + action="store_true", + default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup), help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.", ) @@ -233,8 +186,7 @@ def _get_parser() -> argparse.ArgumentParser: "--tarball", "--snapshot", "--git-ref", - action=_env_arg("DTS_DPDK_TARBALL"), - default=SETTINGS.dpdk_tarball_path, + default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path), type=Path, help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, " "tag ID or tree ID to test. To test local changes, first commit them, " @@ -243,36 +195,71 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "--compile-timeout", - action=_env_arg("DTS_COMPILE_TIMEOUT"), - default=SETTINGS.compile_timeout, + default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout), type=float, help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.", ) parser.add_argument( - "--test-cases", - action=_env_arg("DTS_TESTCASES"), - default="", - help="[DTS_TESTCASES] Comma-separated list of test cases to execute.", + "--test-suite", + action="append", + nargs="+", + metavar=("TEST_SUITE", "TEST_CASES"), + default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites), + help="[DTS_TEST_SUITES] A list containing a test suite with test cases. " + "The first parameter is the test suite name, and the rest are test case names, " + "which are optional. May be specified multiple times. To specify multiple test suites in " + "the environment variable, join the lists with a comma. " + "Examples: " + "--test-suite suite case case --test-suite suite case ... | " + "DTS_TEST_SUITES='suite case case, suite case, ...' | " + "--test-suite suite --test-suite suite case ... | " + "DTS_TEST_SUITES='suite, suite case, ...'", ) parser.add_argument( "--re-run", "--re_run", - action=_env_arg("DTS_RERUN"), - default=SETTINGS.re_run, + default=env_arg("DTS_RERUN", SETTINGS.re_run), type=int, help="[DTS_RERUN] Re-run each test case the specified number of times " - "if a test failure occurs", + "if a test failure occurs.", ) return parser +def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]: + """Process the given argument to a list of :class:`TestSuiteConfig` to execute. + + Args: + args: The arguments to process. The args is a string from an environment variable + or a list of from the user input containing tests suites with tests cases, + each of which is a list of [test_suite, test_case, test_case, ...]. + + Returns: + A list of test suite configurations to execute. + """ + if isinstance(args, str): + # Environment variable in the form of "suite case case, suite case, suite, ..." + args = [suite_with_cases.split() for suite_with_cases in args.split(",")] + + test_suites_to_run = [] + for suite_with_cases in args: + test_suites_to_run.append( + TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:]) + ) + + return test_suites_to_run + + def get_settings() -> Settings: """Create new settings with inputs from the user. The inputs are taken from the command line and from environment variables. + + Returns: + The new settings object. """ parsed_args = _get_parser().parse_args() return Settings( @@ -287,6 +274,6 @@ def get_settings() -> Settings: else Path(parsed_args.tarball) ), compile_timeout=parsed_args.compile_timeout, - test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []), + test_suites=_process_test_suites(parsed_args.test_suite), re_run=parsed_args.re_run, ) diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index 365f80e21a..1957ea7328 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -40,7 +40,7 @@ class TestSuite(object): and functional test cases (all other test cases). By default, all test cases will be executed. A list of testcase names may be specified - in the YAML test run configuration file and in the :option:`--test-cases` command line argument + in the YAML test run configuration file and in the :option:`--test-suite` command line argument or in the :envvar:`DTS_TESTCASES` environment variable to filter which test cases to run. The union of both lists will be used. Any unknown test cases from the latter lists will be silently ignored. -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v3 0/7] test case blocking and logging 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš ` (6 preceding siblings ...) 2024-02-23 7:55 ` [PATCH v3 7/7] dts: improve test suite and case filtering Juraj Linkeš @ 2024-02-27 21:24 ` Jeremy Spewock 7 siblings, 0 replies; 44+ messages in thread From: Jeremy Spewock @ 2024-02-27 21:24 UTC (permalink / raw) To: Juraj Linkeš Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, dev I just had a very small comment which has nothing to do with functionality and is really just about something being checked twice which does no harm. Outside of that however, the whole series looked good to me. ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 0/7] test case blocking and logging 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš ` (7 preceding siblings ...) 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš @ 2024-03-01 10:55 ` Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 1/7] dts: convert dts.py methods to class Juraj Linkeš ` (8 more replies) 8 siblings, 9 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro, npratte Cc: dev, Juraj Linkeš We currently don't record test case results that couldn't be executed because of a previous failure, such as when a test suite setup failed, resulting in no executed test cases. In order to record the test cases that couldn't be executed, we must know the lists of test suites and test cases ahead of the actual test suite execution, as an error could occur before we even start executing test suites. In addition, the patch series contains two refactors and one improvement. The first refactor is closely related. The dts.py was renamed to runner.py and given a clear purpose - running the test suites and all other orchestration needed to run test suites. The logic for this was not all in the original dts.py module and it was brought there. The runner is also responsible for recording results, which is the blocked test cases are recorded. The other refactor, logging, is related to the first refactor. The logging module was simplified while extending capabilities. Each test suite logs into its own log file in addition to the main log file which the runner must handle (as it knows when we start executing particular test suites). The runner also handles the switching between execution stages for the purposes of logging. The one aforementioned improvement is in unifying how we specify test cases in the conf.yaml file and in the environment variable/command line argument. v2: Rebase and update of the whole patch. There are changes in all parts of the code, mainly improving the design and logic. Also added the last patch which improves test suite specification on the cmdline. v3: Improved variables in _get_test_suite_class along with docstring. Fixed smoke test suite not being added into the list of test suites to be executed. v4: Added the checking of CamelCase convention when discovering test cases. Also added additional test stages. The stages were split into setup and teardown. Juraj Linkeš (7): dts: convert dts.py methods to class dts: move test suite execution logic to DTSRunner dts: filter test suites in executions dts: reorganize test result dts: block all test cases when earlier setup fails dts: refactor logging configuration dts: improve test suite and case filtering doc/guides/tools/dts.rst | 14 +- dts/framework/config/__init__.py | 36 +- dts/framework/config/conf_yaml_schema.json | 2 +- dts/framework/dts.py | 338 --------- dts/framework/logger.py | 246 +++--- dts/framework/remote_session/__init__.py | 6 +- .../interactive_remote_session.py | 6 +- .../remote_session/interactive_shell.py | 6 +- .../remote_session/remote_session.py | 8 +- dts/framework/runner.py | 711 ++++++++++++++++++ dts/framework/settings.py | 188 +++-- dts/framework/test_result.py | 565 ++++++++------ dts/framework/test_suite.py | 239 +----- dts/framework/testbed_model/node.py | 11 +- dts/framework/testbed_model/os_session.py | 7 +- .../traffic_generator/traffic_generator.py | 6 +- dts/main.py | 9 +- dts/pyproject.toml | 3 + dts/tests/TestSuite_os_udp.py | 2 +- dts/tests/TestSuite_smoke_tests.py | 2 +- 20 files changed, 1375 insertions(+), 1030 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 1/7] dts: convert dts.py methods to class 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš @ 2024-03-01 10:55 ` Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš ` (7 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro, npratte Cc: dev, Juraj Linkeš The dts.py module deviates from the rest of the code without a clear reason. Converting it into a class and using better naming will improve organization and code readability. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/dts.py | 338 ---------------------------------------- dts/framework/runner.py | 333 +++++++++++++++++++++++++++++++++++++++ dts/main.py | 6 +- 3 files changed, 337 insertions(+), 340 deletions(-) delete mode 100644 dts/framework/dts.py create mode 100644 dts/framework/runner.py diff --git a/dts/framework/dts.py b/dts/framework/dts.py deleted file mode 100644 index e16d4578a0..0000000000 --- a/dts/framework/dts.py +++ /dev/null @@ -1,338 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2010-2019 Intel Corporation -# Copyright(c) 2022-2023 PANTHEON.tech s.r.o. -# Copyright(c) 2022-2023 University of New Hampshire - -r"""Test suite runner module. - -A DTS run is split into stages: - - #. Execution stage, - #. Build target stage, - #. Test suite stage, - #. Test case stage. - -The module is responsible for running tests on testbeds defined in the test run configuration. -Each setup or teardown of each stage is recorded in a :class:`~.test_result.DTSResult` or -one of its subclasses. The test case results are also recorded. - -If an error occurs, the current stage is aborted, the error is recorded and the run continues in -the next iteration of the same stage. The return code is the highest `severity` of all -:class:`~.exception.DTSError`\s. - -Example: - An error occurs in a build target setup. The current build target is aborted and the run - continues with the next build target. If the errored build target was the last one in the given - execution, the next execution begins. - -Attributes: - dts_logger: The logger instance used in this module. - result: The top level result used in the module. -""" - -import sys - -from .config import ( - BuildTargetConfiguration, - ExecutionConfiguration, - TestSuiteConfig, - load_config, -) -from .exception import BlockingTestSuiteError -from .logger import DTSLOG, getLogger -from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result -from .test_suite import get_test_suites -from .testbed_model import SutNode, TGNode - -# dummy defaults to satisfy linters -dts_logger: DTSLOG = None # type: ignore[assignment] -result: DTSResult = DTSResult(dts_logger) - - -def run_all() -> None: - """Run all build targets in all executions from the test run configuration. - - Before running test suites, executions and build targets are first set up. - The executions and build targets defined in the test run configuration are iterated over. - The executions define which tests to run and where to run them and build targets define - the DPDK build setup. - - The tests suites are set up for each execution/build target tuple and each scheduled - test case within the test suite is set up, executed and torn down. After all test cases - have been executed, the test suite is torn down and the next build target will be tested. - - All the nested steps look like this: - - #. Execution setup - - #. Build target setup - - #. Test suite setup - - #. Test case setup - #. Test case logic - #. Test case teardown - - #. Test suite teardown - - #. Build target teardown - - #. Execution teardown - - The test cases are filtered according to the specification in the test run configuration and - the :option:`--test-cases` command line argument or - the :envvar:`DTS_TESTCASES` environment variable. - """ - global dts_logger - global result - - # create a regular DTS logger and create a new result with it - dts_logger = getLogger("DTSRunner") - result = DTSResult(dts_logger) - - # check the python version of the server that run dts - _check_dts_python_version() - - sut_nodes: dict[str, SutNode] = {} - tg_nodes: dict[str, TGNode] = {} - try: - # for all Execution sections - for execution in load_config().executions: - sut_node = sut_nodes.get(execution.system_under_test_node.name) - tg_node = tg_nodes.get(execution.traffic_generator_node.name) - - try: - if not sut_node: - sut_node = SutNode(execution.system_under_test_node) - sut_nodes[sut_node.name] = sut_node - if not tg_node: - tg_node = TGNode(execution.traffic_generator_node) - tg_nodes[tg_node.name] = tg_node - result.update_setup(Result.PASS) - except Exception as e: - failed_node = execution.system_under_test_node.name - if sut_node: - failed_node = execution.traffic_generator_node.name - dts_logger.exception(f"Creation of node {failed_node} failed.") - result.update_setup(Result.FAIL, e) - - else: - _run_execution(sut_node, tg_node, execution, result) - - except Exception as e: - dts_logger.exception("An unexpected error has occurred.") - result.add_error(e) - raise - - finally: - try: - for node in (sut_nodes | tg_nodes).values(): - node.close() - result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Final cleanup of nodes failed.") - result.update_teardown(Result.ERROR, e) - - # we need to put the sys.exit call outside the finally clause to make sure - # that unexpected exceptions will propagate - # in that case, the error that should be reported is the uncaught exception as - # that is a severe error originating from the framework - # at that point, we'll only have partial results which could be impacted by the - # error causing the uncaught exception, making them uninterpretable - _exit_dts() - - -def _check_dts_python_version() -> None: - """Check the required Python version - v3.10.""" - - def RED(text: str) -> str: - return f"\u001B[31;1m{str(text)}\u001B[0m" - - if sys.version_info.major < 3 or (sys.version_info.major == 3 and sys.version_info.minor < 10): - print( - RED( - ( - "WARNING: DTS execution node's python version is lower than" - "python 3.10, is deprecated and will not work in future releases." - ) - ), - file=sys.stderr, - ) - print(RED("Please use Python >= 3.10 instead"), file=sys.stderr) - - -def _run_execution( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - result: DTSResult, -) -> None: - """Run the given execution. - - This involves running the execution setup as well as running all build targets - in the given execution. After that, execution teardown is run. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - execution: An execution's test run configuration. - result: The top level result object. - """ - dts_logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") - execution_result = result.add_execution(sut_node.config) - execution_result.add_sut_info(sut_node.node_info) - - try: - sut_node.set_up_execution(execution) - execution_result.update_setup(Result.PASS) - except Exception as e: - dts_logger.exception("Execution setup failed.") - execution_result.update_setup(Result.FAIL, e) - - else: - for build_target in execution.build_targets: - _run_build_target(sut_node, tg_node, build_target, execution, execution_result) - - finally: - try: - sut_node.tear_down_execution() - execution_result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Execution teardown failed.") - execution_result.update_teardown(Result.FAIL, e) - - -def _run_build_target( - sut_node: SutNode, - tg_node: TGNode, - build_target: BuildTargetConfiguration, - execution: ExecutionConfiguration, - execution_result: ExecutionResult, -) -> None: - """Run the given build target. - - This involves running the build target setup as well as running all test suites - in the given execution the build target is defined in. - After that, build target teardown is run. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - build_target: A build target's test run configuration. - execution: The build target's execution's test run configuration. - execution_result: The execution level result object associated with the execution. - """ - dts_logger.info(f"Running build target '{build_target.name}'.") - build_target_result = execution_result.add_build_target(build_target) - - try: - sut_node.set_up_build_target(build_target) - result.dpdk_version = sut_node.dpdk_version - build_target_result.add_build_target_info(sut_node.get_build_target_info()) - build_target_result.update_setup(Result.PASS) - except Exception as e: - dts_logger.exception("Build target setup failed.") - build_target_result.update_setup(Result.FAIL, e) - - else: - _run_all_suites(sut_node, tg_node, execution, build_target_result) - - finally: - try: - sut_node.tear_down_build_target() - build_target_result.update_teardown(Result.PASS) - except Exception as e: - dts_logger.exception("Build target teardown failed.") - build_target_result.update_teardown(Result.FAIL, e) - - -def _run_all_suites( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, -) -> None: - """Run the execution's (possibly a subset) test suites using the current build target. - - The function assumes the build target we're testing has already been built on the SUT node. - The current build target thus corresponds to the current DPDK build present on the SUT node. - - If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites - in the current build target won't be executed. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - execution: The execution's test run configuration associated with the current build target. - build_target_result: The build target level result object associated - with the current build target. - """ - end_build_target = False - if not execution.skip_smoke_tests: - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] - for test_suite_config in execution.test_suites: - try: - _run_single_suite(sut_node, tg_node, execution, build_target_result, test_suite_config) - except BlockingTestSuiteError as e: - dts_logger.exception( - f"An error occurred within {test_suite_config.test_suite}. Skipping build target." - ) - result.add_error(e) - end_build_target = True - # if a blocking test failed and we need to bail out of suite executions - if end_build_target: - break - - -def _run_single_suite( - sut_node: SutNode, - tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, - test_suite_config: TestSuiteConfig, -) -> None: - """Run all test suite in a single test suite module. - - The function assumes the build target we're testing has already been built on the SUT node. - The current build target thus corresponds to the current DPDK build present on the SUT node. - - Args: - sut_node: The execution's SUT node. - tg_node: The execution's TG node. - execution: The execution's test run configuration associated with the current build target. - build_target_result: The build target level result object associated - with the current build target. - test_suite_config: Test suite test run configuration specifying the test suite module - and possibly a subset of test cases of test suites in that module. - - Raises: - BlockingTestSuiteError: If a blocking test suite fails. - """ - try: - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" - test_suite_classes = get_test_suites(full_suite_path) - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) - dts_logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") - except Exception as e: - dts_logger.exception("An error occurred when searching for test suites.") - result.update_setup(Result.ERROR, e) - - else: - for test_suite_class in test_suite_classes: - test_suite = test_suite_class( - sut_node, - tg_node, - test_suite_config.test_cases, - execution.func, - build_target_result, - ) - test_suite.run() - - -def _exit_dts() -> None: - """Process all errors and exit with the proper exit code.""" - result.process() - - if dts_logger: - dts_logger.info("DTS execution has ended.") - sys.exit(result.get_return_code()) diff --git a/dts/framework/runner.py b/dts/framework/runner.py new file mode 100644 index 0000000000..acc1c4d6db --- /dev/null +++ b/dts/framework/runner.py @@ -0,0 +1,333 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2019 Intel Corporation +# Copyright(c) 2022-2023 PANTHEON.tech s.r.o. +# Copyright(c) 2022-2023 University of New Hampshire + +"""Test suite runner module. + +The module is responsible for running DTS in a series of stages: + + #. Execution stage, + #. Build target stage, + #. Test suite stage, + #. Test case stage. + +The execution and build target stages set up the environment before running test suites. +The test suite stage sets up steps common to all test cases +and the test case stage runs test cases individually. +""" + +import logging +import sys + +from .config import ( + BuildTargetConfiguration, + ExecutionConfiguration, + TestSuiteConfig, + load_config, +) +from .exception import BlockingTestSuiteError +from .logger import DTSLOG, getLogger +from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result +from .test_suite import get_test_suites +from .testbed_model import SutNode, TGNode + + +class DTSRunner: + r"""Test suite runner class. + + The class is responsible for running tests on testbeds defined in the test run configuration. + Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult` + or one of its subclasses. The test case results are also recorded. + + If an error occurs, the current stage is aborted, the error is recorded and the run continues in + the next iteration of the same stage. The return code is the highest `severity` of all + :class:`~.framework.exception.DTSError`\s. + + Example: + An error occurs in a build target setup. The current build target is aborted and the run + continues with the next build target. If the errored build target was the last one in the + given execution, the next execution begins. + """ + + _logger: DTSLOG + _result: DTSResult + + def __init__(self): + """Initialize the instance with logger and result.""" + self._logger = getLogger("DTSRunner") + self._result = DTSResult(self._logger) + + def run(self): + """Run all build targets in all executions from the test run configuration. + + Before running test suites, executions and build targets are first set up. + The executions and build targets defined in the test run configuration are iterated over. + The executions define which tests to run and where to run them and build targets define + the DPDK build setup. + + The tests suites are set up for each execution/build target tuple and each discovered + test case within the test suite is set up, executed and torn down. After all test cases + have been executed, the test suite is torn down and the next build target will be tested. + + All the nested steps look like this: + + #. Execution setup + + #. Build target setup + + #. Test suite setup + + #. Test case setup + #. Test case logic + #. Test case teardown + + #. Test suite teardown + + #. Build target teardown + + #. Execution teardown + + The test cases are filtered according to the specification in the test run configuration and + the :option:`--test-cases` command line argument or + the :envvar:`DTS_TESTCASES` environment variable. + """ + sut_nodes: dict[str, SutNode] = {} + tg_nodes: dict[str, TGNode] = {} + try: + # check the python version of the server that runs dts + self._check_dts_python_version() + + # for all Execution sections + for execution in load_config().executions: + sut_node = sut_nodes.get(execution.system_under_test_node.name) + tg_node = tg_nodes.get(execution.traffic_generator_node.name) + + try: + if not sut_node: + sut_node = SutNode(execution.system_under_test_node) + sut_nodes[sut_node.name] = sut_node + if not tg_node: + tg_node = TGNode(execution.traffic_generator_node) + tg_nodes[tg_node.name] = tg_node + self._result.update_setup(Result.PASS) + except Exception as e: + failed_node = execution.system_under_test_node.name + if sut_node: + failed_node = execution.traffic_generator_node.name + self._logger.exception(f"The Creation of node {failed_node} failed.") + self._result.update_setup(Result.FAIL, e) + + else: + self._run_execution(sut_node, tg_node, execution) + + except Exception as e: + self._logger.exception("An unexpected error has occurred.") + self._result.add_error(e) + raise + + finally: + try: + for node in (sut_nodes | tg_nodes).values(): + node.close() + self._result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("The final cleanup of nodes failed.") + self._result.update_teardown(Result.ERROR, e) + + # we need to put the sys.exit call outside the finally clause to make sure + # that unexpected exceptions will propagate + # in that case, the error that should be reported is the uncaught exception as + # that is a severe error originating from the framework + # at that point, we'll only have partial results which could be impacted by the + # error causing the uncaught exception, making them uninterpretable + self._exit_dts() + + def _check_dts_python_version(self) -> None: + """Check the required Python version - v3.10.""" + if sys.version_info.major < 3 or ( + sys.version_info.major == 3 and sys.version_info.minor < 10 + ): + self._logger.warning( + "DTS execution node's python version is lower than Python 3.10, " + "is deprecated and will not work in future releases." + ) + self._logger.warning("Please use Python >= 3.10 instead.") + + def _run_execution( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + ) -> None: + """Run the given execution. + + This involves running the execution setup as well as running all build targets + in the given execution. After that, execution teardown is run. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: An execution's test run configuration. + """ + self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") + execution_result = self._result.add_execution(sut_node.config) + execution_result.add_sut_info(sut_node.node_info) + + try: + sut_node.set_up_execution(execution) + execution_result.update_setup(Result.PASS) + except Exception as e: + self._logger.exception("Execution setup failed.") + execution_result.update_setup(Result.FAIL, e) + + else: + for build_target in execution.build_targets: + self._run_build_target(sut_node, tg_node, build_target, execution, execution_result) + + finally: + try: + sut_node.tear_down_execution() + execution_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("Execution teardown failed.") + execution_result.update_teardown(Result.FAIL, e) + + def _run_build_target( + self, + sut_node: SutNode, + tg_node: TGNode, + build_target: BuildTargetConfiguration, + execution: ExecutionConfiguration, + execution_result: ExecutionResult, + ) -> None: + """Run the given build target. + + This involves running the build target setup as well as running all test suites + of the build target's execution. + After that, build target teardown is run. + + Args: + sut_node: The execution's sut node. + tg_node: The execution's tg node. + build_target: A build target's test run configuration. + execution: The build target's execution's test run configuration. + execution_result: The execution level result object associated with the execution. + """ + self._logger.info(f"Running build target '{build_target.name}'.") + build_target_result = execution_result.add_build_target(build_target) + + try: + sut_node.set_up_build_target(build_target) + self._result.dpdk_version = sut_node.dpdk_version + build_target_result.add_build_target_info(sut_node.get_build_target_info()) + build_target_result.update_setup(Result.PASS) + except Exception as e: + self._logger.exception("Build target setup failed.") + build_target_result.update_setup(Result.FAIL, e) + + else: + self._run_all_suites(sut_node, tg_node, execution, build_target_result) + + finally: + try: + sut_node.tear_down_build_target() + build_target_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception("Build target teardown failed.") + build_target_result.update_teardown(Result.FAIL, e) + + def _run_all_suites( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + build_target_result: BuildTargetResult, + ) -> None: + """Run the execution's (possibly a subset of) test suites using the current build target. + + The method assumes the build target we're testing has already been built on the SUT node. + The current build target thus corresponds to the current DPDK build present on the SUT node. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: The execution's test run configuration associated + with the current build target. + build_target_result: The build target level result object associated + with the current build target. + """ + end_build_target = False + if not execution.skip_smoke_tests: + execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] + for test_suite_config in execution.test_suites: + try: + self._run_single_suite( + sut_node, tg_node, execution, build_target_result, test_suite_config + ) + except BlockingTestSuiteError as e: + self._logger.exception( + f"An error occurred within {test_suite_config.test_suite}. " + "Skipping build target..." + ) + self._result.add_error(e) + end_build_target = True + # if a blocking test failed and we need to bail out of suite executions + if end_build_target: + break + + def _run_single_suite( + self, + sut_node: SutNode, + tg_node: TGNode, + execution: ExecutionConfiguration, + build_target_result: BuildTargetResult, + test_suite_config: TestSuiteConfig, + ) -> None: + """Run all test suites in a single test suite module. + + The method assumes the build target we're testing has already been built on the SUT node. + The current build target thus corresponds to the current DPDK build present on the SUT node. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: The execution's test run configuration associated + with the current build target. + build_target_result: The build target level result object associated + with the current build target. + test_suite_config: Test suite test run configuration specifying the test suite module + and possibly a subset of test cases of test suites in that module. + + Raises: + BlockingTestSuiteError: If a blocking test suite fails. + """ + try: + full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" + test_suite_classes = get_test_suites(full_suite_path) + suites_str = ", ".join((x.__name__ for x in test_suite_classes)) + self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") + except Exception as e: + self._logger.exception("An error occurred when searching for test suites.") + self._result.update_setup(Result.ERROR, e) + + else: + for test_suite_class in test_suite_classes: + test_suite = test_suite_class( + sut_node, + tg_node, + test_suite_config.test_cases, + execution.func, + build_target_result, + ) + test_suite.run() + + def _exit_dts(self) -> None: + """Process all errors and exit with the proper exit code.""" + self._result.process() + + if self._logger: + self._logger.info("DTS execution has ended.") + + logging.shutdown() + sys.exit(self._result.get_return_code()) diff --git a/dts/main.py b/dts/main.py index f703615d11..1ffe8ff81f 100755 --- a/dts/main.py +++ b/dts/main.py @@ -21,9 +21,11 @@ def main() -> None: be modified before the settings module is imported anywhere else in the framework. """ settings.SETTINGS = settings.get_settings() - from framework import dts - dts.run_all() + from framework.runner import DTSRunner + + dts = DTSRunner() + dts.run() # Main program begins here -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 2/7] dts: move test suite execution logic to DTSRunner 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 1/7] dts: convert dts.py methods to class Juraj Linkeš @ 2024-03-01 10:55 ` Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 3/7] dts: filter test suites in executions Juraj Linkeš ` (6 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro, npratte Cc: dev, Juraj Linkeš Move the code responsible for running the test suite from the TestSuite class to the DTSRunner class. This restructuring decision was made to consolidate and unify the related logic into a single unit. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/runner.py | 175 ++++++++++++++++++++++++++++++++---- dts/framework/test_suite.py | 152 ++----------------------------- 2 files changed, 169 insertions(+), 158 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index acc1c4d6db..933685d638 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -19,6 +19,7 @@ import logging import sys +from types import MethodType from .config import ( BuildTargetConfiguration, @@ -26,10 +27,18 @@ TestSuiteConfig, load_config, ) -from .exception import BlockingTestSuiteError +from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError from .logger import DTSLOG, getLogger -from .test_result import BuildTargetResult, DTSResult, ExecutionResult, Result -from .test_suite import get_test_suites +from .settings import SETTINGS +from .test_result import ( + BuildTargetResult, + DTSResult, + ExecutionResult, + Result, + TestCaseResult, + TestSuiteResult, +) +from .test_suite import TestSuite, get_test_suites from .testbed_model import SutNode, TGNode @@ -227,7 +236,7 @@ def _run_build_target( build_target_result.update_setup(Result.FAIL, e) else: - self._run_all_suites(sut_node, tg_node, execution, build_target_result) + self._run_test_suites(sut_node, tg_node, execution, build_target_result) finally: try: @@ -237,7 +246,7 @@ def _run_build_target( self._logger.exception("Build target teardown failed.") build_target_result.update_teardown(Result.FAIL, e) - def _run_all_suites( + def _run_test_suites( self, sut_node: SutNode, tg_node: TGNode, @@ -249,6 +258,9 @@ def _run_all_suites( The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. + If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites + in the current build target won't be executed. + Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. @@ -262,7 +274,7 @@ def _run_all_suites( execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] for test_suite_config in execution.test_suites: try: - self._run_single_suite( + self._run_test_suite_module( sut_node, tg_node, execution, build_target_result, test_suite_config ) except BlockingTestSuiteError as e: @@ -276,7 +288,7 @@ def _run_all_suites( if end_build_target: break - def _run_single_suite( + def _run_test_suite_module( self, sut_node: SutNode, tg_node: TGNode, @@ -284,11 +296,18 @@ def _run_single_suite( build_target_result: BuildTargetResult, test_suite_config: TestSuiteConfig, ) -> None: - """Run all test suites in a single test suite module. + """Set up, execute and tear down all test suites in a single test suite module. The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. + Test suite execution consists of running the discovered test cases. + A test case run consists of setup, execution and teardown of said test case. + + Record the setup and the teardown and handle failures. + + The test cases to execute are discovered when creating the :class:`TestSuite` object. + Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. @@ -313,14 +332,140 @@ def _run_single_suite( else: for test_suite_class in test_suite_classes: - test_suite = test_suite_class( - sut_node, - tg_node, - test_suite_config.test_cases, - execution.func, - build_target_result, + test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases) + + test_suite_name = test_suite.__class__.__name__ + test_suite_result = build_target_result.add_test_suite(test_suite_name) + try: + self._logger.info(f"Starting test suite setup: {test_suite_name}") + test_suite.set_up_suite() + test_suite_result.update_setup(Result.PASS) + self._logger.info(f"Test suite setup successful: {test_suite_name}") + except Exception as e: + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") + test_suite_result.update_setup(Result.ERROR, e) + + else: + self._execute_test_suite(execution.func, test_suite, test_suite_result) + + finally: + try: + test_suite.tear_down_suite() + sut_node.kill_cleanup_dpdk_apps() + test_suite_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") + self._logger.warning( + f"Test suite '{test_suite_name}' teardown failed, " + f"the next test suite may be affected." + ) + test_suite_result.update_setup(Result.ERROR, e) + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: + raise BlockingTestSuiteError(test_suite_name) + + def _execute_test_suite( + self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult + ) -> None: + """Execute all discovered test cases in `test_suite`. + + If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment + variable is set, in case of a test case failure, the test case will be executed again + until it passes or it fails that many times in addition of the first failure. + + Args: + func: Whether to execute functional test cases. + test_suite: The test suite object. + test_suite_result: The test suite level result object associated + with the current test suite. + """ + if func: + for test_case_method in test_suite._get_functional_test_cases(): + test_case_name = test_case_method.__name__ + test_case_result = test_suite_result.add_test_case(test_case_name) + all_attempts = SETTINGS.re_run + 1 + attempt_nr = 1 + self._run_test_case(test_suite, test_case_method, test_case_result) + while not test_case_result and attempt_nr < all_attempts: + attempt_nr += 1 + self._logger.info( + f"Re-running FAILED test case '{test_case_name}'. " + f"Attempt number {attempt_nr} out of {all_attempts}." + ) + self._run_test_case(test_suite, test_case_method, test_case_result) + + def _run_test_case( + self, + test_suite: TestSuite, + test_case_method: MethodType, + test_case_result: TestCaseResult, + ) -> None: + """Setup, execute and teardown a test case in `test_suite`. + + Record the result of the setup and the teardown and handle failures. + + Args: + test_suite: The test suite object. + test_case_method: The test case method. + test_case_result: The test case level result object associated + with the current test case. + """ + test_case_name = test_case_method.__name__ + + try: + # run set_up function for each case + test_suite.set_up_test_case() + test_case_result.update_setup(Result.PASS) + except SSHTimeoutError as e: + self._logger.exception(f"Test case setup FAILED: {test_case_name}") + test_case_result.update_setup(Result.FAIL, e) + except Exception as e: + self._logger.exception(f"Test case setup ERROR: {test_case_name}") + test_case_result.update_setup(Result.ERROR, e) + + else: + # run test case if setup was successful + self._execute_test_case(test_case_method, test_case_result) + + finally: + try: + test_suite.tear_down_test_case() + test_case_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test case teardown ERROR: {test_case_name}") + self._logger.warning( + f"Test case '{test_case_name}' teardown failed, " + f"the next test case may be affected." ) - test_suite.run() + test_case_result.update_teardown(Result.ERROR, e) + test_case_result.update(Result.ERROR) + + def _execute_test_case( + self, test_case_method: MethodType, test_case_result: TestCaseResult + ) -> None: + """Execute one test case, record the result and handle failures. + + Args: + test_case_method: The test case method. + test_case_result: The test case level result object associated + with the current test case. + """ + test_case_name = test_case_method.__name__ + try: + self._logger.info(f"Starting test case execution: {test_case_name}") + test_case_method() + test_case_result.update(Result.PASS) + self._logger.info(f"Test case execution PASSED: {test_case_name}") + + except TestCaseVerifyError as e: + self._logger.exception(f"Test case execution FAILED: {test_case_name}") + test_case_result.update(Result.FAIL, e) + except Exception as e: + self._logger.exception(f"Test case execution ERROR: {test_case_name}") + test_case_result.update(Result.ERROR, e) + except KeyboardInterrupt: + self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}") + test_case_result.update(Result.SKIP) + raise KeyboardInterrupt("Stop DTS") def _exit_dts(self) -> None: """Process all errors and exit with the proper exit code.""" diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index dfb391ffbd..b02fd36147 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -8,7 +8,6 @@ must be extended by subclasses which add test cases. The :class:`TestSuite` contains the basics needed by subclasses: - * Test suite and test case execution flow, * Testbed (SUT, TG) configuration, * Packet sending and verification, * Test case verification. @@ -28,27 +27,22 @@ from scapy.layers.l2 import Ether # type: ignore[import] from scapy.packet import Packet, Padding # type: ignore[import] -from .exception import ( - BlockingTestSuiteError, - ConfigurationError, - SSHTimeoutError, - TestCaseVerifyError, -) +from .exception import ConfigurationError, TestCaseVerifyError from .logger import DTSLOG, getLogger from .settings import SETTINGS -from .test_result import BuildTargetResult, Result, TestCaseResult, TestSuiteResult from .testbed_model import Port, PortLink, SutNode, TGNode from .utils import get_packet_summaries class TestSuite(object): - """The base class with methods for handling the basic flow of a test suite. + """The base class with building blocks needed by most test cases. * Test case filtering and collection, - * Test suite setup/cleanup, - * Test setup/cleanup, - * Test case execution, - * Error handling and results storage. + * Test suite setup/cleanup methods to override, + * Test case setup/cleanup methods to override, + * Test case verification, + * Testbed configuration, + * Traffic sending and verification. Test cases are implemented by subclasses. Test cases are all methods starting with ``test_``, further divided into performance test cases (starting with ``test_perf_``) @@ -60,10 +54,6 @@ class TestSuite(object): The union of both lists will be used. Any unknown test cases from the latter lists will be silently ignored. - If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable - is set, in case of a test case failure, the test case will be executed again until it passes - or it fails that many times in addition of the first failure. - The methods named ``[set_up|tear_down]_[suite|test_case]`` should be overridden in subclasses if the appropriate test suite/test case fixtures are needed. @@ -82,8 +72,6 @@ class TestSuite(object): is_blocking: ClassVar[bool] = False _logger: DTSLOG _test_cases_to_run: list[str] - _func: bool - _result: TestSuiteResult _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -99,30 +87,23 @@ def __init__( sut_node: SutNode, tg_node: TGNode, test_cases: list[str], - func: bool, - build_target_result: BuildTargetResult, ): """Initialize the test suite testbed information and basic configuration. - Process what test cases to run, create the associated - :class:`~.test_result.TestSuiteResult`, find links between ports - and set up default IP addresses to be used when configuring them. + Process what test cases to run, find links between ports and set up + default IP addresses to be used when configuring them. Args: sut_node: The SUT node where the test suite will run. tg_node: The TG node where the test suite will run. test_cases: The list of test cases to execute. If empty, all test cases will be executed. - func: Whether to run functional tests. - build_target_result: The build target result this test suite is run in. """ self.sut_node = sut_node self.tg_node = tg_node self._logger = getLogger(self.__class__.__name__) self._test_cases_to_run = test_cases self._test_cases_to_run.extend(SETTINGS.test_cases) - self._func = func - self._result = build_target_result.add_test_suite(self.__class__.__name__) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( @@ -384,62 +365,6 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: return False return True - def run(self) -> None: - """Set up, execute and tear down the whole suite. - - Test suite execution consists of running all test cases scheduled to be executed. - A test case run consists of setup, execution and teardown of said test case. - - Record the setup and the teardown and handle failures. - - The list of scheduled test cases is constructed when creating the :class:`TestSuite` object. - """ - test_suite_name = self.__class__.__name__ - - try: - self._logger.info(f"Starting test suite setup: {test_suite_name}") - self.set_up_suite() - self._result.update_setup(Result.PASS) - self._logger.info(f"Test suite setup successful: {test_suite_name}") - except Exception as e: - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") - self._result.update_setup(Result.ERROR, e) - - else: - self._execute_test_suite() - - finally: - try: - self.tear_down_suite() - self.sut_node.kill_cleanup_dpdk_apps() - self._result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") - self._logger.warning( - f"Test suite '{test_suite_name}' teardown failed, " - f"the next test suite may be affected." - ) - self._result.update_setup(Result.ERROR, e) - if len(self._result.get_errors()) > 0 and self.is_blocking: - raise BlockingTestSuiteError(test_suite_name) - - def _execute_test_suite(self) -> None: - """Execute all test cases scheduled to be executed in this suite.""" - if self._func: - for test_case_method in self._get_functional_test_cases(): - test_case_name = test_case_method.__name__ - test_case_result = self._result.add_test_case(test_case_name) - all_attempts = SETTINGS.re_run + 1 - attempt_nr = 1 - self._run_test_case(test_case_method, test_case_result) - while not test_case_result and attempt_nr < all_attempts: - attempt_nr += 1 - self._logger.info( - f"Re-running FAILED test case '{test_case_name}'. " - f"Attempt number {attempt_nr} out of {all_attempts}." - ) - self._run_test_case(test_case_method, test_case_result) - def _get_functional_test_cases(self) -> list[MethodType]: """Get all functional test cases defined in this TestSuite. @@ -471,65 +396,6 @@ def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool return match - def _run_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult - ) -> None: - """Setup, execute and teardown a test case in this suite. - - Record the result of the setup and the teardown and handle failures. - """ - test_case_name = test_case_method.__name__ - - try: - # run set_up function for each case - self.set_up_test_case() - test_case_result.update_setup(Result.PASS) - except SSHTimeoutError as e: - self._logger.exception(f"Test case setup FAILED: {test_case_name}") - test_case_result.update_setup(Result.FAIL, e) - except Exception as e: - self._logger.exception(f"Test case setup ERROR: {test_case_name}") - test_case_result.update_setup(Result.ERROR, e) - - else: - # run test case if setup was successful - self._execute_test_case(test_case_method, test_case_result) - - finally: - try: - self.tear_down_test_case() - test_case_result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test case teardown ERROR: {test_case_name}") - self._logger.warning( - f"Test case '{test_case_name}' teardown failed, " - f"the next test case may be affected." - ) - test_case_result.update_teardown(Result.ERROR, e) - test_case_result.update(Result.ERROR) - - def _execute_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult - ) -> None: - """Execute one test case, record the result and handle failures.""" - test_case_name = test_case_method.__name__ - try: - self._logger.info(f"Starting test case execution: {test_case_name}") - test_case_method() - test_case_result.update(Result.PASS) - self._logger.info(f"Test case execution PASSED: {test_case_name}") - - except TestCaseVerifyError as e: - self._logger.exception(f"Test case execution FAILED: {test_case_name}") - test_case_result.update(Result.FAIL, e) - except Exception as e: - self._logger.exception(f"Test case execution ERROR: {test_case_name}") - test_case_result.update(Result.ERROR, e) - except KeyboardInterrupt: - self._logger.error(f"Test case execution INTERRUPTED by user: {test_case_name}") - test_case_result.update(Result.SKIP) - raise KeyboardInterrupt("Stop DTS") - def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: r"""Find all :class:`TestSuite`\s in a Python module. -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 3/7] dts: filter test suites in executions 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 1/7] dts: convert dts.py methods to class Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš @ 2024-03-01 10:55 ` Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 4/7] dts: reorganize test result Juraj Linkeš ` (5 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro, npratte Cc: dev, Juraj Linkeš We're currently filtering which test cases to run after some setup steps, such as DPDK build, have already been taken. This prohibits us to mark the test suites and cases that were supposed to be run as blocked when an earlier setup fails, as that information is not available at that time. To remedy this, move the filtering to the beginning of each execution. This is the first action taken in each execution and if we can't filter the test cases, such as due to invalid inputs, we abort the whole execution. No test suites nor cases will be marked as blocked as we don't know which were supposed to be run. On top of that, the filtering takes place in the TestSuite class, which should only concern itself with test suite and test case logic, not the processing behind the scenes. The logic has been moved to DTSRunner which should do all the processing needed to run test suites. The filtering itself introduces a few changes/assumptions which are more sensible than before: 1. Assumption: There is just one TestSuite child class in each test suite module. This was an implicit assumption before as we couldn't specify the TestSuite classes in the test run configuration, just the modules. The name of the TestSuite child class starts with "Test" and then corresponds to the name of the module with CamelCase naming. 2. Unknown test cases specified both in the test run configuration and the environment variable/command line argument are no longer silently ignored. This is a quality of life improvement for users, as they could easily be not aware of the silent ignoration. Also, a change in the code results in pycodestyle warning and error: [E] E203 whitespace before ':' [W] W503 line break before binary operator These two are not PEP8 compliant, so they're disabled. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/config/__init__.py | 24 +- dts/framework/config/conf_yaml_schema.json | 2 +- dts/framework/runner.py | 433 +++++++++++++++------ dts/framework/settings.py | 3 +- dts/framework/test_result.py | 34 ++ dts/framework/test_suite.py | 85 +--- dts/pyproject.toml | 3 + dts/tests/TestSuite_os_udp.py | 2 +- dts/tests/TestSuite_smoke_tests.py | 2 +- 9 files changed, 390 insertions(+), 198 deletions(-) diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index 62eded7f04..c6a93b3b89 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -36,7 +36,7 @@ import json import os.path import pathlib -from dataclasses import dataclass +from dataclasses import dataclass, fields from enum import auto, unique from typing import Union @@ -506,6 +506,28 @@ def from_dict( vdevs=vdevs, ) + def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration": + """Create a shallow copy with any of the fields modified. + + The only new data are those passed to this method. + The rest are copied from the object's fields calling the method. + + Args: + **kwargs: The names and types of keyword arguments are defined + by the fields of the :class:`ExecutionConfiguration` class. + + Returns: + The copied and modified execution configuration. + """ + new_config = {} + for field in fields(self): + if field.name in kwargs: + new_config[field.name] = kwargs[field.name] + else: + new_config[field.name] = getattr(self, field.name) + + return ExecutionConfiguration(**new_config) + @dataclass(slots=True, frozen=True) class Configuration: diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json index 84e45fe3c2..051b079fe4 100644 --- a/dts/framework/config/conf_yaml_schema.json +++ b/dts/framework/config/conf_yaml_schema.json @@ -197,7 +197,7 @@ }, "cases": { "type": "array", - "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.", + "description": "If specified, only this subset of test suite's test cases will be run.", "items": { "type": "string" }, diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 933685d638..5f6bcbbb86 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -17,17 +17,27 @@ and the test case stage runs test cases individually. """ +import importlib +import inspect import logging +import re import sys from types import MethodType +from typing import Iterable from .config import ( BuildTargetConfiguration, + Configuration, ExecutionConfiguration, TestSuiteConfig, load_config, ) -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCaseVerifyError +from .exception import ( + BlockingTestSuiteError, + ConfigurationError, + SSHTimeoutError, + TestCaseVerifyError, +) from .logger import DTSLOG, getLogger from .settings import SETTINGS from .test_result import ( @@ -37,8 +47,9 @@ Result, TestCaseResult, TestSuiteResult, + TestSuiteWithCases, ) -from .test_suite import TestSuite, get_test_suites +from .test_suite import TestSuite from .testbed_model import SutNode, TGNode @@ -59,13 +70,23 @@ class DTSRunner: given execution, the next execution begins. """ + _configuration: Configuration _logger: DTSLOG _result: DTSResult + _test_suite_class_prefix: str + _test_suite_module_prefix: str + _func_test_case_regex: str + _perf_test_case_regex: str def __init__(self): - """Initialize the instance with logger and result.""" + """Initialize the instance with configuration, logger, result and string constants.""" + self._configuration = load_config() self._logger = getLogger("DTSRunner") self._result = DTSResult(self._logger) + self._test_suite_class_prefix = "Test" + self._test_suite_module_prefix = "tests.TestSuite_" + self._func_test_case_regex = r"test_(?!perf_)" + self._perf_test_case_regex = r"test_perf_" def run(self): """Run all build targets in all executions from the test run configuration. @@ -106,29 +127,32 @@ def run(self): try: # check the python version of the server that runs dts self._check_dts_python_version() + self._result.update_setup(Result.PASS) # for all Execution sections - for execution in load_config().executions: - sut_node = sut_nodes.get(execution.system_under_test_node.name) - tg_node = tg_nodes.get(execution.traffic_generator_node.name) - + for execution in self._configuration.executions: + self._logger.info( + f"Running execution with SUT '{execution.system_under_test_node.name}'." + ) + execution_result = self._result.add_execution(execution.system_under_test_node) + # we don't want to modify the original config, so create a copy + execution_test_suites = list(execution.test_suites) + if not execution.skip_smoke_tests: + execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] try: - if not sut_node: - sut_node = SutNode(execution.system_under_test_node) - sut_nodes[sut_node.name] = sut_node - if not tg_node: - tg_node = TGNode(execution.traffic_generator_node) - tg_nodes[tg_node.name] = tg_node - self._result.update_setup(Result.PASS) + test_suites_with_cases = self._get_test_suites_with_cases( + execution_test_suites, execution.func, execution.perf + ) except Exception as e: - failed_node = execution.system_under_test_node.name - if sut_node: - failed_node = execution.traffic_generator_node.name - self._logger.exception(f"The Creation of node {failed_node} failed.") - self._result.update_setup(Result.FAIL, e) + self._logger.exception( + f"Invalid test suite configuration found: " f"{execution_test_suites}." + ) + execution_result.update_setup(Result.FAIL, e) else: - self._run_execution(sut_node, tg_node, execution) + self._connect_nodes_and_run_execution( + sut_nodes, tg_nodes, execution, execution_result, test_suites_with_cases + ) except Exception as e: self._logger.exception("An unexpected error has occurred.") @@ -163,11 +187,207 @@ def _check_dts_python_version(self) -> None: ) self._logger.warning("Please use Python >= 3.10 instead.") + def _get_test_suites_with_cases( + self, + test_suite_configs: list[TestSuiteConfig], + func: bool, + perf: bool, + ) -> list[TestSuiteWithCases]: + """Test suites with test cases discovery. + + The test suites with test cases defined in the user configuration are discovered + and stored for future use so that we don't import the modules twice and so that + the list of test suites with test cases is available for recording right away. + + Args: + test_suite_configs: Test suite configurations. + func: Whether to include functional test cases in the final list. + perf: Whether to include performance test cases in the final list. + + Returns: + The discovered test suites, each with test cases. + """ + test_suites_with_cases = [] + + for test_suite_config in test_suite_configs: + test_suite_class = self._get_test_suite_class(test_suite_config.test_suite) + test_cases = [] + func_test_cases, perf_test_cases = self._filter_test_cases( + test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases) + ) + if func: + test_cases.extend(func_test_cases) + if perf: + test_cases.extend(perf_test_cases) + + test_suites_with_cases.append( + TestSuiteWithCases(test_suite_class=test_suite_class, test_cases=test_cases) + ) + + return test_suites_with_cases + + def _get_test_suite_class(self, module_name: str) -> type[TestSuite]: + """Find the :class:`TestSuite` class in `module_name`. + + The full module name is `module_name` prefixed with `self._test_suite_module_prefix`. + The module name is a standard filename with words separated with underscores. + Search the `module_name` for a :class:`TestSuite` class which starts + with `self._test_suite_class_prefix`, continuing with CamelCase `module_name`. + The first matching class is returned. + + The CamelCase convention applies to abbreviations, acronyms, initialisms and so on:: + + OS -> Os + TCP -> Tcp + + Args: + module_name: The module name without prefix where to search for the test suite. + + Returns: + The found test suite class. + + Raises: + ConfigurationError: If the corresponding module is not found or + a valid :class:`TestSuite` is not found in the module. + """ + + def is_test_suite(object) -> bool: + """Check whether `object` is a :class:`TestSuite`. + + The `object` is a subclass of :class:`TestSuite`, but not :class:`TestSuite` itself. + + Args: + object: The object to be checked. + + Returns: + :data:`True` if `object` is a subclass of `TestSuite`. + """ + try: + if issubclass(object, TestSuite) and object is not TestSuite: + return True + except TypeError: + return False + return False + + testsuite_module_path = f"{self._test_suite_module_prefix}{module_name}" + try: + test_suite_module = importlib.import_module(testsuite_module_path) + except ModuleNotFoundError as e: + raise ConfigurationError( + f"Test suite module '{testsuite_module_path}' not found." + ) from e + + camel_case_suite_name = "".join( + [suite_word.capitalize() for suite_word in module_name.split("_")] + ) + full_suite_name_to_find = f"{self._test_suite_class_prefix}{camel_case_suite_name}" + for class_name, class_obj in inspect.getmembers(test_suite_module, is_test_suite): + if class_name == full_suite_name_to_find: + return class_obj + raise ConfigurationError( + f"Couldn't find any valid test suites in {test_suite_module.__name__}." + ) + + def _filter_test_cases( + self, test_suite_class: type[TestSuite], test_cases_to_run: set[str] + ) -> tuple[list[MethodType], list[MethodType]]: + """Filter `test_cases_to_run` from `test_suite_class`. + + There are two rounds of filtering if `test_cases_to_run` is not empty. + The first filters `test_cases_to_run` from all methods of `test_suite_class`. + Then the methods are separated into functional and performance test cases. + If a method matches neither the functional nor performance name prefix, it's an error. + + Args: + test_suite_class: The class of the test suite. + test_cases_to_run: Test case names to filter from `test_suite_class`. + If empty, return all matching test cases. + + Returns: + A list of test case methods that should be executed. + + Raises: + ConfigurationError: If a test case from `test_cases_to_run` is not found + or it doesn't match either the functional nor performance name prefix. + """ + func_test_cases = [] + perf_test_cases = [] + name_method_tuples = inspect.getmembers(test_suite_class, inspect.isfunction) + if test_cases_to_run: + name_method_tuples = [ + (name, method) for name, method in name_method_tuples if name in test_cases_to_run + ] + if len(name_method_tuples) < len(test_cases_to_run): + missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples} + raise ConfigurationError( + f"Test cases {missing_test_cases} not found among methods " + f"of {test_suite_class.__name__}." + ) + + for test_case_name, test_case_method in name_method_tuples: + if re.match(self._func_test_case_regex, test_case_name): + func_test_cases.append(test_case_method) + elif re.match(self._perf_test_case_regex, test_case_name): + perf_test_cases.append(test_case_method) + elif test_cases_to_run: + raise ConfigurationError( + f"Method '{test_case_name}' matches neither " + f"a functional nor a performance test case name." + ) + + return func_test_cases, perf_test_cases + + def _connect_nodes_and_run_execution( + self, + sut_nodes: dict[str, SutNode], + tg_nodes: dict[str, TGNode], + execution: ExecutionConfiguration, + execution_result: ExecutionResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], + ) -> None: + """Connect nodes, then continue to run the given execution. + + Connect the :class:`SutNode` and the :class:`TGNode` of this `execution`. + If either has already been connected, it's going to be in either `sut_nodes` or `tg_nodes`, + respectively. + If not, connect and add the node to the respective `sut_nodes` or `tg_nodes` :class:`dict`. + + Args: + sut_nodes: A dictionary storing connected/to be connected SUT nodes. + tg_nodes: A dictionary storing connected/to be connected TG nodes. + execution: An execution's test run configuration. + execution_result: The execution's result. + test_suites_with_cases: The test suites with test cases to run. + """ + sut_node = sut_nodes.get(execution.system_under_test_node.name) + tg_node = tg_nodes.get(execution.traffic_generator_node.name) + + try: + if not sut_node: + sut_node = SutNode(execution.system_under_test_node) + sut_nodes[sut_node.name] = sut_node + if not tg_node: + tg_node = TGNode(execution.traffic_generator_node) + tg_nodes[tg_node.name] = tg_node + except Exception as e: + failed_node = execution.system_under_test_node.name + if sut_node: + failed_node = execution.traffic_generator_node.name + self._logger.exception(f"The Creation of node {failed_node} failed.") + execution_result.update_setup(Result.FAIL, e) + + else: + self._run_execution( + sut_node, tg_node, execution, execution_result, test_suites_with_cases + ) + def _run_execution( self, sut_node: SutNode, tg_node: TGNode, execution: ExecutionConfiguration, + execution_result: ExecutionResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], ) -> None: """Run the given execution. @@ -178,11 +398,11 @@ def _run_execution( sut_node: The execution's SUT node. tg_node: The execution's TG node. execution: An execution's test run configuration. + execution_result: The execution's result. + test_suites_with_cases: The test suites with test cases to run. """ self._logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") - execution_result = self._result.add_execution(sut_node.config) execution_result.add_sut_info(sut_node.node_info) - try: sut_node.set_up_execution(execution) execution_result.update_setup(Result.PASS) @@ -192,7 +412,10 @@ def _run_execution( else: for build_target in execution.build_targets: - self._run_build_target(sut_node, tg_node, build_target, execution, execution_result) + build_target_result = execution_result.add_build_target(build_target) + self._run_build_target( + sut_node, tg_node, build_target, build_target_result, test_suites_with_cases + ) finally: try: @@ -207,8 +430,8 @@ def _run_build_target( sut_node: SutNode, tg_node: TGNode, build_target: BuildTargetConfiguration, - execution: ExecutionConfiguration, - execution_result: ExecutionResult, + build_target_result: BuildTargetResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], ) -> None: """Run the given build target. @@ -220,11 +443,11 @@ def _run_build_target( sut_node: The execution's sut node. tg_node: The execution's tg node. build_target: A build target's test run configuration. - execution: The build target's execution's test run configuration. - execution_result: The execution level result object associated with the execution. + build_target_result: The build target level result object associated + with the current build target. + test_suites_with_cases: The test suites with test cases to run. """ self._logger.info(f"Running build target '{build_target.name}'.") - build_target_result = execution_result.add_build_target(build_target) try: sut_node.set_up_build_target(build_target) @@ -236,7 +459,7 @@ def _run_build_target( build_target_result.update_setup(Result.FAIL, e) else: - self._run_test_suites(sut_node, tg_node, execution, build_target_result) + self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases) finally: try: @@ -250,10 +473,10 @@ def _run_test_suites( self, sut_node: SutNode, tg_node: TGNode, - execution: ExecutionConfiguration, build_target_result: BuildTargetResult, + test_suites_with_cases: Iterable[TestSuiteWithCases], ) -> None: - """Run the execution's (possibly a subset of) test suites using the current build target. + """Run `test_suites_with_cases` with the current build target. The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. @@ -264,22 +487,20 @@ def _run_test_suites( Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. - execution: The execution's test run configuration associated - with the current build target. build_target_result: The build target level result object associated with the current build target. + test_suites_with_cases: The test suites with test cases to run. """ end_build_target = False - if not execution.skip_smoke_tests: - execution.test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] - for test_suite_config in execution.test_suites: + for test_suite_with_cases in test_suites_with_cases: + test_suite_result = build_target_result.add_test_suite( + test_suite_with_cases.test_suite_class.__name__ + ) try: - self._run_test_suite_module( - sut_node, tg_node, execution, build_target_result, test_suite_config - ) + self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases) except BlockingTestSuiteError as e: self._logger.exception( - f"An error occurred within {test_suite_config.test_suite}. " + f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. " "Skipping build target..." ) self._result.add_error(e) @@ -288,15 +509,14 @@ def _run_test_suites( if end_build_target: break - def _run_test_suite_module( + def _run_test_suite( self, sut_node: SutNode, tg_node: TGNode, - execution: ExecutionConfiguration, - build_target_result: BuildTargetResult, - test_suite_config: TestSuiteConfig, + test_suite_result: TestSuiteResult, + test_suite_with_cases: TestSuiteWithCases, ) -> None: - """Set up, execute and tear down all test suites in a single test suite module. + """Set up, execute and tear down `test_suite_with_cases`. The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. @@ -306,92 +526,79 @@ def _run_test_suite_module( Record the setup and the teardown and handle failures. - The test cases to execute are discovered when creating the :class:`TestSuite` object. - Args: sut_node: The execution's SUT node. tg_node: The execution's TG node. - execution: The execution's test run configuration associated - with the current build target. - build_target_result: The build target level result object associated - with the current build target. - test_suite_config: Test suite test run configuration specifying the test suite module - and possibly a subset of test cases of test suites in that module. + test_suite_result: The test suite level result object associated + with the current test suite. + test_suite_with_cases: The test suite with test cases to run. Raises: BlockingTestSuiteError: If a blocking test suite fails. """ + test_suite_name = test_suite_with_cases.test_suite_class.__name__ + test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) try: - full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" - test_suite_classes = get_test_suites(full_suite_path) - suites_str = ", ".join((x.__name__ for x in test_suite_classes)) - self._logger.debug(f"Found test suites '{suites_str}' in '{full_suite_path}'.") + self._logger.info(f"Starting test suite setup: {test_suite_name}") + test_suite.set_up_suite() + test_suite_result.update_setup(Result.PASS) + self._logger.info(f"Test suite setup successful: {test_suite_name}") except Exception as e: - self._logger.exception("An error occurred when searching for test suites.") - self._result.update_setup(Result.ERROR, e) + self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") + test_suite_result.update_setup(Result.ERROR, e) else: - for test_suite_class in test_suite_classes: - test_suite = test_suite_class(sut_node, tg_node, test_suite_config.test_cases) - - test_suite_name = test_suite.__class__.__name__ - test_suite_result = build_target_result.add_test_suite(test_suite_name) - try: - self._logger.info(f"Starting test suite setup: {test_suite_name}") - test_suite.set_up_suite() - test_suite_result.update_setup(Result.PASS) - self._logger.info(f"Test suite setup successful: {test_suite_name}") - except Exception as e: - self._logger.exception(f"Test suite setup ERROR: {test_suite_name}") - test_suite_result.update_setup(Result.ERROR, e) - - else: - self._execute_test_suite(execution.func, test_suite, test_suite_result) - - finally: - try: - test_suite.tear_down_suite() - sut_node.kill_cleanup_dpdk_apps() - test_suite_result.update_teardown(Result.PASS) - except Exception as e: - self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") - self._logger.warning( - f"Test suite '{test_suite_name}' teardown failed, " - f"the next test suite may be affected." - ) - test_suite_result.update_setup(Result.ERROR, e) - if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: - raise BlockingTestSuiteError(test_suite_name) + self._execute_test_suite( + test_suite, + test_suite_with_cases.test_cases, + test_suite_result, + ) + finally: + try: + test_suite.tear_down_suite() + sut_node.kill_cleanup_dpdk_apps() + test_suite_result.update_teardown(Result.PASS) + except Exception as e: + self._logger.exception(f"Test suite teardown ERROR: {test_suite_name}") + self._logger.warning( + f"Test suite '{test_suite_name}' teardown failed, " + "the next test suite may be affected." + ) + test_suite_result.update_setup(Result.ERROR, e) + if len(test_suite_result.get_errors()) > 0 and test_suite.is_blocking: + raise BlockingTestSuiteError(test_suite_name) def _execute_test_suite( - self, func: bool, test_suite: TestSuite, test_suite_result: TestSuiteResult + self, + test_suite: TestSuite, + test_cases: Iterable[MethodType], + test_suite_result: TestSuiteResult, ) -> None: - """Execute all discovered test cases in `test_suite`. + """Execute all `test_cases` in `test_suite`. If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable is set, in case of a test case failure, the test case will be executed again until it passes or it fails that many times in addition of the first failure. Args: - func: Whether to execute functional test cases. test_suite: The test suite object. + test_cases: The list of test case methods. test_suite_result: The test suite level result object associated with the current test suite. """ - if func: - for test_case_method in test_suite._get_functional_test_cases(): - test_case_name = test_case_method.__name__ - test_case_result = test_suite_result.add_test_case(test_case_name) - all_attempts = SETTINGS.re_run + 1 - attempt_nr = 1 + for test_case_method in test_cases: + test_case_name = test_case_method.__name__ + test_case_result = test_suite_result.add_test_case(test_case_name) + all_attempts = SETTINGS.re_run + 1 + attempt_nr = 1 + self._run_test_case(test_suite, test_case_method, test_case_result) + while not test_case_result and attempt_nr < all_attempts: + attempt_nr += 1 + self._logger.info( + f"Re-running FAILED test case '{test_case_name}'. " + f"Attempt number {attempt_nr} out of {all_attempts}." + ) self._run_test_case(test_suite, test_case_method, test_case_result) - while not test_case_result and attempt_nr < all_attempts: - attempt_nr += 1 - self._logger.info( - f"Re-running FAILED test case '{test_case_name}'. " - f"Attempt number {attempt_nr} out of {all_attempts}." - ) - self._run_test_case(test_suite, test_case_method, test_case_result) def _run_test_case( self, @@ -399,7 +606,7 @@ def _run_test_case( test_case_method: MethodType, test_case_result: TestCaseResult, ) -> None: - """Setup, execute and teardown a test case in `test_suite`. + """Setup, execute and teardown `test_case_method` from `test_suite`. Record the result of the setup and the teardown and handle failures. @@ -424,7 +631,7 @@ def _run_test_case( else: # run test case if setup was successful - self._execute_test_case(test_case_method, test_case_result) + self._execute_test_case(test_suite, test_case_method, test_case_result) finally: try: @@ -440,11 +647,15 @@ def _run_test_case( test_case_result.update(Result.ERROR) def _execute_test_case( - self, test_case_method: MethodType, test_case_result: TestCaseResult + self, + test_suite: TestSuite, + test_case_method: MethodType, + test_case_result: TestCaseResult, ) -> None: - """Execute one test case, record the result and handle failures. + """Execute `test_case_method` from `test_suite`, record the result and handle failures. Args: + test_suite: The test suite object. test_case_method: The test case method. test_case_result: The test case level result object associated with the current test case. @@ -452,7 +663,7 @@ def _execute_test_case( test_case_name = test_case_method.__name__ try: self._logger.info(f"Starting test case execution: {test_case_name}") - test_case_method() + test_case_method(test_suite) test_case_result.update(Result.PASS) self._logger.info(f"Test case execution PASSED: {test_case_name}") diff --git a/dts/framework/settings.py b/dts/framework/settings.py index 609c8d0e62..2b8bfbe0ed 100644 --- a/dts/framework/settings.py +++ b/dts/framework/settings.py @@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser: "--test-cases", action=_env_arg("DTS_TESTCASES"), default="", - help="[DTS_TESTCASES] Comma-separated list of test cases to execute. " - "Unknown test cases will be silently ignored.", + help="[DTS_TESTCASES] Comma-separated list of test cases to execute.", ) parser.add_argument( diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index 4467749a9d..075195fd5b 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -25,7 +25,9 @@ import os.path from collections.abc import MutableSequence +from dataclasses import dataclass from enum import Enum, auto +from types import MethodType from .config import ( OS, @@ -36,10 +38,42 @@ CPUType, NodeConfiguration, NodeInfo, + TestSuiteConfig, ) from .exception import DTSError, ErrorSeverity from .logger import DTSLOG from .settings import SETTINGS +from .test_suite import TestSuite + + +@dataclass(slots=True, frozen=True) +class TestSuiteWithCases: + """A test suite class with test case methods. + + An auxiliary class holding a test case class with test case methods. The intended use of this + class is to hold a subset of test cases (which could be all test cases) because we don't have + all the data to instantiate the class at the point of inspection. The knowledge of this subset + is needed in case an error occurs before the class is instantiated and we need to record + which test cases were blocked by the error. + + Attributes: + test_suite_class: The test suite class. + test_cases: The test case methods. + """ + + test_suite_class: type[TestSuite] + test_cases: list[MethodType] + + def create_config(self) -> TestSuiteConfig: + """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases. + + Returns: + The :class:`TestSuiteConfig` representation. + """ + return TestSuiteConfig( + test_suite=self.test_suite_class.__name__, + test_cases=[test_case.__name__ for test_case in self.test_cases], + ) class Result(Enum): diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index b02fd36147..f9fe88093e 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -11,25 +11,17 @@ * Testbed (SUT, TG) configuration, * Packet sending and verification, * Test case verification. - -The module also defines a function, :func:`get_test_suites`, -for gathering test suites from a Python module. """ -import importlib -import inspect -import re from ipaddress import IPv4Interface, IPv6Interface, ip_interface -from types import MethodType -from typing import Any, ClassVar, Union +from typing import ClassVar, Union from scapy.layers.inet import IP # type: ignore[import] from scapy.layers.l2 import Ether # type: ignore[import] from scapy.packet import Packet, Padding # type: ignore[import] -from .exception import ConfigurationError, TestCaseVerifyError +from .exception import TestCaseVerifyError from .logger import DTSLOG, getLogger -from .settings import SETTINGS from .testbed_model import Port, PortLink, SutNode, TGNode from .utils import get_packet_summaries @@ -37,7 +29,6 @@ class TestSuite(object): """The base class with building blocks needed by most test cases. - * Test case filtering and collection, * Test suite setup/cleanup methods to override, * Test case setup/cleanup methods to override, * Test case verification, @@ -71,7 +62,6 @@ class TestSuite(object): #: will block the execution of all subsequent test suites in the current build target. is_blocking: ClassVar[bool] = False _logger: DTSLOG - _test_cases_to_run: list[str] _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -86,24 +76,19 @@ def __init__( self, sut_node: SutNode, tg_node: TGNode, - test_cases: list[str], ): """Initialize the test suite testbed information and basic configuration. - Process what test cases to run, find links between ports and set up - default IP addresses to be used when configuring them. + Find links between ports and set up default IP addresses to be used when + configuring them. Args: sut_node: The SUT node where the test suite will run. tg_node: The TG node where the test suite will run. - test_cases: The list of test cases to execute. - If empty, all test cases will be executed. """ self.sut_node = sut_node self.tg_node = tg_node self._logger = getLogger(self.__class__.__name__) - self._test_cases_to_run = test_cases - self._test_cases_to_run.extend(SETTINGS.test_cases) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( @@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: if received_packet.src != expected_packet.src or received_packet.dst != expected_packet.dst: return False return True - - def _get_functional_test_cases(self) -> list[MethodType]: - """Get all functional test cases defined in this TestSuite. - - Returns: - The list of functional test cases of this TestSuite. - """ - return self._get_test_cases(r"test_(?!perf_)") - - def _get_test_cases(self, test_case_regex: str) -> list[MethodType]: - """Return a list of test cases matching test_case_regex. - - Returns: - The list of test cases matching test_case_regex of this TestSuite. - """ - self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.") - filtered_test_cases = [] - for test_case_name, test_case in inspect.getmembers(self, inspect.ismethod): - if self._should_be_executed(test_case_name, test_case_regex): - filtered_test_cases.append(test_case) - cases_str = ", ".join((x.__name__ for x in filtered_test_cases)) - self._logger.debug(f"Found test cases '{cases_str}' in {self.__class__.__name__}.") - return filtered_test_cases - - def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool: - """Check whether the test case should be scheduled to be executed.""" - match = bool(re.match(test_case_regex, test_case_name)) - if self._test_cases_to_run: - return match and test_case_name in self._test_cases_to_run - - return match - - -def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: - r"""Find all :class:`TestSuite`\s in a Python module. - - Args: - testsuite_module_path: The path to the Python module. - - Returns: - The list of :class:`TestSuite`\s found within the Python module. - - Raises: - ConfigurationError: The test suite module was not found. - """ - - def is_test_suite(object: Any) -> bool: - try: - if issubclass(object, TestSuite) and object is not TestSuite: - return True - except TypeError: - return False - return False - - try: - testcase_module = importlib.import_module(testsuite_module_path) - except ModuleNotFoundError as e: - raise ConfigurationError(f"Test suite '{testsuite_module_path}' not found.") from e - return [ - test_suite_class - for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite) - ] diff --git a/dts/pyproject.toml b/dts/pyproject.toml index 28bd970ae4..8eb92b4f11 100644 --- a/dts/pyproject.toml +++ b/dts/pyproject.toml @@ -51,6 +51,9 @@ linters = "mccabe,pycodestyle,pydocstyle,pyflakes" format = "pylint" max_line_length = 100 +[tool.pylama.linter.pycodestyle] +ignore = "E203,W503" + [tool.pylama.linter.pydocstyle] convention = "google" diff --git a/dts/tests/TestSuite_os_udp.py b/dts/tests/TestSuite_os_udp.py index 2cf29d37bb..b4784dd95e 100644 --- a/dts/tests/TestSuite_os_udp.py +++ b/dts/tests/TestSuite_os_udp.py @@ -13,7 +13,7 @@ from framework.test_suite import TestSuite -class TestOSUdp(TestSuite): +class TestOsUdp(TestSuite): """IPv4 UDP OS routing test suite.""" def set_up_suite(self) -> None: diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py index 5e2bac14bd..7b2a0e97f8 100644 --- a/dts/tests/TestSuite_smoke_tests.py +++ b/dts/tests/TestSuite_smoke_tests.py @@ -21,7 +21,7 @@ from framework.utils import REGEX_FOR_PCI_ADDRESS -class SmokeTests(TestSuite): +class TestSmokeTests(TestSuite): """DPDK and infrastructure smoke test suite. The test cases validate the most basic DPDK functionality needed for all other test suites. -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 4/7] dts: reorganize test result 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš ` (2 preceding siblings ...) 2024-03-01 10:55 ` [PATCH v4 3/7] dts: filter test suites in executions Juraj Linkeš @ 2024-03-01 10:55 ` Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš ` (4 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro, npratte Cc: dev, Juraj Linkeš The current order of Result classes in the test_suite.py module is guided by the needs of type hints, which is not as intuitively readable as ordering them by the occurrences in code. The order goes from the topmost level to lowermost: BaseResult DTSResult ExecutionResult BuildTargetResult TestSuiteResult TestCaseResult This is the same order as they're used in the runner module and they're also used in the same order between themselves in the test_result module. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/test_result.py | 411 ++++++++++++++++++----------------- 1 file changed, 206 insertions(+), 205 deletions(-) diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index 075195fd5b..abdbafab10 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -28,6 +28,7 @@ from dataclasses import dataclass from enum import Enum, auto from types import MethodType +from typing import Union from .config import ( OS, @@ -129,58 +130,6 @@ def __bool__(self) -> bool: return bool(self.result) -class Statistics(dict): - """How many test cases ended in which result state along some other basic information. - - Subclassing :class:`dict` provides a convenient way to format the data. - - The data are stored in the following keys: - - * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases. - * **DPDK VERSION** (:class:`str`) -- The tested DPDK version. - """ - - def __init__(self, dpdk_version: str | None): - """Extend the constructor with keys in which the data are stored. - - Args: - dpdk_version: The version of tested DPDK. - """ - super(Statistics, self).__init__() - for result in Result: - self[result.name] = 0 - self["PASS RATE"] = 0.0 - self["DPDK VERSION"] = dpdk_version - - def __iadd__(self, other: Result) -> "Statistics": - """Add a Result to the final count. - - Example: - stats: Statistics = Statistics() # empty Statistics - stats += Result.PASS # add a Result to `stats` - - Args: - other: The Result to add to this statistics object. - - Returns: - The modified statistics object. - """ - self[other.name] += 1 - self["PASS RATE"] = ( - float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result) - ) - return self - - def __str__(self) -> str: - """Each line contains the formatted key = value pair.""" - stats_str = "" - for key, value in self.items(): - stats_str += f"{key:<12} = {value}\n" - # according to docs, we should use \n when writing to text files - # on all platforms - return stats_str - - class BaseResult(object): """Common data and behavior of DTS results. @@ -245,7 +194,7 @@ def get_errors(self) -> list[Exception]: """ return self._get_setup_teardown_errors() + self._get_inner_errors() - def add_stats(self, statistics: Statistics) -> None: + def add_stats(self, statistics: "Statistics") -> None: """Collate stats from the whole result hierarchy. Args: @@ -255,91 +204,149 @@ def add_stats(self, statistics: Statistics) -> None: inner_result.add_stats(statistics) -class TestCaseResult(BaseResult, FixtureResult): - r"""The test case specific result. +class DTSResult(BaseResult): + """Stores environment information and test results from a DTS run. - Stores the result of the actual test case. This is done by adding an extra superclass - in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and - the class is itself a record of the test case. + * Execution level information, such as testbed and the test suite list, + * Build target level information, such as compiler, target OS and cpu, + * Test suite and test case results, + * All errors that are caught and recorded during DTS execution. + + The information is stored hierarchically. This is the first level of the hierarchy + and as such is where the data form the whole hierarchy is collated or processed. + + The internal list stores the results of all executions. Attributes: - test_case_name: The test case name. + dpdk_version: The DPDK version to record. """ - test_case_name: str + dpdk_version: str | None + _logger: DTSLOG + _errors: list[Exception] + _return_code: ErrorSeverity + _stats_result: Union["Statistics", None] + _stats_filename: str - def __init__(self, test_case_name: str): - """Extend the constructor with `test_case_name`. + def __init__(self, logger: DTSLOG): + """Extend the constructor with top-level specifics. Args: - test_case_name: The test case's name. + logger: The logger instance the whole result will use. """ - super(TestCaseResult, self).__init__() - self.test_case_name = test_case_name + super(DTSResult, self).__init__() + self.dpdk_version = None + self._logger = logger + self._errors = [] + self._return_code = ErrorSeverity.NO_ERR + self._stats_result = None + self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") - def update(self, result: Result, error: Exception | None = None) -> None: - """Update the test case result. + def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult": + """Add and return the inner result (execution). - This updates the result of the test case itself and doesn't affect - the results of the setup and teardown steps in any way. + Args: + sut_node: The SUT node's test run configuration. + + Returns: + The execution's result. + """ + execution_result = ExecutionResult(sut_node) + self._inner_results.append(execution_result) + return execution_result + + def add_error(self, error: Exception) -> None: + """Record an error that occurred outside any execution. Args: - result: The result of the test case. - error: The error that occurred in case of a failure. + error: The exception to record. """ - self.result = result - self.error = error + self._errors.append(error) - def _get_inner_errors(self) -> list[Exception]: - if self.error: - return [self.error] - return [] + def process(self) -> None: + """Process the data after a whole DTS run. - def add_stats(self, statistics: Statistics) -> None: - r"""Add the test case result to statistics. + The data is added to inner objects during runtime and this object is not updated + at that time. This requires us to process the inner data after it's all been gathered. - The base method goes through the hierarchy recursively and this method is here to stop - the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree. + The processing gathers all errors and the statistics of test case results. + """ + self._errors += self.get_errors() + if self._errors and self._logger: + self._logger.debug("Summary of errors:") + for error in self._errors: + self._logger.debug(repr(error)) - Args: - statistics: The :class:`Statistics` object where the stats will be added. + self._stats_result = Statistics(self.dpdk_version) + self.add_stats(self._stats_result) + with open(self._stats_filename, "w+") as stats_file: + stats_file.write(str(self._stats_result)) + + def get_return_code(self) -> int: + """Go through all stored Exceptions and return the final DTS error code. + + Returns: + The highest error code found. """ - statistics += self.result + for error in self._errors: + error_return_code = ErrorSeverity.GENERIC_ERR + if isinstance(error, DTSError): + error_return_code = error.severity - def __bool__(self) -> bool: - """The test case passed only if setup, teardown and the test case itself passed.""" - return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) + if error_return_code > self._return_code: + self._return_code = error_return_code + return int(self._return_code) -class TestSuiteResult(BaseResult): - """The test suite specific result. - The internal list stores the results of all test cases in a given test suite. +class ExecutionResult(BaseResult): + """The execution specific result. + + The internal list stores the results of all build targets in a given execution. Attributes: - suite_name: The test suite name. + sut_node: The SUT node used in the execution. + sut_os_name: The operating system of the SUT node. + sut_os_version: The operating system version of the SUT node. + sut_kernel_version: The operating system kernel version of the SUT node. """ - suite_name: str + sut_node: NodeConfiguration + sut_os_name: str + sut_os_version: str + sut_kernel_version: str - def __init__(self, suite_name: str): - """Extend the constructor with `suite_name`. + def __init__(self, sut_node: NodeConfiguration): + """Extend the constructor with the `sut_node`'s config. Args: - suite_name: The test suite's name. + sut_node: The SUT node's test run configuration used in the execution. """ - super(TestSuiteResult, self).__init__() - self.suite_name = suite_name + super(ExecutionResult, self).__init__() + self.sut_node = sut_node - def add_test_case(self, test_case_name: str) -> TestCaseResult: - """Add and return the inner result (test case). + def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult": + """Add and return the inner result (build target). + + Args: + build_target: The build target's test run configuration. Returns: - The test case's result. + The build target's result. """ - test_case_result = TestCaseResult(test_case_name) - self._inner_results.append(test_case_result) - return test_case_result + build_target_result = BuildTargetResult(build_target) + self._inner_results.append(build_target_result) + return build_target_result + + def add_sut_info(self, sut_info: NodeInfo) -> None: + """Add SUT information gathered at runtime. + + Args: + sut_info: The additional SUT node information. + """ + self.sut_os_name = sut_info.os_name + self.sut_os_version = sut_info.os_version + self.sut_kernel_version = sut_info.kernel_version class BuildTargetResult(BaseResult): @@ -386,7 +393,7 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None: self.compiler_version = versions.compiler_version self.dpdk_version = versions.dpdk_version - def add_test_suite(self, test_suite_name: str) -> TestSuiteResult: + def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult": """Add and return the inner result (test suite). Returns: @@ -397,146 +404,140 @@ def add_test_suite(self, test_suite_name: str) -> TestSuiteResult: return test_suite_result -class ExecutionResult(BaseResult): - """The execution specific result. +class TestSuiteResult(BaseResult): + """The test suite specific result. - The internal list stores the results of all build targets in a given execution. + The internal list stores the results of all test cases in a given test suite. Attributes: - sut_node: The SUT node used in the execution. - sut_os_name: The operating system of the SUT node. - sut_os_version: The operating system version of the SUT node. - sut_kernel_version: The operating system kernel version of the SUT node. + suite_name: The test suite name. """ - sut_node: NodeConfiguration - sut_os_name: str - sut_os_version: str - sut_kernel_version: str + suite_name: str - def __init__(self, sut_node: NodeConfiguration): - """Extend the constructor with the `sut_node`'s config. + def __init__(self, suite_name: str): + """Extend the constructor with `suite_name`. Args: - sut_node: The SUT node's test run configuration used in the execution. + suite_name: The test suite's name. """ - super(ExecutionResult, self).__init__() - self.sut_node = sut_node - - def add_build_target(self, build_target: BuildTargetConfiguration) -> BuildTargetResult: - """Add and return the inner result (build target). + super(TestSuiteResult, self).__init__() + self.suite_name = suite_name - Args: - build_target: The build target's test run configuration. + def add_test_case(self, test_case_name: str) -> "TestCaseResult": + """Add and return the inner result (test case). Returns: - The build target's result. - """ - build_target_result = BuildTargetResult(build_target) - self._inner_results.append(build_target_result) - return build_target_result - - def add_sut_info(self, sut_info: NodeInfo) -> None: - """Add SUT information gathered at runtime. - - Args: - sut_info: The additional SUT node information. + The test case's result. """ - self.sut_os_name = sut_info.os_name - self.sut_os_version = sut_info.os_version - self.sut_kernel_version = sut_info.kernel_version + test_case_result = TestCaseResult(test_case_name) + self._inner_results.append(test_case_result) + return test_case_result -class DTSResult(BaseResult): - """Stores environment information and test results from a DTS run. - - * Execution level information, such as testbed and the test suite list, - * Build target level information, such as compiler, target OS and cpu, - * Test suite and test case results, - * All errors that are caught and recorded during DTS execution. - - The information is stored hierarchically. This is the first level of the hierarchy - and as such is where the data form the whole hierarchy is collated or processed. +class TestCaseResult(BaseResult, FixtureResult): + r"""The test case specific result. - The internal list stores the results of all executions. + Stores the result of the actual test case. This is done by adding an extra superclass + in :class:`FixtureResult`. The setup and teardown results are :class:`FixtureResult`\s and + the class is itself a record of the test case. Attributes: - dpdk_version: The DPDK version to record. + test_case_name: The test case name. """ - dpdk_version: str | None - _logger: DTSLOG - _errors: list[Exception] - _return_code: ErrorSeverity - _stats_result: Statistics | None - _stats_filename: str + test_case_name: str - def __init__(self, logger: DTSLOG): - """Extend the constructor with top-level specifics. + def __init__(self, test_case_name: str): + """Extend the constructor with `test_case_name`. Args: - logger: The logger instance the whole result will use. + test_case_name: The test case's name. """ - super(DTSResult, self).__init__() - self.dpdk_version = None - self._logger = logger - self._errors = [] - self._return_code = ErrorSeverity.NO_ERR - self._stats_result = None - self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") + super(TestCaseResult, self).__init__() + self.test_case_name = test_case_name - def add_execution(self, sut_node: NodeConfiguration) -> ExecutionResult: - """Add and return the inner result (execution). + def update(self, result: Result, error: Exception | None = None) -> None: + """Update the test case result. - Args: - sut_node: The SUT node's test run configuration. + This updates the result of the test case itself and doesn't affect + the results of the setup and teardown steps in any way. - Returns: - The execution's result. + Args: + result: The result of the test case. + error: The error that occurred in case of a failure. """ - execution_result = ExecutionResult(sut_node) - self._inner_results.append(execution_result) - return execution_result + self.result = result + self.error = error - def add_error(self, error: Exception) -> None: - """Record an error that occurred outside any execution. + def _get_inner_errors(self) -> list[Exception]: + if self.error: + return [self.error] + return [] + + def add_stats(self, statistics: "Statistics") -> None: + r"""Add the test case result to statistics. + + The base method goes through the hierarchy recursively and this method is here to stop + the recursion, as the :class:`TestCaseResult`\s are the leaves of the hierarchy tree. Args: - error: The exception to record. + statistics: The :class:`Statistics` object where the stats will be added. """ - self._errors.append(error) + statistics += self.result - def process(self) -> None: - """Process the data after a whole DTS run. + def __bool__(self) -> bool: + """The test case passed only if setup, teardown and the test case itself passed.""" + return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) - The data is added to inner objects during runtime and this object is not updated - at that time. This requires us to process the inner data after it's all been gathered. - The processing gathers all errors and the statistics of test case results. +class Statistics(dict): + """How many test cases ended in which result state along some other basic information. + + Subclassing :class:`dict` provides a convenient way to format the data. + + The data are stored in the following keys: + + * **PASS RATE** (:class:`int`) -- The FAIL/PASS ratio of all test cases. + * **DPDK VERSION** (:class:`str`) -- The tested DPDK version. + """ + + def __init__(self, dpdk_version: str | None): + """Extend the constructor with keys in which the data are stored. + + Args: + dpdk_version: The version of tested DPDK. """ - self._errors += self.get_errors() - if self._errors and self._logger: - self._logger.debug("Summary of errors:") - for error in self._errors: - self._logger.debug(repr(error)) + super(Statistics, self).__init__() + for result in Result: + self[result.name] = 0 + self["PASS RATE"] = 0.0 + self["DPDK VERSION"] = dpdk_version - self._stats_result = Statistics(self.dpdk_version) - self.add_stats(self._stats_result) - with open(self._stats_filename, "w+") as stats_file: - stats_file.write(str(self._stats_result)) + def __iadd__(self, other: Result) -> "Statistics": + """Add a Result to the final count. - def get_return_code(self) -> int: - """Go through all stored Exceptions and return the final DTS error code. + Example: + stats: Statistics = Statistics() # empty Statistics + stats += Result.PASS # add a Result to `stats` + + Args: + other: The Result to add to this statistics object. Returns: - The highest error code found. + The modified statistics object. """ - for error in self._errors: - error_return_code = ErrorSeverity.GENERIC_ERR - if isinstance(error, DTSError): - error_return_code = error.severity - - if error_return_code > self._return_code: - self._return_code = error_return_code + self[other.name] += 1 + self["PASS RATE"] = ( + float(self[Result.PASS.name]) * 100 / sum(self[result.name] for result in Result) + ) + return self - return int(self._return_code) + def __str__(self) -> str: + """Each line contains the formatted key = value pair.""" + stats_str = "" + for key, value in self.items(): + stats_str += f"{key:<12} = {value}\n" + # according to docs, we should use \n when writing to text files + # on all platforms + return stats_str -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 5/7] dts: block all test cases when earlier setup fails 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš ` (3 preceding siblings ...) 2024-03-01 10:55 ` [PATCH v4 4/7] dts: reorganize test result Juraj Linkeš @ 2024-03-01 10:55 ` Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 6/7] dts: refactor logging configuration Juraj Linkeš ` (3 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro, npratte Cc: dev, Juraj Linkeš In case of a failure before a test suite, the child results will be recursively recorded as blocked, giving us a full report which was missing previously. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/runner.py | 21 ++-- dts/framework/test_result.py | 186 +++++++++++++++++++++++++---------- 2 files changed, 148 insertions(+), 59 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 5f6bcbbb86..864015c350 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -60,13 +60,15 @@ class DTSRunner: Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult` or one of its subclasses. The test case results are also recorded. - If an error occurs, the current stage is aborted, the error is recorded and the run continues in - the next iteration of the same stage. The return code is the highest `severity` of all + If an error occurs, the current stage is aborted, the error is recorded, everything in + the inner stages is marked as blocked and the run continues in the next iteration + of the same stage. The return code is the highest `severity` of all :class:`~.framework.exception.DTSError`\s. Example: - An error occurs in a build target setup. The current build target is aborted and the run - continues with the next build target. If the errored build target was the last one in the + An error occurs in a build target setup. The current build target is aborted, + all test suites and their test cases are marked as blocked and the run continues + with the next build target. If the errored build target was the last one in the given execution, the next execution begins. """ @@ -100,6 +102,10 @@ def run(self): test case within the test suite is set up, executed and torn down. After all test cases have been executed, the test suite is torn down and the next build target will be tested. + In order to properly mark test suites and test cases as blocked in case of a failure, + we need to have discovered which test suites and test cases to run before any failures + happen. The discovery happens at the earliest point at the start of each execution. + All the nested steps look like this: #. Execution setup @@ -134,7 +140,7 @@ def run(self): self._logger.info( f"Running execution with SUT '{execution.system_under_test_node.name}'." ) - execution_result = self._result.add_execution(execution.system_under_test_node) + execution_result = self._result.add_execution(execution) # we don't want to modify the original config, so create a copy execution_test_suites = list(execution.test_suites) if not execution.skip_smoke_tests: @@ -143,6 +149,7 @@ def run(self): test_suites_with_cases = self._get_test_suites_with_cases( execution_test_suites, execution.func, execution.perf ) + execution_result.test_suites_with_cases = test_suites_with_cases except Exception as e: self._logger.exception( f"Invalid test suite configuration found: " f"{execution_test_suites}." @@ -493,9 +500,7 @@ def _run_test_suites( """ end_build_target = False for test_suite_with_cases in test_suites_with_cases: - test_suite_result = build_target_result.add_test_suite( - test_suite_with_cases.test_suite_class.__name__ - ) + test_suite_result = build_target_result.add_test_suite(test_suite_with_cases) try: self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases) except BlockingTestSuiteError as e: diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index abdbafab10..eedb2d20ee 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -37,7 +37,7 @@ BuildTargetInfo, Compiler, CPUType, - NodeConfiguration, + ExecutionConfiguration, NodeInfo, TestSuiteConfig, ) @@ -88,6 +88,8 @@ class Result(Enum): ERROR = auto() #: SKIP = auto() + #: + BLOCK = auto() def __bool__(self) -> bool: """Only PASS is True.""" @@ -141,21 +143,26 @@ class BaseResult(object): Attributes: setup_result: The result of the setup of the particular stage. teardown_result: The results of the teardown of the particular stage. + child_results: The results of the descendants in the results hierarchy. """ setup_result: FixtureResult teardown_result: FixtureResult - _inner_results: MutableSequence["BaseResult"] + child_results: MutableSequence["BaseResult"] def __init__(self): """Initialize the constructor.""" self.setup_result = FixtureResult() self.teardown_result = FixtureResult() - self._inner_results = [] + self.child_results = [] def update_setup(self, result: Result, error: Exception | None = None) -> None: """Store the setup result. + If the result is :attr:`~Result.BLOCK`, :attr:`~Result.ERROR` or :attr:`~Result.FAIL`, + then the corresponding child results in result hierarchy + are also marked with :attr:`~Result.BLOCK`. + Args: result: The result of the setup. error: The error that occurred in case of a failure. @@ -163,6 +170,16 @@ def update_setup(self, result: Result, error: Exception | None = None) -> None: self.setup_result.result = result self.setup_result.error = error + if result in [Result.BLOCK, Result.ERROR, Result.FAIL]: + self.update_teardown(Result.BLOCK) + self._block_result() + + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed. + + The blocking of child results should be done in overloaded methods. + """ + def update_teardown(self, result: Result, error: Exception | None = None) -> None: """Store the teardown result. @@ -181,10 +198,8 @@ def _get_setup_teardown_errors(self) -> list[Exception]: errors.append(self.teardown_result.error) return errors - def _get_inner_errors(self) -> list[Exception]: - return [ - error for inner_result in self._inner_results for error in inner_result.get_errors() - ] + def _get_child_errors(self) -> list[Exception]: + return [error for child_result in self.child_results for error in child_result.get_errors()] def get_errors(self) -> list[Exception]: """Compile errors from the whole result hierarchy. @@ -192,7 +207,7 @@ def get_errors(self) -> list[Exception]: Returns: The errors from setup, teardown and all errors found in the whole result hierarchy. """ - return self._get_setup_teardown_errors() + self._get_inner_errors() + return self._get_setup_teardown_errors() + self._get_child_errors() def add_stats(self, statistics: "Statistics") -> None: """Collate stats from the whole result hierarchy. @@ -200,8 +215,8 @@ def add_stats(self, statistics: "Statistics") -> None: Args: statistics: The :class:`Statistics` object where the stats will be collated. """ - for inner_result in self._inner_results: - inner_result.add_stats(statistics) + for child_result in self.child_results: + child_result.add_stats(statistics) class DTSResult(BaseResult): @@ -242,18 +257,18 @@ def __init__(self, logger: DTSLOG): self._stats_result = None self._stats_filename = os.path.join(SETTINGS.output_dir, "statistics.txt") - def add_execution(self, sut_node: NodeConfiguration) -> "ExecutionResult": - """Add and return the inner result (execution). + def add_execution(self, execution: ExecutionConfiguration) -> "ExecutionResult": + """Add and return the child result (execution). Args: - sut_node: The SUT node's test run configuration. + execution: The execution's test run configuration. Returns: The execution's result. """ - execution_result = ExecutionResult(sut_node) - self._inner_results.append(execution_result) - return execution_result + result = ExecutionResult(execution) + self.child_results.append(result) + return result def add_error(self, error: Exception) -> None: """Record an error that occurred outside any execution. @@ -266,8 +281,8 @@ def add_error(self, error: Exception) -> None: def process(self) -> None: """Process the data after a whole DTS run. - The data is added to inner objects during runtime and this object is not updated - at that time. This requires us to process the inner data after it's all been gathered. + The data is added to child objects during runtime and this object is not updated + at that time. This requires us to process the child data after it's all been gathered. The processing gathers all errors and the statistics of test case results. """ @@ -305,28 +320,30 @@ class ExecutionResult(BaseResult): The internal list stores the results of all build targets in a given execution. Attributes: - sut_node: The SUT node used in the execution. sut_os_name: The operating system of the SUT node. sut_os_version: The operating system version of the SUT node. sut_kernel_version: The operating system kernel version of the SUT node. """ - sut_node: NodeConfiguration sut_os_name: str sut_os_version: str sut_kernel_version: str + _config: ExecutionConfiguration + _parent_result: DTSResult + _test_suites_with_cases: list[TestSuiteWithCases] - def __init__(self, sut_node: NodeConfiguration): - """Extend the constructor with the `sut_node`'s config. + def __init__(self, execution: ExecutionConfiguration): + """Extend the constructor with the execution's config and DTSResult. Args: - sut_node: The SUT node's test run configuration used in the execution. + execution: The execution's test run configuration. """ super(ExecutionResult, self).__init__() - self.sut_node = sut_node + self._config = execution + self._test_suites_with_cases = [] def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTargetResult": - """Add and return the inner result (build target). + """Add and return the child result (build target). Args: build_target: The build target's test run configuration. @@ -334,9 +351,34 @@ def add_build_target(self, build_target: BuildTargetConfiguration) -> "BuildTarg Returns: The build target's result. """ - build_target_result = BuildTargetResult(build_target) - self._inner_results.append(build_target_result) - return build_target_result + result = BuildTargetResult( + self._test_suites_with_cases, + build_target, + ) + self.child_results.append(result) + return result + + @property + def test_suites_with_cases(self) -> list[TestSuiteWithCases]: + """The test suites with test cases to be executed in this execution. + + The test suites can only be assigned once. + + Returns: + The list of test suites with test cases. If an error occurs between + the initialization of :class:`ExecutionResult` and assigning test cases to the instance, + return an empty list, representing that we don't know what to execute. + """ + return self._test_suites_with_cases + + @test_suites_with_cases.setter + def test_suites_with_cases(self, test_suites_with_cases: list[TestSuiteWithCases]) -> None: + if self._test_suites_with_cases: + raise ValueError( + "Attempted to assign test suites to an execution result " + "which already has test suites." + ) + self._test_suites_with_cases = test_suites_with_cases def add_sut_info(self, sut_info: NodeInfo) -> None: """Add SUT information gathered at runtime. @@ -348,6 +390,12 @@ def add_sut_info(self, sut_info: NodeInfo) -> None: self.sut_os_version = sut_info.os_version self.sut_kernel_version = sut_info.kernel_version + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + for build_target in self._config.build_targets: + child_result = self.add_build_target(build_target) + child_result.update_setup(Result.BLOCK) + class BuildTargetResult(BaseResult): """The build target specific result. @@ -369,11 +417,17 @@ class BuildTargetResult(BaseResult): compiler: Compiler compiler_version: str | None dpdk_version: str | None + _test_suites_with_cases: list[TestSuiteWithCases] - def __init__(self, build_target: BuildTargetConfiguration): - """Extend the constructor with the `build_target`'s build target config. + def __init__( + self, + test_suites_with_cases: list[TestSuiteWithCases], + build_target: BuildTargetConfiguration, + ): + """Extend the constructor with the build target's config and ExecutionResult. Args: + test_suites_with_cases: The test suites with test cases to be run in this build target. build_target: The build target's test run configuration. """ super(BuildTargetResult, self).__init__() @@ -383,6 +437,23 @@ def __init__(self, build_target: BuildTargetConfiguration): self.compiler = build_target.compiler self.compiler_version = None self.dpdk_version = None + self._test_suites_with_cases = test_suites_with_cases + + def add_test_suite( + self, + test_suite_with_cases: TestSuiteWithCases, + ) -> "TestSuiteResult": + """Add and return the child result (test suite). + + Args: + test_suite_with_cases: The test suite with test cases. + + Returns: + The test suite's result. + """ + result = TestSuiteResult(test_suite_with_cases) + self.child_results.append(result) + return result def add_build_target_info(self, versions: BuildTargetInfo) -> None: """Add information about the build target gathered at runtime. @@ -393,15 +464,11 @@ def add_build_target_info(self, versions: BuildTargetInfo) -> None: self.compiler_version = versions.compiler_version self.dpdk_version = versions.dpdk_version - def add_test_suite(self, test_suite_name: str) -> "TestSuiteResult": - """Add and return the inner result (test suite). - - Returns: - The test suite's result. - """ - test_suite_result = TestSuiteResult(test_suite_name) - self._inner_results.append(test_suite_result) - return test_suite_result + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + for test_suite_with_cases in self._test_suites_with_cases: + child_result = self.add_test_suite(test_suite_with_cases) + child_result.update_setup(Result.BLOCK) class TestSuiteResult(BaseResult): @@ -410,29 +477,42 @@ class TestSuiteResult(BaseResult): The internal list stores the results of all test cases in a given test suite. Attributes: - suite_name: The test suite name. + test_suite_name: The test suite name. """ - suite_name: str + test_suite_name: str + _test_suite_with_cases: TestSuiteWithCases + _parent_result: BuildTargetResult + _child_configs: list[str] - def __init__(self, suite_name: str): - """Extend the constructor with `suite_name`. + def __init__(self, test_suite_with_cases: TestSuiteWithCases): + """Extend the constructor with test suite's config and BuildTargetResult. Args: - suite_name: The test suite's name. + test_suite_with_cases: The test suite with test cases. """ super(TestSuiteResult, self).__init__() - self.suite_name = suite_name + self.test_suite_name = test_suite_with_cases.test_suite_class.__name__ + self._test_suite_with_cases = test_suite_with_cases def add_test_case(self, test_case_name: str) -> "TestCaseResult": - """Add and return the inner result (test case). + """Add and return the child result (test case). + + Args: + test_case_name: The name of the test case. Returns: The test case's result. """ - test_case_result = TestCaseResult(test_case_name) - self._inner_results.append(test_case_result) - return test_case_result + result = TestCaseResult(test_case_name) + self.child_results.append(result) + return result + + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + for test_case_method in self._test_suite_with_cases.test_cases: + child_result = self.add_test_case(test_case_method.__name__) + child_result.update_setup(Result.BLOCK) class TestCaseResult(BaseResult, FixtureResult): @@ -449,7 +529,7 @@ class TestCaseResult(BaseResult, FixtureResult): test_case_name: str def __init__(self, test_case_name: str): - """Extend the constructor with `test_case_name`. + """Extend the constructor with test case's name and TestSuiteResult. Args: test_case_name: The test case's name. @@ -470,7 +550,7 @@ def update(self, result: Result, error: Exception | None = None) -> None: self.result = result self.error = error - def _get_inner_errors(self) -> list[Exception]: + def _get_child_errors(self) -> list[Exception]: if self.error: return [self.error] return [] @@ -486,6 +566,10 @@ def add_stats(self, statistics: "Statistics") -> None: """ statistics += self.result + def _block_result(self) -> None: + r"""Mark the result as :attr:`~Result.BLOCK`\ed.""" + self.update(Result.BLOCK) + def __bool__(self) -> bool: """The test case passed only if setup, teardown and the test case itself passed.""" return bool(self.setup_result) and bool(self.teardown_result) and bool(self.result) -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 6/7] dts: refactor logging configuration 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš ` (4 preceding siblings ...) 2024-03-01 10:55 ` [PATCH v4 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš @ 2024-03-01 10:55 ` Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 7/7] dts: improve test suite and case filtering Juraj Linkeš ` (2 subsequent siblings) 8 siblings, 0 replies; 44+ messages in thread From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro, npratte Cc: dev, Juraj Linkeš Remove unused parts of the code and add useful features: 1. Add DTS execution stages such as execution and test suite to better identify where in the DTS lifecycle we are when investigating logs, 2. Logging to separate files in specific stages, which is mainly useful for having test suite logs in additional separate files. 3. Remove the dependence on the settings module which enhances the usefulness of the logger module, as it can now be imported in more modules. The execution stages and the files to log to are the same for all DTS loggers. To achieve this, we have one DTS root logger which should be used for handling stage switching and all other loggers are children of this DTS root logger. The DTS root logger is the one where we change the behavior of all loggers (the stage and which files to log to) and the child loggers just log messages under a different name. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- dts/framework/logger.py | 246 +++++++++++------- dts/framework/remote_session/__init__.py | 6 +- .../interactive_remote_session.py | 6 +- .../remote_session/interactive_shell.py | 6 +- .../remote_session/remote_session.py | 8 +- dts/framework/runner.py | 23 +- dts/framework/test_result.py | 6 +- dts/framework/test_suite.py | 6 +- dts/framework/testbed_model/node.py | 11 +- dts/framework/testbed_model/os_session.py | 7 +- .../traffic_generator/traffic_generator.py | 6 +- dts/main.py | 3 - 12 files changed, 197 insertions(+), 137 deletions(-) diff --git a/dts/framework/logger.py b/dts/framework/logger.py index cfa6e8cd72..fc6c50c983 100644 --- a/dts/framework/logger.py +++ b/dts/framework/logger.py @@ -5,141 +5,195 @@ """DTS logger module. -DTS framework and TestSuite logs are saved in different log files. +The module provides several additional features: + + * The storage of DTS execution stages, + * Logging to console, a human-readable log file and a machine-readable log file, + * Optional log files for specific stages. """ import logging -import os.path -from typing import TypedDict +from enum import auto +from logging import FileHandler, StreamHandler +from pathlib import Path +from typing import ClassVar -from .settings import SETTINGS +from .utils import StrEnum date_fmt = "%Y/%m/%d %H:%M:%S" -stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" +stream_fmt = "%(asctime)s - %(stage)s - %(name)s - %(levelname)s - %(message)s" +dts_root_logger_name = "dts" + + +class DtsStage(StrEnum): + """The DTS execution stage.""" + + #: + pre_execution = auto() + #: + execution_setup = auto() + #: + execution_teardown = auto() + #: + build_target_setup = auto() + #: + build_target_teardown = auto() + #: + test_suite_setup = auto() + #: + test_suite = auto() + #: + test_suite_teardown = auto() + #: + post_execution = auto() + + +class DTSLogger(logging.Logger): + """The DTS logger class. + + The class extends the :class:`~logging.Logger` class to add the DTS execution stage information + to log records. The stage is common to all loggers, so it's stored in a class variable. + + Any time we switch to a new stage, we have the ability to log to an additional log file along + with a supplementary log file with machine-readable format. These two log files are used until + a new stage switch occurs. This is useful mainly for logging per test suite. + """ + _stage: ClassVar[DtsStage] = DtsStage.pre_execution + _extra_file_handlers: list[FileHandler] = [] -class DTSLOG(logging.LoggerAdapter): - """DTS logger adapter class for framework and testsuites. + def __init__(self, *args, **kwargs): + """Extend the constructor with extra file handlers.""" + self._extra_file_handlers = [] + super().__init__(*args, **kwargs) - The :option:`--verbose` command line argument and the :envvar:`DTS_VERBOSE` environment - variable control the verbosity of output. If enabled, all messages will be emitted to the - console. + def makeRecord(self, *args, **kwargs) -> logging.LogRecord: + """Generates a record with additional stage information. - The :option:`--output` command line argument and the :envvar:`DTS_OUTPUT_DIR` environment - variable modify the directory where the logs will be stored. + This is the default method for the :class:`~logging.Logger` class. We extend it + to add stage information to the record. - Attributes: - node: The additional identifier. Currently unused. - sh: The handler which emits logs to console. - fh: The handler which emits logs to a file. - verbose_fh: Just as fh, but logs with a different, more verbose, format. - """ + :meta private: + + Returns: + record: The generated record with the stage information. + """ + record = super().makeRecord(*args, **kwargs) + record.stage = DTSLogger._stage # type: ignore[attr-defined] + return record + + def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None: + """Add logger handlers to the DTS root logger. + + This method should be called only on the DTS root logger. + The log records from child loggers will propagate to these handlers. - _logger: logging.Logger - node: str - sh: logging.StreamHandler - fh: logging.FileHandler - verbose_fh: logging.FileHandler + Three handlers are added: - def __init__(self, logger: logging.Logger, node: str = "suite"): - """Extend the constructor with additional handlers. + * A console handler, + * A file handler, + * A supplementary file handler with machine-readable logs + containing more debug information. - One handler logs to the console, the other one to a file, with either a regular or verbose - format. + All log messages will be logged to files. The log level of the console handler + is configurable with `verbose`. Args: - logger: The logger from which to create the logger adapter. - node: An additional identifier. Currently unused. + verbose: If :data:`True`, log all messages to the console. + If :data:`False`, log to console with the :data:`logging.INFO` level. + output_dir: The directory where the log files will be located. + The names of the log files correspond to the name of the logger instance. """ - self._logger = logger - # 1 means log everything, this will be used by file handlers if their level - # is not set - self._logger.setLevel(1) + self.setLevel(1) - self.node = node - - # add handler to emit to stdout - sh = logging.StreamHandler() + sh = StreamHandler() sh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) - sh.setLevel(logging.INFO) # console handler default level + if not verbose: + sh.setLevel(logging.INFO) + self.addHandler(sh) - if SETTINGS.verbose is True: - sh.setLevel(logging.DEBUG) + self._add_file_handlers(Path(output_dir, self.name)) - self._logger.addHandler(sh) - self.sh = sh + def set_stage(self, stage: DtsStage, log_file_path: Path | None = None) -> None: + """Set the DTS execution stage and optionally log to files. - # prepare the output folder - if not os.path.exists(SETTINGS.output_dir): - os.mkdir(SETTINGS.output_dir) + Set the DTS execution stage of the DTSLog class and optionally add + file handlers to the instance if the log file name is provided. - logging_path_prefix = os.path.join(SETTINGS.output_dir, node) + The file handlers log all messages. One is a regular human-readable log file and + the other one is a machine-readable log file with extra debug information. - fh = logging.FileHandler(f"{logging_path_prefix}.log") - fh.setFormatter( - logging.Formatter( - fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s", - datefmt=date_fmt, - ) - ) + Args: + stage: The DTS stage to set. + log_file_path: An optional path of the log file to use. This should be a full path + (either relative or absolute) without suffix (which will be appended). + """ + self._remove_extra_file_handlers() - self._logger.addHandler(fh) - self.fh = fh + if DTSLogger._stage != stage: + self.info(f"Moving from stage '{DTSLogger._stage}' to stage '{stage}'.") + DTSLogger._stage = stage - # This outputs EVERYTHING, intended for post-mortem debugging - # Also optimized for processing via AWK (awk -F '|' ...) - verbose_fh = logging.FileHandler(f"{logging_path_prefix}.verbose.log") + if log_file_path: + self._extra_file_handlers.extend(self._add_file_handlers(log_file_path)) + + def _add_file_handlers(self, log_file_path: Path) -> list[FileHandler]: + """Add file handlers to the DTS root logger. + + Add two type of file handlers: + + * A regular file handler with suffix ".log", + * A machine-readable file handler with suffix ".verbose.log". + This format provides extensive information for debugging and detailed analysis. + + Args: + log_file_path: The full path to the log file without suffix. + + Returns: + The newly created file handlers. + + """ + fh = FileHandler(f"{log_file_path}.log") + fh.setFormatter(logging.Formatter(stream_fmt, date_fmt)) + self.addHandler(fh) + + verbose_fh = FileHandler(f"{log_file_path}.verbose.log") verbose_fh.setFormatter( logging.Formatter( - fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" + "%(asctime)s|%(stage)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|" "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s", datefmt=date_fmt, ) ) + self.addHandler(verbose_fh) - self._logger.addHandler(verbose_fh) - self.verbose_fh = verbose_fh - - super(DTSLOG, self).__init__(self._logger, dict(node=self.node)) - - def logger_exit(self) -> None: - """Remove the stream handler and the logfile handler.""" - for handler in (self.sh, self.fh, self.verbose_fh): - handler.flush() - self._logger.removeHandler(handler) - - -class _LoggerDictType(TypedDict): - logger: DTSLOG - name: str - node: str - + return [fh, verbose_fh] -# List for saving all loggers in use -_Loggers: list[_LoggerDictType] = [] + def _remove_extra_file_handlers(self) -> None: + """Remove any extra file handlers that have been added to the logger.""" + if self._extra_file_handlers: + for extra_file_handler in self._extra_file_handlers: + self.removeHandler(extra_file_handler) + self._extra_file_handlers = [] -def getLogger(name: str, node: str = "suite") -> DTSLOG: - """Get DTS logger adapter identified by name and node. - An existing logger will be returned if one with the exact name and node already exists. - A new one will be created and stored otherwise. +def get_dts_logger(name: str = None) -> DTSLogger: + """Return a DTS logger instance identified by `name`. Args: - name: The name of the logger. - node: An additional identifier for the logger. + name: If :data:`None`, return the DTS root logger. + If specified, return a child of the DTS root logger. Returns: - A logger uniquely identified by both name and node. + The DTS root logger or a child logger identified by `name`. """ - global _Loggers - # return saved logger - logger: _LoggerDictType - for logger in _Loggers: - if logger["name"] == name and logger["node"] == node: - return logger["logger"] - - # return new logger - dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node) - _Loggers.append({"logger": dts_logger, "name": name, "node": node}) - return dts_logger + original_logger_class = logging.getLoggerClass() + logging.setLoggerClass(DTSLogger) + if name: + name = f"{dts_root_logger_name}.{name}" + else: + name = dts_root_logger_name + logger = logging.getLogger(name) + logging.setLoggerClass(original_logger_class) + return logger # type: ignore[return-value] diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py index 51a01d6b5e..1910c81c3c 100644 --- a/dts/framework/remote_session/__init__.py +++ b/dts/framework/remote_session/__init__.py @@ -15,7 +15,7 @@ # pylama:ignore=W0611 from framework.config import NodeConfiguration -from framework.logger import DTSLOG +from framework.logger import DTSLogger from .interactive_remote_session import InteractiveRemoteSession from .interactive_shell import InteractiveShell @@ -26,7 +26,7 @@ def create_remote_session( - node_config: NodeConfiguration, name: str, logger: DTSLOG + node_config: NodeConfiguration, name: str, logger: DTSLogger ) -> RemoteSession: """Factory for non-interactive remote sessions. @@ -45,7 +45,7 @@ def create_remote_session( def create_interactive_session( - node_config: NodeConfiguration, logger: DTSLOG + node_config: NodeConfiguration, logger: DTSLogger ) -> InteractiveRemoteSession: """Factory for interactive remote sessions. diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py index 1cc82e3377..c50790db79 100644 --- a/dts/framework/remote_session/interactive_remote_session.py +++ b/dts/framework/remote_session/interactive_remote_session.py @@ -16,7 +16,7 @@ from framework.config import NodeConfiguration from framework.exception import SSHConnectionError -from framework.logger import DTSLOG +from framework.logger import DTSLogger class InteractiveRemoteSession: @@ -50,11 +50,11 @@ class InteractiveRemoteSession: username: str password: str session: SSHClient - _logger: DTSLOG + _logger: DTSLogger _node_config: NodeConfiguration _transport: Transport | None - def __init__(self, node_config: NodeConfiguration, logger: DTSLOG) -> None: + def __init__(self, node_config: NodeConfiguration, logger: DTSLogger) -> None: """Connect to the node during initialization. Args: diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py index b158f963b6..5cfe202e15 100644 --- a/dts/framework/remote_session/interactive_shell.py +++ b/dts/framework/remote_session/interactive_shell.py @@ -20,7 +20,7 @@ from paramiko import Channel, SSHClient, channel # type: ignore[import] -from framework.logger import DTSLOG +from framework.logger import DTSLogger from framework.settings import SETTINGS @@ -38,7 +38,7 @@ class InteractiveShell(ABC): _stdin: channel.ChannelStdinFile _stdout: channel.ChannelFile _ssh_channel: Channel - _logger: DTSLOG + _logger: DTSLogger _timeout: float _app_args: str @@ -61,7 +61,7 @@ class InteractiveShell(ABC): def __init__( self, interactive_session: SSHClient, - logger: DTSLOG, + logger: DTSLogger, get_privileged_command: Callable[[str], str] | None, app_args: str = "", timeout: float = SETTINGS.timeout, diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py index 2059f9a981..a69dc99400 100644 --- a/dts/framework/remote_session/remote_session.py +++ b/dts/framework/remote_session/remote_session.py @@ -9,14 +9,13 @@ the structure of the result of a command execution. """ - import dataclasses from abc import ABC, abstractmethod from pathlib import PurePath from framework.config import NodeConfiguration from framework.exception import RemoteCommandExecutionError -from framework.logger import DTSLOG +from framework.logger import DTSLogger from framework.settings import SETTINGS @@ -75,14 +74,14 @@ class RemoteSession(ABC): username: str password: str history: list[CommandResult] - _logger: DTSLOG + _logger: DTSLogger _node_config: NodeConfiguration def __init__( self, node_config: NodeConfiguration, session_name: str, - logger: DTSLOG, + logger: DTSLogger, ): """Connect to the node during initialization. @@ -181,7 +180,6 @@ def close(self, force: bool = False) -> None: Args: force: Force the closure of the connection. This may not clean up all resources. """ - self._logger.logger_exit() self._close(force) @abstractmethod diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 864015c350..dfee8ebd7c 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -19,9 +19,10 @@ import importlib import inspect -import logging +import os import re import sys +from pathlib import Path from types import MethodType from typing import Iterable @@ -38,7 +39,7 @@ SSHTimeoutError, TestCaseVerifyError, ) -from .logger import DTSLOG, getLogger +from .logger import DTSLogger, DtsStage, get_dts_logger from .settings import SETTINGS from .test_result import ( BuildTargetResult, @@ -73,7 +74,7 @@ class DTSRunner: """ _configuration: Configuration - _logger: DTSLOG + _logger: DTSLogger _result: DTSResult _test_suite_class_prefix: str _test_suite_module_prefix: str @@ -83,7 +84,10 @@ class DTSRunner: def __init__(self): """Initialize the instance with configuration, logger, result and string constants.""" self._configuration = load_config() - self._logger = getLogger("DTSRunner") + self._logger = get_dts_logger() + if not os.path.exists(SETTINGS.output_dir): + os.makedirs(SETTINGS.output_dir) + self._logger.add_dts_root_logger_handlers(SETTINGS.verbose, SETTINGS.output_dir) self._result = DTSResult(self._logger) self._test_suite_class_prefix = "Test" self._test_suite_module_prefix = "tests.TestSuite_" @@ -137,6 +141,7 @@ def run(self): # for all Execution sections for execution in self._configuration.executions: + self._logger.set_stage(DtsStage.execution_setup) self._logger.info( f"Running execution with SUT '{execution.system_under_test_node.name}'." ) @@ -168,6 +173,7 @@ def run(self): finally: try: + self._logger.set_stage(DtsStage.post_execution) for node in (sut_nodes | tg_nodes).values(): node.close() self._result.update_teardown(Result.PASS) @@ -426,6 +432,7 @@ def _run_execution( finally: try: + self._logger.set_stage(DtsStage.execution_teardown) sut_node.tear_down_execution() execution_result.update_teardown(Result.PASS) except Exception as e: @@ -454,6 +461,7 @@ def _run_build_target( with the current build target. test_suites_with_cases: The test suites with test cases to run. """ + self._logger.set_stage(DtsStage.build_target_setup) self._logger.info(f"Running build target '{build_target.name}'.") try: @@ -470,6 +478,7 @@ def _run_build_target( finally: try: + self._logger.set_stage(DtsStage.build_target_teardown) sut_node.tear_down_build_target() build_target_result.update_teardown(Result.PASS) except Exception as e: @@ -542,6 +551,9 @@ def _run_test_suite( BlockingTestSuiteError: If a blocking test suite fails. """ test_suite_name = test_suite_with_cases.test_suite_class.__name__ + self._logger.set_stage( + DtsStage.test_suite_setup, Path(SETTINGS.output_dir, test_suite_name) + ) test_suite = test_suite_with_cases.test_suite_class(sut_node, tg_node) try: self._logger.info(f"Starting test suite setup: {test_suite_name}") @@ -560,6 +572,7 @@ def _run_test_suite( ) finally: try: + self._logger.set_stage(DtsStage.test_suite_teardown) test_suite.tear_down_suite() sut_node.kill_cleanup_dpdk_apps() test_suite_result.update_teardown(Result.PASS) @@ -591,6 +604,7 @@ def _execute_test_suite( test_suite_result: The test suite level result object associated with the current test suite. """ + self._logger.set_stage(DtsStage.test_suite) for test_case_method in test_cases: test_case_name = test_case_method.__name__ test_case_result = test_suite_result.add_test_case(test_case_name) @@ -690,5 +704,4 @@ def _exit_dts(self) -> None: if self._logger: self._logger.info("DTS execution has ended.") - logging.shutdown() sys.exit(self._result.get_return_code()) diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index eedb2d20ee..28f84fd793 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -42,7 +42,7 @@ TestSuiteConfig, ) from .exception import DTSError, ErrorSeverity -from .logger import DTSLOG +from .logger import DTSLogger from .settings import SETTINGS from .test_suite import TestSuite @@ -237,13 +237,13 @@ class DTSResult(BaseResult): """ dpdk_version: str | None - _logger: DTSLOG + _logger: DTSLogger _errors: list[Exception] _return_code: ErrorSeverity _stats_result: Union["Statistics", None] _stats_filename: str - def __init__(self, logger: DTSLOG): + def __init__(self, logger: DTSLogger): """Extend the constructor with top-level specifics. Args: diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index f9fe88093e..365f80e21a 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -21,7 +21,7 @@ from scapy.packet import Packet, Padding # type: ignore[import] from .exception import TestCaseVerifyError -from .logger import DTSLOG, getLogger +from .logger import DTSLogger, get_dts_logger from .testbed_model import Port, PortLink, SutNode, TGNode from .utils import get_packet_summaries @@ -61,7 +61,7 @@ class TestSuite(object): #: Whether the test suite is blocking. A failure of a blocking test suite #: will block the execution of all subsequent test suites in the current build target. is_blocking: ClassVar[bool] = False - _logger: DTSLOG + _logger: DTSLogger _port_links: list[PortLink] _sut_port_ingress: Port _sut_port_egress: Port @@ -88,7 +88,7 @@ def __init__( """ self.sut_node = sut_node self.tg_node = tg_node - self._logger = getLogger(self.__class__.__name__) + self._logger = get_dts_logger(self.__class__.__name__) self._port_links = [] self._process_links() self._sut_port_ingress, self._tg_port_egress = ( diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py index 1a55fadf78..74061f6262 100644 --- a/dts/framework/testbed_model/node.py +++ b/dts/framework/testbed_model/node.py @@ -23,7 +23,7 @@ NodeConfiguration, ) from framework.exception import ConfigurationError -from framework.logger import DTSLOG, getLogger +from framework.logger import DTSLogger, get_dts_logger from framework.settings import SETTINGS from .cpu import ( @@ -63,7 +63,7 @@ class Node(ABC): name: str lcores: list[LogicalCore] ports: list[Port] - _logger: DTSLOG + _logger: DTSLogger _other_sessions: list[OSSession] _execution_config: ExecutionConfiguration virtual_devices: list[VirtualDevice] @@ -82,7 +82,7 @@ def __init__(self, node_config: NodeConfiguration): """ self.config = node_config self.name = node_config.name - self._logger = getLogger(self.name) + self._logger = get_dts_logger(self.name) self.main_session = create_session(self.config, self.name, self._logger) self._logger.info(f"Connected to node: {self.name}") @@ -189,7 +189,7 @@ def create_session(self, name: str) -> OSSession: connection = create_session( self.config, session_name, - getLogger(session_name, node=self.name), + get_dts_logger(session_name), ) self._other_sessions.append(connection) return connection @@ -299,7 +299,6 @@ def close(self) -> None: self.main_session.close() for session in self._other_sessions: session.close() - self._logger.logger_exit() @staticmethod def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: @@ -314,7 +313,7 @@ def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]: return func -def create_session(node_config: NodeConfiguration, name: str, logger: DTSLOG) -> OSSession: +def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession: """Factory for OS-aware sessions. Args: diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py index ac6bb5e112..6983aa4a77 100644 --- a/dts/framework/testbed_model/os_session.py +++ b/dts/framework/testbed_model/os_session.py @@ -21,7 +21,6 @@ the :attr:`~.node.Node.main_session` translates that to ``rm -rf`` if the node's OS is Linux and other commands for other OSs. It also translates the path to match the underlying OS. """ - from abc import ABC, abstractmethod from collections.abc import Iterable from ipaddress import IPv4Interface, IPv6Interface @@ -29,7 +28,7 @@ from typing import Type, TypeVar, Union from framework.config import Architecture, NodeConfiguration, NodeInfo -from framework.logger import DTSLOG +from framework.logger import DTSLogger from framework.remote_session import ( CommandResult, InteractiveRemoteSession, @@ -62,7 +61,7 @@ class OSSession(ABC): _config: NodeConfiguration name: str - _logger: DTSLOG + _logger: DTSLogger remote_session: RemoteSession interactive_session: InteractiveRemoteSession @@ -70,7 +69,7 @@ def __init__( self, node_config: NodeConfiguration, name: str, - logger: DTSLOG, + logger: DTSLogger, ): """Initialize the OS-aware session. diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py index c49fbff488..d86d7fb532 100644 --- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py +++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py @@ -13,7 +13,7 @@ from scapy.packet import Packet # type: ignore[import] from framework.config import TrafficGeneratorConfig -from framework.logger import DTSLOG, getLogger +from framework.logger import DTSLogger, get_dts_logger from framework.testbed_model.node import Node from framework.testbed_model.port import Port from framework.utils import get_packet_summaries @@ -28,7 +28,7 @@ class TrafficGenerator(ABC): _config: TrafficGeneratorConfig _tg_node: Node - _logger: DTSLOG + _logger: DTSLogger def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): """Initialize the traffic generator. @@ -39,7 +39,7 @@ def __init__(self, tg_node: Node, config: TrafficGeneratorConfig): """ self._config = config self._tg_node = tg_node - self._logger = getLogger(f"{self._tg_node.name} {self._config.traffic_generator_type}") + self._logger = get_dts_logger(f"{self._tg_node.name} {self._config.traffic_generator_type}") def send_packet(self, packet: Packet, port: Port) -> None: """Send `packet` and block until it is fully sent. diff --git a/dts/main.py b/dts/main.py index 1ffe8ff81f..fa878cc16e 100755 --- a/dts/main.py +++ b/dts/main.py @@ -6,8 +6,6 @@ """The DTS executable.""" -import logging - from framework import settings @@ -30,5 +28,4 @@ def main() -> None: # Main program begins here if __name__ == "__main__": - logging.raiseExceptions = True main() -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v4 7/7] dts: improve test suite and case filtering 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš ` (5 preceding siblings ...) 2024-03-01 10:55 ` [PATCH v4 6/7] dts: refactor logging configuration Juraj Linkeš @ 2024-03-01 10:55 ` Juraj Linkeš 2024-03-01 17:41 ` Jeremy Spewock 2024-03-01 16:11 ` [PATCH v4 0/7] test case blocking and logging Patrick Robb 2024-03-07 14:55 ` Thomas Monjalon 8 siblings, 1 reply; 44+ messages in thread From: Juraj Linkeš @ 2024-03-01 10:55 UTC (permalink / raw) To: thomas, Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro, npratte Cc: dev, Juraj Linkeš The two places where we specify which test suite and test cases to run are complimentary and not that intuitive to use. A unified way provides a better user experience. The syntax in test run configuration file has not changed, but the environment variable and the command line arguments was changed to match the config file syntax. This required changes in the settings module which greatly simplified the parsing of the environment variables while retaining the same functionality. Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> --- doc/guides/tools/dts.rst | 14 ++- dts/framework/config/__init__.py | 12 +- dts/framework/runner.py | 18 +-- dts/framework/settings.py | 187 ++++++++++++++----------------- dts/framework/test_suite.py | 2 +- 5 files changed, 114 insertions(+), 119 deletions(-) diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst index f686ca487c..d1c3c2af7a 100644 --- a/doc/guides/tools/dts.rst +++ b/doc/guides/tools/dts.rst @@ -215,28 +215,30 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet .. code-block:: console (dts-py3.10) $ ./main.py --help - usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN] + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN] Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority. options: -h, --help show this help message and exit --config-file CONFIG_FILE - [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml) + [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml) --output-dir OUTPUT_DIR, --output OUTPUT_DIR [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output) -t TIMEOUT, --timeout TIMEOUT [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15) -v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False) - -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None) + -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False) --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz) --compile-timeout COMPILE_TIMEOUT [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200) - --test-cases TEST_CASES - [DTS_TESTCASES] Comma-separated list of test cases to execute. Unknown test cases will be silently ignored. (default: ) + --test-suite TEST_SUITE [TEST_CASES ...] + [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment + variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...' + (default: []) --re-run RE_RUN, --re_run RE_RUN - [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs (default: 0) + [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0) The brackets contain the names of environment variables that set the same thing. diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py index c6a93b3b89..4cb5c74059 100644 --- a/dts/framework/config/__init__.py +++ b/dts/framework/config/__init__.py @@ -35,9 +35,9 @@ import json import os.path -import pathlib from dataclasses import dataclass, fields from enum import auto, unique +from pathlib import Path from typing import Union import warlock # type: ignore[import] @@ -53,7 +53,6 @@ TrafficGeneratorConfigDict, ) from framework.exception import ConfigurationError -from framework.settings import SETTINGS from framework.utils import StrEnum @@ -571,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration": return Configuration(executions=executions) -def load_config() -> Configuration: +def load_config(config_file_path: Path) -> Configuration: """Load DTS test run configuration from a file. Load the YAML test run configuration file @@ -581,13 +580,16 @@ def load_config() -> Configuration: The YAML test run configuration file is specified in the :option:`--config-file` command line argument or the :envvar:`DTS_CFG_FILE` environment variable. + Args: + config_file_path: The path to the YAML test run configuration file. + Returns: The parsed test run configuration. """ - with open(SETTINGS.config_file_path, "r") as f: + with open(config_file_path, "r") as f: config_data = yaml.safe_load(f) - schema_path = os.path.join(pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json") + schema_path = os.path.join(Path(__file__).parent.resolve(), "conf_yaml_schema.json") with open(schema_path, "r") as f: schema = json.load(f) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index dfee8ebd7c..db8e3ba96b 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -24,7 +24,7 @@ import sys from pathlib import Path from types import MethodType -from typing import Iterable +from typing import Iterable, Sequence from .config import ( BuildTargetConfiguration, @@ -83,7 +83,7 @@ class DTSRunner: def __init__(self): """Initialize the instance with configuration, logger, result and string constants.""" - self._configuration = load_config() + self._configuration = load_config(SETTINGS.config_file_path) self._logger = get_dts_logger() if not os.path.exists(SETTINGS.output_dir): os.makedirs(SETTINGS.output_dir) @@ -129,7 +129,7 @@ def run(self): #. Execution teardown The test cases are filtered according to the specification in the test run configuration and - the :option:`--test-cases` command line argument or + the :option:`--test-suite` command line argument or the :envvar:`DTS_TESTCASES` environment variable. """ sut_nodes: dict[str, SutNode] = {} @@ -147,7 +147,9 @@ def run(self): ) execution_result = self._result.add_execution(execution) # we don't want to modify the original config, so create a copy - execution_test_suites = list(execution.test_suites) + execution_test_suites = list( + SETTINGS.test_suites if SETTINGS.test_suites else execution.test_suites + ) if not execution.skip_smoke_tests: execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] try: @@ -226,7 +228,7 @@ def _get_test_suites_with_cases( test_suite_class = self._get_test_suite_class(test_suite_config.test_suite) test_cases = [] func_test_cases, perf_test_cases = self._filter_test_cases( - test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases) + test_suite_class, test_suite_config.test_cases ) if func: test_cases.extend(func_test_cases) @@ -302,7 +304,7 @@ def is_test_suite(object) -> bool: ) def _filter_test_cases( - self, test_suite_class: type[TestSuite], test_cases_to_run: set[str] + self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str] ) -> tuple[list[MethodType], list[MethodType]]: """Filter `test_cases_to_run` from `test_suite_class`. @@ -331,7 +333,9 @@ def _filter_test_cases( (name, method) for name, method in name_method_tuples if name in test_cases_to_run ] if len(name_method_tuples) < len(test_cases_to_run): - missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples} + missing_test_cases = set(test_cases_to_run) - { + name for name, _ in name_method_tuples + } raise ConfigurationError( f"Test cases {missing_test_cases} not found among methods " f"of {test_suite_class.__name__}." diff --git a/dts/framework/settings.py b/dts/framework/settings.py index 2b8bfbe0ed..688e8679a7 100644 --- a/dts/framework/settings.py +++ b/dts/framework/settings.py @@ -48,10 +48,11 @@ The path to a DPDK tarball, git commit ID, tag ID or tree ID to test. -.. option:: --test-cases -.. envvar:: DTS_TESTCASES +.. option:: --test-suite +.. envvar:: DTS_TEST_SUITES - A comma-separated list of test cases to execute. Unknown test cases will be silently ignored. + A test suite with test cases which may be specified multiple times. + In the environment variable, the suites are joined with a comma. .. option:: --re-run, --re_run .. envvar:: DTS_RERUN @@ -71,83 +72,13 @@ import argparse import os -from collections.abc import Callable, Iterable, Sequence from dataclasses import dataclass, field from pathlib import Path -from typing import Any, TypeVar +from typing import Any +from .config import TestSuiteConfig from .utils import DPDKGitTarball -_T = TypeVar("_T") - - -def _env_arg(env_var: str) -> Any: - """A helper method augmenting the argparse Action with environment variables. - - If the supplied environment variable is defined, then the default value - of the argument is modified. This satisfies the priority order of - command line argument > environment variable > default value. - - Arguments with no values (flags) should be defined using the const keyword argument - (True or False). When the argument is specified, it will be set to const, if not specified, - the default will be stored (possibly modified by the corresponding environment variable). - - Other arguments work the same as default argparse arguments, that is using - the default 'store' action. - - Returns: - The modified argparse.Action. - """ - - class _EnvironmentArgument(argparse.Action): - def __init__( - self, - option_strings: Sequence[str], - dest: str, - nargs: str | int | None = None, - const: bool | None = None, - default: Any = None, - type: Callable[[str], _T | argparse.FileType | None] = None, - choices: Iterable[_T] | None = None, - required: bool = False, - help: str | None = None, - metavar: str | tuple[str, ...] | None = None, - ) -> None: - env_var_value = os.environ.get(env_var) - default = env_var_value or default - if const is not None: - nargs = 0 - default = const if env_var_value else default - type = None - choices = None - metavar = None - super(_EnvironmentArgument, self).__init__( - option_strings, - dest, - nargs=nargs, - const=const, - default=default, - type=type, - choices=choices, - required=required, - help=help, - metavar=metavar, - ) - - def __call__( - self, - parser: argparse.ArgumentParser, - namespace: argparse.Namespace, - values: Any, - option_string: str = None, - ) -> None: - if self.const is not None: - setattr(namespace, self.dest, self.const) - else: - setattr(namespace, self.dest, values) - - return _EnvironmentArgument - @dataclass(slots=True) class Settings: @@ -171,7 +102,7 @@ class Settings: #: compile_timeout: float = 1200 #: - test_cases: list[str] = field(default_factory=list) + test_suites: list[TestSuiteConfig] = field(default_factory=list) #: re_run: int = 0 @@ -180,6 +111,31 @@ class Settings: def _get_parser() -> argparse.ArgumentParser: + """Create the argument parser for DTS. + + Command line options take precedence over environment variables, which in turn take precedence + over default values. + + Returns: + argparse.ArgumentParser: The configured argument parser with defined options. + """ + + def env_arg(env_var: str, default: Any) -> Any: + """A helper function augmenting the argparse with environment variables. + + If the supplied environment variable is defined, then the default value + of the argument is modified. This satisfies the priority order of + command line argument > environment variable > default value. + + Args: + env_var: Environment variable name. + default: Default value. + + Returns: + Environment variable or default value. + """ + return os.environ.get(env_var) or default + parser = argparse.ArgumentParser( description="Run DPDK test suites. All options may be specified with the environment " "variables provided in brackets. Command line arguments have higher priority.", @@ -188,25 +144,23 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "--config-file", - action=_env_arg("DTS_CFG_FILE"), - default=SETTINGS.config_file_path, + default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path), type=Path, - help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets.", + help="[DTS_CFG_FILE] The configuration file that describes the test cases, " + "SUTs and targets.", ) parser.add_argument( "--output-dir", "--output", - action=_env_arg("DTS_OUTPUT_DIR"), - default=SETTINGS.output_dir, + default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir), help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.", ) parser.add_argument( "-t", "--timeout", - action=_env_arg("DTS_TIMEOUT"), - default=SETTINGS.timeout, + default=env_arg("DTS_TIMEOUT", SETTINGS.timeout), type=float, help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.", ) @@ -214,9 +168,8 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "-v", "--verbose", - action=_env_arg("DTS_VERBOSE"), - default=SETTINGS.verbose, - const=True, + action="store_true", + default=env_arg("DTS_VERBOSE", SETTINGS.verbose), help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages " "to the console.", ) @@ -224,8 +177,8 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "-s", "--skip-setup", - action=_env_arg("DTS_SKIP_SETUP"), - const=True, + action="store_true", + default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup), help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.", ) @@ -233,8 +186,7 @@ def _get_parser() -> argparse.ArgumentParser: "--tarball", "--snapshot", "--git-ref", - action=_env_arg("DTS_DPDK_TARBALL"), - default=SETTINGS.dpdk_tarball_path, + default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path), type=Path, help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, " "tag ID or tree ID to test. To test local changes, first commit them, " @@ -243,36 +195,71 @@ def _get_parser() -> argparse.ArgumentParser: parser.add_argument( "--compile-timeout", - action=_env_arg("DTS_COMPILE_TIMEOUT"), - default=SETTINGS.compile_timeout, + default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout), type=float, help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.", ) parser.add_argument( - "--test-cases", - action=_env_arg("DTS_TESTCASES"), - default="", - help="[DTS_TESTCASES] Comma-separated list of test cases to execute.", + "--test-suite", + action="append", + nargs="+", + metavar=("TEST_SUITE", "TEST_CASES"), + default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites), + help="[DTS_TEST_SUITES] A list containing a test suite with test cases. " + "The first parameter is the test suite name, and the rest are test case names, " + "which are optional. May be specified multiple times. To specify multiple test suites in " + "the environment variable, join the lists with a comma. " + "Examples: " + "--test-suite suite case case --test-suite suite case ... | " + "DTS_TEST_SUITES='suite case case, suite case, ...' | " + "--test-suite suite --test-suite suite case ... | " + "DTS_TEST_SUITES='suite, suite case, ...'", ) parser.add_argument( "--re-run", "--re_run", - action=_env_arg("DTS_RERUN"), - default=SETTINGS.re_run, + default=env_arg("DTS_RERUN", SETTINGS.re_run), type=int, help="[DTS_RERUN] Re-run each test case the specified number of times " - "if a test failure occurs", + "if a test failure occurs.", ) return parser +def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]: + """Process the given argument to a list of :class:`TestSuiteConfig` to execute. + + Args: + args: The arguments to process. The args is a string from an environment variable + or a list of from the user input containing tests suites with tests cases, + each of which is a list of [test_suite, test_case, test_case, ...]. + + Returns: + A list of test suite configurations to execute. + """ + if isinstance(args, str): + # Environment variable in the form of "suite case case, suite case, suite, ..." + args = [suite_with_cases.split() for suite_with_cases in args.split(",")] + + test_suites_to_run = [] + for suite_with_cases in args: + test_suites_to_run.append( + TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:]) + ) + + return test_suites_to_run + + def get_settings() -> Settings: """Create new settings with inputs from the user. The inputs are taken from the command line and from environment variables. + + Returns: + The new settings object. """ parsed_args = _get_parser().parse_args() return Settings( @@ -287,6 +274,6 @@ def get_settings() -> Settings: else Path(parsed_args.tarball) ), compile_timeout=parsed_args.compile_timeout, - test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []), + test_suites=_process_test_suites(parsed_args.test_suite), re_run=parsed_args.re_run, ) diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index 365f80e21a..1957ea7328 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -40,7 +40,7 @@ class TestSuite(object): and functional test cases (all other test cases). By default, all test cases will be executed. A list of testcase names may be specified - in the YAML test run configuration file and in the :option:`--test-cases` command line argument + in the YAML test run configuration file and in the :option:`--test-suite` command line argument or in the :envvar:`DTS_TESTCASES` environment variable to filter which test cases to run. The union of both lists will be used. Any unknown test cases from the latter lists will be silently ignored. -- 2.34.1 ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 7/7] dts: improve test suite and case filtering 2024-03-01 10:55 ` [PATCH v4 7/7] dts: improve test suite and case filtering Juraj Linkeš @ 2024-03-01 17:41 ` Jeremy Spewock 2024-03-01 17:43 ` Jeremy Spewock 0 siblings, 1 reply; 44+ messages in thread From: Jeremy Spewock @ 2024-03-01 17:41 UTC (permalink / raw) To: Juraj Linkeš Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, npratte, dev For the whole series: Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu> On Fri, Mar 1, 2024 at 5:55 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote: > > The two places where we specify which test suite and test cases to run > are complimentary and not that intuitive to use. A unified way provides > a better user experience. > > The syntax in test run configuration file has not changed, but the > environment variable and the command line arguments was changed to match > the config file syntax. This required changes in the settings module > which greatly simplified the parsing of the environment variables while > retaining the same functionality. > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech> > --- > doc/guides/tools/dts.rst | 14 ++- > dts/framework/config/__init__.py | 12 +- > dts/framework/runner.py | 18 +-- > dts/framework/settings.py | 187 ++++++++++++++----------------- > dts/framework/test_suite.py | 2 +- > 5 files changed, 114 insertions(+), 119 deletions(-) > > diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst > index f686ca487c..d1c3c2af7a 100644 > --- a/doc/guides/tools/dts.rst > +++ b/doc/guides/tools/dts.rst > @@ -215,28 +215,30 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet > .. code-block:: console > > (dts-py3.10) $ ./main.py --help > - usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN] > + usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN] > > Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority. > > options: > -h, --help show this help message and exit > --config-file CONFIG_FILE > - [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml) > + [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml) > --output-dir OUTPUT_DIR, --output OUTPUT_DIR > [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output) > -t TIMEOUT, --timeout TIMEOUT > [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15) > -v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False) > - -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None) > + -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False) > --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL > [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz) > --compile-timeout COMPILE_TIMEOUT > [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200) > - --test-cases TEST_CASES > - [DTS_TESTCASES] Comma-separated list of test cases to execute. Unknown test cases will be silently ignored. (default: ) > + --test-suite TEST_SUITE [TEST_CASES ...] > + [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment > + variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...' > + (default: []) > --re-run RE_RUN, --re_run RE_RUN > - [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs (default: 0) > + [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0) > > > The brackets contain the names of environment variables that set the same thing. > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py > index c6a93b3b89..4cb5c74059 100644 > --- a/dts/framework/config/__init__.py > +++ b/dts/framework/config/__init__.py > @@ -35,9 +35,9 @@ > > import json > import os.path > -import pathlib > from dataclasses import dataclass, fields > from enum import auto, unique > +from pathlib import Path > from typing import Union > > import warlock # type: ignore[import] > @@ -53,7 +53,6 @@ > TrafficGeneratorConfigDict, > ) > from framework.exception import ConfigurationError > -from framework.settings import SETTINGS > from framework.utils import StrEnum > > > @@ -571,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration": > return Configuration(executions=executions) > > > -def load_config() -> Configuration: > +def load_config(config_file_path: Path) -> Configuration: > """Load DTS test run configuration from a file. > > Load the YAML test run configuration file > @@ -581,13 +580,16 @@ def load_config() -> Configuration: > The YAML test run configuration file is specified in the :option:`--config-file` command line > argument or the :envvar:`DTS_CFG_FILE` environment variable. > > + Args: > + config_file_path: The path to the YAML test run configuration file. > + > Returns: > The parsed test run configuration. > """ > - with open(SETTINGS.config_file_path, "r") as f: > + with open(config_file_path, "r") as f: > config_data = yaml.safe_load(f) > > - schema_path = os.path.join(pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json") > + schema_path = os.path.join(Path(__file__).parent.resolve(), "conf_yaml_schema.json") > > with open(schema_path, "r") as f: > schema = json.load(f) > diff --git a/dts/framework/runner.py b/dts/framework/runner.py > index dfee8ebd7c..db8e3ba96b 100644 > --- a/dts/framework/runner.py > +++ b/dts/framework/runner.py > @@ -24,7 +24,7 @@ > import sys > from pathlib import Path > from types import MethodType > -from typing import Iterable > +from typing import Iterable, Sequence > > from .config import ( > BuildTargetConfiguration, > @@ -83,7 +83,7 @@ class DTSRunner: > > def __init__(self): > """Initialize the instance with configuration, logger, result and string constants.""" > - self._configuration = load_config() > + self._configuration = load_config(SETTINGS.config_file_path) > self._logger = get_dts_logger() > if not os.path.exists(SETTINGS.output_dir): > os.makedirs(SETTINGS.output_dir) > @@ -129,7 +129,7 @@ def run(self): > #. Execution teardown > > The test cases are filtered according to the specification in the test run configuration and > - the :option:`--test-cases` command line argument or > + the :option:`--test-suite` command line argument or > the :envvar:`DTS_TESTCASES` environment variable. > """ > sut_nodes: dict[str, SutNode] = {} > @@ -147,7 +147,9 @@ def run(self): > ) > execution_result = self._result.add_execution(execution) > # we don't want to modify the original config, so create a copy > - execution_test_suites = list(execution.test_suites) > + execution_test_suites = list( > + SETTINGS.test_suites if SETTINGS.test_suites else execution.test_suites > + ) > if not execution.skip_smoke_tests: > execution_test_suites[:0] = [TestSuiteConfig.from_dict("smoke_tests")] > try: > @@ -226,7 +228,7 @@ def _get_test_suites_with_cases( > test_suite_class = self._get_test_suite_class(test_suite_config.test_suite) > test_cases = [] > func_test_cases, perf_test_cases = self._filter_test_cases( > - test_suite_class, set(test_suite_config.test_cases + SETTINGS.test_cases) > + test_suite_class, test_suite_config.test_cases > ) > if func: > test_cases.extend(func_test_cases) > @@ -302,7 +304,7 @@ def is_test_suite(object) -> bool: > ) > > def _filter_test_cases( > - self, test_suite_class: type[TestSuite], test_cases_to_run: set[str] > + self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str] > ) -> tuple[list[MethodType], list[MethodType]]: > """Filter `test_cases_to_run` from `test_suite_class`. > > @@ -331,7 +333,9 @@ def _filter_test_cases( > (name, method) for name, method in name_method_tuples if name in test_cases_to_run > ] > if len(name_method_tuples) < len(test_cases_to_run): > - missing_test_cases = test_cases_to_run - {name for name, _ in name_method_tuples} > + missing_test_cases = set(test_cases_to_run) - { > + name for name, _ in name_method_tuples > + } > raise ConfigurationError( > f"Test cases {missing_test_cases} not found among methods " > f"of {test_suite_class.__name__}." > diff --git a/dts/framework/settings.py b/dts/framework/settings.py > index 2b8bfbe0ed..688e8679a7 100644 > --- a/dts/framework/settings.py > +++ b/dts/framework/settings.py > @@ -48,10 +48,11 @@ > > The path to a DPDK tarball, git commit ID, tag ID or tree ID to test. > > -.. option:: --test-cases > -.. envvar:: DTS_TESTCASES > +.. option:: --test-suite > +.. envvar:: DTS_TEST_SUITES > > - A comma-separated list of test cases to execute. Unknown test cases will be silently ignored. > + A test suite with test cases which may be specified multiple times. > + In the environment variable, the suites are joined with a comma. > > .. option:: --re-run, --re_run > .. envvar:: DTS_RERUN > @@ -71,83 +72,13 @@ > > import argparse > import os > -from collections.abc import Callable, Iterable, Sequence > from dataclasses import dataclass, field > from pathlib import Path > -from typing import Any, TypeVar > +from typing import Any > > +from .config import TestSuiteConfig > from .utils import DPDKGitTarball > > -_T = TypeVar("_T") > - > - > -def _env_arg(env_var: str) -> Any: > - """A helper method augmenting the argparse Action with environment variables. > - > - If the supplied environment variable is defined, then the default value > - of the argument is modified. This satisfies the priority order of > - command line argument > environment variable > default value. > - > - Arguments with no values (flags) should be defined using the const keyword argument > - (True or False). When the argument is specified, it will be set to const, if not specified, > - the default will be stored (possibly modified by the corresponding environment variable). > - > - Other arguments work the same as default argparse arguments, that is using > - the default 'store' action. > - > - Returns: > - The modified argparse.Action. > - """ > - > - class _EnvironmentArgument(argparse.Action): > - def __init__( > - self, > - option_strings: Sequence[str], > - dest: str, > - nargs: str | int | None = None, > - const: bool | None = None, > - default: Any = None, > - type: Callable[[str], _T | argparse.FileType | None] = None, > - choices: Iterable[_T] | None = None, > - required: bool = False, > - help: str | None = None, > - metavar: str | tuple[str, ...] | None = None, > - ) -> None: > - env_var_value = os.environ.get(env_var) > - default = env_var_value or default > - if const is not None: > - nargs = 0 > - default = const if env_var_value else default > - type = None > - choices = None > - metavar = None > - super(_EnvironmentArgument, self).__init__( > - option_strings, > - dest, > - nargs=nargs, > - const=const, > - default=default, > - type=type, > - choices=choices, > - required=required, > - help=help, > - metavar=metavar, > - ) > - > - def __call__( > - self, > - parser: argparse.ArgumentParser, > - namespace: argparse.Namespace, > - values: Any, > - option_string: str = None, > - ) -> None: > - if self.const is not None: > - setattr(namespace, self.dest, self.const) > - else: > - setattr(namespace, self.dest, values) > - > - return _EnvironmentArgument > - > > @dataclass(slots=True) > class Settings: > @@ -171,7 +102,7 @@ class Settings: > #: > compile_timeout: float = 1200 > #: > - test_cases: list[str] = field(default_factory=list) > + test_suites: list[TestSuiteConfig] = field(default_factory=list) > #: > re_run: int = 0 > > @@ -180,6 +111,31 @@ class Settings: > > > def _get_parser() -> argparse.ArgumentParser: > + """Create the argument parser for DTS. > + > + Command line options take precedence over environment variables, which in turn take precedence > + over default values. > + > + Returns: > + argparse.ArgumentParser: The configured argument parser with defined options. > + """ > + > + def env_arg(env_var: str, default: Any) -> Any: > + """A helper function augmenting the argparse with environment variables. > + > + If the supplied environment variable is defined, then the default value > + of the argument is modified. This satisfies the priority order of > + command line argument > environment variable > default value. > + > + Args: > + env_var: Environment variable name. > + default: Default value. > + > + Returns: > + Environment variable or default value. > + """ > + return os.environ.get(env_var) or default > + > parser = argparse.ArgumentParser( > description="Run DPDK test suites. All options may be specified with the environment " > "variables provided in brackets. Command line arguments have higher priority.", > @@ -188,25 +144,23 @@ def _get_parser() -> argparse.ArgumentParser: > > parser.add_argument( > "--config-file", > - action=_env_arg("DTS_CFG_FILE"), > - default=SETTINGS.config_file_path, > + default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path), > type=Path, > - help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets.", > + help="[DTS_CFG_FILE] The configuration file that describes the test cases, " > + "SUTs and targets.", > ) > > parser.add_argument( > "--output-dir", > "--output", > - action=_env_arg("DTS_OUTPUT_DIR"), > - default=SETTINGS.output_dir, > + default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir), > help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.", > ) > > parser.add_argument( > "-t", > "--timeout", > - action=_env_arg("DTS_TIMEOUT"), > - default=SETTINGS.timeout, > + default=env_arg("DTS_TIMEOUT", SETTINGS.timeout), > type=float, > help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.", > ) > @@ -214,9 +168,8 @@ def _get_parser() -> argparse.ArgumentParser: > parser.add_argument( > "-v", > "--verbose", > - action=_env_arg("DTS_VERBOSE"), > - default=SETTINGS.verbose, > - const=True, > + action="store_true", > + default=env_arg("DTS_VERBOSE", SETTINGS.verbose), > help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages " > "to the console.", > ) > @@ -224,8 +177,8 @@ def _get_parser() -> argparse.ArgumentParser: > parser.add_argument( > "-s", > "--skip-setup", > - action=_env_arg("DTS_SKIP_SETUP"), > - const=True, > + action="store_true", > + default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup), > help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.", > ) > > @@ -233,8 +186,7 @@ def _get_parser() -> argparse.ArgumentParser: > "--tarball", > "--snapshot", > "--git-ref", > - action=_env_arg("DTS_DPDK_TARBALL"), > - default=SETTINGS.dpdk_tarball_path, > + default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path), > type=Path, > help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, " > "tag ID or tree ID to test. To test local changes, first commit them, " > @@ -243,36 +195,71 @@ def _get_parser() -> argparse.ArgumentParser: > > parser.add_argument( > "--compile-timeout", > - action=_env_arg("DTS_COMPILE_TIMEOUT"), > - default=SETTINGS.compile_timeout, > + default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout), > type=float, > help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.", > ) > > parser.add_argument( > - "--test-cases", > - action=_env_arg("DTS_TESTCASES"), > - default="", > - help="[DTS_TESTCASES] Comma-separated list of test cases to execute.", > + "--test-suite", > + action="append", > + nargs="+", > + metavar=("TEST_SUITE", "TEST_CASES"), > + default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites), > + help="[DTS_TEST_SUITES] A list containing a test suite with test cases. " > + "The first parameter is the test suite name, and the rest are test case names, " > + "which are optional. May be specified multiple times. To specify multiple test suites in " > + "the environment variable, join the lists with a comma. " > + "Examples: " > + "--test-suite suite case case --test-suite suite case ... | " > + "DTS_TEST_SUITES='suite case case, suite case, ...' | " > + "--test-suite suite --test-suite suite case ... | " > + "DTS_TEST_SUITES='suite, suite case, ...'", > ) > > parser.add_argument( > "--re-run", > "--re_run", > - action=_env_arg("DTS_RERUN"), > - default=SETTINGS.re_run, > + default=env_arg("DTS_RERUN", SETTINGS.re_run), > type=int, > help="[DTS_RERUN] Re-run each test case the specified number of times " > - "if a test failure occurs", > + "if a test failure occurs.", > ) > > return parser > > > +def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]: > + """Process the given argument to a list of :class:`TestSuiteConfig` to execute. > + > + Args: > + args: The arguments to process. The args is a string from an environment variable > + or a list of from the user input containing tests suites with tests cases, > + each of which is a list of [test_suite, test_case, test_case, ...]. > + > + Returns: > + A list of test suite configurations to execute. > + """ > + if isinstance(args, str): > + # Environment variable in the form of "suite case case, suite case, suite, ..." > + args = [suite_with_cases.split() for suite_with_cases in args.split(",")] > + > + test_suites_to_run = [] > + for suite_with_cases in args: > + test_suites_to_run.append( > + TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:]) > + ) > + > + return test_suites_to_run > + > + > def get_settings() -> Settings: > """Create new settings with inputs from the user. > > The inputs are taken from the command line and from environment variables. > + > + Returns: > + The new settings object. > """ > parsed_args = _get_parser().parse_args() > return Settings( > @@ -287,6 +274,6 @@ def get_settings() -> Settings: > else Path(parsed_args.tarball) > ), > compile_timeout=parsed_args.compile_timeout, > - test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []), > + test_suites=_process_test_suites(parsed_args.test_suite), > re_run=parsed_args.re_run, > ) > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py > index 365f80e21a..1957ea7328 100644 > --- a/dts/framework/test_suite.py > +++ b/dts/framework/test_suite.py > @@ -40,7 +40,7 @@ class TestSuite(object): > and functional test cases (all other test cases). > > By default, all test cases will be executed. A list of testcase names may be specified > - in the YAML test run configuration file and in the :option:`--test-cases` command line argument > + in the YAML test run configuration file and in the :option:`--test-suite` command line argument > or in the :envvar:`DTS_TESTCASES` environment variable to filter which test cases to run. > The union of both lists will be used. Any unknown test cases from the latter lists > will be silently ignored. > -- > 2.34.1 > ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 7/7] dts: improve test suite and case filtering 2024-03-01 17:41 ` Jeremy Spewock @ 2024-03-01 17:43 ` Jeremy Spewock 2024-03-07 11:04 ` Thomas Monjalon 0 siblings, 1 reply; 44+ messages in thread From: Jeremy Spewock @ 2024-03-01 17:43 UTC (permalink / raw) To: Juraj Linkeš Cc: thomas, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, npratte, dev On Fri, Mar 1, 2024 at 12:41 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote: > > For the whole series: > > Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu> > Apologies, I forgot to remove the content of the patch when I added my tag. ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 7/7] dts: improve test suite and case filtering 2024-03-01 17:43 ` Jeremy Spewock @ 2024-03-07 11:04 ` Thomas Monjalon 0 siblings, 0 replies; 44+ messages in thread From: Thomas Monjalon @ 2024-03-07 11:04 UTC (permalink / raw) To: Jeremy Spewock Cc: Juraj Linkeš, dev, Honnappa.Nagarahalli, probb, paul.szczepanek, Luca.Vizzarro, npratte 01/03/2024 18:43, Jeremy Spewock: > On Fri, Mar 1, 2024 at 12:41 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote: > > > > For the whole series: > > > > Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu> > > > Apologies, I forgot to remove the content of the patch when I added my tag. The best is to keep the commit message, and put your tag below others so we collect review/ack/test tags. ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 0/7] test case blocking and logging 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš ` (6 preceding siblings ...) 2024-03-01 10:55 ` [PATCH v4 7/7] dts: improve test suite and case filtering Juraj Linkeš @ 2024-03-01 16:11 ` Patrick Robb 2024-03-07 14:55 ` Thomas Monjalon 8 siblings, 0 replies; 44+ messages in thread From: Patrick Robb @ 2024-03-01 16:11 UTC (permalink / raw) To: Juraj Linkeš Cc: thomas, Honnappa.Nagarahalli, jspewock, paul.szczepanek, Luca.Vizzarro, npratte, dev [-- Attachment #1: Type: text/plain, Size: 4111 bytes --] The Community CI Testing Lab had an infra failure this morning and some patches including yours were affected with false failures. The issue is now resolved and we are rerunning the tests in question for all patches submitted today. On Fri, Mar 1, 2024 at 5:55 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote: > We currently don't record test case results that couldn't be executed > because of a previous failure, such as when a test suite setup failed, > resulting in no executed test cases. > > In order to record the test cases that couldn't be executed, we must > know the lists of test suites and test cases ahead of the actual test > suite execution, as an error could occur before we even start executing > test suites. > > In addition, the patch series contains two refactors and one > improvement. > > The first refactor is closely related. The dts.py was renamed to > runner.py and given a clear purpose - running the test suites and all > other orchestration needed to run test suites. The logic for this was > not all in the original dts.py module and it was brought there. The > runner is also responsible for recording results, which is the blocked > test cases are recorded. > > The other refactor, logging, is related to the first refactor. The > logging module was simplified while extending capabilities. Each test > suite logs into its own log file in addition to the main log file which > the runner must handle (as it knows when we start executing particular > test suites). The runner also handles the switching between execution > stages for the purposes of logging. > > The one aforementioned improvement is in unifying how we specify test > cases in the conf.yaml file and in the environment variable/command line > argument. > > v2: > Rebase and update of the whole patch. There are changes in all parts of > the code, mainly improving the design and logic. > Also added the last patch which improves test suite specification on the > cmdline. > > v3: > Improved variables in _get_test_suite_class along with docstring. > Fixed smoke test suite not being added into the list of test suites to > be executed. > > v4: > Added the checking of CamelCase convention when discovering test cases. > Also added additional test stages. The stages were split into setup and > teardown. > > Juraj Linkeš (7): > dts: convert dts.py methods to class > dts: move test suite execution logic to DTSRunner > dts: filter test suites in executions > dts: reorganize test result > dts: block all test cases when earlier setup fails > dts: refactor logging configuration > dts: improve test suite and case filtering > > doc/guides/tools/dts.rst | 14 +- > dts/framework/config/__init__.py | 36 +- > dts/framework/config/conf_yaml_schema.json | 2 +- > dts/framework/dts.py | 338 --------- > dts/framework/logger.py | 246 +++--- > dts/framework/remote_session/__init__.py | 6 +- > .../interactive_remote_session.py | 6 +- > .../remote_session/interactive_shell.py | 6 +- > .../remote_session/remote_session.py | 8 +- > dts/framework/runner.py | 711 ++++++++++++++++++ > dts/framework/settings.py | 188 +++-- > dts/framework/test_result.py | 565 ++++++++------ > dts/framework/test_suite.py | 239 +----- > dts/framework/testbed_model/node.py | 11 +- > dts/framework/testbed_model/os_session.py | 7 +- > .../traffic_generator/traffic_generator.py | 6 +- > dts/main.py | 9 +- > dts/pyproject.toml | 3 + > dts/tests/TestSuite_os_udp.py | 2 +- > dts/tests/TestSuite_smoke_tests.py | 2 +- > 20 files changed, 1375 insertions(+), 1030 deletions(-) > delete mode 100644 dts/framework/dts.py > create mode 100644 dts/framework/runner.py > > -- > 2.34.1 > > [-- Attachment #2: Type: text/html, Size: 4804 bytes --] ^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v4 0/7] test case blocking and logging 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš ` (7 preceding siblings ...) 2024-03-01 16:11 ` [PATCH v4 0/7] test case blocking and logging Patrick Robb @ 2024-03-07 14:55 ` Thomas Monjalon 8 siblings, 0 replies; 44+ messages in thread From: Thomas Monjalon @ 2024-03-07 14:55 UTC (permalink / raw) To: Juraj Linkeš Cc: Honnappa.Nagarahalli, jspewock, probb, paul.szczepanek, Luca.Vizzarro, npratte, dev > Juraj Linkeš (7): > dts: convert dts.py methods to class > dts: move test suite execution logic to DTSRunner > dts: filter test suites in executions > dts: reorganize test result > dts: block all test cases when earlier setup fails > dts: refactor logging configuration > dts: improve test suite and case filtering Applied, thanks. ^ permalink raw reply [flat|nested] 44+ messages in thread
end of thread, other threads:[~2024-03-07 14:55 UTC | newest] Thread overview: 44+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-12-20 10:33 [RFC PATCH v1 0/5] test case blocking and logging Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 1/5] dts: convert dts.py methods to class Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 2/5] dts: move test suite execution logic to DTSRunner Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 3/5] dts: process test suites at the beginning of run Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 4/5] dts: block all testcases when earlier setup fails Juraj Linkeš 2023-12-20 10:33 ` [RFC PATCH v1 5/5] dts: refactor logging configuration Juraj Linkeš 2024-01-08 18:47 ` [RFC PATCH v1 0/5] test case blocking and logging Jeremy Spewock 2024-02-06 14:57 ` [PATCH v2 0/7] " Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 1/7] dts: convert dts.py methods to class Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 3/7] dts: filter test suites in executions Juraj Linkeš 2024-02-12 16:44 ` Jeremy Spewock 2024-02-14 9:55 ` Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 4/7] dts: reorganize test result Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš 2024-02-06 14:57 ` [PATCH v2 6/7] dts: refactor logging configuration Juraj Linkeš 2024-02-12 16:45 ` Jeremy Spewock 2024-02-14 7:49 ` Juraj Linkeš 2024-02-14 16:51 ` Jeremy Spewock 2024-02-06 14:57 ` [PATCH v2 7/7] dts: improve test suite and case filtering Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 0/7] test case blocking and logging Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 1/7] dts: convert dts.py methods to class Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 3/7] dts: filter test suites in executions Juraj Linkeš 2024-02-27 21:21 ` Jeremy Spewock 2024-02-28 9:16 ` Juraj Linkeš 2024-02-23 7:54 ` [PATCH v3 4/7] dts: reorganize test result Juraj Linkeš 2024-02-23 7:55 ` [PATCH v3 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš 2024-02-23 7:55 ` [PATCH v3 6/7] dts: refactor logging configuration Juraj Linkeš 2024-02-23 7:55 ` [PATCH v3 7/7] dts: improve test suite and case filtering Juraj Linkeš 2024-02-27 21:24 ` [PATCH v3 0/7] test case blocking and logging Jeremy Spewock 2024-03-01 10:55 ` [PATCH v4 " Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 1/7] dts: convert dts.py methods to class Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 2/7] dts: move test suite execution logic to DTSRunner Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 3/7] dts: filter test suites in executions Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 4/7] dts: reorganize test result Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 5/7] dts: block all test cases when earlier setup fails Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 6/7] dts: refactor logging configuration Juraj Linkeš 2024-03-01 10:55 ` [PATCH v4 7/7] dts: improve test suite and case filtering Juraj Linkeš 2024-03-01 17:41 ` Jeremy Spewock 2024-03-01 17:43 ` Jeremy Spewock 2024-03-07 11:04 ` Thomas Monjalon 2024-03-01 16:11 ` [PATCH v4 0/7] test case blocking and logging Patrick Robb 2024-03-07 14:55 ` Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).