From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8F65543B01; Mon, 12 Feb 2024 17:45:00 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 19C45402C8; Mon, 12 Feb 2024 17:45:00 +0100 (CET) Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) by mails.dpdk.org (Postfix) with ESMTP id 8CD8540279 for ; Mon, 12 Feb 2024 17:44:58 +0100 (CET) Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-5cddc5455aeso2374789a12.1 for ; Mon, 12 Feb 2024 08:44:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1707756297; x=1708361097; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=laZIcWDGN2n3QdnTyFmtIHQN/9ZeXXP2W8dh/CySW/8=; b=UfPluNQHyrQaWt2U9VDoJmqXrQv4BJ9Mwti8KMCw3Z3i/VTsUXBR6P38cMb8w8xPtu HtbgbzikHgysk6TJm8dZKscKsPCEEeCFp9QjtYkORigQvOpfzB3lwcAPeKdiLfQTJNXc YHyP+6cvcFrcnY8xI6Fr5nb9DZ7xVAPuvRf/o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707756297; x=1708361097; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=laZIcWDGN2n3QdnTyFmtIHQN/9ZeXXP2W8dh/CySW/8=; b=eA5iqkdY4WcvCCmk2Au9aXM14U9sCSFdkrauy3C4k8BorUJKBZdR6GapxqW0H8PUKD ftP89eRwvJuPIzZtUOOBbKEfQNWcqKQuxuzIV9FyHBc7DJmYMSNzenwge/p/ZrscYkU7 ZpGU8yqMbVNTHx9jzLwht/x+5oC47m2mRFqBsPOWf03jAQgHTuHUibgk/YMNkWPzcjAg VOC99hHQ7/mTi+6gpEKnmlGecsZ/aIoJAiAjLa9hD05emDCzap7lNND5IEeba14uDwHY 0T+D9iDdngq4n0QWC5fJmd9cpyY45MpIEqd2NqfyAi2aMsI4DhD3zugdCTNXq7rPCe4c DbFQ== X-Forwarded-Encrypted: i=1; AJvYcCV4bxppgN4qLer2G+GovaqHGXROJm+0cJwJl/lIX97TS+P0LOAELmmkJJ6KUSEKBuYOpxAIpAg3JrEOaeI= X-Gm-Message-State: AOJu0YwMMJ+8ck9I9DTHv/TaKyoea4wf83TRX652IztWVpUJoCGUAjfG xK8EJPqGIHQkaqgpfcMmXPU4YSjAFp0ltzY+9KA76+0gHjU/o32W8JYsx1eDpcytHDyUd3pM1H7 UGQ5RSV+FPvo6o4V/XUu8SFQoSkLfdTza7MCyRA== X-Google-Smtp-Source: AGHT+IFy5xU3D94NTeHf4HcREZrk1NbuOXjA8LhbghdAhG9RT8Mj35pLdeTEhiK9cJYJn+6FuNihjtdDLozvgUTUgDY= X-Received: by 2002:a17:90a:bc4b:b0:28c:fb86:23ce with SMTP id t11-20020a17090abc4b00b0028cfb8623cemr3940588pjv.44.1707756297083; Mon, 12 Feb 2024 08:44:57 -0800 (PST) MIME-Version: 1.0 References: <20231220103331.60888-1-juraj.linkes@pantheon.tech> <20240206145716.71435-1-juraj.linkes@pantheon.tech> <20240206145716.71435-4-juraj.linkes@pantheon.tech> In-Reply-To: <20240206145716.71435-4-juraj.linkes@pantheon.tech> From: Jeremy Spewock Date: Mon, 12 Feb 2024 11:44:45 -0500 Message-ID: Subject: Re: [PATCH v2 3/7] dts: filter test suites in executions To: =?UTF-8?Q?Juraj_Linke=C5=A1?= Cc: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, probb@iol.unh.edu, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com, dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Feb 6, 2024 at 9:57=E2=80=AFAM Juraj Linke=C5=A1 wrote: > > We're currently filtering which test cases to run after some setup > steps, such as DPDK build, have already been taken. This prohibits us to > mark the test suites and cases that were supposed to be run as blocked > when an earlier setup fails, as that information is not available at > that time. > > To remedy this, move the filtering to the beginning of each execution. > This is the first action taken in each execution and if we can't filter > the test cases, such as due to invalid inputs, we abort the whole > execution. No test suites nor cases will be marked as blocked as we > don't know which were supposed to be run. > > On top of that, the filtering takes place in the TestSuite class, which > should only concern itself with test suite and test case logic, not the > processing behind the scenes. The logic has been moved to DTSRunner > which should do all the processing needed to run test suites. > > The filtering itself introduces a few changes/assumptions which are more > sensible than before: > 1. Assumption: There is just one TestSuite child class in each test > suite module. This was an implicit assumption before as we couldn't > specify the TestSuite classes in the test run configuration, just the > modules. The name of the TestSuite child class starts with "Test" and > then corresponds to the name of the module with CamelCase naming. > 2. Unknown test cases specified both in the test run configuration and > the environment variable/command line argument are no longer silently > ignored. This is a quality of life improvement for users, as they > could easily be not aware of the silent ignoration. > > Also, a change in the code results in pycodestyle warning and error: > [E] E203 whitespace before ':' > [W] W503 line break before binary operator > > These two are not PEP8 compliant, so they're disabled. > > Signed-off-by: Juraj Linke=C5=A1 > --- > dts/framework/config/__init__.py | 24 +- > dts/framework/config/conf_yaml_schema.json | 2 +- > dts/framework/runner.py | 426 +++++++++++++++------ > dts/framework/settings.py | 3 +- > dts/framework/test_result.py | 34 ++ > dts/framework/test_suite.py | 85 +--- > dts/pyproject.toml | 3 + > dts/tests/TestSuite_smoke_tests.py | 2 +- > 8 files changed, 382 insertions(+), 197 deletions(-) > > diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__in= it__.py > index 62eded7f04..c6a93b3b89 100644 > --- a/dts/framework/config/__init__.py > +++ b/dts/framework/config/__init__.py > @@ -36,7 +36,7 @@ > import json > import os.path > import pathlib > -from dataclasses import dataclass > +from dataclasses import dataclass, fields > from enum import auto, unique > from typing import Union > > @@ -506,6 +506,28 @@ def from_dict( > vdevs=3Dvdevs, > ) > > + def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration": > + """Create a shallow copy with any of the fields modified. > + > + The only new data are those passed to this method. > + The rest are copied from the object's fields calling the method. > + > + Args: > + **kwargs: The names and types of keyword arguments are defin= ed > + by the fields of the :class:`ExecutionConfiguration` cla= ss. > + > + Returns: > + The copied and modified execution configuration. > + """ > + new_config =3D {} > + for field in fields(self): > + if field.name in kwargs: > + new_config[field.name] =3D kwargs[field.name] > + else: > + new_config[field.name] =3D getattr(self, field.name) > + > + return ExecutionConfiguration(**new_config) > + > > @dataclass(slots=3DTrue, frozen=3DTrue) > class Configuration: > diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/c= onfig/conf_yaml_schema.json > index 84e45fe3c2..051b079fe4 100644 > --- a/dts/framework/config/conf_yaml_schema.json > +++ b/dts/framework/config/conf_yaml_schema.json > @@ -197,7 +197,7 @@ > }, > "cases": { > "type": "array", > - "description": "If specified, only this subset of test suite's= test cases will be run. Unknown test cases will be silently ignored.", > + "description": "If specified, only this subset of test suite's= test cases will be run.", > "items": { > "type": "string" > }, > diff --git a/dts/framework/runner.py b/dts/framework/runner.py > index 933685d638..3e95cf9e26 100644 > --- a/dts/framework/runner.py > +++ b/dts/framework/runner.py > @@ -17,17 +17,27 @@ > and the test case stage runs test cases individually. > """ > > +import importlib > +import inspect > import logging > +import re > import sys > from types import MethodType > +from typing import Iterable > > from .config import ( > BuildTargetConfiguration, > + Configuration, > ExecutionConfiguration, > TestSuiteConfig, > load_config, > ) > -from .exception import BlockingTestSuiteError, SSHTimeoutError, TestCase= VerifyError > +from .exception import ( > + BlockingTestSuiteError, > + ConfigurationError, > + SSHTimeoutError, > + TestCaseVerifyError, > +) > from .logger import DTSLOG, getLogger > from .settings import SETTINGS > from .test_result import ( > @@ -37,8 +47,9 @@ > Result, > TestCaseResult, > TestSuiteResult, > + TestSuiteWithCases, > ) > -from .test_suite import TestSuite, get_test_suites > +from .test_suite import TestSuite > from .testbed_model import SutNode, TGNode > > > @@ -59,13 +70,23 @@ class DTSRunner: > given execution, the next execution begins. > """ > > + _configuration: Configuration > _logger: DTSLOG > _result: DTSResult > + _test_suite_class_prefix: str > + _test_suite_module_prefix: str > + _func_test_case_regex: str > + _perf_test_case_regex: str > > def __init__(self): > - """Initialize the instance with logger and result.""" > + """Initialize the instance with configuration, logger, result an= d string constants.""" > + self._configuration =3D load_config() > self._logger =3D getLogger("DTSRunner") > self._result =3D DTSResult(self._logger) > + self._test_suite_class_prefix =3D "Test" > + self._test_suite_module_prefix =3D "tests.TestSuite_" > + self._func_test_case_regex =3D r"test_(?!perf_)" > + self._perf_test_case_regex =3D r"test_perf_" > > def run(self): > """Run all build targets in all executions from the test run con= figuration. > @@ -106,29 +127,28 @@ def run(self): > try: > # check the python version of the server that runs dts > self._check_dts_python_version() > + self._result.update_setup(Result.PASS) > > # for all Execution sections > - for execution in load_config().executions: > - sut_node =3D sut_nodes.get(execution.system_under_test_n= ode.name) > - tg_node =3D tg_nodes.get(execution.traffic_generator_nod= e.name) > - > + for execution in self._configuration.executions: > + self._logger.info( > + f"Running execution with SUT '{execution.system_unde= r_test_node.name}'." > + ) > + execution_result =3D self._result.add_execution(executio= n.system_under_test_node) > try: > - if not sut_node: > - sut_node =3D SutNode(execution.system_under_test= _node) > - sut_nodes[sut_node.name] =3D sut_node > - if not tg_node: > - tg_node =3D TGNode(execution.traffic_generator_n= ode) > - tg_nodes[tg_node.name] =3D tg_node > - self._result.update_setup(Result.PASS) > + test_suites_with_cases =3D self._get_test_suites_wit= h_cases( > + execution.test_suites, execution.func, execution= .perf > + ) > except Exception as e: > - failed_node =3D execution.system_under_test_node.nam= e > - if sut_node: > - failed_node =3D execution.traffic_generator_node= .name > - self._logger.exception(f"The Creation of node {faile= d_node} failed.") > - self._result.update_setup(Result.FAIL, e) > + self._logger.exception( > + f"Invalid test suite configuration found: " f"{e= xecution.test_suites}." > + ) > + execution_result.update_setup(Result.FAIL, e) > > else: > - self._run_execution(sut_node, tg_node, execution) > + self._connect_nodes_and_run_execution( > + sut_nodes, tg_nodes, execution, execution_result= , test_suites_with_cases > + ) > > except Exception as e: > self._logger.exception("An unexpected error has occurred.") > @@ -163,11 +183,204 @@ def _check_dts_python_version(self) -> None: > ) > self._logger.warning("Please use Python >=3D 3.10 instead.") > > + def _get_test_suites_with_cases( > + self, > + test_suite_configs: list[TestSuiteConfig], > + func: bool, > + perf: bool, > + ) -> list[TestSuiteWithCases]: > + """Test suites with test cases discovery. > + > + The test suites with test cases defined in the user configuratio= n are discovered > + and stored for future use so that we don't import the modules tw= ice and so that > + the list of test suites with test cases is available for recordi= ng right away. > + > + Args: > + test_suite_configs: Test suite configurations. > + func: Whether to include functional test cases in the final = list. > + perf: Whether to include performance test cases in the final= list. > + > + Returns: > + The discovered test suites, each with test cases. > + """ > + test_suites_with_cases =3D [] > + > + for test_suite_config in test_suite_configs: > + test_suite_class =3D self._get_test_suite_class(test_suite_c= onfig.test_suite) > + test_cases =3D [] > + func_test_cases, perf_test_cases =3D self._filter_test_cases= ( > + test_suite_class, set(test_suite_config.test_cases + SET= TINGS.test_cases) > + ) > + if func: > + test_cases.extend(func_test_cases) > + if perf: > + test_cases.extend(perf_test_cases) > + > + test_suites_with_cases.append( > + TestSuiteWithCases(test_suite_class=3Dtest_suite_class, = test_cases=3Dtest_cases) > + ) > + > + return test_suites_with_cases > + > + def _get_test_suite_class(self, test_suite_name: str) -> type[TestSu= ite]: > + """Find the :class:`TestSuite` class with `test_suite_name` in t= he corresponding module. > + > + The method assumes that the :class:`TestSuite` class starts > + with `self._test_suite_class_prefix`, > + continuing with `test_suite_name` with CamelCase convention. > + It also assumes there's only one test suite in each module and t= he module name > + is `test_suite_name` prefixed with `self._test_suite_module_pref= ix`. > + > + The CamelCase convention is not tested, only lowercase strings a= re compared. > + > + Args: > + test_suite_name: The name of the test suite to find. > + > + Returns: > + The found test suite. > + > + Raises: > + ConfigurationError: If the corresponding module is not found= or > + a valid :class:`TestSuite` is not found in the module. > + """ > + > + def is_test_suite(object) -> bool: > + """Check whether `object` is a :class:`TestSuite`. > + > + The `object` is a subclass of :class:`TestSuite`, but not :c= lass:`TestSuite` itself. > + > + Args: > + object: The object to be checked. > + > + Returns: > + :data:`True` if `object` is a subclass of `TestSuite`. > + """ > + try: > + if issubclass(object, TestSuite) and object is not TestS= uite: > + return True > + except TypeError: > + return False > + return False > + > + testsuite_module_path =3D f"{self._test_suite_module_prefix}{tes= t_suite_name}" > + try: > + test_suite_module =3D importlib.import_module(testsuite_modu= le_path) > + except ModuleNotFoundError as e: > + raise ConfigurationError( > + f"Test suite module '{testsuite_module_path}' not found.= " > + ) from e > + > + lowercase_suite_name =3D test_suite_name.replace("_", "").lower(= ) > + for class_name, class_obj in inspect.getmembers(test_suite_modul= e, is_test_suite): > + if ( > + class_name.startswith(self._test_suite_class_prefix) > + and lowercase_suite_name =3D=3D class_name[len(self._tes= t_suite_class_prefix) :].lower() > + ): Would it be simpler to instead just make lowercase_suite_name =3D f"{self._test_suite_class_prefix}{test_suite_name.replace("_", "").lower()}" so that you can just directly compare class_name =3D=3D lowercase_suite_name? Both ways should have the exact same result of course so it isn't important, I was just curious. > + return class_obj > + raise ConfigurationError( > + f"Couldn't find any valid test suites in {test_suite_module.= __name__}." > + ) > + > + def _filter_test_cases( > + self, test_suite_class: type[TestSuite], test_cases_to_run: set[= str] > + ) -> tuple[list[MethodType], list[MethodType]]: > + """Filter `test_cases_to_run` from `test_suite_class`. > + > + There are two rounds of filtering if `test_cases_to_run` is not = empty. > + The first filters `test_cases_to_run` from all methods of `test_= suite_class`. > + Then the methods are separated into functional and performance t= est cases. > + If a method doesn't match neither the functional nor performance= name prefix, it's an error. I think this is a double negative but could be either "if a method doesn't match either ... or ..." or "if a method matches neither ... nor ...". I have a small preference to the second of the two options though because the "neither" makes the negative more clear in my mind. > + > + Args: > + test_suite_class: The class of the test suite. > + test_cases_to_run: Test case names to filter from `test_suit= e_class`. > + If empty, return all matching test cases. > + > + Returns: > + A list of test case methods that should be executed. > + > + Raises: > + ConfigurationError: If a test case from `test_cases_to_run` = is not found > + or it doesn't match either the functional nor performanc= e name prefix. > + """ > + func_test_cases =3D [] > + perf_test_cases =3D [] > + name_method_tuples =3D inspect.getmembers(test_suite_class, insp= ect.isfunction) > + if test_cases_to_run: > + name_method_tuples =3D [ > + (name, method) for name, method in name_method_tuples if= name in test_cases_to_run > + ] > + if len(name_method_tuples) < len(test_cases_to_run): > + missing_test_cases =3D test_cases_to_run - {name for nam= e, _ in name_method_tuples} > + raise ConfigurationError( > + f"Test cases {missing_test_cases} not found among me= thods " > + f"of {test_suite_class.__name__}." > + ) > + > + for test_case_name, test_case_method in name_method_tuples: > + if re.match(self._func_test_case_regex, test_case_name): > + func_test_cases.append(test_case_method) > + elif re.match(self._perf_test_case_regex, test_case_name): > + perf_test_cases.append(test_case_method) > + elif test_cases_to_run: > + raise ConfigurationError( > + f"Method '{test_case_name}' doesn't match neither " > + f"a functional nor a performance test case name." Same thing here with the double negative. > + ) > + > + return func_test_cases, perf_test_cases > + > + def _connect_nodes_and_run_execution( > + self, > + sut_nodes: dict[str, SutNode], > + tg_nodes: dict[str, TGNode], > + execution: ExecutionConfiguration, > + execution_result: ExecutionResult, > + test_suites_with_cases: Iterable[TestSuiteWithCases], > + ) -> None: > + """Connect nodes, then continue to run the given execution. > + > + Connect the :class:`SutNode` and the :class:`TGNode` of this `ex= ecution`. > + If either has already been connected, it's going to be in either= `sut_nodes` or `tg_nodes`, > + respectively. > + If not, connect and add the node to the respective `sut_nodes` o= r `tg_nodes` :class:`dict`. > + > + Args: > + sut_nodes: A dictionary storing connected/to be connected SU= T nodes. > + tg_nodes: A dictionary storing connected/to be connected TG = nodes. > + execution: An execution's test run configuration. > + execution_result: The execution's result. > + test_suites_with_cases: The test suites with test cases to r= un. > + """ > + sut_node =3D sut_nodes.get(execution.system_under_test_node.name= ) > + tg_node =3D tg_nodes.get(execution.traffic_generator_node.name) > + > + try: > + if not sut_node: > + sut_node =3D SutNode(execution.system_under_test_node) > + sut_nodes[sut_node.name] =3D sut_node > + if not tg_node: > + tg_node =3D TGNode(execution.traffic_generator_node) > + tg_nodes[tg_node.name] =3D tg_node > + except Exception as e: > + failed_node =3D execution.system_under_test_node.name > + if sut_node: > + failed_node =3D execution.traffic_generator_node.name > + self._logger.exception(f"The Creation of node {failed_node} = failed.") > + execution_result.update_setup(Result.FAIL, e) > + > + else: > + self._run_execution( > + sut_node, tg_node, execution, execution_result, test_sui= tes_with_cases > + ) > + > def _run_execution( > self, > sut_node: SutNode, > tg_node: TGNode, > execution: ExecutionConfiguration, > + execution_result: ExecutionResult, > + test_suites_with_cases: Iterable[TestSuiteWithCases], > ) -> None: > """Run the given execution. > > @@ -178,11 +391,11 @@ def _run_execution( > sut_node: The execution's SUT node. > tg_node: The execution's TG node. > execution: An execution's test run configuration. > + execution_result: The execution's result. > + test_suites_with_cases: The test suites with test cases to r= un. > """ > self._logger.info(f"Running execution with SUT '{execution.syste= m_under_test_node.name}'.") > - execution_result =3D self._result.add_execution(sut_node.config) > execution_result.add_sut_info(sut_node.node_info) > - > try: > sut_node.set_up_execution(execution) > execution_result.update_setup(Result.PASS) > @@ -192,7 +405,10 @@ def _run_execution( > > else: > for build_target in execution.build_targets: > - self._run_build_target(sut_node, tg_node, build_target, = execution, execution_result) > + build_target_result =3D execution_result.add_build_targe= t(build_target) > + self._run_build_target( > + sut_node, tg_node, build_target, build_target_result= , test_suites_with_cases > + ) > > finally: > try: > @@ -207,8 +423,8 @@ def _run_build_target( > sut_node: SutNode, > tg_node: TGNode, > build_target: BuildTargetConfiguration, > - execution: ExecutionConfiguration, > - execution_result: ExecutionResult, > + build_target_result: BuildTargetResult, > + test_suites_with_cases: Iterable[TestSuiteWithCases], > ) -> None: > """Run the given build target. > > @@ -220,11 +436,11 @@ def _run_build_target( > sut_node: The execution's sut node. > tg_node: The execution's tg node. > build_target: A build target's test run configuration. > - execution: The build target's execution's test run configura= tion. > - execution_result: The execution level result object associat= ed with the execution. > + build_target_result: The build target level result object as= sociated > + with the current build target. > + test_suites_with_cases: The test suites with test cases to r= un. > """ > self._logger.info(f"Running build target '{build_target.name}'."= ) > - build_target_result =3D execution_result.add_build_target(build_= target) > > try: > sut_node.set_up_build_target(build_target) > @@ -236,7 +452,7 @@ def _run_build_target( > build_target_result.update_setup(Result.FAIL, e) > > else: > - self._run_test_suites(sut_node, tg_node, execution, build_ta= rget_result) > + self._run_test_suites(sut_node, tg_node, build_target_result= , test_suites_with_cases) > > finally: > try: > @@ -250,10 +466,10 @@ def _run_test_suites( > self, > sut_node: SutNode, > tg_node: TGNode, > - execution: ExecutionConfiguration, > build_target_result: BuildTargetResult, > + test_suites_with_cases: Iterable[TestSuiteWithCases], > ) -> None: > - """Run the execution's (possibly a subset of) test suites using = the current build target. > + """Run `test_suites_with_cases` with the current build target. > > The method assumes the build target we're testing has already be= en built on the SUT node. > The current build target thus corresponds to the current DPDK bu= ild present on the SUT node. > @@ -264,22 +480,20 @@ def _run_test_suites( > Args: > sut_node: The execution's SUT node. > tg_node: The execution's TG node. > - execution: The execution's test run configuration associated > - with the current build target. > build_target_result: The build target level result object as= sociated > with the current build target. > + test_suites_with_cases: The test suites with test cases to r= un. > """ > end_build_target =3D False > - if not execution.skip_smoke_tests: > - execution.test_suites[:0] =3D [TestSuiteConfig.from_dict("sm= oke_tests")] > - for test_suite_config in execution.test_suites: > + for test_suite_with_cases in test_suites_with_cases: > + test_suite_result =3D build_target_result.add_test_suite( > + test_suite_with_cases.test_suite_class.__name__ > + ) > try: > - self._run_test_suite_module( > - sut_node, tg_node, execution, build_target_result, t= est_suite_config > - ) > + self._run_test_suite(sut_node, tg_node, test_suite_resul= t, test_suite_with_cases) > except BlockingTestSuiteError as e: > self._logger.exception( > - f"An error occurred within {test_suite_config.test_s= uite}. " > + f"An error occurred within {test_suite_with_cases.te= st_suite_class.__name__}. " > "Skipping build target..." > ) > self._result.add_error(e) > @@ -288,15 +502,14 @@ def _run_test_suites( > if end_build_target: > break > > - def _run_test_suite_module( > + def _run_test_suite( > self, > sut_node: SutNode, > tg_node: TGNode, > - execution: ExecutionConfiguration, > - build_target_result: BuildTargetResult, > - test_suite_config: TestSuiteConfig, > + test_suite_result: TestSuiteResult, > + test_suite_with_cases: TestSuiteWithCases, > ) -> None: > - """Set up, execute and tear down all test suites in a single tes= t suite module. > + """Set up, execute and tear down `test_suite_with_cases`. > > The method assumes the build target we're testing has already be= en built on the SUT node. > The current build target thus corresponds to the current DPDK bu= ild present on the SUT node. > @@ -306,92 +519,79 @@ def _run_test_suite_module( > > Record the setup and the teardown and handle failures. > > - The test cases to execute are discovered when creating the :clas= s:`TestSuite` object. > - > Args: > sut_node: The execution's SUT node. > tg_node: The execution's TG node. > - execution: The execution's test run configuration associated > - with the current build target. > - build_target_result: The build target level result object as= sociated > - with the current build target. > - test_suite_config: Test suite test run configuration specify= ing the test suite module > - and possibly a subset of test cases of test suites in th= at module. > + test_suite_result: The test suite level result object associ= ated > + with the current test suite. > + test_suite_with_cases: The test suite with test cases to run= . > > Raises: > BlockingTestSuiteError: If a blocking test suite fails. > """ > + test_suite_name =3D test_suite_with_cases.test_suite_class.__nam= e__ > + test_suite =3D test_suite_with_cases.test_suite_class(sut_node, = tg_node) > try: > - full_suite_path =3D f"tests.TestSuite_{test_suite_config.tes= t_suite}" > - test_suite_classes =3D get_test_suites(full_suite_path) > - suites_str =3D ", ".join((x.__name__ for x in test_suite_cla= sses)) > - self._logger.debug(f"Found test suites '{suites_str}' in '{f= ull_suite_path}'.") > + self._logger.info(f"Starting test suite setup: {test_suite_n= ame}") > + test_suite.set_up_suite() > + test_suite_result.update_setup(Result.PASS) > + self._logger.info(f"Test suite setup successful: {test_suite= _name}") > except Exception as e: > - self._logger.exception("An error occurred when searching for= test suites.") > - self._result.update_setup(Result.ERROR, e) > + self._logger.exception(f"Test suite setup ERROR: {test_suite= _name}") > + test_suite_result.update_setup(Result.ERROR, e) > > else: > - for test_suite_class in test_suite_classes: > - test_suite =3D test_suite_class(sut_node, tg_node, test_= suite_config.test_cases) > - > - test_suite_name =3D test_suite.__class__.__name__ > - test_suite_result =3D build_target_result.add_test_suite= (test_suite_name) > - try: > - self._logger.info(f"Starting test suite setup: {test= _suite_name}") > - test_suite.set_up_suite() > - test_suite_result.update_setup(Result.PASS) > - self._logger.info(f"Test suite setup successful: {te= st_suite_name}") > - except Exception as e: > - self._logger.exception(f"Test suite setup ERROR: {te= st_suite_name}") > - test_suite_result.update_setup(Result.ERROR, e) > - > - else: > - self._execute_test_suite(execution.func, test_suite,= test_suite_result) > - > - finally: > - try: > - test_suite.tear_down_suite() > - sut_node.kill_cleanup_dpdk_apps() > - test_suite_result.update_teardown(Result.PASS) > - except Exception as e: > - self._logger.exception(f"Test suite teardown ERR= OR: {test_suite_name}") > - self._logger.warning( > - f"Test suite '{test_suite_name}' teardown fa= iled, " > - f"the next test suite may be affected." > - ) > - test_suite_result.update_setup(Result.ERROR, e) > - if len(test_suite_result.get_errors()) > 0 and test_= suite.is_blocking: > - raise BlockingTestSuiteError(test_suite_name) > + self._execute_test_suite( > + test_suite, > + test_suite_with_cases.test_cases, > + test_suite_result, > + ) > + finally: > + try: > + test_suite.tear_down_suite() > + sut_node.kill_cleanup_dpdk_apps() > + test_suite_result.update_teardown(Result.PASS) > + except Exception as e: > + self._logger.exception(f"Test suite teardown ERROR: {tes= t_suite_name}") > + self._logger.warning( > + f"Test suite '{test_suite_name}' teardown failed, " > + "the next test suite may be affected." > + ) > + test_suite_result.update_setup(Result.ERROR, e) > + if len(test_suite_result.get_errors()) > 0 and test_suite.is= _blocking: > + raise BlockingTestSuiteError(test_suite_name) > > def _execute_test_suite( > - self, func: bool, test_suite: TestSuite, test_suite_result: Test= SuiteResult > + self, > + test_suite: TestSuite, > + test_cases: Iterable[MethodType], > + test_suite_result: TestSuiteResult, > ) -> None: > - """Execute all discovered test cases in `test_suite`. > + """Execute all `test_cases` in `test_suite`. > > If the :option:`--re-run` command line argument or the :envvar:`= DTS_RERUN` environment > variable is set, in case of a test case failure, the test case w= ill be executed again > until it passes or it fails that many times in addition of the f= irst failure. > > Args: > - func: Whether to execute functional test cases. > test_suite: The test suite object. > + test_cases: The list of test case methods. > test_suite_result: The test suite level result object associ= ated > with the current test suite. > """ > - if func: > - for test_case_method in test_suite._get_functional_test_case= s(): > - test_case_name =3D test_case_method.__name__ > - test_case_result =3D test_suite_result.add_test_case(tes= t_case_name) > - all_attempts =3D SETTINGS.re_run + 1 > - attempt_nr =3D 1 > + for test_case_method in test_cases: > + test_case_name =3D test_case_method.__name__ > + test_case_result =3D test_suite_result.add_test_case(test_ca= se_name) > + all_attempts =3D SETTINGS.re_run + 1 > + attempt_nr =3D 1 > + self._run_test_case(test_suite, test_case_method, test_case_= result) > + while not test_case_result and attempt_nr < all_attempts: > + attempt_nr +=3D 1 > + self._logger.info( > + f"Re-running FAILED test case '{test_case_name}'. " > + f"Attempt number {attempt_nr} out of {all_attempts}.= " > + ) > self._run_test_case(test_suite, test_case_method, test_c= ase_result) > - while not test_case_result and attempt_nr < all_attempts= : > - attempt_nr +=3D 1 > - self._logger.info( > - f"Re-running FAILED test case '{test_case_name}'= . " > - f"Attempt number {attempt_nr} out of {all_attemp= ts}." > - ) > - self._run_test_case(test_suite, test_case_method, te= st_case_result) > > def _run_test_case( > self, > @@ -399,7 +599,7 @@ def _run_test_case( > test_case_method: MethodType, > test_case_result: TestCaseResult, > ) -> None: > - """Setup, execute and teardown a test case in `test_suite`. > + """Setup, execute and teardown `test_case_method` from `test_sui= te`. > > Record the result of the setup and the teardown and handle failu= res. > > @@ -424,7 +624,7 @@ def _run_test_case( > > else: > # run test case if setup was successful > - self._execute_test_case(test_case_method, test_case_result) > + self._execute_test_case(test_suite, test_case_method, test_c= ase_result) > > finally: > try: > @@ -440,11 +640,15 @@ def _run_test_case( > test_case_result.update(Result.ERROR) > > def _execute_test_case( > - self, test_case_method: MethodType, test_case_result: TestCaseRe= sult > + self, > + test_suite: TestSuite, > + test_case_method: MethodType, > + test_case_result: TestCaseResult, > ) -> None: > - """Execute one test case, record the result and handle failures. > + """Execute `test_case_method` from `test_suite`, record the resu= lt and handle failures. > > Args: > + test_suite: The test suite object. > test_case_method: The test case method. > test_case_result: The test case level result object associat= ed > with the current test case. > @@ -452,7 +656,7 @@ def _execute_test_case( > test_case_name =3D test_case_method.__name__ > try: > self._logger.info(f"Starting test case execution: {test_case= _name}") > - test_case_method() > + test_case_method(test_suite) > test_case_result.update(Result.PASS) > self._logger.info(f"Test case execution PASSED: {test_case_n= ame}") > > diff --git a/dts/framework/settings.py b/dts/framework/settings.py > index 609c8d0e62..2b8bfbe0ed 100644 > --- a/dts/framework/settings.py > +++ b/dts/framework/settings.py > @@ -253,8 +253,7 @@ def _get_parser() -> argparse.ArgumentParser: > "--test-cases", > action=3D_env_arg("DTS_TESTCASES"), > default=3D"", > - help=3D"[DTS_TESTCASES] Comma-separated list of test cases to ex= ecute. " > - "Unknown test cases will be silently ignored.", > + help=3D"[DTS_TESTCASES] Comma-separated list of test cases to ex= ecute.", > ) > > parser.add_argument( > diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py > index 4467749a9d..075195fd5b 100644 > --- a/dts/framework/test_result.py > +++ b/dts/framework/test_result.py > @@ -25,7 +25,9 @@ > > import os.path > from collections.abc import MutableSequence > +from dataclasses import dataclass > from enum import Enum, auto > +from types import MethodType > > from .config import ( > OS, > @@ -36,10 +38,42 @@ > CPUType, > NodeConfiguration, > NodeInfo, > + TestSuiteConfig, > ) > from .exception import DTSError, ErrorSeverity > from .logger import DTSLOG > from .settings import SETTINGS > +from .test_suite import TestSuite > + > + > +@dataclass(slots=3DTrue, frozen=3DTrue) > +class TestSuiteWithCases: > + """A test suite class with test case methods. > + > + An auxiliary class holding a test case class with test case methods.= The intended use of this > + class is to hold a subset of test cases (which could be all test cas= es) because we don't have > + all the data to instantiate the class at the point of inspection. Th= e knowledge of this subset > + is needed in case an error occurs before the class is instantiated a= nd we need to record > + which test cases were blocked by the error. > + > + Attributes: > + test_suite_class: The test suite class. > + test_cases: The test case methods. > + """ > + > + test_suite_class: type[TestSuite] > + test_cases: list[MethodType] > + > + def create_config(self) -> TestSuiteConfig: > + """Generate a :class:`TestSuiteConfig` from the stored test suit= e with test cases. > + > + Returns: > + The :class:`TestSuiteConfig` representation. > + """ > + return TestSuiteConfig( > + test_suite=3Dself.test_suite_class.__name__, > + test_cases=3D[test_case.__name__ for test_case in self.test_= cases], > + ) > > > class Result(Enum): > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py > index b02fd36147..f9fe88093e 100644 > --- a/dts/framework/test_suite.py > +++ b/dts/framework/test_suite.py > @@ -11,25 +11,17 @@ > * Testbed (SUT, TG) configuration, > * Packet sending and verification, > * Test case verification. > - > -The module also defines a function, :func:`get_test_suites`, > -for gathering test suites from a Python module. > """ > > -import importlib > -import inspect > -import re > from ipaddress import IPv4Interface, IPv6Interface, ip_interface > -from types import MethodType > -from typing import Any, ClassVar, Union > +from typing import ClassVar, Union > > from scapy.layers.inet import IP # type: ignore[import] > from scapy.layers.l2 import Ether # type: ignore[import] > from scapy.packet import Packet, Padding # type: ignore[import] > > -from .exception import ConfigurationError, TestCaseVerifyError > +from .exception import TestCaseVerifyError > from .logger import DTSLOG, getLogger > -from .settings import SETTINGS > from .testbed_model import Port, PortLink, SutNode, TGNode > from .utils import get_packet_summaries > > @@ -37,7 +29,6 @@ > class TestSuite(object): > """The base class with building blocks needed by most test cases. > > - * Test case filtering and collection, > * Test suite setup/cleanup methods to override, > * Test case setup/cleanup methods to override, > * Test case verification, > @@ -71,7 +62,6 @@ class TestSuite(object): > #: will block the execution of all subsequent test suites in the cur= rent build target. > is_blocking: ClassVar[bool] =3D False > _logger: DTSLOG > - _test_cases_to_run: list[str] > _port_links: list[PortLink] > _sut_port_ingress: Port > _sut_port_egress: Port > @@ -86,24 +76,19 @@ def __init__( > self, > sut_node: SutNode, > tg_node: TGNode, > - test_cases: list[str], > ): > """Initialize the test suite testbed information and basic confi= guration. > > - Process what test cases to run, find links between ports and set= up > - default IP addresses to be used when configuring them. > + Find links between ports and set up default IP addresses to be u= sed when > + configuring them. > > Args: > sut_node: The SUT node where the test suite will run. > tg_node: The TG node where the test suite will run. > - test_cases: The list of test cases to execute. > - If empty, all test cases will be executed. > """ > self.sut_node =3D sut_node > self.tg_node =3D tg_node > self._logger =3D getLogger(self.__class__.__name__) > - self._test_cases_to_run =3D test_cases > - self._test_cases_to_run.extend(SETTINGS.test_cases) > self._port_links =3D [] > self._process_links() > self._sut_port_ingress, self._tg_port_egress =3D ( > @@ -364,65 +349,3 @@ def _verify_l3_packet(self, received_packet: IP, exp= ected_packet: IP) -> bool: > if received_packet.src !=3D expected_packet.src or received_pack= et.dst !=3D expected_packet.dst: > return False > return True > - > - def _get_functional_test_cases(self) -> list[MethodType]: > - """Get all functional test cases defined in this TestSuite. > - > - Returns: > - The list of functional test cases of this TestSuite. > - """ > - return self._get_test_cases(r"test_(?!perf_)") > - > - def _get_test_cases(self, test_case_regex: str) -> list[MethodType]: > - """Return a list of test cases matching test_case_regex. > - > - Returns: > - The list of test cases matching test_case_regex of this Test= Suite. > - """ > - self._logger.debug(f"Searching for test cases in {self.__class__= .__name__}.") > - filtered_test_cases =3D [] > - for test_case_name, test_case in inspect.getmembers(self, inspec= t.ismethod): > - if self._should_be_executed(test_case_name, test_case_regex)= : > - filtered_test_cases.append(test_case) > - cases_str =3D ", ".join((x.__name__ for x in filtered_test_cases= )) > - self._logger.debug(f"Found test cases '{cases_str}' in {self.__c= lass__.__name__}.") > - return filtered_test_cases > - > - def _should_be_executed(self, test_case_name: str, test_case_regex: = str) -> bool: > - """Check whether the test case should be scheduled to be execute= d.""" > - match =3D bool(re.match(test_case_regex, test_case_name)) > - if self._test_cases_to_run: > - return match and test_case_name in self._test_cases_to_run > - > - return match > - > - > -def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]= : > - r"""Find all :class:`TestSuite`\s in a Python module. > - > - Args: > - testsuite_module_path: The path to the Python module. > - > - Returns: > - The list of :class:`TestSuite`\s found within the Python module. > - > - Raises: > - ConfigurationError: The test suite module was not found. > - """ > - > - def is_test_suite(object: Any) -> bool: > - try: > - if issubclass(object, TestSuite) and object is not TestSuite= : > - return True > - except TypeError: > - return False > - return False > - > - try: > - testcase_module =3D importlib.import_module(testsuite_module_pat= h) > - except ModuleNotFoundError as e: > - raise ConfigurationError(f"Test suite '{testsuite_module_path}' = not found.") from e > - return [ > - test_suite_class > - for _, test_suite_class in inspect.getmembers(testcase_module, i= s_test_suite) > - ] > diff --git a/dts/pyproject.toml b/dts/pyproject.toml > index 28bd970ae4..8eb92b4f11 100644 > --- a/dts/pyproject.toml > +++ b/dts/pyproject.toml > @@ -51,6 +51,9 @@ linters =3D "mccabe,pycodestyle,pydocstyle,pyflakes" > format =3D "pylint" > max_line_length =3D 100 > > +[tool.pylama.linter.pycodestyle] > +ignore =3D "E203,W503" > + > [tool.pylama.linter.pydocstyle] > convention =3D "google" > > diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smo= ke_tests.py > index 5e2bac14bd..7b2a0e97f8 100644 > --- a/dts/tests/TestSuite_smoke_tests.py > +++ b/dts/tests/TestSuite_smoke_tests.py > @@ -21,7 +21,7 @@ > from framework.utils import REGEX_FOR_PCI_ADDRESS > > > -class SmokeTests(TestSuite): > +class TestSmokeTests(TestSuite): > """DPDK and infrastructure smoke test suite. > > The test cases validate the most basic DPDK functionality needed for= all other test suites. > -- > 2.34.1 >