From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6EFC543380; Mon, 20 Nov 2023 17:13:17 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1122742DDE; Mon, 20 Nov 2023 17:13:17 +0100 (CET) Received: from mail-ed1-f51.google.com (mail-ed1-f51.google.com [209.85.208.51]) by mails.dpdk.org (Postfix) with ESMTP id 5176042DD2 for ; Mon, 20 Nov 2023 17:13:15 +0100 (CET) Received: by mail-ed1-f51.google.com with SMTP id 4fb4d7f45d1cf-5446c9f3a77so6729577a12.0 for ; Mon, 20 Nov 2023 08:13:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1700496795; x=1701101595; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=/+UqHKUdjNtznBNCXwIiZIq4uxujgSo3DL0xqoibo8Y=; b=mBh1ar7yJdDxWiHAXKwsT3mQ2+vZI0vhNx1vNnIvw1Xwbqn/cZTvxDCEI3xiN8QUWb S3fOk2YkRI9K5+SlET6YJw4d+UtnMeb/JVAG0cm2rQ+2miF5eQfX3uSdxp+clzZMXl8X JnPWmG9wijEXxqhu8dmmI947xeparcuuXKQqR5liD1LZTJBVCX4xzRjMjcXpX8u5nWBy WTCEdxbhaYRV8oFc2fIgiJEKjPi3WwI1UcAh1QbOLuojzjEoldEmDmv8dh7lJ72XORzM WcaedT2ZAsnCr2pGuAE+ajKyFO6zzwHepMjaD/pAbj4ActNlbGjQA/j1AgNhkh1/8MHq Mddw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700496795; x=1701101595; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/+UqHKUdjNtznBNCXwIiZIq4uxujgSo3DL0xqoibo8Y=; b=mWv6GbOJL5LqTOrxI+XlBtM4MSHeoOGrvjCMsSDhkhu7CVMhs5BL+kXcWBidLetEFs evWer5E8f9EMA4QA+RDpsSf10/iCRy3+FAhm7ivoPEixCg6kml7yfGhxGUGCw4h1R6+S ZUKIknUL69YSsuX6+zBCdRDrDPDY8NJrt4OgRrtu/Ji+Yv8SSRdJ8P4Y7znjZhEcisMF qcMBn8g8+ncVl0RypOLA+fndZYlBuVJWmffqAhvXYiUfmGmvx5Y9VVzl0MyR9c1MwT0C CaZ7nQd6Mc9TI2xc3uZrpeQKEuMrLU3jyrq+xLNV1TG0deBOU+JPHbcQPmhF0JPCPZzk 2dXA== X-Gm-Message-State: AOJu0Yy9XRc7fnsWzb+VWfdEiAOts8AvAphvijkz1ByNw90t5qMVODgL KHwZGCNcI+apLXNDUaVZOOCWqdVJtCve++/00SqhSQ== X-Google-Smtp-Source: AGHT+IGS2q03Au0CRRYLuZ0YApAEo3oNiG3i70Qx6LqLV6MMrYb1uKWLeRATO9k5HZZq0e7rWifoogLH/TiGnYX5iLk= X-Received: by 2002:a17:906:3f1b:b0:9fe:a881:81ab with SMTP id c27-20020a1709063f1b00b009fea88181abmr2515070ejj.53.1700496794923; Mon, 20 Nov 2023 08:13:14 -0800 (PST) MIME-Version: 1.0 References: <20231108125324.191005-23-juraj.linkes@pantheon.tech> <20231115130959.39420-1-juraj.linkes@pantheon.tech> <20231115130959.39420-8-juraj.linkes@pantheon.tech> In-Reply-To: From: =?UTF-8?Q?Juraj_Linke=C5=A1?= Date: Mon, 20 Nov 2023 17:13:04 +0100 Message-ID: Subject: Re: [PATCH v7 07/21] dts: dts runner and main docstring update To: Jeremy Spewock Cc: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, probb@iol.unh.edu, paul.szczepanek@arm.com, yoan.picchi@foss.arm.com, dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Thu, Nov 16, 2023 at 10:51=E2=80=AFPM Jeremy Spewock wrote: > > > > On Wed, Nov 15, 2023 at 8:11=E2=80=AFAM Juraj Linke=C5=A1 wrote: >> >> Format according to the Google format and PEP257, with slight >> deviations. >> >> Signed-off-by: Juraj Linke=C5=A1 >> --- >> dts/framework/dts.py | 128 ++++++++++++++++++++++++++++++++++++------- >> dts/main.py | 8 ++- >> 2 files changed, 112 insertions(+), 24 deletions(-) >> >> diff --git a/dts/framework/dts.py b/dts/framework/dts.py >> index 4c7fb0c40a..331fed7dc4 100644 >> --- a/dts/framework/dts.py >> +++ b/dts/framework/dts.py >> @@ -3,6 +3,33 @@ >> # Copyright(c) 2022-2023 PANTHEON.tech s.r.o. >> # Copyright(c) 2022-2023 University of New Hampshire >> >> +r"""Test suite runner module. >> + >> +A DTS run is split into stages: >> + >> + #. Execution stage, >> + #. Build target stage, >> + #. Test suite stage, >> + #. Test case stage. >> + >> +The module is responsible for running tests on testbeds defined in the = test run configuration. >> +Each setup or teardown of each stage is recorded in a :class:`~framewor= k.test_result.DTSResult` or >> +one of its subclasses. The test case results are also recorded. >> + >> +If an error occurs, the current stage is aborted, the error is recorded= and the run continues in >> +the next iteration of the same stage. The return code is the highest `s= everity` of all >> +:class:`~.framework.exception.DTSError`\s. >> + >> +Example: >> + An error occurs in a build target setup. The current build target i= s aborted and the run >> + continues with the next build target. If the errored build target w= as the last one in the given >> + execution, the next execution begins. >> + >> +Attributes: >> + dts_logger: The logger instance used in this module. >> + result: The top level result used in the module. >> +""" >> + >> import sys >> >> from .config import ( >> @@ -23,9 +50,38 @@ >> >> >> def run_all() -> None: >> - """ >> - The main process of DTS. Runs all build targets in all executions f= rom the main >> - config file. >> + """Run all build targets in all executions from the test run config= uration. >> + >> + Before running test suites, executions and build targets are first = set up. >> + The executions and build targets defined in the test run configurat= ion are iterated over. >> + The executions define which tests to run and where to run them and = build targets define >> + the DPDK build setup. >> + >> + The tests suites are set up for each execution/build target tuple a= nd each scheduled >> + test case within the test suite is set up, executed and torn down. = After all test cases >> + have been executed, the test suite is torn down and the next build = target will be tested. >> + >> + All the nested steps look like this: >> + >> + #. Execution setup >> + >> + #. Build target setup >> + >> + #. Test suite setup >> + >> + #. Test case setup >> + #. Test case logic >> + #. Test case teardown >> + >> + #. Test suite teardown >> + >> + #. Build target teardown >> + >> + #. Execution teardown >> + >> + The test cases are filtered according to the specification in the t= est run configuration and >> + the :option:`--test-cases` command line argument or >> + the :envvar:`DTS_TESTCASES` environment variable. >> """ >> global dts_logger >> global result >> @@ -87,6 +143,8 @@ def run_all() -> None: >> >> >> def _check_dts_python_version() -> None: >> + """Check the required Python version - v3.10.""" >> + >> def RED(text: str) -> str: >> return f"\u001B[31;1m{str(text)}\u001B[0m" >> >> @@ -111,9 +169,16 @@ def _run_execution( >> execution: ExecutionConfiguration, >> result: DTSResult, >> ) -> None: >> - """ >> - Run the given execution. This involves running the execution setup = as well as >> - running all build targets in the given execution. >> + """Run the given execution. >> + >> + This involves running the execution setup as well as running all bu= ild targets >> + in the given execution. After that, execution teardown is run. >> + >> + Args: >> + sut_node: The execution's SUT node. >> + tg_node: The execution's TG node. >> + execution: An execution's test run configuration. >> + result: The top level result object. >> """ >> dts_logger.info( >> f"Running execution with SUT '{execution.system_under_test_node= .name}'." >> @@ -150,8 +215,18 @@ def _run_build_target( >> execution: ExecutionConfiguration, >> execution_result: ExecutionResult, >> ) -> None: >> - """ >> - Run the given build target. >> + """Run the given build target. >> + >> + This involves running the build target setup as well as running all= test suites >> + in the given execution the build target is defined in. >> + After that, build target teardown is run. >> + >> + Args: >> + sut_node: The execution's SUT node. >> + tg_node: The execution's TG node. >> + build_target: A build target's test run configuration. >> + execution: The build target's execution's test run configuratio= n. >> + execution_result: The execution level result object associated = with the execution. >> """ >> dts_logger.info(f"Running build target '{build_target.name}'.") >> build_target_result =3D execution_result.add_build_target(build_tar= get) >> @@ -183,10 +258,17 @@ def _run_all_suites( >> execution: ExecutionConfiguration, >> build_target_result: BuildTargetResult, >> ) -> None: >> - """ >> - Use the given build_target to run execution's test suites >> - with possibly only a subset of test cases. >> - If no subset is specified, run all test cases. >> + """Run the execution's (possibly a subset) test suites using the cu= rrent build_target. >> + >> + The function assumes the build target we're testing has already bee= n built on the SUT node. >> + The current build target thus corresponds to the current DPDK build= present on the SUT node. >> + >> + Args: >> + sut_node: The execution's SUT node. >> + tg_node: The execution's TG node. >> + execution: The execution's test run configuration associated wi= th the current build target. >> + build_target_result: The build target level result object assoc= iated >> + with the current build target. >> """ > > > Is it worth mentioning in this method or the _run_build_target method tha= t when a blocking suite fails that no more suites will be run on that build= target? > Absolutely, I'll add that. Thanks for the catch. >> >> end_build_target =3D False >> if not execution.skip_smoke_tests: >> @@ -215,16 +297,22 @@ def _run_single_suite( >> build_target_result: BuildTargetResult, >> test_suite_config: TestSuiteConfig, >> ) -> None: >> - """Runs a single test suite. >> + """Run all test suite in a single test suite module. >> + >> + The function assumes the build target we're testing has already bee= n built on the SUT node. >> + The current build target thus corresponds to the current DPDK build= present on the SUT node. >> >> Args: >> - sut_node: Node to run tests on. >> - execution: Execution the test case belongs to. >> - build_target_result: Build target configuration test case is ru= n on >> - test_suite_config: Test suite configuration >> + sut_node: The execution's SUT node. >> + tg_node: The execution's TG node. >> + execution: The execution's test run configuration associated wi= th the current build target. >> + build_target_result: The build target level result object assoc= iated >> + with the current build target. >> + test_suite_config: Test suite test run configuration specifying= the test suite module >> + and possibly a subset of test cases of test suites in that = module. >> >> Raises: >> - BlockingTestSuiteError: If a test suite that was marked as bloc= king fails. >> + BlockingTestSuiteError: If a blocking test suite fails. >> """ >> try: >> full_suite_path =3D f"tests.TestSuite_{test_suite_config.test_s= uite}" >> @@ -248,9 +336,7 @@ def _run_single_suite( >> >> >> def _exit_dts() -> None: >> - """ >> - Process all errors and exit with the proper exit code. >> - """ >> + """Process all errors and exit with the proper exit code.""" >> result.process() >> >> if dts_logger: >> diff --git a/dts/main.py b/dts/main.py >> index 5d4714b0c3..f703615d11 100755 >> --- a/dts/main.py >> +++ b/dts/main.py >> @@ -4,9 +4,7 @@ >> # Copyright(c) 2022 PANTHEON.tech s.r.o. >> # Copyright(c) 2022 University of New Hampshire >> >> -""" >> -A test framework for testing DPDK. >> -""" >> +"""The DTS executable.""" >> >> import logging >> >> @@ -17,6 +15,10 @@ def main() -> None: >> """Set DTS settings, then run DTS. >> >> The DTS settings are taken from the command line arguments and the = environment variables. >> + The settings object is stored in the module-level variable settings= .SETTINGS which the entire >> + framework uses. After importing the module (or the variable), any c= hanges to the variable are >> + not going to be reflected without a re-import. This means that the = SETTINGS variable must >> + be modified before the settings module is imported anywhere else in= the framework. >> """ >> settings.SETTINGS =3D settings.get_settings() >> from framework import dts >> -- >> 2.34.1 >>