From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 71E4F432D6;
Wed, 8 Nov 2023 13:54:26 +0100 (CET)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 6747C42E35;
Wed, 8 Nov 2023 13:53:38 +0100 (CET)
Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com
[209.85.208.44]) by mails.dpdk.org (Postfix) with ESMTP id 4177E402DA
for <dev@dpdk.org>; Wed, 8 Nov 2023 13:53:34 +0100 (CET)
Received: by mail-ed1-f44.google.com with SMTP id
4fb4d7f45d1cf-53d9f001b35so11500832a12.2
for <dev@dpdk.org>; Wed, 08 Nov 2023 04:53:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=pantheon.tech; s=google; t=1699448014; x=1700052814; darn=dpdk.org;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:from:to:cc:subject:date
:message-id:reply-to;
bh=2ZNMam0a1R4XoTdcPdnNVy5Fpo9kpNHkFyw2wLxXUoc=;
b=e73rX1VNa/pnUCMfYt5A5nuYJQBVL44hby+mLLrij1Mwb6C2vOE+IZyEgVIc6NQ0jZ
o32Uw/23n+SkSRmXPTizj2CggWd9mqL/51lNASVJm4dWKbQbRhLB28uYh6w9zX4SLrr5
KpKXvF93Xh1oxbmrF4aoML9oXnoiWUWOiMt/UHurvMGK39GipjARdSbCofFpxxxhwYzN
LC9SyOcHzSMPBzwDAsSMjIKo3t80tGoIznR9lEyv77EsT5tohP899XwEH+g2cdKUr1zD
AciRNq2pVY8UmRl1gLABjMAFLvTH0OQL5eR+cloqS7wg3e/jQbtbg3sbb2u5TtXdpgfK
pDXQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1699448014; x=1700052814;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=2ZNMam0a1R4XoTdcPdnNVy5Fpo9kpNHkFyw2wLxXUoc=;
b=TTafWNCmFwM+vq3OH3VJyvuV4xEIlQWDFv+BCAY98qWaKX/uFfFPFOE2CI0BHkrtsG
hs8nb5xviTjh566RamTXeCFEU/Z3Bpqz+BGhZ+zK/RTpwbZm7MexEIOs6I9fvsAsfAvT
pa35YZ+vtdyUCS/KadmIkvirW+qcrjZAR4gr78GgnKgpvJ11WXHbagUJaDMR/iW8s8hr
v+6T/bjNgy5JQg7sL4O8skcc50rYtiL0c4W+ogOSOONBTEnUX6Qdi2tauu2w9ee2J303
eZ64B7bSnACdGZgZ02iyj7+cT/paQTyXHN2ulPu+ClPvOlEU3e0FcYdii/viyToTKEgO
d64Q==
X-Gm-Message-State: AOJu0YxSyAmws5tmtN8aWHJooDtpwrRVLg/KSh6NEeRfKSqg9zeaVlhr
fI4s5b341TS4BO1PnGcMpyFvfp41erUySFN2yF/6Zw==
X-Google-Smtp-Source: AGHT+IE0JeO/8adVks+0wAYSxXVd1Q+i5LaN50di9sPYzZX9OwdIyKqP0+YIbDBSsqe7abEQUfo2iQ==
X-Received: by 2002:a50:cc8a:0:b0:543:5d2e:a9c3 with SMTP id
q10-20020a50cc8a000000b005435d2ea9c3mr1512636edi.20.1699448013876;
Wed, 08 Nov 2023 04:53:33 -0800 (PST)
Received: from jlinkes-PT-Latitude-5530.pantheon.local
(81.89.53.154.host.vnet.sk. [81.89.53.154])
by smtp.gmail.com with ESMTPSA id
v10-20020aa7dbca000000b0052ff9bae873sm6589289edt.5.2023.11.08.04.53.32
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Wed, 08 Nov 2023 04:53:33 -0800 (PST)
From: =?UTF-8?q?Juraj=20Linke=C5=A1?= <juraj.linkes@pantheon.tech>
To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com,
bruce.richardson@intel.com, jspewock@iol.unh.edu, probb@iol.unh.edu,
paul.szczepanek@arm.com, yoan.picchi@foss.arm.com
Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= <juraj.linkes@pantheon.tech>
Subject: [PATCH v6 07/23] dts: dts runner and main docstring update
Date: Wed, 8 Nov 2023 13:53:08 +0100
Message-Id: <20231108125324.191005-7-juraj.linkes@pantheon.tech>
X-Mailer: git-send-email 2.34.1
In-Reply-To: <20231108125324.191005-1-juraj.linkes@pantheon.tech>
References: <20231106171601.160749-1-juraj.linkes@pantheon.tech>
<20231108125324.191005-1-juraj.linkes@pantheon.tech>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Format according to the Google format and PEP257, with slight
deviations.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/framework/dts.py | 128 ++++++++++++++++++++++++++++++++++++-------
dts/main.py | 8 ++-
2 files changed, 112 insertions(+), 24 deletions(-)
diff --git a/dts/framework/dts.py b/dts/framework/dts.py
index 4c7fb0c40a..331fed7dc4 100644
--- a/dts/framework/dts.py
+++ b/dts/framework/dts.py
@@ -3,6 +3,33 @@
# Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
# Copyright(c) 2022-2023 University of New Hampshire
+r"""Test suite runner module.
+
+A DTS run is split into stages:
+
+ #. Execution stage,
+ #. Build target stage,
+ #. Test suite stage,
+ #. Test case stage.
+
+The module is responsible for running tests on testbeds defined in the test run configuration.
+Each setup or teardown of each stage is recorded in a :class:`~framework.test_result.DTSResult` or
+one of its subclasses. The test case results are also recorded.
+
+If an error occurs, the current stage is aborted, the error is recorded and the run continues in
+the next iteration of the same stage. The return code is the highest `severity` of all
+:class:`~.framework.exception.DTSError`\s.
+
+Example:
+ An error occurs in a build target setup. The current build target is aborted and the run
+ continues with the next build target. If the errored build target was the last one in the given
+ execution, the next execution begins.
+
+Attributes:
+ dts_logger: The logger instance used in this module.
+ result: The top level result used in the module.
+"""
+
import sys
from .config import (
@@ -23,9 +50,38 @@
def run_all() -> None:
- """
- The main process of DTS. Runs all build targets in all executions from the main
- config file.
+ """Run all build targets in all executions from the test run configuration.
+
+ Before running test suites, executions and build targets are first set up.
+ The executions and build targets defined in the test run configuration are iterated over.
+ The executions define which tests to run and where to run them and build targets define
+ the DPDK build setup.
+
+ The tests suites are set up for each execution/build target tuple and each scheduled
+ test case within the test suite is set up, executed and torn down. After all test cases
+ have been executed, the test suite is torn down and the next build target will be tested.
+
+ All the nested steps look like this:
+
+ #. Execution setup
+
+ #. Build target setup
+
+ #. Test suite setup
+
+ #. Test case setup
+ #. Test case logic
+ #. Test case teardown
+
+ #. Test suite teardown
+
+ #. Build target teardown
+
+ #. Execution teardown
+
+ The test cases are filtered according to the specification in the test run configuration and
+ the :option:`--test-cases` command line argument or
+ the :envvar:`DTS_TESTCASES` environment variable.
"""
global dts_logger
global result
@@ -87,6 +143,8 @@ def run_all() -> None:
def _check_dts_python_version() -> None:
+ """Check the required Python version - v3.10."""
+
def RED(text: str) -> str:
return f"\u001B[31;1m{str(text)}\u001B[0m"
@@ -111,9 +169,16 @@ def _run_execution(
execution: ExecutionConfiguration,
result: DTSResult,
) -> None:
- """
- Run the given execution. This involves running the execution setup as well as
- running all build targets in the given execution.
+ """Run the given execution.
+
+ This involves running the execution setup as well as running all build targets
+ in the given execution. After that, execution teardown is run.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: An execution's test run configuration.
+ result: The top level result object.
"""
dts_logger.info(
f"Running execution with SUT '{execution.system_under_test_node.name}'."
@@ -150,8 +215,18 @@ def _run_build_target(
execution: ExecutionConfiguration,
execution_result: ExecutionResult,
) -> None:
- """
- Run the given build target.
+ """Run the given build target.
+
+ This involves running the build target setup as well as running all test suites
+ in the given execution the build target is defined in.
+ After that, build target teardown is run.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ build_target: A build target's test run configuration.
+ execution: The build target's execution's test run configuration.
+ execution_result: The execution level result object associated with the execution.
"""
dts_logger.info(f"Running build target '{build_target.name}'.")
build_target_result = execution_result.add_build_target(build_target)
@@ -183,10 +258,17 @@ def _run_all_suites(
execution: ExecutionConfiguration,
build_target_result: BuildTargetResult,
) -> None:
- """
- Use the given build_target to run execution's test suites
- with possibly only a subset of test cases.
- If no subset is specified, run all test cases.
+ """Run the execution's (possibly a subset) test suites using the current build_target.
+
+ The function assumes the build target we're testing has already been built on the SUT node.
+ The current build target thus corresponds to the current DPDK build present on the SUT node.
+
+ Args:
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: The execution's test run configuration associated with the current build target.
+ build_target_result: The build target level result object associated
+ with the current build target.
"""
end_build_target = False
if not execution.skip_smoke_tests:
@@ -215,16 +297,22 @@ def _run_single_suite(
build_target_result: BuildTargetResult,
test_suite_config: TestSuiteConfig,
) -> None:
- """Runs a single test suite.
+ """Run all test suite in a single test suite module.
+
+ The function assumes the build target we're testing has already been built on the SUT node.
+ The current build target thus corresponds to the current DPDK build present on the SUT node.
Args:
- sut_node: Node to run tests on.
- execution: Execution the test case belongs to.
- build_target_result: Build target configuration test case is run on
- test_suite_config: Test suite configuration
+ sut_node: The execution's SUT node.
+ tg_node: The execution's TG node.
+ execution: The execution's test run configuration associated with the current build target.
+ build_target_result: The build target level result object associated
+ with the current build target.
+ test_suite_config: Test suite test run configuration specifying the test suite module
+ and possibly a subset of test cases of test suites in that module.
Raises:
- BlockingTestSuiteError: If a test suite that was marked as blocking fails.
+ BlockingTestSuiteError: If a blocking test suite fails.
"""
try:
full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
@@ -248,9 +336,7 @@ def _run_single_suite(
def _exit_dts() -> None:
- """
- Process all errors and exit with the proper exit code.
- """
+ """Process all errors and exit with the proper exit code."""
result.process()
if dts_logger:
diff --git a/dts/main.py b/dts/main.py
index 5d4714b0c3..f703615d11 100755
--- a/dts/main.py
+++ b/dts/main.py
@@ -4,9 +4,7 @@
# Copyright(c) 2022 PANTHEON.tech s.r.o.
# Copyright(c) 2022 University of New Hampshire
-"""
-A test framework for testing DPDK.
-"""
+"""The DTS executable."""
import logging
@@ -17,6 +15,10 @@ def main() -> None:
"""Set DTS settings, then run DTS.
The DTS settings are taken from the command line arguments and the environment variables.
+ The settings object is stored in the module-level variable settings.SETTINGS which the entire
+ framework uses. After importing the module (or the variable), any changes to the variable are
+ not going to be reflected without a re-import. This means that the SETTINGS variable must
+ be modified before the settings module is imported anywhere else in the framework.
"""
settings.SETTINGS = settings.get_settings()
from framework import dts
--
2.34.1