From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id DDDF041C8A;
Mon, 13 Feb 2023 16:29:54 +0100 (CET)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 3D56D42D4B;
Mon, 13 Feb 2023 16:29:09 +0100 (CET)
Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com
[209.85.221.43]) by mails.dpdk.org (Postfix) with ESMTP id A413742D2D
for <dev@dpdk.org>; Mon, 13 Feb 2023 16:29:00 +0100 (CET)
Received: by mail-wr1-f43.google.com with SMTP id a2so12650137wrd.6
for <dev@dpdk.org>; Mon, 13 Feb 2023 07:29:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=pantheon-tech.20210112.gappssmtp.com; s=20210112;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:from:to:cc:subject:date
:message-id:reply-to;
bh=AbGn4fIxmG0q2izl0ccaCBDn0/VaogkIvQiFJO0XPLo=;
b=jGDkJfMO7hW8YBu605YSr3XgIwkemgs2vYuQt1C/Ap++KbtJauwW3/885egdO9OqCk
e/kqjaEA5u03Z3pmjs5ZwyRs3J3rQ1rzDEXa/BwGvZ/p3P1Fd6Wwyw0LX/EzGM9nDTWX
/1sgUa0hl1dAstCL7xgvQIUGVS8GJxEivboIEOd3RlFNLTbKxKWA3xBSePkVzUXa/aB5
vJgsRCJKUxDa0vjW7IneXPI5cHW0kn1dQvt/pjoIu+yua9Wx0xUkIAcKAypkWr79ksMR
bxBKsdCoV6gPENm/j00RUZui4y5DpaN4Doar506DsQ73+4EtEQfEyI7wj5e2Xa3+LWsV
WcCw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=AbGn4fIxmG0q2izl0ccaCBDn0/VaogkIvQiFJO0XPLo=;
b=L1S0V3+UzJxnJhLcg1SEyH6jBiXCXUQHmKB0m9l03zUme98lzVuAZblaI8hEQTO6/7
8dIddJ/EjrCgbaoDs5ajM7FT23mqHsGa0Dwacws0vBwVNm2iYEjPnJTgSKwI91HLP+c5
p2mVJ7Qg1LZCXHXWNFXG7fd6AKKVEKozdMp8RKyVJ2GLgXHSjYJ7qBgLQLlliu0tD9Ra
E9ajq1wm/akbnn3E1h8QvQcZtei0a0+liK+aBLaR0rU9nC8AUwg8dbezAwhfzosdkeRp
2fKBAZILRopbgTYU/hFGo6iOXzvgqzcIMTxs3eyLCBwzwsBEnlv5w/yOy7baWreaeoNP
RTZA==
X-Gm-Message-State: AO0yUKUdzDNiMV556/6QGS4kMJ2C4Gy3PE49ON/gXexc3kxiVOtvB7oV
6Q4PSJ5cHL1TJP2hLC6vlc6GCQ==
X-Google-Smtp-Source: AK7set9cCJ3JacgNj+l1asRBSv+4vLpv1dpIXi5PWFuAYgmb7/daccTIIGCuqqgT8BmL2Ne29XXo/w==
X-Received: by 2002:a5d:51c2:0:b0:2c3:e5f5:9fb4 with SMTP id
n2-20020a5d51c2000000b002c3e5f59fb4mr21060647wrv.16.1676302140357;
Mon, 13 Feb 2023 07:29:00 -0800 (PST)
Received: from localhost.localdomain ([84.245.121.112])
by smtp.gmail.com with ESMTPSA id
d13-20020adfe88d000000b002c54f4d0f71sm5848613wrm.38.2023.02.13.07.28.59
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Mon, 13 Feb 2023 07:29:00 -0800 (PST)
From: =?UTF-8?q?Juraj=20Linke=C5=A1?= <juraj.linkes@pantheon.tech>
To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu,
lijuan.tu@intel.com, bruce.richardson@intel.com,
wathsala.vithanage@arm.com, probb@iol.unh.edu
Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= <juraj.linkes@pantheon.tech>
Subject: [PATCH v4 08/10] dts: add test suite config and runner
Date: Mon, 13 Feb 2023 16:28:44 +0100
Message-Id: <20230213152846.284191-9-juraj.linkes@pantheon.tech>
X-Mailer: git-send-email 2.30.2
In-Reply-To: <20230213152846.284191-1-juraj.linkes@pantheon.tech>
References: <20230117154906.860916-1-juraj.linkes@pantheon.tech>
<20230213152846.284191-1-juraj.linkes@pantheon.tech>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
The config allows users to specify which test suites and test cases
within test suites to run.
Also add test suite running capabilities to dts runner.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/conf.yaml | 2 ++
dts/framework/config/__init__.py | 29 +++++++++++++++-
dts/framework/config/conf_yaml_schema.json | 40 ++++++++++++++++++++++
dts/framework/dts.py | 19 ++++++++++
dts/framework/test_suite.py | 24 ++++++++++++-
5 files changed, 112 insertions(+), 2 deletions(-)
diff --git a/dts/conf.yaml b/dts/conf.yaml
index b606aa0df1..70989be528 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -10,6 +10,8 @@ executions:
compiler_wrapper: ccache
perf: false
func: true
+ test_suites:
+ - hello_world
system_under_test: "SUT 1"
nodes:
- name: "SUT 1"
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 544fceca6a..ebb0823ff5 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -12,7 +12,7 @@
import pathlib
from dataclasses import dataclass
from enum import Enum, auto, unique
-from typing import Any
+from typing import Any, TypedDict
import warlock # type: ignore
import yaml
@@ -128,11 +128,34 @@ def from_dict(d: dict) -> "BuildTargetConfiguration":
)
+class TestSuiteConfigDict(TypedDict):
+ suite: str
+ cases: list[str]
+
+
+@dataclass(slots=True, frozen=True)
+class TestSuiteConfig:
+ test_suite: str
+ test_cases: list[str]
+
+ @staticmethod
+ def from_dict(
+ entry: str | TestSuiteConfigDict,
+ ) -> "TestSuiteConfig":
+ if isinstance(entry, str):
+ return TestSuiteConfig(test_suite=entry, test_cases=[])
+ elif isinstance(entry, dict):
+ return TestSuiteConfig(test_suite=entry["suite"], test_cases=entry["cases"])
+ else:
+ raise TypeError(f"{type(entry)} is not valid for a test suite config.")
+
+
@dataclass(slots=True, frozen=True)
class ExecutionConfiguration:
build_targets: list[BuildTargetConfiguration]
perf: bool
func: bool
+ test_suites: list[TestSuiteConfig]
system_under_test: NodeConfiguration
@staticmethod
@@ -140,6 +163,9 @@ def from_dict(d: dict, node_map: dict) -> "ExecutionConfiguration":
build_targets: list[BuildTargetConfiguration] = list(
map(BuildTargetConfiguration.from_dict, d["build_targets"])
)
+ test_suites: list[TestSuiteConfig] = list(
+ map(TestSuiteConfig.from_dict, d["test_suites"])
+ )
sut_name = d["system_under_test"]
assert sut_name in node_map, f"Unknown SUT {sut_name} in execution {d}"
@@ -147,6 +173,7 @@ def from_dict(d: dict, node_map: dict) -> "ExecutionConfiguration":
build_targets=build_targets,
perf=d["perf"],
func=d["func"],
+ test_suites=test_suites,
system_under_test=node_map[sut_name],
)
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 878ca3aec2..ca2d4a1ef2 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -93,6 +93,32 @@
"required": [
"amount"
]
+ },
+ "test_suite": {
+ "type": "string",
+ "enum": [
+ "hello_world"
+ ]
+ },
+ "test_target": {
+ "type": "object",
+ "properties": {
+ "suite": {
+ "$ref": "#/definitions/test_suite"
+ },
+ "cases": {
+ "type": "array",
+ "description": "If specified, only this subset of test suite's test cases will be run. Unknown test cases will be silently ignored.",
+ "items": {
+ "type": "string"
+ },
+ "minimum": 1
+ }
+ },
+ "required": [
+ "suite"
+ ],
+ "additionalProperties": false
}
},
"type": "object",
@@ -172,6 +198,19 @@
"type": "boolean",
"description": "Enable functional testing."
},
+ "test_suites": {
+ "type": "array",
+ "items": {
+ "oneOf": [
+ {
+ "$ref": "#/definitions/test_suite"
+ },
+ {
+ "$ref": "#/definitions/test_target"
+ }
+ ]
+ }
+ },
"system_under_test": {
"$ref": "#/definitions/node_name"
}
@@ -181,6 +220,7 @@
"build_targets",
"perf",
"func",
+ "test_suites",
"system_under_test"
]
},
diff --git a/dts/framework/dts.py b/dts/framework/dts.py
index 6ea7c6e736..f98000450f 100644
--- a/dts/framework/dts.py
+++ b/dts/framework/dts.py
@@ -8,6 +8,7 @@
from .config import CONFIGURATION, BuildTargetConfiguration, ExecutionConfiguration
from .exception import DTSError, ErrorSeverity
from .logger import DTSLOG, getLogger
+from .test_suite import get_test_suites
from .testbed_model import SutNode
from .utils import check_dts_python_version
@@ -132,6 +133,24 @@ def _run_suites(
with possibly only a subset of test cases.
If no subset is specified, run all test cases.
"""
+ for test_suite_config in execution.test_suites:
+ try:
+ full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}"
+ test_suite_classes = get_test_suites(full_suite_path)
+ suites_str = ", ".join((x.__name__ for x in test_suite_classes))
+ dts_logger.debug(
+ f"Found test suites '{suites_str}' in '{full_suite_path}'."
+ )
+ except Exception as e:
+ dts_logger.exception("An error occurred when searching for test suites.")
+ errors.append(e)
+
+ else:
+ for test_suite_class in test_suite_classes:
+ test_suite = test_suite_class(
+ sut_node, test_suite_config.test_cases, execution.func, errors
+ )
+ test_suite.run()
def _exit_dts() -> None:
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 0972a70c14..57118f7c14 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -6,12 +6,13 @@
Base class for creating DTS test cases.
"""
+import importlib
import inspect
import re
from collections.abc import MutableSequence
from types import MethodType
-from .exception import SSHTimeoutError, TestCaseVerifyError
+from .exception import ConfigurationError, SSHTimeoutError, TestCaseVerifyError
from .logger import DTSLOG, getLogger
from .settings import SETTINGS
from .testbed_model import SutNode
@@ -226,3 +227,24 @@ def _execute_test_case(self, test_case_method: MethodType) -> bool:
raise KeyboardInterrupt("Stop DTS")
return result
+
+
+def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]:
+ def is_test_suite(object) -> bool:
+ try:
+ if issubclass(object, TestSuite) and object is not TestSuite:
+ return True
+ except TypeError:
+ return False
+ return False
+
+ try:
+ testcase_module = importlib.import_module(testsuite_module_path)
+ except ModuleNotFoundError as e:
+ raise ConfigurationError(
+ f"Test suite '{testsuite_module_path}' not found."
+ ) from e
+ return [
+ test_suite_class
+ for _, test_suite_class in inspect.getmembers(testcase_module, is_test_suite)
+ ]
--
2.30.2