From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8AE4545830; Wed, 21 Aug 2024 16:54:04 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 34D3342759; Wed, 21 Aug 2024 16:53:29 +0200 (CEST) Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) by mails.dpdk.org (Postfix) with ESMTP id B332D411F3 for ; Wed, 21 Aug 2024 16:53:25 +0200 (CEST) Received: by mail-ed1-f54.google.com with SMTP id 4fb4d7f45d1cf-5bec78c3f85so6215216a12.1 for ; Wed, 21 Aug 2024 07:53:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1724252005; x=1724856805; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hCqVS2m2INp/6NcDjRm0craXL99gKfi6g/uL3HEqK9I=; b=EEctc258fPqJyO/y8ZwACykNVUYcGb/P4P7WPezrm/iDb61ZAL0d8YMPWv6HQ8aFuW HJyLDZhGArjz+hkcCIrRdz5a+94fwOe8cHwf+YuaERBUW7oWaBflNL/MCfZszfQocaL5 vs3yC72J/90Ur6BDxzIpmSlXWuDlZm/jc15YcsYlzZvDEDd9pRBa1meOAY7Qu76D+G3f rJgIawzvuh2T/0Lvg1hybShr3yUSxwt64PYQKMgvsGG6OUHwFbhf2JQzIjtCQQoqrbjN 46640uTbdqMJJqDgb4vZ6qCvbAYKkrt1TvYmUtpTg7sXlU8Q99XEd9A0Fn7BD3tCz+IG SiXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724252005; x=1724856805; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hCqVS2m2INp/6NcDjRm0craXL99gKfi6g/uL3HEqK9I=; b=Ld3xYjXGv2B8j+qOYZkuAKtICic5mOM9Db7B+pGA6BAAZpoAfW0go7U63acJ0L5sFz X1ASSDGqJFFxmP4RV0EpJ2DItVqGY8fsKp9qxKk/8G8NgctLcI622/n7Ca9f4uSuOfY6 05kKS8recoqvSMzL1FBl/UiJL6oEY0/O4y5Ra2PlkMpQR55g6ILXujQL0Lh7gNb9nXe8 DOHU78FDWExXzcNqISKain01TMM4qQzwF517+d/35eZggNcWxDUfgXa9Lg6nlODVIQYR 8a0ktOuraDhntfizzQUU9o32gduuV++1bmLJFdKOd9nOALbEGY07yZaYbY8okZiXqvSd ua7Q== X-Gm-Message-State: AOJu0YwTyLRTTi/oW1GSJPJDaERE14G7cNs6zIfs9vdsRaKesDV/9rHj VsayuGsM9Wo8Cq0f+VQfM2vkD2M+HFqwS9KN/pXHdRp7HXPARYfpEg1sQeSgqN4= X-Google-Smtp-Source: AGHT+IG1nMep6EcRV+0BYu8W6Jfn2LEmEJCwSHseowELg93cxs9N/Aj8gZYgSmENIT6nIpvRjyIxSA== X-Received: by 2002:a17:907:86a3:b0:a86:67e7:c740 with SMTP id a640c23a62f3a-a866f2930ffmr170308566b.17.1724252005185; Wed, 21 Aug 2024 07:53:25 -0700 (PDT) Received: from localhost.localdomain ([84.245.121.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8680c4725csm52434866b.91.2024.08.21.07.53.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 07:53:24 -0700 (PDT) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com, npratte@iol.unh.edu, dmarx@iol.unh.edu, alex.chapman@arm.com Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [PATCH v3 06/12] dst: add basic capability support Date: Wed, 21 Aug 2024 16:53:09 +0200 Message-Id: <20240821145315.97974-7-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240821145315.97974-1-juraj.linkes@pantheon.tech> References: <20240301155416.96960-1-juraj.linkes@pantheon.tech> <20240821145315.97974-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org A test case or suite may require certain capabilities to be present in the tested environment. Add the basic infrastructure for checking the support status of capabilities: * The Capability ABC defining the common capability API * Extension of the TestProtocol with required capabilities (each test suite or case stores the capabilities it requires) * Integration with the runner which calls the new APIs to get which capabilities are supported. Signed-off-by: Juraj Linkeš --- dts/framework/runner.py | 26 +++++ dts/framework/test_result.py | 38 ++++++- dts/framework/test_suite.py | 1 + dts/framework/testbed_model/capability.py | 117 +++++++++++++++++++++- 4 files changed, 179 insertions(+), 3 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index 48ae9cc215..43bb2bc830 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -25,6 +25,7 @@ from types import MethodType from typing import Iterable +from framework.testbed_model.capability import Capability, get_supported_capabilities from framework.testbed_model.sut_node import SutNode from framework.testbed_model.tg_node import TGNode @@ -452,6 +453,21 @@ def _run_build_target( self._logger.exception("Build target teardown failed.") build_target_result.update_teardown(Result.FAIL, e) + def _get_supported_capabilities( + self, + sut_node: SutNode, + topology_config: Topology, + test_suites_with_cases: Iterable[TestSuiteWithCases], + ) -> set[Capability]: + + capabilities_to_check = set() + for test_suite_with_cases in test_suites_with_cases: + capabilities_to_check.update(test_suite_with_cases.required_capabilities) + + self._logger.debug(f"Found capabilities to check: {capabilities_to_check}") + + return get_supported_capabilities(sut_node, topology_config, capabilities_to_check) + def _run_test_suites( self, sut_node: SutNode, @@ -464,6 +480,12 @@ def _run_test_suites( The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. + Before running any suites, the method determines whether they should be skipped + by inspecting any required capabilities the test suite needs and comparing those + to capabilities supported by the tested environment. If all capabilities are supported, + the suite is run. If all test cases in a test suite would be skipped, the whole test suite + is skipped (the setup and teardown is not run). + If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites in the current build target won't be executed. @@ -476,7 +498,11 @@ def _run_test_suites( """ end_build_target = False topology = Topology(sut_node.ports, tg_node.ports) + supported_capabilities = self._get_supported_capabilities( + sut_node, topology, test_suites_with_cases + ) for test_suite_with_cases in test_suites_with_cases: + test_suite_with_cases.mark_skip_unsupported(supported_capabilities) test_suite_result = build_target_result.add_test_suite(test_suite_with_cases) try: if not test_suite_with_cases.skip: diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index 306b100bc6..b4b58ef348 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -25,10 +25,12 @@ import os.path from collections.abc import MutableSequence -from dataclasses import dataclass +from dataclasses import dataclass, field from enum import Enum, auto from typing import Union +from framework.testbed_model.capability import Capability + from .config import ( OS, Architecture, @@ -63,6 +65,12 @@ class is to hold a subset of test cases (which could be all test cases) because test_suite_class: type[TestSuite] test_cases: list[type[TestCase]] + required_capabilities: set[Capability] = field(default_factory=set, init=False) + + def __post_init__(self): + """Gather the required capabilities of the test suite and all test cases.""" + for test_object in [self.test_suite_class] + self.test_cases: + self.required_capabilities.update(test_object.required_capabilities) def create_config(self) -> TestSuiteConfig: """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases. @@ -75,6 +83,34 @@ def create_config(self) -> TestSuiteConfig: test_cases=[test_case.__name__ for test_case in self.test_cases], ) + def mark_skip_unsupported(self, supported_capabilities: set[Capability]) -> None: + """Mark the test suite and test cases to be skipped. + + The mark is applied if object to be skipped requires any capabilities and at least one of + them is not among `supported_capabilities`. + + Args: + supported_capabilities: The supported capabilities. + """ + for test_object in [self.test_suite_class, *self.test_cases]: + capabilities_not_supported = test_object.required_capabilities - supported_capabilities + if capabilities_not_supported: + test_object.skip = True + capability_str = ( + "capability" if len(capabilities_not_supported) == 1 else "capabilities" + ) + test_object.skip_reason = ( + f"Required {capability_str} '{capabilities_not_supported}' not found." + ) + if not self.test_suite_class.skip: + if all(test_case.skip for test_case in self.test_cases): + self.test_suite_class.skip = True + + self.test_suite_class.skip_reason = ( + "All test cases are marked to be skipped with reasons: " + f"{' '.join(test_case.skip_reason for test_case in self.test_cases)}" + ) + @property def skip(self) -> bool: """Skip the test suite if all test cases or the suite itself are to be skipped. diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index 56f153bda6..5c393ce8bf 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -467,6 +467,7 @@ def _decorator(func: TestSuiteMethodType) -> type[TestCase]: test_case = cast(type[TestCase], func) test_case.skip = cls.skip test_case.skip_reason = cls.skip_reason + test_case.required_capabilities = set() test_case.test_type = test_case_type return test_case diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py index 662f691a0e..8899f07f76 100644 --- a/dts/framework/testbed_model/capability.py +++ b/dts/framework/testbed_model/capability.py @@ -3,11 +3,97 @@ """Testbed capabilities. -This module provides a protocol that defines the common attributes of test cases and suites. +This module provides a protocol that defines the common attributes of test cases and suites +and support for test environment capabilities. """ +from abc import ABC, abstractmethod from collections.abc import Sequence -from typing import ClassVar, Protocol +from typing import Callable, ClassVar, Protocol + +from typing_extensions import Self + +from .sut_node import SutNode +from .topology import Topology + + +class Capability(ABC): + """The base class for various capabilities. + + The same capability should always be represented by the same object, + meaning the same capability required by different test cases or suites + should point to the same object. + + Example: + ``test_case1`` and ``test_case2`` each require ``capability1`` + and in both instances, ``capability1`` should point to the same capability object. + + It is up to the subclasses how they implement this. + + The instances are used in sets so they must be hashable. + """ + + #: A set storing the capabilities whose support should be checked. + capabilities_to_check: ClassVar[set[Self]] = set() + + def register_to_check(self) -> Callable[[SutNode, "Topology"], set[Self]]: + """Register the capability to be checked for support. + + Returns: + The callback function that checks the support of capabilities of the particular subclass + which should be called after all capabilities have been registered. + """ + if not type(self).capabilities_to_check: + type(self).capabilities_to_check = set() + type(self).capabilities_to_check.add(self) + return type(self)._get_and_reset + + def add_to_required(self, test_case_or_suite: type["TestProtocol"]) -> None: + """Add the capability instance to the required test case or suite's capabilities. + + Args: + test_case_or_suite: The test case or suite among whose required capabilities + to add this instance. + """ + if not test_case_or_suite.required_capabilities: + test_case_or_suite.required_capabilities = set() + self._preprocess_required(test_case_or_suite) + test_case_or_suite.required_capabilities.add(self) + + def _preprocess_required(self, test_case_or_suite: type["TestProtocol"]) -> None: + """An optional method that modifies the required capabilities.""" + + @classmethod + def _get_and_reset(cls, sut_node: SutNode, topology: "Topology") -> set[Self]: + """The callback method to be called after all capabilities have been registered. + + Not only does this method check the support of capabilities, + but it also reset the internal set of registered capabilities + so that the "register, then get support" workflow works in subsequent test runs. + """ + supported_capabilities = cls.get_supported_capabilities(sut_node, topology) + cls.capabilities_to_check = set() + return supported_capabilities + + @classmethod + @abstractmethod + def get_supported_capabilities(cls, sut_node: SutNode, topology: "Topology") -> set[Self]: + """Get the support status of each registered capability. + + Each subclass must implement this method and return the subset of supported capabilities + of :attr:`capabilities_to_check`. + + Args: + sut_node: The SUT node of the current test run. + topology: The topology of the current test run. + + Returns: + The supported capabilities. + """ + + @abstractmethod + def __hash__(self) -> int: + """The subclasses must be hashable so that they can be stored in sets.""" class TestProtocol(Protocol): @@ -17,6 +103,8 @@ class TestProtocol(Protocol): skip: ClassVar[bool] = False #: The reason for skipping the test case or suite. skip_reason: ClassVar[str] = "" + #: The capabilities the test case or suite requires in order to be executed. + required_capabilities: ClassVar[set[Capability]] = set() @classmethod def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple[set, set]: @@ -26,3 +114,28 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple NotImplementedError: The subclass does not implement the method. """ raise NotImplementedError() + + +def get_supported_capabilities( + sut_node: SutNode, + topology_config: Topology, + capabilities_to_check: set[Capability], +) -> set[Capability]: + """Probe the environment for `capabilities_to_check` and return the supported ones. + + Args: + sut_node: The SUT node to check for capabilities. + topology_config: The topology config to check for capabilities. + capabilities_to_check: The capabilities to check. + + Returns: + The capabilities supported by the environment. + """ + callbacks = set() + for capability_to_check in capabilities_to_check: + callbacks.add(capability_to_check.register_to_check()) + supported_capabilities = set() + for callback in callbacks: + supported_capabilities.update(callback(sut_node, topology_config)) + + return supported_capabilities -- 2.34.1