From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D28C45A11; Mon, 23 Sep 2024 17:03:03 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 55B6C40608; Mon, 23 Sep 2024 17:02:23 +0200 (CEST) Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by mails.dpdk.org (Postfix) with ESMTP id 69D70402E6 for ; Mon, 23 Sep 2024 17:02:20 +0200 (CEST) Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a8b155b5e9eso663834066b.1 for ; Mon, 23 Sep 2024 08:02:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1727103740; x=1727708540; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e4WCH3XBNEZ6Fwtthi/qfYKJ9n58UGSFQpMXcPdtQ8E=; b=s9tb+gQ7vEu+MVPtY4JPDLThSMywOPsgiLBZZCfrCarHvppVRgWzdzabDg+jMH+6ho dKMIQAXAfGtFDryq9Qrno4v07en3EDcr4tau9+zNhtI019BoXB8hS2SLrrmMPBTls2dw opLjtrHKAyNX2MNaq07PUppwwZwOIFonk2LZlXFX6pp4NnzKbavvlL5K1C2P4oUMsix0 0JtITVUCTXz0mbRUQeMUssm7D53Ek1H/uVJaAWxhXuo3yYHOETkd4JZLgQPsusWzvQDR uenJj+Y/PGnAj/E7MLRPDFfxBuzk5YfMNAr+jLOKNbeIodqitXwGHoJzdsIk+eYp61oS 6eRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727103740; x=1727708540; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e4WCH3XBNEZ6Fwtthi/qfYKJ9n58UGSFQpMXcPdtQ8E=; b=eEd2YO0wnO/7nriEDKUqKYAYDlxQx3ElRg/9LTaVmExR0iZ60PQOMkCJ4106QnNY0p GcbH0uGf8xQp68WkzZr8DSDymR8uo0SeNcRrP8gvg7KINty/rC20Rh9EPB+OrAxEj0ql mYEVB7PdA+zLPu9IMCy1MK6o1+ltep9FKBe/vZmgbM5EAi5RiwOc5EHlIKA4uDrHXL04 /YmJq2cS+unT9ZpL2RiHpm1B81OeMWGVXVc9SWVuiFJw2e5uRu88o0Tya/drdHeAWJVI n/6Jfjg1EHE54gx6XiDPP+5Ywc+ppsEsBLBY3srJQeGdU4OGSDTaE+QGES3EhCZUddnD QRRw== X-Gm-Message-State: AOJu0Yw96kpUNOxYwYDxsnA0TFrvkJSTbfvpke+YkJBfcznAxB0Klzsm rSixSDHEc8RKKmJj+7SpYV1rqZNRtIJOxISrEu9sFc3D+DjRpnAalZ/FwXlQAGbs8Fnc5HObfPO jt8Q= X-Google-Smtp-Source: AGHT+IHWQto+dSYRBHrH/nMxEl0rDM85HwDNrdzbRjmNoxleEGrCP35/64nFvcCe2snb1rDY4+WwFQ== X-Received: by 2002:a17:907:f704:b0:a86:80ef:4fe5 with SMTP id a640c23a62f3a-a90d50ee84bmr1116159166b.47.1727103739717; Mon, 23 Sep 2024 08:02:19 -0700 (PDT) Received: from jlinkes-PT-Latitude-5530.. ([84.245.121.62]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a90612b3ba7sm1235318866b.119.2024.09.23.08.02.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 08:02:18 -0700 (PDT) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, paul.szczepanek@arm.com, Luca.Vizzarro@arm.com, alex.chapman@arm.com, probb@iol.unh.edu, jspewock@iol.unh.edu, npratte@iol.unh.edu, dmarx@iol.unh.edu Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [PATCH v4 05/11] dts: add basic capability support Date: Mon, 23 Sep 2024 17:02:04 +0200 Message-ID: <20240923150210.57269-5-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240923150210.57269-1-juraj.linkes@pantheon.tech> References: <20240301155416.96960-1-juraj.linkes@pantheon.tech> <20240923150210.57269-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org A test case or suite may require certain capabilities to be present in the tested environment. Add the basic infrastructure for checking the support status of capabilities: * The Capability ABC defining the common capability API * Extension of the TestProtocol with required capabilities (each test suite or case stores the capabilities it requires) * Integration with the runner which calls the new APIs to get which capabilities are supported. Signed-off-by: Juraj Linkeš Reviewed-by: Jeremy Spewock Reviewed-by: Dean Marx --- dts/framework/runner.py | 26 +++++ dts/framework/test_result.py | 40 +++++++- dts/framework/test_suite.py | 1 + dts/framework/testbed_model/capability.py | 117 +++++++++++++++++++++- 4 files changed, 181 insertions(+), 3 deletions(-) diff --git a/dts/framework/runner.py b/dts/framework/runner.py index bf2bfa156f..fff7f1c0a1 100644 --- a/dts/framework/runner.py +++ b/dts/framework/runner.py @@ -26,6 +26,7 @@ from types import MethodType from typing import Iterable +from framework.testbed_model.capability import Capability, get_supported_capabilities from framework.testbed_model.sut_node import SutNode from framework.testbed_model.tg_node import TGNode @@ -454,6 +455,21 @@ def _run_build_target( self._logger.exception("Build target teardown failed.") build_target_result.update_teardown(Result.FAIL, e) + def _get_supported_capabilities( + self, + sut_node: SutNode, + topology_config: Topology, + test_suites_with_cases: Iterable[TestSuiteWithCases], + ) -> set[Capability]: + + capabilities_to_check = set() + for test_suite_with_cases in test_suites_with_cases: + capabilities_to_check.update(test_suite_with_cases.required_capabilities) + + self._logger.debug(f"Found capabilities to check: {capabilities_to_check}") + + return get_supported_capabilities(sut_node, topology_config, capabilities_to_check) + def _run_test_suites( self, sut_node: SutNode, @@ -466,6 +482,12 @@ def _run_test_suites( The method assumes the build target we're testing has already been built on the SUT node. The current build target thus corresponds to the current DPDK build present on the SUT node. + Before running any suites, the method determines whether they should be skipped + by inspecting any required capabilities the test suite needs and comparing those + to capabilities supported by the tested environment. If all capabilities are supported, + the suite is run. If all test cases in a test suite would be skipped, the whole test suite + is skipped (the setup and teardown is not run). + If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites in the current build target won't be executed. @@ -478,7 +500,11 @@ def _run_test_suites( """ end_build_target = False topology = Topology(sut_node.ports, tg_node.ports) + supported_capabilities = self._get_supported_capabilities( + sut_node, topology, test_suites_with_cases + ) for test_suite_with_cases in test_suites_with_cases: + test_suite_with_cases.mark_skip_unsupported(supported_capabilities) test_suite_result = build_target_result.add_test_suite(test_suite_with_cases) try: if not test_suite_with_cases.skip: diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py index 50633eee4e..40f3fbb77e 100644 --- a/dts/framework/test_result.py +++ b/dts/framework/test_result.py @@ -25,10 +25,12 @@ import os.path from collections.abc import MutableSequence -from dataclasses import dataclass +from dataclasses import dataclass, field from enum import Enum, auto from typing import Union +from framework.testbed_model.capability import Capability + from .config import ( OS, Architecture, @@ -59,10 +61,18 @@ class is to hold a subset of test cases (which could be all test cases) because Attributes: test_suite_class: The test suite class. test_cases: The test case methods. + required_capabilities: The combined required capabilities of both the test suite + and the subset of test cases. """ test_suite_class: type[TestSuite] test_cases: list[type[TestCase]] + required_capabilities: set[Capability] = field(default_factory=set, init=False) + + def __post_init__(self): + """Gather the required capabilities of the test suite and all test cases.""" + for test_object in [self.test_suite_class] + self.test_cases: + self.required_capabilities.update(test_object.required_capabilities) def create_config(self) -> TestSuiteConfig: """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases. @@ -75,6 +85,34 @@ def create_config(self) -> TestSuiteConfig: test_cases=[test_case.__name__ for test_case in self.test_cases], ) + def mark_skip_unsupported(self, supported_capabilities: set[Capability]) -> None: + """Mark the test suite and test cases to be skipped. + + The mark is applied if object to be skipped requires any capabilities and at least one of + them is not among `supported_capabilities`. + + Args: + supported_capabilities: The supported capabilities. + """ + for test_object in [self.test_suite_class, *self.test_cases]: + capabilities_not_supported = test_object.required_capabilities - supported_capabilities + if capabilities_not_supported: + test_object.skip = True + capability_str = ( + "capability" if len(capabilities_not_supported) == 1 else "capabilities" + ) + test_object.skip_reason = ( + f"Required {capability_str} '{capabilities_not_supported}' not found." + ) + if not self.test_suite_class.skip: + if all(test_case.skip for test_case in self.test_cases): + self.test_suite_class.skip = True + + self.test_suite_class.skip_reason = ( + "All test cases are marked to be skipped with reasons: " + f"{' '.join(test_case.skip_reason for test_case in self.test_cases)}" + ) + @property def skip(self) -> bool: """Skip the test suite if all test cases or the suite itself are to be skipped. diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index 367203f67e..bdc57a6339 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -527,6 +527,7 @@ def _decorator(func: TestSuiteMethodType) -> type[TestCase]: test_case = cast(type[TestCase], func) test_case.skip = cls.skip test_case.skip_reason = cls.skip_reason + test_case.required_capabilities = set() test_case.test_type = test_case_type return test_case diff --git a/dts/framework/testbed_model/capability.py b/dts/framework/testbed_model/capability.py index 662f691a0e..8899f07f76 100644 --- a/dts/framework/testbed_model/capability.py +++ b/dts/framework/testbed_model/capability.py @@ -3,11 +3,97 @@ """Testbed capabilities. -This module provides a protocol that defines the common attributes of test cases and suites. +This module provides a protocol that defines the common attributes of test cases and suites +and support for test environment capabilities. """ +from abc import ABC, abstractmethod from collections.abc import Sequence -from typing import ClassVar, Protocol +from typing import Callable, ClassVar, Protocol + +from typing_extensions import Self + +from .sut_node import SutNode +from .topology import Topology + + +class Capability(ABC): + """The base class for various capabilities. + + The same capability should always be represented by the same object, + meaning the same capability required by different test cases or suites + should point to the same object. + + Example: + ``test_case1`` and ``test_case2`` each require ``capability1`` + and in both instances, ``capability1`` should point to the same capability object. + + It is up to the subclasses how they implement this. + + The instances are used in sets so they must be hashable. + """ + + #: A set storing the capabilities whose support should be checked. + capabilities_to_check: ClassVar[set[Self]] = set() + + def register_to_check(self) -> Callable[[SutNode, "Topology"], set[Self]]: + """Register the capability to be checked for support. + + Returns: + The callback function that checks the support of capabilities of the particular subclass + which should be called after all capabilities have been registered. + """ + if not type(self).capabilities_to_check: + type(self).capabilities_to_check = set() + type(self).capabilities_to_check.add(self) + return type(self)._get_and_reset + + def add_to_required(self, test_case_or_suite: type["TestProtocol"]) -> None: + """Add the capability instance to the required test case or suite's capabilities. + + Args: + test_case_or_suite: The test case or suite among whose required capabilities + to add this instance. + """ + if not test_case_or_suite.required_capabilities: + test_case_or_suite.required_capabilities = set() + self._preprocess_required(test_case_or_suite) + test_case_or_suite.required_capabilities.add(self) + + def _preprocess_required(self, test_case_or_suite: type["TestProtocol"]) -> None: + """An optional method that modifies the required capabilities.""" + + @classmethod + def _get_and_reset(cls, sut_node: SutNode, topology: "Topology") -> set[Self]: + """The callback method to be called after all capabilities have been registered. + + Not only does this method check the support of capabilities, + but it also reset the internal set of registered capabilities + so that the "register, then get support" workflow works in subsequent test runs. + """ + supported_capabilities = cls.get_supported_capabilities(sut_node, topology) + cls.capabilities_to_check = set() + return supported_capabilities + + @classmethod + @abstractmethod + def get_supported_capabilities(cls, sut_node: SutNode, topology: "Topology") -> set[Self]: + """Get the support status of each registered capability. + + Each subclass must implement this method and return the subset of supported capabilities + of :attr:`capabilities_to_check`. + + Args: + sut_node: The SUT node of the current test run. + topology: The topology of the current test run. + + Returns: + The supported capabilities. + """ + + @abstractmethod + def __hash__(self) -> int: + """The subclasses must be hashable so that they can be stored in sets.""" class TestProtocol(Protocol): @@ -17,6 +103,8 @@ class TestProtocol(Protocol): skip: ClassVar[bool] = False #: The reason for skipping the test case or suite. skip_reason: ClassVar[str] = "" + #: The capabilities the test case or suite requires in order to be executed. + required_capabilities: ClassVar[set[Capability]] = set() @classmethod def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple[set, set]: @@ -26,3 +114,28 @@ def get_test_cases(cls, test_case_sublist: Sequence[str] | None = None) -> tuple NotImplementedError: The subclass does not implement the method. """ raise NotImplementedError() + + +def get_supported_capabilities( + sut_node: SutNode, + topology_config: Topology, + capabilities_to_check: set[Capability], +) -> set[Capability]: + """Probe the environment for `capabilities_to_check` and return the supported ones. + + Args: + sut_node: The SUT node to check for capabilities. + topology_config: The topology config to check for capabilities. + capabilities_to_check: The capabilities to check. + + Returns: + The capabilities supported by the environment. + """ + callbacks = set() + for capability_to_check in capabilities_to_check: + callbacks.add(capability_to_check.register_to_check()) + supported_capabilities = set() + for callback in callbacks: + supported_capabilities.update(callback(sut_node, topology_config)) + + return supported_capabilities -- 2.43.0