* [PATCH 0/7] DTS external DPDK build
@ 2024-09-27 16:08 Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 1/7] dts: rename build target to " Tomáš Ďurovec
` (6 more replies)
0 siblings, 7 replies; 9+ messages in thread
From: Tomáš Ďurovec @ 2024-09-27 16:08 UTC (permalink / raw)
To: dev, Luca.Vizzarro, probb, npratte, dmarx; +Cc: Tomáš Ďurovec
This patch series adds ability to use pre-build DPDK in various options of usage. User can specify
this option from config file or cmd arguments/environment variables:
* The source of DPDK (tarball or tree),
* The build dir witch will be located in a subdirectory of DPDK tree root directory, otherwise will
be used build configuration from config file,
* Location where the DPDK source will be stored either in SUT node or local filesystem.
Tomáš Ďurovec (7):
dts: rename build target to DPDK build
dts: one dpdk build per test run
dts: fix remote session file transfer vars
dts: add the ability to copy directories via remote
dts: add support for externally compiled DPDK
doc: update argument options for external DPDK build
dts: remove git ref option
doc/guides/tools/dts.rst | 18 +-
dts/conf.yaml | 17 +-
dts/framework/config/__init__.py | 140 ++++++-
dts/framework/config/conf_yaml_schema.json | 72 +++-
dts/framework/config/types.py | 19 +-
dts/framework/exception.py | 4 +-
dts/framework/logger.py | 4 -
dts/framework/remote_session/dpdk_shell.py | 2 +-
.../remote_session/remote_session.py | 24 +-
dts/framework/remote_session/ssh_session.py | 18 +-
dts/framework/runner.py | 139 ++-----
dts/framework/settings.py | 200 +++++++---
dts/framework/test_result.py | 124 ++----
dts/framework/test_suite.py | 2 +-
dts/framework/testbed_model/node.py | 22 +-
dts/framework/testbed_model/os_session.py | 209 ++++++++--
dts/framework/testbed_model/posix_session.py | 141 ++++++-
dts/framework/testbed_model/sut_node.py | 366 ++++++++++++------
dts/framework/utils.py | 164 ++++----
dts/tests/TestSuite_smoke_tests.py | 2 +-
20 files changed, 1078 insertions(+), 609 deletions(-)
--
2.46.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/7] dts: rename build target to DPDK build
2024-09-27 16:08 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
@ 2024-09-27 16:08 ` Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 2/7] dts: one dpdk build per test run Tomáš Ďurovec
` (5 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Tomáš Ďurovec @ 2024-09-27 16:08 UTC (permalink / raw)
To: dev, Luca.Vizzarro, probb, npratte, dmarx; +Cc: Tomáš Ďurovec
Since the DPDK may already be built, some more general name
is needed that includes both the DPDK location and the build
config (if we are going to build).
Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
---
dts/conf.yaml | 2 +-
dts/framework/config/__init__.py | 26 ++---
dts/framework/config/conf_yaml_schema.json | 10 +-
dts/framework/config/types.py | 4 +-
dts/framework/logger.py | 4 +-
dts/framework/runner.py | 112 ++++++++++-----------
dts/framework/settings.py | 2 +-
dts/framework/test_result.py | 72 +++++++------
dts/framework/test_suite.py | 2 +-
dts/framework/testbed_model/sut_node.py | 55 +++++-----
dts/tests/TestSuite_smoke_tests.py | 2 +-
11 files changed, 142 insertions(+), 149 deletions(-)
diff --git a/dts/conf.yaml b/dts/conf.yaml
index ca5e87636e..1363e93580 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -4,7 +4,7 @@
test_runs:
# define one test run environment
- - build_targets:
+ - dpdk_builds:
- arch: x86_64
os: linux
cpu: native
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 269d9ec318..c243716010 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -45,8 +45,8 @@
from typing_extensions import Self
from framework.config.types import (
- BuildTargetConfigDict,
ConfigurationDict,
+ DPDKBuildConfigDict,
NodeConfigDict,
PortConfigDict,
TestRunConfigDict,
@@ -335,7 +335,7 @@ class NodeInfo:
@dataclass(slots=True, frozen=True)
-class BuildTargetConfiguration:
+class DPDKBuildConfiguration:
"""DPDK build configuration.
The configuration used for building DPDK.
@@ -358,7 +358,7 @@ class BuildTargetConfiguration:
name: str
@classmethod
- def from_dict(cls, d: BuildTargetConfigDict) -> Self:
+ def from_dict(cls, d: DPDKBuildConfigDict) -> Self:
r"""A convenience method that processes the inputs before creating an instance.
`arch`, `os`, `cpu` and `compiler` are converted to :class:`Enum`\s and
@@ -368,7 +368,7 @@ def from_dict(cls, d: BuildTargetConfigDict) -> Self:
d: The configuration dictionary.
Returns:
- The build target configuration instance.
+ The DPDK build configuration instance.
"""
return cls(
arch=Architecture(d["arch"]),
@@ -381,8 +381,8 @@ def from_dict(cls, d: BuildTargetConfigDict) -> Self:
@dataclass(slots=True, frozen=True)
-class BuildTargetInfo:
- """Various versions and other information about a build target.
+class DPDKBuildInfo:
+ """Various versions and other information about a DPDK build.
Attributes:
dpdk_version: The DPDK version that was built.
@@ -437,7 +437,7 @@ class TestRunConfiguration:
and with what DPDK build.
Attributes:
- build_targets: A list of DPDK builds to test.
+ dpdk_builds: A list of DPDK builds to test.
perf: Whether to run performance tests.
func: Whether to run functional tests.
skip_smoke_tests: Whether to skip smoke tests.
@@ -448,7 +448,7 @@ class TestRunConfiguration:
random_seed: The seed to use for pseudo-random generation.
"""
- build_targets: list[BuildTargetConfiguration]
+ dpdk_builds: list[DPDKBuildConfiguration]
perf: bool
func: bool
skip_smoke_tests: bool
@@ -466,7 +466,7 @@ def from_dict(
) -> Self:
"""A convenience method that processes the inputs before creating an instance.
- The build target and the test suite config are transformed into their respective objects.
+ The DPDK build and the test suite config are transformed into their respective objects.
SUT and TG configurations are taken from `node_map`. The other (:class:`bool`) attributes
are just stored.
@@ -477,8 +477,8 @@ def from_dict(
Returns:
The test run configuration instance.
"""
- build_targets: list[BuildTargetConfiguration] = list(
- map(BuildTargetConfiguration.from_dict, d["build_targets"])
+ dpdk_builds: list[DPDKBuildConfiguration] = list(
+ map(DPDKBuildConfiguration.from_dict, d["dpdk_builds"])
)
test_suites: list[TestSuiteConfig] = list(map(TestSuiteConfig.from_dict, d["test_suites"]))
sut_name = d["system_under_test_node"]["node_name"]
@@ -501,7 +501,7 @@ def from_dict(
)
random_seed = d.get("random_seed", None)
return cls(
- build_targets=build_targets,
+ dpdk_builds=dpdk_builds,
perf=d["perf"],
func=d["func"],
skip_smoke_tests=skip_smoke_tests,
@@ -552,7 +552,7 @@ class Configuration:
def from_dict(cls, d: ConfigurationDict) -> Self:
"""A convenience method that processes the inputs before creating an instance.
- Build target and test suite config are transformed into their respective objects.
+ DPDK build and test suite config are transformed into their respective objects.
SUT and TG configurations are taken from `node_map`. The other (:class:`bool`) attributes
are just stored.
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index df390e8ae2..927a73ac6c 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -110,9 +110,9 @@
"mscv"
]
},
- "build_target": {
+ "dpdk_build": {
"type": "object",
- "description": "Targets supported by DTS",
+ "description": "DPDK build configuration supported by DTS.",
"properties": {
"arch": {
"type": "string",
@@ -327,10 +327,10 @@
"items": {
"type": "object",
"properties": {
- "build_targets": {
+ "dpdk_builds": {
"type": "array",
"items": {
- "$ref": "#/definitions/build_target"
+ "$ref": "#/definitions/dpdk_build"
},
"minimum": 1
},
@@ -387,7 +387,7 @@
},
"additionalProperties": false,
"required": [
- "build_targets",
+ "dpdk_builds",
"perf",
"func",
"test_suites",
diff --git a/dts/framework/config/types.py b/dts/framework/config/types.py
index ce7b784ac8..4f450267d1 100644
--- a/dts/framework/config/types.py
+++ b/dts/framework/config/types.py
@@ -71,7 +71,7 @@ class NodeConfigDict(TypedDict):
traffic_generator: TrafficGeneratorConfigDict
-class BuildTargetConfigDict(TypedDict):
+class DPDKBuildConfigDict(TypedDict):
"""Allowed keys and values."""
#:
@@ -108,7 +108,7 @@ class TestRunConfigDict(TypedDict):
"""Allowed keys and values."""
#:
- build_targets: list[BuildTargetConfigDict]
+ dpdk_builds: list[DPDKBuildConfigDict]
#:
perf: bool
#:
diff --git a/dts/framework/logger.py b/dts/framework/logger.py
index 9420323d38..3fbe618219 100644
--- a/dts/framework/logger.py
+++ b/dts/framework/logger.py
@@ -33,7 +33,7 @@ class DtsStage(StrEnum):
#:
test_run_setup = auto()
#:
- build_target_setup = auto()
+ dpdk_build_setup = auto()
#:
test_suite_setup = auto()
#:
@@ -41,7 +41,7 @@ class DtsStage(StrEnum):
#:
test_suite_teardown = auto()
#:
- build_target_teardown = auto()
+ dpdk_build_teardown = auto()
#:
test_run_teardown = auto()
#:
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index ab98de8353..d63b1821e7 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -8,11 +8,11 @@
The module is responsible for running DTS in a series of stages:
#. Test run stage,
- #. Build target stage,
+ #. DPDK build stage,
#. Test suite stage,
#. Test case stage.
-The test run and build target stages set up the environment before running test suites.
+The test run and DPDK build stages set up the environment before running test suites.
The test suite stage sets up steps common to all test cases
and the test case stage runs test cases individually.
"""
@@ -31,8 +31,8 @@
from framework.testbed_model.tg_node import TGNode
from .config import (
- BuildTargetConfiguration,
Configuration,
+ DPDKBuildConfiguration,
TestRunConfiguration,
TestSuiteConfig,
load_config,
@@ -46,7 +46,7 @@
from .logger import DTSLogger, DtsStage, get_dts_logger
from .settings import SETTINGS
from .test_result import (
- BuildTargetResult,
+ DPDKBuildResult,
DTSResult,
Result,
TestCaseResult,
@@ -70,9 +70,9 @@ class DTSRunner:
:class:`~.framework.exception.DTSError`\s.
Example:
- An error occurs in a build target setup. The current build target is aborted,
+ An error occurs in a DPDK build setup. The current DPDK build is aborted,
all test suites and their test cases are marked as blocked and the run continues
- with the next build target. If the errored build target was the last one in the
+ with the next DPDK build. If the errored DPDK build was the last one in the
given test run, the next test run begins.
"""
@@ -98,16 +98,16 @@ def __init__(self):
self._perf_test_case_regex = r"test_perf_"
def run(self) -> None:
- """Run all build targets in all test runs from the test run configuration.
+ """Run all DPDK build in all test runs from the test run configuration.
- Before running test suites, test runs and build targets are first set up.
- The test runs and build targets defined in the test run configuration are iterated over.
- The test runs define which tests to run and where to run them and build targets define
+ Before running test suites, test runs and DPDK builds are first set up.
+ The test runs and DPDK builds defined in the test run configuration are iterated over.
+ The test runs define which tests to run and where to run them and DPDK builds define
the DPDK build setup.
- The tests suites are set up for each test run/build target tuple and each discovered
+ The tests suites are set up for each test run/DPDK build tuple and each discovered
test case within the test suite is set up, executed and torn down. After all test cases
- have been executed, the test suite is torn down and the next build target will be tested.
+ have been executed, the test suite is torn down and the next DPDK build will be tested.
In order to properly mark test suites and test cases as blocked in case of a failure,
we need to have discovered which test suites and test cases to run before any failures
@@ -117,7 +117,7 @@ def run(self) -> None:
#. Test run setup
- #. Build target setup
+ #. DPDK build setup
#. Test suite setup
@@ -127,7 +127,7 @@ def run(self) -> None:
#. Test suite teardown
- #. Build target teardown
+ #. DPDK build teardown
#. Test run teardown
@@ -416,7 +416,7 @@ def _run_test_run(
) -> None:
"""Run the given test run.
- This involves running the test run setup as well as running all build targets
+ This involves running the test run setup as well as running all DPDK builds
in the given test run. After that, the test run teardown is run.
Args:
@@ -439,13 +439,13 @@ def _run_test_run(
test_run_result.update_setup(Result.FAIL, e)
else:
- for build_target_config in test_run_config.build_targets:
- build_target_result = test_run_result.add_build_target(build_target_config)
- self._run_build_target(
+ for dpdk_build_config in test_run_config.dpdk_builds:
+ dpdk_build_result = test_run_result.add_dpdk_build(dpdk_build_config)
+ self._run_dpdk_build(
sut_node,
tg_node,
- build_target_config,
- build_target_result,
+ dpdk_build_config,
+ dpdk_build_result,
test_suites_with_cases,
)
@@ -459,88 +459,87 @@ def _run_test_run(
self._logger.exception("Test run teardown failed.")
test_run_result.update_teardown(Result.FAIL, e)
- def _run_build_target(
+ def _run_dpdk_build(
self,
sut_node: SutNode,
tg_node: TGNode,
- build_target_config: BuildTargetConfiguration,
- build_target_result: BuildTargetResult,
+ dpdk_build_config: DPDKBuildConfiguration,
+ dpdk_build_result: DPDKBuildResult,
test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
- """Run the given build target.
+ """Run the given DPDK build.
- This involves running the build target setup as well as running all test suites
- of the build target's test run.
- After that, build target teardown is run.
+ This involves running the DPDK build setup as well as running all test suites
+ of the DPDK build's test run.
+ After that, DPDK build teardown is run.
Args:
sut_node: The test run's sut node.
tg_node: The test run's tg node.
- build_target_config: A build target's test run configuration.
- build_target_result: The build target level result object associated
- with the current build target.
+ dpdk_build_config: A DPDK build's test run configuration.
+ dpdk_build_result: The DPDK build level result object associated
+ with the current DPDK build.
test_suites_with_cases: The test suites with test cases to run.
"""
- self._logger.set_stage(DtsStage.build_target_setup)
- self._logger.info(f"Running build target '{build_target_config.name}'.")
+ self._logger.set_stage(DtsStage.dpdk_build_setup)
+ self._logger.info(f"Running DPDK build '{dpdk_build_config.name}'.")
try:
- sut_node.set_up_build_target(build_target_config)
+ sut_node.set_up_dpdk(dpdk_build_config)
self._result.dpdk_version = sut_node.dpdk_version
- build_target_result.add_build_target_info(sut_node.get_build_target_info())
- build_target_result.update_setup(Result.PASS)
+ dpdk_build_result.add_dpdk_build_info(sut_node.get_dpdk_build_info())
+ dpdk_build_result.update_setup(Result.PASS)
except Exception as e:
- self._logger.exception("Build target setup failed.")
- build_target_result.update_setup(Result.FAIL, e)
+ self._logger.exception("DPDK build setup failed.")
+ dpdk_build_result.update_setup(Result.FAIL, e)
else:
- self._run_test_suites(sut_node, tg_node, build_target_result, test_suites_with_cases)
+ self._run_test_suites(sut_node, tg_node, dpdk_build_result, test_suites_with_cases)
finally:
try:
- self._logger.set_stage(DtsStage.build_target_teardown)
- sut_node.tear_down_build_target()
- build_target_result.update_teardown(Result.PASS)
+ self._logger.set_stage(DtsStage.dpdk_build_teardown)
+ sut_node.tear_down_dpdk()
+ dpdk_build_result.update_teardown(Result.PASS)
except Exception as e:
- self._logger.exception("Build target teardown failed.")
- build_target_result.update_teardown(Result.FAIL, e)
+ self._logger.exception("DPDK build teardown failed.")
+ dpdk_build_result.update_teardown(Result.FAIL, e)
def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
- build_target_result: BuildTargetResult,
+ dpdk_build_result: DPDKBuildResult,
test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
- """Run `test_suites_with_cases` with the current build target.
+ """Run `test_suites_with_cases` with the current DPDK build.
- The method assumes the build target we're testing has already been built on the SUT node.
- The current build target thus corresponds to the current DPDK build present on the SUT node.
+ The method assumes the DPDK we're testing has already been built on the SUT node.
If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
- in the current build target won't be executed.
+ in the current DPDK build won't be executed.
Args:
sut_node: The test run's SUT node.
tg_node: The test run's TG node.
- build_target_result: The build target level result object associated
- with the current build target.
+ dpdk_build_result: The DPDK build level result object associated
+ with the current DPDK build.
test_suites_with_cases: The test suites with test cases to run.
"""
- end_build_target = False
+ end_dpdk_build = False
for test_suite_with_cases in test_suites_with_cases:
- test_suite_result = build_target_result.add_test_suite(test_suite_with_cases)
+ test_suite_result = dpdk_build_result.add_test_suite(test_suite_with_cases)
try:
self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
except BlockingTestSuiteError as e:
self._logger.exception(
f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
- "Skipping build target..."
+ "Skipping DPDK build ..."
)
self._result.add_error(e)
- end_build_target = True
+ end_dpdk_build = True
# if a blocking test failed and we need to bail out of suite executions
- if end_build_target:
+ if end_dpdk_build:
break
def _run_test_suite(
@@ -552,8 +551,7 @@ def _run_test_suite(
) -> None:
"""Set up, execute and tear down `test_suite_with_cases`.
- The method assumes the build target we're testing has already been built on the SUT node.
- The current build target thus corresponds to the current DPDK build present on the SUT node.
+ The method assumes the DPDK we're testing has already been built on the SUT node.
Test suite execution consists of running the discovered test cases.
A test case run consists of setup, execution and teardown of said test case.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 7744e37f54..52a1582d5c 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -278,7 +278,7 @@ def _get_parser() -> _DTSArgumentParser:
"--config-file",
default=SETTINGS.config_file_path,
type=Path,
- help="The configuration file that describes the test cases, SUTs and targets.",
+ help="The configuration file that describes the test cases, SUTs and DPDK build configs.",
metavar="FILE_PATH",
dest="config_file_path",
)
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 5694a2482b..95788b7d2e 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -8,12 +8,12 @@
* :class:`DTSResult` contains
* :class:`TestRunResult` contains
- * :class:`BuildTargetResult` contains
+ * :class:`DPDKBuildResult` contains
* :class:`TestSuiteResult` contains
* :class:`TestCaseResult`
Each result may contain multiple lower level results, e.g. there are multiple
-:class:`TestSuiteResult`\s in a :class:`BuildTargetResult`.
+:class:`TestSuiteResult`\s in a :class:`DPDKBuildResult`.
The results have common parts, such as setup and teardown results, captured in :class:`BaseResult`,
which also defines some common behaviors in its methods.
@@ -33,10 +33,10 @@
from .config import (
OS,
Architecture,
- BuildTargetConfiguration,
- BuildTargetInfo,
Compiler,
CPUType,
+ DPDKBuildConfiguration,
+ DPDKBuildInfo,
NodeInfo,
TestRunConfiguration,
TestSuiteConfig,
@@ -138,7 +138,7 @@ class BaseResult:
Stores the results of the setup and teardown portions of the corresponding stage.
The hierarchical nature of DTS results is captured recursively in an internal list.
A stage is each level in this particular hierarchy (pre-run or the top-most level,
- test run, build target, test suite and test case.)
+ test run, DPDK build, test suite and test case.)
Attributes:
setup_result: The result of the setup of the particular stage.
@@ -223,7 +223,7 @@ class DTSResult(BaseResult):
"""Stores environment information and test results from a DTS run.
* Test run level information, such as testbed and the test suite list,
- * Build target level information, such as compiler, target OS and cpu,
+ * DPDK build level information, such as compiler, target OS and cpu,
* Test suite and test case results,
* All errors that are caught and recorded during DTS execution.
@@ -317,7 +317,7 @@ def get_return_code(self) -> int:
class TestRunResult(BaseResult):
"""The test run specific result.
- The internal list stores the results of all build targets in a given test run.
+ The internal list stores the results of all DPDK builds in a given test run.
Attributes:
sut_os_name: The operating system of the SUT node.
@@ -342,20 +342,18 @@ def __init__(self, test_run_config: TestRunConfiguration):
self._config = test_run_config
self._test_suites_with_cases = []
- def add_build_target(
- self, build_target_config: BuildTargetConfiguration
- ) -> "BuildTargetResult":
- """Add and return the child result (build target).
+ def add_dpdk_build(self, dpdk_build_config: DPDKBuildConfiguration) -> "DPDKBuildResult":
+ """Add and return the child result (DPDK build).
Args:
- build_target_config: The build target's test run configuration.
+ dpdk_build: The DPDK build's test run configuration.
Returns:
- The build target's result.
+ The DPDK build's result.
"""
- result = BuildTargetResult(
+ result = DPDKBuildResult(
self._test_suites_with_cases,
- build_target_config,
+ dpdk_build_config,
)
self.child_results.append(result)
return result
@@ -394,22 +392,22 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
def _block_result(self) -> None:
r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
- for build_target in self._config.build_targets:
- child_result = self.add_build_target(build_target)
+ for dpdk_build in self._config.dpdk_builds:
+ child_result = self.add_dpdk_build(dpdk_build)
child_result.update_setup(Result.BLOCK)
-class BuildTargetResult(BaseResult):
- """The build target specific result.
+class DPDKBuildResult(BaseResult):
+ """The DPDK build specific result.
- The internal list stores the results of all test suites in a given build target.
+ The internal list stores the results of all test suites in a given DPDK build.
Attributes:
- arch: The DPDK build target architecture.
- os: The DPDK build target operating system.
- cpu: The DPDK build target CPU.
- compiler: The DPDK build target compiler.
- compiler_version: The DPDK build target compiler version.
+ arch: The DPDK DPDK build architecture.
+ os: The DPDK DPDK build operating system.
+ cpu: The DPDK DPDK build CPU.
+ compiler: The DPDK DPDK build compiler.
+ compiler_version: The DPDK DPDK build compiler version.
dpdk_version: The built DPDK version.
"""
@@ -424,19 +422,19 @@ class BuildTargetResult(BaseResult):
def __init__(
self,
test_suites_with_cases: list[TestSuiteWithCases],
- build_target_config: BuildTargetConfiguration,
+ dpdk_build_config: DPDKBuildConfiguration,
):
- """Extend the constructor with the build target's config and test suites with cases.
+ """Extend the constructor with the DPDK build's config and test suites with cases.
Args:
- test_suites_with_cases: The test suites with test cases to be run in this build target.
- build_target_config: The build target's test run configuration.
+ test_suites_with_cases: The test suites with test cases to be run in this DPDK build.
+ dpdk_build_config: The DPDK build's test run configuration.
"""
super().__init__()
- self.arch = build_target_config.arch
- self.os = build_target_config.os
- self.cpu = build_target_config.cpu
- self.compiler = build_target_config.compiler
+ self.arch = dpdk_build_config.arch
+ self.os = dpdk_build_config.os
+ self.cpu = dpdk_build_config.cpu
+ self.compiler = dpdk_build_config.compiler
self.compiler_version = None
self.dpdk_version = None
self._test_suites_with_cases = test_suites_with_cases
@@ -457,8 +455,8 @@ def add_test_suite(
self.child_results.append(result)
return result
- def add_build_target_info(self, versions: BuildTargetInfo) -> None:
- """Add information about the build target gathered at runtime.
+ def add_dpdk_build_info(self, versions: DPDKBuildInfo) -> None:
+ """Add information about the DPDK build gathered at runtime.
Args:
versions: The additional information.
@@ -484,11 +482,11 @@ class TestSuiteResult(BaseResult):
test_suite_name: str
_test_suite_with_cases: TestSuiteWithCases
- _parent_result: BuildTargetResult
+ _parent_result: DPDKBuildResult
_child_configs: list[str]
def __init__(self, test_suite_with_cases: TestSuiteWithCases):
- """Extend the constructor with test suite's config and BuildTargetResult.
+ """Extend the constructor with test suite's config and DPDKBuildResult.
Args:
test_suite_with_cases: The test suite with test cases.
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 051509fb86..dbf33c3bcf 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -66,7 +66,7 @@ class TestSuite:
sut_node: SutNode
tg_node: TGNode
#: Whether the test suite is blocking. A failure of a blocking test suite
- #: will block the execution of all subsequent test suites in the current build target.
+ #: will block the execution of all subsequent test suites in the current DPDK build.
is_blocking: ClassVar[bool] = False
_logger: DTSLogger
_port_links: list[PortLink]
diff --git a/dts/framework/testbed_model/sut_node.py b/dts/framework/testbed_model/sut_node.py
index 2855fe0276..6b6fb894ca 100644
--- a/dts/framework/testbed_model/sut_node.py
+++ b/dts/framework/testbed_model/sut_node.py
@@ -18,8 +18,8 @@
from pathlib import PurePath
from framework.config import (
- BuildTargetConfiguration,
- BuildTargetInfo,
+ DPDKBuildConfiguration,
+ DPDKBuildInfo,
NodeInfo,
SutNodeConfiguration,
TestRunConfiguration,
@@ -57,7 +57,7 @@ class SutNode(Node):
virtual_devices: list[VirtualDevice]
dpdk_prefix_list: list[str]
dpdk_timestamp: str
- _build_target_config: BuildTargetConfiguration | None
+ _dpdk_build_config: DPDKBuildConfiguration | None
_env_vars: dict
_remote_tmp_dir: PurePath
__remote_dpdk_dir: PurePath | None
@@ -77,7 +77,7 @@ def __init__(self, node_config: SutNodeConfiguration):
super().__init__(node_config)
self.virtual_devices = []
self.dpdk_prefix_list = []
- self._build_target_config = None
+ self._dpdk_build_config = None
self._env_vars = {}
self._remote_tmp_dir = self.main_session.get_remote_tmp_dir()
self.__remote_dpdk_dir = None
@@ -115,9 +115,9 @@ def remote_dpdk_build_dir(self) -> PurePath:
This is the directory where DPDK was built.
We assume it was built in a subdirectory of the extracted tarball.
"""
- if self._build_target_config:
+ if self._dpdk_build_config:
return self.main_session.join_remote_path(
- self._remote_dpdk_dir, self._build_target_config.name
+ self._remote_dpdk_dir, self._dpdk_build_config.name
)
else:
return self.main_session.join_remote_path(self._remote_dpdk_dir, "build")
@@ -140,13 +140,13 @@ def node_info(self) -> NodeInfo:
def compiler_version(self) -> str:
"""The node's compiler version."""
if self._compiler_version is None:
- if self._build_target_config is not None:
+ if self._dpdk_build_config is not None:
self._compiler_version = self.main_session.get_compiler_version(
- self._build_target_config.compiler.name
+ self._dpdk_build_config.compiler.name
)
else:
self._logger.warning(
- "Failed to get compiler version because _build_target_config is None."
+ "Failed to get compiler version because _dpdk_build_config is None."
)
return ""
return self._compiler_version
@@ -160,15 +160,13 @@ def path_to_devbind_script(self) -> PurePath:
)
return self._path_to_devbind_script
- def get_build_target_info(self) -> BuildTargetInfo:
- """Get additional build target information.
+ def get_dpdk_build_info(self) -> DPDKBuildInfo:
+ """Get additional DPDK build information.
Returns:
- The build target information,
+ The DPDK build information,
"""
- return BuildTargetInfo(
- dpdk_version=self.dpdk_version, compiler_version=self.compiler_version
- )
+ return DPDKBuildInfo(dpdk_version=self.dpdk_version, compiler_version=self.compiler_version)
def _guess_dpdk_remote_dir(self) -> PurePath:
return self.main_session.guess_dpdk_remote_dir(self._remote_tmp_dir)
@@ -189,40 +187,39 @@ def tear_down_test_run(self) -> None:
super().tear_down_test_run()
self.virtual_devices = []
- def set_up_build_target(self, build_target_config: BuildTargetConfiguration) -> None:
+ def set_up_dpdk(self, dpdk_build_config: DPDKBuildConfiguration) -> None:
"""Set up DPDK the SUT node and bind ports.
DPDK setup includes setting all internals needed for the build, the copying of DPDK tarball
and then building DPDK. The drivers are bound to those that DPDK needs.
Args:
- build_target_config: The build target test run configuration according to which
+ dpdk_build_config: The DPDK build test run configuration according to which
the setup steps will be taken.
"""
- self._configure_build_target(build_target_config)
+ self._configure_dpdk_build(dpdk_build_config)
self._copy_dpdk_tarball()
self._build_dpdk()
self.bind_ports_to_driver()
- def tear_down_build_target(self) -> None:
+ def tear_down_dpdk(self) -> None:
"""Reset DPDK variables and bind port driver to the OS driver."""
self._env_vars = {}
- self._build_target_config = None
+ self._dpdk_build_config = None
self.__remote_dpdk_dir = None
self._dpdk_version = None
self._compiler_version = None
self.bind_ports_to_driver(for_dpdk=False)
- def _configure_build_target(self, build_target_config: BuildTargetConfiguration) -> None:
- """Populate common environment variables and set build target config."""
+ def _configure_dpdk_build(self, dpdk_build_config: DPDKBuildConfiguration) -> None:
+ """Populate common environment variables and set DPDK build config."""
self._env_vars = {}
- self._build_target_config = build_target_config
- self._env_vars.update(self.main_session.get_dpdk_build_env_vars(build_target_config.arch))
- self._env_vars["CC"] = build_target_config.compiler.name
- if build_target_config.compiler_wrapper:
- self._env_vars["CC"] = (
- f"'{build_target_config.compiler_wrapper} {build_target_config.compiler.name}'"
- ) # fmt: skip
+ self._dpdk_build_config = dpdk_build_config
+ self._env_vars.update(self.main_session.get_dpdk_build_env_vars(dpdk_build_config.arch))
+ self._env_vars["CC"] = dpdk_build_config.compiler.name
+ if dpdk_build_config.compiler_wrapper:
+ self._env_vars["CC"] = f"'{self._dpdk_build_config.compiler_wrapper} "
+ f"{self._dpdk_build_config.compiler.name}'"
@Node.skip_setup
def _copy_dpdk_tarball(self) -> None:
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index c0b0e6bb00..ac661472b9 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -29,7 +29,7 @@ class TestSmokeTests(TestSuite):
Attributes:
is_blocking: This test suite will block the execution of all other test suites
- in the build target after it.
+ in the DPDK build after it.
nics_in_node: The NICs present on the SUT node.
"""
--
2.46.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/7] dts: one dpdk build per test run
2024-09-27 16:08 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 1/7] dts: rename build target to " Tomáš Ďurovec
@ 2024-09-27 16:08 ` Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 3/7] dts: fix remote session file transfer vars Tomáš Ďurovec
` (4 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Tomáš Ďurovec @ 2024-09-27 16:08 UTC (permalink / raw)
To: dev, Luca.Vizzarro, probb, npratte, dmarx; +Cc: Tomáš Ďurovec
When the DPDK build can be already pre-build, there is
not a need for defining multiple build targets. To make
it cleaner we decide to use one DPDK build wheater can
be pre-build or DTS will build it.
Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
---
dts/conf.yaml | 14 +--
dts/framework/config/__init__.py | 9 +-
dts/framework/config/conf_yaml_schema.json | 10 +-
dts/framework/config/types.py | 2 +-
dts/framework/logger.py | 4 -
dts/framework/runner.py | 117 +++++---------------
dts/framework/test_result.py | 119 ++++++---------------
dts/framework/test_suite.py | 2 +-
dts/framework/testbed_model/sut_node.py | 6 +-
dts/tests/TestSuite_smoke_tests.py | 2 +-
10 files changed, 80 insertions(+), 205 deletions(-)
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 1363e93580..814744a1fc 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -4,13 +4,13 @@
test_runs:
# define one test run environment
- - dpdk_builds:
- - arch: x86_64
- os: linux
- cpu: native
- # the combination of the following two makes CC="ccache gcc"
- compiler: gcc
- compiler_wrapper: ccache
+ - dpdk_build:
+ arch: x86_64
+ os: linux
+ cpu: native
+ # the combination of the following two makes CC="ccache gcc"
+ compiler: gcc
+ compiler_wrapper: ccache
perf: false # disable performance testing
func: true # enable functional testing
skip_smoke_tests: false # optional
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index c243716010..49b2e8d016 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -437,7 +437,7 @@ class TestRunConfiguration:
and with what DPDK build.
Attributes:
- dpdk_builds: A list of DPDK builds to test.
+ dpdk_build: A DPDK build to test.
perf: Whether to run performance tests.
func: Whether to run functional tests.
skip_smoke_tests: Whether to skip smoke tests.
@@ -448,7 +448,7 @@ class TestRunConfiguration:
random_seed: The seed to use for pseudo-random generation.
"""
- dpdk_builds: list[DPDKBuildConfiguration]
+ dpdk_build: DPDKBuildConfiguration
perf: bool
func: bool
skip_smoke_tests: bool
@@ -477,9 +477,6 @@ def from_dict(
Returns:
The test run configuration instance.
"""
- dpdk_builds: list[DPDKBuildConfiguration] = list(
- map(DPDKBuildConfiguration.from_dict, d["dpdk_builds"])
- )
test_suites: list[TestSuiteConfig] = list(map(TestSuiteConfig.from_dict, d["test_suites"]))
sut_name = d["system_under_test_node"]["node_name"]
skip_smoke_tests = d.get("skip_smoke_tests", False)
@@ -501,7 +498,7 @@ def from_dict(
)
random_seed = d.get("random_seed", None)
return cls(
- dpdk_builds=dpdk_builds,
+ dpdk_build=DPDKBuildConfiguration.from_dict(d["dpdk_build"]),
perf=d["perf"],
func=d["func"],
skip_smoke_tests=skip_smoke_tests,
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 927a73ac6c..94d7efa5f5 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -327,12 +327,8 @@
"items": {
"type": "object",
"properties": {
- "dpdk_builds": {
- "type": "array",
- "items": {
- "$ref": "#/definitions/dpdk_build"
- },
- "minimum": 1
+ "dpdk_build": {
+ "$ref": "#/definitions/dpdk_build"
},
"perf": {
"type": "boolean",
@@ -387,7 +383,7 @@
},
"additionalProperties": false,
"required": [
- "dpdk_builds",
+ "dpdk_build",
"perf",
"func",
"test_suites",
diff --git a/dts/framework/config/types.py b/dts/framework/config/types.py
index 4f450267d1..a710c20d6a 100644
--- a/dts/framework/config/types.py
+++ b/dts/framework/config/types.py
@@ -108,7 +108,7 @@ class TestRunConfigDict(TypedDict):
"""Allowed keys and values."""
#:
- dpdk_builds: list[DPDKBuildConfigDict]
+ dpdk_build: DPDKBuildConfigDict
#:
perf: bool
#:
diff --git a/dts/framework/logger.py b/dts/framework/logger.py
index 3fbe618219..d2b8e37da4 100644
--- a/dts/framework/logger.py
+++ b/dts/framework/logger.py
@@ -33,16 +33,12 @@ class DtsStage(StrEnum):
#:
test_run_setup = auto()
#:
- dpdk_build_setup = auto()
- #:
test_suite_setup = auto()
#:
test_suite = auto()
#:
test_suite_teardown = auto()
#:
- dpdk_build_teardown = auto()
- #:
test_run_teardown = auto()
#:
post_run = auto()
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index d63b1821e7..100dd75adb 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -12,7 +12,7 @@
#. Test suite stage,
#. Test case stage.
-The test run and DPDK build stages set up the environment before running test suites.
+The test run stage sets up the environment before running test suites.
The test suite stage sets up steps common to all test cases
and the test case stage runs test cases individually.
"""
@@ -30,13 +30,7 @@
from framework.testbed_model.sut_node import SutNode
from framework.testbed_model.tg_node import TGNode
-from .config import (
- Configuration,
- DPDKBuildConfiguration,
- TestRunConfiguration,
- TestSuiteConfig,
- load_config,
-)
+from .config import Configuration, TestRunConfiguration, TestSuiteConfig, load_config
from .exception import (
BlockingTestSuiteError,
ConfigurationError,
@@ -46,7 +40,6 @@
from .logger import DTSLogger, DtsStage, get_dts_logger
from .settings import SETTINGS
from .test_result import (
- DPDKBuildResult,
DTSResult,
Result,
TestCaseResult,
@@ -70,9 +63,9 @@ class DTSRunner:
:class:`~.framework.exception.DTSError`\s.
Example:
- An error occurs in a DPDK build setup. The current DPDK build is aborted,
- all test suites and their test cases are marked as blocked and the run continues
- with the next DPDK build. If the errored DPDK build was the last one in the
+ An error occurs in a test suite setup. The current test suite is aborted,
+ all its test cases are marked as blocked and the run continues
+ with the next test suite. If the errored test suite was the last one in the
given test run, the next test run begins.
"""
@@ -98,16 +91,16 @@ def __init__(self):
self._perf_test_case_regex = r"test_perf_"
def run(self) -> None:
- """Run all DPDK build in all test runs from the test run configuration.
+ """Run all test runs from the test run configuration.
- Before running test suites, test runs and DPDK builds are first set up.
- The test runs and DPDK builds defined in the test run configuration are iterated over.
- The test runs define which tests to run and where to run them and DPDK builds define
- the DPDK build setup.
+ Before running test suites, test runs are first set up.
+ The test runs defined in the test run configuration are iterated over.
+ The test runs define which tests to run and where to run them.
- The tests suites are set up for each test run/DPDK build tuple and each discovered
+ The test suites are set up for each test run and each discovered
test case within the test suite is set up, executed and torn down. After all test cases
- have been executed, the test suite is torn down and the next DPDK build will be tested.
+ have been executed, the test suite is torn down and the next test suite will be run. Once
+ all test suites have been run, the next test run will be tested.
In order to properly mark test suites and test cases as blocked in case of a failure,
we need to have discovered which test suites and test cases to run before any failures
@@ -117,17 +110,13 @@ def run(self) -> None:
#. Test run setup
- #. DPDK build setup
-
- #. Test suite setup
+ #. Test suite setup
- #. Test case setup
- #. Test case logic
- #. Test case teardown
+ #. Test case setup
+ #. Test case logic
+ #. Test case teardown
- #. Test suite teardown
-
- #. DPDK build teardown
+ #. Test suite teardown
#. Test run teardown
@@ -416,7 +405,7 @@ def _run_test_run(
) -> None:
"""Run the given test run.
- This involves running the test run setup as well as running all DPDK builds
+ This involves running the test run setup as well as running all test suites
in the given test run. After that, the test run teardown is run.
Args:
@@ -432,6 +421,7 @@ def _run_test_run(
test_run_result.add_sut_info(sut_node.node_info)
try:
sut_node.set_up_test_run(test_run_config)
+ test_run_result.add_dpdk_build_info(sut_node.get_dpdk_build_info())
tg_node.set_up_test_run(test_run_config)
test_run_result.update_setup(Result.PASS)
except Exception as e:
@@ -439,15 +429,7 @@ def _run_test_run(
test_run_result.update_setup(Result.FAIL, e)
else:
- for dpdk_build_config in test_run_config.dpdk_builds:
- dpdk_build_result = test_run_result.add_dpdk_build(dpdk_build_config)
- self._run_dpdk_build(
- sut_node,
- tg_node,
- dpdk_build_config,
- dpdk_build_result,
- test_suites_with_cases,
- )
+ self._run_test_suites(sut_node, tg_node, test_run_result, test_suites_with_cases)
finally:
try:
@@ -459,82 +441,35 @@ def _run_test_run(
self._logger.exception("Test run teardown failed.")
test_run_result.update_teardown(Result.FAIL, e)
- def _run_dpdk_build(
- self,
- sut_node: SutNode,
- tg_node: TGNode,
- dpdk_build_config: DPDKBuildConfiguration,
- dpdk_build_result: DPDKBuildResult,
- test_suites_with_cases: Iterable[TestSuiteWithCases],
- ) -> None:
- """Run the given DPDK build.
-
- This involves running the DPDK build setup as well as running all test suites
- of the DPDK build's test run.
- After that, DPDK build teardown is run.
-
- Args:
- sut_node: The test run's sut node.
- tg_node: The test run's tg node.
- dpdk_build_config: A DPDK build's test run configuration.
- dpdk_build_result: The DPDK build level result object associated
- with the current DPDK build.
- test_suites_with_cases: The test suites with test cases to run.
- """
- self._logger.set_stage(DtsStage.dpdk_build_setup)
- self._logger.info(f"Running DPDK build '{dpdk_build_config.name}'.")
-
- try:
- sut_node.set_up_dpdk(dpdk_build_config)
- self._result.dpdk_version = sut_node.dpdk_version
- dpdk_build_result.add_dpdk_build_info(sut_node.get_dpdk_build_info())
- dpdk_build_result.update_setup(Result.PASS)
- except Exception as e:
- self._logger.exception("DPDK build setup failed.")
- dpdk_build_result.update_setup(Result.FAIL, e)
-
- else:
- self._run_test_suites(sut_node, tg_node, dpdk_build_result, test_suites_with_cases)
-
- finally:
- try:
- self._logger.set_stage(DtsStage.dpdk_build_teardown)
- sut_node.tear_down_dpdk()
- dpdk_build_result.update_teardown(Result.PASS)
- except Exception as e:
- self._logger.exception("DPDK build teardown failed.")
- dpdk_build_result.update_teardown(Result.FAIL, e)
-
def _run_test_suites(
self,
sut_node: SutNode,
tg_node: TGNode,
- dpdk_build_result: DPDKBuildResult,
+ test_run_result: TestRunResult,
test_suites_with_cases: Iterable[TestSuiteWithCases],
) -> None:
- """Run `test_suites_with_cases` with the current DPDK build.
+ """Run `test_suites_with_cases` with the current test run.
The method assumes the DPDK we're testing has already been built on the SUT node.
If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites
- in the current DPDK build won't be executed.
+ in the current test run won't be executed.
Args:
sut_node: The test run's SUT node.
tg_node: The test run's TG node.
- dpdk_build_result: The DPDK build level result object associated
- with the current DPDK build.
+ test_run_result: The test run's result.
test_suites_with_cases: The test suites with test cases to run.
"""
end_dpdk_build = False
for test_suite_with_cases in test_suites_with_cases:
- test_suite_result = dpdk_build_result.add_test_suite(test_suite_with_cases)
+ test_suite_result = test_run_result.add_test_suite(test_suite_with_cases)
try:
self._run_test_suite(sut_node, tg_node, test_suite_result, test_suite_with_cases)
except BlockingTestSuiteError as e:
self._logger.exception(
f"An error occurred within {test_suite_with_cases.test_suite_class.__name__}. "
- "Skipping DPDK build ..."
+ "Skipping the rest of the test suites in this test run."
)
self._result.add_error(e)
end_dpdk_build = True
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 95788b7d2e..31560f6704 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -8,12 +8,11 @@
* :class:`DTSResult` contains
* :class:`TestRunResult` contains
- * :class:`DPDKBuildResult` contains
* :class:`TestSuiteResult` contains
* :class:`TestCaseResult`
Each result may contain multiple lower level results, e.g. there are multiple
-:class:`TestSuiteResult`\s in a :class:`DPDKBuildResult`.
+:class:`TestSuiteResult`\s in a :class:`TestRunResult`.
The results have common parts, such as setup and teardown results, captured in :class:`BaseResult`,
which also defines some common behaviors in its methods.
@@ -35,7 +34,6 @@
Architecture,
Compiler,
CPUType,
- DPDKBuildConfiguration,
DPDKBuildInfo,
NodeInfo,
TestRunConfiguration,
@@ -138,7 +136,7 @@ class BaseResult:
Stores the results of the setup and teardown portions of the corresponding stage.
The hierarchical nature of DTS results is captured recursively in an internal list.
A stage is each level in this particular hierarchy (pre-run or the top-most level,
- test run, DPDK build, test suite and test case.)
+ test run, test suite and test case).
Attributes:
setup_result: The result of the setup of the particular stage.
@@ -222,8 +220,8 @@ def add_stats(self, statistics: "Statistics") -> None:
class DTSResult(BaseResult):
"""Stores environment information and test results from a DTS run.
- * Test run level information, such as testbed and the test suite list,
- * DPDK build level information, such as compiler, target OS and cpu,
+ * Test run level information, such as testbed, the test suite list and
+ DPDK build configuration (compiler, target OS and cpu),
* Test suite and test case results,
* All errors that are caught and recorded during DTS execution.
@@ -317,44 +315,61 @@ def get_return_code(self) -> int:
class TestRunResult(BaseResult):
"""The test run specific result.
- The internal list stores the results of all DPDK builds in a given test run.
+ The internal list stores the results of all test suites in a given test run.
Attributes:
+ arch: The DPDK build architecture.
+ os: The DPDK build operating system.
+ cpu: The DPDK build CPU.
+ compiler: The DPDK build compiler.
+ compiler_version: The DPDK build compiler version.
+ dpdk_version: The built DPDK version.
sut_os_name: The operating system of the SUT node.
sut_os_version: The operating system version of the SUT node.
sut_kernel_version: The operating system kernel version of the SUT node.
"""
+ arch: Architecture
+ os: OS
+ cpu: CPUType
+ compiler: Compiler
+ compiler_version: str | None
+ dpdk_version: str | None
sut_os_name: str
sut_os_version: str
sut_kernel_version: str
_config: TestRunConfiguration
- _parent_result: DTSResult
_test_suites_with_cases: list[TestSuiteWithCases]
def __init__(self, test_run_config: TestRunConfiguration):
- """Extend the constructor with the test run's config and DTSResult.
+ """Extend the constructor with the test run's config.
Args:
test_run_config: A test run configuration.
"""
super().__init__()
+ self.arch = test_run_config.dpdk_build.arch
+ self.os = test_run_config.dpdk_build.os
+ self.cpu = test_run_config.dpdk_build.cpu
+ self.compiler = test_run_config.dpdk_build.compiler
+ self.compiler_version = None
+ self.dpdk_version = None
self._config = test_run_config
self._test_suites_with_cases = []
- def add_dpdk_build(self, dpdk_build_config: DPDKBuildConfiguration) -> "DPDKBuildResult":
- """Add and return the child result (DPDK build).
+ def add_test_suite(
+ self,
+ test_suite_with_cases: TestSuiteWithCases,
+ ) -> "TestSuiteResult":
+ """Add and return the child result (test suite).
Args:
- dpdk_build: The DPDK build's test run configuration.
+ test_suite_with_cases: The test suite with test cases.
Returns:
- The DPDK build's result.
+ The test suite's result.
"""
- result = DPDKBuildResult(
- self._test_suites_with_cases,
- dpdk_build_config,
- )
+ result = TestSuiteResult(test_suite_with_cases)
self.child_results.append(result)
return result
@@ -390,71 +405,6 @@ def add_sut_info(self, sut_info: NodeInfo) -> None:
self.sut_os_version = sut_info.os_version
self.sut_kernel_version = sut_info.kernel_version
- def _block_result(self) -> None:
- r"""Mark the result as :attr:`~Result.BLOCK`\ed."""
- for dpdk_build in self._config.dpdk_builds:
- child_result = self.add_dpdk_build(dpdk_build)
- child_result.update_setup(Result.BLOCK)
-
-
-class DPDKBuildResult(BaseResult):
- """The DPDK build specific result.
-
- The internal list stores the results of all test suites in a given DPDK build.
-
- Attributes:
- arch: The DPDK DPDK build architecture.
- os: The DPDK DPDK build operating system.
- cpu: The DPDK DPDK build CPU.
- compiler: The DPDK DPDK build compiler.
- compiler_version: The DPDK DPDK build compiler version.
- dpdk_version: The built DPDK version.
- """
-
- arch: Architecture
- os: OS
- cpu: CPUType
- compiler: Compiler
- compiler_version: str | None
- dpdk_version: str | None
- _test_suites_with_cases: list[TestSuiteWithCases]
-
- def __init__(
- self,
- test_suites_with_cases: list[TestSuiteWithCases],
- dpdk_build_config: DPDKBuildConfiguration,
- ):
- """Extend the constructor with the DPDK build's config and test suites with cases.
-
- Args:
- test_suites_with_cases: The test suites with test cases to be run in this DPDK build.
- dpdk_build_config: The DPDK build's test run configuration.
- """
- super().__init__()
- self.arch = dpdk_build_config.arch
- self.os = dpdk_build_config.os
- self.cpu = dpdk_build_config.cpu
- self.compiler = dpdk_build_config.compiler
- self.compiler_version = None
- self.dpdk_version = None
- self._test_suites_with_cases = test_suites_with_cases
-
- def add_test_suite(
- self,
- test_suite_with_cases: TestSuiteWithCases,
- ) -> "TestSuiteResult":
- """Add and return the child result (test suite).
-
- Args:
- test_suite_with_cases: The test suite with test cases.
-
- Returns:
- The test suite's result.
- """
- result = TestSuiteResult(test_suite_with_cases)
- self.child_results.append(result)
- return result
-
def add_dpdk_build_info(self, versions: DPDKBuildInfo) -> None:
"""Add information about the DPDK build gathered at runtime.
@@ -482,11 +432,10 @@ class TestSuiteResult(BaseResult):
test_suite_name: str
_test_suite_with_cases: TestSuiteWithCases
- _parent_result: DPDKBuildResult
_child_configs: list[str]
def __init__(self, test_suite_with_cases: TestSuiteWithCases):
- """Extend the constructor with test suite's config and DPDKBuildResult.
+ """Extend the constructor with test suite's config.
Args:
test_suite_with_cases: The test suite with test cases.
@@ -529,7 +478,7 @@ class TestCaseResult(BaseResult, FixtureResult):
test_case_name: str
def __init__(self, test_case_name: str):
- """Extend the constructor with test case's name and TestSuiteResult.
+ """Extend the constructor with test case's name.
Args:
test_case_name: The test case's name.
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index dbf33c3bcf..6e14d47ae6 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -66,7 +66,7 @@ class TestSuite:
sut_node: SutNode
tg_node: TGNode
#: Whether the test suite is blocking. A failure of a blocking test suite
- #: will block the execution of all subsequent test suites in the current DPDK build.
+ #: will block the execution of all subsequent test suites in the current test run.
is_blocking: ClassVar[bool] = False
_logger: DTSLogger
_port_links: list[PortLink]
diff --git a/dts/framework/testbed_model/sut_node.py b/dts/framework/testbed_model/sut_node.py
index 6b6fb894ca..9bfb91816e 100644
--- a/dts/framework/testbed_model/sut_node.py
+++ b/dts/framework/testbed_model/sut_node.py
@@ -181,13 +181,15 @@ def set_up_test_run(self, test_run_config: TestRunConfiguration) -> None:
super().set_up_test_run(test_run_config)
for vdev in test_run_config.vdevs:
self.virtual_devices.append(VirtualDevice(vdev))
+ self._set_up_dpdk(test_run_config.dpdk_build)
def tear_down_test_run(self) -> None:
"""Extend the test run teardown with virtual device teardown."""
super().tear_down_test_run()
self.virtual_devices = []
+ self._tear_down_dpdk()
- def set_up_dpdk(self, dpdk_build_config: DPDKBuildConfiguration) -> None:
+ def _set_up_dpdk(self, dpdk_build_config: DPDKBuildConfiguration) -> None:
"""Set up DPDK the SUT node and bind ports.
DPDK setup includes setting all internals needed for the build, the copying of DPDK tarball
@@ -202,7 +204,7 @@ def set_up_dpdk(self, dpdk_build_config: DPDKBuildConfiguration) -> None:
self._build_dpdk()
self.bind_ports_to_driver()
- def tear_down_dpdk(self) -> None:
+ def _tear_down_dpdk(self) -> None:
"""Reset DPDK variables and bind port driver to the OS driver."""
self._env_vars = {}
self._dpdk_build_config = None
diff --git a/dts/tests/TestSuite_smoke_tests.py b/dts/tests/TestSuite_smoke_tests.py
index ac661472b9..99fa8d19c7 100644
--- a/dts/tests/TestSuite_smoke_tests.py
+++ b/dts/tests/TestSuite_smoke_tests.py
@@ -29,7 +29,7 @@ class TestSmokeTests(TestSuite):
Attributes:
is_blocking: This test suite will block the execution of all other test suites
- in the DPDK build after it.
+ in the test run after it.
nics_in_node: The NICs present on the SUT node.
"""
--
2.46.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 3/7] dts: fix remote session file transfer vars
2024-09-27 16:08 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 1/7] dts: rename build target to " Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 2/7] dts: one dpdk build per test run Tomáš Ďurovec
@ 2024-09-27 16:08 ` Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 4/7] dts: add the ability to copy directories via remote Tomáš Ďurovec
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Tomáš Ďurovec @ 2024-09-27 16:08 UTC (permalink / raw)
To: dev, Luca.Vizzarro, probb, npratte, dmarx; +Cc: Tomáš Ďurovec
The OSSession (and its subclasses) should accept PurePaths
for remote paths to translate from OS-unaware (PurePath)
to OS-aware (Path) only on the remote side. For local paths,
they should accept Paths, as Python is OS-aware locally.
Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
---
.../remote_session/remote_session.py | 24 ++++++----------
dts/framework/remote_session/ssh_session.py | 18 ++++--------
dts/framework/testbed_model/os_session.py | 28 ++++++++-----------
dts/framework/testbed_model/posix_session.py | 18 ++++--------
4 files changed, 30 insertions(+), 58 deletions(-)
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
index 8c580b070f..ab83f5b266 100644
--- a/dts/framework/remote_session/remote_session.py
+++ b/dts/framework/remote_session/remote_session.py
@@ -12,7 +12,7 @@
from abc import ABC, abstractmethod
from dataclasses import InitVar, dataclass, field
-from pathlib import PurePath
+from pathlib import Path, PurePath
from framework.config import NodeConfiguration
from framework.exception import RemoteCommandExecutionError
@@ -196,35 +196,29 @@ def is_alive(self) -> bool:
"""Check whether the remote session is still responding."""
@abstractmethod
- def copy_from(
- self,
- source_file: str | PurePath,
- destination_file: str | PurePath,
- ) -> None:
+ def copy_from(self, source_file: str | PurePath, destination_dir: str | Path) -> None:
"""Copy a file from the remote Node to the local filesystem.
Copy `source_file` from the remote Node associated with this remote session
- to `destination_file` on the local filesystem.
+ to `destination_dir` on the local filesystem.
Args:
source_file: The file on the remote Node.
- destination_file: A file or directory path on the local filesystem.
+ destination_dir: The directory path on the local filesystem where the `source_file`
+ will be saved.
"""
@abstractmethod
- def copy_to(
- self,
- source_file: str | PurePath,
- destination_file: str | PurePath,
- ) -> None:
+ def copy_to(self, source_file: str | Path, destination_dir: str | PurePath) -> None:
"""Copy a file from local filesystem to the remote Node.
- Copy `source_file` from local filesystem to `destination_file` on the remote Node
+ Copy `source_file` from local filesystem to `destination_dir` on the remote Node
associated with this remote session.
Args:
source_file: The file on the local filesystem.
- destination_file: A file or directory path on the remote Node.
+ destination_dir: The directory path on the remote Node where the `source_file`
+ will be saved.
"""
@abstractmethod
diff --git a/dts/framework/remote_session/ssh_session.py b/dts/framework/remote_session/ssh_session.py
index 66f8176833..329121913f 100644
--- a/dts/framework/remote_session/ssh_session.py
+++ b/dts/framework/remote_session/ssh_session.py
@@ -5,7 +5,7 @@
import socket
import traceback
-from pathlib import PurePath
+from pathlib import Path, PurePath
from fabric import Connection # type: ignore[import-untyped]
from invoke.exceptions import ( # type: ignore[import-untyped]
@@ -103,21 +103,13 @@ def is_alive(self) -> bool:
"""Overrides :meth:`~.remote_session.RemoteSession.is_alive`."""
return self.session.is_connected
- def copy_from(
- self,
- source_file: str | PurePath,
- destination_file: str | PurePath,
- ) -> None:
+ def copy_from(self, source_file: str | PurePath, destination_dir: str | Path) -> None:
"""Overrides :meth:`~.remote_session.RemoteSession.copy_from`."""
- self.session.get(str(destination_file), str(source_file))
+ self.session.get(str(source_file), str(destination_dir))
- def copy_to(
- self,
- source_file: str | PurePath,
- destination_file: str | PurePath,
- ) -> None:
+ def copy_to(self, source_file: str | Path, destination_dir: str | PurePath) -> None:
"""Overrides :meth:`~.remote_session.RemoteSession.copy_to`."""
- self.session.put(str(source_file), str(destination_file))
+ self.session.put(str(source_file), str(destination_dir))
def close(self) -> None:
"""Overrides :meth:`~.remote_session.RemoteSession.close`."""
diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
index 79f56b289b..1aac3659bf 100644
--- a/dts/framework/testbed_model/os_session.py
+++ b/dts/framework/testbed_model/os_session.py
@@ -25,7 +25,7 @@
from abc import ABC, abstractmethod
from collections.abc import Iterable
from ipaddress import IPv4Interface, IPv6Interface
-from pathlib import PurePath
+from pathlib import Path, PurePath
from typing import Union
from framework.config import Architecture, NodeConfiguration, NodeInfo
@@ -178,35 +178,29 @@ def join_remote_path(self, *args: str | PurePath) -> PurePath:
"""
@abstractmethod
- def copy_from(
- self,
- source_file: str | PurePath,
- destination_file: str | PurePath,
- ) -> None:
+ def copy_from(self, source_file: str | PurePath, destination_dir: str | Path) -> None:
"""Copy a file from the remote node to the local filesystem.
Copy `source_file` from the remote node associated with this remote
- session to `destination_file` on the local filesystem.
+ session to `destination_dir` on the local filesystem.
Args:
- source_file: the file on the remote node.
- destination_file: a file or directory path on the local filesystem.
+ source_file: The file on the remote node.
+ destination_dir: The directory path on the local filesystem where the `source_file`
+ will be saved.
"""
@abstractmethod
- def copy_to(
- self,
- source_file: str | PurePath,
- destination_file: str | PurePath,
- ) -> None:
+ def copy_to(self, source_file: str | Path, destination_dir: str | PurePath) -> None:
"""Copy a file from local filesystem to the remote node.
- Copy `source_file` from local filesystem to `destination_file`
+ Copy `source_file` from local filesystem to `destination_dir`
on the remote node associated with this remote session.
Args:
- source_file: the file on the local filesystem.
- destination_file: a file or directory path on the remote node.
+ source_file: The file on the local filesystem.
+ destination_dir: The directory path on the remote Node where the `source_file`
+ will be saved.
"""
@abstractmethod
diff --git a/dts/framework/testbed_model/posix_session.py b/dts/framework/testbed_model/posix_session.py
index d279bb8b53..2449c0ab35 100644
--- a/dts/framework/testbed_model/posix_session.py
+++ b/dts/framework/testbed_model/posix_session.py
@@ -13,7 +13,7 @@
import re
from collections.abc import Iterable
-from pathlib import PurePath, PurePosixPath
+from pathlib import Path, PurePath, PurePosixPath
from framework.config import Architecture, NodeInfo
from framework.exception import DPDKBuildError, RemoteCommandExecutionError
@@ -85,21 +85,13 @@ def join_remote_path(self, *args: str | PurePath) -> PurePosixPath:
"""Overrides :meth:`~.os_session.OSSession.join_remote_path`."""
return PurePosixPath(*args)
- def copy_from(
- self,
- source_file: str | PurePath,
- destination_file: str | PurePath,
- ) -> None:
+ def copy_from(self, source_file: str | PurePath, destination_dir: str | Path) -> None:
"""Overrides :meth:`~.os_session.OSSession.copy_from`."""
- self.remote_session.copy_from(source_file, destination_file)
+ self.remote_session.copy_from(source_file, destination_dir)
- def copy_to(
- self,
- source_file: str | PurePath,
- destination_file: str | PurePath,
- ) -> None:
+ def copy_to(self, source_file: str | Path, destination_dir: str | PurePath) -> None:
"""Overrides :meth:`~.os_session.OSSession.copy_to`."""
- self.remote_session.copy_to(source_file, destination_file)
+ self.remote_session.copy_to(source_file, destination_dir)
def remove_remote_dir(
self,
--
2.46.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 4/7] dts: add the ability to copy directories via remote
2024-09-27 16:08 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
` (2 preceding siblings ...)
2024-09-27 16:08 ` [PATCH 3/7] dts: fix remote session file transfer vars Tomáš Ďurovec
@ 2024-09-27 16:08 ` Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 5/7] dts: add support for externally compiled DPDK Tomáš Ďurovec
` (2 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Tomáš Ďurovec @ 2024-09-27 16:08 UTC (permalink / raw)
To: dev, Luca.Vizzarro, probb, npratte, dmarx; +Cc: Tomáš Ďurovec
Before, the remote session did't allows to copy directories,
only files. This feature will be used in future commit.
Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
---
dts/framework/testbed_model/os_session.py | 120 ++++++++++++++++++-
dts/framework/testbed_model/posix_session.py | 88 +++++++++++++-
dts/framework/utils.py | 91 +++++++++++++-
3 files changed, 287 insertions(+), 12 deletions(-)
diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
index 1aac3659bf..6c3f84dec1 100644
--- a/dts/framework/testbed_model/os_session.py
+++ b/dts/framework/testbed_model/os_session.py
@@ -25,7 +25,7 @@
from abc import ABC, abstractmethod
from collections.abc import Iterable
from ipaddress import IPv4Interface, IPv6Interface
-from pathlib import Path, PurePath
+from pathlib import Path, PurePath, PurePosixPath
from typing import Union
from framework.config import Architecture, NodeConfiguration, NodeInfo
@@ -38,7 +38,7 @@
)
from framework.remote_session.remote_session import CommandResult
from framework.settings import SETTINGS
-from framework.utils import MesonArgs
+from framework.utils import MesonArgs, TarCompressionFormat
from .cpu import LogicalCore
from .port import Port
@@ -203,6 +203,95 @@ def copy_to(self, source_file: str | Path, destination_dir: str | PurePath) -> N
will be saved.
"""
+ @abstractmethod
+ def copy_dir_from(
+ self,
+ source_dir: str | PurePath,
+ destination_dir: str | Path,
+ compress_format: TarCompressionFormat = TarCompressionFormat.none,
+ exclude: str | list[str] | None = None,
+ ) -> None:
+ """Copy a directory from the remote node to the local filesystem.
+
+ Copy `source_dir` from the remote node associated with this remote session to
+ `destination_dir` on the local filesystem. The new local directory will be created
+ at `destination_dir` path.
+
+ Example:
+ source_dir = '/remote/path/to/source'
+ destination_dir = '/local/path/to/destination'
+ compress_format = TarCompressionFormat.xz
+
+ The method will:
+ 1. Create a tarball from `source_dir`, resulting in:
+ '/remote/path/to/source.tar.xz',
+ 2. Copy '/remote/path/to/source.tar.xz' to
+ '/local/path/to/destination/source.tar.xz',
+ 3. Extract the contents of the tarball, resulting in:
+ '/local/path/to/destination/source/',
+ 4. Remove the tarball after extraction
+ ('/local/path/to/destination/source.tar.xz').
+
+ Final Path Structure:
+ '/local/path/to/destination/source/'
+
+ Args:
+ source_dir: The directory on the remote node.
+ destination_dir: The directory path on the local filesystem.
+ compress_format: The compression format to use. Defaults to no compression.
+ exclude: Patterns for files or directories to exclude from the tarball.
+ These patterns are used with `tar`'s `--exclude` option.
+ """
+
+ @abstractmethod
+ def copy_dir_to(
+ self,
+ source_dir: str | Path,
+ destination_dir: str | PurePath,
+ compress_format: TarCompressionFormat = TarCompressionFormat.none,
+ exclude: str | list[str] | None = None,
+ ) -> None:
+ """Copy a directory from the local filesystem to the remote node.
+
+ Copy `source_dir` from the local filesystem to `destination_dir` on the remote node
+ associated with this remote session. The new remote directory will be created at
+ `destination_dir` path.
+
+ Example:
+ source_dir = '/local/path/to/source'
+ destination_dir = '/remote/path/to/destination'
+ compress_format = TarCompressionFormat.xz
+
+ The method will:
+ 1. Create a tarball from `source_dir`, resulting in:
+ '/local/path/to/source.tar.xz',
+ 2. Copy '/local/path/to/source.tar.xz' to
+ '/remote/path/to/destination/source.tar.xz',
+ 3. Extract the contents of the tarball, resulting in:
+ '/remote/path/to/destination/source/',
+ 4. Remove the tarball after extraction
+ ('/remote/path/to/destination/source.tar.xz').
+
+ Final Path Structure:
+ '/remote/path/to/destination/source/'
+
+ Args:
+ source_dir: The directory on the local filesystem.
+ destination_dir: The directory path on the remote node.
+ compress_format: The compression format to use. Defaults to no compression.
+ exclude: Patterns for files or directories to exclude from the tarball.
+ These patterns are used with `fnmatch.fnmatch` to filter out files.
+ """
+
+ @abstractmethod
+ def remove_remote_file(self, remote_file_path: str | PurePath, force: bool = True) -> None:
+ """Remove remote file, by default remove forcefully.
+
+ Args:
+ remote_file_path: The file path to remove.
+ force: If :data:`True`, ignore all warnings and try to remove at all costs.
+ """
+
@abstractmethod
def remove_remote_dir(
self,
@@ -213,11 +302,34 @@ def remove_remote_dir(
"""Remove remote directory, by default remove recursively and forcefully.
Args:
- remote_dir_path: The path of the directory to remove.
+ remote_dir_path: The directory path to remove.
recursive: If :data:`True`, also remove all contents inside the directory.
force: If :data:`True`, ignore all warnings and try to remove at all costs.
"""
+ @abstractmethod
+ def create_remote_tarball(
+ self,
+ remote_dir_path: str | PurePath,
+ compress_format: TarCompressionFormat = TarCompressionFormat.none,
+ exclude: str | list[str] | None = None,
+ ) -> PurePosixPath:
+ """Create a tarball from the contents of the specified remote directory.
+
+ This method creates a tarball containing all files and directories
+ within `remote_dir_path`. The tarball will be saved in the directory of
+ `remote_dir_path` and will be named based on `remote_dir_path`.
+
+ Args:
+ remote_dir_path: The directory path on the remote node.
+ compress_format: The compression format to use. Defaults to no compression.
+ exclude: Patterns for files or directories to exclude from the tarball.
+ These patterns are used with `tar`'s `--exclude` option.
+
+ Returns:
+ The path to the created tarball on the remote node.
+ """
+
@abstractmethod
def extract_remote_tarball(
self,
@@ -227,7 +339,7 @@ def extract_remote_tarball(
"""Extract remote tarball in its remote directory.
Args:
- remote_tarball_path: The path of the tarball on the remote node.
+ remote_tarball_path: The tarball path on the remote node.
expected_dir: If non-empty, check whether `expected_dir` exists after extracting
the archive.
"""
diff --git a/dts/framework/testbed_model/posix_session.py b/dts/framework/testbed_model/posix_session.py
index 2449c0ab35..94e721da61 100644
--- a/dts/framework/testbed_model/posix_session.py
+++ b/dts/framework/testbed_model/posix_session.py
@@ -18,7 +18,13 @@
from framework.config import Architecture, NodeInfo
from framework.exception import DPDKBuildError, RemoteCommandExecutionError
from framework.settings import SETTINGS
-from framework.utils import MesonArgs
+from framework.utils import (
+ MesonArgs,
+ TarCompressionFormat,
+ convert_to_list_of_string,
+ create_tarball,
+ extract_tarball,
+)
from .os_session import OSSession
@@ -93,6 +99,48 @@ def copy_to(self, source_file: str | Path, destination_dir: str | PurePath) -> N
"""Overrides :meth:`~.os_session.OSSession.copy_to`."""
self.remote_session.copy_to(source_file, destination_dir)
+ def copy_dir_from(
+ self,
+ source_dir: str | PurePath,
+ destination_dir: str | Path,
+ compress_format: TarCompressionFormat = TarCompressionFormat.none,
+ exclude: str | list[str] | None = None,
+ ) -> None:
+ """Overrides :meth:`~.os_session.OSSession.copy_dir_from`."""
+ source_dir = PurePath(source_dir)
+ remote_tarball_path = self.create_remote_tarball(source_dir, compress_format, exclude)
+
+ self.copy_from(remote_tarball_path, destination_dir)
+ self.remove_remote_file(remote_tarball_path)
+
+ tarball_path = Path(destination_dir, f"{source_dir.name}.{compress_format.extension}")
+ extract_tarball(tarball_path)
+ tarball_path.unlink()
+
+ def copy_dir_to(
+ self,
+ source_dir: str | Path,
+ destination_dir: str | PurePath,
+ compress_format: TarCompressionFormat = TarCompressionFormat.none,
+ exclude: str | list[str] | None = None,
+ ) -> None:
+ """Overrides :meth:`~.os_session.OSSession.copy_dir_to`."""
+ source_dir = Path(source_dir)
+ tarball_path = create_tarball(source_dir, compress_format, exclude=exclude)
+ self.copy_to(tarball_path, destination_dir)
+ tarball_path.unlink()
+
+ remote_tar_path = self.join_remote_path(
+ destination_dir, f"{source_dir.name}.{compress_format.extension}"
+ )
+ self.extract_remote_tarball(remote_tar_path)
+ self.remove_remote_file(remote_tar_path)
+
+ def remove_remote_file(self, remote_file_path: str | PurePath, force: bool = True) -> None:
+ """Overrides :meth:`~.os_session.OSSession.remove_remote_dir`."""
+ opts = PosixSession.combine_short_options(f=force)
+ self.send_command(f"rm{opts} {remote_file_path}")
+
def remove_remote_dir(
self,
remote_dir_path: str | PurePath,
@@ -103,10 +151,42 @@ def remove_remote_dir(
opts = PosixSession.combine_short_options(r=recursive, f=force)
self.send_command(f"rm{opts} {remote_dir_path}")
- def extract_remote_tarball(
+ def create_remote_tarball(
self,
- remote_tarball_path: str | PurePath,
- expected_dir: str | PurePath | None = None,
+ remote_dir_path: str | PurePath,
+ compress_format: TarCompressionFormat = TarCompressionFormat.none,
+ exclude: str | list[str] | None = None,
+ ) -> PurePosixPath:
+ """Overrides :meth:`~.os_session.OSSession.create_remote_tarball`."""
+
+ def generate_tar_exclude_args(exclude_patterns) -> str:
+ """Generate args to exclude patterns when creating a tarball.
+
+ Args:
+ exclude_patterns: Patterns for files or directories to exclude from the tarball.
+ These patterns are used with `tar`'s `--exclude` option.
+
+ Returns:
+ The generated string args to exclude the specified patterns.
+ """
+ if exclude_patterns:
+ exclude_patterns = convert_to_list_of_string(exclude_patterns)
+ return "".join([f" --exclude={pattern}" for pattern in exclude_patterns])
+ return ""
+
+ posix_remote_dir_path = PurePosixPath(remote_dir_path)
+ target_tarball_path = PurePosixPath(f"{remote_dir_path}.{compress_format.extension}")
+
+ self.send_command(
+ f"tar caf {target_tarball_path}{generate_tar_exclude_args(exclude)} "
+ f"-C {posix_remote_dir_path.parent} {posix_remote_dir_path.name}",
+ 60,
+ )
+
+ return target_tarball_path
+
+ def extract_remote_tarball(
+ self, remote_tarball_path: str | PurePath, expected_dir: str | PurePath | None = None
) -> None:
"""Overrides :meth:`~.os_session.OSSession.extract_remote_tarball`."""
self.send_command(
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index c768dd0c99..382357ffe8 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -15,13 +15,16 @@
"""
import atexit
+import fnmatch
import json
import os
import random
import subprocess
+import tarfile
from enum import Enum, Flag
from pathlib import Path
from subprocess import SubprocessError
+from typing import Any, Callable
from scapy.layers.inet import IP, TCP, UDP, Ether # type: ignore[import-untyped]
from scapy.packet import Packet # type: ignore[import-untyped]
@@ -142,13 +145,17 @@ def __str__(self) -> str:
return " ".join(f"{self._default_library} {self._dpdk_args}".split())
-class _TarCompressionFormat(StrEnum):
+class TarCompressionFormat(StrEnum):
"""Compression formats that tar can use.
Enum names are the shell compression commands
and Enum values are the associated file extensions.
+
+ The 'none' member represents no compression, only archiving with tar.
+ Its value is set to 'tar' to indicate that the file is an uncompressed tar archive.
"""
+ none = "tar"
gzip = "gz"
compress = "Z"
bzip2 = "bz2"
@@ -158,6 +165,16 @@ class _TarCompressionFormat(StrEnum):
xz = "xz"
zstd = "zst"
+ @property
+ def extension(self):
+ """Return the extension associated with the compression format.
+
+ If the compression format is 'none', the extension will be in the format 'tar'.
+ For other compression formats, the extension will be in the format
+ 'tar.{compression format}'.
+ """
+ return f"{self.value}" if self == self.none else f"{self.none.value}.{self.value}"
+
class DPDKGitTarball:
"""Compressed tarball of DPDK from the repository.
@@ -171,7 +188,7 @@ class DPDKGitTarball:
"""
_git_ref: str
- _tar_compression_format: _TarCompressionFormat
+ _tar_compression_format: TarCompressionFormat
_tarball_dir: Path
_tarball_name: str
_tarball_path: Path | None
@@ -180,7 +197,7 @@ def __init__(
self,
git_ref: str,
output_dir: str,
- tar_compression_format: _TarCompressionFormat = _TarCompressionFormat.xz,
+ tar_compression_format: TarCompressionFormat = TarCompressionFormat.xz,
):
"""Create the tarball during initialization.
@@ -201,7 +218,7 @@ def __init__(
self._create_tarball_dir()
self._tarball_name = (
- f"dpdk-tarball-{self._git_ref}.tar.{self._tar_compression_format.value}"
+ f"dpdk-tarball-{self._git_ref}.{self._tar_compression_format.extension}"
)
self._tarball_path = self._check_tarball_path()
if not self._tarball_path:
@@ -248,6 +265,72 @@ def __fspath__(self) -> str:
return str(self._tarball_path)
+def convert_to_list_of_string(value: Any | list[Any]) -> list[str]:
+ """Convert the input to the list of strings."""
+ return list(map(str, value) if isinstance(value, list) else str(value))
+
+
+def create_tarball(
+ dir_path: str | Path,
+ compress_format: TarCompressionFormat = TarCompressionFormat.none,
+ exclude: Any | list[Any] | None = None,
+) -> Path:
+ """Create a tarball from the contents of the specified directory.
+
+ This method creates a tarball containing all files and directories within `dir_path`.
+ The tarball will be saved in the directory of `dir_path` and will be named based on `dir_path`.
+
+ Args:
+ dir_path: The directory path.
+ compress_format: The compression format to use. Defaults to no compression.
+ exclude: Patterns for files or directories to exclude from the tarball.
+ These patterns are used with `fnmatch.fnmatch` to filter out files.
+
+ Returns:
+ The path to the created tarball.
+ """
+
+ def create_filter_function(exclude_patterns: str | list[str] | None) -> Callable | None:
+ """Create a filter function based on the provided exclude patterns.
+
+ Args:
+ exclude_patterns: Patterns for files or directories to exclude from the tarball.
+ These patterns are used with `fnmatch.fnmatch` to filter out files.
+
+ Returns:
+ The filter function that excludes files based on the patterns.
+ """
+ if exclude_patterns:
+ exclude_patterns = convert_to_list_of_string(exclude_patterns)
+
+ def filter_func(tarinfo: tarfile.TarInfo) -> tarfile.TarInfo | None:
+ file_name = os.path.basename(tarinfo.name)
+ if any(fnmatch.fnmatch(file_name, pattern) for pattern in exclude_patterns):
+ return None
+ return tarinfo
+
+ return filter_func
+ return None
+
+ target_tarball_path = Path(f"{dir_path}.{compress_format.extension}")
+ with tarfile.open(target_tarball_path, f"w:{compress_format.value}") as tar:
+ tar.add(dir_path, arcname=target_tarball_path.stem, filter=create_filter_function(exclude))
+
+ return target_tarball_path
+
+
+def extract_tarball(tar_path: str | Path):
+ """Extract the contents of a tarball.
+
+ The tarball will be extracted in the same path as `tar_path` parent path.
+
+ Args:
+ tar_path: The path to the tarball file to extract.
+ """
+ with tarfile.open(tar_path, "r") as tar:
+ tar.extractall(path=Path(tar_path).parent)
+
+
class PacketProtocols(Flag):
"""Flag specifying which protocols to use for packet generation."""
--
2.46.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 5/7] dts: add support for externally compiled DPDK
2024-09-27 16:08 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
` (3 preceding siblings ...)
2024-09-27 16:08 ` [PATCH 4/7] dts: add the ability to copy directories via remote Tomáš Ďurovec
@ 2024-09-27 16:08 ` Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 6/7] doc: update argument options for external DPDK build Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 7/7] dts: remove git ref option Tomáš Ďurovec
6 siblings, 0 replies; 9+ messages in thread
From: Tomáš Ďurovec @ 2024-09-27 16:08 UTC (permalink / raw)
To: dev, Luca.Vizzarro, probb, npratte, dmarx; +Cc: Tomáš Ďurovec
Add support for using DPDK source tree directory as well as DPDK
tarball with the pre-build directory that can user specify and
type of location, it can be stored in the local filesystem or SUT
node. Additionally, this can be set up with the config file or
cmd arguments/environment variables.
Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
---
dts/conf.yaml | 23 +-
dts/framework/config/__init__.py | 121 ++++++-
dts/framework/config/conf_yaml_schema.json | 62 +++-
dts/framework/config/types.py | 17 +-
dts/framework/exception.py | 4 +-
dts/framework/remote_session/dpdk_shell.py | 2 +-
dts/framework/runner.py | 8 +-
dts/framework/settings.py | 193 +++++++++--
dts/framework/test_result.py | 23 +-
dts/framework/testbed_model/node.py | 22 +-
dts/framework/testbed_model/os_session.py | 63 +++-
dts/framework/testbed_model/posix_session.py | 39 ++-
dts/framework/testbed_model/sut_node.py | 345 +++++++++++++------
13 files changed, 718 insertions(+), 204 deletions(-)
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 814744a1fc..2f3010204d 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -5,12 +5,23 @@
test_runs:
# define one test run environment
- dpdk_build:
- arch: x86_64
- os: linux
- cpu: native
- # the combination of the following two makes CC="ccache gcc"
- compiler: gcc
- compiler_wrapper: ccache
+ # dpdk_tree: Commented out because `tarball` is defined.
+ tarball: dpdk-tarball.tar.xz
+ # Either `dpdk_tree` or `tarball` can be defined, but not both.
+ remote: false # Optional, defaults to false. If it's true, the `dpdk_tree` or `tarball`
+ # is located on the SUT node, instead of the execution host.
+
+ # dir_name: Commented out because `build` is defined.
+ build:
+ arch: x86_64
+ os: linux
+ cpu: native
+ # the combination of the following two makes CC="ccache gcc"
+ compiler: gcc
+ compiler_wrapper: ccache # Optional.
+ # If `dir_name` is defined, DPDK has been pre-built and the build directory is located in a
+ # subdirectory of DPDK tree root directory. Otherwise, will be using a `build` to build the
+ # DPDK from source. Either `dir_name` or `build` can be defined, but not both.
perf: false # disable performance testing
func: true # enable functional testing
skip_smoke_tests: false # optional
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 49b2e8d016..1bbc1c8700 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -35,6 +35,7 @@
import json
import os.path
+import tarfile
from dataclasses import dataclass, fields
from enum import auto, unique
from pathlib import Path
@@ -47,6 +48,7 @@
from framework.config.types import (
ConfigurationDict,
DPDKBuildConfigDict,
+ DPDKConfigurationDict,
NodeConfigDict,
PortConfigDict,
TestRunConfigDict,
@@ -380,6 +382,115 @@ def from_dict(cls, d: DPDKBuildConfigDict) -> Self:
)
+@dataclass(slots=True, frozen=True)
+class DPDKLocation:
+ """DPDK location.
+
+ The path to the DPDK sources, build dir and type of location.
+
+ Attributes:
+ dpdk_tree: The path to the DPDK source tree directory. Only one of `dpdk_tree` or `tarball`
+ must be provided.
+ tarball: The path to the DPDK tarball. Only one of `dpdk_tree` or `tarball` must be
+ provided.
+ remote: Optional, defaults to :data:`False`. If :data:`True`, `dpdk_tree` or `tarball` is
+ located on the SUT node, instead of the execution host.
+ build_dir: If it's defined, DPDK has been pre-built and the build directory is located in a
+ subdirectory of `dpdk_tree` or `tarball` root directory. Otherwise, will be using a
+ `build` from configuration to build the DPDK from source.
+ """
+
+ dpdk_tree: str | None
+ tarball: str | None
+ remote: bool
+ build_dir: str | None
+
+ @classmethod
+ def from_dict(cls, d: DPDKConfigurationDict) -> Self:
+ """A convenience method that processes and validates the inputs before creating an instance.
+
+ Validate existence and format of `dpdk_tree` or `tarball` on local filesystem, if
+ `remote` is False.
+
+ Args:
+ d: The configuration dictionary.
+
+ Returns:
+ The DPDK location instance.
+
+ Raises:
+ ConfigurationError: If `dpdk_tree` or `tarball` not found in local filesystem or they
+ aren't in the right format.
+ """
+ dpdk_tree = d.get("dpdk_tree")
+ tarball = d.get("tarball")
+ remote = d.get("remote", False)
+
+ if not remote:
+ if dpdk_tree:
+ if not Path(dpdk_tree).exists():
+ raise ConfigurationError(
+ f"DPDK tree '{dpdk_tree}' not found in local filesystem."
+ )
+
+ if not Path(dpdk_tree).is_dir():
+ raise ConfigurationError(
+ f"DPDK tree '{dpdk_tree}' had not valid format, must be directory."
+ )
+
+ if tarball:
+ if not Path(tarball).exists():
+ raise ConfigurationError(
+ f"DPDK tarball '{tarball}' not found in local filesystem."
+ )
+
+ if not tarfile.is_tarfile(tarball):
+ raise ConfigurationError(
+ f"DPDK tarball '{tarball}' had not valid format, must be tar archive."
+ )
+
+ return cls(
+ dpdk_tree=dpdk_tree,
+ tarball=tarball,
+ remote=remote,
+ build_dir=d.get("dir_name"),
+ )
+
+
+@dataclass
+class DPDKConfiguration:
+ """The configuration of the DPDK build.
+
+ The configuration contain the location of the DPDK and configuration used for
+ building it.
+
+ Attributes:
+ dpdk_location: The location of the DPDK tree.
+ dpdk_build_config: A DPDK build configuration to test. If :data:`None`,
+ DTS will use pre-built DPDK from `build_dir` in a :dataclass:`DPDKLocation`.
+ """
+
+ dpdk_location: DPDKLocation
+ dpdk_build_config: DPDKBuildConfiguration | None
+
+ @classmethod
+ def from_dict(cls, d: DPDKConfigurationDict) -> Self:
+ """A convenience method that processes the inputs before creating an instance.
+
+ Args:
+ d: The configuration dictionary.
+
+ Returns:
+ The DPDK configuration.
+ """
+ return cls(
+ dpdk_location=DPDKLocation.from_dict(d),
+ dpdk_build_config=DPDKBuildConfiguration.from_dict(d["build"])
+ if d.get("build")
+ else None,
+ )
+
+
@dataclass(slots=True, frozen=True)
class DPDKBuildInfo:
"""Various versions and other information about a DPDK build.
@@ -389,8 +500,8 @@ class DPDKBuildInfo:
compiler_version: The version of the compiler used to build DPDK.
"""
- dpdk_version: str
- compiler_version: str
+ dpdk_version: str | None
+ compiler_version: str | None
@dataclass(slots=True, frozen=True)
@@ -437,7 +548,7 @@ class TestRunConfiguration:
and with what DPDK build.
Attributes:
- dpdk_build: A DPDK build to test.
+ dpdk_config: The DPDK configuration used to test.
perf: Whether to run performance tests.
func: Whether to run functional tests.
skip_smoke_tests: Whether to skip smoke tests.
@@ -448,7 +559,7 @@ class TestRunConfiguration:
random_seed: The seed to use for pseudo-random generation.
"""
- dpdk_build: DPDKBuildConfiguration
+ dpdk_config: DPDKConfiguration
perf: bool
func: bool
skip_smoke_tests: bool
@@ -498,7 +609,7 @@ def from_dict(
)
random_seed = d.get("random_seed", None)
return cls(
- dpdk_build=DPDKBuildConfiguration.from_dict(d["dpdk_build"]),
+ dpdk_config=DPDKConfiguration.from_dict(d["dpdk_build"]),
perf=d["perf"],
func=d["func"],
skip_smoke_tests=skip_smoke_tests,
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 94d7efa5f5..f5cd6d6075 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -110,9 +110,8 @@
"mscv"
]
},
- "dpdk_build": {
+ "build": {
"type": "object",
- "description": "DPDK build configuration supported by DTS.",
"properties": {
"arch": {
"type": "string",
@@ -133,7 +132,7 @@
"compiler": {
"$ref": "#/definitions/compiler"
},
- "compiler_wrapper": {
+ "compiler_wrapper": {
"type": "string",
"description": "This will be added before compiler to the CC variable when building DPDK. Optional."
}
@@ -146,6 +145,63 @@
"compiler"
]
},
+ "dpdk_build": {
+ "type": "object",
+ "description": "DPDK source and build configuration.",
+ "properties": {
+ "dpdk_tree": {
+ "type": "string",
+ "description": "The path to the DPDK source tree directory to test. Only one of `dpdk_tree` or `tarball` must be provided."
+ },
+ "tarball": {
+ "type": "string",
+ "description": "The path to the DPDK source tarball to test. Only one of `dpdk_tree` or `tarball` must be provided."
+ },
+ "remote": {
+ "type": "boolean",
+ "description": "Optional, defaults to false. If it's true, the `dpdk_tree` or `tarball` is located on the SUT node, instead of the execution host."
+ },
+ "dir_name": {
+ "type": "string",
+ "description": "If it's defined, DPDK has been pre-built and the build directory is located in a subdirectory of DPDK tree root directory. Otherwise, will be using a `build` to build the DPDK from source. Either this or `build` must be defined, but not both."
+ },
+ "build": {
+ "$ref": "#/definitions/build",
+ "description": "Either this or `dir_name` must be defined, but not both. DPDK build configuration supported by DTS."
+ }
+ },
+ "allOf": [
+ {
+ "oneOf": [
+ {
+ "required": [
+ "dpdk_tree"
+ ]
+ },
+ {
+ "required": [
+ "tarball"
+ ]
+ }
+ ]
+ },
+ {
+ "oneOf": [
+ {
+ "required": [
+ "dir_name"
+ ]
+ },
+ {
+ "required": [
+ "build"
+ ]
+ }
+ ]
+ }
+ ],
+ "additionalProperties": false
+ },
"hugepages_2mb": {
"type": "object",
"description": "Optional hugepage configuration. If not specified, hugepages won't be configured and DTS will use system configuration.",
diff --git a/dts/framework/config/types.py b/dts/framework/config/types.py
index a710c20d6a..24884381cc 100644
--- a/dts/framework/config/types.py
+++ b/dts/framework/config/types.py
@@ -86,6 +86,21 @@ class DPDKBuildConfigDict(TypedDict):
compiler_wrapper: str
+class DPDKConfigurationDict(TypedDict):
+ """Allowed keys and values."""
+
+ #:
+ dpdk_tree: str | None
+ #:
+ tarball: str | None
+ #:
+ remote: bool
+ #:
+ dir_name: str | None
+ #:
+ build: DPDKBuildConfigDict
+
+
class TestSuiteConfigDict(TypedDict):
"""Allowed keys and values."""
@@ -108,7 +123,7 @@ class TestRunConfigDict(TypedDict):
"""Allowed keys and values."""
#:
- dpdk_build: DPDKBuildConfigDict
+ dpdk_build: DPDKConfigurationDict
#:
perf: bool
#:
diff --git a/dts/framework/exception.py b/dts/framework/exception.py
index f45f789825..d967ede09b 100644
--- a/dts/framework/exception.py
+++ b/dts/framework/exception.py
@@ -184,8 +184,8 @@ class InteractiveCommandExecutionError(DTSError):
severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
-class RemoteDirectoryExistsError(DTSError):
- """A directory that exists on a remote node."""
+class RemoteFileNotFoundError(DTSError):
+ """A remote file or directory is requested but doesn’t exist."""
#:
severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
diff --git a/dts/framework/remote_session/dpdk_shell.py b/dts/framework/remote_session/dpdk_shell.py
index c5f5c2d116..b39132cc42 100644
--- a/dts/framework/remote_session/dpdk_shell.py
+++ b/dts/framework/remote_session/dpdk_shell.py
@@ -104,4 +104,4 @@ def _update_real_path(self, path: PurePath) -> None:
Adds the remote DPDK build directory to the path.
"""
- super()._update_real_path(self._node.remote_dpdk_build_dir.joinpath(path))
+ super()._update_real_path(PurePath(self._node.remote_dpdk_build_dir).joinpath(path))
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 100dd75adb..7d463c1fa1 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -414,15 +414,19 @@ def _run_test_run(
test_run_config: A test run configuration.
test_run_result: The test run's result.
test_suites_with_cases: The test suites with test cases to run.
+
+ Raises:
+ ConfigurationError: If the DPDK sources or build is not set up from config or settings.
"""
self._logger.info(
f"Running test run with SUT '{test_run_config.system_under_test_node.name}'."
)
test_run_result.add_sut_info(sut_node.node_info)
try:
- sut_node.set_up_test_run(test_run_config)
+ dpdk_location = SETTINGS.dpdk_location or test_run_config.dpdk_config.dpdk_location
+ sut_node.set_up_test_run(test_run_config, dpdk_location)
test_run_result.add_dpdk_build_info(sut_node.get_dpdk_build_info())
- tg_node.set_up_test_run(test_run_config)
+ tg_node.set_up_test_run(test_run_config, dpdk_location)
test_run_result.update_setup(Result.PASS)
except Exception as e:
self._logger.exception("Test run setup failed.")
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 52a1582d5c..17594ecb15 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -39,21 +39,36 @@
Set to any value to enable logging everything to the console.
-.. option:: -s, --skip-setup
-.. envvar:: DTS_SKIP_SETUP
+.. option:: --dpdk-tree
+.. envvar:: DTS_DPDK_TREE
- Set to any value to skip building DPDK.
+ The path to DPDK source tree directory to test. Only this or tarball or revision can be
+ provided.
.. option:: --tarball, --snapshot
.. envvar:: DTS_DPDK_TARBALL
- Path to DPDK source code tarball to test.
+ The path to DPDK source tarball to test. Only this or DPDK tree or revision can be provided.
.. option:: --revision, --rev, --git-ref
.. envvar:: DTS_DPDK_REVISION_ID
Git revision ID to test. Could be commit, tag, tree ID etc.
To test local changes, first commit them, then use their commit ID.
+ Only this or DPDK tree or tarball can be provided.
+
+.. option:: --remote-source
+.. envvar:: DTS_REMOTE_SOURCE
+
+ Set when the DPDK source tree or tarball is located on the SUT node, instead of the
+ execution host. This can be provided only with DPDK tree or tarball.
+
+.. option:: --build-dir
+.. envvar:: DTS_BUILD_DIR
+
+ A directory name. Optional, if it's defined, DPDK has been pre-built and the build directory
+ is located in a subdirectory of DPDK tree root directory. Otherwise, will be using a `build`
+ This can be provided only with DPDK tree or tarball.
.. option:: --test-suite
.. envvar:: DTS_TEST_SUITES
@@ -86,12 +101,13 @@
import argparse
import os
import sys
+import tarfile
from argparse import Action, ArgumentDefaultsHelpFormatter, _get_action_name
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable
-from .config import TestSuiteConfig
+from .config import DPDKLocation, TestSuiteConfig
from .exception import ConfigurationError
from .utils import DPDKGitTarball, get_commit_id
@@ -112,9 +128,7 @@ class Settings:
#:
verbose: bool = False
#:
- skip_setup: bool = False
- #:
- dpdk_tarball_path: Path | str = ""
+ dpdk_location: DPDKLocation | None = None
#:
compile_timeout: float = 1200
#:
@@ -242,14 +256,6 @@ def _get_help_string(self, action):
return help
-def _parse_tarball_path(file_path: str) -> Path:
- """Validate whether `file_path` is valid and return a Path object."""
- path = Path(file_path)
- if not path.exists() or not path.is_file():
- raise argparse.ArgumentTypeError("The file path provided is not a valid file")
- return path
-
-
def _parse_revision_id(rev_id: str) -> str:
"""Validate revision ID and retrieve corresponding commit ID."""
try:
@@ -258,6 +264,47 @@ def _parse_revision_id(rev_id: str) -> str:
raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")
+def _required_with_one_of(parser: _DTSArgumentParser, action: Action, *required_dests: str) -> None:
+ """Verify that `action` is listed together with at least one of `required_dests`.
+
+ Verify that when `action` is among the command-line arguments or
+ environment variables, at least one of `required_dests` is also among
+ the command-line arguments or environment variables.
+
+ Args:
+ parser: The custom ArgumentParser object which contains `action`.
+ action: The action to be verified.
+ *required_dests: Destination variable names of the required arguments.
+
+ Raises:
+ argparse.ArgumentTypeError: When none of the required_dest are defined.
+
+ Example:
+ We have ``--option1`` and we only want it to be a passed alongside
+ either ``--option2`` or ``--option3`` (meaning if ``--option1`` is
+ passed without either ``--option2`` or ``--option3``, that's an error).
+
+ parser = _DTSArgumentParser()
+ option1_arg = parser.add_argument('--option1', dest='option1', action='store_true')
+ option2_arg = parser.add_argument('--option2', dest='option2', action='store_true')
+ option2_arg = parser.add_argument('--option3', dest='option3', action='store_true')
+
+ _required_with_one_of(parser, option1_arg, 'option2', 'option3')
+ """
+ if _is_action_in_args(action):
+ for required_dest in required_dests:
+ required_action = parser.find_action(required_dest)
+ if required_action is None:
+ continue
+
+ if _is_action_in_args(required_action):
+ return None
+
+ raise argparse.ArgumentTypeError(
+ f"The '{action.dest}' is required at least with one of '{', '.join(required_dests)}'."
+ )
+
+
def _get_parser() -> _DTSArgumentParser:
"""Create the argument parser for DTS.
@@ -312,22 +359,29 @@ def _get_parser() -> _DTSArgumentParser:
)
_add_env_var_to_action(action)
- action = parser.add_argument(
- "-s",
- "--skip-setup",
- action="store_true",
- default=SETTINGS.skip_setup,
- help="Specify to skip all setup steps on SUT and TG nodes.",
+ dpdk_build = parser.add_argument_group(
+ "DPDK Build Options",
+ description="Arguments in this group (and subgroup) will be applied to a "
+ ":class:`DPDKLocation` when the DPDK tree, tarball or revision will be provided, "
+ "other arguments like remote source and build dir are optional. A :class:`DPDKLocation` "
+ "from settings are used instead of from config if construct successful.",
)
- _add_env_var_to_action(action)
- dpdk_source = parser.add_mutually_exclusive_group(required=True)
+ dpdk_source = dpdk_build.add_mutually_exclusive_group()
+ action = dpdk_source.add_argument(
+ "--dpdk-tree",
+ help="The path to DPDK source tree directory to test. Only this or tarball or revision "
+ "can be provided.",
+ metavar="DIR_PATH",
+ dest="dpdk_tree_path",
+ )
+ _add_env_var_to_action(action, "DPDK_TREE")
action = dpdk_source.add_argument(
"--tarball",
"--snapshot",
- type=_parse_tarball_path,
- help="Path to DPDK source code tarball to test.",
+ help="The path to DPDK source tarball to test. Only this or DPDK tree or revision "
+ "can be provided.",
metavar="FILE_PATH",
dest="dpdk_tarball_path",
)
@@ -339,12 +393,36 @@ def _get_parser() -> _DTSArgumentParser:
"--git-ref",
type=_parse_revision_id,
help="Git revision ID to test. Could be commit, tag, tree ID etc. "
- "To test local changes, first commit them, then use their commit ID.",
+ "To test local changes, first commit them, then use their commit ID."
+ "Only this or DPDK tree or tarball can be provided.",
metavar="ID",
dest="dpdk_revision_id",
)
_add_env_var_to_action(action)
+ action = dpdk_build.add_argument(
+ "--remote-source",
+ action="store_true",
+ default=False,
+ help="Optional. Set when the DPDK source tree or tarball is located on the SUT node, "
+ "instead of the execution host. This can be provided only with DPDK tree or tarball.",
+ )
+ _add_env_var_to_action(action)
+ _required_with_one_of(
+ parser, action, "dpdk_tarball_path", "dpdk_tree_path"
+ ) # ignored if passed with git-ref
+
+ action = dpdk_build.add_argument(
+ "--build-dir",
+ help="A directory name. Optional, if it's defined, DPDK has been pre-built and the build "
+ "directory is located in a subdirectory of DPDK tree root directory. Otherwise DPDK will "
+ "be built from scratch with DPDK build configuration. This can be provided only with DPDK "
+ "tree or tarball.",
+ metavar="DIR_NAME",
+ )
+ _add_env_var_to_action(action)
+ _required_with_one_of(parser, action, "dpdk_tarball_path", "dpdk_tree_path")
+
action = parser.add_argument(
"--compile-timeout",
default=SETTINGS.compile_timeout,
@@ -395,6 +473,64 @@ def _get_parser() -> _DTSArgumentParser:
return parser
+def _process_dpdk_location(
+ dpdk_tree: str | None,
+ tarball: str | None,
+ remote: bool,
+ build_dir: str | None,
+):
+ """Process and validate DPDK build arguments.
+
+ Ensures that either `dpdk_tree` or `tarball` is provided. Validate existence and format of
+ `dpdk_tree` or `tarball` on local filesystem, if `remote` is False. Constructs and returns
+ the :class:`DPDKLocation` with the provided parameters if validation is successful.
+
+ Args:
+ dpdk_tree: The path to the DPDK source tree directory. Only one of `dpdk_tree` or `tarball`
+ must be provided.
+ tarball: The path to the DPDK tarball. Only one of `dpdk_tree` or `tarball` must be
+ provided.
+ remote: If :data:`True`, `dpdk_tree` or `tarball` is located on the SUT node, instead of the
+ execution host.
+ build_dir: If it's defined, DPDK has been pre-built and the build directory is located in a
+ subdirectory of `dpdk_tree` or `tarball` root directory.
+
+ Returns:
+ A DPDK location if construction is successful, otherwise None.
+
+ Raises:
+ argparse.ArgumentTypeError: If `dpdk_tree` or `tarball` not found in local filesystem or
+ they aren't in the right format.
+ """
+ if not (dpdk_tree or tarball):
+ return None
+
+ if not remote:
+ if dpdk_tree:
+ if not Path(dpdk_tree).exists():
+ raise argparse.ArgumentTypeError(
+ f"DPDK tree '{dpdk_tree}' not found in local filesystem."
+ )
+
+ if not Path(dpdk_tree).is_dir():
+ raise argparse.ArgumentTypeError(
+ f"DPDK tree '{dpdk_tree}' had not valid format, must be directory."
+ )
+
+ if tarball:
+ if not Path(tarball).exists():
+ raise argparse.ArgumentTypeError(
+ f"DPDK tarball '{tarball}' not found in local filesystem."
+ )
+
+ if not tarfile.is_tarfile(tarball):
+ raise argparse.ArgumentTypeError(
+ f"DPDK tarball '{tarball}' had not valid format, must be tar archive."
+ )
+
+ return DPDKLocation(dpdk_tree=dpdk_tree, tarball=tarball, remote=remote, build_dir=build_dir)
+
+
def _process_test_suites(
parser: _DTSArgumentParser, args: list[list[str]]
) -> list[TestSuiteConfig]:
@@ -434,6 +570,9 @@ def get_settings() -> Settings:
if args.dpdk_revision_id:
args.dpdk_tarball_path = Path(DPDKGitTarball(args.dpdk_revision_id, args.output_dir))
+ args.dpdk_location = _process_dpdk_location(
+ args.dpdk_tree_path, args.dpdk_tarball_path, args.remote_source, args.build_dir
+ )
args.test_suites = _process_test_suites(parser, args.test_suites)
kwargs = {k: v for k, v in vars(args).items() if hasattr(SETTINGS, k)}
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 31560f6704..0a10723098 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -29,16 +29,7 @@
from types import FunctionType
from typing import Union
-from .config import (
- OS,
- Architecture,
- Compiler,
- CPUType,
- DPDKBuildInfo,
- NodeInfo,
- TestRunConfiguration,
- TestSuiteConfig,
-)
+from .config import DPDKBuildInfo, NodeInfo, TestRunConfiguration, TestSuiteConfig
from .exception import DTSError, ErrorSeverity
from .logger import DTSLogger
from .settings import SETTINGS
@@ -318,10 +309,6 @@ class TestRunResult(BaseResult):
The internal list stores the results of all test suites in a given test run.
Attributes:
- arch: The DPDK build architecture.
- os: The DPDK build operating system.
- cpu: The DPDK build CPU.
- compiler: The DPDK build compiler.
compiler_version: The DPDK build compiler version.
dpdk_version: The built DPDK version.
sut_os_name: The operating system of the SUT node.
@@ -329,10 +316,6 @@ class TestRunResult(BaseResult):
sut_kernel_version: The operating system kernel version of the SUT node.
"""
- arch: Architecture
- os: OS
- cpu: CPUType
- compiler: Compiler
compiler_version: str | None
dpdk_version: str | None
sut_os_name: str
@@ -348,10 +331,6 @@ def __init__(self, test_run_config: TestRunConfiguration):
test_run_config: A test run configuration.
"""
super().__init__()
- self.arch = test_run_config.dpdk_build.arch
- self.os = test_run_config.dpdk_build.os
- self.cpu = test_run_config.dpdk_build.cpu
- self.compiler = test_run_config.dpdk_build.compiler
self.compiler_version = None
self.dpdk_version = None
self._config = test_run_config
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
index 12a40170ac..f048b57ed5 100644
--- a/dts/framework/testbed_model/node.py
+++ b/dts/framework/testbed_model/node.py
@@ -15,12 +15,11 @@
from abc import ABC
from ipaddress import IPv4Interface, IPv6Interface
-from typing import Any, Callable, Union
+from typing import Union
-from framework.config import OS, NodeConfiguration, TestRunConfiguration
+from framework.config import OS, DPDKLocation, NodeConfiguration, TestRunConfiguration
from framework.exception import ConfigurationError
from framework.logger import DTSLogger, get_dts_logger
-from framework.settings import SETTINGS
from .cpu import (
LogicalCore,
@@ -95,7 +94,9 @@ def _init_ports(self) -> None:
for port in self.ports:
self.configure_port_state(port)
- def set_up_test_run(self, test_run_config: TestRunConfiguration) -> None:
+ def set_up_test_run(
+ self, test_run_config: TestRunConfiguration, dpdk_location: DPDKLocation
+ ) -> None:
"""Test run setup steps.
Configure hugepages on all DTS node types. Additional steps can be added by
@@ -104,6 +105,7 @@ def set_up_test_run(self, test_run_config: TestRunConfiguration) -> None:
Args:
test_run_config: A test run configuration according to which
the setup steps will be taken.
+ dpdk_location: The target source of the DPDK tree.
"""
self._setup_hugepages()
@@ -216,18 +218,6 @@ def close(self) -> None:
for session in self._other_sessions:
session.close()
- @staticmethod
- def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
- """Skip the decorated function.
-
- The :option:`--skip-setup` command line argument and the :envvar:`DTS_SKIP_SETUP`
- environment variable enable the decorator.
- """
- if SETTINGS.skip_setup:
- return lambda *args: None
- else:
- return func
-
def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession:
"""Factory for OS-aware sessions.
diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
index 6c3f84dec1..6194ddb989 100644
--- a/dts/framework/testbed_model/os_session.py
+++ b/dts/framework/testbed_model/os_session.py
@@ -137,17 +137,6 @@ def _get_privileged_command(command: str) -> str:
The modified command that executes with administrative privileges.
"""
- @abstractmethod
- def guess_dpdk_remote_dir(self, remote_dir: str | PurePath) -> PurePath:
- """Try to find DPDK directory in `remote_dir`.
-
- The directory is the one which is created after the extraction of the tarball. The files
- are usually extracted into a directory starting with ``dpdk-``.
-
- Returns:
- The absolute path of the DPDK remote directory, empty path if not found.
- """
-
@abstractmethod
def get_remote_tmp_dir(self) -> PurePath:
"""Get the path of the temporary directory of the remote OS.
@@ -177,6 +166,17 @@ def join_remote_path(self, *args: str | PurePath) -> PurePath:
The resulting joined path.
"""
+ @abstractmethod
+ def remote_path_exists(self, remote_path: str | PurePath) -> bool:
+ """Check whether `remote_path` exists on the remote system.
+
+ Args:
+ remote_path: The path to check.
+
+ Returns:
+ :data:`True` if the path exists, :data:`False` otherwise.
+ """
+
@abstractmethod
def copy_from(self, source_file: str | PurePath, destination_dir: str | Path) -> None:
"""Copy a file from the remote node to the local filesystem.
@@ -344,6 +344,47 @@ def extract_remote_tarball(
the archive.
"""
+ @abstractmethod
+ def is_remote_dir(self, remote_path: str) -> bool:
+ """Check if the `remote_path` is a directory.
+
+ Args:
+ remote_tarball_path: The path to the remote tarball.
+
+ Returns:
+ If :data:`True` the `remote_path` is a directory, otherwise :data:`False`.
+ """
+
+ @abstractmethod
+ def is_remote_tarfile(self, remote_tarball_path: str) -> bool:
+ """Check if the `remote_tarball_path` is a tar archive.
+
+ Args:
+ remote_tarball_path: The path to the remote tarball.
+
+ Returns:
+ If :data:`True` the `remote_tarball_path` is a tar archive, otherwise :data:`False`.
+ """
+
+ @abstractmethod
+ def get_tarball_top_dir(
+ self, remote_tarball_path: str | PurePath
+ ) -> str | PurePosixPath | None:
+ """Get the top directory of the remote tarball.
+
+ Examines the contents of a tarball located at the given `remote_tarball_path` and
+ determines the top-level directory. If all files and directories in the tarball share
+ the same top-level directory, that directory name is returned. If the tarball contains
+ multiple top-level directories or is empty, the method return None.
+
+ Args:
+ remote_tarball_path: The path to the remote tarball.
+
+ Returns:
+ The top directory of the tarball. If there are multiple top directories
+ or the tarball is empty, returns :data:`None`.
+ """
+
@abstractmethod
def build_dpdk(
self,
diff --git a/dts/framework/testbed_model/posix_session.py b/dts/framework/testbed_model/posix_session.py
index 94e721da61..5ab7c18fb7 100644
--- a/dts/framework/testbed_model/posix_session.py
+++ b/dts/framework/testbed_model/posix_session.py
@@ -91,6 +91,11 @@ def join_remote_path(self, *args: str | PurePath) -> PurePosixPath:
"""Overrides :meth:`~.os_session.OSSession.join_remote_path`."""
return PurePosixPath(*args)
+ def remote_path_exists(self, remote_path: str | PurePath) -> bool:
+ """Overrides :meth:`~.os_session.OSSession.remote_path_exists`."""
+ result = self.send_command(f"test -e {remote_path}")
+ return not result.return_code
+
def copy_from(self, source_file: str | PurePath, destination_dir: str | Path) -> None:
"""Overrides :meth:`~.os_session.OSSession.copy_from`."""
self.remote_session.copy_from(source_file, destination_dir)
@@ -196,6 +201,32 @@ def extract_remote_tarball(
if expected_dir:
self.send_command(f"ls {expected_dir}", verify=True)
+ def is_remote_dir(self, remote_path: str) -> bool:
+ """Overrides :meth:`~.os_session.OSSession.is_remote_dir`."""
+ result = self.send_command(f"test -d {remote_path}")
+ return not result.return_code
+
+ def is_remote_tarfile(self, remote_tarball_path: str) -> bool:
+ """Overrides :meth:`~.os_session.OSSession.is_remote_tarfile`."""
+ result = self.send_command(f"tar -tvf {remote_tarball_path}")
+ return not result.return_code
+
+ def get_tarball_top_dir(
+ self, remote_tarball_path: str | PurePath
+ ) -> str | PurePosixPath | None:
+ """Overrides :meth:`~.os_session.OSSession.get_tarball_top_dir`."""
+ members = self.send_command(f"tar tf {remote_tarball_path}").stdout.split()
+
+ top_dirs = []
+ for member in members:
+ parts_of_member = PurePosixPath(member).parts
+ if parts_of_member:
+ top_dirs.append(parts_of_member[0])
+
+ if len(set(top_dirs)) == 1:
+ return top_dirs[0]
+ return None
+
def build_dpdk(
self,
env_vars: dict,
@@ -301,7 +332,7 @@ def _get_dpdk_pids(self, dpdk_runtime_dirs: Iterable[str | PurePath]) -> list[in
pid_regex = r"p(\d+)"
for dpdk_runtime_dir in dpdk_runtime_dirs:
dpdk_config_file = PurePosixPath(dpdk_runtime_dir, "config")
- if self._remote_files_exists(dpdk_config_file):
+ if self.remote_path_exists(dpdk_config_file):
out = self.send_command(f"lsof -Fp {dpdk_config_file}").stdout
if out and "No such file or directory" not in out:
for out_line in out.splitlines():
@@ -310,10 +341,6 @@ def _get_dpdk_pids(self, dpdk_runtime_dirs: Iterable[str | PurePath]) -> list[in
pids.append(int(match.group(1)))
return pids
- def _remote_files_exists(self, remote_path: PurePath) -> bool:
- result = self.send_command(f"test -e {remote_path}")
- return not result.return_code
-
def _check_dpdk_hugepages(self, dpdk_runtime_dirs: Iterable[str | PurePath]) -> None:
"""Check there aren't any leftover hugepages.
@@ -325,7 +352,7 @@ def _check_dpdk_hugepages(self, dpdk_runtime_dirs: Iterable[str | PurePath]) ->
"""
for dpdk_runtime_dir in dpdk_runtime_dirs:
hugepage_info = PurePosixPath(dpdk_runtime_dir, "hugepage_info")
- if self._remote_files_exists(hugepage_info):
+ if self.remote_path_exists(hugepage_info):
out = self.send_command(f"lsof -Fp {hugepage_info}").stdout
if out and "No such file or directory" not in out:
self._logger.warning("Some DPDK processes did not free hugepages.")
diff --git a/dts/framework/testbed_model/sut_node.py b/dts/framework/testbed_model/sut_node.py
index 9bfb91816e..a84129d86b 100644
--- a/dts/framework/testbed_model/sut_node.py
+++ b/dts/framework/testbed_model/sut_node.py
@@ -13,20 +13,20 @@
import os
-import tarfile
import time
from pathlib import PurePath
from framework.config import (
DPDKBuildConfiguration,
DPDKBuildInfo,
+ DPDKLocation,
NodeInfo,
SutNodeConfiguration,
TestRunConfiguration,
)
+from framework.exception import ConfigurationError, RemoteFileNotFoundError
from framework.params.eal import EalParams
from framework.remote_session.remote_session import CommandResult
-from framework.settings import SETTINGS
from framework.utils import MesonArgs
from .node import Node
@@ -39,14 +39,13 @@ class SutNode(Node):
The SUT node extends :class:`Node` with DPDK specific features:
- * DPDK build,
+ * Managing DPDK source tree on the remote SUT,
+ * Building the DPDK from source or using a pre-built version,
* Gathering of DPDK build info,
* The running of DPDK apps, interactively or one-time execution,
* DPDK apps cleanup.
- The :option:`--tarball` command line argument and the :envvar:`DTS_DPDK_TARBALL`
- environment variable configure the path to the DPDK tarball
- or the git commit ID, tag ID or tree ID to test.
+ Building DPDK from source uses `build` configuration inside `dpdk_build` of configuration.
Attributes:
config: The SUT node configuration.
@@ -57,10 +56,10 @@ class SutNode(Node):
virtual_devices: list[VirtualDevice]
dpdk_prefix_list: list[str]
dpdk_timestamp: str
- _dpdk_build_config: DPDKBuildConfiguration | None
_env_vars: dict
_remote_tmp_dir: PurePath
- __remote_dpdk_dir: PurePath | None
+ __remote_dpdk_tree_path: str | PurePath | None
+ _remote_dpdk_build_dir: PurePath | None
_app_compile_timeout: float
_dpdk_kill_session: OSSession | None
_dpdk_version: str | None
@@ -77,10 +76,10 @@ def __init__(self, node_config: SutNodeConfiguration):
super().__init__(node_config)
self.virtual_devices = []
self.dpdk_prefix_list = []
- self._dpdk_build_config = None
self._env_vars = {}
self._remote_tmp_dir = self.main_session.get_remote_tmp_dir()
- self.__remote_dpdk_dir = None
+ self.__remote_dpdk_tree_path = None
+ self._remote_dpdk_build_dir = None
self._app_compile_timeout = 90
self._dpdk_kill_session = None
self.dpdk_timestamp = (
@@ -93,40 +92,34 @@ def __init__(self, node_config: SutNodeConfiguration):
self._logger.info(f"Created node: {self.name}")
@property
- def _remote_dpdk_dir(self) -> PurePath:
- """The remote DPDK dir.
-
- This internal property should be set after extracting the DPDK tarball. If it's not set,
- that implies the DPDK setup step has been skipped, in which case we can guess where
- a previous build was located.
- """
- if self.__remote_dpdk_dir is None:
- self.__remote_dpdk_dir = self._guess_dpdk_remote_dir()
- return self.__remote_dpdk_dir
-
- @_remote_dpdk_dir.setter
- def _remote_dpdk_dir(self, value: PurePath) -> None:
- self.__remote_dpdk_dir = value
+ def _remote_dpdk_tree_path(self) -> str | PurePath:
+ """The remote DPDK tree path."""
+ if self.__remote_dpdk_tree_path:
+ return self.__remote_dpdk_tree_path
+
+ self._logger.warning(
+ "Failed to get remote dpdk tree path because we don't know the "
+ "location on the SUT node."
+ )
+ return ""
@property
- def remote_dpdk_build_dir(self) -> PurePath:
- """The remote DPDK build directory.
-
- This is the directory where DPDK was built.
- We assume it was built in a subdirectory of the extracted tarball.
- """
- if self._dpdk_build_config:
- return self.main_session.join_remote_path(
- self._remote_dpdk_dir, self._dpdk_build_config.name
- )
- else:
- return self.main_session.join_remote_path(self._remote_dpdk_dir, "build")
+ def remote_dpdk_build_dir(self) -> str | PurePath:
+ """The remote DPDK build dir path."""
+ if self._remote_dpdk_build_dir:
+ return self._remote_dpdk_build_dir
+
+ self._logger.warning(
+ "Failed to get remote dpdk build dir because we don't know the "
+ "location on the SUT node."
+ )
+ return ""
@property
- def dpdk_version(self) -> str:
+ def dpdk_version(self) -> str | None:
"""Last built DPDK version."""
if self._dpdk_version is None:
- self._dpdk_version = self.main_session.get_dpdk_version(self._remote_dpdk_dir)
+ self._dpdk_version = self.main_session.get_dpdk_version(self._remote_dpdk_tree_path)
return self._dpdk_version
@property
@@ -137,26 +130,28 @@ def node_info(self) -> NodeInfo:
return self._node_info
@property
- def compiler_version(self) -> str:
+ def compiler_version(self) -> str | None:
"""The node's compiler version."""
if self._compiler_version is None:
- if self._dpdk_build_config is not None:
- self._compiler_version = self.main_session.get_compiler_version(
- self._dpdk_build_config.compiler.name
- )
- else:
- self._logger.warning(
- "Failed to get compiler version because _dpdk_build_config is None."
- )
- return ""
+ self._logger.warning("The `complier_version` is None because of using pre-built DPDK.")
+
return self._compiler_version
+ @compiler_version.setter
+ def compiler_version(self, value: str) -> None:
+ """Set the `compiler_version` used on the SUT node.
+
+ Args:
+ value: The node's compiler version.
+ """
+ self._compiler_version = value
+
@property
- def path_to_devbind_script(self) -> PurePath:
+ def path_to_devbind_script(self) -> PurePath | str:
"""The path to the dpdk-devbind.py script on the node."""
if self._path_to_devbind_script is None:
self._path_to_devbind_script = self.main_session.join_remote_path(
- self._remote_dpdk_dir, "usertools", "dpdk-devbind.py"
+ self._remote_dpdk_tree_path, "usertools", "dpdk-devbind.py"
)
return self._path_to_devbind_script
@@ -168,101 +163,247 @@ def get_dpdk_build_info(self) -> DPDKBuildInfo:
"""
return DPDKBuildInfo(dpdk_version=self.dpdk_version, compiler_version=self.compiler_version)
- def _guess_dpdk_remote_dir(self) -> PurePath:
- return self.main_session.guess_dpdk_remote_dir(self._remote_tmp_dir)
+ def set_up_test_run(
+ self, test_run_config: TestRunConfiguration, dpdk_location: DPDKLocation
+ ) -> None:
+ """Extend the test run setup with vdev config and DPDK build set up.
- def set_up_test_run(self, test_run_config: TestRunConfiguration) -> None:
- """Extend the test run setup with vdev config.
+ This method extends the setup process by configuring virtual devices and preparing the DPDK
+ environment based on the provided configuration.
Args:
test_run_config: A test run configuration according to which
the setup steps will be taken.
+ dpdk_location: The target source of the DPDK tree.
"""
- super().set_up_test_run(test_run_config)
+ super().set_up_test_run(test_run_config, dpdk_location)
for vdev in test_run_config.vdevs:
self.virtual_devices.append(VirtualDevice(vdev))
- self._set_up_dpdk(test_run_config.dpdk_build)
+ self._set_up_dpdk(dpdk_location, test_run_config.dpdk_config.dpdk_build_config)
def tear_down_test_run(self) -> None:
- """Extend the test run teardown with virtual device teardown."""
+ """Extend the test run teardown with virtual device teardown and DPDK teardown."""
super().tear_down_test_run()
self.virtual_devices = []
self._tear_down_dpdk()
- def _set_up_dpdk(self, dpdk_build_config: DPDKBuildConfiguration) -> None:
+ def _set_up_dpdk(
+ self, dpdk_location: DPDKLocation, dpdk_build_config: DPDKBuildConfiguration | None
+ ) -> None:
"""Set up DPDK the SUT node and bind ports.
- DPDK setup includes setting all internals needed for the build, the copying of DPDK tarball
- and then building DPDK. The drivers are bound to those that DPDK needs.
+ DPDK setup includes setting all internals needed for the build, the copying of DPDK
+ sources and then building DPDK or used the exist ones from the `dpdk_location`. The drivers
+ are bound to those that DPDK needs.
Args:
- dpdk_build_config: The DPDK build test run configuration according to which
- the setup steps will be taken.
+ dpdk_location: The location of the DPDK tree.
+ dpdk_build_config: A DPDK build configuration to test. If :data:`None`,
+ DTS will use pre-built DPDK from a :dataclass:`DPDKLocation`.
"""
- self._configure_dpdk_build(dpdk_build_config)
- self._copy_dpdk_tarball()
- self._build_dpdk()
+ self._set_remote_dpdk_tree_path(dpdk_location.dpdk_tree, dpdk_location.remote)
+ if not self._remote_dpdk_tree_path:
+ if dpdk_location.dpdk_tree:
+ self._copy_dpdk_tree(dpdk_location.dpdk_tree)
+ elif dpdk_location.tarball:
+ self._prepare_and_extract_dpdk_tarball(dpdk_location.tarball, dpdk_location.remote)
+
+ self._set_remote_dpdk_build_dir(dpdk_location.build_dir)
+ if not self.remote_dpdk_build_dir and dpdk_build_config:
+ self._configure_dpdk_build(dpdk_build_config)
+ self._build_dpdk()
+
self.bind_ports_to_driver()
def _tear_down_dpdk(self) -> None:
"""Reset DPDK variables and bind port driver to the OS driver."""
self._env_vars = {}
- self._dpdk_build_config = None
- self.__remote_dpdk_dir = None
+ self.__remote_dpdk_tree_path = None
+ self._remote_dpdk_build_dir = None
self._dpdk_version = None
- self._compiler_version = None
+ self.compiler_version = None
self.bind_ports_to_driver(for_dpdk=False)
+ def _set_remote_dpdk_tree_path(self, dpdk_tree: str | None, remote: bool):
+ """Set the path to the remote DPDK source tree based on the provided DPDK location.
+
+ If :data:`dpdk_tree` and :data:`remote` is defined, check existence of :data:`dpdk_tree`
+ on SUT node and sets the `_remote_dpdk_tree_path` property. Otherwise, sets nothing.
+
+ Verify DPDK source tree existence on the SUT node, if exists sets the
+ `_remote_dpdk_tree_path` property, otherwise sets nothing.
+
+ Args:
+ dpdk_tree: The path to the DPDK source tree directory.
+ remote: Indicates whether the `dpdk_tree` is already on the SUT node, instead of the
+ execution host.
+
+ Raises:
+ RemoteFileNotFoundError: If the DPDK source tree is expected to be on the SUT node but
+ is not found.
+ """
+ if remote and dpdk_tree:
+ if not self.main_session.remote_path_exists(dpdk_tree):
+ raise RemoteFileNotFoundError(
+ f"Remote DPDK source tree '{dpdk_tree}' not found in SUT node."
+ )
+ if not self.main_session.is_remote_dir(dpdk_tree):
+ raise ConfigurationError(
+ f"Remote DPDK source tree '{dpdk_tree}' had not valid format, must be "
+ "directory."
+ )
+
+ self.__remote_dpdk_tree_path = PurePath(dpdk_tree)
+
+ def _copy_dpdk_tree(self, dpdk_tree_path: str) -> None:
+ """Copy the DPDK source tree to the SUT.
+
+ Args:
+ dpdk_tree_path: The path to DPDK source tree on local filesystem.
+ """
+ self._logger.info(
+ f"Copying DPDK source tree to SUT: '{dpdk_tree_path}' into '{self._remote_tmp_dir}'."
+ )
+ self.main_session.copy_dir_to(dpdk_tree_path, self._remote_tmp_dir, exclude=".git")
+
+ self.__remote_dpdk_tree_path = self.main_session.join_remote_path(
+ self._remote_tmp_dir, PurePath(dpdk_tree_path).name
+ )
+
+ def _prepare_and_extract_dpdk_tarball(self, dpdk_tarball: str, remote: bool) -> None:
+ """Ensure the DPDK tarball is available on the SUT node and extract it.
+
+ This method ensures that the DPDK source tree tarball is available on the
+ SUT node. If the `dpdk_tarball` is local, it is copied to the SUT node. If the
+ `dpdk_tarball` is already on the SUT node, it verifies its existence.
+ The `dpdk_tarball` is then extracted on the SUT node.
+
+ This method sets the `_remote_dpdk_tree_path` property to the path of the
+ extracted DPDK tree on the SUT node.
+
+ Args:
+ dpdk_tarball: The path to the DPDK tarball, either locally or on the SUT node.
+ remote: Indicates whether the `dpdk_tarball` is already on the SUT node, instead of the
+ execution host.
+
+ Raises:
+ RemoteFileNotFoundError: If the `dpdk_tarball` is expected to be on the SUT node but
+ is not found.
+ """
+
+ def remove_tarball_suffix(remote_tarball_path: PurePath) -> PurePath:
+ """Remove the tarball suffix from the path.
+
+ Args:
+ remote_tarball_path: The path to the remote tarball.
+
+ Returns:
+ The path without the tarball suffix.
+ """
+ if len(remote_tarball_path.suffixes) > 1:
+ if remote_tarball_path.suffixes[-2] == ".tar":
+ suffixes_to_remove = "".join(remote_tarball_path.suffixes[-2:])
+ return PurePath(str(remote_tarball_path).replace(suffixes_to_remove, ""))
+ return remote_tarball_path.with_suffix("")
+
+ if remote:
+ if not self.main_session.remote_path_exists(dpdk_tarball):
+ raise RemoteFileNotFoundError(
+ f"Remote DPDK tarball '{dpdk_tarball}' not found in SUT."
+ )
+ if not self.main_session.is_remote_tarfile(dpdk_tarball):
+ raise ConfigurationError(
+ f"Remote DPDK tarball '{dpdk_tarball}' had not valid format, must be tar "
+ "archive."
+ )
+
+ remote_tarball_path = PurePath(dpdk_tarball)
+ else:
+ self._logger.info(
+ f"Copying DPDK tarball to SUT: '{dpdk_tarball}' into '{self._remote_tmp_dir}'."
+ )
+ self.main_session.copy_to(dpdk_tarball, self._remote_tmp_dir)
+
+ remote_tarball_path = self.main_session.join_remote_path(
+ self._remote_tmp_dir, PurePath(dpdk_tarball).name
+ )
+
+ tarball_top_dir = self.main_session.get_tarball_top_dir(remote_tarball_path)
+ self.__remote_dpdk_tree_path = self.main_session.join_remote_path(
+ PurePath(remote_tarball_path).parent,
+ tarball_top_dir or remove_tarball_suffix(remote_tarball_path),
+ )
+
+ self._logger.info(
+ "Extracting DPDK tarball on SUT: "
+ f"'{remote_tarball_path}' into '{self._remote_dpdk_tree_path}'."
+ )
+ self.main_session.extract_remote_tarball(
+ remote_tarball_path,
+ self._remote_dpdk_tree_path,
+ )
+
+ def _set_remote_dpdk_build_dir(self, build_dir: str | None):
+ """Set the `remote_dpdk_build_dir` on the SUT.
+
+ If :data:`build_dir` is defined, check existence on the SUT node and sets the
+ `remote_dpdk_build_dir` property by joining the `_remote_dpdk_tree_path` and `build_dir`.
+ Otherwise, sets nothing.
+
+ Args:
+ build_dir: If it's defined, DPDK has been pre-built and the build directory is located
+ in a subdirectory of `dpdk_tree` or `tarball` root directory.
+
+ Raises:
+ RemoteFileNotFoundError: If the `build_dir` is expected but does not exist on the SUT
+ node.
+ """
+ if build_dir:
+ remote_dpdk_build_dir = self.main_session.join_remote_path(
+ self._remote_dpdk_tree_path, build_dir
+ )
+ if not self.main_session.remote_path_exists(remote_dpdk_build_dir):
+ raise RemoteFileNotFoundError(
+ f"Remote DPDK build dir '{remote_dpdk_build_dir}' not found in SUT node."
+ )
+
+ self._remote_dpdk_build_dir = PurePath(remote_dpdk_build_dir)
+
def _configure_dpdk_build(self, dpdk_build_config: DPDKBuildConfiguration) -> None:
- """Populate common environment variables and set DPDK build config."""
+ """Populate common environment variables and set the DPDK build related properties.
+
+ This method sets `compiler_version` for additional information and `remote_dpdk_build_dir`
+ from DPDK build config name.
+
+ Args:
+ dpdk_build_config: A DPDK build configuration to test.
+ """
self._env_vars = {}
- self._dpdk_build_config = dpdk_build_config
self._env_vars.update(self.main_session.get_dpdk_build_env_vars(dpdk_build_config.arch))
self._env_vars["CC"] = dpdk_build_config.compiler.name
if dpdk_build_config.compiler_wrapper:
- self._env_vars["CC"] = f"'{self._dpdk_build_config.compiler_wrapper} "
- f"{self._dpdk_build_config.compiler.name}'"
-
- @Node.skip_setup
- def _copy_dpdk_tarball(self) -> None:
- """Copy to and extract DPDK tarball on the SUT node."""
- self._logger.info("Copying DPDK tarball to SUT.")
- self.main_session.copy_to(SETTINGS.dpdk_tarball_path, self._remote_tmp_dir)
-
- # construct remote tarball path
- # the basename is the same on local host and on remote Node
- remote_tarball_path = self.main_session.join_remote_path(
- self._remote_tmp_dir, os.path.basename(SETTINGS.dpdk_tarball_path)
- )
+ self._env_vars[
+ "CC"
+ ] = f"'{dpdk_build_config.compiler_wrapper} {dpdk_build_config.compiler.name}'"
- # construct remote path after extracting
- with tarfile.open(SETTINGS.dpdk_tarball_path) as dpdk_tar:
- dpdk_top_dir = dpdk_tar.getnames()[0]
- self._remote_dpdk_dir = self.main_session.join_remote_path(
- self._remote_tmp_dir, dpdk_top_dir
+ self.compiler_version = self.main_session.get_compiler_version(
+ dpdk_build_config.compiler.name
)
- self._logger.info(
- f"Extracting DPDK tarball on SUT: "
- f"'{remote_tarball_path}' into '{self._remote_dpdk_dir}'."
+ self._remote_dpdk_build_dir = self.main_session.join_remote_path(
+ self._remote_dpdk_tree_path, dpdk_build_config.name
)
- # clean remote path where we're extracting
- self.main_session.remove_remote_dir(self._remote_dpdk_dir)
-
- # then extract to remote path
- self.main_session.extract_remote_tarball(remote_tarball_path, self._remote_dpdk_dir)
- @Node.skip_setup
def _build_dpdk(self) -> None:
"""Build DPDK.
- Uses the already configured target. Assumes that the tarball has
- already been copied to and extracted on the SUT node.
+ Uses the already configured DPDK build configuration. Assumes that the
+ `_remote_dpdk_tree_path` has already been set on the SUT node.
"""
self.main_session.build_dpdk(
self._env_vars,
MesonArgs(default_library="static", enable_kmods=True, libdir="lib"),
- self._remote_dpdk_dir,
+ self._remote_dpdk_tree_path,
self.remote_dpdk_build_dir,
)
@@ -285,7 +426,7 @@ def build_dpdk_app(self, app_name: str, **meson_dpdk_args: str | bool) -> PurePa
self._env_vars,
MesonArgs(examples=app_name, **meson_dpdk_args), # type: ignore [arg-type]
# ^^ https://github.com/python/mypy/issues/11583
- self._remote_dpdk_dir,
+ self._remote_dpdk_tree_path,
self.remote_dpdk_build_dir,
rebuild=True,
timeout=self._app_compile_timeout,
--
2.46.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 6/7] doc: update argument options for external DPDK build
2024-09-27 16:08 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
` (4 preceding siblings ...)
2024-09-27 16:08 ` [PATCH 5/7] dts: add support for externally compiled DPDK Tomáš Ďurovec
@ 2024-09-27 16:08 ` Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 7/7] dts: remove git ref option Tomáš Ďurovec
6 siblings, 0 replies; 9+ messages in thread
From: Tomáš Ďurovec @ 2024-09-27 16:08 UTC (permalink / raw)
To: dev, Luca.Vizzarro, probb, npratte, dmarx; +Cc: Tomáš Ďurovec
By adding support for external build, we extend the
argument documentation for supported options.
Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
---
doc/guides/tools/dts.rst | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 65cce9e5ed..20d4d18b18 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -236,12 +236,14 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
-t SECONDS, --timeout SECONDS
[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
- -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
+ --dpdk-tree DIR_PATH [DTS_DPDK_TREE] Path to DPDK source code tree to test. (default: None)
--tarball FILE_PATH, --snapshot FILE_PATH
[DTS_DPDK_TARBALL] Path to DPDK source code tarball to test. (default: None)
--revision ID, --rev ID, --git-ref ID
[DTS_DPDK_REVISION_ID] Git revision ID to test. Could be commit, tag, tree ID etc. To test local changes, first
commit them, then use their commit ID. (default: None)
+ --remote-source [DTS_REMOTE_SOURCE] Set when the DPDK source tree or tarball is located on the SUT node. (default: False)
+ --build-dir DIR_NAME [DTS_BUILD_DIR] A directory name, which would be located in the `dpdk tree` or `tarball`. (default: None)
--compile-timeout SECONDS
[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
--test-suite TEST_SUITE [TEST_CASES ...]
@@ -257,8 +259,9 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
The brackets contain the names of environment variables that set the same thing.
-The minimum DTS needs is a config file and a DPDK tarball or git ref ID.
-You may pass those to DTS using the command line arguments or use the default paths.
+The minimum DTS needs is a config file and a pre-built DPDK or DPDK
+sources location which can be specified in said config file or on the
+command line or environment variables.
Example command for running DTS with the template configuration and DPDK tag v23.11:
--
2.46.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 7/7] dts: remove git ref option
2024-09-27 16:08 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
` (5 preceding siblings ...)
2024-09-27 16:08 ` [PATCH 6/7] doc: update argument options for external DPDK build Tomáš Ďurovec
@ 2024-09-27 16:08 ` Tomáš Ďurovec
6 siblings, 0 replies; 9+ messages in thread
From: Tomáš Ďurovec @ 2024-09-27 16:08 UTC (permalink / raw)
To: dev, Luca.Vizzarro, probb, npratte, dmarx; +Cc: Tomáš Ďurovec
In the previous commits we're adding the support for copying
the whole local DPDK tree directory and git-ref option was
meant to do the same thing.
Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
---
doc/guides/tools/dts.rst | 9 ---
dts/framework/settings.py | 51 +++-------------
dts/framework/utils.py | 119 +-------------------------------------
3 files changed, 8 insertions(+), 171 deletions(-)
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 20d4d18b18..d806dce2ae 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -239,9 +239,6 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
--dpdk-tree DIR_PATH [DTS_DPDK_TREE] Path to DPDK source code tree to test. (default: None)
--tarball FILE_PATH, --snapshot FILE_PATH
[DTS_DPDK_TARBALL] Path to DPDK source code tarball to test. (default: None)
- --revision ID, --rev ID, --git-ref ID
- [DTS_DPDK_REVISION_ID] Git revision ID to test. Could be commit, tag, tree ID etc. To test local changes, first
- commit them, then use their commit ID. (default: None)
--remote-source [DTS_REMOTE_SOURCE] Set when the DPDK source tree or tarball is located on the SUT node. (default: False)
--build-dir DIR_NAME [DTS_BUILD_DIR] A directory name, which would be located in the `dpdk tree` or `tarball`. (default: None)
--compile-timeout SECONDS
@@ -263,12 +260,6 @@ The minimum DTS needs is a config file and a pre-built DPDK or DPDK
sources location which can be specified in said config file or on the
command line or environment variables.
-Example command for running DTS with the template configuration and DPDK tag v23.11:
-
-.. code-block:: console
-
- (dts-py3.10) $ ./main.py --git-ref v23.11
-
DTS Results
~~~~~~~~~~~
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 17594ecb15..a4ab674189 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -42,20 +42,12 @@
.. option:: --dpdk-tree
.. envvar:: DTS_DPDK_TREE
- The path to DPDK source tree directory to test. Only this or tarball or revision can be
- provided.
+ The path to DPDK source tree directory to test. Only this or tarball can be provided.
.. option:: --tarball, --snapshot
.. envvar:: DTS_DPDK_TARBALL
- The path to DPDK source tarball to test. Only this or DPDK tree or revision can be provided.
-
-.. option:: --revision, --rev, --git-ref
-.. envvar:: DTS_DPDK_REVISION_ID
-
- Git revision ID to test. Could be commit, tag, tree ID etc.
- To test local changes, first commit them, then use their commit ID.
- Only this or DPDK tree or tarball can be provided.
+ The path to DPDK source tarball to test. Only this or DPDK tree can be provided.
.. option:: --remote-source
.. envvar:: DTS_REMOTE_SOURCE
@@ -108,8 +100,6 @@
from typing import Callable
from .config import DPDKLocation, TestSuiteConfig
-from .exception import ConfigurationError
-from .utils import DPDKGitTarball, get_commit_id
@dataclass(slots=True)
@@ -256,14 +246,6 @@ def _get_help_string(self, action):
return help
-def _parse_revision_id(rev_id: str) -> str:
- """Validate revision ID and retrieve corresponding commit ID."""
- try:
- return get_commit_id(rev_id)
- except ConfigurationError:
- raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")
-
-
def _required_with_one_of(parser: _DTSArgumentParser, action: Action, *required_dests: str) -> None:
"""Verify that `action` is listed together with at least one of `required_dests`.
@@ -362,15 +344,15 @@ def _get_parser() -> _DTSArgumentParser:
dpdk_build = parser.add_argument_group(
"DPDK Build Options",
description="Arguments in this group (and subgroup) will be applied to a "
- ":class:`DPDKLocation` when the DPDK tree, tarball or revision will be provided, "
- "other arguments like remote source and build dir are optional. A :class:`DPDKLocation` "
+ ":class:`DPDKLocation` when the DPDK tree, tarball will be provided, other "
+ "arguments like remote source and build dir are optional. A :class:`DPDKLocation` "
"from settings are used instead of from config if construct successful.",
)
dpdk_source = dpdk_build.add_mutually_exclusive_group()
action = dpdk_source.add_argument(
"--dpdk-tree",
- help="The path to DPDK source tree directory to test. Only this or tarball or revision "
+ help="The path to DPDK source tree directory to test. Only this or tarball "
"can be provided.",
metavar="DIR_PATH",
dest="dpdk_tree_path",
@@ -380,26 +362,12 @@ def _get_parser() -> _DTSArgumentParser:
action = dpdk_source.add_argument(
"--tarball",
"--snapshot",
- help="The path to DPDK source tarball to test. Only this or DPDK tree or revision "
- "can be provided.",
+ help="The path to DPDK source tarball to test. Only this or DPDK tree " "can be provided.",
metavar="FILE_PATH",
dest="dpdk_tarball_path",
)
_add_env_var_to_action(action, "DPDK_TARBALL")
- action = dpdk_source.add_argument(
- "--revision",
- "--rev",
- "--git-ref",
- type=_parse_revision_id,
- help="Git revision ID to test. Could be commit, tag, tree ID etc. "
- "To test local changes, first commit them, then use their commit ID."
- "Only this or DPDK tree or tarball can be provided.",
- metavar="ID",
- dest="dpdk_revision_id",
- )
- _add_env_var_to_action(action)
-
action = dpdk_build.add_argument(
"--remote-source",
action="store_true",
@@ -408,9 +376,7 @@ def _get_parser() -> _DTSArgumentParser:
"instead of the execution host. This can be provided only with DPDK tree or tarball.",
)
_add_env_var_to_action(action)
- _required_with_one_of(
- parser, action, "dpdk_tarball_path", "dpdk_tree_path"
- ) # ignored if passed with git-ref
+ _required_with_one_of(parser, action, "dpdk_tarball_path", "dpdk_tree_path")
action = dpdk_build.add_argument(
"--build-dir",
@@ -567,9 +533,6 @@ def get_settings() -> Settings:
args = parser.parse_args()
- if args.dpdk_revision_id:
- args.dpdk_tarball_path = Path(DPDKGitTarball(args.dpdk_revision_id, args.output_dir))
-
args.dpdk_location = _process_dpdk_location(
args.dpdk_tree_path, args.dpdk_tarball_path, args.remote_source, args.build_dir
)
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index 382357ffe8..4b8843bf20 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -14,22 +14,19 @@
REGEX_FOR_PCI_ADDRESS: The regex representing a PCI address, e.g. ``0000:00:08.0``.
"""
-import atexit
import fnmatch
import json
import os
import random
-import subprocess
import tarfile
from enum import Enum, Flag
from pathlib import Path
-from subprocess import SubprocessError
from typing import Any, Callable
from scapy.layers.inet import IP, TCP, UDP, Ether # type: ignore[import-untyped]
from scapy.packet import Packet # type: ignore[import-untyped]
-from .exception import ConfigurationError, InternalError
+from .exception import InternalError
REGEX_FOR_PCI_ADDRESS: str = "/[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}.[0-9]{1}/"
@@ -76,31 +73,6 @@ def get_packet_summaries(packets: list[Packet]) -> str:
return f"Packet contents: \n{packet_summaries}"
-def get_commit_id(rev_id: str) -> str:
- """Given a Git revision ID, return the corresponding commit ID.
-
- Args:
- rev_id: The Git revision ID.
-
- Raises:
- ConfigurationError: The ``git rev-parse`` command failed, suggesting
- an invalid or ambiguous revision ID was supplied.
- """
- result = subprocess.run(
- ["git", "rev-parse", "--verify", rev_id],
- text=True,
- capture_output=True,
- )
- if result.returncode != 0:
- raise ConfigurationError(
- f"{rev_id} is not a valid git reference.\n"
- f"Command: {result.args}\n"
- f"Stdout: {result.stdout}\n"
- f"Stderr: {result.stderr}"
- )
- return result.stdout.strip()
-
-
class StrEnum(Enum):
"""Enum with members stored as strings."""
@@ -176,95 +148,6 @@ def extension(self):
return f"{self.value}" if self == self.none else f"{self.none.value}.{self.value}"
-class DPDKGitTarball:
- """Compressed tarball of DPDK from the repository.
-
- The class supports the :class:`os.PathLike` protocol,
- which is used to get the Path of the tarball::
-
- from pathlib import Path
- tarball = DPDKGitTarball("HEAD", "output")
- tarball_path = Path(tarball)
- """
-
- _git_ref: str
- _tar_compression_format: TarCompressionFormat
- _tarball_dir: Path
- _tarball_name: str
- _tarball_path: Path | None
-
- def __init__(
- self,
- git_ref: str,
- output_dir: str,
- tar_compression_format: TarCompressionFormat = TarCompressionFormat.xz,
- ):
- """Create the tarball during initialization.
-
- The DPDK version is specified with `git_ref`. The tarball will be compressed with
- `tar_compression_format`, which must be supported by the DTS execution environment.
- The resulting tarball will be put into `output_dir`.
-
- Args:
- git_ref: A git commit ID, tag ID or tree ID.
- output_dir: The directory where to put the resulting tarball.
- tar_compression_format: The compression format to use.
- """
- self._git_ref = git_ref
- self._tar_compression_format = tar_compression_format
-
- self._tarball_dir = Path(output_dir, "tarball")
-
- self._create_tarball_dir()
-
- self._tarball_name = (
- f"dpdk-tarball-{self._git_ref}.{self._tar_compression_format.extension}"
- )
- self._tarball_path = self._check_tarball_path()
- if not self._tarball_path:
- self._create_tarball()
-
- def _create_tarball_dir(self) -> None:
- os.makedirs(self._tarball_dir, exist_ok=True)
-
- def _check_tarball_path(self) -> Path | None:
- if self._tarball_name in os.listdir(self._tarball_dir):
- return Path(self._tarball_dir, self._tarball_name)
- return None
-
- def _create_tarball(self) -> None:
- self._tarball_path = Path(self._tarball_dir, self._tarball_name)
-
- atexit.register(self._delete_tarball)
-
- result = subprocess.run(
- 'git -C "$(git rev-parse --show-toplevel)" archive '
- f'{self._git_ref} --prefix="dpdk-tarball-{self._git_ref + os.sep}" | '
- f"{self._tar_compression_format} > {Path(self._tarball_path.absolute())}",
- shell=True,
- text=True,
- capture_output=True,
- )
-
- if result.returncode != 0:
- raise SubprocessError(
- f"Git archive creation failed with exit code {result.returncode}.\n"
- f"Command: {result.args}\n"
- f"Stdout: {result.stdout}\n"
- f"Stderr: {result.stderr}"
- )
-
- atexit.unregister(self._delete_tarball)
-
- def _delete_tarball(self) -> None:
- if self._tarball_path and os.path.exists(self._tarball_path):
- os.remove(self._tarball_path)
-
- def __fspath__(self) -> str:
- """The os.PathLike protocol implementation."""
- return str(self._tarball_path)
-
-
def convert_to_list_of_string(value: Any | list[Any]) -> list[str]:
"""Convert the input to the list of strings."""
return list(map(str, value) if isinstance(value, list) else str(value))
--
2.46.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 7/7] dts: remove git ref option
2024-09-27 15:38 [PATCH 1/7] dts: rename build target to DPDK build Tomáš Ďurovec
@ 2024-09-27 15:38 ` Tomáš Ďurovec
0 siblings, 0 replies; 9+ messages in thread
From: Tomáš Ďurovec @ 2024-09-27 15:38 UTC (permalink / raw)
To: dev; +Cc: Tomáš Ďurovec
In the previous commits we're adding the support for copying
the whole local DPDK tree directory and git-ref option was
meant to do the same thing.
Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
---
doc/guides/tools/dts.rst | 9 ---
dts/framework/settings.py | 51 +++-------------
dts/framework/utils.py | 119 +-------------------------------------
3 files changed, 8 insertions(+), 171 deletions(-)
diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 20d4d18b18..d806dce2ae 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -239,9 +239,6 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
--dpdk-tree DIR_PATH [DTS_DPDK_TREE] Path to DPDK source code tree to test. (default: None)
--tarball FILE_PATH, --snapshot FILE_PATH
[DTS_DPDK_TARBALL] Path to DPDK source code tarball to test. (default: None)
- --revision ID, --rev ID, --git-ref ID
- [DTS_DPDK_REVISION_ID] Git revision ID to test. Could be commit, tag, tree ID etc. To test local changes, first
- commit them, then use their commit ID. (default: None)
--remote-source [DTS_REMOTE_SOURCE] Set when the DPDK source tree or tarball is located on the SUT node. (default: False)
--build-dir DIR_NAME [DTS_BUILD_DIR] A directory name, which would be located in the `dpdk tree` or `tarball`. (default: None)
--compile-timeout SECONDS
@@ -263,12 +260,6 @@ The minimum DTS needs is a config file and a pre-built DPDK or DPDK
sources location which can be specified in said config file or on the
command line or environment variables.
-Example command for running DTS with the template configuration and DPDK tag v23.11:
-
-.. code-block:: console
-
- (dts-py3.10) $ ./main.py --git-ref v23.11
-
DTS Results
~~~~~~~~~~~
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 17594ecb15..a4ab674189 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -42,20 +42,12 @@
.. option:: --dpdk-tree
.. envvar:: DTS_DPDK_TREE
- The path to DPDK source tree directory to test. Only this or tarball or revision can be
- provided.
+ The path to DPDK source tree directory to test. Only this or tarball can be provided.
.. option:: --tarball, --snapshot
.. envvar:: DTS_DPDK_TARBALL
- The path to DPDK source tarball to test. Only this or DPDK tree or revision can be provided.
-
-.. option:: --revision, --rev, --git-ref
-.. envvar:: DTS_DPDK_REVISION_ID
-
- Git revision ID to test. Could be commit, tag, tree ID etc.
- To test local changes, first commit them, then use their commit ID.
- Only this or DPDK tree or tarball can be provided.
+ The path to DPDK source tarball to test. Only this or DPDK tree can be provided.
.. option:: --remote-source
.. envvar:: DTS_REMOTE_SOURCE
@@ -108,8 +100,6 @@
from typing import Callable
from .config import DPDKLocation, TestSuiteConfig
-from .exception import ConfigurationError
-from .utils import DPDKGitTarball, get_commit_id
@dataclass(slots=True)
@@ -256,14 +246,6 @@ def _get_help_string(self, action):
return help
-def _parse_revision_id(rev_id: str) -> str:
- """Validate revision ID and retrieve corresponding commit ID."""
- try:
- return get_commit_id(rev_id)
- except ConfigurationError:
- raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")
-
-
def _required_with_one_of(parser: _DTSArgumentParser, action: Action, *required_dests: str) -> None:
"""Verify that `action` is listed together with at least one of `required_dests`.
@@ -362,15 +344,15 @@ def _get_parser() -> _DTSArgumentParser:
dpdk_build = parser.add_argument_group(
"DPDK Build Options",
description="Arguments in this group (and subgroup) will be applied to a "
- ":class:`DPDKLocation` when the DPDK tree, tarball or revision will be provided, "
- "other arguments like remote source and build dir are optional. A :class:`DPDKLocation` "
+ ":class:`DPDKLocation` when the DPDK tree, tarball will be provided, other "
+ "arguments like remote source and build dir are optional. A :class:`DPDKLocation` "
"from settings are used instead of from config if construct successful.",
)
dpdk_source = dpdk_build.add_mutually_exclusive_group()
action = dpdk_source.add_argument(
"--dpdk-tree",
- help="The path to DPDK source tree directory to test. Only this or tarball or revision "
+ help="The path to DPDK source tree directory to test. Only this or tarball "
"can be provided.",
metavar="DIR_PATH",
dest="dpdk_tree_path",
@@ -380,26 +362,12 @@ def _get_parser() -> _DTSArgumentParser:
action = dpdk_source.add_argument(
"--tarball",
"--snapshot",
- help="The path to DPDK source tarball to test. Only this or DPDK tree or revision "
- "can be provided.",
+ help="The path to DPDK source tarball to test. Only this or DPDK tree " "can be provided.",
metavar="FILE_PATH",
dest="dpdk_tarball_path",
)
_add_env_var_to_action(action, "DPDK_TARBALL")
- action = dpdk_source.add_argument(
- "--revision",
- "--rev",
- "--git-ref",
- type=_parse_revision_id,
- help="Git revision ID to test. Could be commit, tag, tree ID etc. "
- "To test local changes, first commit them, then use their commit ID."
- "Only this or DPDK tree or tarball can be provided.",
- metavar="ID",
- dest="dpdk_revision_id",
- )
- _add_env_var_to_action(action)
-
action = dpdk_build.add_argument(
"--remote-source",
action="store_true",
@@ -408,9 +376,7 @@ def _get_parser() -> _DTSArgumentParser:
"instead of the execution host. This can be provided only with DPDK tree or tarball.",
)
_add_env_var_to_action(action)
- _required_with_one_of(
- parser, action, "dpdk_tarball_path", "dpdk_tree_path"
- ) # ignored if passed with git-ref
+ _required_with_one_of(parser, action, "dpdk_tarball_path", "dpdk_tree_path")
action = dpdk_build.add_argument(
"--build-dir",
@@ -567,9 +533,6 @@ def get_settings() -> Settings:
args = parser.parse_args()
- if args.dpdk_revision_id:
- args.dpdk_tarball_path = Path(DPDKGitTarball(args.dpdk_revision_id, args.output_dir))
-
args.dpdk_location = _process_dpdk_location(
args.dpdk_tree_path, args.dpdk_tarball_path, args.remote_source, args.build_dir
)
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index 382357ffe8..4b8843bf20 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -14,22 +14,19 @@
REGEX_FOR_PCI_ADDRESS: The regex representing a PCI address, e.g. ``0000:00:08.0``.
"""
-import atexit
import fnmatch
import json
import os
import random
-import subprocess
import tarfile
from enum import Enum, Flag
from pathlib import Path
-from subprocess import SubprocessError
from typing import Any, Callable
from scapy.layers.inet import IP, TCP, UDP, Ether # type: ignore[import-untyped]
from scapy.packet import Packet # type: ignore[import-untyped]
-from .exception import ConfigurationError, InternalError
+from .exception import InternalError
REGEX_FOR_PCI_ADDRESS: str = "/[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}.[0-9]{1}/"
@@ -76,31 +73,6 @@ def get_packet_summaries(packets: list[Packet]) -> str:
return f"Packet contents: \n{packet_summaries}"
-def get_commit_id(rev_id: str) -> str:
- """Given a Git revision ID, return the corresponding commit ID.
-
- Args:
- rev_id: The Git revision ID.
-
- Raises:
- ConfigurationError: The ``git rev-parse`` command failed, suggesting
- an invalid or ambiguous revision ID was supplied.
- """
- result = subprocess.run(
- ["git", "rev-parse", "--verify", rev_id],
- text=True,
- capture_output=True,
- )
- if result.returncode != 0:
- raise ConfigurationError(
- f"{rev_id} is not a valid git reference.\n"
- f"Command: {result.args}\n"
- f"Stdout: {result.stdout}\n"
- f"Stderr: {result.stderr}"
- )
- return result.stdout.strip()
-
-
class StrEnum(Enum):
"""Enum with members stored as strings."""
@@ -176,95 +148,6 @@ def extension(self):
return f"{self.value}" if self == self.none else f"{self.none.value}.{self.value}"
-class DPDKGitTarball:
- """Compressed tarball of DPDK from the repository.
-
- The class supports the :class:`os.PathLike` protocol,
- which is used to get the Path of the tarball::
-
- from pathlib import Path
- tarball = DPDKGitTarball("HEAD", "output")
- tarball_path = Path(tarball)
- """
-
- _git_ref: str
- _tar_compression_format: TarCompressionFormat
- _tarball_dir: Path
- _tarball_name: str
- _tarball_path: Path | None
-
- def __init__(
- self,
- git_ref: str,
- output_dir: str,
- tar_compression_format: TarCompressionFormat = TarCompressionFormat.xz,
- ):
- """Create the tarball during initialization.
-
- The DPDK version is specified with `git_ref`. The tarball will be compressed with
- `tar_compression_format`, which must be supported by the DTS execution environment.
- The resulting tarball will be put into `output_dir`.
-
- Args:
- git_ref: A git commit ID, tag ID or tree ID.
- output_dir: The directory where to put the resulting tarball.
- tar_compression_format: The compression format to use.
- """
- self._git_ref = git_ref
- self._tar_compression_format = tar_compression_format
-
- self._tarball_dir = Path(output_dir, "tarball")
-
- self._create_tarball_dir()
-
- self._tarball_name = (
- f"dpdk-tarball-{self._git_ref}.{self._tar_compression_format.extension}"
- )
- self._tarball_path = self._check_tarball_path()
- if not self._tarball_path:
- self._create_tarball()
-
- def _create_tarball_dir(self) -> None:
- os.makedirs(self._tarball_dir, exist_ok=True)
-
- def _check_tarball_path(self) -> Path | None:
- if self._tarball_name in os.listdir(self._tarball_dir):
- return Path(self._tarball_dir, self._tarball_name)
- return None
-
- def _create_tarball(self) -> None:
- self._tarball_path = Path(self._tarball_dir, self._tarball_name)
-
- atexit.register(self._delete_tarball)
-
- result = subprocess.run(
- 'git -C "$(git rev-parse --show-toplevel)" archive '
- f'{self._git_ref} --prefix="dpdk-tarball-{self._git_ref + os.sep}" | '
- f"{self._tar_compression_format} > {Path(self._tarball_path.absolute())}",
- shell=True,
- text=True,
- capture_output=True,
- )
-
- if result.returncode != 0:
- raise SubprocessError(
- f"Git archive creation failed with exit code {result.returncode}.\n"
- f"Command: {result.args}\n"
- f"Stdout: {result.stdout}\n"
- f"Stderr: {result.stderr}"
- )
-
- atexit.unregister(self._delete_tarball)
-
- def _delete_tarball(self) -> None:
- if self._tarball_path and os.path.exists(self._tarball_path):
- os.remove(self._tarball_path)
-
- def __fspath__(self) -> str:
- """The os.PathLike protocol implementation."""
- return str(self._tarball_path)
-
-
def convert_to_list_of_string(value: Any | list[Any]) -> list[str]:
"""Convert the input to the list of strings."""
return list(map(str, value) if isinstance(value, list) else str(value))
--
2.46.1
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2024-09-27 22:16 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-09-27 16:08 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 1/7] dts: rename build target to " Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 2/7] dts: one dpdk build per test run Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 3/7] dts: fix remote session file transfer vars Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 4/7] dts: add the ability to copy directories via remote Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 5/7] dts: add support for externally compiled DPDK Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 6/7] doc: update argument options for external DPDK build Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 7/7] dts: remove git ref option Tomáš Ďurovec
-- strict thread matches above, loose matches on Subject: below --
2024-09-27 15:38 [PATCH 1/7] dts: rename build target to DPDK build Tomáš Ďurovec
2024-09-27 15:38 ` [PATCH 7/7] dts: remove git ref option Tomáš Ďurovec
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).