From: "Tomáš Ďurovec" <tomas.durovec@pantheon.tech>
To: dev@dpdk.org
Cc: "Tomáš Ďurovec" <tomas.durovec@pantheon.tech>
Subject: [PATCH 5/7] dts: add support for externally compiled DPDK
Date: Mon, 30 Sep 2024 18:18:12 +0200 [thread overview]
Message-ID: <20240930161814.26070-6-tomas.durovec@pantheon.tech> (raw)
In-Reply-To: <20240930161814.26070-1-tomas.durovec@pantheon.tech>
Add support for using DPDK source tree directory as well as DPDK
tarball with the pre-build directory that can user specify and
type of location, it can be stored in the local filesystem or SUT
node. Additionally, this can be set up with the config file or
cmd arguments/environment variables.
Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
---
dts/conf.yaml | 23 +-
dts/framework/config/__init__.py | 121 ++++++-
dts/framework/config/conf_yaml_schema.json | 62 +++-
dts/framework/config/types.py | 17 +-
dts/framework/exception.py | 4 +-
dts/framework/remote_session/dpdk_shell.py | 2 +-
dts/framework/runner.py | 8 +-
dts/framework/settings.py | 193 +++++++++--
dts/framework/test_result.py | 23 +-
dts/framework/testbed_model/node.py | 22 +-
dts/framework/testbed_model/os_session.py | 63 +++-
dts/framework/testbed_model/posix_session.py | 39 ++-
dts/framework/testbed_model/sut_node.py | 345 +++++++++++++------
13 files changed, 718 insertions(+), 204 deletions(-)
diff --git a/dts/conf.yaml b/dts/conf.yaml
index 814744a1fc..2f3010204d 100644
--- a/dts/conf.yaml
+++ b/dts/conf.yaml
@@ -5,12 +5,23 @@
test_runs:
# define one test run environment
- dpdk_build:
- arch: x86_64
- os: linux
- cpu: native
- # the combination of the following two makes CC="ccache gcc"
- compiler: gcc
- compiler_wrapper: ccache
+ # dpdk_tree: Commented out because `tarball` is defined.
+ tarball: dpdk-tarball.tar.xz
+ # Either `dpdk_tree` or `tarball` can be defined, but not both.
+ remote: false # Optional, defaults to false. If it's true, the `dpdk_tree` or `tarball`
+ # is located on the SUT node, instead of the execution host.
+
+ # dir_name: Commented out because `build` is defined.
+ build:
+ arch: x86_64
+ os: linux
+ cpu: native
+ # the combination of the following two makes CC="ccache gcc"
+ compiler: gcc
+ compiler_wrapper: ccache # Optional.
+ # If `dir_name` is defined, DPDK has been pre-built and the build directory is located in a
+ # subdirectory of DPDK tree root directory. Otherwise, will be using a `build` to build the
+ # DPDK from source. Either `dir_name` or `build` can be defined, but not both.
perf: false # disable performance testing
func: true # enable functional testing
skip_smoke_tests: false # optional
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 49b2e8d016..1bbc1c8700 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -35,6 +35,7 @@
import json
import os.path
+import tarfile
from dataclasses import dataclass, fields
from enum import auto, unique
from pathlib import Path
@@ -47,6 +48,7 @@
from framework.config.types import (
ConfigurationDict,
DPDKBuildConfigDict,
+ DPDKConfigurationDict,
NodeConfigDict,
PortConfigDict,
TestRunConfigDict,
@@ -380,6 +382,115 @@ def from_dict(cls, d: DPDKBuildConfigDict) -> Self:
)
+@dataclass(slots=True, frozen=True)
+class DPDKLocation:
+ """DPDK location.
+
+ The path to the DPDK sources, build dir and type of location.
+
+ Attributes:
+ dpdk_tree: The path to the DPDK source tree directory. Only one of `dpdk_tree` or `tarball`
+ must be provided.
+ tarball: The path to the DPDK tarball. Only one of `dpdk_tree` or `tarball` must be
+ provided.
+ remote: Optional, defaults to :data:`False`. If :data:`True`, `dpdk_tree` or `tarball` is
+ located on the SUT node, instead of the execution host.
+ build_dir: If it's defined, DPDK has been pre-built and the build directory is located in a
+ subdirectory of `dpdk_tree` or `tarball` root directory. Otherwise, will be using a
+ `build` from configuration to build the DPDK from source.
+ """
+
+ dpdk_tree: str | None
+ tarball: str | None
+ remote: bool
+ build_dir: str | None
+
+ @classmethod
+ def from_dict(cls, d: DPDKConfigurationDict) -> Self:
+ """A convenience method that processes and validates the inputs before creating an instance.
+
+ Validate existence and format of `dpdk_tree` or `tarball` on local filesystem, if
+ `remote` is False.
+
+ Args:
+ d: The configuration dictionary.
+
+ Returns:
+ The DPDK location instance.
+
+ Raises:
+ ConfigurationError: If `dpdk_tree` or `tarball` not found in local filesystem or they
+ aren't in the right format.
+ """
+ dpdk_tree = d.get("dpdk_tree")
+ tarball = d.get("tarball")
+ remote = d.get("remote", False)
+
+ if not remote:
+ if dpdk_tree:
+ if not Path(dpdk_tree).exists():
+ raise ConfigurationError(
+ f"DPDK tree '{dpdk_tree}' not found in local filesystem."
+ )
+
+ if not Path(dpdk_tree).is_dir():
+ raise ConfigurationError(
+ f"DPDK tree '{dpdk_tree}' had not valid format, must be directory."
+ )
+
+ if tarball:
+ if not Path(tarball).exists():
+ raise ConfigurationError(
+ f"DPDK tarball '{tarball}' not found in local filesystem."
+ )
+
+ if not tarfile.is_tarfile(tarball):
+ raise ConfigurationError(
+ f"DPDK tarball '{tarball}' had not valid format, must be tar archive."
+ )
+
+ return cls(
+ dpdk_tree=dpdk_tree,
+ tarball=tarball,
+ remote=remote,
+ build_dir=d.get("dir_name"),
+ )
+
+
+@dataclass
+class DPDKConfiguration:
+ """The configuration of the DPDK build.
+
+ The configuration contain the location of the DPDK and configuration used for
+ building it.
+
+ Attributes:
+ dpdk_location: The location of the DPDK tree.
+ dpdk_build_config: A DPDK build configuration to test. If :data:`None`,
+ DTS will use pre-built DPDK from `build_dir` in a :dataclass:`DPDKLocation`.
+ """
+
+ dpdk_location: DPDKLocation
+ dpdk_build_config: DPDKBuildConfiguration | None
+
+ @classmethod
+ def from_dict(cls, d: DPDKConfigurationDict) -> Self:
+ """A convenience method that processes the inputs before creating an instance.
+
+ Args:
+ d: The configuration dictionary.
+
+ Returns:
+ The DPDK configuration.
+ """
+ return cls(
+ dpdk_location=DPDKLocation.from_dict(d),
+ dpdk_build_config=DPDKBuildConfiguration.from_dict(d["build"])
+ if d.get("build")
+ else None,
+ )
+
+
@dataclass(slots=True, frozen=True)
class DPDKBuildInfo:
"""Various versions and other information about a DPDK build.
@@ -389,8 +500,8 @@ class DPDKBuildInfo:
compiler_version: The version of the compiler used to build DPDK.
"""
- dpdk_version: str
- compiler_version: str
+ dpdk_version: str | None
+ compiler_version: str | None
@dataclass(slots=True, frozen=True)
@@ -437,7 +548,7 @@ class TestRunConfiguration:
and with what DPDK build.
Attributes:
- dpdk_build: A DPDK build to test.
+ dpdk_config: The DPDK configuration used to test.
perf: Whether to run performance tests.
func: Whether to run functional tests.
skip_smoke_tests: Whether to skip smoke tests.
@@ -448,7 +559,7 @@ class TestRunConfiguration:
random_seed: The seed to use for pseudo-random generation.
"""
- dpdk_build: DPDKBuildConfiguration
+ dpdk_config: DPDKConfiguration
perf: bool
func: bool
skip_smoke_tests: bool
@@ -498,7 +609,7 @@ def from_dict(
)
random_seed = d.get("random_seed", None)
return cls(
- dpdk_build=DPDKBuildConfiguration.from_dict(d["dpdk_build"]),
+ dpdk_config=DPDKConfiguration.from_dict(d["dpdk_build"]),
perf=d["perf"],
func=d["func"],
skip_smoke_tests=skip_smoke_tests,
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 94d7efa5f5..f5cd6d6075 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -110,9 +110,8 @@
"mscv"
]
},
- "dpdk_build": {
+ "build": {
"type": "object",
- "description": "DPDK build configuration supported by DTS.",
"properties": {
"arch": {
"type": "string",
@@ -133,7 +132,7 @@
"compiler": {
"$ref": "#/definitions/compiler"
},
- "compiler_wrapper": {
+ "compiler_wrapper": {
"type": "string",
"description": "This will be added before compiler to the CC variable when building DPDK. Optional."
}
@@ -146,6 +145,63 @@
"compiler"
]
},
+ "dpdk_build": {
+ "type": "object",
+ "description": "DPDK source and build configuration.",
+ "properties": {
+ "dpdk_tree": {
+ "type": "string",
+ "description": "The path to the DPDK source tree directory to test. Only one of `dpdk_tree` or `tarball` must be provided."
+ },
+ "tarball": {
+ "type": "string",
+ "description": "The path to the DPDK source tarball to test. Only one of `dpdk_tree` or `tarball` must be provided."
+ },
+ "remote": {
+ "type": "boolean",
+ "description": "Optional, defaults to false. If it's true, the `dpdk_tree` or `tarball` is located on the SUT node, instead of the execution host."
+ },
+ "dir_name": {
+ "type": "string",
+ "description": "If it's defined, DPDK has been pre-built and the build directory is located in a subdirectory of DPDK tree root directory. Otherwise, will be using a `build` to build the DPDK from source. Either this or `build` must be defined, but not both."
+ },
+ "build": {
+ "$ref": "#/definitions/build",
+ "description": "Either this or `dir_name` must be defined, but not both. DPDK build configuration supported by DTS."
+ }
+ },
+ "allOf": [
+ {
+ "oneOf": [
+ {
+ "required": [
+ "dpdk_tree"
+ ]
+ },
+ {
+ "required": [
+ "tarball"
+ ]
+ }
+ ]
+ },
+ {
+ "oneOf": [
+ {
+ "required": [
+ "dir_name"
+ ]
+ },
+ {
+ "required": [
+ "build"
+ ]
+ }
+ ]
+ }
+ ],
+ "additionalProperties": false
+ },
"hugepages_2mb": {
"type": "object",
"description": "Optional hugepage configuration. If not specified, hugepages won't be configured and DTS will use system configuration.",
diff --git a/dts/framework/config/types.py b/dts/framework/config/types.py
index a710c20d6a..24884381cc 100644
--- a/dts/framework/config/types.py
+++ b/dts/framework/config/types.py
@@ -86,6 +86,21 @@ class DPDKBuildConfigDict(TypedDict):
compiler_wrapper: str
+class DPDKConfigurationDict(TypedDict):
+ """Allowed keys and values."""
+
+ #:
+ dpdk_tree: str | None
+ #:
+ tarball: str | None
+ #:
+ remote: bool
+ #:
+ dir_name: str | None
+ #:
+ build: DPDKBuildConfigDict
+
+
class TestSuiteConfigDict(TypedDict):
"""Allowed keys and values."""
@@ -108,7 +123,7 @@ class TestRunConfigDict(TypedDict):
"""Allowed keys and values."""
#:
- dpdk_build: DPDKBuildConfigDict
+ dpdk_build: DPDKConfigurationDict
#:
perf: bool
#:
diff --git a/dts/framework/exception.py b/dts/framework/exception.py
index f45f789825..d967ede09b 100644
--- a/dts/framework/exception.py
+++ b/dts/framework/exception.py
@@ -184,8 +184,8 @@ class InteractiveCommandExecutionError(DTSError):
severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
-class RemoteDirectoryExistsError(DTSError):
- """A directory that exists on a remote node."""
+class RemoteFileNotFoundError(DTSError):
+ """A remote file or directory is requested but doesn’t exist."""
#:
severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
diff --git a/dts/framework/remote_session/dpdk_shell.py b/dts/framework/remote_session/dpdk_shell.py
index c5f5c2d116..b39132cc42 100644
--- a/dts/framework/remote_session/dpdk_shell.py
+++ b/dts/framework/remote_session/dpdk_shell.py
@@ -104,4 +104,4 @@ def _update_real_path(self, path: PurePath) -> None:
Adds the remote DPDK build directory to the path.
"""
- super()._update_real_path(self._node.remote_dpdk_build_dir.joinpath(path))
+ super()._update_real_path(PurePath(self._node.remote_dpdk_build_dir).joinpath(path))
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index 100dd75adb..7d463c1fa1 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -414,15 +414,19 @@ def _run_test_run(
test_run_config: A test run configuration.
test_run_result: The test run's result.
test_suites_with_cases: The test suites with test cases to run.
+
+ Raises:
+ ConfigurationError: If the DPDK sources or build is not set up from config or settings.
"""
self._logger.info(
f"Running test run with SUT '{test_run_config.system_under_test_node.name}'."
)
test_run_result.add_sut_info(sut_node.node_info)
try:
- sut_node.set_up_test_run(test_run_config)
+ dpdk_location = SETTINGS.dpdk_location or test_run_config.dpdk_config.dpdk_location
+ sut_node.set_up_test_run(test_run_config, dpdk_location)
test_run_result.add_dpdk_build_info(sut_node.get_dpdk_build_info())
- tg_node.set_up_test_run(test_run_config)
+ tg_node.set_up_test_run(test_run_config, dpdk_location)
test_run_result.update_setup(Result.PASS)
except Exception as e:
self._logger.exception("Test run setup failed.")
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 52a1582d5c..17594ecb15 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -39,21 +39,36 @@
Set to any value to enable logging everything to the console.
-.. option:: -s, --skip-setup
-.. envvar:: DTS_SKIP_SETUP
+.. option:: --dpdk-tree
+.. envvar:: DTS_DPDK_TREE
- Set to any value to skip building DPDK.
+ The path to DPDK source tree directory to test. Only this or tarball or revision can be
+ provided.
.. option:: --tarball, --snapshot
.. envvar:: DTS_DPDK_TARBALL
- Path to DPDK source code tarball to test.
+ The path to DPDK source tarball to test. Only this or DPDK tree or revision can be provided.
.. option:: --revision, --rev, --git-ref
.. envvar:: DTS_DPDK_REVISION_ID
Git revision ID to test. Could be commit, tag, tree ID etc.
To test local changes, first commit them, then use their commit ID.
+ Only this or DPDK tree or tarball can be provided.
+
+.. option:: --remote-source
+.. envvar:: DTS_REMOTE_SOURCE
+
+ Set when the DPDK source tree or tarball is located on the SUT node, instead of the
+ execution host. This can be provided only with DPDK tree or tarball.
+
+.. option:: --build-dir
+.. envvar:: DTS_BUILD_DIR
+
+ A directory name. Optional, if it's defined, DPDK has been pre-built and the build directory
+ is located in a subdirectory of DPDK tree root directory. Otherwise, will be using a `build`
+ This can be provided only with DPDK tree or tarball.
.. option:: --test-suite
.. envvar:: DTS_TEST_SUITES
@@ -86,12 +101,13 @@
import argparse
import os
import sys
+import tarfile
from argparse import Action, ArgumentDefaultsHelpFormatter, _get_action_name
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable
-from .config import TestSuiteConfig
+from .config import DPDKLocation, TestSuiteConfig
from .exception import ConfigurationError
from .utils import DPDKGitTarball, get_commit_id
@@ -112,9 +128,7 @@ class Settings:
#:
verbose: bool = False
#:
- skip_setup: bool = False
- #:
- dpdk_tarball_path: Path | str = ""
+ dpdk_location: DPDKLocation | None = None
#:
compile_timeout: float = 1200
#:
@@ -242,14 +256,6 @@ def _get_help_string(self, action):
return help
-def _parse_tarball_path(file_path: str) -> Path:
- """Validate whether `file_path` is valid and return a Path object."""
- path = Path(file_path)
- if not path.exists() or not path.is_file():
- raise argparse.ArgumentTypeError("The file path provided is not a valid file")
- return path
-
-
def _parse_revision_id(rev_id: str) -> str:
"""Validate revision ID and retrieve corresponding commit ID."""
try:
@@ -258,6 +264,47 @@ def _parse_revision_id(rev_id: str) -> str:
raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")
+def _required_with_one_of(parser: _DTSArgumentParser, action: Action, *required_dests: str) -> None:
+ """Verify that `action` is listed together with at least one of `required_dests`.
+
+ Verify that when `action` is among the command-line arguments or
+ environment variables, at least one of `required_dests` is also among
+ the command-line arguments or environment variables.
+
+ Args:
+ parser: The custom ArgumentParser object which contains `action`.
+ action: The action to be verified.
+ *required_dests: Destination variable names of the required arguments.
+
+ Raises:
+ argparse.ArgumentTypeError: When none of the required_dest are defined.
+
+ Example:
+ We have ``--option1`` and we only want it to be a passed alongside
+ either ``--option2`` or ``--option3`` (meaning if ``--option1`` is
+ passed without either ``--option2`` or ``--option3``, that's an error).
+
+ parser = _DTSArgumentParser()
+ option1_arg = parser.add_argument('--option1', dest='option1', action='store_true')
+ option2_arg = parser.add_argument('--option2', dest='option2', action='store_true')
+ option2_arg = parser.add_argument('--option3', dest='option3', action='store_true')
+
+ _required_with_one_of(parser, option1_arg, 'option2', 'option3')
+ """
+ if _is_action_in_args(action):
+ for required_dest in required_dests:
+ required_action = parser.find_action(required_dest)
+ if required_action is None:
+ continue
+
+ if _is_action_in_args(required_action):
+ return None
+
+ raise argparse.ArgumentTypeError(
+ f"The '{action.dest}' is required at least with one of '{', '.join(required_dests)}'."
+ )
+
+
def _get_parser() -> _DTSArgumentParser:
"""Create the argument parser for DTS.
@@ -312,22 +359,29 @@ def _get_parser() -> _DTSArgumentParser:
)
_add_env_var_to_action(action)
- action = parser.add_argument(
- "-s",
- "--skip-setup",
- action="store_true",
- default=SETTINGS.skip_setup,
- help="Specify to skip all setup steps on SUT and TG nodes.",
+ dpdk_build = parser.add_argument_group(
+ "DPDK Build Options",
+ description="Arguments in this group (and subgroup) will be applied to a "
+ ":class:`DPDKLocation` when the DPDK tree, tarball or revision will be provided, "
+ "other arguments like remote source and build dir are optional. A :class:`DPDKLocation` "
+ "from settings are used instead of from config if construct successful.",
)
- _add_env_var_to_action(action)
- dpdk_source = parser.add_mutually_exclusive_group(required=True)
+ dpdk_source = dpdk_build.add_mutually_exclusive_group()
+ action = dpdk_source.add_argument(
+ "--dpdk-tree",
+ help="The path to DPDK source tree directory to test. Only this or tarball or revision "
+ "can be provided.",
+ metavar="DIR_PATH",
+ dest="dpdk_tree_path",
+ )
+ _add_env_var_to_action(action, "DPDK_TREE")
action = dpdk_source.add_argument(
"--tarball",
"--snapshot",
- type=_parse_tarball_path,
- help="Path to DPDK source code tarball to test.",
+ help="The path to DPDK source tarball to test. Only this or DPDK tree or revision "
+ "can be provided.",
metavar="FILE_PATH",
dest="dpdk_tarball_path",
)
@@ -339,12 +393,36 @@ def _get_parser() -> _DTSArgumentParser:
"--git-ref",
type=_parse_revision_id,
help="Git revision ID to test. Could be commit, tag, tree ID etc. "
- "To test local changes, first commit them, then use their commit ID.",
+ "To test local changes, first commit them, then use their commit ID."
+ "Only this or DPDK tree or tarball can be provided.",
metavar="ID",
dest="dpdk_revision_id",
)
_add_env_var_to_action(action)
+ action = dpdk_build.add_argument(
+ "--remote-source",
+ action="store_true",
+ default=False,
+ help="Optional. Set when the DPDK source tree or tarball is located on the SUT node, "
+ "instead of the execution host. This can be provided only with DPDK tree or tarball.",
+ )
+ _add_env_var_to_action(action)
+ _required_with_one_of(
+ parser, action, "dpdk_tarball_path", "dpdk_tree_path"
+ ) # ignored if passed with git-ref
+
+ action = dpdk_build.add_argument(
+ "--build-dir",
+ help="A directory name. Optional, if it's defined, DPDK has been pre-built and the build "
+ "directory is located in a subdirectory of DPDK tree root directory. Otherwise DPDK will "
+ "be built from scratch with DPDK build configuration. This can be provided only with DPDK "
+ "tree or tarball.",
+ metavar="DIR_NAME",
+ )
+ _add_env_var_to_action(action)
+ _required_with_one_of(parser, action, "dpdk_tarball_path", "dpdk_tree_path")
+
action = parser.add_argument(
"--compile-timeout",
default=SETTINGS.compile_timeout,
@@ -395,6 +473,64 @@ def _get_parser() -> _DTSArgumentParser:
return parser
+def _process_dpdk_location(
+ dpdk_tree: str | None,
+ tarball: str | None,
+ remote: bool,
+ build_dir: str | None,
+):
+ """Process and validate DPDK build arguments.
+
+ Ensures that either `dpdk_tree` or `tarball` is provided. Validate existence and format of
+ `dpdk_tree` or `tarball` on local filesystem, if `remote` is False. Constructs and returns
+ the :class:`DPDKLocation` with the provided parameters if validation is successful.
+
+ Args:
+ dpdk_tree: The path to the DPDK source tree directory. Only one of `dpdk_tree` or `tarball`
+ must be provided.
+ tarball: The path to the DPDK tarball. Only one of `dpdk_tree` or `tarball` must be
+ provided.
+ remote: If :data:`True`, `dpdk_tree` or `tarball` is located on the SUT node, instead of the
+ execution host.
+ build_dir: If it's defined, DPDK has been pre-built and the build directory is located in a
+ subdirectory of `dpdk_tree` or `tarball` root directory.
+
+ Returns:
+ A DPDK location if construction is successful, otherwise None.
+
+ Raises:
+ argparse.ArgumentTypeError: If `dpdk_tree` or `tarball` not found in local filesystem or
+ they aren't in the right format.
+ """
+ if not (dpdk_tree or tarball):
+ return None
+
+ if not remote:
+ if dpdk_tree:
+ if not Path(dpdk_tree).exists():
+ raise argparse.ArgumentTypeError(
+ f"DPDK tree '{dpdk_tree}' not found in local filesystem."
+ )
+
+ if not Path(dpdk_tree).is_dir():
+ raise argparse.ArgumentTypeError(
+ f"DPDK tree '{dpdk_tree}' had not valid format, must be directory."
+ )
+
+ if tarball:
+ if not Path(tarball).exists():
+ raise argparse.ArgumentTypeError(
+ f"DPDK tarball '{tarball}' not found in local filesystem."
+ )
+
+ if not tarfile.is_tarfile(tarball):
+ raise argparse.ArgumentTypeError(
+ f"DPDK tarball '{tarball}' had not valid format, must be tar archive."
+ )
+
+ return DPDKLocation(dpdk_tree=dpdk_tree, tarball=tarball, remote=remote, build_dir=build_dir)
+
+
def _process_test_suites(
parser: _DTSArgumentParser, args: list[list[str]]
) -> list[TestSuiteConfig]:
@@ -434,6 +570,9 @@ def get_settings() -> Settings:
if args.dpdk_revision_id:
args.dpdk_tarball_path = Path(DPDKGitTarball(args.dpdk_revision_id, args.output_dir))
+ args.dpdk_location = _process_dpdk_location(
+ args.dpdk_tree_path, args.dpdk_tarball_path, args.remote_source, args.build_dir
+ )
args.test_suites = _process_test_suites(parser, args.test_suites)
kwargs = {k: v for k, v in vars(args).items() if hasattr(SETTINGS, k)}
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 31560f6704..0a10723098 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -29,16 +29,7 @@
from types import FunctionType
from typing import Union
-from .config import (
- OS,
- Architecture,
- Compiler,
- CPUType,
- DPDKBuildInfo,
- NodeInfo,
- TestRunConfiguration,
- TestSuiteConfig,
-)
+from .config import DPDKBuildInfo, NodeInfo, TestRunConfiguration, TestSuiteConfig
from .exception import DTSError, ErrorSeverity
from .logger import DTSLogger
from .settings import SETTINGS
@@ -318,10 +309,6 @@ class TestRunResult(BaseResult):
The internal list stores the results of all test suites in a given test run.
Attributes:
- arch: The DPDK build architecture.
- os: The DPDK build operating system.
- cpu: The DPDK build CPU.
- compiler: The DPDK build compiler.
compiler_version: The DPDK build compiler version.
dpdk_version: The built DPDK version.
sut_os_name: The operating system of the SUT node.
@@ -329,10 +316,6 @@ class TestRunResult(BaseResult):
sut_kernel_version: The operating system kernel version of the SUT node.
"""
- arch: Architecture
- os: OS
- cpu: CPUType
- compiler: Compiler
compiler_version: str | None
dpdk_version: str | None
sut_os_name: str
@@ -348,10 +331,6 @@ def __init__(self, test_run_config: TestRunConfiguration):
test_run_config: A test run configuration.
"""
super().__init__()
- self.arch = test_run_config.dpdk_build.arch
- self.os = test_run_config.dpdk_build.os
- self.cpu = test_run_config.dpdk_build.cpu
- self.compiler = test_run_config.dpdk_build.compiler
self.compiler_version = None
self.dpdk_version = None
self._config = test_run_config
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
index 12a40170ac..f048b57ed5 100644
--- a/dts/framework/testbed_model/node.py
+++ b/dts/framework/testbed_model/node.py
@@ -15,12 +15,11 @@
from abc import ABC
from ipaddress import IPv4Interface, IPv6Interface
-from typing import Any, Callable, Union
+from typing import Union
-from framework.config import OS, NodeConfiguration, TestRunConfiguration
+from framework.config import OS, DPDKLocation, NodeConfiguration, TestRunConfiguration
from framework.exception import ConfigurationError
from framework.logger import DTSLogger, get_dts_logger
-from framework.settings import SETTINGS
from .cpu import (
LogicalCore,
@@ -95,7 +94,9 @@ def _init_ports(self) -> None:
for port in self.ports:
self.configure_port_state(port)
- def set_up_test_run(self, test_run_config: TestRunConfiguration) -> None:
+ def set_up_test_run(
+ self, test_run_config: TestRunConfiguration, dpdk_location: DPDKLocation
+ ) -> None:
"""Test run setup steps.
Configure hugepages on all DTS node types. Additional steps can be added by
@@ -104,6 +105,7 @@ def set_up_test_run(self, test_run_config: TestRunConfiguration) -> None:
Args:
test_run_config: A test run configuration according to which
the setup steps will be taken.
+ dpdk_location: The target source of the DPDK tree.
"""
self._setup_hugepages()
@@ -216,18 +218,6 @@ def close(self) -> None:
for session in self._other_sessions:
session.close()
- @staticmethod
- def skip_setup(func: Callable[..., Any]) -> Callable[..., Any]:
- """Skip the decorated function.
-
- The :option:`--skip-setup` command line argument and the :envvar:`DTS_SKIP_SETUP`
- environment variable enable the decorator.
- """
- if SETTINGS.skip_setup:
- return lambda *args: None
- else:
- return func
-
def create_session(node_config: NodeConfiguration, name: str, logger: DTSLogger) -> OSSession:
"""Factory for OS-aware sessions.
diff --git a/dts/framework/testbed_model/os_session.py b/dts/framework/testbed_model/os_session.py
index 6c3f84dec1..6194ddb989 100644
--- a/dts/framework/testbed_model/os_session.py
+++ b/dts/framework/testbed_model/os_session.py
@@ -137,17 +137,6 @@ def _get_privileged_command(command: str) -> str:
The modified command that executes with administrative privileges.
"""
- @abstractmethod
- def guess_dpdk_remote_dir(self, remote_dir: str | PurePath) -> PurePath:
- """Try to find DPDK directory in `remote_dir`.
-
- The directory is the one which is created after the extraction of the tarball. The files
- are usually extracted into a directory starting with ``dpdk-``.
-
- Returns:
- The absolute path of the DPDK remote directory, empty path if not found.
- """
-
@abstractmethod
def get_remote_tmp_dir(self) -> PurePath:
"""Get the path of the temporary directory of the remote OS.
@@ -177,6 +166,17 @@ def join_remote_path(self, *args: str | PurePath) -> PurePath:
The resulting joined path.
"""
+ @abstractmethod
+ def remote_path_exists(self, remote_path: str | PurePath) -> bool:
+ """Check whether `remote_path` exists on the remote system.
+
+ Args:
+ remote_path: The path to check.
+
+ Returns:
+ :data:`True` if the path exists, :data:`False` otherwise.
+ """
+
@abstractmethod
def copy_from(self, source_file: str | PurePath, destination_dir: str | Path) -> None:
"""Copy a file from the remote node to the local filesystem.
@@ -344,6 +344,47 @@ def extract_remote_tarball(
the archive.
"""
+ @abstractmethod
+ def is_remote_dir(self, remote_path: str) -> bool:
+ """Check if the `remote_path` is a directory.
+
+ Args:
+ remote_tarball_path: The path to the remote tarball.
+
+ Returns:
+ If :data:`True` the `remote_path` is a directory, otherwise :data:`False`.
+ """
+
+ @abstractmethod
+ def is_remote_tarfile(self, remote_tarball_path: str) -> bool:
+ """Check if the `remote_tarball_path` is a tar archive.
+
+ Args:
+ remote_tarball_path: The path to the remote tarball.
+
+ Returns:
+ If :data:`True` the `remote_tarball_path` is a tar archive, otherwise :data:`False`.
+ """
+
+ @abstractmethod
+ def get_tarball_top_dir(
+ self, remote_tarball_path: str | PurePath
+ ) -> str | PurePosixPath | None:
+ """Get the top directory of the remote tarball.
+
+ Examines the contents of a tarball located at the given `remote_tarball_path` and
+ determines the top-level directory. If all files and directories in the tarball share
+ the same top-level directory, that directory name is returned. If the tarball contains
+ multiple top-level directories or is empty, the method return None.
+
+ Args:
+ remote_tarball_path: The path to the remote tarball.
+
+ Returns:
+ The top directory of the tarball. If there are multiple top directories
+ or the tarball is empty, returns :data:`None`.
+ """
+
@abstractmethod
def build_dpdk(
self,
diff --git a/dts/framework/testbed_model/posix_session.py b/dts/framework/testbed_model/posix_session.py
index 94e721da61..5ab7c18fb7 100644
--- a/dts/framework/testbed_model/posix_session.py
+++ b/dts/framework/testbed_model/posix_session.py
@@ -91,6 +91,11 @@ def join_remote_path(self, *args: str | PurePath) -> PurePosixPath:
"""Overrides :meth:`~.os_session.OSSession.join_remote_path`."""
return PurePosixPath(*args)
+ def remote_path_exists(self, remote_path: str | PurePath) -> bool:
+ """Overrides :meth:`~.os_session.OSSession.remote_path_exists`."""
+ result = self.send_command(f"test -e {remote_path}")
+ return not result.return_code
+
def copy_from(self, source_file: str | PurePath, destination_dir: str | Path) -> None:
"""Overrides :meth:`~.os_session.OSSession.copy_from`."""
self.remote_session.copy_from(source_file, destination_dir)
@@ -196,6 +201,32 @@ def extract_remote_tarball(
if expected_dir:
self.send_command(f"ls {expected_dir}", verify=True)
+ def is_remote_dir(self, remote_path: str) -> bool:
+ """Overrides :meth:`~.os_session.OSSession.is_remote_dir`."""
+ result = self.send_command(f"test -d {remote_path}")
+ return not result.return_code
+
+ def is_remote_tarfile(self, remote_tarball_path: str) -> bool:
+ """Overrides :meth:`~.os_session.OSSession.is_remote_tarfile`."""
+ result = self.send_command(f"tar -tvf {remote_tarball_path}")
+ return not result.return_code
+
+ def get_tarball_top_dir(
+ self, remote_tarball_path: str | PurePath
+ ) -> str | PurePosixPath | None:
+ """Overrides :meth:`~.os_session.OSSession.get_tarball_top_dir`."""
+ members = self.send_command(f"tar tf {remote_tarball_path}").stdout.split()
+
+ top_dirs = []
+ for member in members:
+ parts_of_member = PurePosixPath(member).parts
+ if parts_of_member:
+ top_dirs.append(parts_of_member[0])
+
+ if len(set(top_dirs)) == 1:
+ return top_dirs[0]
+ return None
+
def build_dpdk(
self,
env_vars: dict,
@@ -301,7 +332,7 @@ def _get_dpdk_pids(self, dpdk_runtime_dirs: Iterable[str | PurePath]) -> list[in
pid_regex = r"p(\d+)"
for dpdk_runtime_dir in dpdk_runtime_dirs:
dpdk_config_file = PurePosixPath(dpdk_runtime_dir, "config")
- if self._remote_files_exists(dpdk_config_file):
+ if self.remote_path_exists(dpdk_config_file):
out = self.send_command(f"lsof -Fp {dpdk_config_file}").stdout
if out and "No such file or directory" not in out:
for out_line in out.splitlines():
@@ -310,10 +341,6 @@ def _get_dpdk_pids(self, dpdk_runtime_dirs: Iterable[str | PurePath]) -> list[in
pids.append(int(match.group(1)))
return pids
- def _remote_files_exists(self, remote_path: PurePath) -> bool:
- result = self.send_command(f"test -e {remote_path}")
- return not result.return_code
-
def _check_dpdk_hugepages(self, dpdk_runtime_dirs: Iterable[str | PurePath]) -> None:
"""Check there aren't any leftover hugepages.
@@ -325,7 +352,7 @@ def _check_dpdk_hugepages(self, dpdk_runtime_dirs: Iterable[str | PurePath]) ->
"""
for dpdk_runtime_dir in dpdk_runtime_dirs:
hugepage_info = PurePosixPath(dpdk_runtime_dir, "hugepage_info")
- if self._remote_files_exists(hugepage_info):
+ if self.remote_path_exists(hugepage_info):
out = self.send_command(f"lsof -Fp {hugepage_info}").stdout
if out and "No such file or directory" not in out:
self._logger.warning("Some DPDK processes did not free hugepages.")
diff --git a/dts/framework/testbed_model/sut_node.py b/dts/framework/testbed_model/sut_node.py
index 9bfb91816e..a84129d86b 100644
--- a/dts/framework/testbed_model/sut_node.py
+++ b/dts/framework/testbed_model/sut_node.py
@@ -13,20 +13,20 @@
import os
-import tarfile
import time
from pathlib import PurePath
from framework.config import (
DPDKBuildConfiguration,
DPDKBuildInfo,
+ DPDKLocation,
NodeInfo,
SutNodeConfiguration,
TestRunConfiguration,
)
+from framework.exception import ConfigurationError, RemoteFileNotFoundError
from framework.params.eal import EalParams
from framework.remote_session.remote_session import CommandResult
-from framework.settings import SETTINGS
from framework.utils import MesonArgs
from .node import Node
@@ -39,14 +39,13 @@ class SutNode(Node):
The SUT node extends :class:`Node` with DPDK specific features:
- * DPDK build,
+ * Managing DPDK source tree on the remote SUT,
+ * Building the DPDK from source or using a pre-built version,
* Gathering of DPDK build info,
* The running of DPDK apps, interactively or one-time execution,
* DPDK apps cleanup.
- The :option:`--tarball` command line argument and the :envvar:`DTS_DPDK_TARBALL`
- environment variable configure the path to the DPDK tarball
- or the git commit ID, tag ID or tree ID to test.
+ Building DPDK from source uses `build` configuration inside `dpdk_build` of configuration.
Attributes:
config: The SUT node configuration.
@@ -57,10 +56,10 @@ class SutNode(Node):
virtual_devices: list[VirtualDevice]
dpdk_prefix_list: list[str]
dpdk_timestamp: str
- _dpdk_build_config: DPDKBuildConfiguration | None
_env_vars: dict
_remote_tmp_dir: PurePath
- __remote_dpdk_dir: PurePath | None
+ __remote_dpdk_tree_path: str | PurePath | None
+ _remote_dpdk_build_dir: PurePath | None
_app_compile_timeout: float
_dpdk_kill_session: OSSession | None
_dpdk_version: str | None
@@ -77,10 +76,10 @@ def __init__(self, node_config: SutNodeConfiguration):
super().__init__(node_config)
self.virtual_devices = []
self.dpdk_prefix_list = []
- self._dpdk_build_config = None
self._env_vars = {}
self._remote_tmp_dir = self.main_session.get_remote_tmp_dir()
- self.__remote_dpdk_dir = None
+ self.__remote_dpdk_tree_path = None
+ self._remote_dpdk_build_dir = None
self._app_compile_timeout = 90
self._dpdk_kill_session = None
self.dpdk_timestamp = (
@@ -93,40 +92,34 @@ def __init__(self, node_config: SutNodeConfiguration):
self._logger.info(f"Created node: {self.name}")
@property
- def _remote_dpdk_dir(self) -> PurePath:
- """The remote DPDK dir.
-
- This internal property should be set after extracting the DPDK tarball. If it's not set,
- that implies the DPDK setup step has been skipped, in which case we can guess where
- a previous build was located.
- """
- if self.__remote_dpdk_dir is None:
- self.__remote_dpdk_dir = self._guess_dpdk_remote_dir()
- return self.__remote_dpdk_dir
-
- @_remote_dpdk_dir.setter
- def _remote_dpdk_dir(self, value: PurePath) -> None:
- self.__remote_dpdk_dir = value
+ def _remote_dpdk_tree_path(self) -> str | PurePath:
+ """The remote DPDK tree path."""
+ if self.__remote_dpdk_tree_path:
+ return self.__remote_dpdk_tree_path
+
+ self._logger.warning(
+ "Failed to get remote dpdk tree path because we don't know the "
+ "location on the SUT node."
+ )
+ return ""
@property
- def remote_dpdk_build_dir(self) -> PurePath:
- """The remote DPDK build directory.
-
- This is the directory where DPDK was built.
- We assume it was built in a subdirectory of the extracted tarball.
- """
- if self._dpdk_build_config:
- return self.main_session.join_remote_path(
- self._remote_dpdk_dir, self._dpdk_build_config.name
- )
- else:
- return self.main_session.join_remote_path(self._remote_dpdk_dir, "build")
+ def remote_dpdk_build_dir(self) -> str | PurePath:
+ """The remote DPDK build dir path."""
+ if self._remote_dpdk_build_dir:
+ return self._remote_dpdk_build_dir
+
+ self._logger.warning(
+ "Failed to get remote dpdk build dir because we don't know the "
+ "location on the SUT node."
+ )
+ return ""
@property
- def dpdk_version(self) -> str:
+ def dpdk_version(self) -> str | None:
"""Last built DPDK version."""
if self._dpdk_version is None:
- self._dpdk_version = self.main_session.get_dpdk_version(self._remote_dpdk_dir)
+ self._dpdk_version = self.main_session.get_dpdk_version(self._remote_dpdk_tree_path)
return self._dpdk_version
@property
@@ -137,26 +130,28 @@ def node_info(self) -> NodeInfo:
return self._node_info
@property
- def compiler_version(self) -> str:
+ def compiler_version(self) -> str | None:
"""The node's compiler version."""
if self._compiler_version is None:
- if self._dpdk_build_config is not None:
- self._compiler_version = self.main_session.get_compiler_version(
- self._dpdk_build_config.compiler.name
- )
- else:
- self._logger.warning(
- "Failed to get compiler version because _dpdk_build_config is None."
- )
- return ""
+ self._logger.warning("The `complier_version` is None because of using pre-built DPDK.")
+
return self._compiler_version
+ @compiler_version.setter
+ def compiler_version(self, value: str) -> None:
+ """Set the `compiler_version` used on the SUT node.
+
+ Args:
+ value: The node's compiler version.
+ """
+ self._compiler_version = value
+
@property
- def path_to_devbind_script(self) -> PurePath:
+ def path_to_devbind_script(self) -> PurePath | str:
"""The path to the dpdk-devbind.py script on the node."""
if self._path_to_devbind_script is None:
self._path_to_devbind_script = self.main_session.join_remote_path(
- self._remote_dpdk_dir, "usertools", "dpdk-devbind.py"
+ self._remote_dpdk_tree_path, "usertools", "dpdk-devbind.py"
)
return self._path_to_devbind_script
@@ -168,101 +163,247 @@ def get_dpdk_build_info(self) -> DPDKBuildInfo:
"""
return DPDKBuildInfo(dpdk_version=self.dpdk_version, compiler_version=self.compiler_version)
- def _guess_dpdk_remote_dir(self) -> PurePath:
- return self.main_session.guess_dpdk_remote_dir(self._remote_tmp_dir)
+ def set_up_test_run(
+ self, test_run_config: TestRunConfiguration, dpdk_location: DPDKLocation
+ ) -> None:
+ """Extend the test run setup with vdev config and DPDK build set up.
- def set_up_test_run(self, test_run_config: TestRunConfiguration) -> None:
- """Extend the test run setup with vdev config.
+ This method extends the setup process by configuring virtual devices and preparing the DPDK
+ environment based on the provided configuration.
Args:
test_run_config: A test run configuration according to which
the setup steps will be taken.
+ dpdk_location: The target source of the DPDK tree.
"""
- super().set_up_test_run(test_run_config)
+ super().set_up_test_run(test_run_config, dpdk_location)
for vdev in test_run_config.vdevs:
self.virtual_devices.append(VirtualDevice(vdev))
- self._set_up_dpdk(test_run_config.dpdk_build)
+ self._set_up_dpdk(dpdk_location, test_run_config.dpdk_config.dpdk_build_config)
def tear_down_test_run(self) -> None:
- """Extend the test run teardown with virtual device teardown."""
+ """Extend the test run teardown with virtual device teardown and DPDK teardown."""
super().tear_down_test_run()
self.virtual_devices = []
self._tear_down_dpdk()
- def _set_up_dpdk(self, dpdk_build_config: DPDKBuildConfiguration) -> None:
+ def _set_up_dpdk(
+ self, dpdk_location: DPDKLocation, dpdk_build_config: DPDKBuildConfiguration | None
+ ) -> None:
"""Set up DPDK the SUT node and bind ports.
- DPDK setup includes setting all internals needed for the build, the copying of DPDK tarball
- and then building DPDK. The drivers are bound to those that DPDK needs.
+ DPDK setup includes setting all internals needed for the build, the copying of DPDK
+ sources and then building DPDK or used the exist ones from the `dpdk_location`. The drivers
+ are bound to those that DPDK needs.
Args:
- dpdk_build_config: The DPDK build test run configuration according to which
- the setup steps will be taken.
+ dpdk_location: The location of the DPDK tree.
+ dpdk_build_config: A DPDK build configuration to test. If :data:`None`,
+ DTS will use pre-built DPDK from a :dataclass:`DPDKLocation`.
"""
- self._configure_dpdk_build(dpdk_build_config)
- self._copy_dpdk_tarball()
- self._build_dpdk()
+ self._set_remote_dpdk_tree_path(dpdk_location.dpdk_tree, dpdk_location.remote)
+ if not self._remote_dpdk_tree_path:
+ if dpdk_location.dpdk_tree:
+ self._copy_dpdk_tree(dpdk_location.dpdk_tree)
+ elif dpdk_location.tarball:
+ self._prepare_and_extract_dpdk_tarball(dpdk_location.tarball, dpdk_location.remote)
+
+ self._set_remote_dpdk_build_dir(dpdk_location.build_dir)
+ if not self.remote_dpdk_build_dir and dpdk_build_config:
+ self._configure_dpdk_build(dpdk_build_config)
+ self._build_dpdk()
+
self.bind_ports_to_driver()
def _tear_down_dpdk(self) -> None:
"""Reset DPDK variables and bind port driver to the OS driver."""
self._env_vars = {}
- self._dpdk_build_config = None
- self.__remote_dpdk_dir = None
+ self.__remote_dpdk_tree_path = None
+ self._remote_dpdk_build_dir = None
self._dpdk_version = None
- self._compiler_version = None
+ self.compiler_version = None
self.bind_ports_to_driver(for_dpdk=False)
+ def _set_remote_dpdk_tree_path(self, dpdk_tree: str | None, remote: bool):
+ """Set the path to the remote DPDK source tree based on the provided DPDK location.
+
+ If :data:`dpdk_tree` and :data:`remote` is defined, check existence of :data:`dpdk_tree`
+ on SUT node and sets the `_remote_dpdk_tree_path` property. Otherwise, sets nothing.
+
+ Verify DPDK source tree existence on the SUT node, if exists sets the
+ `_remote_dpdk_tree_path` property, otherwise sets nothing.
+
+ Args:
+ dpdk_tree: The path to the DPDK source tree directory.
+ remote: Indicates whether the `dpdk_tree` is already on the SUT node, instead of the
+ execution host.
+
+ Raises:
+ RemoteFileNotFoundError: If the DPDK source tree is expected to be on the SUT node but
+ is not found.
+ """
+ if remote and dpdk_tree:
+ if not self.main_session.remote_path_exists(dpdk_tree):
+ raise RemoteFileNotFoundError(
+ f"Remote DPDK source tree '{dpdk_tree}' not found in SUT node."
+ )
+ if not self.main_session.is_remote_dir(dpdk_tree):
+ raise ConfigurationError(
+ f"Remote DPDK source tree '{dpdk_tree}' had not valid format, must be "
+ "directory."
+ )
+
+ self.__remote_dpdk_tree_path = PurePath(dpdk_tree)
+
+ def _copy_dpdk_tree(self, dpdk_tree_path: str) -> None:
+ """Copy the DPDK source tree to the SUT.
+
+ Args:
+ dpdk_tree_path: The path to DPDK source tree on local filesystem.
+ """
+ self._logger.info(
+ f"Copying DPDK source tree to SUT: '{dpdk_tree_path}' into '{self._remote_tmp_dir}'."
+ )
+ self.main_session.copy_dir_to(dpdk_tree_path, self._remote_tmp_dir, exclude=".git")
+
+ self.__remote_dpdk_tree_path = self.main_session.join_remote_path(
+ self._remote_tmp_dir, PurePath(dpdk_tree_path).name
+ )
+
+ def _prepare_and_extract_dpdk_tarball(self, dpdk_tarball: str, remote: bool) -> None:
+ """Ensure the DPDK tarball is available on the SUT node and extract it.
+
+ This method ensures that the DPDK source tree tarball is available on the
+ SUT node. If the `dpdk_tarball` is local, it is copied to the SUT node. If the
+ `dpdk_tarball` is already on the SUT node, it verifies its existence.
+ The `dpdk_tarball` is then extracted on the SUT node.
+
+ This method sets the `_remote_dpdk_tree_path` property to the path of the
+ extracted DPDK tree on the SUT node.
+
+ Args:
+ dpdk_tarball: The path to the DPDK tarball, either locally or on the SUT node.
+ remote: Indicates whether the `dpdk_tarball` is already on the SUT node, instead of the
+ execution host.
+
+ Raises:
+ RemoteFileNotFoundError: If the `dpdk_tarball` is expected to be on the SUT node but
+ is not found.
+ """
+
+ def remove_tarball_suffix(remote_tarball_path: PurePath) -> PurePath:
+ """Remove the tarball suffix from the path.
+
+ Args:
+ remote_tarball_path: The path to the remote tarball.
+
+ Returns:
+ The path without the tarball suffix.
+ """
+ if len(remote_tarball_path.suffixes) > 1:
+ if remote_tarball_path.suffixes[-2] == ".tar":
+ suffixes_to_remove = "".join(remote_tarball_path.suffixes[-2:])
+ return PurePath(str(remote_tarball_path).replace(suffixes_to_remove, ""))
+ return remote_tarball_path.with_suffix("")
+
+ if remote:
+ if not self.main_session.remote_path_exists(dpdk_tarball):
+ raise RemoteFileNotFoundError(
+ f"Remote DPDK tarball '{dpdk_tarball}' not found in SUT."
+ )
+ if not self.main_session.is_remote_tarfile(dpdk_tarball):
+ raise ConfigurationError(
+ f"Remote DPDK tarball '{dpdk_tarball}' had not valid format, must be tar "
+ "archive."
+ )
+
+ remote_tarball_path = PurePath(dpdk_tarball)
+ else:
+ self._logger.info(
+ f"Copying DPDK tarball to SUT: '{dpdk_tarball}' into '{self._remote_tmp_dir}'."
+ )
+ self.main_session.copy_to(dpdk_tarball, self._remote_tmp_dir)
+
+ remote_tarball_path = self.main_session.join_remote_path(
+ self._remote_tmp_dir, PurePath(dpdk_tarball).name
+ )
+
+ tarball_top_dir = self.main_session.get_tarball_top_dir(remote_tarball_path)
+ self.__remote_dpdk_tree_path = self.main_session.join_remote_path(
+ PurePath(remote_tarball_path).parent,
+ tarball_top_dir or remove_tarball_suffix(remote_tarball_path),
+ )
+
+ self._logger.info(
+ "Extracting DPDK tarball on SUT: "
+ f"'{remote_tarball_path}' into '{self._remote_dpdk_tree_path}'."
+ )
+ self.main_session.extract_remote_tarball(
+ remote_tarball_path,
+ self._remote_dpdk_tree_path,
+ )
+
+ def _set_remote_dpdk_build_dir(self, build_dir: str | None):
+ """Set the `remote_dpdk_build_dir` on the SUT.
+
+ If :data:`build_dir` is defined, check existence on the SUT node and sets the
+ `remote_dpdk_build_dir` property by joining the `_remote_dpdk_tree_path` and `build_dir`.
+ Otherwise, sets nothing.
+
+ Args:
+ build_dir: If it's defined, DPDK has been pre-built and the build directory is located
+ in a subdirectory of `dpdk_tree` or `tarball` root directory.
+
+ Raises:
+ RemoteFileNotFoundError: If the `build_dir` is expected but does not exist on the SUT
+ node.
+ """
+ if build_dir:
+ remote_dpdk_build_dir = self.main_session.join_remote_path(
+ self._remote_dpdk_tree_path, build_dir
+ )
+ if not self.main_session.remote_path_exists(remote_dpdk_build_dir):
+ raise RemoteFileNotFoundError(
+ f"Remote DPDK build dir '{remote_dpdk_build_dir}' not found in SUT node."
+ )
+
+ self._remote_dpdk_build_dir = PurePath(remote_dpdk_build_dir)
+
def _configure_dpdk_build(self, dpdk_build_config: DPDKBuildConfiguration) -> None:
- """Populate common environment variables and set DPDK build config."""
+ """Populate common environment variables and set the DPDK build related properties.
+
+ This method sets `compiler_version` for additional information and `remote_dpdk_build_dir`
+ from DPDK build config name.
+
+ Args:
+ dpdk_build_config: A DPDK build configuration to test.
+ """
self._env_vars = {}
- self._dpdk_build_config = dpdk_build_config
self._env_vars.update(self.main_session.get_dpdk_build_env_vars(dpdk_build_config.arch))
self._env_vars["CC"] = dpdk_build_config.compiler.name
if dpdk_build_config.compiler_wrapper:
- self._env_vars["CC"] = f"'{self._dpdk_build_config.compiler_wrapper} "
- f"{self._dpdk_build_config.compiler.name}'"
-
- @Node.skip_setup
- def _copy_dpdk_tarball(self) -> None:
- """Copy to and extract DPDK tarball on the SUT node."""
- self._logger.info("Copying DPDK tarball to SUT.")
- self.main_session.copy_to(SETTINGS.dpdk_tarball_path, self._remote_tmp_dir)
-
- # construct remote tarball path
- # the basename is the same on local host and on remote Node
- remote_tarball_path = self.main_session.join_remote_path(
- self._remote_tmp_dir, os.path.basename(SETTINGS.dpdk_tarball_path)
- )
+ self._env_vars[
+ "CC"
+ ] = f"'{dpdk_build_config.compiler_wrapper} {dpdk_build_config.compiler.name}'"
- # construct remote path after extracting
- with tarfile.open(SETTINGS.dpdk_tarball_path) as dpdk_tar:
- dpdk_top_dir = dpdk_tar.getnames()[0]
- self._remote_dpdk_dir = self.main_session.join_remote_path(
- self._remote_tmp_dir, dpdk_top_dir
+ self.compiler_version = self.main_session.get_compiler_version(
+ dpdk_build_config.compiler.name
)
- self._logger.info(
- f"Extracting DPDK tarball on SUT: "
- f"'{remote_tarball_path}' into '{self._remote_dpdk_dir}'."
+ self._remote_dpdk_build_dir = self.main_session.join_remote_path(
+ self._remote_dpdk_tree_path, dpdk_build_config.name
)
- # clean remote path where we're extracting
- self.main_session.remove_remote_dir(self._remote_dpdk_dir)
-
- # then extract to remote path
- self.main_session.extract_remote_tarball(remote_tarball_path, self._remote_dpdk_dir)
- @Node.skip_setup
def _build_dpdk(self) -> None:
"""Build DPDK.
- Uses the already configured target. Assumes that the tarball has
- already been copied to and extracted on the SUT node.
+ Uses the already configured DPDK build configuration. Assumes that the
+ `_remote_dpdk_tree_path` has already been set on the SUT node.
"""
self.main_session.build_dpdk(
self._env_vars,
MesonArgs(default_library="static", enable_kmods=True, libdir="lib"),
- self._remote_dpdk_dir,
+ self._remote_dpdk_tree_path,
self.remote_dpdk_build_dir,
)
@@ -285,7 +426,7 @@ def build_dpdk_app(self, app_name: str, **meson_dpdk_args: str | bool) -> PurePa
self._env_vars,
MesonArgs(examples=app_name, **meson_dpdk_args), # type: ignore [arg-type]
# ^^ https://github.com/python/mypy/issues/11583
- self._remote_dpdk_dir,
+ self._remote_dpdk_tree_path,
self.remote_dpdk_build_dir,
rebuild=True,
timeout=self._app_compile_timeout,
--
2.46.1
next prev parent reply other threads:[~2024-09-30 16:18 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-09-30 16:18 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
2024-09-30 16:18 ` [PATCH 1/7] dts: rename build target to " Tomáš Ďurovec
2024-09-30 16:18 ` [PATCH 2/7] dts: one dpdk build per test run Tomáš Ďurovec
2024-09-30 16:18 ` [PATCH 3/7] dts: fix remote session file transfer vars Tomáš Ďurovec
2024-09-30 16:18 ` [PATCH 4/7] dts: add the ability to copy directories via remote Tomáš Ďurovec
2024-09-30 16:18 ` Tomáš Ďurovec [this message]
2024-09-30 16:18 ` [PATCH 6/7] doc: update argument options for external DPDK build Tomáš Ďurovec
2024-09-30 16:18 ` [PATCH 7/7] dts: remove git ref option Tomáš Ďurovec
-- strict thread matches above, loose matches on Subject: below --
2024-09-30 16:01 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
2024-09-30 16:02 ` [PATCH 5/7] dts: add support for externally compiled DPDK Tomáš Ďurovec
2024-09-27 16:08 [PATCH 0/7] DTS external DPDK build Tomáš Ďurovec
2024-09-27 16:08 ` [PATCH 5/7] dts: add support for externally compiled DPDK Tomáš Ďurovec
2024-09-27 15:38 [PATCH 1/7] dts: rename build target to DPDK build Tomáš Ďurovec
2024-09-27 15:38 ` [PATCH 5/7] dts: add support for externally compiled DPDK Tomáš Ďurovec
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240930161814.26070-6-tomas.durovec@pantheon.tech \
--to=tomas.durovec@pantheon.tech \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).