[PATCH 0/4] dts: error and usage improvements

[PATCH 0/4] dts: error and usage improvements

[Luca Vizzarro]

As mentioned in my previous DTS docs improvement patch series, here are
some usage improvements to DTS. The main purpose is to give the
first-time user of DTS some more meaningful messages of its usage.

Secondly, report back stderr to the user when remote commands fail. For
example, if DTS tries to run any program which is not installed on the
target node, it will just say that it failed with its return code. The
only way to see the actual error message is through the DEBUG level of
verbosity. Rightfully though, errors should be logged as ERROR.

Best,
Luca

Luca Vizzarro (4):
  dts: constrain DPDK source flag
  dts: customise argparse error message
  dts: show help when DTS is ran without args
  dts: log stderr with failed remote commands

 doc/guides/tools/dts.rst                      |  8 +-
 dts/framework/exception.py                    | 10 ++-
 .../remote_session/remote_session.py          |  2 +-
 dts/framework/settings.py                     | 83 ++++++++++++++-----
 dts/framework/utils.py                        | 43 ++++++----
 5 files changed, 104 insertions(+), 42 deletions(-)

-- 
2.34.1

[PATCH 1/4] dts: constrain DPDK source flag

[Luca Vizzarro]

DTS needs an input to gather the DPDK source code from. This is then
built on the remote target. This commit makes sure that this input is
more constrained, separating the Git revision ID – used to create a
tarball using Git – and providing tarballed source code directly, while
retaining mutual exclusion.

This makes the code more readable and easier to handle for input
validation, of which this commit introduces a basic one based on the
pre-existing code.

Moreover it ensures that these flags are explicitly required to be set
by the user, dropping a default value. It also aids the user understand
how to use the DTS in the scenario it is ran without any arguments set.

Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
---
 doc/guides/tools/dts.rst  |  8 +++--
 dts/framework/settings.py | 64 ++++++++++++++++++++++++++++-----------
 dts/framework/utils.py    | 43 ++++++++++++++++----------
 3 files changed, 79 insertions(+), 36 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 846696e14e..6e2da317e8 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,12 +215,16 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN]
+   usage: main.py [-h] (--tarball FILEPATH | --revision ID) [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN]
 
    Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
 
    options:
    -h, --help            show this help message and exit
+   --tarball FILEPATH, --snapshot FILEPATH
+                         Path to DPDK source code tarball to test. (default: None)
+   --revision ID, --rev ID, --git-ref ID
+                         Git revision ID to test. Could be commit, tag, tree ID and vice versa. To test local changes, first commit them, then use their commit ID (default: None)
    --config-file CONFIG_FILE
                          [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
    --output-dir OUTPUT_DIR, --output OUTPUT_DIR
@@ -229,8 +233,6 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
                          [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
    -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
    -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None)
-   --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
-                         [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
    --compile-timeout COMPILE_TIMEOUT
                          [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
    --test-cases TEST_CASES
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 609c8d0e62..2d0365e763 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -76,7 +76,8 @@
 from pathlib import Path
 from typing import Any, TypeVar
 
-from .utils import DPDKGitTarball
+from .exception import ConfigurationError
+from .utils import DPDKGitTarball, get_commit_id
 
 _T = TypeVar("_T")
 
@@ -149,6 +150,26 @@ def __call__(
     return _EnvironmentArgument
 
 
+def _parse_tarball(filepath: str) -> Path:
+    """Validate if the filepath is valid and return a Path object."""
+    path = Path(filepath)
+    if not path.exists() or not path.is_file():
+        raise argparse.ArgumentTypeError(
+            "the file path provided is not a valid file")
+    return path
+
+
+def _parse_revision_id(rev_id: str) -> str:
+    """Retrieve effective commit ID from a revision ID. While validating it."""
+
+    try:
+        return get_commit_id(rev_id)
+    except ConfigurationError:
+        raise argparse.ArgumentTypeError(
+            "the Git revision ID supplied is invalid or ambiguous"
+        )
+
+
 @dataclass(slots=True)
 class Settings:
     """Default framework-wide user settings.
@@ -167,7 +188,7 @@ class Settings:
     #:
     skip_setup: bool = False
     #:
-    dpdk_tarball_path: Path | str = "dpdk.tar.xz"
+    dpdk_tarball_path: Path | str = ""
     #:
     compile_timeout: float = 1200
     #:
@@ -186,6 +207,28 @@ def _get_parser() -> argparse.ArgumentParser:
         formatter_class=argparse.ArgumentDefaultsHelpFormatter,
     )
 
+    dpdk_source = parser.add_mutually_exclusive_group(required=True)
+
+    dpdk_source.add_argument(
+        "--tarball",
+        "--snapshot",
+        action='store',
+        type=_parse_tarball,
+        help="Path to DPDK source code tarball to test.",
+        metavar="FILEPATH",
+    )
+
+    dpdk_source.add_argument(
+        "--revision",
+        "--rev",
+        "--git-ref",
+        type=_parse_revision_id,
+        metavar="ID",
+        help="Git revision ID to test. Could be commit, tag, tree ID and "
+        "vice versa. To test local changes, first commit them, then use their "
+        "commit ID",
+    )
+
     parser.add_argument(
         "--config-file",
         action=_env_arg("DTS_CFG_FILE"),
@@ -229,18 +272,6 @@ def _get_parser() -> argparse.ArgumentParser:
         help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
     )
 
-    parser.add_argument(
-        "--tarball",
-        "--snapshot",
-        "--git-ref",
-        action=_env_arg("DTS_DPDK_TARBALL"),
-        default=SETTINGS.dpdk_tarball_path,
-        type=Path,
-        help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
-        "tag ID or tree ID to test. To test local changes, first commit them, "
-        "then use the commit ID with this option.",
-    )
-
     parser.add_argument(
         "--compile-timeout",
         action=_env_arg("DTS_COMPILE_TIMEOUT"),
@@ -283,9 +314,8 @@ def get_settings() -> Settings:
         verbose=parsed_args.verbose,
         skip_setup=parsed_args.skip_setup,
         dpdk_tarball_path=Path(
-            Path(DPDKGitTarball(parsed_args.tarball, parsed_args.output_dir))
-            if not os.path.exists(parsed_args.tarball)
-            else Path(parsed_args.tarball)
+            parsed_args.tarball or
+            DPDKGitTarball(parsed_args.revision, parsed_args.output_dir)
         ),
         compile_timeout=parsed_args.compile_timeout,
         test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []),
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index cc5e458cc8..dbbec8b00a 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -70,6 +70,32 @@ def get_packet_summaries(packets: list[Packet]) -> str:
     return f"Packet contents: \n{packet_summaries}"
 
 
+def get_commit_id(rev_id: str) -> str:
+    """Given a Git revision ID, return the corresponding commit ID.
+
+    Args:
+        rev_id: The Git revision ID.
+
+    Raises:
+        ConfigurationError: The ``git rev-parse`` command failed. Suggesting
+            an invalid or ambiguous revision ID was supplied.
+    """
+    result = subprocess.run(
+        ["git", "rev-parse", "--verify", rev_id],
+        text=True,
+        capture_output=True,
+    )
+    if result.returncode != 0:
+        raise ConfigurationError(
+            f"{rev_id} is neither a path to an existing DPDK "
+            "archive nor a valid git reference.\n"
+            f"Command: {result.args}\n"
+            f"Stdout: {result.stdout}\n"
+            f"Stderr: {result.stderr}"
+        )
+    return result.stdout.strip()
+
+
 class StrEnum(Enum):
     """Enum with members stored as strings."""
 
@@ -170,7 +196,6 @@ def __init__(
 
         self._tarball_dir = Path(output_dir, "tarball")
 
-        self._get_commit_id()
         self._create_tarball_dir()
 
         self._tarball_name = (
@@ -180,21 +205,7 @@ def __init__(
         if not self._tarball_path:
             self._create_tarball()
 
-    def _get_commit_id(self) -> None:
-        result = subprocess.run(
-            ["git", "rev-parse", "--verify", self._git_ref],
-            text=True,
-            capture_output=True,
-        )
-        if result.returncode != 0:
-            raise ConfigurationError(
-                f"{self._git_ref} is neither a path to an existing DPDK "
-                "archive nor a valid git reference.\n"
-                f"Command: {result.args}\n"
-                f"Stdout: {result.stdout}\n"
-                f"Stderr: {result.stderr}"
-            )
-        self._git_ref = result.stdout.strip()
+
 
     def _create_tarball_dir(self) -> None:
         os.makedirs(self._tarball_dir, exist_ok=True)
-- 
2.34.1

Re: [PATCH 1/4] dts: constrain DPDK source flag

[Juraj Linkeš]

On Mon, Jan 22, 2024 at 7:26 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> DTS needs an input to gather the DPDK source code from. This is then
> built on the remote target. This commit makes sure that this input is
> more constrained, separating the Git revision ID – used to create a
> tarball using Git – and providing tarballed source code directly, while
> retaining mutual exclusion.
>

I didn't see the mutual exclusion being enforced in the code. From
what I can tell, I could pass both --tarball FILEPATH and --revision
and the former would be used without checking the latter.

> This makes the code more readable and easier to handle for input
> validation, of which this commit introduces a basic one based on the
> pre-existing code.
>

I wanted to have just one argument for basically the same thing, but
the input is pretty different so a two argument solution actually
sounds better.

> Moreover it ensures that these flags are explicitly required to be set
> by the user, dropping a default value. It also aids the user understand
> how to use the DTS in the scenario it is ran without any arguments set.
>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> ---
>  doc/guides/tools/dts.rst  |  8 +++--
>  dts/framework/settings.py | 64 ++++++++++++++++++++++++++++-----------
>  dts/framework/utils.py    | 43 ++++++++++++++++----------
>  3 files changed, 79 insertions(+), 36 deletions(-)
>
> diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
> index 846696e14e..6e2da317e8 100644
> --- a/doc/guides/tools/dts.rst
> +++ b/doc/guides/tools/dts.rst
> @@ -215,12 +215,16 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
>  .. code-block:: console
>
>     (dts-py3.10) $ ./main.py --help
> -   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN]
> +   usage: main.py [-h] (--tarball FILEPATH | --revision ID) [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN]
>
>     Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
>
>     options:
>     -h, --help            show this help message and exit
> +   --tarball FILEPATH, --snapshot FILEPATH
> +                         Path to DPDK source code tarball to test. (default: None)
> +   --revision ID, --rev ID, --git-ref ID
> +                         Git revision ID to test. Could be commit, tag, tree ID and vice versa. To test local changes, first commit them, then use their commit ID (default: None)
>     --config-file CONFIG_FILE
>                           [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
>     --output-dir OUTPUT_DIR, --output OUTPUT_DIR
> @@ -229,8 +233,6 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
>                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
>     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
>     -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None)
> -   --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
> -                         [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
>     --compile-timeout COMPILE_TIMEOUT
>                           [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
>     --test-cases TEST_CASES
> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> index 609c8d0e62..2d0365e763 100644
> --- a/dts/framework/settings.py
> +++ b/dts/framework/settings.py
> @@ -76,7 +76,8 @@
>  from pathlib import Path
>  from typing import Any, TypeVar
>
> -from .utils import DPDKGitTarball
> +from .exception import ConfigurationError
> +from .utils import DPDKGitTarball, get_commit_id
>
>  _T = TypeVar("_T")
>
> @@ -149,6 +150,26 @@ def __call__(
>      return _EnvironmentArgument
>
>
> +def _parse_tarball(filepath: str) -> Path:
> +    """Validate if the filepath is valid and return a Path object."""

whether `filepath` is valid

Even though private methods don't get included in the API docs, I like
to be consistent. In this case, it doesn't really detract (but maybe
some disability would prove this wrong) while adding a bit (the fact
that we're referencing the argument).

I think the name should either be _validate_tarball or
_parse_tarball_path. The argument name is two words, so let's put an
underscore between them.

> +    path = Path(filepath)
> +    if not path.exists() or not path.is_file():
> +        raise argparse.ArgumentTypeError(
> +            "the file path provided is not a valid file")

Typo: capital T

> +    return path
> +
> +
> +def _parse_revision_id(rev_id: str) -> str:
> +    """Retrieve effective commit ID from a revision ID. While validating it."""

I think this would read better as one sentence.

> +
> +    try:
> +        return get_commit_id(rev_id)
> +    except ConfigurationError:
> +        raise argparse.ArgumentTypeError(
> +            "the Git revision ID supplied is invalid or ambiguous"
> +        )
> +
> +
>  @dataclass(slots=True)
>  class Settings:
>      """Default framework-wide user settings.
> @@ -167,7 +188,7 @@ class Settings:
>      #:
>      skip_setup: bool = False
>      #:
> -    dpdk_tarball_path: Path | str = "dpdk.tar.xz"
> +    dpdk_tarball_path: Path | str = ""
>      #:
>      compile_timeout: float = 1200
>      #:
> @@ -186,6 +207,28 @@ def _get_parser() -> argparse.ArgumentParser:
>          formatter_class=argparse.ArgumentDefaultsHelpFormatter,
>      )
>
> +    dpdk_source = parser.add_mutually_exclusive_group(required=True)
> +
> +    dpdk_source.add_argument(
> +        "--tarball",
> +        "--snapshot",
> +        action='store',
> +        type=_parse_tarball,
> +        help="Path to DPDK source code tarball to test.",
> +        metavar="FILEPATH",
> +    )
> +
> +    dpdk_source.add_argument(
> +        "--revision",
> +        "--rev",
> +        "--git-ref",
> +        type=_parse_revision_id,
> +        metavar="ID",

Since this is a patch with improvements, maybe we could add metavars
to other arguments as well. It looks pretty good.

> +        help="Git revision ID to test. Could be commit, tag, tree ID and "
> +        "vice versa. To test local changes, first commit them, then use their "
> +        "commit ID",
> +    )
> +

This removes the support for environment variables. It's possible we
don't want the support for these two arguments. Maybe we don't need
the support for variables at all. I'm leaning towards supporting the
env variables, but we probably should refactor how they're done. The
easiest would be to not do them with action, but just modifying the
default value if set. That would be a worthwhile improvement.

>      parser.add_argument(
>          "--config-file",
>          action=_env_arg("DTS_CFG_FILE"),
> @@ -229,18 +272,6 @@ def _get_parser() -> argparse.ArgumentParser:
>          help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
>      )
>
> -    parser.add_argument(
> -        "--tarball",
> -        "--snapshot",
> -        "--git-ref",
> -        action=_env_arg("DTS_DPDK_TARBALL"),
> -        default=SETTINGS.dpdk_tarball_path,
> -        type=Path,
> -        help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
> -        "tag ID or tree ID to test. To test local changes, first commit them, "
> -        "then use the commit ID with this option.",
> -    )
> -
>      parser.add_argument(
>          "--compile-timeout",
>          action=_env_arg("DTS_COMPILE_TIMEOUT"),
> @@ -283,9 +314,8 @@ def get_settings() -> Settings:
>          verbose=parsed_args.verbose,
>          skip_setup=parsed_args.skip_setup,
>          dpdk_tarball_path=Path(
> -            Path(DPDKGitTarball(parsed_args.tarball, parsed_args.output_dir))
> -            if not os.path.exists(parsed_args.tarball)
> -            else Path(parsed_args.tarball)
> +            parsed_args.tarball or
> +            DPDKGitTarball(parsed_args.revision, parsed_args.output_dir)
>          ),
>          compile_timeout=parsed_args.compile_timeout,
>          test_cases=(parsed_args.test_cases.split(",") if parsed_args.test_cases else []),
> diff --git a/dts/framework/utils.py b/dts/framework/utils.py
> index cc5e458cc8..dbbec8b00a 100644
> --- a/dts/framework/utils.py
> +++ b/dts/framework/utils.py
> @@ -70,6 +70,32 @@ def get_packet_summaries(packets: list[Packet]) -> str:
>      return f"Packet contents: \n{packet_summaries}"
>
>
> +def get_commit_id(rev_id: str) -> str:
> +    """Given a Git revision ID, return the corresponding commit ID.
> +
> +    Args:
> +        rev_id: The Git revision ID.
> +
> +    Raises:
> +        ConfigurationError: The ``git rev-parse`` command failed. Suggesting
> +            an invalid or ambiguous revision ID was supplied.

This would also probably read better as one sentence.

> +    """
> +    result = subprocess.run(
> +        ["git", "rev-parse", "--verify", rev_id],
> +        text=True,
> +        capture_output=True,
> +    )
> +    if result.returncode != 0:
> +        raise ConfigurationError(
> +            f"{rev_id} is neither a path to an existing DPDK "
> +            "archive nor a valid git reference.\n"
> +            f"Command: {result.args}\n"
> +            f"Stdout: {result.stdout}\n"
> +            f"Stderr: {result.stderr}"
> +        )

We shuffled the order of operations a bit and now the error message
doesn't correspond.

> +    return result.stdout.strip()
> +
> +
>  class StrEnum(Enum):
>      """Enum with members stored as strings."""
>
> @@ -170,7 +196,6 @@ def __init__(
>
>          self._tarball_dir = Path(output_dir, "tarball")
>
> -        self._get_commit_id()

Makes sense to move this outside.

>          self._create_tarball_dir()
>
>          self._tarball_name = (
> @@ -180,21 +205,7 @@ def __init__(
>          if not self._tarball_path:
>              self._create_tarball()
>
> -    def _get_commit_id(self) -> None:
> -        result = subprocess.run(
> -            ["git", "rev-parse", "--verify", self._git_ref],
> -            text=True,
> -            capture_output=True,
> -        )
> -        if result.returncode != 0:
> -            raise ConfigurationError(
> -                f"{self._git_ref} is neither a path to an existing DPDK "
> -                "archive nor a valid git reference.\n"
> -                f"Command: {result.args}\n"
> -                f"Stdout: {result.stdout}\n"
> -                f"Stderr: {result.stderr}"
> -            )
> -        self._git_ref = result.stdout.strip()
> +
>
>      def _create_tarball_dir(self) -> None:
>          os.makedirs(self._tarball_dir, exist_ok=True)
> --
> 2.34.1
>

Re: [PATCH 1/4] dts: constrain DPDK source flag

[Luca Vizzarro]

Hi Juraj,

Thank you for your review!

On 29/01/2024 11:47, Juraj Linkeš wrote:
> I didn't see the mutual exclusion being enforced in the code. From
> what I can tell, I could pass both --tarball FILEPATH and --revision
> and the former would be used without checking the latter.

Yep, it is enforced in the code, you may have missed it. The two 
arguments are under the same mutual exclusion group in line 220:

dpdk_source = parser.add_mutually_exclusive_group(required=True)

When using both arguments `argparse` will automatically complain that 
you can only use one or the other.

> whether `filepath` is valid
> Even though private methods don't get included in the API docs, I like
> to be consistent. In this case, it doesn't really detract (but maybe
> some disability would prove this wrong) while adding a bit (the fact
> that we're referencing the argument).

Yes, it is a good idea. Especially since this will work within IDEs.

> I think the name should either be _validate_tarball or
> _parse_tarball_path. The argument name is two words, so let's put an
> underscore between them.

Ack.

> I think this would read better as one sentence.

Ack.

> Since this is a patch with improvements, maybe we could add metavars
> to other arguments as well. It looks pretty good.

Sure, no problem!

> This removes the support for environment variables. It's possible we
> don't want the support for these two arguments. Maybe we don't need
> the support for variables at all. I'm leaning towards supporting the
> env variables, but we probably should refactor how they're done. The
> easiest would be to not do them with action, but just modifying the
> default value if set. That would be a worthwhile improvement.

I've tried to find a way to still keep them. But with arguments done 
this way, it is somewhat hard to understand the provenance of the value, 
whether it's set in the arguments, an environment variable or just the 
default value. Therefore, give a useful error message to the user when 
using something invalid. I'll try to come up with something as you 
suggested, although I am not entirely convinced it'll be ideal.

> This would also probably read better as one sentence.

Ack.

> We shuffled the order of operations a bit and now the error message
> doesn't correspond.

Sorry, I don't think I am understanding what you are referring to 
exactly. What do you mean?

Best,
Luca

Re: [PATCH 1/4] dts: constrain DPDK source flag

[Juraj Linkeš]

On Fri, Feb 23, 2024 at 8:09 PM Luca Vizzarro <Luca.Vizzarro@arm.com> wrote:
>
> Hi Juraj,
>
> Thank you for your review!
>
> On 29/01/2024 11:47, Juraj Linkeš wrote:
> > I didn't see the mutual exclusion being enforced in the code. From
> > what I can tell, I could pass both --tarball FILEPATH and --revision
> > and the former would be used without checking the latter.
>
> Yep, it is enforced in the code, you may have missed it. The two
> arguments are under the same mutual exclusion group in line 220:
>
> dpdk_source = parser.add_mutually_exclusive_group(required=True)
>
> When using both arguments `argparse` will automatically complain that
> you can only use one or the other.
>

Yes, I missed this. This looks great.

> > whether `filepath` is valid
> > Even though private methods don't get included in the API docs, I like
> > to be consistent. In this case, it doesn't really detract (but maybe
> > some disability would prove this wrong) while adding a bit (the fact
> > that we're referencing the argument).
>
> Yes, it is a good idea. Especially since this will work within IDEs.
>
> > I think the name should either be _validate_tarball or
> > _parse_tarball_path. The argument name is two words, so let's put an
> > underscore between them.
>
> Ack.
>
> > I think this would read better as one sentence.
>
> Ack.
>
> > Since this is a patch with improvements, maybe we could add metavars
> > to other arguments as well. It looks pretty good.
>
> Sure, no problem!
>
> > This removes the support for environment variables. It's possible we
> > don't want the support for these two arguments. Maybe we don't need
> > the support for variables at all. I'm leaning towards supporting the
> > env variables, but we probably should refactor how they're done. The
> > easiest would be to not do them with action, but just modifying the
> > default value if set. That would be a worthwhile improvement.
>
> I've tried to find a way to still keep them. But with arguments done
> this way, it is somewhat hard to understand the provenance of the value,
> whether it's set in the arguments, an environment variable or just the
> default value. Therefore, give a useful error message to the user when
> using something invalid. I'll try to come up with something as you
> suggested, although I am not entirely convinced it'll be ideal.
>

For reference, my test case blocking patch implements an alternative
approach: https://patches.dpdk.org/project/dpdk/patch/20240223075502.60485-8-juraj.linkes@pantheon.tech/

It's the same thing, significantly simplified. Looks like it could work here.

> > This would also probably read better as one sentence.
>
> Ack.
>
> > We shuffled the order of operations a bit and now the error message
> > doesn't correspond.
>
> Sorry, I don't think I am understanding what you are referring to
> exactly. What do you mean?
>

You removed what I commented on (in the future, please keep that so
what we know what the original comment relates to), so here's what I
commented on:

+    if result.returncode != 0:
+        raise ConfigurationError(
+            f"{rev_id} is neither a path to an existing DPDK "
+            "archive nor a valid git reference.\n"
+            f"Command: {result.args}\n"
+            f"Stdout: {result.stdout}\n"
+            f"Stderr: {result.stderr}"
+        )

Previously, this was inside DPDKGitTarball which assumed it was passed
git ref because the command line arg is not a path. In this patch, the
git ref arg is decoupled from the tarball path, so the function
shouldn't reference the path.

> Best,
> Luca

[PATCH 2/4] dts: customise argparse error message

[Luca Vizzarro]

This commit customises the arguments parsing class' error message,
making it so the confusing usage is not displayed in these occurrences,
but the user is redirected to use the help argument instead.

Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
---
 dts/framework/settings.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 2d0365e763..acfe5cad44 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -170,6 +170,15 @@ def _parse_revision_id(rev_id: str) -> str:
         )
 
 
+class ArgumentParser(argparse.ArgumentParser):
+    """ArgumentParser with a custom error message."""
+    def error(self, message):
+        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
+        self.exit(2,
+                  "For help and usage, "
+                  "run the command with the --help flag.\n")
+
+
 @dataclass(slots=True)
 class Settings:
     """Default framework-wide user settings.
@@ -200,8 +209,8 @@ class Settings:
 SETTINGS: Settings = Settings()
 
 
-def _get_parser() -> argparse.ArgumentParser:
-    parser = argparse.ArgumentParser(
+def _get_parser() -> ArgumentParser:
+    parser = ArgumentParser(
         description="Run DPDK test suites. All options may be specified with the environment "
         "variables provided in brackets. Command line arguments have higher priority.",
         formatter_class=argparse.ArgumentDefaultsHelpFormatter,
-- 
2.34.1

Re: [PATCH 2/4] dts: customise argparse error message

[Juraj Linkeš]

On Mon, Jan 22, 2024 at 7:26 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> This commit customises the arguments parsing class' error message,
> making it so the confusing usage is not displayed in these occurrences,

I'm curious, what exactly is confusing about the message?

> but the user is redirected to use the help argument instead.
>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> ---
>  dts/framework/settings.py | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> index 2d0365e763..acfe5cad44 100644
> --- a/dts/framework/settings.py
> +++ b/dts/framework/settings.py
> @@ -170,6 +170,15 @@ def _parse_revision_id(rev_id: str) -> str:
>          )
>
>
> +class ArgumentParser(argparse.ArgumentParser):
> +    """ArgumentParser with a custom error message."""
> +    def error(self, message):
> +        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
> +        self.exit(2,
> +                  "For help and usage, "
> +                  "run the command with the --help flag.\n")
> +
> +
>  @dataclass(slots=True)
>  class Settings:
>      """Default framework-wide user settings.
> @@ -200,8 +209,8 @@ class Settings:
>  SETTINGS: Settings = Settings()
>
>
> -def _get_parser() -> argparse.ArgumentParser:
> -    parser = argparse.ArgumentParser(
> +def _get_parser() -> ArgumentParser:
> +    parser = ArgumentParser(
>          description="Run DPDK test suites. All options may be specified with the environment "
>          "variables provided in brackets. Command line arguments have higher priority.",
>          formatter_class=argparse.ArgumentDefaultsHelpFormatter,
> --
> 2.34.1
>

Re: [PATCH 2/4] dts: customise argparse error message

[Luca Vizzarro]

On 29/01/2024 13:04, Juraj Linkeš wrote:
> I'm curious, what exactly is confusing about the message?

Unfortunately a bit too much time has passed... but if I remember 
correctly I think that given the great amount of arguments, whenever the 
message is printed a bit too much information is given to the user. So 
bottomline, too crowded

Re: [PATCH 2/4] dts: customise argparse error message

[Juraj Linkeš]

On Fri, Feb 23, 2024 at 8:12 PM Luca Vizzarro <Luca.Vizzarro@arm.com> wrote:
>
> On 29/01/2024 13:04, Juraj Linkeš wrote:
> > I'm curious, what exactly is confusing about the message?
>
> Unfortunately a bit too much time has passed... but if I remember
> correctly I think that given the great amount of arguments, whenever the
> message is printed a bit too much information is given to the user. So
> bottomline, too crowded

The original message is:
./main.py -wdghf
usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir
OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL]
[--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE
[TEST_CASES ...]] [--re-run RE_RUN]
main.py: error: unrecognized arguments: -wdghf

So this is what's confusing. I guess it doesn't mention that the user
should use the help argument and that's where the confusion was? From
my point of view that's just standard (to run a command with -h in
case of an error such as the one above), but maybe it is better to
state it explicitly.

[PATCH 3/4] dts: show help when DTS is ran without args

[Luca Vizzarro]

This commit changes the default behaviour of DTS, making it so that the
user automatically sees the help and usage page when running it without
any arguments set. Instead of being welcomed by an error message.

Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
---
 dts/framework/settings.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index acfe5cad44..5809fd4e91 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -71,6 +71,7 @@
 
 import argparse
 import os
+import sys
 from collections.abc import Callable, Iterable, Sequence
 from dataclasses import dataclass, field
 from pathlib import Path
@@ -315,6 +316,11 @@ def get_settings() -> Settings:
 
     The inputs are taken from the command line and from environment variables.
     """
+
+    if len(sys.argv) == 1:
+        _get_parser().print_help()
+        sys.exit(1)
+
     parsed_args = _get_parser().parse_args()
     return Settings(
         config_file_path=parsed_args.config_file,
-- 
2.34.1

[PATCH 4/4] dts: log stderr with failed remote commands

[Luca Vizzarro]

Add the executed command stderr to RemoteCommandExecutionError. So that
it is logged as an error, instead of just as debug.

Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
---
 dts/framework/exception.py                     | 10 +++++++---
 dts/framework/remote_session/remote_session.py |  2 +-
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/dts/framework/exception.py b/dts/framework/exception.py
index 658eee2c38..9fc3fa096a 100644
--- a/dts/framework/exception.py
+++ b/dts/framework/exception.py
@@ -130,20 +130,24 @@ class RemoteCommandExecutionError(DTSError):
     #: The executed command.
     command: str
     _command_return_code: int
+    _command_stderr: str
 
-    def __init__(self, command: str, command_return_code: int):
+    def __init__(self, command: str, command_return_code: int, command_stderr: str):
         """Define the meaning of the first two arguments.
 
         Args:
             command: The executed command.
             command_return_code: The return code of the executed command.
+            command_stderr: The stderr of the executed command.
         """
         self.command = command
         self._command_return_code = command_return_code
+        self._command_stderr = command_stderr
 
     def __str__(self) -> str:
-        """Include both the command and return code in the string representation."""
-        return f"Command {self.command} returned a non-zero exit code: {self._command_return_code}"
+        """Include the command, its return code and stderr in the string representation."""
+        return (f"Command '{self.command}' returned a non-zero exit code: "
+                f"{self._command_return_code}\n{self._command_stderr}")
 
 
 class RemoteDirectoryExistsError(DTSError):
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
index 2059f9a981..345439f2de 100644
--- a/dts/framework/remote_session/remote_session.py
+++ b/dts/framework/remote_session/remote_session.py
@@ -157,7 +157,7 @@ def send_command(
             )
             self._logger.debug(f"stdout: '{result.stdout}'")
             self._logger.debug(f"stderr: '{result.stderr}'")
-            raise RemoteCommandExecutionError(command, result.return_code)
+            raise RemoteCommandExecutionError(command, result.return_code, result.stderr)
         self._logger.debug(f"Received from '{command}':\n{result}")
         self.history.append(result)
         return result
-- 
2.34.1

Re: [PATCH 4/4] dts: log stderr with failed remote commands

[Juraj Linkeš]

On Mon, Jan 22, 2024 at 7:26 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> Add the executed command stderr to RemoteCommandExecutionError. So that
> it is logged as an error, instead of just as debug.

Here's I'd add logged additionally as an error, as this sounds as if
we're changing debug to error.

>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> ---
>  dts/framework/exception.py                     | 10 +++++++---
>  dts/framework/remote_session/remote_session.py |  2 +-
>  2 files changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/dts/framework/exception.py b/dts/framework/exception.py
> index 658eee2c38..9fc3fa096a 100644
> --- a/dts/framework/exception.py
> +++ b/dts/framework/exception.py
> @@ -130,20 +130,24 @@ class RemoteCommandExecutionError(DTSError):
>      #: The executed command.
>      command: str
>      _command_return_code: int
> +    _command_stderr: str
>

I'd change the order here (and all other places) so that stderr is
before the return code.

> -    def __init__(self, command: str, command_return_code: int):
> +    def __init__(self, command: str, command_return_code: int, command_stderr: str):
>          """Define the meaning of the first two arguments.
>
>          Args:
>              command: The executed command.
>              command_return_code: The return code of the executed command.
> +            command_stderr: The stderr of the executed command.
>          """
>          self.command = command
>          self._command_return_code = command_return_code
> +        self._command_stderr = command_stderr
>
>      def __str__(self) -> str:
> -        """Include both the command and return code in the string representation."""
> -        return f"Command {self.command} returned a non-zero exit code: {self._command_return_code}"
> +        """Include the command, its return code and stderr in the string representation."""
> +        return (f"Command '{self.command}' returned a non-zero exit code: "
> +                f"{self._command_return_code}\n{self._command_stderr}")

We should mention that the last string is the stderr output. Maybe we
just add 'Stderr:' before {self._command_stderr}. And maybe we should
put quotes around {self._command_stderr}.

>
>
>  class RemoteDirectoryExistsError(DTSError):
> diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
> index 2059f9a981..345439f2de 100644
> --- a/dts/framework/remote_session/remote_session.py
> +++ b/dts/framework/remote_session/remote_session.py
> @@ -157,7 +157,7 @@ def send_command(
>              )
>              self._logger.debug(f"stdout: '{result.stdout}'")
>              self._logger.debug(f"stderr: '{result.stderr}'")
> -            raise RemoteCommandExecutionError(command, result.return_code)
> +            raise RemoteCommandExecutionError(command, result.return_code, result.stderr)
>          self._logger.debug(f"Received from '{command}':\n{result}")
>          self.history.append(result)
>          return result
> --
> 2.34.1
>

Re: [PATCH 4/4] dts: log stderr with failed remote commands

[Luca Vizzarro]

On 29/01/2024 13:10, Juraj Linkeš wrote:
> Here's I'd add logged additionally as an error, as this sounds as if
> we're changing debug to error

That is also a way of doing this, but an error is an error. If we wanted 
to log the same thing in debug and error, then when we go read the debug 
we get duplicates... making it less readable. What do you say?

> I'd change the order here (and all other places) so that stderr is
> before the return code.
Ack.

> We should mention that the last string is the stderr output. Maybe we
> just add 'Stderr:' before {self._command_stderr}. And maybe we should
> put quotes around {self._command_stderr}.

Since you mentioned "quotes", I'd think that it'd be even better to 
indent it as if it's a quote. With logs as busy as the ones DTS prints, 
adding some quotes may not change much as it's all already very crowded. 
Can prefix with 'Stderr: ' though.

Re: [PATCH 4/4] dts: log stderr with failed remote commands

[Juraj Linkeš]

On Fri, Feb 23, 2024 at 8:19 PM Luca Vizzarro <Luca.Vizzarro@arm.com> wrote:
>
> On 29/01/2024 13:10, Juraj Linkeš wrote:
> > Here's I'd add logged additionally as an error, as this sounds as if
> > we're changing debug to error
>
> That is also a way of doing this, but an error is an error. If we wanted
> to log the same thing in debug and error, then when we go read the debug
> we get duplicates... making it less readable. What do you say?
>

I meant let's change the commit message wording to better reflect what
the patch does - it adds stderr to the exception, not doing something
instead of logging it as debug (it could be understood this way). But
it doesn't really matter much. Maybe a better wording of the second
sentence would be "So that, instead of logging it just as debug, it is
also stored in an error, ."

> > I'd change the order here (and all other places) so that stderr is
> > before the return code.
> Ack.
>
> > We should mention that the last string is the stderr output. Maybe we
> > just add 'Stderr:' before {self._command_stderr}. And maybe we should
> > put quotes around {self._command_stderr}.
>
> Since you mentioned "quotes", I'd think that it'd be even better to
> indent it as if it's a quote. With logs as busy as the ones DTS prints,
> adding some quotes may not change much as it's all already very crowded.
> Can prefix with 'Stderr: ' though.

The prefix is essential so that we know what the output actually is.
Indenting it sounds great.

[PATCH v2 0/3] dts: error and usage improvements

[Luca Vizzarro]

Hello!

Sending in v2 for my patch series, which changes a lot compared to v1.
The main and big change was the reworking of the arguments handling,
this can potentially be seen as a controversial change but I tried to
explain it as much as I could in the commit body message.

v2:
- complete rework of the arguments handling, to retain
  the environment variables and gain control over them
- prefixing 'Stderr: ' to RemoteCommandExecutionError
- rebased

Luca Vizzarro (3):
  dts: rework arguments framework
  dts: constrain DPDK source argument
  dts: store stderr in RemoteCommandExecutionError

 doc/guides/tools/dts.rst                      |  55 ++-
 dts/framework/exception.py                    |  13 +-
 .../remote_session/remote_session.py          |   3 +-
 dts/framework/settings.py                     | 459 +++++++++++++-----
 dts/framework/utils.py                        |  44 +-
 5 files changed, 417 insertions(+), 157 deletions(-)

-- 
2.34.1

[PATCH v2 1/3] dts: rework arguments framework

[Luca Vizzarro]

The existing argument handling in the code relies on basic argparse
functionality and a custom argparse action to integrate environment
variables. This commit improves the current handling by augmenting
argparse through a custom implementation of the arguments.

This rework implements the following improvements:
- There are duplicate expressions scattered throughout the code. To
  improve readability and maintainability, these are refactored
  into list/dict comprehensions or factory functions.
- Arguments are augmented with their own class allowing for more
  flexibility, enabling manipulation while reducing redundancy.
- Arguments are grouped within a new class. This class would
  track the origin of each argument, ensuring consistent input
  handling and facilitating debugging.
- Instead of relying solely on argument flags, error messages now
  accurately reference environment variables when applicable, enhancing
  user experience. For instance:

    error: environment variable DTS_DPDK_TARBALL: Invalid file

- Knowing the number of environment variables and arguments set
  allow for a useful help page display when none are provided.
- A display of which environment variables have been detected and their
  corresponding values in the help page, aiding user awareness.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
---
 doc/guides/tools/dts.rst  |  53 +++--
 dts/framework/settings.py | 393 ++++++++++++++++++++++++++++----------
 2 files changed, 324 insertions(+), 122 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 47b218b2c6..6993443389 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,30 +215,41 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
+   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
+                  [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
 
-   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
+   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
+   line arguments have higher priority.
 
    options:
-   -h, --help            show this help message and exit
-   --config-file CONFIG_FILE
-                         [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
-   --output-dir OUTPUT_DIR, --output OUTPUT_DIR
-                         [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
-   -t TIMEOUT, --timeout TIMEOUT
-                         [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-   -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
-   -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-   --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
-                         [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
-   --compile-timeout COMPILE_TIMEOUT
-                         [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
-   --test-suite TEST_SUITE [TEST_CASES ...]
-                         [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
-                         variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
-                         (default: [])
-   --re-run RE_RUN, --re_run RE_RUN
-                         [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
+     -h, --help            show this help message and exit
+     --config-file FILE_PATH
+                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
+                           (default: conf.yaml)
+     --output-dir DIR_PATH, --output DIR_PATH
+                           [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
+     -t SECONDS, --timeout SECONDS
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
+                           (default: 15)
+     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
+                           (default: False)
+     -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
+     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
+                           test. To test local changes, first commit them, then use the commit ID with this option.
+                           (default: dpdk.tar.xz)
+     --compile-timeout SECONDS
+                           [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
+     --test-suite TEST_SUITE [TEST_CASES ...]
+                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
+                           the test suite name, and the rest are test case names, which are optional. May be specified
+                           multiple times. To specify multiple test suites in the environment variable, join the lists
+                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
+                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
+                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
+     --re-run N_TIMES, --re_run N_TIMES
+                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
+                           (default: 0)
 
 
 The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 688e8679a7..421a9cb15b 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2021 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Environment variables and command line arguments parsing.
 
@@ -62,6 +63,7 @@
 The module provides one key module-level variable:
 
 Attributes:
+    ARGS: The module level variable storing the state of the DTS arguments.
     SETTINGS: The module level variable storing framework-wide DTS settings.
 
 Typical usage example::
@@ -72,14 +74,30 @@
 
 import argparse
 import os
+import sys
 from dataclasses import dataclass, field
 from pathlib import Path
-from typing import Any
+from typing import Any, Generator, NamedTuple
 
 from .config import TestSuiteConfig
 from .utils import DPDKGitTarball
 
 
+#: The prefix to be added to all of the environment variables.
+ENV_PREFIX = "DTS_"
+
+
+DPDK_TARBALL_PATH_ARGUMENT_NAME = "dpdk_tarball_path"
+CONFIG_FILE_ARGUMENT_NAME = "config_file"
+OUTPUT_DIR_ARGUMENT_NAME = "output_dir"
+TIMEOUT_ARGUMENT_NAME = "timeout"
+VERBOSE_ARGUMENT_NAME = "verbose"
+SKIP_SETUP_ARGUMENT_NAME = "skip_setup"
+COMPILE_TIMEOUT_ARGUMENT_NAME = "compile_timeout"
+TEST_SUITES_ARGUMENT_NAME = "test_suites"
+RERUN_ARGUMENT_NAME = "re_run"
+
+
 @dataclass(slots=True)
 class Settings:
     """Default framework-wide user settings.
@@ -110,126 +128,296 @@ class Settings:
 SETTINGS: Settings = Settings()
 
 
-def _get_parser() -> argparse.ArgumentParser:
-    """Create the argument parser for DTS.
+class ArgumentEnvPair(NamedTuple):
+    """A named tuple pairing the argument identifiers with its environment variable."""
 
-    Command line options take precedence over environment variables, which in turn take precedence
-    over default values.
+    #: The argument name.
+    arg_name: str
 
-    Returns:
-        argparse.ArgumentParser: The configured argument parser with defined options.
-    """
+    #: The argument's associated environment variable name.
+    env_var_name: str
+
+    #: The argument's associated :class:`argparse.Action` name.
+    action_name: str | None
+
+
+@dataclass
+class Argument:
+    """A class representing a DTS argument."""
+
+    #: The identifying name of the argument.
+    #: It also translates into the corresponding :class:`Settings` attribute.
+    name: str
+
+    #: A list of flags to pass to :meth:`argparse.ArgumentParser.add_argument`.
+    flags: tuple[str, ...]
+
+    #: Any other keyword arguments to pass to :meth:`argparse.ArgumentParser.add_argument`.
+    kwargs: dict[str, Any]
+
+    #: The corresponding environment variable name.
+    #: It is prefixed with the value stored in `ENV_PREFIX`.
+    #: If not specified, it is automatically generated from the :attr:`~name`.
+    env_var_name: str
+
+    _from_env: bool = False
+
+    #: A reference to its corresponding :class:`argparse.Action`.
+    _action: argparse.Action | None = None
+
+    def __init__(self, name: str, *flags: str, env_var_name: str | None = None, **kwargs: Any):
+        """Class constructor.
+
+        If the `help` argument is passed, this is prefixed with the
+        argument's corresponding environment variable in square brackets."""
+
+        self.name = name
+        self.flags = flags
+        self.kwargs = kwargs
+        self.env_var_name = self._make_env_var(env_var_name)
+
+        if "help" in self.kwargs:
+            self.kwargs["help"] = f"[{self.env_var_name}] {self.kwargs['help']}"
+
+    def add_to(self, parser: argparse._ActionsContainer):
+        """Adds this argument to an :class:`argparse.ArgumentParser` instance."""
+        self._action = parser.add_argument(*self.flags, dest=self.name, **self.kwargs)
+
+    def _make_env_var(self, env_var_name: str | None) -> str:
+        """Make the environment variable name."""
+        return f"{ENV_PREFIX}{env_var_name or self.name.upper()}"
+
+    def get_env_var(self) -> str | None:
+        """Get environment variable if it was supplied instead of a command line flag."""
+
+        env_var_value = os.environ.get(self.env_var_name)
 
-    def env_arg(env_var: str, default: Any) -> Any:
-        """A helper function augmenting the argparse with environment variables.
+        if env_var_value:
+            # check if the user has supplied any of this argument's flags in the command line
+            for flag in self.flags:
+                if flag in sys.argv:
+                    return None
 
-        If the supplied environment variable is defined, then the default value
-        of the argument is modified. This satisfies the priority order of
-        command line argument > environment variable > default value.
+        return env_var_value
 
-        Args:
-            env_var: Environment variable name.
-            default: Default value.
+    @property
+    def from_env(self) -> bool:
+        """Indicates if the argument value originated from the environment."""
+        return self._from_env
 
-        Returns:
-            Environment variable or default value.
+    def inject_env_var(self, env_value: str) -> ArgumentEnvPair:
+        """Injects the environment variable as a program argument.
+
+        Injects this argument's flag with the supplied environment variable's value and
+        returns an :class:`ArgumentEnvPair` object pairing this argument to its environment
+        variable and :class:`argparse.Action`.
+
+        The help notice of the argument is updated to display that the environment variable
+        has been correctly picked up by showing its recorded value.
+
+        .. note:: This method **must** be called after the argument has been added to the parser.
         """
-        return os.environ.get(env_var) or default
 
-    parser = argparse.ArgumentParser(
-        description="Run DPDK test suites. All options may be specified with the environment "
-        "variables provided in brackets. Command line arguments have higher priority.",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
-    )
+        assert self._action is not None
+
+        sys.argv[1:0] = [self.flags[0], env_value]
+
+        self._from_env = True
+
+        if "help" in self.kwargs:
+            self.kwargs["help"] = f"{self.kwargs['help']} (env value: {env_value})"
+        else:
+            self.kwargs["help"] = f"(env value: {env_value})"
+
+        self._action.help = self.kwargs["help"]
+
+        return ArgumentEnvPair(
+            arg_name=self.name,
+            env_var_name=self.env_var_name,
+            action_name=argparse._get_action_name(self._action),
+        )
+
+
+@dataclass
+class ArgumentGroup:
+    """A class grouping all the instances of :class:`Argument`.
+
+    This class provides indexing to access an :class:`Argument` by name:
+
+    >>> args["dpdk_revision_id"].env_var_name
+    DTS_DPDK_REVISION_ID
+
+    And can be iterated to access all the arguments:
+
+    >>> arg_env_vars = [arg.env_var_name for arg in args]
+    ['DPDK_TARBALL', ..]
+    """
+
+    #: The arguments values as parsed by :class:`argparse.ArgumentParse`.
+    values: argparse.Namespace
+
+    #: A dictionary pairing argument names to :class:`Argument` instances.
+    _args: dict[str, Argument]
+
+    #: A list of :class:`ArgumentEnvPair` containing all the successfully injected environment variables.
+    _env_vars: list[ArgumentEnvPair]
+
+    def __init__(self, *args: Argument):
+        self._args = {arg.name: arg for arg in args}
+        self._env_vars = []
+
+    def __getitem__(self, arg_name: str) -> Argument:
+        return self._args.__getitem__(arg_name)
 
-    parser.add_argument(
+    def __iter__(self) -> Generator[Argument, None, None]:
+        yield from self._args.values()
+
+    def add_environment_fed_argument(self, env_pair: ArgumentEnvPair):
+        """Add an injected environment variable."""
+        self._env_vars.append(env_pair)
+
+    @property
+    def environment_fed_arguments(self) -> list[ArgumentEnvPair]:
+        """Returns the list of all the successfully injected environment variables."""
+        return self._env_vars
+
+
+ARGS = ArgumentGroup(
+    Argument(
+        CONFIG_FILE_ARGUMENT_NAME,
         "--config-file",
-        default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
+        default=SETTINGS.config_file_path,
         type=Path,
-        help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
-        "SUTs and targets.",
-    )
-
-    parser.add_argument(
+        help="The configuration file that describes the test cases, SUTs and targets.",
+        metavar="FILE_PATH",
+        env_var_name="CFG_FILE",
+    ),
+    Argument(
+        OUTPUT_DIR_ARGUMENT_NAME,
         "--output-dir",
         "--output",
-        default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
-        help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
-    )
-
-    parser.add_argument(
+        default=SETTINGS.output_dir,
+        help="Output directory where DTS logs and results are saved.",
+        metavar="DIR_PATH",
+    ),
+    Argument(
+        TIMEOUT_ARGUMENT_NAME,
         "-t",
         "--timeout",
-        default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
+        default=SETTINGS.timeout,
         type=float,
-        help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
-    )
-
-    parser.add_argument(
+        help="The default timeout for all DTS operations except for compiling DPDK.",
+        metavar="SECONDS",
+    ),
+    Argument(
+        VERBOSE_ARGUMENT_NAME,
         "-v",
         "--verbose",
         action="store_true",
-        default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
-        help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
-        "to the console.",
-    )
-
-    parser.add_argument(
+        help="Specify to enable verbose output, logging all messages " "to the console.",
+    ),
+    Argument(
+        SKIP_SETUP_ARGUMENT_NAME,
         "-s",
         "--skip-setup",
         action="store_true",
-        default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
-        help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
-    )
-
-    parser.add_argument(
+        help="Specify to skip all setup steps on SUT and TG nodes.",
+    ),
+    Argument(
+        DPDK_TARBALL_PATH_ARGUMENT_NAME,
         "--tarball",
         "--snapshot",
         "--git-ref",
-        default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
         type=Path,
-        help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
+        default=SETTINGS.dpdk_tarball_path,
+        help="Path to DPDK source code tarball or a git commit ID,"
         "tag ID or tree ID to test. To test local changes, first commit them, "
         "then use the commit ID with this option.",
-    )
-
-    parser.add_argument(
+        metavar="FILE_PATH",
+        env_var_name="DPDK_TARBALL",
+    ),
+    Argument(
+        COMPILE_TIMEOUT_ARGUMENT_NAME,
         "--compile-timeout",
-        default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
+        default=SETTINGS.compile_timeout,
         type=float,
-        help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
-    )
-
-    parser.add_argument(
+        help="The timeout for compiling DPDK.",
+        metavar="SECONDS",
+    ),
+    Argument(
+        TEST_SUITES_ARGUMENT_NAME,
         "--test-suite",
         action="append",
         nargs="+",
-        metavar=("TEST_SUITE", "TEST_CASES"),
-        default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
-        help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
+        default=SETTINGS.test_suites,
+        help="A list containing a test suite with test cases. "
         "The first parameter is the test suite name, and the rest are test case names, "
         "which are optional. May be specified multiple times. To specify multiple test suites in "
         "the environment variable, join the lists with a comma. "
         "Examples: "
-        "--test-suite suite case case --test-suite suite case ... | "
-        "DTS_TEST_SUITES='suite case case, suite case, ...' | "
-        "--test-suite suite --test-suite suite case ... | "
-        "DTS_TEST_SUITES='suite, suite case, ...'",
-    )
-
-    parser.add_argument(
+        "--test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... | "
+        "DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | "
+        "--test-suite SUITE1 --test-suite SUITE2 CASE1 ... | "
+        "DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...'",
+        metavar=("TEST_SUITE", "TEST_CASES"),
+    ),
+    Argument(
+        RERUN_ARGUMENT_NAME,
         "--re-run",
         "--re_run",
-        default=env_arg("DTS_RERUN", SETTINGS.re_run),
+        default=SETTINGS.re_run,
         type=int,
-        help="[DTS_RERUN] Re-run each test case the specified number of times "
-        "if a test failure occurs.",
+        help="Re-run each test case the specified number of times if a test failure occurs.",
+        env_var_name="RERUN",
+        metavar="N_TIMES",
+    ),
+)
+
+
+class ArgumentParser(argparse.ArgumentParser):
+    """ArgumentParser with a custom error message.
+
+    This custom version of ArgumentParser changes the error message to
+    accurately reflect its origin if an environment variable is used
+    as an argument.
+
+    Instead of printing usage on every error, it prints instructions
+    on how to do it.
+    """
+
+    def error(self, message):
+        for _, env_var_name, action_name in ARGS.environment_fed_arguments:
+            message = message.replace(
+                f"argument {action_name}", f"environment variable {env_var_name}"
+            )
+
+        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
+        self.exit(2, "For help and usage, " "run the command with the --help flag.\n")
+
+
+def _get_parser() -> ArgumentParser:
+    """Create the argument parser for DTS.
+
+    Command line options take precedence over environment variables, which in turn take precedence
+    over default values.
+
+    Returns:
+        ArgumentParser: The configured argument parser with defined options.
+    """
+
+    parser = ArgumentParser(
+        description="Run DPDK test suites. All options may be specified with the environment "
+        "variables provided in brackets. Command line arguments have higher priority.",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+        allow_abbrev=False,
     )
+    for arg in ARGS:
+        arg.add_to(parser)
 
     return parser
 
 
-def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
+def _process_test_suites(args: list[list[str]]) -> list[TestSuiteConfig]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -240,17 +428,11 @@ def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
     Returns:
         A list of test suite configurations to execute.
     """
-    if isinstance(args, str):
-        # Environment variable in the form of "suite case case, suite case, suite, ..."
-        args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
-
-    test_suites_to_run = []
-    for suite_with_cases in args:
-        test_suites_to_run.append(
-            TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
-        )
+    if ARGS[TEST_SUITES_ARGUMENT_NAME].from_env:
+        # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
+        args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    return test_suites_to_run
+    return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
 
 
 def get_settings() -> Settings:
@@ -261,19 +443,28 @@ def get_settings() -> Settings:
     Returns:
         The new settings object.
     """
-    parsed_args = _get_parser().parse_args()
-    return Settings(
-        config_file_path=parsed_args.config_file,
-        output_dir=parsed_args.output_dir,
-        timeout=parsed_args.timeout,
-        verbose=parsed_args.verbose,
-        skip_setup=parsed_args.skip_setup,
-        dpdk_tarball_path=Path(
-            Path(DPDKGitTarball(parsed_args.tarball, parsed_args.output_dir))
-            if not os.path.exists(parsed_args.tarball)
-            else Path(parsed_args.tarball)
-        ),
-        compile_timeout=parsed_args.compile_timeout,
-        test_suites=_process_test_suites(parsed_args.test_suite),
-        re_run=parsed_args.re_run,
+
+    parser = _get_parser()
+
+    for arg in ARGS:
+        env_value = arg.get_env_var()
+        if env_value:
+            env_pair = arg.inject_env_var(env_value)
+            ARGS.add_environment_fed_argument(env_pair)
+
+    if len(sys.argv) == 1:
+        parser.print_help()
+        sys.exit(1)
+
+    ARGS.values = parser.parse_args()
+
+    ARGS.values.dpdk_tarball_path = Path(
+        Path(DPDKGitTarball(ARGS.values.dpdk_tarball_path, ARGS.values.output_dir))
+        if not os.path.exists(ARGS.values.dpdk_tarball_path)
+        else Path(ARGS.values.dpdk_tarball_path)
     )
+
+    ARGS.values.test_suites = _process_test_suites(ARGS.values.test_suites)
+
+    kwargs = {k: v for k, v in vars(ARGS.values).items() if hasattr(SETTINGS, k)}
+    return Settings(**kwargs)
-- 
2.34.1

Re: [PATCH v2 1/3] dts: rework arguments framework

[Juraj Linkeš]

On Mon, Mar 18, 2024 at 6:17 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> The existing argument handling in the code relies on basic argparse
> functionality and a custom argparse action to integrate environment
> variables. This commit improves the current handling by augmenting
> argparse through a custom implementation of the arguments.
>
> This rework implements the following improvements:
> - There are duplicate expressions scattered throughout the code. To
>   improve readability and maintainability, these are refactored
>   into list/dict comprehensions or factory functions.
> - Arguments are augmented with their own class allowing for more
>   flexibility, enabling manipulation while reducing redundancy.
> - Arguments are grouped within a new class. This class would
>   track the origin of each argument, ensuring consistent input
>   handling and facilitating debugging.
> - Instead of relying solely on argument flags, error messages now
>   accurately reference environment variables when applicable, enhancing
>   user experience. For instance:
>
>     error: environment variable DTS_DPDK_TARBALL: Invalid file
>
> - Knowing the number of environment variables and arguments set
>   allow for a useful help page display when none are provided.
> - A display of which environment variables have been detected and their
>   corresponding values in the help page, aiding user awareness.
>

Judging from the code, this patch seems like a convoluted way to implement:
1. An association between an Argument and the corresponding
environment variable,
2. A better way to add the env vars names to the help message of each
argument as well as adding the current value if set,
3. A better error message where we replace argument names with env var
names for args where env vars are used.

But maybe there's more.

In any case, isn't there a simpler way to implement the above? I
originally thought extending something in the argparse module (like
the add_argument methods in ArgumentParser and
_MutuallyExclusiveGroup), but that doesn't seem as simple as maybe
just augmenting the actions that are returned with the add_argument
methods (maybe we could subclass ArgumentParser and add some method
for this, such as add_dts_argument?), where we could just add the
env_var_name (and maybe arg_name if needed) and modify the help
message the same way we do now (modifying the self._action.help
attribute). This should also be enough for 3, but even if not, we
could store what we need in the subclass.

Also, what seems to be missing is the modification of actual values of
SETTING with the env var values (this could be done somewhere in the
add_dts_argument method).

But overall I like this approach as it gives us the knowledge of
whether an env var was used or not. I have some comments that
illustrate why I think the patch is a bit convoluted.

> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
> Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
> ---
>  doc/guides/tools/dts.rst  |  53 +++--
>  dts/framework/settings.py | 393 ++++++++++++++++++++++++++++----------
>  2 files changed, 324 insertions(+), 122 deletions(-)
>
> diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
> index 47b218b2c6..6993443389 100644
> --- a/doc/guides/tools/dts.rst
> +++ b/doc/guides/tools/dts.rst
> @@ -215,30 +215,41 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
>  .. code-block:: console
>
>     (dts-py3.10) $ ./main.py --help
> -   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
> +   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
> +                  [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
>
> -   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
> +   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
> +   line arguments have higher priority.
>
>     options:
> -   -h, --help            show this help message and exit
> -   --config-file CONFIG_FILE
> -                         [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
> -   --output-dir OUTPUT_DIR, --output OUTPUT_DIR
> -                         [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
> -   -t TIMEOUT, --timeout TIMEOUT
> -                         [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
> -   -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
> -   -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
> -   --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
> -                         [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
> -   --compile-timeout COMPILE_TIMEOUT
> -                         [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
> -   --test-suite TEST_SUITE [TEST_CASES ...]
> -                         [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
> -                         variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
> -                         (default: [])
> -   --re-run RE_RUN, --re_run RE_RUN
> -                         [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
> +     -h, --help            show this help message and exit
> +     --config-file FILE_PATH
> +                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
> +                           (default: conf.yaml)
> +     --output-dir DIR_PATH, --output DIR_PATH
> +                           [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
> +     -t SECONDS, --timeout SECONDS
> +                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
> +                           (default: 15)
> +     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
> +                           (default: False)
> +     -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
> +     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
> +                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
> +                           test. To test local changes, first commit them, then use the commit ID with this option.
> +                           (default: dpdk.tar.xz)
> +     --compile-timeout SECONDS
> +                           [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
> +     --test-suite TEST_SUITE [TEST_CASES ...]
> +                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
> +                           the test suite name, and the rest are test case names, which are optional. May be specified
> +                           multiple times. To specify multiple test suites in the environment variable, join the lists
> +                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
> +                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
> +                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
> +     --re-run N_TIMES, --re_run N_TIMES
> +                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
> +                           (default: 0)
>
>
>  The brackets contain the names of environment variables that set the same thing.
> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> index 688e8679a7..421a9cb15b 100644
> --- a/dts/framework/settings.py
> +++ b/dts/framework/settings.py
> @@ -2,6 +2,7 @@
>  # Copyright(c) 2010-2021 Intel Corporation
>  # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
>  # Copyright(c) 2022 University of New Hampshire
> +# Copyright(c) 2024 Arm Limited
>
>  """Environment variables and command line arguments parsing.
>
> @@ -62,6 +63,7 @@
>  The module provides one key module-level variable:
>
>  Attributes:
> +    ARGS: The module level variable storing the state of the DTS arguments.
>      SETTINGS: The module level variable storing framework-wide DTS settings.
>
>  Typical usage example::
> @@ -72,14 +74,30 @@
>
>  import argparse
>  import os
> +import sys
>  from dataclasses import dataclass, field
>  from pathlib import Path
> -from typing import Any
> +from typing import Any, Generator, NamedTuple
>
>  from .config import TestSuiteConfig
>  from .utils import DPDKGitTarball
>
>
> +#: The prefix to be added to all of the environment variables.
> +ENV_PREFIX = "DTS_"
> +
> +
> +DPDK_TARBALL_PATH_ARGUMENT_NAME = "dpdk_tarball_path"
> +CONFIG_FILE_ARGUMENT_NAME = "config_file"
> +OUTPUT_DIR_ARGUMENT_NAME = "output_dir"
> +TIMEOUT_ARGUMENT_NAME = "timeout"
> +VERBOSE_ARGUMENT_NAME = "verbose"
> +SKIP_SETUP_ARGUMENT_NAME = "skip_setup"
> +COMPILE_TIMEOUT_ARGUMENT_NAME = "compile_timeout"
> +TEST_SUITES_ARGUMENT_NAME = "test_suites"
> +RERUN_ARGUMENT_NAME = "re_run"
> +
> +
>  @dataclass(slots=True)
>  class Settings:
>      """Default framework-wide user settings.
> @@ -110,126 +128,296 @@ class Settings:
>  SETTINGS: Settings = Settings()
>
>
> -def _get_parser() -> argparse.ArgumentParser:
> -    """Create the argument parser for DTS.
> +class ArgumentEnvPair(NamedTuple):
> +    """A named tuple pairing the argument identifiers with its environment variable."""
>
> -    Command line options take precedence over environment variables, which in turn take precedence
> -    over default values.
> +    #: The argument name.
> +    arg_name: str
>
> -    Returns:
> -        argparse.ArgumentParser: The configured argument parser with defined options.
> -    """
> +    #: The argument's associated environment variable name.
> +    env_var_name: str
> +
> +    #: The argument's associated :class:`argparse.Action` name.
> +    action_name: str | None
> +
> +
> +@dataclass
> +class Argument:
> +    """A class representing a DTS argument."""
> +
> +    #: The identifying name of the argument.
> +    #: It also translates into the corresponding :class:`Settings` attribute.
> +    name: str
> +
> +    #: A list of flags to pass to :meth:`argparse.ArgumentParser.add_argument`.
> +    flags: tuple[str, ...]
> +
> +    #: Any other keyword arguments to pass to :meth:`argparse.ArgumentParser.add_argument`.
> +    kwargs: dict[str, Any]
> +
> +    #: The corresponding environment variable name.
> +    #: It is prefixed with the value stored in `ENV_PREFIX`.
> +    #: If not specified, it is automatically generated from the :attr:`~name`.
> +    env_var_name: str
> +
> +    _from_env: bool = False
> +
> +    #: A reference to its corresponding :class:`argparse.Action`.
> +    _action: argparse.Action | None = None
> +
> +    def __init__(self, name: str, *flags: str, env_var_name: str | None = None, **kwargs: Any):
> +        """Class constructor.
> +
> +        If the `help` argument is passed, this is prefixed with the
> +        argument's corresponding environment variable in square brackets."""
> +
> +        self.name = name
> +        self.flags = flags
> +        self.kwargs = kwargs
> +        self.env_var_name = self._make_env_var(env_var_name)
> +
> +        if "help" in self.kwargs:
> +            self.kwargs["help"] = f"[{self.env_var_name}] {self.kwargs['help']}"
> +
> +    def add_to(self, parser: argparse._ActionsContainer):
> +        """Adds this argument to an :class:`argparse.ArgumentParser` instance."""
> +        self._action = parser.add_argument(*self.flags, dest=self.name, **self.kwargs)
> +
> +    def _make_env_var(self, env_var_name: str | None) -> str:
> +        """Make the environment variable name."""
> +        return f"{ENV_PREFIX}{env_var_name or self.name.upper()}"
> +
> +    def get_env_var(self) -> str | None:
> +        """Get environment variable if it was supplied instead of a command line flag."""
> +
> +        env_var_value = os.environ.get(self.env_var_name)
>
> -    def env_arg(env_var: str, default: Any) -> Any:
> -        """A helper function augmenting the argparse with environment variables.
> +        if env_var_value:
> +            # check if the user has supplied any of this argument's flags in the command line
> +            for flag in self.flags:
> +                if flag in sys.argv:
> +                    return None
>
> -        If the supplied environment variable is defined, then the default value
> -        of the argument is modified. This satisfies the priority order of
> -        command line argument > environment variable > default value.
> +        return env_var_value
>
> -        Args:
> -            env_var: Environment variable name.
> -            default: Default value.
> +    @property
> +    def from_env(self) -> bool:
> +        """Indicates if the argument value originated from the environment."""
> +        return self._from_env
>
> -        Returns:
> -            Environment variable or default value.
> +    def inject_env_var(self, env_value: str) -> ArgumentEnvPair:
> +        """Injects the environment variable as a program argument.
> +
> +        Injects this argument's flag with the supplied environment variable's value and
> +        returns an :class:`ArgumentEnvPair` object pairing this argument to its environment
> +        variable and :class:`argparse.Action`.
> +
> +        The help notice of the argument is updated to display that the environment variable
> +        has been correctly picked up by showing its recorded value.
> +
> +        .. note:: This method **must** be called after the argument has been added to the parser.
>          """
> -        return os.environ.get(env_var) or default
>
> -    parser = argparse.ArgumentParser(
> -        description="Run DPDK test suites. All options may be specified with the environment "
> -        "variables provided in brackets. Command line arguments have higher priority.",
> -        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
> -    )
> +        assert self._action is not None
> +
> +        sys.argv[1:0] = [self.flags[0], env_value]
> +
> +        self._from_env = True
> +
> +        if "help" in self.kwargs:
> +            self.kwargs["help"] = f"{self.kwargs['help']} (env value: {env_value})"
> +        else:
> +            self.kwargs["help"] = f"(env value: {env_value})"
> +
> +        self._action.help = self.kwargs["help"]
> +
> +        return ArgumentEnvPair(
> +            arg_name=self.name,
> +            env_var_name=self.env_var_name,
> +            action_name=argparse._get_action_name(self._action),
> +        )
> +
> +
> +@dataclass
> +class ArgumentGroup:
> +    """A class grouping all the instances of :class:`Argument`.
> +
> +    This class provides indexing to access an :class:`Argument` by name:
> +
> +    >>> args["dpdk_revision_id"].env_var_name
> +    DTS_DPDK_REVISION_ID
> +
> +    And can be iterated to access all the arguments:
> +
> +    >>> arg_env_vars = [arg.env_var_name for arg in args]
> +    ['DPDK_TARBALL', ..]
> +    """
> +
> +    #: The arguments values as parsed by :class:`argparse.ArgumentParse`.
> +    values: argparse.Namespace
> +
> +    #: A dictionary pairing argument names to :class:`Argument` instances.
> +    _args: dict[str, Argument]
> +
> +    #: A list of :class:`ArgumentEnvPair` containing all the successfully injected environment variables.
> +    _env_vars: list[ArgumentEnvPair]
> +
> +    def __init__(self, *args: Argument):
> +        self._args = {arg.name: arg for arg in args}
> +        self._env_vars = []
> +
> +    def __getitem__(self, arg_name: str) -> Argument:
> +        return self._args.__getitem__(arg_name)

Looking at this, this class could've been just a subclassed dict. We
could set the attributes with setattr in __init__(). But at that
point, it looks to be the same as the namespace returned by
parser.parse_args(), apart from the environment_fed_arguments property
(more below), which we could do without.

>
> -    parser.add_argument(
> +    def __iter__(self) -> Generator[Argument, None, None]:
> +        yield from self._args.values()
> +
> +    def add_environment_fed_argument(self, env_pair: ArgumentEnvPair):
> +        """Add an injected environment variable."""
> +        self._env_vars.append(env_pair)
> +

We're already storing the arguments in the class, so we could just add
whatever is in ArgumentEnvPair to the argument and we have the
correspondence (looking at the Argument class again, we already have
that). The pair class seems redundant.

> +    @property
> +    def environment_fed_arguments(self) -> list[ArgumentEnvPair]:
> +        """Returns the list of all the successfully injected environment variables."""
> +        return self._env_vars
> +

And then we could get all of this from the stored arguments. Could be
just a tuple of (var_name, arg_name) of args with from_env == True.

> +
> +ARGS = ArgumentGroup(
> +    Argument(
> +        CONFIG_FILE_ARGUMENT_NAME,
>          "--config-file",
> -        default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
> +        default=SETTINGS.config_file_path,
>          type=Path,
> -        help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
> -        "SUTs and targets.",
> -    )
> -
> -    parser.add_argument(
> +        help="The configuration file that describes the test cases, SUTs and targets.",
> +        metavar="FILE_PATH",
> +        env_var_name="CFG_FILE",
> +    ),
> +    Argument(
> +        OUTPUT_DIR_ARGUMENT_NAME,
>          "--output-dir",
>          "--output",
> -        default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
> -        help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
> -    )
> -
> -    parser.add_argument(
> +        default=SETTINGS.output_dir,
> +        help="Output directory where DTS logs and results are saved.",
> +        metavar="DIR_PATH",
> +    ),
> +    Argument(
> +        TIMEOUT_ARGUMENT_NAME,
>          "-t",
>          "--timeout",
> -        default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
> +        default=SETTINGS.timeout,
>          type=float,
> -        help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
> -    )
> -
> -    parser.add_argument(
> +        help="The default timeout for all DTS operations except for compiling DPDK.",
> +        metavar="SECONDS",
> +    ),
> +    Argument(
> +        VERBOSE_ARGUMENT_NAME,
>          "-v",
>          "--verbose",
>          action="store_true",
> -        default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
> -        help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
> -        "to the console.",
> -    )
> -
> -    parser.add_argument(
> +        help="Specify to enable verbose output, logging all messages " "to the console.",
> +    ),
> +    Argument(
> +        SKIP_SETUP_ARGUMENT_NAME,
>          "-s",
>          "--skip-setup",
>          action="store_true",
> -        default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
> -        help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
> -    )
> -
> -    parser.add_argument(
> +        help="Specify to skip all setup steps on SUT and TG nodes.",
> +    ),
> +    Argument(
> +        DPDK_TARBALL_PATH_ARGUMENT_NAME,
>          "--tarball",
>          "--snapshot",
>          "--git-ref",
> -        default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
>          type=Path,
> -        help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
> +        default=SETTINGS.dpdk_tarball_path,
> +        help="Path to DPDK source code tarball or a git commit ID,"
>          "tag ID or tree ID to test. To test local changes, first commit them, "
>          "then use the commit ID with this option.",
> -    )
> -
> -    parser.add_argument(
> +        metavar="FILE_PATH",
> +        env_var_name="DPDK_TARBALL",
> +    ),
> +    Argument(
> +        COMPILE_TIMEOUT_ARGUMENT_NAME,
>          "--compile-timeout",
> -        default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
> +        default=SETTINGS.compile_timeout,
>          type=float,
> -        help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
> -    )
> -
> -    parser.add_argument(
> +        help="The timeout for compiling DPDK.",
> +        metavar="SECONDS",
> +    ),
> +    Argument(
> +        TEST_SUITES_ARGUMENT_NAME,
>          "--test-suite",
>          action="append",
>          nargs="+",
> -        metavar=("TEST_SUITE", "TEST_CASES"),
> -        default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
> -        help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
> +        default=SETTINGS.test_suites,
> +        help="A list containing a test suite with test cases. "
>          "The first parameter is the test suite name, and the rest are test case names, "
>          "which are optional. May be specified multiple times. To specify multiple test suites in "
>          "the environment variable, join the lists with a comma. "
>          "Examples: "
> -        "--test-suite suite case case --test-suite suite case ... | "
> -        "DTS_TEST_SUITES='suite case case, suite case, ...' | "
> -        "--test-suite suite --test-suite suite case ... | "
> -        "DTS_TEST_SUITES='suite, suite case, ...'",
> -    )
> -
> -    parser.add_argument(
> +        "--test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... | "
> +        "DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | "
> +        "--test-suite SUITE1 --test-suite SUITE2 CASE1 ... | "
> +        "DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...'",
> +        metavar=("TEST_SUITE", "TEST_CASES"),
> +    ),
> +    Argument(
> +        RERUN_ARGUMENT_NAME,
>          "--re-run",
>          "--re_run",
> -        default=env_arg("DTS_RERUN", SETTINGS.re_run),
> +        default=SETTINGS.re_run,
>          type=int,
> -        help="[DTS_RERUN] Re-run each test case the specified number of times "
> -        "if a test failure occurs.",
> +        help="Re-run each test case the specified number of times if a test failure occurs.",
> +        env_var_name="RERUN",
> +        metavar="N_TIMES",
> +    ),
> +)
> +
> +
> +class ArgumentParser(argparse.ArgumentParser):
> +    """ArgumentParser with a custom error message.
> +
> +    This custom version of ArgumentParser changes the error message to
> +    accurately reflect its origin if an environment variable is used
> +    as an argument.
> +
> +    Instead of printing usage on every error, it prints instructions
> +    on how to do it.
> +    """
> +
> +    def error(self, message):
> +        for _, env_var_name, action_name in ARGS.environment_fed_arguments:
> +            message = message.replace(
> +                f"argument {action_name}", f"environment variable {env_var_name}"
> +            )

I think this should also contain the env var value to be consistent
with the help message.

> +
> +        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
> +        self.exit(2, "For help and usage, " "run the command with the --help flag.\n")
> +
> +
> +def _get_parser() -> ArgumentParser:
> +    """Create the argument parser for DTS.
> +
> +    Command line options take precedence over environment variables, which in turn take precedence
> +    over default values.
> +
> +    Returns:
> +        ArgumentParser: The configured argument parser with defined options.
> +    """
> +
> +    parser = ArgumentParser(
> +        description="Run DPDK test suites. All options may be specified with the environment "
> +        "variables provided in brackets. Command line arguments have higher priority.",
> +        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
> +        allow_abbrev=False,
>      )
> +    for arg in ARGS:
> +        arg.add_to(parser)
>
>      return parser
>
>
> -def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
> +def _process_test_suites(args: list[list[str]]) -> list[TestSuiteConfig]:
>      """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
>
>      Args:
> @@ -240,17 +428,11 @@ def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
>      Returns:
>          A list of test suite configurations to execute.
>      """
> -    if isinstance(args, str):
> -        # Environment variable in the form of "suite case case, suite case, suite, ..."
> -        args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
> -
> -    test_suites_to_run = []
> -    for suite_with_cases in args:
> -        test_suites_to_run.append(
> -            TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
> -        )
> +    if ARGS[TEST_SUITES_ARGUMENT_NAME].from_env:
> +        # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
> +        args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
>
> -    return test_suites_to_run
> +    return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
>
>
>  def get_settings() -> Settings:
> @@ -261,19 +443,28 @@ def get_settings() -> Settings:
>      Returns:
>          The new settings object.
>      """
> -    parsed_args = _get_parser().parse_args()
> -    return Settings(
> -        config_file_path=parsed_args.config_file,
> -        output_dir=parsed_args.output_dir,
> -        timeout=parsed_args.timeout,
> -        verbose=parsed_args.verbose,
> -        skip_setup=parsed_args.skip_setup,
> -        dpdk_tarball_path=Path(
> -            Path(DPDKGitTarball(parsed_args.tarball, parsed_args.output_dir))
> -            if not os.path.exists(parsed_args.tarball)
> -            else Path(parsed_args.tarball)
> -        ),
> -        compile_timeout=parsed_args.compile_timeout,
> -        test_suites=_process_test_suites(parsed_args.test_suite),
> -        re_run=parsed_args.re_run,
> +
> +    parser = _get_parser()
> +
> +    for arg in ARGS:
> +        env_value = arg.get_env_var()
> +        if env_value:
> +            env_pair = arg.inject_env_var(env_value)
> +            ARGS.add_environment_fed_argument(env_pair)

We're going through all of the args here so we could just do this when
creating the argument. I guess we'd need to modify the top-level error
message afterwards with the current class structure, but I'm sure even
that could be done when creating the argument with some refactoring.

> +
> +    if len(sys.argv) == 1:
> +        parser.print_help()
> +        sys.exit(1)
> +
> +    ARGS.values = parser.parse_args()
> +

The values attribute seems superfluous as we're only using it locally.

> +    ARGS.values.dpdk_tarball_path = Path(
> +        Path(DPDKGitTarball(ARGS.values.dpdk_tarball_path, ARGS.values.output_dir))
> +        if not os.path.exists(ARGS.values.dpdk_tarball_path)
> +        else Path(ARGS.values.dpdk_tarball_path)
>      )
> +
> +    ARGS.values.test_suites = _process_test_suites(ARGS.values.test_suites)
> +
> +    kwargs = {k: v for k, v in vars(ARGS.values).items() if hasattr(SETTINGS, k)}

This is here so that we don't use both arguments from the mutual
group, right? We could add a short comment explaining this.
Also, I think we don't need to use vars() here, the result should be the same.

> +    return Settings(**kwargs)
> --
> 2.34.1
>

Re: [PATCH v2 1/3] dts: rework arguments framework

[Luca Vizzarro]

On 04/04/2024 10:25, Juraj Linkeš wrote:
> Judging from the code, this patch seems like a convoluted way to implement:
> 1. An association between an Argument and the corresponding
> environment variable,
> 2. A better way to add the env vars names to the help message of each
> argument as well as adding the current value if set,
> 3. A better error message where we replace argument names with env var
> names for args where env vars are used.
> 
> But maybe there's more.
> 
> In any case, isn't there a simpler way to implement the above? I
> originally thought extending something in the argparse module (like
> the add_argument methods in ArgumentParser and
> _MutuallyExclusiveGroup), but that doesn't seem as simple as maybe
> just augmenting the actions that are returned with the add_argument
> methods (maybe we could subclass ArgumentParser and add some method
> for this, such as add_dts_argument?), where we could just add the
> env_var_name (and maybe arg_name if needed) and modify the help
> message the same way we do now (modifying the self._action.help
> attribute). This should also be enough for 3, but even if not, we
> could store what we need in the subclass.

I agree that the solutions appears somewhat convoluted, and I am indeed 
open to ideas on how to simplify the approach. I initially considered 
extending the argparse module, but as you said it's no particularly 
simple, and it wasn't written to be particularly extendable either. If
we want to go down this route, it would be somewhat hacky. I am not 
against it though. But, yeah, in retrospective some things can be easily 
integrated.

> Also, what seems to be missing is the modification of actual values of
> SETTING with the env var values (this could be done somewhere in the
> add_dts_argument method).

I don't think I am following what you mean by this. If you refer to 
updating the values of `SETTING` with the environment ones, this is done 
using `inject_env_variable` before the arguments are parsed. In a few 
words, that method just injects the environment arguments in sys.argv. 
Therefore to the ArgumentParser it just looks like they were supplied on 
the command line.

> But overall I like this approach as it gives us the knowledge of
> whether an env var was used or not. I have some comments that
> illustrate why I think the patch is a bit convoluted.
> 
<snip>
> 
> Looking at this, this class could've been just a subclassed dict. We
> could set the attributes with setattr in __init__(). But at that
> point, it looks to be the same as the namespace returned by
> parser.parse_args(), apart from the environment_fed_arguments property
> (more below), which we could do without.
> 

Yes, definitely. The main reason to store the namespaced values was in 
case this could turn out to be useful when debugging in the future. But 
if we think it's not worthwhile, it can reduce complexity.

> 
> We're already storing the arguments in the class, so we could just add
> whatever is in ArgumentEnvPair to the argument and we have the
> correspondence (looking at the Argument class again, we already have
> that). The pair class seems redundant.
>
> 
> And then we could get all of this from the stored arguments. Could be
> just a tuple of (var_name, arg_name) of args with from_env == True.
> 

And storing a pre-made list of environment-fed arguments. Yes, we could 
definitely walk through every argument as needed. Given the context and 
usage of this, I guess yeah, you are right in saying it's redundant. 
Happy to remove it.

> 
> I think this should also contain the env var value to be consistent
> with the help message.
> 

Ack.

> 
> We're going through all of the args here so we could just do this when
> creating the argument. I guess we'd need to modify the top-level error
> message afterwards with the current class structure, but I'm sure even
> that could be done when creating the argument with some refactoring.
> 

Yeah. I decided to do this separately to avoid duplicating the code for 
the mutual exclusion group (as per the next commit).

> 
> The values attribute seems superfluous as we're only using it locally.
> 

Ack.

> 
> This is here so that we don't use both arguments from the mutual
> group, right? We could add a short comment explaining this.
> Also, I think we don't need to use vars() here, the result should be the same.
> 

Yes, and any other argument we may want to add in the future that don't 
belong in SETTINGS. It's only selecting the arguments that already exist 
in SETTING and write values for that. Can add a comment.

Namespace doesn't appear to implement iteration. As per the doc page[1], 
the suggested way is to use vars to extract a dict, which we can iterate 
over.

[1] https://docs.python.org/3/library/argparse.html#argparse.Namespace

Re: [PATCH v2 1/3] dts: rework arguments framework

[Juraj Linkeš]

On Tue, Apr 9, 2024 at 5:14 PM Luca Vizzarro <Luca.Vizzarro@arm.com> wrote:
>
> On 04/04/2024 10:25, Juraj Linkeš wrote:
> > Judging from the code, this patch seems like a convoluted way to implement:
> > 1. An association between an Argument and the corresponding
> > environment variable,
> > 2. A better way to add the env vars names to the help message of each
> > argument as well as adding the current value if set,
> > 3. A better error message where we replace argument names with env var
> > names for args where env vars are used.
> >
> > But maybe there's more.
> >
> > In any case, isn't there a simpler way to implement the above? I
> > originally thought extending something in the argparse module (like
> > the add_argument methods in ArgumentParser and
> > _MutuallyExclusiveGroup), but that doesn't seem as simple as maybe
> > just augmenting the actions that are returned with the add_argument
> > methods (maybe we could subclass ArgumentParser and add some method
> > for this, such as add_dts_argument?), where we could just add the
> > env_var_name (and maybe arg_name if needed) and modify the help
> > message the same way we do now (modifying the self._action.help
> > attribute). This should also be enough for 3, but even if not, we
> > could store what we need in the subclass.
>
> I agree that the solutions appears somewhat convoluted, and I am indeed
> open to ideas on how to simplify the approach. I initially considered
> extending the argparse module, but as you said it's no particularly
> simple, and it wasn't written to be particularly extendable either. If
> we want to go down this route, it would be somewhat hacky. I am not
> against it though. But, yeah, in retrospective some things can be easily
> integrated.
>

Great, let's get a v2 and see where we end up.

> > Also, what seems to be missing is the modification of actual values of
> > SETTING with the env var values (this could be done somewhere in the
> > add_dts_argument method).
>
> I don't think I am following what you mean by this. If you refer to
> updating the values of `SETTING` with the environment ones, this is done
> using `inject_env_variable` before the arguments are parsed. In a few
> words, that method just injects the environment arguments in sys.argv.
> Therefore to the ArgumentParser it just looks like they were supplied on
> the command line.
>

My bad, I must've made a mistake when verifying this. It is working fine.

> > But overall I like this approach as it gives us the knowledge of
> > whether an env var was used or not. I have some comments that
> > illustrate why I think the patch is a bit convoluted.
> >
> <snip>
> >
> > Looking at this, this class could've been just a subclassed dict. We
> > could set the attributes with setattr in __init__(). But at that
> > point, it looks to be the same as the namespace returned by
> > parser.parse_args(), apart from the environment_fed_arguments property
> > (more below), which we could do without.
> >
>
> Yes, definitely. The main reason to store the namespaced values was in
> case this could turn out to be useful when debugging in the future. But
> if we think it's not worthwhile, it can reduce complexity.
>
> >
> > We're already storing the arguments in the class, so we could just add
> > whatever is in ArgumentEnvPair to the argument and we have the
> > correspondence (looking at the Argument class again, we already have
> > that). The pair class seems redundant.
> >
> >
> > And then we could get all of this from the stored arguments. Could be
> > just a tuple of (var_name, arg_name) of args with from_env == True.
> >
>
> And storing a pre-made list of environment-fed arguments. Yes, we could
> definitely walk through every argument as needed. Given the context and
> usage of this, I guess yeah, you are right in saying it's redundant.
> Happy to remove it.
>
> >
> > I think this should also contain the env var value to be consistent
> > with the help message.
> >
>
> Ack.
>
> >
> > We're going through all of the args here so we could just do this when
> > creating the argument. I guess we'd need to modify the top-level error
> > message afterwards with the current class structure, but I'm sure even
> > that could be done when creating the argument with some refactoring.
> >
>
> Yeah. I decided to do this separately to avoid duplicating the code for
> the mutual exclusion group (as per the next commit).
>
> >
> > The values attribute seems superfluous as we're only using it locally.
> >
>
> Ack.
>
> >
> > This is here so that we don't use both arguments from the mutual
> > group, right? We could add a short comment explaining this.
> > Also, I think we don't need to use vars() here, the result should be the same.
> >
>
> Yes, and any other argument we may want to add in the future that don't
> belong in SETTINGS. It's only selecting the arguments that already exist
> in SETTING and write values for that. Can add a comment.
>
> Namespace doesn't appear to implement iteration. As per the doc page[1],
> the suggested way is to use vars to extract a dict, which we can iterate
> over.
>
> [1] https://docs.python.org/3/library/argparse.html#argparse.Namespace

Ah, I get it now. Thanks for pointing this out.

[PATCH v2 2/3] dts: constrain DPDK source argument

[Luca Vizzarro]

DTS needs an input to gather the DPDK source code from. This is then
built on the remote target. This commit makes sure that this input is
more constrained, separating the Git revision ID – used to create a
tarball using Git – and providing tarballed source code directly, while
retaining mutual exclusion.

This makes the code more readable and easier to handle for input
validation, of which this commit introduces a basic one based on the
pre-existing code.

Moreover it ensures that these flags are explicitly required to be set
by the user, dropping a default value.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
---
 doc/guides/tools/dts.rst  | 14 +++---
 dts/framework/settings.py | 90 ++++++++++++++++++++++++++++-----------
 dts/framework/utils.py    | 43 +++++++++++--------
 3 files changed, 98 insertions(+), 49 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 6993443389..ea2bb34ddc 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,14 +215,20 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
-                  [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
+   usage: main.py [-h] (--tarball FILE_PATH | --revision ID) [--config-file FILE_PATH] [--output-dir DIR_PATH]
+                  [-t SECONDS] [-v] [-s] [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]]
+                  [--re-run N_TIMES]
 
    Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
    line arguments have higher priority.
 
    options:
      -h, --help            show this help message and exit
+     --tarball FILE_PATH, --snapshot FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball to test. (default: None)
+     --revision ID, --rev ID, --git-ref ID
+                           [DTS_DPDK_REVISION_ID] Git revision ID to test. Could be commit, tag, tree ID etc. To test
+                           local changes, first commit them, then use their commit ID. (default: None)
      --config-file FILE_PATH
                            [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
                            (default: conf.yaml)
@@ -234,10 +240,6 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
      -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
                            (default: False)
      -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
-                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
-                           test. To test local changes, first commit them, then use the commit ID with this option.
-                           (default: dpdk.tar.xz)
      --compile-timeout SECONDS
                            [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
      --test-suite TEST_SUITE [TEST_CASES ...]
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 421a9cb15b..ec238cea33 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -14,6 +14,17 @@
 
 The command line arguments along with the supported environment variables are:
 
+.. option:: --tarball, --snapshot
+.. envvar:: DTS_DPDK_TARBALL
+
+    Path to DPDK source code tarball to test.
+
+.. option:: --revision, --rev, --git-ref
+.. envvar:: DTS_DPDK_REVISION_ID
+
+    Git revision ID to test. Could be commit, tag, tree ID etc.
+    To test local changes, first commit them, then use their commit ID.
+
 .. option:: --config-file
 .. envvar:: DTS_CFG_FILE
 
@@ -44,11 +55,6 @@
 
     Set to any value to skip building DPDK.
 
-.. option:: --tarball, --snapshot, --git-ref
-.. envvar:: DTS_DPDK_TARBALL
-
-    The path to a DPDK tarball, git commit ID, tag ID or tree ID to test.
-
 .. option:: --test-suite
 .. envvar:: DTS_TEST_SUITES
 
@@ -79,8 +85,9 @@
 from pathlib import Path
 from typing import Any, Generator, NamedTuple
 
+from .exception import ConfigurationError
 from .config import TestSuiteConfig
-from .utils import DPDKGitTarball
+from .utils import DPDKGitTarball, get_commit_id
 
 
 #: The prefix to be added to all of the environment variables.
@@ -88,6 +95,7 @@
 
 
 DPDK_TARBALL_PATH_ARGUMENT_NAME = "dpdk_tarball_path"
+DPDK_REVISION_ID_ARGUMENT_NAME = "dpdk_revision_id"
 CONFIG_FILE_ARGUMENT_NAME = "config_file"
 OUTPUT_DIR_ARGUMENT_NAME = "output_dir"
 TIMEOUT_ARGUMENT_NAME = "timeout"
@@ -98,6 +106,24 @@
 RERUN_ARGUMENT_NAME = "re_run"
 
 
+def _parse_tarball_path(file_path: str) -> Path:
+    """Validate whether `file_path` is valid and return a Path object."""
+
+    path = Path(file_path)
+    if not path.exists() or not path.is_file():
+        raise argparse.ArgumentTypeError("The file path provided is not a valid file")
+    return path
+
+
+def _parse_revision_id(rev_id: str) -> str:
+    """Validate revision ID and retrieve corresponding commit ID."""
+
+    try:
+        return get_commit_id(rev_id)
+    except ConfigurationError:
+        raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")
+
+
 @dataclass(slots=True)
 class Settings:
     """Default framework-wide user settings.
@@ -116,7 +142,7 @@ class Settings:
     #:
     skip_setup: bool = False
     #:
-    dpdk_tarball_path: Path | str = "dpdk.tar.xz"
+    dpdk_tarball_path: Path | str = ""
     #:
     compile_timeout: float = 1200
     #:
@@ -283,6 +309,25 @@ def environment_fed_arguments(self) -> list[ArgumentEnvPair]:
 
 
 ARGS = ArgumentGroup(
+    Argument(
+        DPDK_TARBALL_PATH_ARGUMENT_NAME,
+        "--tarball",
+        "--snapshot",
+        type=_parse_tarball_path,
+        help="Path to DPDK source code tarball to test.",
+        metavar="FILE_PATH",
+        env_var_name="DPDK_TARBALL",
+    ),
+    Argument(
+        DPDK_REVISION_ID_ARGUMENT_NAME,
+        "--revision",
+        "--rev",
+        "--git-ref",
+        type=_parse_revision_id,
+        help="Git revision ID to test. Could be commit, tag, tree ID etc. "
+        "To test local changes, first commit them, then use their commit ID.",
+        metavar="ID",
+    ),
     Argument(
         CONFIG_FILE_ARGUMENT_NAME,
         "--config-file",
@@ -323,19 +368,6 @@ def environment_fed_arguments(self) -> list[ArgumentEnvPair]:
         action="store_true",
         help="Specify to skip all setup steps on SUT and TG nodes.",
     ),
-    Argument(
-        DPDK_TARBALL_PATH_ARGUMENT_NAME,
-        "--tarball",
-        "--snapshot",
-        "--git-ref",
-        type=Path,
-        default=SETTINGS.dpdk_tarball_path,
-        help="Path to DPDK source code tarball or a git commit ID,"
-        "tag ID or tree ID to test. To test local changes, first commit them, "
-        "then use the commit ID with this option.",
-        metavar="FILE_PATH",
-        env_var_name="DPDK_TARBALL",
-    ),
     Argument(
         COMPILE_TIMEOUT_ARGUMENT_NAME,
         "--compile-timeout",
@@ -411,7 +443,14 @@ def _get_parser() -> ArgumentParser:
         formatter_class=argparse.ArgumentDefaultsHelpFormatter,
         allow_abbrev=False,
     )
-    for arg in ARGS:
+
+    dpdk_source = parser.add_mutually_exclusive_group(required=True)
+    dpdk_source_arg_group = [DPDK_TARBALL_PATH_ARGUMENT_NAME, DPDK_REVISION_ID_ARGUMENT_NAME]
+    for arg_name in dpdk_source_arg_group:
+        ARGS[arg_name].add_to(dpdk_source)
+
+    arg_group = [arg for arg in ARGS if arg.name not in dpdk_source_arg_group]
+    for arg in arg_group:
         arg.add_to(parser)
 
     return parser
@@ -458,11 +497,10 @@ def get_settings() -> Settings:
 
     ARGS.values = parser.parse_args()
 
-    ARGS.values.dpdk_tarball_path = Path(
-        Path(DPDKGitTarball(ARGS.values.dpdk_tarball_path, ARGS.values.output_dir))
-        if not os.path.exists(ARGS.values.dpdk_tarball_path)
-        else Path(ARGS.values.dpdk_tarball_path)
-    )
+    if ARGS.values.dpdk_revision_id:
+        ARGS.values.dpdk_tarball_path = DPDKGitTarball(
+            ARGS.values.dpdk_revision_id, ARGS.values.output_dir
+        )
 
     ARGS.values.test_suites = _process_test_suites(ARGS.values.test_suites)
 
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index cc5e458cc8..3976f92335 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Various utility classes and functions.
 
@@ -70,6 +71,31 @@ def get_packet_summaries(packets: list[Packet]) -> str:
     return f"Packet contents: \n{packet_summaries}"
 
 
+def get_commit_id(rev_id: str) -> str:
+    """Given a Git revision ID, return the corresponding commit ID.
+
+    Args:
+        rev_id: The Git revision ID.
+
+    Raises:
+        ConfigurationError: The ``git rev-parse`` command failed, suggesting
+            an invalid or ambiguous revision ID was supplied.
+    """
+    result = subprocess.run(
+        ["git", "rev-parse", "--verify", rev_id],
+        text=True,
+        capture_output=True,
+    )
+    if result.returncode != 0:
+        raise ConfigurationError(
+            f"{rev_id} is not a valid git reference.\n"
+            f"Command: {result.args}\n"
+            f"Stdout: {result.stdout}\n"
+            f"Stderr: {result.stderr}"
+        )
+    return result.stdout.strip()
+
+
 class StrEnum(Enum):
     """Enum with members stored as strings."""
 
@@ -170,7 +196,6 @@ def __init__(
 
         self._tarball_dir = Path(output_dir, "tarball")
 
-        self._get_commit_id()
         self._create_tarball_dir()
 
         self._tarball_name = (
@@ -180,22 +205,6 @@ def __init__(
         if not self._tarball_path:
             self._create_tarball()
 
-    def _get_commit_id(self) -> None:
-        result = subprocess.run(
-            ["git", "rev-parse", "--verify", self._git_ref],
-            text=True,
-            capture_output=True,
-        )
-        if result.returncode != 0:
-            raise ConfigurationError(
-                f"{self._git_ref} is neither a path to an existing DPDK "
-                "archive nor a valid git reference.\n"
-                f"Command: {result.args}\n"
-                f"Stdout: {result.stdout}\n"
-                f"Stderr: {result.stderr}"
-            )
-        self._git_ref = result.stdout.strip()
-
     def _create_tarball_dir(self) -> None:
         os.makedirs(self._tarball_dir, exist_ok=True)
 
-- 
2.34.1

[PATCH v2 3/3] dts: store stderr in RemoteCommandExecutionError

[Luca Vizzarro]

Store the stderr of an executed command in RemoteCommandExecutionError.
Consequently, when the exception is logged the error message includes
the stderr.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
---
 dts/framework/exception.py                     | 13 ++++++++++---
 dts/framework/remote_session/remote_session.py |  3 ++-
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/dts/framework/exception.py b/dts/framework/exception.py
index cce1e0231a..50724acdf2 100644
--- a/dts/framework/exception.py
+++ b/dts/framework/exception.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """DTS exceptions.
 
@@ -129,21 +130,27 @@ class RemoteCommandExecutionError(DTSError):
     severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
     #: The executed command.
     command: str
+    _command_stderr: str
     _command_return_code: int
 
-    def __init__(self, command: str, command_return_code: int):
+    def __init__(self, command: str, command_return_code: int, command_stderr: str):
         """Define the meaning of the first two arguments.
 
         Args:
             command: The executed command.
             command_return_code: The return code of the executed command.
+            command_stderr: The stderr of the executed command.
         """
         self.command = command
         self._command_return_code = command_return_code
+        self._command_stderr = command_stderr
 
     def __str__(self) -> str:
-        """Include both the command and return code in the string representation."""
-        return f"Command {self.command} returned a non-zero exit code: {self._command_return_code}"
+        """Include the command, its return code and stderr in the string representation."""
+        return (
+            f"Command '{self.command}' returned a non-zero exit code: "
+            f"{self._command_return_code}\nStderr: {self._command_stderr}"
+        )
 
 
 class InteractiveCommandExecutionError(DTSError):
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
index ad0f53720a..9aaa8c8a04 100644
--- a/dts/framework/remote_session/remote_session.py
+++ b/dts/framework/remote_session/remote_session.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Base remote session.
 
@@ -172,7 +173,7 @@ def send_command(
             )
             self._logger.debug(f"stdout: '{result.stdout}'")
             self._logger.debug(f"stderr: '{result.stderr}'")
-            raise RemoteCommandExecutionError(command, result.return_code)
+            raise RemoteCommandExecutionError(command, result.return_code, result.stderr)
         self._logger.debug(f"Received from '{command}':\n{result}")
         self.history.append(result)
         return result
-- 
2.34.1

[PATCH v3 0/3] error and usage improvements

[Luca Vizzarro]

v3:
- amended arguments rework so that it retains the original functional
  style
- re-implemetend functionalities in v2 by augmenting argparse classes
  and functions 
v2:
- complete rework of the arguments handling, to retain
  the environment variables and gain control over them
- prefixing 'Stderr: ' to RemoteCommandExecutionError
- rebased

---
Depends-on: series-31920 ("dts: update mypy and clean up")
---

Luca Vizzarro (3):
  dts: rework arguments framework
  dts: constrain DPDK source argument
  dts: store stderr in RemoteCommandExecutionError

 doc/guides/tools/dts.rst                      |  50 +--
 dts/framework/exception.py                    |  13 +-
 .../remote_session/remote_session.py          |   3 +-
 dts/framework/settings.py                     | 328 +++++++++++++-----
 dts/framework/utils.py                        |  43 ++-
 5 files changed, 309 insertions(+), 128 deletions(-)

-- 
2.34.1

[PATCH v3 1/3] dts: rework arguments framework

[Luca Vizzarro]

The existing argument handling in the code relies on basic argparse
functionality and a custom argparse action to integrate environment
variables. This commit improves the current handling by augmenting
argparse.

This rework implements the following improvements:
- There are duplicate expressions scattered throughout the code. To
  improve readability and maintainability, these are refactored
  into list/dict comprehensions or factory functions.
- Instead of relying solely on argument flags, error messages now
  accurately reference environment variables when applicable, enhancing
  user experience. For instance:

    error: environment variable DTS_DPDK_TARBALL: Invalid file

- Knowing the number of environment variables and arguments set
  allow for a useful help page display when none are provided.
- A display of which environment variables have been detected and their
  corresponding values in the help page, aiding user awareness.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/tools/dts.rst  |  53 +++++---
 dts/framework/settings.py | 280 +++++++++++++++++++++++++++-----------
 2 files changed, 235 insertions(+), 98 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 47b218b2c6..6993443389 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,30 +215,41 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
+   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
+                  [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
 
-   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
+   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
+   line arguments have higher priority.
 
    options:
-   -h, --help            show this help message and exit
-   --config-file CONFIG_FILE
-                         [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
-   --output-dir OUTPUT_DIR, --output OUTPUT_DIR
-                         [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
-   -t TIMEOUT, --timeout TIMEOUT
-                         [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-   -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
-   -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-   --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
-                         [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
-   --compile-timeout COMPILE_TIMEOUT
-                         [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
-   --test-suite TEST_SUITE [TEST_CASES ...]
-                         [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
-                         variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
-                         (default: [])
-   --re-run RE_RUN, --re_run RE_RUN
-                         [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
+     -h, --help            show this help message and exit
+     --config-file FILE_PATH
+                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
+                           (default: conf.yaml)
+     --output-dir DIR_PATH, --output DIR_PATH
+                           [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
+     -t SECONDS, --timeout SECONDS
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
+                           (default: 15)
+     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
+                           (default: False)
+     -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
+     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
+                           test. To test local changes, first commit them, then use the commit ID with this option.
+                           (default: dpdk.tar.xz)
+     --compile-timeout SECONDS
+                           [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
+     --test-suite TEST_SUITE [TEST_CASES ...]
+                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
+                           the test suite name, and the rest are test case names, which are optional. May be specified
+                           multiple times. To specify multiple test suites in the environment variable, join the lists
+                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
+                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
+                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
+     --re-run N_TIMES, --re_run N_TIMES
+                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
+                           (default: 0)
 
 
 The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 688e8679a7..d7af5adf2d 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2021 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Environment variables and command line arguments parsing.
 
@@ -72,9 +73,15 @@
 
 import argparse
 import os
+import sys
+from argparse import (
+    Action,
+    ArgumentDefaultsHelpFormatter,
+    _get_action_name,
+)
 from dataclasses import dataclass, field
 from pathlib import Path
-from typing import Any
+from typing import Callable, ParamSpec
 
 from .config import TestSuiteConfig
 from .utils import DPDKGitTarball
@@ -109,104 +116,224 @@ class Settings:
 
 SETTINGS: Settings = Settings()
 
+P = ParamSpec("P")
 
-def _get_parser() -> argparse.ArgumentParser:
-    """Create the argument parser for DTS.
+#: Attribute name representing the env variable name to augment :class:`~argparse.Action` with.
+_ENV_VAR_NAME_ATTR = "env_var_name"
+#: Attribute name representing the value origin to augment :class:`~argparse.Action` with.
+_IS_FROM_ENV_ATTR = "is_from_env"
 
-    Command line options take precedence over environment variables, which in turn take precedence
-    over default values.
+#: The prefix to be added to all of the environment variables.
+ENV_PREFIX = "DTS_"
+
+
+def is_action_in_args(action: Action) -> bool:
+    """Check if the action is invoked in the command line arguments."""
+    for option in action.option_strings:
+        if option in sys.argv:
+            return True
+    return False
+
+
+def make_env_var_name(action: Action, env_var_name: str | None) -> str:
+    """Make and assign an environment variable nam to the given action."""
+    env_var_name = f"{ENV_PREFIX}{env_var_name or action.dest.upper()}"
+    setattr(action, _ENV_VAR_NAME_ATTR, env_var_name)
+    return env_var_name
+
+
+def get_env_var_name(action: Action) -> str | None:
+    """Get the environment variable name of the given action."""
+    return getattr(action, _ENV_VAR_NAME_ATTR, None)
+
+
+def set_is_from_env(action: Action) -> None:
+    """Make the environment the given action's value origin."""
+    setattr(action, _IS_FROM_ENV_ATTR, True)
 
-    Returns:
-        argparse.ArgumentParser: The configured argument parser with defined options.
-    """
 
-    def env_arg(env_var: str, default: Any) -> Any:
-        """A helper function augmenting the argparse with environment variables.
+def is_from_env(action: Action) -> bool:
+    """Check if the given action's value originated from the environment."""
+    return getattr(action, _IS_FROM_ENV_ATTR, False)
 
-        If the supplied environment variable is defined, then the default value
-        of the argument is modified. This satisfies the priority order of
-        command line argument > environment variable > default value.
 
-        Args:
-            env_var: Environment variable name.
-            default: Default value.
+def augment_add_argument_with_env(
+    add_argument_fn: Callable[P, Action],
+):
+    """Augment any :class:`~argparse._ActionsContainer.add_argument` with environment variables."""
 
-        Returns:
-            Environment variable or default value.
+    def _add_argument(
+        *args: P.args,
+        env_var_name: str | None = None,
+        **kwargs: P.kwargs,
+    ) -> Action:
+        """Add an argument with an environment variable to the parser."""
+        action = add_argument_fn(*args, **kwargs)
+        env_var_name = make_env_var_name(action, env_var_name)
+
+        if not is_action_in_args(action):
+            env_var_value = os.environ.get(env_var_name)
+            if env_var_value:
+                set_is_from_env(action)
+                sys.argv[1:0] = [action.format_usage(), env_var_value]
+
+        return action
+
+    return _add_argument
+
+
+class ArgumentParser(argparse.ArgumentParser):
+    """ArgumentParser with a custom error message.
+
+    This custom version of ArgumentParser changes the error message to
+    accurately reflect its origin if an environment variable is used
+    as an argument.
+
+    Instead of printing usage on every error, it prints instructions
+    on how to do it.
+    """
+
+    def find_action(
+        self, action_name: str, filter_fn: Callable[[Action], bool] | None = None
+    ) -> Action | None:
+        """Find and return an action by its name.
+
+        Arguments:
+            action_name: the name of the action to find.
+            filter_fn: a filter function to use in the search.
         """
-        return os.environ.get(env_var) or default
+        it = (action for action in self._actions if action_name == _get_action_name(action))
+        action = next(it, None)
+
+        if action is not None and filter_fn is not None:
+            return action if filter_fn(action) else None
+
+        return action
 
-    parser = argparse.ArgumentParser(
+    def error(self, message):
+        """Augments :meth:`~argparse.ArgumentParser.error` with environment variable awareness."""
+        for action in self._actions:
+            if is_from_env(action):
+                action_name = _get_action_name(action)
+                env_var_name = get_env_var_name(action)
+                env_var_value = os.environ.get(env_var_name)
+
+                message = message.replace(
+                    f"argument {action_name}",
+                    f"environment variable {env_var_name} (value: {env_var_value})",
+                )
+
+        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
+        self.exit(2, "For help and usage, " "run the command with the --help flag.\n")
+
+
+class EnvVarHelpFormatter(ArgumentDefaultsHelpFormatter):
+    """Custom formatter to add environment variables in the help page."""
+
+    def _get_help_string(self, action):
+        """Overrides :meth:`ArgumentDefaultsHelpFormatter._get_help_string`."""
+        help = super()._get_help_string(action)
+
+        env_var_name = get_env_var_name(action)
+        if env_var_name is not None:
+            help = f"[{env_var_name}] {help}"
+
+            env_var_value = os.environ.get(env_var_name)
+            if env_var_value is not None:
+                help += f" (env value: {env_var_value})"
+
+        return help
+
+
+def _get_parser() -> ArgumentParser:
+    """Create the argument parser for DTS.
+
+    Command line options take precedence over environment variables, which in turn take precedence
+    over default values.
+
+    Returns:
+        ArgumentParser: The configured argument parser with defined options.
+    """
+    parser = ArgumentParser(
         description="Run DPDK test suites. All options may be specified with the environment "
         "variables provided in brackets. Command line arguments have higher priority.",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+        formatter_class=EnvVarHelpFormatter,
+        allow_abbrev=False,
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env = augment_add_argument_with_env(parser.add_argument)
+
+    add_argument_to_parser_with_env(
         "--config-file",
-        default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
+        default=SETTINGS.config_file_path,
         type=Path,
-        help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
-        "SUTs and targets.",
+        help="The configuration file that describes the test cases, SUTs and targets.",
+        metavar="FILE_PATH",
+        env_var_name="CFG_FILE",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--output-dir",
         "--output",
-        default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
-        help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
+        default=SETTINGS.output_dir,
+        help="Output directory where DTS logs and results are saved.",
+        metavar="DIR_PATH",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-t",
         "--timeout",
-        default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
+        default=SETTINGS.timeout,
         type=float,
-        help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
+        help="The default timeout for all DTS operations except for compiling DPDK.",
+        metavar="SECONDS",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-v",
         "--verbose",
         action="store_true",
-        default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
-        help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
-        "to the console.",
+        default=SETTINGS.verbose,
+        help="Specify to enable verbose output, logging all messages to the console.",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-s",
         "--skip-setup",
         action="store_true",
-        default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
-        help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
+        default=SETTINGS.skip_setup,
+        help="Specify to skip all setup steps on SUT and TG nodes.",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--tarball",
         "--snapshot",
         "--git-ref",
-        default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
+        default=SETTINGS.dpdk_tarball_path,
         type=Path,
-        help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
+        help="Path to DPDK source code tarball or a git commit ID, "
         "tag ID or tree ID to test. To test local changes, first commit them, "
         "then use the commit ID with this option.",
+        metavar="FILE_PATH",
+        dest="dpdk_tarball_path",
+        env_var_name="DPDK_TARBALL",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--compile-timeout",
-        default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
+        default=SETTINGS.compile_timeout,
         type=float,
-        help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
+        help="The timeout for compiling DPDK.",
+        metavar="SECONDS",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--test-suite",
         action="append",
         nargs="+",
         metavar=("TEST_SUITE", "TEST_CASES"),
-        default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
-        help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
+        default=SETTINGS.test_suites,
+        help="A list containing a test suite with test cases. "
         "The first parameter is the test suite name, and the rest are test case names, "
         "which are optional. May be specified multiple times. To specify multiple test suites in "
         "the environment variable, join the lists with a comma. "
@@ -215,21 +342,23 @@ def env_arg(env_var: str, default: Any) -> Any:
         "DTS_TEST_SUITES='suite case case, suite case, ...' | "
         "--test-suite suite --test-suite suite case ... | "
         "DTS_TEST_SUITES='suite, suite case, ...'",
+        dest="test_suites",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--re-run",
         "--re_run",
-        default=env_arg("DTS_RERUN", SETTINGS.re_run),
+        default=SETTINGS.re_run,
         type=int,
-        help="[DTS_RERUN] Re-run each test case the specified number of times "
-        "if a test failure occurs.",
+        help="Re-run each test case the specified number of times if a test failure occurs.",
+        env_var_name="RERUN",
+        metavar="N_TIMES",
     )
 
     return parser
 
 
-def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
+def _process_test_suites(parser: ArgumentParser, args: list[list[str]]) -> list[TestSuiteConfig]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -240,17 +369,12 @@ def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
     Returns:
         A list of test suite configurations to execute.
     """
-    if isinstance(args, str):
-        # Environment variable in the form of "suite case case, suite case, suite, ..."
-        args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
+    test_suites = parser.find_action("test_suites", is_from_env)
+    if test_suites is not None:
+        # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
+        args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    test_suites_to_run = []
-    for suite_with_cases in args:
-        test_suites_to_run.append(
-            TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
-        )
-
-    return test_suites_to_run
+    return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
 
 
 def get_settings() -> Settings:
@@ -261,19 +385,21 @@ def get_settings() -> Settings:
     Returns:
         The new settings object.
     """
-    parsed_args = _get_parser().parse_args()
-    return Settings(
-        config_file_path=parsed_args.config_file,
-        output_dir=parsed_args.output_dir,
-        timeout=parsed_args.timeout,
-        verbose=parsed_args.verbose,
-        skip_setup=parsed_args.skip_setup,
-        dpdk_tarball_path=Path(
-            Path(DPDKGitTarball(parsed_args.tarball, parsed_args.output_dir))
-            if not os.path.exists(parsed_args.tarball)
-            else Path(parsed_args.tarball)
-        ),
-        compile_timeout=parsed_args.compile_timeout,
-        test_suites=_process_test_suites(parsed_args.test_suite),
-        re_run=parsed_args.re_run,
+    parser = _get_parser()
+
+    if len(sys.argv) == 1:
+        parser.print_help()
+        sys.exit(1)
+
+    args = parser.parse_args()
+
+    args.dpdk_tarball_path = Path(
+        Path(DPDKGitTarball(args.dpdk_tarball_path, args.output_dir))
+        if not os.path.exists(args.dpdk_tarball_path)
+        else Path(args.dpdk_tarball_path)
     )
+
+    args.test_suites = _process_test_suites(parser, args.test_suites)
+
+    kwargs = {k: v for k, v in vars(args).items() if hasattr(SETTINGS, k)}
+    return Settings(**kwargs)
-- 
2.34.1

[PATCH v3] dts: rework arguments framework

[Luca Vizzarro]

The existing argument handling in the code relies on basic argparse
functionality and a custom argparse action to integrate environment
variables. This commit improves the current handling by augmenting
argparse.

This rework implements the following improvements:
- There are duplicate expressions scattered throughout the code. To
  improve readability and maintainability, these are refactored
  into list/dict comprehensions or factory functions.
- Instead of relying solely on argument flags, error messages now
  accurately reference environment variables when applicable, enhancing
  user experience. For instance:

    error: environment variable DTS_DPDK_TARBALL: Invalid file

- Knowing the number of environment variables and arguments set
  allow for a useful help page display when none are provided.
- A display of which environment variables have been detected and their
  corresponding values in the help page, aiding user awareness.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
sorry for the re-send. fixed docstring typo.
---
 doc/guides/tools/dts.rst  |  53 +++++---
 dts/framework/settings.py | 280 +++++++++++++++++++++++++++-----------
 2 files changed, 235 insertions(+), 98 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 47b218b2c6..6993443389 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,30 +215,41 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
+   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
+                  [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
 
-   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
+   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
+   line arguments have higher priority.
 
    options:
-   -h, --help            show this help message and exit
-   --config-file CONFIG_FILE
-                         [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
-   --output-dir OUTPUT_DIR, --output OUTPUT_DIR
-                         [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
-   -t TIMEOUT, --timeout TIMEOUT
-                         [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-   -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
-   -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-   --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
-                         [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
-   --compile-timeout COMPILE_TIMEOUT
-                         [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
-   --test-suite TEST_SUITE [TEST_CASES ...]
-                         [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
-                         variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
-                         (default: [])
-   --re-run RE_RUN, --re_run RE_RUN
-                         [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
+     -h, --help            show this help message and exit
+     --config-file FILE_PATH
+                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
+                           (default: conf.yaml)
+     --output-dir DIR_PATH, --output DIR_PATH
+                           [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
+     -t SECONDS, --timeout SECONDS
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
+                           (default: 15)
+     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
+                           (default: False)
+     -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
+     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
+                           test. To test local changes, first commit them, then use the commit ID with this option.
+                           (default: dpdk.tar.xz)
+     --compile-timeout SECONDS
+                           [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
+     --test-suite TEST_SUITE [TEST_CASES ...]
+                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
+                           the test suite name, and the rest are test case names, which are optional. May be specified
+                           multiple times. To specify multiple test suites in the environment variable, join the lists
+                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
+                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
+                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
+     --re-run N_TIMES, --re_run N_TIMES
+                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
+                           (default: 0)
 
 
 The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 688e8679a7..b19f274f9d 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2021 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Environment variables and command line arguments parsing.
 
@@ -72,9 +73,15 @@
 
 import argparse
 import os
+import sys
+from argparse import (
+    Action,
+    ArgumentDefaultsHelpFormatter,
+    _get_action_name,
+)
 from dataclasses import dataclass, field
 from pathlib import Path
-from typing import Any
+from typing import Callable, ParamSpec
 
 from .config import TestSuiteConfig
 from .utils import DPDKGitTarball
@@ -109,104 +116,224 @@ class Settings:
 
 SETTINGS: Settings = Settings()
 
+P = ParamSpec("P")
 
-def _get_parser() -> argparse.ArgumentParser:
-    """Create the argument parser for DTS.
+#: Attribute name representing the env variable name to augment :class:`~argparse.Action` with.
+_ENV_VAR_NAME_ATTR = "env_var_name"
+#: Attribute name representing the value origin to augment :class:`~argparse.Action` with.
+_IS_FROM_ENV_ATTR = "is_from_env"
 
-    Command line options take precedence over environment variables, which in turn take precedence
-    over default values.
+#: The prefix to be added to all of the environment variables.
+ENV_PREFIX = "DTS_"
+
+
+def is_action_in_args(action: Action) -> bool:
+    """Check if the action is invoked in the command line arguments."""
+    for option in action.option_strings:
+        if option in sys.argv:
+            return True
+    return False
+
+
+def make_env_var_name(action: Action, env_var_name: str | None) -> str:
+    """Make and assign an environment variable name to the given action."""
+    env_var_name = f"{ENV_PREFIX}{env_var_name or action.dest.upper()}"
+    setattr(action, _ENV_VAR_NAME_ATTR, env_var_name)
+    return env_var_name
+
+
+def get_env_var_name(action: Action) -> str | None:
+    """Get the environment variable name of the given action."""
+    return getattr(action, _ENV_VAR_NAME_ATTR, None)
+
+
+def set_is_from_env(action: Action) -> None:
+    """Make the environment the given action's value origin."""
+    setattr(action, _IS_FROM_ENV_ATTR, True)
 
-    Returns:
-        argparse.ArgumentParser: The configured argument parser with defined options.
-    """
 
-    def env_arg(env_var: str, default: Any) -> Any:
-        """A helper function augmenting the argparse with environment variables.
+def is_from_env(action: Action) -> bool:
+    """Check if the given action's value originated from the environment."""
+    return getattr(action, _IS_FROM_ENV_ATTR, False)
 
-        If the supplied environment variable is defined, then the default value
-        of the argument is modified. This satisfies the priority order of
-        command line argument > environment variable > default value.
 
-        Args:
-            env_var: Environment variable name.
-            default: Default value.
+def augment_add_argument_with_env(
+    add_argument_fn: Callable[P, Action],
+):
+    """Augment any :class:`~argparse._ActionsContainer.add_argument` with environment variables."""
 
-        Returns:
-            Environment variable or default value.
+    def _add_argument(
+        *args: P.args,
+        env_var_name: str | None = None,
+        **kwargs: P.kwargs,
+    ) -> Action:
+        """Add an argument with an environment variable to the parser."""
+        action = add_argument_fn(*args, **kwargs)
+        env_var_name = make_env_var_name(action, env_var_name)
+
+        if not is_action_in_args(action):
+            env_var_value = os.environ.get(env_var_name)
+            if env_var_value:
+                set_is_from_env(action)
+                sys.argv[1:0] = [action.format_usage(), env_var_value]
+
+        return action
+
+    return _add_argument
+
+
+class ArgumentParser(argparse.ArgumentParser):
+    """ArgumentParser with a custom error message.
+
+    This custom version of ArgumentParser changes the error message to
+    accurately reflect its origin if an environment variable is used
+    as an argument.
+
+    Instead of printing usage on every error, it prints instructions
+    on how to do it.
+    """
+
+    def find_action(
+        self, action_name: str, filter_fn: Callable[[Action], bool] | None = None
+    ) -> Action | None:
+        """Find and return an action by its name.
+
+        Arguments:
+            action_name: the name of the action to find.
+            filter_fn: a filter function to use in the search.
         """
-        return os.environ.get(env_var) or default
+        it = (action for action in self._actions if action_name == _get_action_name(action))
+        action = next(it, None)
+
+        if action is not None and filter_fn is not None:
+            return action if filter_fn(action) else None
+
+        return action
 
-    parser = argparse.ArgumentParser(
+    def error(self, message):
+        """Augments :meth:`~argparse.ArgumentParser.error` with environment variable awareness."""
+        for action in self._actions:
+            if is_from_env(action):
+                action_name = _get_action_name(action)
+                env_var_name = get_env_var_name(action)
+                env_var_value = os.environ.get(env_var_name)
+
+                message = message.replace(
+                    f"argument {action_name}",
+                    f"environment variable {env_var_name} (value: {env_var_value})",
+                )
+
+        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
+        self.exit(2, "For help and usage, " "run the command with the --help flag.\n")
+
+
+class EnvVarHelpFormatter(ArgumentDefaultsHelpFormatter):
+    """Custom formatter to add environment variables in the help page."""
+
+    def _get_help_string(self, action):
+        """Overrides :meth:`ArgumentDefaultsHelpFormatter._get_help_string`."""
+        help = super()._get_help_string(action)
+
+        env_var_name = get_env_var_name(action)
+        if env_var_name is not None:
+            help = f"[{env_var_name}] {help}"
+
+            env_var_value = os.environ.get(env_var_name)
+            if env_var_value is not None:
+                help += f" (env value: {env_var_value})"
+
+        return help
+
+
+def _get_parser() -> ArgumentParser:
+    """Create the argument parser for DTS.
+
+    Command line options take precedence over environment variables, which in turn take precedence
+    over default values.
+
+    Returns:
+        ArgumentParser: The configured argument parser with defined options.
+    """
+    parser = ArgumentParser(
         description="Run DPDK test suites. All options may be specified with the environment "
         "variables provided in brackets. Command line arguments have higher priority.",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+        formatter_class=EnvVarHelpFormatter,
+        allow_abbrev=False,
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env = augment_add_argument_with_env(parser.add_argument)
+
+    add_argument_to_parser_with_env(
         "--config-file",
-        default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
+        default=SETTINGS.config_file_path,
         type=Path,
-        help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
-        "SUTs and targets.",
+        help="The configuration file that describes the test cases, SUTs and targets.",
+        metavar="FILE_PATH",
+        env_var_name="CFG_FILE",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--output-dir",
         "--output",
-        default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
-        help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
+        default=SETTINGS.output_dir,
+        help="Output directory where DTS logs and results are saved.",
+        metavar="DIR_PATH",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-t",
         "--timeout",
-        default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
+        default=SETTINGS.timeout,
         type=float,
-        help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
+        help="The default timeout for all DTS operations except for compiling DPDK.",
+        metavar="SECONDS",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-v",
         "--verbose",
         action="store_true",
-        default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
-        help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
-        "to the console.",
+        default=SETTINGS.verbose,
+        help="Specify to enable verbose output, logging all messages to the console.",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-s",
         "--skip-setup",
         action="store_true",
-        default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
-        help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
+        default=SETTINGS.skip_setup,
+        help="Specify to skip all setup steps on SUT and TG nodes.",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--tarball",
         "--snapshot",
         "--git-ref",
-        default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
+        default=SETTINGS.dpdk_tarball_path,
         type=Path,
-        help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
+        help="Path to DPDK source code tarball or a git commit ID, "
         "tag ID or tree ID to test. To test local changes, first commit them, "
         "then use the commit ID with this option.",
+        metavar="FILE_PATH",
+        dest="dpdk_tarball_path",
+        env_var_name="DPDK_TARBALL",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--compile-timeout",
-        default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
+        default=SETTINGS.compile_timeout,
         type=float,
-        help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
+        help="The timeout for compiling DPDK.",
+        metavar="SECONDS",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--test-suite",
         action="append",
         nargs="+",
         metavar=("TEST_SUITE", "TEST_CASES"),
-        default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
-        help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
+        default=SETTINGS.test_suites,
+        help="A list containing a test suite with test cases. "
         "The first parameter is the test suite name, and the rest are test case names, "
         "which are optional. May be specified multiple times. To specify multiple test suites in "
         "the environment variable, join the lists with a comma. "
@@ -215,21 +342,23 @@ def env_arg(env_var: str, default: Any) -> Any:
         "DTS_TEST_SUITES='suite case case, suite case, ...' | "
         "--test-suite suite --test-suite suite case ... | "
         "DTS_TEST_SUITES='suite, suite case, ...'",
+        dest="test_suites",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--re-run",
         "--re_run",
-        default=env_arg("DTS_RERUN", SETTINGS.re_run),
+        default=SETTINGS.re_run,
         type=int,
-        help="[DTS_RERUN] Re-run each test case the specified number of times "
-        "if a test failure occurs.",
+        help="Re-run each test case the specified number of times if a test failure occurs.",
+        env_var_name="RERUN",
+        metavar="N_TIMES",
     )
 
     return parser
 
 
-def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
+def _process_test_suites(parser: ArgumentParser, args: list[list[str]]) -> list[TestSuiteConfig]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -240,17 +369,12 @@ def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
     Returns:
         A list of test suite configurations to execute.
     """
-    if isinstance(args, str):
-        # Environment variable in the form of "suite case case, suite case, suite, ..."
-        args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
+    test_suites = parser.find_action("test_suites", is_from_env)
+    if test_suites is not None:
+        # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
+        args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    test_suites_to_run = []
-    for suite_with_cases in args:
-        test_suites_to_run.append(
-            TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
-        )
-
-    return test_suites_to_run
+    return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
 
 
 def get_settings() -> Settings:
@@ -261,19 +385,21 @@ def get_settings() -> Settings:
     Returns:
         The new settings object.
     """
-    parsed_args = _get_parser().parse_args()
-    return Settings(
-        config_file_path=parsed_args.config_file,
-        output_dir=parsed_args.output_dir,
-        timeout=parsed_args.timeout,
-        verbose=parsed_args.verbose,
-        skip_setup=parsed_args.skip_setup,
-        dpdk_tarball_path=Path(
-            Path(DPDKGitTarball(parsed_args.tarball, parsed_args.output_dir))
-            if not os.path.exists(parsed_args.tarball)
-            else Path(parsed_args.tarball)
-        ),
-        compile_timeout=parsed_args.compile_timeout,
-        test_suites=_process_test_suites(parsed_args.test_suite),
-        re_run=parsed_args.re_run,
+    parser = _get_parser()
+
+    if len(sys.argv) == 1:
+        parser.print_help()
+        sys.exit(1)
+
+    args = parser.parse_args()
+
+    args.dpdk_tarball_path = Path(
+        Path(DPDKGitTarball(args.dpdk_tarball_path, args.output_dir))
+        if not os.path.exists(args.dpdk_tarball_path)
+        else Path(args.dpdk_tarball_path)
     )
+
+    args.test_suites = _process_test_suites(parser, args.test_suites)
+
+    kwargs = {k: v for k, v in vars(args).items() if hasattr(SETTINGS, k)}
+    return Settings(**kwargs)
-- 
2.34.1

[PATCH v3 2/3] dts: constrain DPDK source argument

[Luca Vizzarro]

DTS needs an input to gather the DPDK source code from. This is then
built on the remote target. This commit makes sure that this input is
more constrained, separating the Git revision ID – used to create a
tarball using Git – and providing tarballed source code directly, while
retaining mutual exclusion.

This makes the code more readable and easier to handle for input
validation, of which this commit introduces a basic one based on the
pre-existing code.

Moreover it ensures that these flags are explicitly required to be set
by the user, dropping a default value.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/tools/dts.rst  | 39 ++++++++++++------------
 dts/framework/settings.py | 62 +++++++++++++++++++++++++++++----------
 dts/framework/utils.py    | 43 ++++++++++++++++-----------
 3 files changed, 90 insertions(+), 54 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 6993443389..f64ab7f732 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,41 +215,38 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
+   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] (--tarball FILE_PATH | --revision ID)
                   [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
 
-   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
-   line arguments have higher priority.
+   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher
+   priority.
 
    options:
      -h, --help            show this help message and exit
      --config-file FILE_PATH
-                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
-                           (default: conf.yaml)
+                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default:
+                           /home/lucviz01/dpdk/dts/conf.yaml)
      --output-dir DIR_PATH, --output DIR_PATH
                            [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
      -t SECONDS, --timeout SECONDS
-                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
-                           (default: 15)
-     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
-                           (default: False)
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
+     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
      -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
-                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
-                           test. To test local changes, first commit them, then use the commit ID with this option.
-                           (default: dpdk.tar.xz)
+     --tarball FILE_PATH, --snapshot FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball to test. (default: None)
+     --revision ID, --rev ID, --git-ref ID
+                           [DTS_DPDK_REVISION_ID] Git revision ID to test. Could be commit, tag, tree ID etc. To test local changes, first
+                           commit them, then use their commit ID. (default: None)
      --compile-timeout SECONDS
                            [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
      --test-suite TEST_SUITE [TEST_CASES ...]
-                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
-                           the test suite name, and the rest are test case names, which are optional. May be specified
-                           multiple times. To specify multiple test suites in the environment variable, join the lists
-                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
-                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
-                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
+                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and
+                           the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites
+                           in the environment variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite
+                           suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case
+                           ... | DTS_TEST_SUITES='suite, suite case, ...' (default: [])
      --re-run N_TIMES, --re_run N_TIMES
-                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
-                           (default: 0)
+                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
 
 
 The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index d7af5adf2d..0bb7f57ed7 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -44,10 +44,16 @@
 
     Set to any value to skip building DPDK.
 
-.. option:: --tarball, --snapshot, --git-ref
+.. option:: --tarball, --snapshot
 .. envvar:: DTS_DPDK_TARBALL
 
-    The path to a DPDK tarball, git commit ID, tag ID or tree ID to test.
+    Path to DPDK source code tarball to test.
+
+.. option:: --revision, --rev, --git-ref
+.. envvar:: DTS_DPDK_REVISION_ID
+
+    Git revision ID to test. Could be commit, tag, tree ID etc.
+    To test local changes, first commit them, then use their commit ID.
 
 .. option:: --test-suite
 .. envvar:: DTS_TEST_SUITES
@@ -84,7 +90,24 @@
 from typing import Callable, ParamSpec
 
 from .config import TestSuiteConfig
-from .utils import DPDKGitTarball
+from .exception import ConfigurationError
+from .utils import DPDKGitTarball, get_commit_id
+
+
+def _parse_tarball_path(file_path: str) -> Path:
+    """Validate whether `file_path` is valid and return a Path object."""
+    path = Path(file_path)
+    if not path.exists() or not path.is_file():
+        raise argparse.ArgumentTypeError("The file path provided is not a valid file")
+    return path
+
+
+def _parse_revision_id(rev_id: str) -> str:
+    """Validate revision ID and retrieve corresponding commit ID."""
+    try:
+        return get_commit_id(rev_id)
+    except ConfigurationError:
+        raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")
 
 
 @dataclass(slots=True)
@@ -105,7 +128,7 @@ class Settings:
     #:
     skip_setup: bool = False
     #:
-    dpdk_tarball_path: Path | str = "dpdk.tar.xz"
+    dpdk_tarball_path: Path | str = ""
     #:
     compile_timeout: float = 1200
     #:
@@ -305,20 +328,30 @@ def _get_parser() -> ArgumentParser:
         help="Specify to skip all setup steps on SUT and TG nodes.",
     )
 
-    add_argument_to_parser_with_env(
+    dpdk_source = parser.add_mutually_exclusive_group(required=True)
+    add_argument_to_dpdk_source_with_env = augment_add_argument_with_env(dpdk_source.add_argument)
+
+    add_argument_to_dpdk_source_with_env(
         "--tarball",
         "--snapshot",
-        "--git-ref",
-        default=SETTINGS.dpdk_tarball_path,
-        type=Path,
-        help="Path to DPDK source code tarball or a git commit ID, "
-        "tag ID or tree ID to test. To test local changes, first commit them, "
-        "then use the commit ID with this option.",
+        type=_parse_tarball_path,
+        help="Path to DPDK source code tarball to test.",
         metavar="FILE_PATH",
         dest="dpdk_tarball_path",
         env_var_name="DPDK_TARBALL",
     )
 
+    add_argument_to_dpdk_source_with_env(
+        "--revision",
+        "--rev",
+        "--git-ref",
+        type=_parse_revision_id,
+        help="Git revision ID to test. Could be commit, tag, tree ID etc. "
+        "To test local changes, first commit them, then use their commit ID.",
+        metavar="ID",
+        dest="dpdk_revision_id",
+    ),
+
     add_argument_to_parser_with_env(
         "--compile-timeout",
         default=SETTINGS.compile_timeout,
@@ -393,11 +426,8 @@ def get_settings() -> Settings:
 
     args = parser.parse_args()
 
-    args.dpdk_tarball_path = Path(
-        Path(DPDKGitTarball(args.dpdk_tarball_path, args.output_dir))
-        if not os.path.exists(args.dpdk_tarball_path)
-        else Path(args.dpdk_tarball_path)
-    )
+    if args.dpdk_revision_id:
+        args.dpdk_tarball_path = DPDKGitTarball(args.dpdk_revision_id, args.output_dir)
 
     args.test_suites = _process_test_suites(parser, args.test_suites)
 
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index 74a11f1aaf..862bafb46c 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Various utility classes and functions.
 
@@ -70,6 +71,31 @@ def get_packet_summaries(packets: list[Packet]) -> str:
     return f"Packet contents: \n{packet_summaries}"
 
 
+def get_commit_id(rev_id: str) -> str:
+    """Given a Git revision ID, return the corresponding commit ID.
+
+    Args:
+        rev_id: The Git revision ID.
+
+    Raises:
+        ConfigurationError: The ``git rev-parse`` command failed, suggesting
+            an invalid or ambiguous revision ID was supplied.
+    """
+    result = subprocess.run(
+        ["git", "rev-parse", "--verify", rev_id],
+        text=True,
+        capture_output=True,
+    )
+    if result.returncode != 0:
+        raise ConfigurationError(
+            f"{rev_id} is not a valid git reference.\n"
+            f"Command: {result.args}\n"
+            f"Stdout: {result.stdout}\n"
+            f"Stderr: {result.stderr}"
+        )
+    return result.stdout.strip()
+
+
 class StrEnum(Enum):
     """Enum with members stored as strings."""
 
@@ -170,7 +196,6 @@ def __init__(
 
         self._tarball_dir = Path(output_dir, "tarball")
 
-        self._get_commit_id()
         self._create_tarball_dir()
 
         self._tarball_name = (
@@ -180,22 +205,6 @@ def __init__(
         if not self._tarball_path:
             self._create_tarball()
 
-    def _get_commit_id(self) -> None:
-        result = subprocess.run(
-            ["git", "rev-parse", "--verify", self._git_ref],
-            text=True,
-            capture_output=True,
-        )
-        if result.returncode != 0:
-            raise ConfigurationError(
-                f"{self._git_ref} is neither a path to an existing DPDK "
-                "archive nor a valid git reference.\n"
-                f"Command: {result.args}\n"
-                f"Stdout: {result.stdout}\n"
-                f"Stderr: {result.stderr}"
-            )
-        self._git_ref = result.stdout.strip()
-
     def _create_tarball_dir(self) -> None:
         os.makedirs(self._tarball_dir, exist_ok=True)
 
-- 
2.34.1

[PATCH v3 3/3] dts: store stderr in RemoteCommandExecutionError

[Luca Vizzarro]

Store the stderr of an executed command in RemoteCommandExecutionError.
Consequently, when the exception is logged the error message includes
the stderr.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/exception.py                     | 13 ++++++++++---
 dts/framework/remote_session/remote_session.py |  3 ++-
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/dts/framework/exception.py b/dts/framework/exception.py
index cce1e0231a..50724acdf2 100644
--- a/dts/framework/exception.py
+++ b/dts/framework/exception.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """DTS exceptions.
 
@@ -129,21 +130,27 @@ class RemoteCommandExecutionError(DTSError):
     severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
     #: The executed command.
     command: str
+    _command_stderr: str
     _command_return_code: int
 
-    def __init__(self, command: str, command_return_code: int):
+    def __init__(self, command: str, command_return_code: int, command_stderr: str):
         """Define the meaning of the first two arguments.
 
         Args:
             command: The executed command.
             command_return_code: The return code of the executed command.
+            command_stderr: The stderr of the executed command.
         """
         self.command = command
         self._command_return_code = command_return_code
+        self._command_stderr = command_stderr
 
     def __str__(self) -> str:
-        """Include both the command and return code in the string representation."""
-        return f"Command {self.command} returned a non-zero exit code: {self._command_return_code}"
+        """Include the command, its return code and stderr in the string representation."""
+        return (
+            f"Command '{self.command}' returned a non-zero exit code: "
+            f"{self._command_return_code}\nStderr: {self._command_stderr}"
+        )
 
 
 class InteractiveCommandExecutionError(DTSError):
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
index ad0f53720a..9aaa8c8a04 100644
--- a/dts/framework/remote_session/remote_session.py
+++ b/dts/framework/remote_session/remote_session.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Base remote session.
 
@@ -172,7 +173,7 @@ def send_command(
             )
             self._logger.debug(f"stdout: '{result.stdout}'")
             self._logger.debug(f"stderr: '{result.stderr}'")
-            raise RemoteCommandExecutionError(command, result.return_code)
+            raise RemoteCommandExecutionError(command, result.return_code, result.stderr)
         self._logger.debug(f"Received from '{command}':\n{result}")
         self.history.append(result)
         return result
-- 
2.34.1

[PATCH v4 0/3] error and usage improvements

[Luca Vizzarro]

Hi,

please ignore v3 as I sent it by mistake and also forgot to checkpatch.

v4:
- fix spelling typo
v3:
- amended arguments rework so that it retains the original functional
  style
- re-implemetend functionalities in v2 by augmenting argparse classes
  and functions 
v2:
- complete rework of the arguments handling, to retain
  the environment variables and gain control over them
- prefixing 'Stderr: ' to RemoteCommandExecutionError
- rebased

---
Depends-on: series-31920 ("dts: update mypy and clean up")
---

Luca Vizzarro (3):
  dts: update mypy static checker
  dts: clean up config types
  dts: rework arguments framework

 doc/guides/tools/dts.rst                      |  53 ++--
 dts/framework/config/__init__.py              |  49 +--
 dts/framework/logger.py                       |   4 +-
 .../interactive_remote_session.py             |   4 +-
 .../remote_session/interactive_shell.py       |   2 +-
 dts/framework/remote_session/ssh_session.py   |   6 +-
 dts/framework/runner.py                       |  14 +-
 dts/framework/settings.py                     | 280 +++++++++++++-----
 dts/framework/test_result.py                  |   4 +-
 dts/framework/test_suite.py                   |   6 +-
 dts/framework/testbed_model/tg_node.py        |   2 +-
 .../traffic_generator/__init__.py             |  10 +-
 .../capturing_traffic_generator.py            |   4 +-
 .../testbed_model/traffic_generator/scapy.py  |   6 +-
 .../traffic_generator/traffic_generator.py    |   2 +-
 dts/framework/utils.py                        |   2 +-
 dts/poetry.lock                               |  86 +++---
 dts/pyproject.toml                            |   3 +-
 dts/tests/TestSuite_os_udp.py                 |   4 +-
 dts/tests/TestSuite_pmd_buffer_scatter.py     |   8 +-
 20 files changed, 353 insertions(+), 196 deletions(-)

-- 
2.34.1

[PATCH v4 1/3] dts: update mypy static checker

[Luca Vizzarro]

Update the mypy static checker to the latest version and fix all the
reported errors.

Bugzilla ID: 1433

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/config/__init__.py              |  2 +-
 dts/framework/logger.py                       |  4 +-
 .../interactive_remote_session.py             |  4 +-
 .../remote_session/interactive_shell.py       |  2 +-
 dts/framework/remote_session/ssh_session.py   |  6 +-
 dts/framework/runner.py                       | 14 +--
 dts/framework/test_result.py                  |  4 +-
 dts/framework/test_suite.py                   |  6 +-
 dts/framework/testbed_model/tg_node.py        |  2 +-
 .../capturing_traffic_generator.py            |  4 +-
 .../testbed_model/traffic_generator/scapy.py  |  6 +-
 .../traffic_generator/traffic_generator.py    |  2 +-
 dts/framework/utils.py                        |  2 +-
 dts/poetry.lock                               | 86 +++++++++++--------
 dts/pyproject.toml                            |  2 +-
 dts/tests/TestSuite_os_udp.py                 |  4 +-
 dts/tests/TestSuite_pmd_buffer_scatter.py     |  8 +-
 17 files changed, 87 insertions(+), 71 deletions(-)

diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 4cb5c74059..6b2ad2b16d 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -40,7 +40,7 @@
 from pathlib import Path
 from typing import Union
 
-import warlock  # type: ignore[import]
+import warlock  # type: ignore[import-untyped]
 import yaml
 
 from framework.config.types import (
diff --git a/dts/framework/logger.py b/dts/framework/logger.py
index fc6c50c983..5b88725481 100644
--- a/dts/framework/logger.py
+++ b/dts/framework/logger.py
@@ -79,7 +79,7 @@ def makeRecord(self, *args, **kwargs) -> logging.LogRecord:
             record: The generated record with the stage information.
         """
         record = super().makeRecord(*args, **kwargs)
-        record.stage = DTSLogger._stage  # type: ignore[attr-defined]
+        record.stage = DTSLogger._stage
         return record
 
     def add_dts_root_logger_handlers(self, verbose: bool, output_dir: str) -> None:
@@ -178,7 +178,7 @@ def _remove_extra_file_handlers(self) -> None:
             self._extra_file_handlers = []
 
 
-def get_dts_logger(name: str = None) -> DTSLogger:
+def get_dts_logger(name: str | None = None) -> DTSLogger:
     """Return a DTS logger instance identified by `name`.
 
     Args:
diff --git a/dts/framework/remote_session/interactive_remote_session.py b/dts/framework/remote_session/interactive_remote_session.py
index c50790db79..97194e6af8 100644
--- a/dts/framework/remote_session/interactive_remote_session.py
+++ b/dts/framework/remote_session/interactive_remote_session.py
@@ -6,8 +6,8 @@
 import socket
 import traceback
 
-from paramiko import AutoAddPolicy, SSHClient, Transport  # type: ignore[import]
-from paramiko.ssh_exception import (  # type: ignore[import]
+from paramiko import AutoAddPolicy, SSHClient, Transport  # type: ignore[import-untyped]
+from paramiko.ssh_exception import (  # type: ignore[import-untyped]
     AuthenticationException,
     BadHostKeyException,
     NoValidConnectionsError,
diff --git a/dts/framework/remote_session/interactive_shell.py b/dts/framework/remote_session/interactive_shell.py
index 5cfe202e15..074a541279 100644
--- a/dts/framework/remote_session/interactive_shell.py
+++ b/dts/framework/remote_session/interactive_shell.py
@@ -18,7 +18,7 @@
 from pathlib import PurePath
 from typing import Callable, ClassVar
 
-from paramiko import Channel, SSHClient, channel  # type: ignore[import]
+from paramiko import Channel, SSHClient, channel  # type: ignore[import-untyped]
 
 from framework.logger import DTSLogger
 from framework.settings import SETTINGS
diff --git a/dts/framework/remote_session/ssh_session.py b/dts/framework/remote_session/ssh_session.py
index 782220092c..216bd25aed 100644
--- a/dts/framework/remote_session/ssh_session.py
+++ b/dts/framework/remote_session/ssh_session.py
@@ -7,13 +7,13 @@
 import traceback
 from pathlib import PurePath
 
-from fabric import Connection  # type: ignore[import]
-from invoke.exceptions import (  # type: ignore[import]
+from fabric import Connection  # type: ignore[import-untyped]
+from invoke.exceptions import (  # type: ignore[import-untyped]
     CommandTimedOut,
     ThreadException,
     UnexpectedExit,
 )
-from paramiko.ssh_exception import (  # type: ignore[import]
+from paramiko.ssh_exception import (  # type: ignore[import-untyped]
     AuthenticationException,
     BadHostKeyException,
     NoValidConnectionsError,
diff --git a/dts/framework/runner.py b/dts/framework/runner.py
index db8e3ba96b..d74f1871db 100644
--- a/dts/framework/runner.py
+++ b/dts/framework/runner.py
@@ -23,7 +23,7 @@
 import re
 import sys
 from pathlib import Path
-from types import MethodType
+from types import FunctionType
 from typing import Iterable, Sequence
 
 from .config import (
@@ -132,8 +132,8 @@ def run(self):
         the :option:`--test-suite` command line argument or
         the :envvar:`DTS_TESTCASES` environment variable.
         """
-        sut_nodes: dict[str, SutNode] = {}
-        tg_nodes: dict[str, TGNode] = {}
+        sut_nodes = {}
+        tg_nodes = {}
         try:
             # check the python version of the server that runs dts
             self._check_dts_python_version()
@@ -305,7 +305,7 @@ def is_test_suite(object) -> bool:
 
     def _filter_test_cases(
         self, test_suite_class: type[TestSuite], test_cases_to_run: Sequence[str]
-    ) -> tuple[list[MethodType], list[MethodType]]:
+    ) -> tuple[list[FunctionType], list[FunctionType]]:
         """Filter `test_cases_to_run` from `test_suite_class`.
 
         There are two rounds of filtering if `test_cases_to_run` is not empty.
@@ -593,7 +593,7 @@ def _run_test_suite(
     def _execute_test_suite(
         self,
         test_suite: TestSuite,
-        test_cases: Iterable[MethodType],
+        test_cases: Iterable[FunctionType],
         test_suite_result: TestSuiteResult,
     ) -> None:
         """Execute all `test_cases` in `test_suite`.
@@ -626,7 +626,7 @@ def _execute_test_suite(
     def _run_test_case(
         self,
         test_suite: TestSuite,
-        test_case_method: MethodType,
+        test_case_method: FunctionType,
         test_case_result: TestCaseResult,
     ) -> None:
         """Setup, execute and teardown `test_case_method` from `test_suite`.
@@ -672,7 +672,7 @@ def _run_test_case(
     def _execute_test_case(
         self,
         test_suite: TestSuite,
-        test_case_method: MethodType,
+        test_case_method: FunctionType,
         test_case_result: TestCaseResult,
     ) -> None:
         """Execute `test_case_method` from `test_suite`, record the result and handle failures.
diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index 28f84fd793..d8d0fe2b2b 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -27,7 +27,7 @@
 from collections.abc import MutableSequence
 from dataclasses import dataclass
 from enum import Enum, auto
-from types import MethodType
+from types import FunctionType
 from typing import Union
 
 from .config import (
@@ -63,7 +63,7 @@ class is to hold a subset of test cases (which could be all test cases) because
     """
 
     test_suite_class: type[TestSuite]
-    test_cases: list[MethodType]
+    test_cases: list[FunctionType]
 
     def create_config(self) -> TestSuiteConfig:
         """Generate a :class:`TestSuiteConfig` from the stored test suite with test cases.
diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 9c3b516002..8768f756a6 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -16,9 +16,9 @@
 from ipaddress import IPv4Interface, IPv6Interface, ip_interface
 from typing import ClassVar, Union
 
-from scapy.layers.inet import IP  # type: ignore[import]
-from scapy.layers.l2 import Ether  # type: ignore[import]
-from scapy.packet import Packet, Padding  # type: ignore[import]
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Packet, Padding  # type: ignore[import-untyped]
 
 from .exception import TestCaseVerifyError
 from .logger import DTSLogger, get_dts_logger
diff --git a/dts/framework/testbed_model/tg_node.py b/dts/framework/testbed_model/tg_node.py
index d3206e87e0..164f790383 100644
--- a/dts/framework/testbed_model/tg_node.py
+++ b/dts/framework/testbed_model/tg_node.py
@@ -9,7 +9,7 @@
 A TG node is where the TG runs.
 """
 
-from scapy.packet import Packet  # type: ignore[import]
+from scapy.packet import Packet  # type: ignore[import-untyped]
 
 from framework.config import TGNodeConfiguration
 
diff --git a/dts/framework/testbed_model/traffic_generator/capturing_traffic_generator.py b/dts/framework/testbed_model/traffic_generator/capturing_traffic_generator.py
index e5a1560e90..c8380b7d57 100644
--- a/dts/framework/testbed_model/traffic_generator/capturing_traffic_generator.py
+++ b/dts/framework/testbed_model/traffic_generator/capturing_traffic_generator.py
@@ -13,8 +13,8 @@
 from abc import abstractmethod
 from dataclasses import dataclass
 
-import scapy.utils  # type: ignore[import]
-from scapy.packet import Packet  # type: ignore[import]
+import scapy.utils  # type: ignore[import-untyped]
+from scapy.packet import Packet  # type: ignore[import-untyped]
 
 from framework.settings import SETTINGS
 from framework.testbed_model.port import Port
diff --git a/dts/framework/testbed_model/traffic_generator/scapy.py b/dts/framework/testbed_model/traffic_generator/scapy.py
index df3069d516..ed5467d825 100644
--- a/dts/framework/testbed_model/traffic_generator/scapy.py
+++ b/dts/framework/testbed_model/traffic_generator/scapy.py
@@ -20,9 +20,9 @@
 import xmlrpc.client
 from xmlrpc.server import SimpleXMLRPCServer
 
-import scapy.all  # type: ignore[import]
-from scapy.layers.l2 import Ether  # type: ignore[import]
-from scapy.packet import Packet  # type: ignore[import]
+import scapy.all  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Packet  # type: ignore[import-untyped]
 
 from framework.config import OS, ScapyTrafficGeneratorConfig
 from framework.remote_session import PythonShell
diff --git a/dts/framework/testbed_model/traffic_generator/traffic_generator.py b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
index d86d7fb532..4ce1148706 100644
--- a/dts/framework/testbed_model/traffic_generator/traffic_generator.py
+++ b/dts/framework/testbed_model/traffic_generator/traffic_generator.py
@@ -10,7 +10,7 @@
 
 from abc import ABC, abstractmethod
 
-from scapy.packet import Packet  # type: ignore[import]
+from scapy.packet import Packet  # type: ignore[import-untyped]
 
 from framework.config import TrafficGeneratorConfig
 from framework.logger import DTSLogger, get_dts_logger
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index cc5e458cc8..74a11f1aaf 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -21,7 +21,7 @@
 from pathlib import Path
 from subprocess import SubprocessError
 
-from scapy.packet import Packet  # type: ignore[import]
+from scapy.packet import Packet  # type: ignore[import-untyped]
 
 from .exception import ConfigurationError
 
diff --git a/dts/poetry.lock b/dts/poetry.lock
index a734fa71f0..df9cecb7e0 100644
--- a/dts/poetry.lock
+++ b/dts/poetry.lock
@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.5.1 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
 
 [[package]]
 name = "attrs"
@@ -353,44 +353,49 @@ files = [
 
 [[package]]
 name = "mypy"
-version = "0.961"
+version = "1.10.0"
 description = "Optional static typing for Python"
 optional = false
-python-versions = ">=3.6"
+python-versions = ">=3.8"
 files = [
-    {file = "mypy-0.961-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:697540876638ce349b01b6786bc6094ccdaba88af446a9abb967293ce6eaa2b0"},
-    {file = "mypy-0.961-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:b117650592e1782819829605a193360a08aa99f1fc23d1d71e1a75a142dc7e15"},
-    {file = "mypy-0.961-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:bdd5ca340beffb8c44cb9dc26697628d1b88c6bddf5c2f6eb308c46f269bb6f3"},
-    {file = "mypy-0.961-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:3e09f1f983a71d0672bbc97ae33ee3709d10c779beb613febc36805a6e28bb4e"},
-    {file = "mypy-0.961-cp310-cp310-win_amd64.whl", hash = "sha256:e999229b9f3198c0c880d5e269f9f8129c8862451ce53a011326cad38b9ccd24"},
-    {file = "mypy-0.961-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:b24be97351084b11582fef18d79004b3e4db572219deee0212078f7cf6352723"},
-    {file = "mypy-0.961-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f4a21d01fc0ba4e31d82f0fff195682e29f9401a8bdb7173891070eb260aeb3b"},
-    {file = "mypy-0.961-cp36-cp36m-win_amd64.whl", hash = "sha256:439c726a3b3da7ca84a0199a8ab444cd8896d95012c4a6c4a0d808e3147abf5d"},
-    {file = "mypy-0.961-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:5a0b53747f713f490affdceef835d8f0cb7285187a6a44c33821b6d1f46ed813"},
-    {file = "mypy-0.961-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:0e9f70df36405c25cc530a86eeda1e0867863d9471fe76d1273c783df3d35c2e"},
-    {file = "mypy-0.961-cp37-cp37m-win_amd64.whl", hash = "sha256:b88f784e9e35dcaa075519096dc947a388319cb86811b6af621e3523980f1c8a"},
-    {file = "mypy-0.961-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:d5aaf1edaa7692490f72bdb9fbd941fbf2e201713523bdb3f4038be0af8846c6"},
-    {file = "mypy-0.961-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:9f5f5a74085d9a81a1f9c78081d60a0040c3efb3f28e5c9912b900adf59a16e6"},
-    {file = "mypy-0.961-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:f4b794db44168a4fc886e3450201365c9526a522c46ba089b55e1f11c163750d"},
-    {file = "mypy-0.961-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:64759a273d590040a592e0f4186539858c948302c653c2eac840c7a3cd29e51b"},
-    {file = "mypy-0.961-cp38-cp38-win_amd64.whl", hash = "sha256:63e85a03770ebf403291ec50097954cc5caf2a9205c888ce3a61bd3f82e17569"},
-    {file = "mypy-0.961-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:5f1332964963d4832a94bebc10f13d3279be3ce8f6c64da563d6ee6e2eeda932"},
-    {file = "mypy-0.961-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:006be38474216b833eca29ff6b73e143386f352e10e9c2fbe76aa8549e5554f5"},
-    {file = "mypy-0.961-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:9940e6916ed9371809b35b2154baf1f684acba935cd09928952310fbddaba648"},
-    {file = "mypy-0.961-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:a5ea0875a049de1b63b972456542f04643daf320d27dc592d7c3d9cd5d9bf950"},
-    {file = "mypy-0.961-cp39-cp39-win_amd64.whl", hash = "sha256:1ece702f29270ec6af25db8cf6185c04c02311c6bb21a69f423d40e527b75c56"},
-    {file = "mypy-0.961-py3-none-any.whl", hash = "sha256:03c6cc893e7563e7b2949b969e63f02c000b32502a1b4d1314cabe391aa87d66"},
-    {file = "mypy-0.961.tar.gz", hash = "sha256:f730d56cb924d371c26b8eaddeea3cc07d78ff51c521c6d04899ac6904b75492"},
+    {file = "mypy-1.10.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:da1cbf08fb3b851ab3b9523a884c232774008267b1f83371ace57f412fe308c2"},
+    {file = "mypy-1.10.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:12b6bfc1b1a66095ab413160a6e520e1dc076a28f3e22f7fb25ba3b000b4ef99"},
+    {file = "mypy-1.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9e36fb078cce9904c7989b9693e41cb9711e0600139ce3970c6ef814b6ebc2b2"},
+    {file = "mypy-1.10.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:2b0695d605ddcd3eb2f736cd8b4e388288c21e7de85001e9f85df9187f2b50f9"},
+    {file = "mypy-1.10.0-cp310-cp310-win_amd64.whl", hash = "sha256:cd777b780312ddb135bceb9bc8722a73ec95e042f911cc279e2ec3c667076051"},
+    {file = "mypy-1.10.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:3be66771aa5c97602f382230165b856c231d1277c511c9a8dd058be4784472e1"},
+    {file = "mypy-1.10.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8b2cbaca148d0754a54d44121b5825ae71868c7592a53b7292eeb0f3fdae95ee"},
+    {file = "mypy-1.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1ec404a7cbe9fc0e92cb0e67f55ce0c025014e26d33e54d9e506a0f2d07fe5de"},
+    {file = "mypy-1.10.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:e22e1527dc3d4aa94311d246b59e47f6455b8729f4968765ac1eacf9a4760bc7"},
+    {file = "mypy-1.10.0-cp311-cp311-win_amd64.whl", hash = "sha256:a87dbfa85971e8d59c9cc1fcf534efe664d8949e4c0b6b44e8ca548e746a8d53"},
+    {file = "mypy-1.10.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:a781f6ad4bab20eef8b65174a57e5203f4be627b46291f4589879bf4e257b97b"},
+    {file = "mypy-1.10.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b808e12113505b97d9023b0b5e0c0705a90571c6feefc6f215c1df9381256e30"},
+    {file = "mypy-1.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8f55583b12156c399dce2df7d16f8a5095291354f1e839c252ec6c0611e86e2e"},
+    {file = "mypy-1.10.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:4cf18f9d0efa1b16478c4c129eabec36148032575391095f73cae2e722fcf9d5"},
+    {file = "mypy-1.10.0-cp312-cp312-win_amd64.whl", hash = "sha256:bc6ac273b23c6b82da3bb25f4136c4fd42665f17f2cd850771cb600bdd2ebeda"},
+    {file = "mypy-1.10.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:9fd50226364cd2737351c79807775136b0abe084433b55b2e29181a4c3c878c0"},
+    {file = "mypy-1.10.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:f90cff89eea89273727d8783fef5d4a934be2fdca11b47def50cf5d311aff727"},
+    {file = "mypy-1.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fcfc70599efde5c67862a07a1aaf50e55bce629ace26bb19dc17cece5dd31ca4"},
+    {file = "mypy-1.10.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:075cbf81f3e134eadaf247de187bd604748171d6b79736fa9b6c9685b4083061"},
+    {file = "mypy-1.10.0-cp38-cp38-win_amd64.whl", hash = "sha256:3f298531bca95ff615b6e9f2fc0333aae27fa48052903a0ac90215021cdcfa4f"},
+    {file = "mypy-1.10.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:fa7ef5244615a2523b56c034becde4e9e3f9b034854c93639adb667ec9ec2976"},
+    {file = "mypy-1.10.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:3236a4c8f535a0631f85f5fcdffba71c7feeef76a6002fcba7c1a8e57c8be1ec"},
+    {file = "mypy-1.10.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4a2b5cdbb5dd35aa08ea9114436e0d79aceb2f38e32c21684dcf8e24e1e92821"},
+    {file = "mypy-1.10.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:92f93b21c0fe73dc00abf91022234c79d793318b8a96faac147cd579c1671746"},
+    {file = "mypy-1.10.0-cp39-cp39-win_amd64.whl", hash = "sha256:28d0e038361b45f099cc086d9dd99c15ff14d0188f44ac883010e172ce86c38a"},
+    {file = "mypy-1.10.0-py3-none-any.whl", hash = "sha256:f8c083976eb530019175aabadb60921e73b4f45736760826aa1689dda8208aee"},
+    {file = "mypy-1.10.0.tar.gz", hash = "sha256:3d087fcbec056c4ee34974da493a826ce316947485cef3901f511848e687c131"},
 ]
 
 [package.dependencies]
-mypy-extensions = ">=0.4.3"
+mypy-extensions = ">=1.0.0"
 tomli = {version = ">=1.1.0", markers = "python_version < \"3.11\""}
-typing-extensions = ">=3.10"
+typing-extensions = ">=4.1.0"
 
 [package.extras]
 dmypy = ["psutil (>=4.0)"]
-python2 = ["typed-ast (>=1.4.0,<2)"]
+install-types = ["pip"]
+mypyc = ["setuptools (>=50)"]
 reports = ["lxml"]
 
 [[package]]
@@ -580,6 +585,7 @@ files = [
     {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:69b023b2b4daa7548bcfbd4aa3da05b3a74b772db9e23b982788168117739938"},
     {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:81e0b275a9ecc9c0c0c07b4b90ba548307583c125f54d5b6946cfee6360c733d"},
     {file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ba336e390cd8e4d1739f42dfe9bb83a3cc2e80f567d8805e11b46f4a943f5515"},
+    {file = "PyYAML-6.0.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:326c013efe8048858a6d312ddd31d56e468118ad4cdeda36c719bf5bb6192290"},
     {file = "PyYAML-6.0.1-cp310-cp310-win32.whl", hash = "sha256:bd4af7373a854424dabd882decdc5579653d7868b8fb26dc7d0e99f823aa5924"},
     {file = "PyYAML-6.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:fd1592b3fdf65fff2ad0004b5e363300ef59ced41c2e6b3a99d4089fa8c5435d"},
     {file = "PyYAML-6.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6965a7bc3cf88e5a1c3bd2e0b5c22f8d677dc88a455344035f03399034eb3007"},
@@ -587,8 +593,16 @@ files = [
     {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:42f8152b8dbc4fe7d96729ec2b99c7097d656dc1213a3229ca5383f973a5ed6d"},
     {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:062582fca9fabdd2c8b54a3ef1c978d786e0f6b3a1510e0ac93ef59e0ddae2bc"},
     {file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d2b04aac4d386b172d5b9692e2d2da8de7bfb6c387fa4f801fbf6fb2e6ba4673"},
+    {file = "PyYAML-6.0.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:e7d73685e87afe9f3b36c799222440d6cf362062f78be1013661b00c5c6f678b"},
     {file = "PyYAML-6.0.1-cp311-cp311-win32.whl", hash = "sha256:1635fd110e8d85d55237ab316b5b011de701ea0f29d07611174a1b42f1444741"},
     {file = "PyYAML-6.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34"},
+    {file = "PyYAML-6.0.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:855fb52b0dc35af121542a76b9a84f8d1cd886ea97c84703eaa6d88e37a2ad28"},
+    {file = "PyYAML-6.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:40df9b996c2b73138957fe23a16a4f0ba614f4c0efce1e9406a184b6d07fa3a9"},
+    {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a08c6f0fe150303c1c6b71ebcd7213c2858041a7e01975da3a99aed1e7a378ef"},
+    {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c22bec3fbe2524cde73d7ada88f6566758a8f7227bfbf93a408a9d86bcc12a0"},
+    {file = "PyYAML-6.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8d4e9c88387b0f5c7d5f281e55304de64cf7f9c0021a3525bd3b1c542da3b0e4"},
+    {file = "PyYAML-6.0.1-cp312-cp312-win32.whl", hash = "sha256:d483d2cdf104e7c9fa60c544d92981f12ad66a457afae824d146093b8c294c54"},
+    {file = "PyYAML-6.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:0d3304d8c0adc42be59c5f8a4d9e3d7379e6955ad754aa9d6ab7a398b59dd1df"},
     {file = "PyYAML-6.0.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:50550eb667afee136e9a77d6dc71ae76a44df8b3e51e41b77f6de2932bfe0f47"},
     {file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1fe35611261b29bd1de0070f0b2f47cb6ff71fa6595c077e42bd0c419fa27b98"},
     {file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:704219a11b772aea0d8ecd7058d0082713c3562b4e271b849ad7dc4a5c90c13c"},
@@ -605,6 +619,7 @@ files = [
     {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a0cd17c15d3bb3fa06978b4e8958dcdc6e0174ccea823003a106c7d4d7899ac5"},
     {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:28c119d996beec18c05208a8bd78cbe4007878c6dd15091efb73a30e90539696"},
     {file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7e07cbde391ba96ab58e532ff4803f79c4129397514e1413a7dc761ccd755735"},
+    {file = "PyYAML-6.0.1-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:49a183be227561de579b4a36efbb21b3eab9651dd81b1858589f796549873dd6"},
     {file = "PyYAML-6.0.1-cp38-cp38-win32.whl", hash = "sha256:184c5108a2aca3c5b3d3bf9395d50893a7ab82a38004c8f61c258d4428e80206"},
     {file = "PyYAML-6.0.1-cp38-cp38-win_amd64.whl", hash = "sha256:1e2722cc9fbb45d9b87631ac70924c11d3a401b2d7f410cc0e3bbf249f2dca62"},
     {file = "PyYAML-6.0.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9eb6caa9a297fc2c2fb8862bc5370d0303ddba53ba97e71f08023b6cd73d16a8"},
@@ -612,6 +627,7 @@ files = [
     {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5773183b6446b2c99bb77e77595dd486303b4faab2b086e7b17bc6bef28865f6"},
     {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b786eecbdf8499b9ca1d697215862083bd6d2a99965554781d0d8d1ad31e13a0"},
     {file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bc1bf2925a1ecd43da378f4db9e4f799775d6367bdb94671027b73b393a7c42c"},
+    {file = "PyYAML-6.0.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:04ac92ad1925b2cff1db0cfebffb6ffc43457495c9b3c39d3fcae417d7125dc5"},
     {file = "PyYAML-6.0.1-cp39-cp39-win32.whl", hash = "sha256:faca3bdcf85b2fc05d06ff3fbc1f83e1391b3e724afa3feba7d13eeab355484c"},
     {file = "PyYAML-6.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:510c9deebc5c0225e8c96813043e62b680ba2f9c50a08d3724c7f28a747d1486"},
     {file = "PyYAML-6.0.1.tar.gz", hash = "sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43"},
@@ -810,13 +826,13 @@ files = [
 
 [[package]]
 name = "typing-extensions"
-version = "4.7.1"
-description = "Backported and Experimental Type Hints for Python 3.7+"
+version = "4.11.0"
+description = "Backported and Experimental Type Hints for Python 3.8+"
 optional = false
-python-versions = ">=3.7"
+python-versions = ">=3.8"
 files = [
-    {file = "typing_extensions-4.7.1-py3-none-any.whl", hash = "sha256:440d5dd3af93b060174bf433bccd69b0babc3b15b1a8dca43789fd7f61514b36"},
-    {file = "typing_extensions-4.7.1.tar.gz", hash = "sha256:b75ddc264f0ba5615db7ba217daeb99701ad295353c45f9e95963337ceeeffb2"},
+    {file = "typing_extensions-4.11.0-py3-none-any.whl", hash = "sha256:c1f94d72897edaf4ce775bb7558d5b79d8126906a14ea5ed1635921406c0387a"},
+    {file = "typing_extensions-4.11.0.tar.gz", hash = "sha256:83f085bd5ca59c80295fc2a82ab5dac679cbe02b9f33f7d83af68e241bea51b0"},
 ]
 
 [[package]]
@@ -837,4 +853,4 @@ jsonschema = ">=4,<5"
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.10"
-content-hash = "3501e97b3dadc19fe8ae179fe21b1edd2488001da9a8e86ff2bca0b86b99b89b"
+content-hash = "1572cb14f0bf88cddc6b6b225312ad468d01916b32067d17a7776af76c6d466c"
diff --git a/dts/pyproject.toml b/dts/pyproject.toml
index a81e46fc07..05c91ef9be 100644
--- a/dts/pyproject.toml
+++ b/dts/pyproject.toml
@@ -28,7 +28,7 @@ scapy = "^2.5.0"
 pydocstyle = "6.1.1"
 
 [tool.poetry.group.dev.dependencies]
-mypy = "^0.961"
+mypy = "^1.10.0"
 black = "^22.6.0"
 isort = "^5.10.1"
 pylama = "^8.4.1"
diff --git a/dts/tests/TestSuite_os_udp.py b/dts/tests/TestSuite_os_udp.py
index b4784dd95e..a78bd74139 100644
--- a/dts/tests/TestSuite_os_udp.py
+++ b/dts/tests/TestSuite_os_udp.py
@@ -7,8 +7,8 @@
 Send a packet to the SUT node, verify it comes back on the second port on the TG node.
 """
 
-from scapy.layers.inet import IP, UDP  # type: ignore[import]
-from scapy.layers.l2 import Ether  # type: ignore[import]
+from scapy.layers.inet import IP, UDP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
 
 from framework.test_suite import TestSuite
 
diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/TestSuite_pmd_buffer_scatter.py
index 3701c47408..a020682e8d 100644
--- a/dts/tests/TestSuite_pmd_buffer_scatter.py
+++ b/dts/tests/TestSuite_pmd_buffer_scatter.py
@@ -17,10 +17,10 @@
 
 import struct
 
-from scapy.layers.inet import IP  # type: ignore[import]
-from scapy.layers.l2 import Ether  # type: ignore[import]
-from scapy.packet import Raw  # type: ignore[import]
-from scapy.utils import hexstr  # type: ignore[import]
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+from scapy.utils import hexstr  # type: ignore[import-untyped]
 
 from framework.remote_session.testpmd_shell import TestPmdForwardingModes, TestPmdShell
 from framework.test_suite import TestSuite
-- 
2.34.1

[PATCH v4 2/3] dts: clean up config types

[Luca Vizzarro]

Clean up types used with the configuration classes, and use Self from
the newly added typing_extensions module.

Methods that instantiate their own class should be @classmethod instead
of @staticmethod.

Bugzilla ID: 1433

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/config/__init__.py              | 47 ++++++++++---------
 .../traffic_generator/__init__.py             | 10 ++--
 dts/poetry.lock                               |  2 +-
 dts/pyproject.toml                            |  1 +
 4 files changed, 32 insertions(+), 28 deletions(-)

diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
index 6b2ad2b16d..41b2fbc94d 100644
--- a/dts/framework/config/__init__.py
+++ b/dts/framework/config/__init__.py
@@ -42,6 +42,7 @@
 
 import warlock  # type: ignore[import-untyped]
 import yaml
+from typing_extensions import Self
 
 from framework.config.types import (
     BuildTargetConfigDict,
@@ -156,8 +157,8 @@ class PortConfig:
     peer_node: str
     peer_pci: str
 
-    @staticmethod
-    def from_dict(node: str, d: PortConfigDict) -> "PortConfig":
+    @classmethod
+    def from_dict(cls, node: str, d: PortConfigDict) -> Self:
         """A convenience method that creates the object from fewer inputs.
 
         Args:
@@ -167,7 +168,7 @@ def from_dict(node: str, d: PortConfigDict) -> "PortConfig":
         Returns:
             The port configuration instance.
         """
-        return PortConfig(node=node, **d)
+        return cls(node=node, **d)
 
 
 @dataclass(slots=True, frozen=True)
@@ -183,7 +184,7 @@ class TrafficGeneratorConfig:
     traffic_generator_type: TrafficGeneratorType
 
     @staticmethod
-    def from_dict(d: TrafficGeneratorConfigDict) -> "ScapyTrafficGeneratorConfig":
+    def from_dict(d: TrafficGeneratorConfigDict) -> "TrafficGeneratorConfig":
         """A convenience method that produces traffic generator config of the proper type.
 
         Args:
@@ -312,7 +313,7 @@ class TGNodeConfiguration(NodeConfiguration):
         traffic_generator: The configuration of the traffic generator present on the TG node.
     """
 
-    traffic_generator: ScapyTrafficGeneratorConfig
+    traffic_generator: TrafficGeneratorConfig
 
 
 @dataclass(slots=True, frozen=True)
@@ -356,8 +357,8 @@ class BuildTargetConfiguration:
     compiler_wrapper: str
     name: str
 
-    @staticmethod
-    def from_dict(d: BuildTargetConfigDict) -> "BuildTargetConfiguration":
+    @classmethod
+    def from_dict(cls, d: BuildTargetConfigDict) -> Self:
         r"""A convenience method that processes the inputs before creating an instance.
 
         `arch`, `os`, `cpu` and `compiler` are converted to :class:`Enum`\s and
@@ -369,7 +370,7 @@ def from_dict(d: BuildTargetConfigDict) -> "BuildTargetConfiguration":
         Returns:
             The build target configuration instance.
         """
-        return BuildTargetConfiguration(
+        return cls(
             arch=Architecture(d["arch"]),
             os=OS(d["os"]),
             cpu=CPUType(d["cpu"]),
@@ -407,10 +408,11 @@ class TestSuiteConfig:
     test_suite: str
     test_cases: list[str]
 
-    @staticmethod
+    @classmethod
     def from_dict(
+        cls,
         entry: str | TestSuiteConfigDict,
-    ) -> "TestSuiteConfig":
+    ) -> Self:
         """Create an instance from two different types.
 
         Args:
@@ -420,9 +422,9 @@ def from_dict(
             The test suite configuration instance.
         """
         if isinstance(entry, str):
-            return TestSuiteConfig(test_suite=entry, test_cases=[])
+            return cls(test_suite=entry, test_cases=[])
         elif isinstance(entry, dict):
-            return TestSuiteConfig(test_suite=entry["suite"], test_cases=entry["cases"])
+            return cls(test_suite=entry["suite"], test_cases=entry["cases"])
         else:
             raise TypeError(f"{type(entry)} is not valid for a test suite config.")
 
@@ -454,11 +456,12 @@ class ExecutionConfiguration:
     traffic_generator_node: TGNodeConfiguration
     vdevs: list[str]
 
-    @staticmethod
+    @classmethod
     def from_dict(
+        cls,
         d: ExecutionConfigDict,
-        node_map: dict[str, Union[SutNodeConfiguration | TGNodeConfiguration]],
-    ) -> "ExecutionConfiguration":
+        node_map: dict[str, SutNodeConfiguration | TGNodeConfiguration],
+    ) -> Self:
         """A convenience method that processes the inputs before creating an instance.
 
         The build target and the test suite config are transformed into their respective objects.
@@ -494,7 +497,7 @@ def from_dict(
         vdevs = (
             d["system_under_test_node"]["vdevs"] if "vdevs" in d["system_under_test_node"] else []
         )
-        return ExecutionConfiguration(
+        return cls(
             build_targets=build_targets,
             perf=d["perf"],
             func=d["func"],
@@ -505,7 +508,7 @@ def from_dict(
             vdevs=vdevs,
         )
 
-    def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration":
+    def copy_and_modify(self, **kwargs) -> Self:
         """Create a shallow copy with any of the fields modified.
 
         The only new data are those passed to this method.
@@ -525,7 +528,7 @@ def copy_and_modify(self, **kwargs) -> "ExecutionConfiguration":
             else:
                 new_config[field.name] = getattr(self, field.name)
 
-        return ExecutionConfiguration(**new_config)
+        return type(self)(**new_config)
 
 
 @dataclass(slots=True, frozen=True)
@@ -541,8 +544,8 @@ class Configuration:
 
     executions: list[ExecutionConfiguration]
 
-    @staticmethod
-    def from_dict(d: ConfigurationDict) -> "Configuration":
+    @classmethod
+    def from_dict(cls, d: ConfigurationDict) -> Self:
         """A convenience method that processes the inputs before creating an instance.
 
         Build target and test suite config are transformed into their respective objects.
@@ -555,7 +558,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration":
         Returns:
             The whole configuration instance.
         """
-        nodes: list[Union[SutNodeConfiguration | TGNodeConfiguration]] = list(
+        nodes: list[SutNodeConfiguration | TGNodeConfiguration] = list(
             map(NodeConfiguration.from_dict, d["nodes"])
         )
         assert len(nodes) > 0, "There must be a node to test"
@@ -567,7 +570,7 @@ def from_dict(d: ConfigurationDict) -> "Configuration":
             map(ExecutionConfiguration.from_dict, d["executions"], [node_map for _ in d])
         )
 
-        return Configuration(executions=executions)
+        return cls(executions=executions)
 
 
 def load_config(config_file_path: Path) -> Configuration:
diff --git a/dts/framework/testbed_model/traffic_generator/__init__.py b/dts/framework/testbed_model/traffic_generator/__init__.py
index 0eaf0355cd..03e57a77fc 100644
--- a/dts/framework/testbed_model/traffic_generator/__init__.py
+++ b/dts/framework/testbed_model/traffic_generator/__init__.py
@@ -16,7 +16,7 @@
 
 # pylama:ignore=W0611
 
-from framework.config import ScapyTrafficGeneratorConfig, TrafficGeneratorType
+from framework.config import ScapyTrafficGeneratorConfig, TrafficGeneratorConfig
 from framework.exception import ConfigurationError
 from framework.testbed_model.node import Node
 
@@ -28,7 +28,7 @@
 
 
 def create_traffic_generator(
-    tg_node: Node, traffic_generator_config: ScapyTrafficGeneratorConfig
+    tg_node: Node, traffic_generator_config: TrafficGeneratorConfig
 ) -> CapturingTrafficGenerator:
     """The factory function for creating traffic generator objects from the test run configuration.
 
@@ -39,10 +39,10 @@ def create_traffic_generator(
     Returns:
         A traffic generator capable of capturing received packets.
     """
-    match traffic_generator_config.traffic_generator_type:
-        case TrafficGeneratorType.SCAPY:
+    match traffic_generator_config:
+        case ScapyTrafficGeneratorConfig():
             return ScapyTrafficGenerator(tg_node, traffic_generator_config)
         case _:
             raise ConfigurationError(
-                "Unknown traffic generator: {traffic_generator_config.traffic_generator_type}"
+                f"Unknown traffic generator: {traffic_generator_config.traffic_generator_type}"
             )
diff --git a/dts/poetry.lock b/dts/poetry.lock
index df9cecb7e0..5f8fa03933 100644
--- a/dts/poetry.lock
+++ b/dts/poetry.lock
@@ -853,4 +853,4 @@ jsonschema = ">=4,<5"
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.10"
-content-hash = "1572cb14f0bf88cddc6b6b225312ad468d01916b32067d17a7776af76c6d466c"
+content-hash = "4af4dd49c59e5bd6ed99e8c19c6756aaf00125339d26cfad2ef98551dc765f8b"
diff --git a/dts/pyproject.toml b/dts/pyproject.toml
index 05c91ef9be..a160d2fa02 100644
--- a/dts/pyproject.toml
+++ b/dts/pyproject.toml
@@ -26,6 +26,7 @@ types-PyYAML = "^6.0.8"
 fabric = "^2.7.1"
 scapy = "^2.5.0"
 pydocstyle = "6.1.1"
+typing-extensions = "^4.11.0"
 
 [tool.poetry.group.dev.dependencies]
 mypy = "^1.10.0"
-- 
2.34.1

[PATCH v4 3/3] dts: rework arguments framework

[Luca Vizzarro]

The existing argument handling in the code relies on basic argparse
functionality and a custom argparse action to integrate environment
variables. This commit improves the current handling by augmenting
argparse.

This rework implements the following improvements:
- There are duplicate expressions scattered throughout the code. To
  improve readability and maintainability, these are refactored
  into list/dict comprehensions or factory functions.
- Instead of relying solely on argument flags, error messages now
  accurately reference environment variables when applicable, enhancing
  user experience. For instance:

    error: environment variable DTS_DPDK_TARBALL: Invalid file

- Knowing the number of environment variables and arguments set
  allow for a useful help page display when none are provided.
- A display of which environment variables have been detected and their
  corresponding values in the help page, aiding user awareness.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
---
 doc/guides/tools/dts.rst  |  53 +++++---
 dts/framework/settings.py | 280 +++++++++++++++++++++++++++-----------
 2 files changed, 235 insertions(+), 98 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 47b218b2c6..6993443389 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,30 +215,41 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
+   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
+                  [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
 
-   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
+   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
+   line arguments have higher priority.
 
    options:
-   -h, --help            show this help message and exit
-   --config-file CONFIG_FILE
-                         [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
-   --output-dir OUTPUT_DIR, --output OUTPUT_DIR
-                         [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
-   -t TIMEOUT, --timeout TIMEOUT
-                         [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-   -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
-   -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-   --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
-                         [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
-   --compile-timeout COMPILE_TIMEOUT
-                         [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
-   --test-suite TEST_SUITE [TEST_CASES ...]
-                         [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
-                         variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
-                         (default: [])
-   --re-run RE_RUN, --re_run RE_RUN
-                         [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
+     -h, --help            show this help message and exit
+     --config-file FILE_PATH
+                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
+                           (default: conf.yaml)
+     --output-dir DIR_PATH, --output DIR_PATH
+                           [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
+     -t SECONDS, --timeout SECONDS
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
+                           (default: 15)
+     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
+                           (default: False)
+     -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
+     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
+                           test. To test local changes, first commit them, then use the commit ID with this option.
+                           (default: dpdk.tar.xz)
+     --compile-timeout SECONDS
+                           [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
+     --test-suite TEST_SUITE [TEST_CASES ...]
+                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
+                           the test suite name, and the rest are test case names, which are optional. May be specified
+                           multiple times. To specify multiple test suites in the environment variable, join the lists
+                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
+                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
+                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
+     --re-run N_TIMES, --re_run N_TIMES
+                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
+                           (default: 0)
 
 
 The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 688e8679a7..b19f274f9d 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2021 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Environment variables and command line arguments parsing.
 
@@ -72,9 +73,15 @@
 
 import argparse
 import os
+import sys
+from argparse import (
+    Action,
+    ArgumentDefaultsHelpFormatter,
+    _get_action_name,
+)
 from dataclasses import dataclass, field
 from pathlib import Path
-from typing import Any
+from typing import Callable, ParamSpec
 
 from .config import TestSuiteConfig
 from .utils import DPDKGitTarball
@@ -109,104 +116,224 @@ class Settings:
 
 SETTINGS: Settings = Settings()
 
+P = ParamSpec("P")
 
-def _get_parser() -> argparse.ArgumentParser:
-    """Create the argument parser for DTS.
+#: Attribute name representing the env variable name to augment :class:`~argparse.Action` with.
+_ENV_VAR_NAME_ATTR = "env_var_name"
+#: Attribute name representing the value origin to augment :class:`~argparse.Action` with.
+_IS_FROM_ENV_ATTR = "is_from_env"
 
-    Command line options take precedence over environment variables, which in turn take precedence
-    over default values.
+#: The prefix to be added to all of the environment variables.
+ENV_PREFIX = "DTS_"
+
+
+def is_action_in_args(action: Action) -> bool:
+    """Check if the action is invoked in the command line arguments."""
+    for option in action.option_strings:
+        if option in sys.argv:
+            return True
+    return False
+
+
+def make_env_var_name(action: Action, env_var_name: str | None) -> str:
+    """Make and assign an environment variable name to the given action."""
+    env_var_name = f"{ENV_PREFIX}{env_var_name or action.dest.upper()}"
+    setattr(action, _ENV_VAR_NAME_ATTR, env_var_name)
+    return env_var_name
+
+
+def get_env_var_name(action: Action) -> str | None:
+    """Get the environment variable name of the given action."""
+    return getattr(action, _ENV_VAR_NAME_ATTR, None)
+
+
+def set_is_from_env(action: Action) -> None:
+    """Make the environment the given action's value origin."""
+    setattr(action, _IS_FROM_ENV_ATTR, True)
 
-    Returns:
-        argparse.ArgumentParser: The configured argument parser with defined options.
-    """
 
-    def env_arg(env_var: str, default: Any) -> Any:
-        """A helper function augmenting the argparse with environment variables.
+def is_from_env(action: Action) -> bool:
+    """Check if the given action's value originated from the environment."""
+    return getattr(action, _IS_FROM_ENV_ATTR, False)
 
-        If the supplied environment variable is defined, then the default value
-        of the argument is modified. This satisfies the priority order of
-        command line argument > environment variable > default value.
 
-        Args:
-            env_var: Environment variable name.
-            default: Default value.
+def augment_add_argument_with_env(
+    add_argument_fn: Callable[P, Action],
+):
+    """Augment any :class:`~argparse._ActionsContainer.add_argument` with environment variables."""
 
-        Returns:
-            Environment variable or default value.
+    def _add_argument(
+        *args: P.args,
+        env_var_name: str | None = None,
+        **kwargs: P.kwargs,
+    ) -> Action:
+        """Add an argument with an environment variable to the parser."""
+        action = add_argument_fn(*args, **kwargs)
+        env_var_name = make_env_var_name(action, env_var_name)
+
+        if not is_action_in_args(action):
+            env_var_value = os.environ.get(env_var_name)
+            if env_var_value:
+                set_is_from_env(action)
+                sys.argv[1:0] = [action.format_usage(), env_var_value]
+
+        return action
+
+    return _add_argument
+
+
+class ArgumentParser(argparse.ArgumentParser):
+    """ArgumentParser with a custom error message.
+
+    This custom version of ArgumentParser changes the error message to
+    accurately reflect its origin if an environment variable is used
+    as an argument.
+
+    Instead of printing usage on every error, it prints instructions
+    on how to do it.
+    """
+
+    def find_action(
+        self, action_name: str, filter_fn: Callable[[Action], bool] | None = None
+    ) -> Action | None:
+        """Find and return an action by its name.
+
+        Arguments:
+            action_name: the name of the action to find.
+            filter_fn: a filter function to use in the search.
         """
-        return os.environ.get(env_var) or default
+        it = (action for action in self._actions if action_name == _get_action_name(action))
+        action = next(it, None)
+
+        if action is not None and filter_fn is not None:
+            return action if filter_fn(action) else None
+
+        return action
 
-    parser = argparse.ArgumentParser(
+    def error(self, message):
+        """Augments :meth:`~argparse.ArgumentParser.error` with environment variable awareness."""
+        for action in self._actions:
+            if is_from_env(action):
+                action_name = _get_action_name(action)
+                env_var_name = get_env_var_name(action)
+                env_var_value = os.environ.get(env_var_name)
+
+                message = message.replace(
+                    f"argument {action_name}",
+                    f"environment variable {env_var_name} (value: {env_var_value})",
+                )
+
+        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
+        self.exit(2, "For help and usage, " "run the command with the --help flag.\n")
+
+
+class EnvVarHelpFormatter(ArgumentDefaultsHelpFormatter):
+    """Custom formatter to add environment variables in the help page."""
+
+    def _get_help_string(self, action):
+        """Overrides :meth:`ArgumentDefaultsHelpFormatter._get_help_string`."""
+        help = super()._get_help_string(action)
+
+        env_var_name = get_env_var_name(action)
+        if env_var_name is not None:
+            help = f"[{env_var_name}] {help}"
+
+            env_var_value = os.environ.get(env_var_name)
+            if env_var_value is not None:
+                help += f" (env value: {env_var_value})"
+
+        return help
+
+
+def _get_parser() -> ArgumentParser:
+    """Create the argument parser for DTS.
+
+    Command line options take precedence over environment variables, which in turn take precedence
+    over default values.
+
+    Returns:
+        ArgumentParser: The configured argument parser with defined options.
+    """
+    parser = ArgumentParser(
         description="Run DPDK test suites. All options may be specified with the environment "
         "variables provided in brackets. Command line arguments have higher priority.",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+        formatter_class=EnvVarHelpFormatter,
+        allow_abbrev=False,
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env = augment_add_argument_with_env(parser.add_argument)
+
+    add_argument_to_parser_with_env(
         "--config-file",
-        default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
+        default=SETTINGS.config_file_path,
         type=Path,
-        help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
-        "SUTs and targets.",
+        help="The configuration file that describes the test cases, SUTs and targets.",
+        metavar="FILE_PATH",
+        env_var_name="CFG_FILE",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--output-dir",
         "--output",
-        default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
-        help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
+        default=SETTINGS.output_dir,
+        help="Output directory where DTS logs and results are saved.",
+        metavar="DIR_PATH",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-t",
         "--timeout",
-        default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
+        default=SETTINGS.timeout,
         type=float,
-        help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
+        help="The default timeout for all DTS operations except for compiling DPDK.",
+        metavar="SECONDS",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-v",
         "--verbose",
         action="store_true",
-        default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
-        help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
-        "to the console.",
+        default=SETTINGS.verbose,
+        help="Specify to enable verbose output, logging all messages to the console.",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-s",
         "--skip-setup",
         action="store_true",
-        default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
-        help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
+        default=SETTINGS.skip_setup,
+        help="Specify to skip all setup steps on SUT and TG nodes.",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--tarball",
         "--snapshot",
         "--git-ref",
-        default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
+        default=SETTINGS.dpdk_tarball_path,
         type=Path,
-        help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
+        help="Path to DPDK source code tarball or a git commit ID, "
         "tag ID or tree ID to test. To test local changes, first commit them, "
         "then use the commit ID with this option.",
+        metavar="FILE_PATH",
+        dest="dpdk_tarball_path",
+        env_var_name="DPDK_TARBALL",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--compile-timeout",
-        default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
+        default=SETTINGS.compile_timeout,
         type=float,
-        help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
+        help="The timeout for compiling DPDK.",
+        metavar="SECONDS",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--test-suite",
         action="append",
         nargs="+",
         metavar=("TEST_SUITE", "TEST_CASES"),
-        default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
-        help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
+        default=SETTINGS.test_suites,
+        help="A list containing a test suite with test cases. "
         "The first parameter is the test suite name, and the rest are test case names, "
         "which are optional. May be specified multiple times. To specify multiple test suites in "
         "the environment variable, join the lists with a comma. "
@@ -215,21 +342,23 @@ def env_arg(env_var: str, default: Any) -> Any:
         "DTS_TEST_SUITES='suite case case, suite case, ...' | "
         "--test-suite suite --test-suite suite case ... | "
         "DTS_TEST_SUITES='suite, suite case, ...'",
+        dest="test_suites",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--re-run",
         "--re_run",
-        default=env_arg("DTS_RERUN", SETTINGS.re_run),
+        default=SETTINGS.re_run,
         type=int,
-        help="[DTS_RERUN] Re-run each test case the specified number of times "
-        "if a test failure occurs.",
+        help="Re-run each test case the specified number of times if a test failure occurs.",
+        env_var_name="RERUN",
+        metavar="N_TIMES",
     )
 
     return parser
 
 
-def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
+def _process_test_suites(parser: ArgumentParser, args: list[list[str]]) -> list[TestSuiteConfig]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -240,17 +369,12 @@ def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
     Returns:
         A list of test suite configurations to execute.
     """
-    if isinstance(args, str):
-        # Environment variable in the form of "suite case case, suite case, suite, ..."
-        args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
+    test_suites = parser.find_action("test_suites", is_from_env)
+    if test_suites is not None:
+        # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
+        args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    test_suites_to_run = []
-    for suite_with_cases in args:
-        test_suites_to_run.append(
-            TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
-        )
-
-    return test_suites_to_run
+    return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
 
 
 def get_settings() -> Settings:
@@ -261,19 +385,21 @@ def get_settings() -> Settings:
     Returns:
         The new settings object.
     """
-    parsed_args = _get_parser().parse_args()
-    return Settings(
-        config_file_path=parsed_args.config_file,
-        output_dir=parsed_args.output_dir,
-        timeout=parsed_args.timeout,
-        verbose=parsed_args.verbose,
-        skip_setup=parsed_args.skip_setup,
-        dpdk_tarball_path=Path(
-            Path(DPDKGitTarball(parsed_args.tarball, parsed_args.output_dir))
-            if not os.path.exists(parsed_args.tarball)
-            else Path(parsed_args.tarball)
-        ),
-        compile_timeout=parsed_args.compile_timeout,
-        test_suites=_process_test_suites(parsed_args.test_suite),
-        re_run=parsed_args.re_run,
+    parser = _get_parser()
+
+    if len(sys.argv) == 1:
+        parser.print_help()
+        sys.exit(1)
+
+    args = parser.parse_args()
+
+    args.dpdk_tarball_path = Path(
+        Path(DPDKGitTarball(args.dpdk_tarball_path, args.output_dir))
+        if not os.path.exists(args.dpdk_tarball_path)
+        else Path(args.dpdk_tarball_path)
     )
+
+    args.test_suites = _process_test_suites(parser, args.test_suites)
+
+    kwargs = {k: v for k, v in vars(args).items() if hasattr(SETTINGS, k)}
+    return Settings(**kwargs)
-- 
2.34.1

[PATCH v5 0/3] error and usage improvements

[Luca Vizzarro]

Hi again,

apologies again. Looks like it's not my good day as I sent the wrong
patches in the series...

v5:
- re-sent correct patches
v4:
- fix spelling typo
v3:
- amended arguments rework so that it retains the original functional
  style
- re-implemetend functionalities in v2 by augmenting argparse classes
  and functions 
v2:
- complete rework of the arguments handling, to retain
  the environment variables and gain control over them
- prefixing 'Stderr: ' to RemoteCommandExecutionError
- rebased

Luca Vizzarro (3):
  dts: rework arguments framework
  dts: constrain DPDK source argument
  dts: store stderr in RemoteCommandExecutionError

 doc/guides/tools/dts.rst                      |  50 +--
 dts/framework/exception.py                    |  13 +-
 .../remote_session/remote_session.py          |   3 +-
 dts/framework/settings.py                     | 328 +++++++++++++-----
 dts/framework/utils.py                        |  43 ++-
 5 files changed, 309 insertions(+), 128 deletions(-)

-- 
2.34.1

[PATCH v5 1/3] dts: rework arguments framework

[Luca Vizzarro]

The existing argument handling in the code relies on basic argparse
functionality and a custom argparse action to integrate environment
variables. This commit improves the current handling by augmenting
argparse.

This rework implements the following improvements:
- There are duplicate expressions scattered throughout the code. To
  improve readability and maintainability, these are refactored
  into list/dict comprehensions or factory functions.
- Instead of relying solely on argument flags, error messages now
  accurately reference environment variables when applicable, enhancing
  user experience. For instance:

    error: environment variable DTS_DPDK_TARBALL: Invalid file

- Knowing the number of environment variables and arguments set
  allow for a useful help page display when none are provided.
- A display of which environment variables have been detected and their
  corresponding values in the help page, aiding user awareness.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/tools/dts.rst  |  53 +++++---
 dts/framework/settings.py | 280 +++++++++++++++++++++++++++-----------
 2 files changed, 235 insertions(+), 98 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 47b218b2c6..6993443389 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,30 +215,41 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
+   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
+                  [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
 
-   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
+   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
+   line arguments have higher priority.
 
    options:
-   -h, --help            show this help message and exit
-   --config-file CONFIG_FILE
-                         [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
-   --output-dir OUTPUT_DIR, --output OUTPUT_DIR
-                         [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
-   -t TIMEOUT, --timeout TIMEOUT
-                         [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-   -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
-   -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-   --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
-                         [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
-   --compile-timeout COMPILE_TIMEOUT
-                         [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
-   --test-suite TEST_SUITE [TEST_CASES ...]
-                         [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
-                         variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
-                         (default: [])
-   --re-run RE_RUN, --re_run RE_RUN
-                         [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
+     -h, --help            show this help message and exit
+     --config-file FILE_PATH
+                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
+                           (default: conf.yaml)
+     --output-dir DIR_PATH, --output DIR_PATH
+                           [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
+     -t SECONDS, --timeout SECONDS
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
+                           (default: 15)
+     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
+                           (default: False)
+     -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
+     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
+                           test. To test local changes, first commit them, then use the commit ID with this option.
+                           (default: dpdk.tar.xz)
+     --compile-timeout SECONDS
+                           [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
+     --test-suite TEST_SUITE [TEST_CASES ...]
+                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
+                           the test suite name, and the rest are test case names, which are optional. May be specified
+                           multiple times. To specify multiple test suites in the environment variable, join the lists
+                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
+                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
+                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
+     --re-run N_TIMES, --re_run N_TIMES
+                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
+                           (default: 0)
 
 
 The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 688e8679a7..b19f274f9d 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2021 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Environment variables and command line arguments parsing.
 
@@ -72,9 +73,15 @@
 
 import argparse
 import os
+import sys
+from argparse import (
+    Action,
+    ArgumentDefaultsHelpFormatter,
+    _get_action_name,
+)
 from dataclasses import dataclass, field
 from pathlib import Path
-from typing import Any
+from typing import Callable, ParamSpec
 
 from .config import TestSuiteConfig
 from .utils import DPDKGitTarball
@@ -109,104 +116,224 @@ class Settings:
 
 SETTINGS: Settings = Settings()
 
+P = ParamSpec("P")
 
-def _get_parser() -> argparse.ArgumentParser:
-    """Create the argument parser for DTS.
+#: Attribute name representing the env variable name to augment :class:`~argparse.Action` with.
+_ENV_VAR_NAME_ATTR = "env_var_name"
+#: Attribute name representing the value origin to augment :class:`~argparse.Action` with.
+_IS_FROM_ENV_ATTR = "is_from_env"
 
-    Command line options take precedence over environment variables, which in turn take precedence
-    over default values.
+#: The prefix to be added to all of the environment variables.
+ENV_PREFIX = "DTS_"
+
+
+def is_action_in_args(action: Action) -> bool:
+    """Check if the action is invoked in the command line arguments."""
+    for option in action.option_strings:
+        if option in sys.argv:
+            return True
+    return False
+
+
+def make_env_var_name(action: Action, env_var_name: str | None) -> str:
+    """Make and assign an environment variable name to the given action."""
+    env_var_name = f"{ENV_PREFIX}{env_var_name or action.dest.upper()}"
+    setattr(action, _ENV_VAR_NAME_ATTR, env_var_name)
+    return env_var_name
+
+
+def get_env_var_name(action: Action) -> str | None:
+    """Get the environment variable name of the given action."""
+    return getattr(action, _ENV_VAR_NAME_ATTR, None)
+
+
+def set_is_from_env(action: Action) -> None:
+    """Make the environment the given action's value origin."""
+    setattr(action, _IS_FROM_ENV_ATTR, True)
 
-    Returns:
-        argparse.ArgumentParser: The configured argument parser with defined options.
-    """
 
-    def env_arg(env_var: str, default: Any) -> Any:
-        """A helper function augmenting the argparse with environment variables.
+def is_from_env(action: Action) -> bool:
+    """Check if the given action's value originated from the environment."""
+    return getattr(action, _IS_FROM_ENV_ATTR, False)
 
-        If the supplied environment variable is defined, then the default value
-        of the argument is modified. This satisfies the priority order of
-        command line argument > environment variable > default value.
 
-        Args:
-            env_var: Environment variable name.
-            default: Default value.
+def augment_add_argument_with_env(
+    add_argument_fn: Callable[P, Action],
+):
+    """Augment any :class:`~argparse._ActionsContainer.add_argument` with environment variables."""
 
-        Returns:
-            Environment variable or default value.
+    def _add_argument(
+        *args: P.args,
+        env_var_name: str | None = None,
+        **kwargs: P.kwargs,
+    ) -> Action:
+        """Add an argument with an environment variable to the parser."""
+        action = add_argument_fn(*args, **kwargs)
+        env_var_name = make_env_var_name(action, env_var_name)
+
+        if not is_action_in_args(action):
+            env_var_value = os.environ.get(env_var_name)
+            if env_var_value:
+                set_is_from_env(action)
+                sys.argv[1:0] = [action.format_usage(), env_var_value]
+
+        return action
+
+    return _add_argument
+
+
+class ArgumentParser(argparse.ArgumentParser):
+    """ArgumentParser with a custom error message.
+
+    This custom version of ArgumentParser changes the error message to
+    accurately reflect its origin if an environment variable is used
+    as an argument.
+
+    Instead of printing usage on every error, it prints instructions
+    on how to do it.
+    """
+
+    def find_action(
+        self, action_name: str, filter_fn: Callable[[Action], bool] | None = None
+    ) -> Action | None:
+        """Find and return an action by its name.
+
+        Arguments:
+            action_name: the name of the action to find.
+            filter_fn: a filter function to use in the search.
         """
-        return os.environ.get(env_var) or default
+        it = (action for action in self._actions if action_name == _get_action_name(action))
+        action = next(it, None)
+
+        if action is not None and filter_fn is not None:
+            return action if filter_fn(action) else None
+
+        return action
 
-    parser = argparse.ArgumentParser(
+    def error(self, message):
+        """Augments :meth:`~argparse.ArgumentParser.error` with environment variable awareness."""
+        for action in self._actions:
+            if is_from_env(action):
+                action_name = _get_action_name(action)
+                env_var_name = get_env_var_name(action)
+                env_var_value = os.environ.get(env_var_name)
+
+                message = message.replace(
+                    f"argument {action_name}",
+                    f"environment variable {env_var_name} (value: {env_var_value})",
+                )
+
+        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
+        self.exit(2, "For help and usage, " "run the command with the --help flag.\n")
+
+
+class EnvVarHelpFormatter(ArgumentDefaultsHelpFormatter):
+    """Custom formatter to add environment variables in the help page."""
+
+    def _get_help_string(self, action):
+        """Overrides :meth:`ArgumentDefaultsHelpFormatter._get_help_string`."""
+        help = super()._get_help_string(action)
+
+        env_var_name = get_env_var_name(action)
+        if env_var_name is not None:
+            help = f"[{env_var_name}] {help}"
+
+            env_var_value = os.environ.get(env_var_name)
+            if env_var_value is not None:
+                help += f" (env value: {env_var_value})"
+
+        return help
+
+
+def _get_parser() -> ArgumentParser:
+    """Create the argument parser for DTS.
+
+    Command line options take precedence over environment variables, which in turn take precedence
+    over default values.
+
+    Returns:
+        ArgumentParser: The configured argument parser with defined options.
+    """
+    parser = ArgumentParser(
         description="Run DPDK test suites. All options may be specified with the environment "
         "variables provided in brackets. Command line arguments have higher priority.",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+        formatter_class=EnvVarHelpFormatter,
+        allow_abbrev=False,
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env = augment_add_argument_with_env(parser.add_argument)
+
+    add_argument_to_parser_with_env(
         "--config-file",
-        default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
+        default=SETTINGS.config_file_path,
         type=Path,
-        help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
-        "SUTs and targets.",
+        help="The configuration file that describes the test cases, SUTs and targets.",
+        metavar="FILE_PATH",
+        env_var_name="CFG_FILE",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--output-dir",
         "--output",
-        default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
-        help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
+        default=SETTINGS.output_dir,
+        help="Output directory where DTS logs and results are saved.",
+        metavar="DIR_PATH",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-t",
         "--timeout",
-        default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
+        default=SETTINGS.timeout,
         type=float,
-        help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
+        help="The default timeout for all DTS operations except for compiling DPDK.",
+        metavar="SECONDS",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-v",
         "--verbose",
         action="store_true",
-        default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
-        help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
-        "to the console.",
+        default=SETTINGS.verbose,
+        help="Specify to enable verbose output, logging all messages to the console.",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "-s",
         "--skip-setup",
         action="store_true",
-        default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
-        help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
+        default=SETTINGS.skip_setup,
+        help="Specify to skip all setup steps on SUT and TG nodes.",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--tarball",
         "--snapshot",
         "--git-ref",
-        default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
+        default=SETTINGS.dpdk_tarball_path,
         type=Path,
-        help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
+        help="Path to DPDK source code tarball or a git commit ID, "
         "tag ID or tree ID to test. To test local changes, first commit them, "
         "then use the commit ID with this option.",
+        metavar="FILE_PATH",
+        dest="dpdk_tarball_path",
+        env_var_name="DPDK_TARBALL",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--compile-timeout",
-        default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
+        default=SETTINGS.compile_timeout,
         type=float,
-        help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
+        help="The timeout for compiling DPDK.",
+        metavar="SECONDS",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--test-suite",
         action="append",
         nargs="+",
         metavar=("TEST_SUITE", "TEST_CASES"),
-        default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
-        help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
+        default=SETTINGS.test_suites,
+        help="A list containing a test suite with test cases. "
         "The first parameter is the test suite name, and the rest are test case names, "
         "which are optional. May be specified multiple times. To specify multiple test suites in "
         "the environment variable, join the lists with a comma. "
@@ -215,21 +342,23 @@ def env_arg(env_var: str, default: Any) -> Any:
         "DTS_TEST_SUITES='suite case case, suite case, ...' | "
         "--test-suite suite --test-suite suite case ... | "
         "DTS_TEST_SUITES='suite, suite case, ...'",
+        dest="test_suites",
     )
 
-    parser.add_argument(
+    add_argument_to_parser_with_env(
         "--re-run",
         "--re_run",
-        default=env_arg("DTS_RERUN", SETTINGS.re_run),
+        default=SETTINGS.re_run,
         type=int,
-        help="[DTS_RERUN] Re-run each test case the specified number of times "
-        "if a test failure occurs.",
+        help="Re-run each test case the specified number of times if a test failure occurs.",
+        env_var_name="RERUN",
+        metavar="N_TIMES",
     )
 
     return parser
 
 
-def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
+def _process_test_suites(parser: ArgumentParser, args: list[list[str]]) -> list[TestSuiteConfig]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -240,17 +369,12 @@ def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
     Returns:
         A list of test suite configurations to execute.
     """
-    if isinstance(args, str):
-        # Environment variable in the form of "suite case case, suite case, suite, ..."
-        args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
+    test_suites = parser.find_action("test_suites", is_from_env)
+    if test_suites is not None:
+        # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
+        args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    test_suites_to_run = []
-    for suite_with_cases in args:
-        test_suites_to_run.append(
-            TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
-        )
-
-    return test_suites_to_run
+    return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
 
 
 def get_settings() -> Settings:
@@ -261,19 +385,21 @@ def get_settings() -> Settings:
     Returns:
         The new settings object.
     """
-    parsed_args = _get_parser().parse_args()
-    return Settings(
-        config_file_path=parsed_args.config_file,
-        output_dir=parsed_args.output_dir,
-        timeout=parsed_args.timeout,
-        verbose=parsed_args.verbose,
-        skip_setup=parsed_args.skip_setup,
-        dpdk_tarball_path=Path(
-            Path(DPDKGitTarball(parsed_args.tarball, parsed_args.output_dir))
-            if not os.path.exists(parsed_args.tarball)
-            else Path(parsed_args.tarball)
-        ),
-        compile_timeout=parsed_args.compile_timeout,
-        test_suites=_process_test_suites(parsed_args.test_suite),
-        re_run=parsed_args.re_run,
+    parser = _get_parser()
+
+    if len(sys.argv) == 1:
+        parser.print_help()
+        sys.exit(1)
+
+    args = parser.parse_args()
+
+    args.dpdk_tarball_path = Path(
+        Path(DPDKGitTarball(args.dpdk_tarball_path, args.output_dir))
+        if not os.path.exists(args.dpdk_tarball_path)
+        else Path(args.dpdk_tarball_path)
     )
+
+    args.test_suites = _process_test_suites(parser, args.test_suites)
+
+    kwargs = {k: v for k, v in vars(args).items() if hasattr(SETTINGS, k)}
+    return Settings(**kwargs)
-- 
2.34.1

Re: [PATCH v5 1/3] dts: rework arguments framework

[Juraj Linkeš]

The overall approach looks solid. Maybe we can do some minor
improvements on it, but it's honestly fine the way it's now.

There is a difference in behavior when I pass no arguments and then I
either have or don't have an env var set:
./main.py
usage: main.py [-h] [--config-file FILE_PATH] ...
...

-----------------------------
DTS_SKIP_SETUP=Y ./main.py
main.py: error: one of the arguments --tarball/--snapshot
--revision/--rev/--git-ref is required

For help and usage, run the command with the --help flag.

I think this is because we're adding the env vars as actions (the
second scenario thus has at least one) and the first scenario doesn't
have any actions, which leads to the different output. I don't know
whether we want to unify these, but it's a bit confusing, as the error
applies to both cases.

<snip>

> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> index 688e8679a7..b19f274f9d 100644
> --- a/dts/framework/settings.py
> +++ b/dts/framework/settings.py

> @@ -109,104 +116,224 @@ class Settings:
>
>  SETTINGS: Settings = Settings()
>
> +P = ParamSpec("P")
>
> -def _get_parser() -> argparse.ArgumentParser:
> -    """Create the argument parser for DTS.
> +#: Attribute name representing the env variable name to augment :class:`~argparse.Action` with.
> +_ENV_VAR_NAME_ATTR = "env_var_name"
> +#: Attribute name representing the value origin to augment :class:`~argparse.Action` with.
> +_IS_FROM_ENV_ATTR = "is_from_env"
>
> -    Command line options take precedence over environment variables, which in turn take precedence
> -    over default values.
> +#: The prefix to be added to all of the environment variables.
> +ENV_PREFIX = "DTS_"
> +
> +
> +def is_action_in_args(action: Action) -> bool:

I don't think there's any expectation that these functions will be
used outside this module, so I'd make them all private, in which case
the short docstring would be fine.

The ParamSpec also maybe should be private. Same goes for ENV_PREFIX.

I'm also looking at the order of the various functions, classes and
variables in the module and it looks all over the place. Maybe we can
tidy it up a bit.

> +    """Check if the action is invoked in the command line arguments."""
> +    for option in action.option_strings:
> +        if option in sys.argv:
> +            return True
> +    return False
> +
> +
> +def make_env_var_name(action: Action, env_var_name: str | None) -> str:
> +    """Make and assign an environment variable name to the given action."""
> +    env_var_name = f"{ENV_PREFIX}{env_var_name or action.dest.upper()}"
> +    setattr(action, _ENV_VAR_NAME_ATTR, env_var_name)
> +    return env_var_name
> +
> +
> +def get_env_var_name(action: Action) -> str | None:
> +    """Get the environment variable name of the given action."""
> +    return getattr(action, _ENV_VAR_NAME_ATTR, None)
> +
> +
> +def set_is_from_env(action: Action) -> None:
> +    """Make the environment the given action's value origin."""
> +    setattr(action, _IS_FROM_ENV_ATTR, True)
>
> -    Returns:
> -        argparse.ArgumentParser: The configured argument parser with defined options.
> -    """
>
> -    def env_arg(env_var: str, default: Any) -> Any:
> -        """A helper function augmenting the argparse with environment variables.
> +def is_from_env(action: Action) -> bool:
> +    """Check if the given action's value originated from the environment."""
> +    return getattr(action, _IS_FROM_ENV_ATTR, False)
>
> -        If the supplied environment variable is defined, then the default value
> -        of the argument is modified. This satisfies the priority order of
> -        command line argument > environment variable > default value.
>
> -        Args:
> -            env_var: Environment variable name.
> -            default: Default value.
> +def augment_add_argument_with_env(
> +    add_argument_fn: Callable[P, Action],
> +):
> +    """Augment any :class:`~argparse._ActionsContainer.add_argument` with environment variables."""
>
> -        Returns:
> -            Environment variable or default value.
> +    def _add_argument(
> +        *args: P.args,
> +        env_var_name: str | None = None,
> +        **kwargs: P.kwargs,
> +    ) -> Action:
> +        """Add an argument with an environment variable to the parser."""
> +        action = add_argument_fn(*args, **kwargs)
> +        env_var_name = make_env_var_name(action, env_var_name)
> +
> +        if not is_action_in_args(action):
> +            env_var_value = os.environ.get(env_var_name)
> +            if env_var_value:
> +                set_is_from_env(action)
> +                sys.argv[1:0] = [action.format_usage(), env_var_value]
> +
> +        return action
> +
> +    return _add_argument
> +
> +
> +class ArgumentParser(argparse.ArgumentParser):

I'd rename this to DTSArgumentParser and maybe also make the classes
(this one and the formatter) private.

> +    """ArgumentParser with a custom error message.
> +
> +    This custom version of ArgumentParser changes the error message to
> +    accurately reflect its origin if an environment variable is used
> +    as an argument.
> +
> +    Instead of printing usage on every error, it prints instructions
> +    on how to do it.

This sentence is confusing - how to do what?

> +    """
> +
> +    def find_action(
> +        self, action_name: str, filter_fn: Callable[[Action], bool] | None = None
> +    ) -> Action | None:
> +        """Find and return an action by its name.
> +
> +        Arguments:
> +            action_name: the name of the action to find.
> +            filter_fn: a filter function to use in the search.

It's not very clear from the description how this filter is applied.
We should mention that the found action (and not action_name) is going
to be passed to filter_fn.

>          """
> -        return os.environ.get(env_var) or default
> +        it = (action for action in self._actions if action_name == _get_action_name(action))
> +        action = next(it, None)
> +
> +        if action is not None and filter_fn is not None:

Would a simplified condition "if action and filter_fn" work?

> +            return action if filter_fn(action) else None
> +
> +        return action
>
> -    parser = argparse.ArgumentParser(
> +    def error(self, message):
> +        """Augments :meth:`~argparse.ArgumentParser.error` with environment variable awareness."""
> +        for action in self._actions:
> +            if is_from_env(action):
> +                action_name = _get_action_name(action)
> +                env_var_name = get_env_var_name(action)
> +                env_var_value = os.environ.get(env_var_name)
> +
> +                message = message.replace(
> +                    f"argument {action_name}",
> +                    f"environment variable {env_var_name} (value: {env_var_value})",
> +                )
> +
> +        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
> +        self.exit(2, "For help and usage, " "run the command with the --help flag.\n")
> +
> +
> +class EnvVarHelpFormatter(ArgumentDefaultsHelpFormatter):
> +    """Custom formatter to add environment variables in the help page."""

to the help page

> +
> +    def _get_help_string(self, action):
> +        """Overrides :meth:`ArgumentDefaultsHelpFormatter._get_help_string`."""
> +        help = super()._get_help_string(action)
> +
> +        env_var_name = get_env_var_name(action)
> +        if env_var_name is not None:
> +            help = f"[{env_var_name}] {help}"
> +
> +            env_var_value = os.environ.get(env_var_name)
> +            if env_var_value is not None:
> +                help += f" (env value: {env_var_value})"

Let's do this the same way as four lines above: help = f"{help} (env
value: {env_var_value})"

> +
> +        return help
> +
> +
> +def _get_parser() -> ArgumentParser:
> +    """Create the argument parser for DTS.
> +
> +    Command line options take precedence over environment variables, which in turn take precedence
> +    over default values.
> +
> +    Returns:
> +        ArgumentParser: The configured argument parser with defined options.
> +    """
> +    parser = ArgumentParser(
>          description="Run DPDK test suites. All options may be specified with the environment "
>          "variables provided in brackets. Command line arguments have higher priority.",
> -        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
> +        formatter_class=EnvVarHelpFormatter,
> +        allow_abbrev=False,
>      )
>
> -    parser.add_argument(
> +    add_argument_to_parser_with_env = augment_add_argument_with_env(parser.add_argument)

I'm wondering whether we could modify the actual add_argument methods
of ArgumentParser and _MutuallyExclusiveGroup, either the class
methods or instance methods. Have you tried that? It could be easier
to understand.

Or, alternatively, we could do:

action = parser.add_argument(
    "--config-file",
    default=SETTINGS.config_file_path,
    type=Path,
    help="The configuration file that describes the test cases, SUTs
and targets.",
    metavar="FILE_PATH",
)
add_env_var_to_action(action, env_var_name="CFG_FILE")

This makes what we're trying to do a bit clearer, but requires two
calls instead of one, so maybe it's not better. I'm not sure.

> +
> +    add_argument_to_parser_with_env(
>          "--config-file",

We should rename Settings.config_file_path to correspond with this.
Otherwise it's going to be ignored by get_settings().

> -        default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
> +        default=SETTINGS.config_file_path,
>          type=Path,
> -        help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
> -        "SUTs and targets.",
> +        help="The configuration file that describes the test cases, SUTs and targets.",
> +        metavar="FILE_PATH",
> +        env_var_name="CFG_FILE",
>      )
>

> @@ -240,17 +369,12 @@ def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
>      Returns:
>          A list of test suite configurations to execute.
>      """
> -    if isinstance(args, str):
> -        # Environment variable in the form of "suite case case, suite case, suite, ..."
> -        args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
> +    test_suites = parser.find_action("test_suites", is_from_env)
> +    if test_suites is not None:
> +        # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
> +        args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
>
> -    test_suites_to_run = []
> -    for suite_with_cases in args:
> -        test_suites_to_run.append(
> -            TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
> -        )
> -
> -    return test_suites_to_run
> +    return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]

This doesn't work properly, if I use:
DTS_TEST_SUITES="hello_world test_hello_world_single_core, os_udp test_os_udp"
I'm getting:
execution_setup - dts - ERROR - Invalid test suite configuration
found: [TestSuiteConfig(test_suite='hello_world
test_hello_world_single_core, os_udp test_os_udp', test_cases=[])].
...
ModuleNotFoundError: No module named 'tests.TestSuite_hello_world
test_hello_world_single_core, os_udp test_os_udp'

Re: [PATCH v5 1/3] dts: rework arguments framework

[Luca Vizzarro]

On 30/05/2024 16:30, Juraj Linkeš wrote:
> There is a difference in behavior when I pass no arguments and then I
> either have or don't have an env var set:
> ./main.py
> usage: main.py [-h] [--config-file FILE_PATH] ...
> ...
> 
> -----------------------------
> DTS_SKIP_SETUP=Y ./main.py
> main.py: error: one of the arguments --tarball/--snapshot
> --revision/--rev/--git-ref is required
> 
> For help and usage, run the command with the --help flag.
> 
> I think this is because we're adding the env vars as actions (the
> second scenario thus has at least one) and the first scenario doesn't
> have any actions, which leads to the different output. I don't know
> whether we want to unify these, but it's a bit confusing, as the error
> applies to both cases.

The behaviour is just as expected. You get the same exact results if you 
were to run:
   ./main.py -s
which is equivalent to:
   DTS_SKIP_SETUP= ./main.py

The true equivalent to your example would be:
   ./main.py -s Y

The only anomaly is that no matter the value passed (true, false, y, 
n...) it's always going to be set because the action is `store_true`.
If we care to interpret a possible value then we need to implement a new 
action to cater for this.

>> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
>> index 688e8679a7..b19f274f9d 100644
>> --- a/dts/framework/settings.py
>> +++ b/dts/framework/settings.py
> 
>> @@ -109,104 +116,224 @@ class Settings:
>>
>>   SETTINGS: Settings = Settings()
>>
>> +P = ParamSpec("P")
>>
>> -def _get_parser() -> argparse.ArgumentParser:
>> -    """Create the argument parser for DTS.
>> +#: Attribute name representing the env variable name to augment :class:`~argparse.Action` with.
>> +_ENV_VAR_NAME_ATTR = "env_var_name"
>> +#: Attribute name representing the value origin to augment :class:`~argparse.Action` with.
>> +_IS_FROM_ENV_ATTR = "is_from_env"
>>
>> -    Command line options take precedence over environment variables, which in turn take precedence
>> -    over default values.
>> +#: The prefix to be added to all of the environment variables.
>> +ENV_PREFIX = "DTS_"
>> +
>> +
>> +def is_action_in_args(action: Action) -> bool:
> 
> I don't think there's any expectation that these functions will be
> used outside this module, so I'd make them all private, in which case
> the short docstring would be fine.
> 
> The ParamSpec also maybe should be private. Same goes for ENV_PREFIX.

Ack.

> I'm also looking at the order of the various functions, classes and
> variables in the module and it looks all over the place. Maybe we can
> tidy it up a bit.

Could you please elaborate on this? It is also important to define 
what's the ordering expectation. At the moment they are mostly in order 
of usage, except for the Settings which I purposefully left on top as 
they are easier to find and probably the most "needed" for a user. As 
for the env var helper functions, most of them are used rightaway, but I 
left a couple that are used later there so that they are all grouped 
together (and related).

>> +class ArgumentParser(argparse.ArgumentParser):
> 
> I'd rename this to DTSArgumentParser and maybe also make the classes
> (this one and the formatter) private.
> 

Ack.
>> +    Instead of printing usage on every error, it prints instructions
>> +    on how to do it.
> 
> This sentence is confusing - how to do what?

This could do without the sentence to be honest, I'll just remove it.

>> +    def find_action(
>> +        self, action_name: str, filter_fn: Callable[[Action], bool] | None = None
>> +    ) -> Action | None:
>> +        """Find and return an action by its name.
>> +
>> +        Arguments:
>> +            action_name: the name of the action to find.
>> +            filter_fn: a filter function to use in the search.
> 
> It's not very clear from the description how this filter is applied.
> We should mention that the found action (and not action_name) is going
> to be passed to filter_fn.

Ack.

>> -        return os.environ.get(env_var) or default
>> +        it = (action for action in self._actions if action_name == _get_action_name(action))
>> +        action = next(it, None)
>> +
>> +        if action is not None and filter_fn is not None:
> 
> Would a simplified condition "if action and filter_fn" work?

Should actually work in this case. Will change it.


>> +class EnvVarHelpFormatter(ArgumentDefaultsHelpFormatter):
>> +    """Custom formatter to add environment variables in the help page."""
> 
> to the help page

Ack.

>> +        if env_var_name is not None:
>> +            help = f"[{env_var_name}] {help}"
>> +
>> +            env_var_value = os.environ.get(env_var_name)
>> +            if env_var_value is not None:
>> +                help += f" (env value: {env_var_value})"
> 
> Let's do this the same way as four lines above: help = f"{help} (env
> value: {env_var_value})"

Ack.

>> -    parser.add_argument(
>> +    add_argument_to_parser_with_env = augment_add_argument_with_env(parser.add_argument)
> 
> I'm wondering whether we could modify the actual add_argument methods
> of ArgumentParser and _MutuallyExclusiveGroup, either the class
> methods or instance methods. Have you tried that? It could be easier
> to understand.

I tried this, but it becomes overly complicated because add_argument is 
implemented in _ActionsContainer which in turn is inherited by several 
classes, which we ultimately use. The easiest and clearest approach is 
just a wrapper.

> Or, alternatively, we could do:
> 
> action = parser.add_argument(
>      "--config-file",
>      default=SETTINGS.config_file_path,
>      type=Path,
>      help="The configuration file that describes the test cases, SUTs
> and targets.",
>      metavar="FILE_PATH",
> )
> add_env_var_to_action(action, env_var_name="CFG_FILE")
> 
> This makes what we're trying to do a bit clearer, but requires two
> calls instead of one, so maybe it's not better. I'm not sure.

I can drop the HOF and use it as `_add_env_var_to_action` as a clearer 
replacement.

>> +
>> +    add_argument_to_parser_with_env(
>>           "--config-file",
> 
> We should rename Settings.config_file_path to correspond with this.
> Otherwise it's going to be ignored by get_settings().

This was actually a miss from me. Nice catch! I've solved this issue by 
adding `dest` to the other arguments, but clearly forgot about this one.

>> @@ -240,17 +369,12 @@ def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
>>       Returns:
>>           A list of test suite configurations to execute.
>>       """
>> -    if isinstance(args, str):
>> -        # Environment variable in the form of "suite case case, suite case, suite, ..."
>> -        args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
>> +    test_suites = parser.find_action("test_suites", is_from_env)
>> +    if test_suites is not None:
>> +        # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
>> +        args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
>>
>> -    test_suites_to_run = []
>> -    for suite_with_cases in args:
>> -        test_suites_to_run.append(
>> -            TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
>> -        )
>> -
>> -    return test_suites_to_run
>> +    return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
> 
> This doesn't work properly, if I use:
> DTS_TEST_SUITES="hello_world test_hello_world_single_core, os_udp test_os_udp"
> I'm getting:
> execution_setup - dts - ERROR - Invalid test suite configuration
> found: [TestSuiteConfig(test_suite='hello_world
> test_hello_world_single_core, os_udp test_os_udp', test_cases=[])].
> ...
> ModuleNotFoundError: No module named 'tests.TestSuite_hello_world
> test_hello_world_single_core, os_udp test_os_udp'

yes, nice catch! Must have forgot to test this bit again. The culprit is 
find_action as it's meant to find by dest and not action name.

Re: [PATCH v5 1/3] dts: rework arguments framework

[Juraj Linkeš]

On Thu, May 30, 2024 at 8:43 PM Luca Vizzarro <Luca.Vizzarro@arm.com> wrote:
>
> On 30/05/2024 16:30, Juraj Linkeš wrote:
> > There is a difference in behavior when I pass no arguments and then I
> > either have or don't have an env var set:
> > ./main.py
> > usage: main.py [-h] [--config-file FILE_PATH] ...
> > ...
> >
> > -----------------------------
> > DTS_SKIP_SETUP=Y ./main.py
> > main.py: error: one of the arguments --tarball/--snapshot
> > --revision/--rev/--git-ref is required
> >
> > For help and usage, run the command with the --help flag.
> >
> > I think this is because we're adding the env vars as actions (the
> > second scenario thus has at least one) and the first scenario doesn't
> > have any actions, which leads to the different output. I don't know
> > whether we want to unify these, but it's a bit confusing, as the error
> > applies to both cases.
>
> The behaviour is just as expected. You get the same exact results if you
> were to run:
>    ./main.py -s
> which is equivalent to:
>    DTS_SKIP_SETUP= ./main.py
>
> The true equivalent to your example would be:
>    ./main.py -s Y
>

Oh, so this is expected. I'm fine with this, but even though it's
equivalent, it doesn't look that way (at first glance), especially
when used with export:
export DTS_SKIP_SETUP=Y
... # other commands
./main.py -s

But I don't think that's a problem, so let's leave it as is.

> The only anomaly is that no matter the value passed (true, false, y,
> n...) it's always going to be set because the action is `store_true`.
> If we care to interpret a possible value then we need to implement a new
> action to cater for this.
>

The only interpretation we want to do is whether the variable is set
or not and it's documented as such (the docs say "set to any value"),
so there's not much of a point to interpret it. And this way it
mirrors the cmdline args (it either is there or isn't).

> >> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> >> index 688e8679a7..b19f274f9d 100644
> >> --- a/dts/framework/settings.py
> >> +++ b/dts/framework/settings.py
> >
> >> @@ -109,104 +116,224 @@ class Settings:
> >>
> >>   SETTINGS: Settings = Settings()
> >>
> >> +P = ParamSpec("P")
> >>
> >> -def _get_parser() -> argparse.ArgumentParser:
> >> -    """Create the argument parser for DTS.
> >> +#: Attribute name representing the env variable name to augment :class:`~argparse.Action` with.
> >> +_ENV_VAR_NAME_ATTR = "env_var_name"
> >> +#: Attribute name representing the value origin to augment :class:`~argparse.Action` with.
> >> +_IS_FROM_ENV_ATTR = "is_from_env"
> >>
> >> -    Command line options take precedence over environment variables, which in turn take precedence
> >> -    over default values.
> >> +#: The prefix to be added to all of the environment variables.
> >> +ENV_PREFIX = "DTS_"
> >> +
> >> +
> >> +def is_action_in_args(action: Action) -> bool:
> >
> > I don't think there's any expectation that these functions will be
> > used outside this module, so I'd make them all private, in which case
> > the short docstring would be fine.
> >
> > The ParamSpec also maybe should be private. Same goes for ENV_PREFIX.
>
> Ack.
>
> > I'm also looking at the order of the various functions, classes and
> > variables in the module and it looks all over the place. Maybe we can
> > tidy it up a bit.
>
> Could you please elaborate on this? It is also important to define
> what's the ordering expectation. At the moment they are mostly in order
> of usage, except for the Settings which I purposefully left on top as
> they are easier to find and probably the most "needed" for a user. As
> for the env var helper functions, most of them are used rightaway, but I
> left a couple that are used later there so that they are all grouped
> together (and related).
>

I just wanted to make sure there's some logic to it and the way you
described it sounds good.
Also, I was looking at the file with all of the patches applied, so
that made it look more jumbled than it is. Without the two functions
added later (or with them in the right place), the order actually
looks fine.

> >> +class ArgumentParser(argparse.ArgumentParser):
> >
> > I'd rename this to DTSArgumentParser and maybe also make the classes
> > (this one and the formatter) private.
> >
>
> Ack.
> >> +    Instead of printing usage on every error, it prints instructions
> >> +    on how to do it.
> >
> > This sentence is confusing - how to do what?
>
> This could do without the sentence to be honest, I'll just remove it.
>

Ok.

<snip>

> >> -    parser.add_argument(
> >> +    add_argument_to_parser_with_env = augment_add_argument_with_env(parser.add_argument)
> >
> > I'm wondering whether we could modify the actual add_argument methods
> > of ArgumentParser and _MutuallyExclusiveGroup, either the class
> > methods or instance methods. Have you tried that? It could be easier
> > to understand.
>
> I tried this, but it becomes overly complicated because add_argument is
> implemented in _ActionsContainer which in turn is inherited by several
> classes, which we ultimately use. The easiest and clearest approach is
> just a wrapper.
>

Ok, let's go with the wrapper then.

> > Or, alternatively, we could do:
> >
> > action = parser.add_argument(
> >      "--config-file",
> >      default=SETTINGS.config_file_path,
> >      type=Path,
> >      help="The configuration file that describes the test cases, SUTs
> > and targets.",
> >      metavar="FILE_PATH",
> > )
> > add_env_var_to_action(action, env_var_name="CFG_FILE")
> >
> > This makes what we're trying to do a bit clearer, but requires two
> > calls instead of one, so maybe it's not better. I'm not sure.
>
> I can drop the HOF and use it as `_add_env_var_to_action` as a clearer
> replacement.
>

Let's try it.

[PATCH v5 2/3] dts: constrain DPDK source argument

[Luca Vizzarro]

DTS needs an input to gather the DPDK source code from. This is then
built on the remote target. This commit makes sure that this input is
more constrained, separating the Git revision ID – used to create a
tarball using Git – and providing tarballed source code directly, while
retaining mutual exclusion.

This makes the code more readable and easier to handle for input
validation, of which this commit introduces a basic one based on the
pre-existing code.

Moreover it ensures that these flags are explicitly required to be set
by the user, dropping a default value.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/tools/dts.rst  | 39 ++++++++++++------------
 dts/framework/settings.py | 62 +++++++++++++++++++++++++++++----------
 dts/framework/utils.py    | 43 ++++++++++++++++-----------
 3 files changed, 90 insertions(+), 54 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 6993443389..f64ab7f732 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,41 +215,38 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
+   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] (--tarball FILE_PATH | --revision ID)
                   [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
 
-   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
-   line arguments have higher priority.
+   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher
+   priority.
 
    options:
      -h, --help            show this help message and exit
      --config-file FILE_PATH
-                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
-                           (default: conf.yaml)
+                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default:
+                           /home/lucviz01/dpdk/dts/conf.yaml)
      --output-dir DIR_PATH, --output DIR_PATH
                            [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
      -t SECONDS, --timeout SECONDS
-                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
-                           (default: 15)
-     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
-                           (default: False)
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
+     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
      -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
-                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
-                           test. To test local changes, first commit them, then use the commit ID with this option.
-                           (default: dpdk.tar.xz)
+     --tarball FILE_PATH, --snapshot FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball to test. (default: None)
+     --revision ID, --rev ID, --git-ref ID
+                           [DTS_DPDK_REVISION_ID] Git revision ID to test. Could be commit, tag, tree ID etc. To test local changes, first
+                           commit them, then use their commit ID. (default: None)
      --compile-timeout SECONDS
                            [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
      --test-suite TEST_SUITE [TEST_CASES ...]
-                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
-                           the test suite name, and the rest are test case names, which are optional. May be specified
-                           multiple times. To specify multiple test suites in the environment variable, join the lists
-                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
-                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
-                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
+                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and
+                           the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites
+                           in the environment variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite
+                           suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case
+                           ... | DTS_TEST_SUITES='suite, suite case, ...' (default: [])
      --re-run N_TIMES, --re_run N_TIMES
-                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
-                           (default: 0)
+                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
 
 
 The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index b19f274f9d..50d8929450 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -44,10 +44,16 @@
 
     Set to any value to skip building DPDK.
 
-.. option:: --tarball, --snapshot, --git-ref
+.. option:: --tarball, --snapshot
 .. envvar:: DTS_DPDK_TARBALL
 
-    The path to a DPDK tarball, git commit ID, tag ID or tree ID to test.
+    Path to DPDK source code tarball to test.
+
+.. option:: --revision, --rev, --git-ref
+.. envvar:: DTS_DPDK_REVISION_ID
+
+    Git revision ID to test. Could be commit, tag, tree ID etc.
+    To test local changes, first commit them, then use their commit ID.
 
 .. option:: --test-suite
 .. envvar:: DTS_TEST_SUITES
@@ -84,7 +90,24 @@
 from typing import Callable, ParamSpec
 
 from .config import TestSuiteConfig
-from .utils import DPDKGitTarball
+from .exception import ConfigurationError
+from .utils import DPDKGitTarball, get_commit_id
+
+
+def _parse_tarball_path(file_path: str) -> Path:
+    """Validate whether `file_path` is valid and return a Path object."""
+    path = Path(file_path)
+    if not path.exists() or not path.is_file():
+        raise argparse.ArgumentTypeError("The file path provided is not a valid file")
+    return path
+
+
+def _parse_revision_id(rev_id: str) -> str:
+    """Validate revision ID and retrieve corresponding commit ID."""
+    try:
+        return get_commit_id(rev_id)
+    except ConfigurationError:
+        raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")
 
 
 @dataclass(slots=True)
@@ -105,7 +128,7 @@ class Settings:
     #:
     skip_setup: bool = False
     #:
-    dpdk_tarball_path: Path | str = "dpdk.tar.xz"
+    dpdk_tarball_path: Path | str = ""
     #:
     compile_timeout: float = 1200
     #:
@@ -305,20 +328,30 @@ def _get_parser() -> ArgumentParser:
         help="Specify to skip all setup steps on SUT and TG nodes.",
     )
 
-    add_argument_to_parser_with_env(
+    dpdk_source = parser.add_mutually_exclusive_group(required=True)
+    add_argument_to_dpdk_source_with_env = augment_add_argument_with_env(dpdk_source.add_argument)
+
+    add_argument_to_dpdk_source_with_env(
         "--tarball",
         "--snapshot",
-        "--git-ref",
-        default=SETTINGS.dpdk_tarball_path,
-        type=Path,
-        help="Path to DPDK source code tarball or a git commit ID, "
-        "tag ID or tree ID to test. To test local changes, first commit them, "
-        "then use the commit ID with this option.",
+        type=_parse_tarball_path,
+        help="Path to DPDK source code tarball to test.",
         metavar="FILE_PATH",
         dest="dpdk_tarball_path",
         env_var_name="DPDK_TARBALL",
     )
 
+    add_argument_to_dpdk_source_with_env(
+        "--revision",
+        "--rev",
+        "--git-ref",
+        type=_parse_revision_id,
+        help="Git revision ID to test. Could be commit, tag, tree ID etc. "
+        "To test local changes, first commit them, then use their commit ID.",
+        metavar="ID",
+        dest="dpdk_revision_id",
+    ),
+
     add_argument_to_parser_with_env(
         "--compile-timeout",
         default=SETTINGS.compile_timeout,
@@ -393,11 +426,8 @@ def get_settings() -> Settings:
 
     args = parser.parse_args()
 
-    args.dpdk_tarball_path = Path(
-        Path(DPDKGitTarball(args.dpdk_tarball_path, args.output_dir))
-        if not os.path.exists(args.dpdk_tarball_path)
-        else Path(args.dpdk_tarball_path)
-    )
+    if args.dpdk_revision_id:
+        args.dpdk_tarball_path = DPDKGitTarball(args.dpdk_revision_id, args.output_dir)
 
     args.test_suites = _process_test_suites(parser, args.test_suites)
 
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index 74a11f1aaf..862bafb46c 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Various utility classes and functions.
 
@@ -70,6 +71,31 @@ def get_packet_summaries(packets: list[Packet]) -> str:
     return f"Packet contents: \n{packet_summaries}"
 
 
+def get_commit_id(rev_id: str) -> str:
+    """Given a Git revision ID, return the corresponding commit ID.
+
+    Args:
+        rev_id: The Git revision ID.
+
+    Raises:
+        ConfigurationError: The ``git rev-parse`` command failed, suggesting
+            an invalid or ambiguous revision ID was supplied.
+    """
+    result = subprocess.run(
+        ["git", "rev-parse", "--verify", rev_id],
+        text=True,
+        capture_output=True,
+    )
+    if result.returncode != 0:
+        raise ConfigurationError(
+            f"{rev_id} is not a valid git reference.\n"
+            f"Command: {result.args}\n"
+            f"Stdout: {result.stdout}\n"
+            f"Stderr: {result.stderr}"
+        )
+    return result.stdout.strip()
+
+
 class StrEnum(Enum):
     """Enum with members stored as strings."""
 
@@ -170,7 +196,6 @@ def __init__(
 
         self._tarball_dir = Path(output_dir, "tarball")
 
-        self._get_commit_id()
         self._create_tarball_dir()
 
         self._tarball_name = (
@@ -180,22 +205,6 @@ def __init__(
         if not self._tarball_path:
             self._create_tarball()
 
-    def _get_commit_id(self) -> None:
-        result = subprocess.run(
-            ["git", "rev-parse", "--verify", self._git_ref],
-            text=True,
-            capture_output=True,
-        )
-        if result.returncode != 0:
-            raise ConfigurationError(
-                f"{self._git_ref} is neither a path to an existing DPDK "
-                "archive nor a valid git reference.\n"
-                f"Command: {result.args}\n"
-                f"Stdout: {result.stdout}\n"
-                f"Stderr: {result.stderr}"
-            )
-        self._git_ref = result.stdout.strip()
-
     def _create_tarball_dir(self) -> None:
         os.makedirs(self._tarball_dir, exist_ok=True)
 
-- 
2.34.1

Re: [PATCH v5 2/3] dts: constrain DPDK source argument

[Juraj Linkeš]

> diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
> index 6993443389..f64ab7f732 100644
> --- a/doc/guides/tools/dts.rst
> +++ b/doc/guides/tools/dts.rst
> @@ -215,41 +215,38 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
>  .. code-block:: console
>
>     (dts-py3.10) $ ./main.py --help
> -   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
> +   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] (--tarball FILE_PATH | --revision ID)
>                    [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
>
> -   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
> -   line arguments have higher priority.
> +   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher
> +   priority.
>
>     options:
>       -h, --help            show this help message and exit
>       --config-file FILE_PATH
> -                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
> -                           (default: conf.yaml)
> +                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default:
> +                           /home/lucviz01/dpdk/dts/conf.yaml)

The path has changed.

> diff --git a/dts/framework/settings.py b/dts/framework/settings.py
> index b19f274f9d..50d8929450 100644
> --- a/dts/framework/settings.py
> +++ b/dts/framework/settings.py
> @@ -84,7 +90,24 @@
>  from typing import Callable, ParamSpec
>
>  from .config import TestSuiteConfig
> -from .utils import DPDKGitTarball
> +from .exception import ConfigurationError
> +from .utils import DPDKGitTarball, get_commit_id
> +
> +
> +def _parse_tarball_path(file_path: str) -> Path:
> +    """Validate whether `file_path` is valid and return a Path object."""
> +    path = Path(file_path)
> +    if not path.exists() or not path.is_file():
> +        raise argparse.ArgumentTypeError("The file path provided is not a valid file")
> +    return path
> +
> +
> +def _parse_revision_id(rev_id: str) -> str:
> +    """Validate revision ID and retrieve corresponding commit ID."""
> +    try:
> +        return get_commit_id(rev_id)
> +    except ConfigurationError:
> +        raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")

I made the comment about ordering in the other patch with these
functions in mind, so let's not forget these.

Re: [PATCH v5 2/3] dts: constrain DPDK source argument

[Luca Vizzarro]

On 30/05/2024 16:41, Juraj Linkeš wrote:
>> -                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
>> -                           (default: conf.yaml)
>> +                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default:
>> +                           /home/lucviz01/dpdk/dts/conf.yaml)
> 
> The path has changed.

whoops! nice catch.

>> +def _parse_tarball_path(file_path: str) -> Path:
>> +    """Validate whether `file_path` is valid and return a Path object."""
>> +    path = Path(file_path)
>> +    if not path.exists() or not path.is_file():
>> +        raise argparse.ArgumentTypeError("The file path provided is not a valid file")
>> +    return path
>> +
>> +
>> +def _parse_revision_id(rev_id: str) -> str:
>> +    """Validate revision ID and retrieve corresponding commit ID."""
>> +    try:
>> +        return get_commit_id(rev_id)
>> +    except ConfigurationError:
>> +        raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")
> 
> I made the comment about ordering in the other patch with these
> functions in mind, so let's not forget these.

Yes, this are actually in the wrong place. Will move.

[PATCH v5 3/3] dts: store stderr in RemoteCommandExecutionError

[Luca Vizzarro]

Store the stderr of an executed command in RemoteCommandExecutionError.
Consequently, when the exception is logged the error message includes
the stderr.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/exception.py                     | 13 ++++++++++---
 dts/framework/remote_session/remote_session.py |  3 ++-
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/dts/framework/exception.py b/dts/framework/exception.py
index cce1e0231a..50724acdf2 100644
--- a/dts/framework/exception.py
+++ b/dts/framework/exception.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """DTS exceptions.
 
@@ -129,21 +130,27 @@ class RemoteCommandExecutionError(DTSError):
     severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
     #: The executed command.
     command: str
+    _command_stderr: str
     _command_return_code: int
 
-    def __init__(self, command: str, command_return_code: int):
+    def __init__(self, command: str, command_return_code: int, command_stderr: str):
         """Define the meaning of the first two arguments.
 
         Args:
             command: The executed command.
             command_return_code: The return code of the executed command.
+            command_stderr: The stderr of the executed command.
         """
         self.command = command
         self._command_return_code = command_return_code
+        self._command_stderr = command_stderr
 
     def __str__(self) -> str:
-        """Include both the command and return code in the string representation."""
-        return f"Command {self.command} returned a non-zero exit code: {self._command_return_code}"
+        """Include the command, its return code and stderr in the string representation."""
+        return (
+            f"Command '{self.command}' returned a non-zero exit code: "
+            f"{self._command_return_code}\nStderr: {self._command_stderr}"
+        )
 
 
 class InteractiveCommandExecutionError(DTSError):
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
index ad0f53720a..9aaa8c8a04 100644
--- a/dts/framework/remote_session/remote_session.py
+++ b/dts/framework/remote_session/remote_session.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Base remote session.
 
@@ -172,7 +173,7 @@ def send_command(
             )
             self._logger.debug(f"stdout: '{result.stdout}'")
             self._logger.debug(f"stderr: '{result.stderr}'")
-            raise RemoteCommandExecutionError(command, result.return_code)
+            raise RemoteCommandExecutionError(command, result.return_code, result.stderr)
         self._logger.debug(f"Received from '{command}':\n{result}")
         self.history.append(result)
         return result
-- 
2.34.1

Re: [PATCH v5 3/3] dts: store stderr in RemoteCommandExecutionError

[Juraj Linkeš]

> diff --git a/dts/framework/exception.py b/dts/framework/exception.py
> index cce1e0231a..50724acdf2 100644
> --- a/dts/framework/exception.py
> +++ b/dts/framework/exception.py
> @@ -2,6 +2,7 @@
>  # Copyright(c) 2010-2014 Intel Corporation
>  # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
>  # Copyright(c) 2022-2023 University of New Hampshire
> +# Copyright(c) 2024 Arm Limited
>
>  """DTS exceptions.
>
> @@ -129,21 +130,27 @@ class RemoteCommandExecutionError(DTSError):
>      severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
>      #: The executed command.
>      command: str
> +    _command_stderr: str
>      _command_return_code: int
>
> -    def __init__(self, command: str, command_return_code: int):
> +    def __init__(self, command: str, command_return_code: int, command_stderr: str):

I wanted to change the order here as well to command, command_stderr
and command_return_code last. The API change doesn't matter as DTS is
still experimental

>          """Define the meaning of the first two arguments.
>
>          Args:
>              command: The executed command.
>              command_return_code: The return code of the executed command.
> +            command_stderr: The stderr of the executed command.

And here.

>          """
>          self.command = command
>          self._command_return_code = command_return_code
> +        self._command_stderr = command_stderr

And here.

>
>      def __str__(self) -> str:
> -        """Include both the command and return code in the string representation."""
> -        return f"Command {self.command} returned a non-zero exit code: {self._command_return_code}"
> +        """Include the command, its return code and stderr in the string representation."""
> +        return (
> +            f"Command '{self.command}' returned a non-zero exit code: "
> +            f"{self._command_return_code}\nStderr: {self._command_stderr}"
> +        )
>
>
>  class InteractiveCommandExecutionError(DTSError):

Re: [PATCH v5 3/3] dts: store stderr in RemoteCommandExecutionError

[Luca Vizzarro]

On 30/05/2024 16:47, Juraj Linkeš wrote:
>> @@ -129,21 +130,27 @@ class RemoteCommandExecutionError(DTSError):
>>       severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
>>       #: The executed command.
>>       command: str
>> +    _command_stderr: str
>>       _command_return_code: int
>>
>> -    def __init__(self, command: str, command_return_code: int):
>> +    def __init__(self, command: str, command_return_code: int, command_stderr: str):
> 
> I wanted to change the order here as well to command, command_stderr
> and command_return_code last. The API change doesn't matter as DTS is
> still experimental
> 
Ack.
>>           """Define the meaning of the first two arguments.
>>
>>           Args:
>>               command: The executed command.
>>               command_return_code: The return code of the executed command.
>> +            command_stderr: The stderr of the executed command.
> 
> And here.
> 
Ack.
>>           """
>>           self.command = command
>>           self._command_return_code = command_return_code
>> +        self._command_stderr = command_stderr
> 
> And here.
> 
Ack.

Re: [PATCH v5 0/3] error and usage improvements

[Luca Vizzarro]

Depends-on: series-31920 ("dts: update mypy and clean up")

[PATCH v6 0/3] error and usage improvements

[Luca Vizzarro]

v6:
- made members of settings.py private
- fixed find_action bug
- exchanged the env var HOF for a regular function
- minor refactoring
v5:
- re-sent correct patches
v4:
- fix spelling typo
v3:
- amended arguments rework so that it retains the original functional
  style
- re-implemetend functionalities in v2 by augmenting argparse classes
  and functions 
v2:
- complete rework of the arguments handling, to retain
  the environment variables and gain control over them
- prefixing 'Stderr: ' to RemoteCommandExecutionError
- rebased

---
Depends-on: series-32026 ("dts: update mypy and clean up")
---

Luca Vizzarro (3):
  dts: rework arguments framework
  dts: constrain DPDK source argument
  dts: store stderr in RemoteCommandExecutionError

 doc/guides/tools/dts.rst                      |  49 +--
 dts/framework/exception.py                    |  13 +-
 .../remote_session/remote_session.py          |   3 +-
 dts/framework/settings.py                     | 314 +++++++++++++-----
 dts/framework/utils.py                        |  43 ++-
 5 files changed, 295 insertions(+), 127 deletions(-)

-- 
2.34.1

[PATCH v6 1/3] dts: rework arguments framework

[Luca Vizzarro]

The existing argument handling in the code relies on basic argparse
functionality and a custom argparse action to integrate environment
variables. This commit improves the current handling by augmenting
argparse.

This rework implements the following improvements:
- There are duplicate expressions scattered throughout the code. To
  improve readability and maintainability, these are refactored
  into list/dict comprehensions or factory functions.
- Instead of relying solely on argument flags, error messages now
  accurately reference environment variables when applicable, enhancing
  user experience. For instance:

    error: environment variable DTS_DPDK_TARBALL: Invalid file

- Knowing the number of environment variables and arguments set
  allow for a useful help page display when none are provided.
- A display of which environment variables have been detected and their
  corresponding values in the help page, aiding user awareness.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/tools/dts.rst  |  53 +++++---
 dts/framework/settings.py | 268 +++++++++++++++++++++++++++-----------
 2 files changed, 223 insertions(+), 98 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 91560ee326..6c350f8667 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,30 +215,41 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run RE_RUN]
+   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
+                  [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
 
-   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority.
+   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
+   line arguments have higher priority.
 
    options:
-   -h, --help            show this help message and exit
-   --config-file CONFIG_FILE
-                         [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml)
-   --output-dir OUTPUT_DIR, --output OUTPUT_DIR
-                         [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
-   -t TIMEOUT, --timeout TIMEOUT
-                         [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
-   -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
-   -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-   --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL
-                         [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz)
-   --compile-timeout COMPILE_TIMEOUT
-                         [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
-   --test-suite TEST_SUITE [TEST_CASES ...]
-                         [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites in the environment
-                         variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case ... | DTS_TEST_SUITES='suite, suite case, ...'
-                         (default: [])
-   --re-run RE_RUN, --re_run RE_RUN
-                         [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
+     -h, --help            show this help message and exit
+     --config-file FILE_PATH
+                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
+                           (default: conf.yaml)
+     --output-dir DIR_PATH, --output DIR_PATH
+                           [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
+     -t SECONDS, --timeout SECONDS
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
+                           (default: 15)
+     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
+                           (default: False)
+     -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
+     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
+                           test. To test local changes, first commit them, then use the commit ID with this option.
+                           (default: dpdk.tar.xz)
+     --compile-timeout SECONDS
+                           [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
+     --test-suite TEST_SUITE [TEST_CASES ...]
+                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
+                           the test suite name, and the rest are test case names, which are optional. May be specified
+                           multiple times. To specify multiple test suites in the environment variable, join the lists
+                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
+                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
+                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
+     --re-run N_TIMES, --re_run N_TIMES
+                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
+                           (default: 0)
 
 
 The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 688e8679a7..7ce7ae2466 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2021 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Environment variables and command line arguments parsing.
 
@@ -72,9 +73,11 @@
 
 import argparse
 import os
+import sys
+from argparse import Action, ArgumentDefaultsHelpFormatter, _get_action_name
 from dataclasses import dataclass, field
 from pathlib import Path
-from typing import Any
+from typing import Callable
 
 from .config import TestSuiteConfig
 from .utils import DPDKGitTarball
@@ -110,103 +113,213 @@ class Settings:
 SETTINGS: Settings = Settings()
 
 
-def _get_parser() -> argparse.ArgumentParser:
-    """Create the argument parser for DTS.
+#: Attribute name representing the env variable name to augment :class:`~argparse.Action` with.
+_ENV_VAR_NAME_ATTR = "env_var_name"
+#: Attribute name representing the value origin to augment :class:`~argparse.Action` with.
+_IS_FROM_ENV_ATTR = "is_from_env"
 
-    Command line options take precedence over environment variables, which in turn take precedence
-    over default values.
+#: The prefix to be added to all of the environment variables.
+_ENV_PREFIX = "DTS_"
+
+
+def _make_env_var_name(action: Action, env_var_name: str | None) -> str:
+    """Make and assign an environment variable name to the given action."""
+    env_var_name = f"{_ENV_PREFIX}{env_var_name or action.dest.upper()}"
+    setattr(action, _ENV_VAR_NAME_ATTR, env_var_name)
+    return env_var_name
+
+
+def _get_env_var_name(action: Action) -> str | None:
+    """Get the environment variable name of the given action."""
+    return getattr(action, _ENV_VAR_NAME_ATTR, None)
+
+
+def _set_is_from_env(action: Action) -> None:
+    """Make the environment the given action's value origin."""
+    setattr(action, _IS_FROM_ENV_ATTR, True)
 
-    Returns:
-        argparse.ArgumentParser: The configured argument parser with defined options.
-    """
 
-    def env_arg(env_var: str, default: Any) -> Any:
-        """A helper function augmenting the argparse with environment variables.
+def _is_from_env(action: Action) -> bool:
+    """Check if the given action's value originated from the environment."""
+    return getattr(action, _IS_FROM_ENV_ATTR, False)
 
-        If the supplied environment variable is defined, then the default value
-        of the argument is modified. This satisfies the priority order of
-        command line argument > environment variable > default value.
 
-        Args:
-            env_var: Environment variable name.
-            default: Default value.
+def _is_action_in_args(action: Action) -> bool:
+    """Check if the action is invoked in the command line arguments."""
+    for option in action.option_strings:
+        if option in sys.argv:
+            return True
+    return False
 
-        Returns:
-            Environment variable or default value.
+
+def _add_env_var_to_action(
+    action: Action,
+    env_var_name: str | None = None,
+) -> None:
+    """Add an argument with an environment variable to the parser."""
+    env_var_name = _make_env_var_name(action, env_var_name)
+
+    if not _is_action_in_args(action):
+        env_var_value = os.environ.get(env_var_name)
+        if env_var_value is not None:
+            _set_is_from_env(action)
+            sys.argv[1:0] = [action.format_usage(), env_var_value]
+
+
+class _DTSArgumentParser(argparse.ArgumentParser):
+    """ArgumentParser with a custom error message.
+
+    This custom version of ArgumentParser changes the error message to accurately reflect the origin
+    of the value of its arguments. If it was supplied through the command line nothing changes, but
+    if it was supplied as an environment variable this is correctly communicated.
+    """
+
+    def find_action(
+        self, action_dest: str, filter_fn: Callable[[Action], bool] | None = None
+    ) -> Action | None:
+        """Find and return an action by its destination variable name.
+
+        Arguments:
+            action_dest: the destination variable name of the action to find.
+            filter_fn: if an action is found it is passed to this filter function, which must
+                return a boolean value.
         """
-        return os.environ.get(env_var) or default
+        it = (action for action in self._actions if action.dest == action_dest)
+        action = next(it, None)
+
+        if action and filter_fn:
+            return action if filter_fn(action) else None
+
+        return action
+
+    def error(self, message):
+        """Augments :meth:`~argparse.ArgumentParser.error` with environment variable awareness."""
+        for action in self._actions:
+            if _is_from_env(action):
+                action_name = _get_action_name(action)
+                env_var_name = _get_env_var_name(action)
+                env_var_value = os.environ.get(env_var_name)
 
-    parser = argparse.ArgumentParser(
+                message = message.replace(
+                    f"argument {action_name}",
+                    f"environment variable {env_var_name} (value: {env_var_value})",
+                )
+
+        print(f"{self.prog}: error: {message}\n", file=sys.stderr)
+        self.exit(2, "For help and usage, " "run the command with the --help flag.\n")
+
+
+class _EnvVarHelpFormatter(ArgumentDefaultsHelpFormatter):
+    """Custom formatter to add environment variables to the help page."""
+
+    def _get_help_string(self, action):
+        """Overrides :meth:`ArgumentDefaultsHelpFormatter._get_help_string`."""
+        help = super()._get_help_string(action)
+
+        env_var_name = _get_env_var_name(action)
+        if env_var_name is not None:
+            help = f"[{env_var_name}] {help}"
+
+            env_var_value = os.environ.get(env_var_name)
+            if env_var_value is not None:
+                help = f"{help} (env value: {env_var_value})"
+
+        return help
+
+
+def _get_parser() -> _DTSArgumentParser:
+    """Create the argument parser for DTS.
+
+    Command line options take precedence over environment variables, which in turn take precedence
+    over default values.
+
+    Returns:
+        _DTSArgumentParser: The configured argument parser with defined options.
+    """
+    parser = _DTSArgumentParser(
         description="Run DPDK test suites. All options may be specified with the environment "
         "variables provided in brackets. Command line arguments have higher priority.",
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+        formatter_class=_EnvVarHelpFormatter,
+        allow_abbrev=False,
     )
 
-    parser.add_argument(
+    action = parser.add_argument(
         "--config-file",
-        default=env_arg("DTS_CFG_FILE", SETTINGS.config_file_path),
+        default=SETTINGS.config_file_path,
         type=Path,
-        help="[DTS_CFG_FILE] The configuration file that describes the test cases, "
-        "SUTs and targets.",
+        help="The configuration file that describes the test cases, SUTs and targets.",
+        metavar="FILE_PATH",
+        dest="config_file_path",
     )
+    _add_env_var_to_action(action, "CFG_FILE")
 
-    parser.add_argument(
+    action = parser.add_argument(
         "--output-dir",
         "--output",
-        default=env_arg("DTS_OUTPUT_DIR", SETTINGS.output_dir),
-        help="[DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved.",
+        default=SETTINGS.output_dir,
+        help="Output directory where DTS logs and results are saved.",
+        metavar="DIR_PATH",
     )
+    _add_env_var_to_action(action)
 
-    parser.add_argument(
+    action = parser.add_argument(
         "-t",
         "--timeout",
-        default=env_arg("DTS_TIMEOUT", SETTINGS.timeout),
+        default=SETTINGS.timeout,
         type=float,
-        help="[DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.",
+        help="The default timeout for all DTS operations except for compiling DPDK.",
+        metavar="SECONDS",
     )
+    _add_env_var_to_action(action)
 
-    parser.add_argument(
+    action = parser.add_argument(
         "-v",
         "--verbose",
         action="store_true",
-        default=env_arg("DTS_VERBOSE", SETTINGS.verbose),
-        help="[DTS_VERBOSE] Specify to enable verbose output, logging all messages "
-        "to the console.",
+        default=SETTINGS.verbose,
+        help="Specify to enable verbose output, logging all messages to the console.",
     )
+    _add_env_var_to_action(action)
 
-    parser.add_argument(
+    action = parser.add_argument(
         "-s",
         "--skip-setup",
         action="store_true",
-        default=env_arg("DTS_SKIP_SETUP", SETTINGS.skip_setup),
-        help="[DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes.",
+        default=SETTINGS.skip_setup,
+        help="Specify to skip all setup steps on SUT and TG nodes.",
     )
+    _add_env_var_to_action(action)
 
-    parser.add_argument(
+    action = parser.add_argument(
         "--tarball",
         "--snapshot",
         "--git-ref",
-        default=env_arg("DTS_DPDK_TARBALL", SETTINGS.dpdk_tarball_path),
+        default=SETTINGS.dpdk_tarball_path,
         type=Path,
-        help="[DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, "
+        help="Path to DPDK source code tarball or a git commit ID, "
         "tag ID or tree ID to test. To test local changes, first commit them, "
         "then use the commit ID with this option.",
+        metavar="FILE_PATH",
+        dest="dpdk_tarball_path",
     )
+    _add_env_var_to_action(action, "DPDK_TARBALL")
 
-    parser.add_argument(
+    action = parser.add_argument(
         "--compile-timeout",
-        default=env_arg("DTS_COMPILE_TIMEOUT", SETTINGS.compile_timeout),
+        default=SETTINGS.compile_timeout,
         type=float,
-        help="[DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK.",
+        help="The timeout for compiling DPDK.",
+        metavar="SECONDS",
     )
+    _add_env_var_to_action(action)
 
-    parser.add_argument(
+    action = parser.add_argument(
         "--test-suite",
         action="append",
         nargs="+",
         metavar=("TEST_SUITE", "TEST_CASES"),
-        default=env_arg("DTS_TEST_SUITES", SETTINGS.test_suites),
-        help="[DTS_TEST_SUITES] A list containing a test suite with test cases. "
+        default=SETTINGS.test_suites,
+        help="A list containing a test suite with test cases. "
         "The first parameter is the test suite name, and the rest are test case names, "
         "which are optional. May be specified multiple times. To specify multiple test suites in "
         "the environment variable, join the lists with a comma. "
@@ -215,21 +328,26 @@ def env_arg(env_var: str, default: Any) -> Any:
         "DTS_TEST_SUITES='suite case case, suite case, ...' | "
         "--test-suite suite --test-suite suite case ... | "
         "DTS_TEST_SUITES='suite, suite case, ...'",
+        dest="test_suites",
     )
+    _add_env_var_to_action(action)
 
-    parser.add_argument(
+    action = parser.add_argument(
         "--re-run",
         "--re_run",
-        default=env_arg("DTS_RERUN", SETTINGS.re_run),
+        default=SETTINGS.re_run,
         type=int,
-        help="[DTS_RERUN] Re-run each test case the specified number of times "
-        "if a test failure occurs.",
+        help="Re-run each test case the specified number of times if a test failure occurs.",
+        metavar="N_TIMES",
     )
+    _add_env_var_to_action(action, "RERUN")
 
     return parser
 
 
-def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
+def _process_test_suites(
+    parser: _DTSArgumentParser, args: list[list[str]]
+) -> list[TestSuiteConfig]:
     """Process the given argument to a list of :class:`TestSuiteConfig` to execute.
 
     Args:
@@ -240,17 +358,11 @@ def _process_test_suites(args: str | list[list[str]]) -> list[TestSuiteConfig]:
     Returns:
         A list of test suite configurations to execute.
     """
-    if isinstance(args, str):
-        # Environment variable in the form of "suite case case, suite case, suite, ..."
-        args = [suite_with_cases.split() for suite_with_cases in args.split(",")]
-
-    test_suites_to_run = []
-    for suite_with_cases in args:
-        test_suites_to_run.append(
-            TestSuiteConfig(test_suite=suite_with_cases[0], test_cases=suite_with_cases[1:])
-        )
+    if parser.find_action("test_suites", _is_from_env):
+        # Environment variable in the form of "SUITE1 CASE1 CASE2, SUITE2 CASE1, SUITE3, ..."
+        args = [suite_with_cases.split() for suite_with_cases in args[0][0].split(",")]
 
-    return test_suites_to_run
+    return [TestSuiteConfig(test_suite, test_cases) for [test_suite, *test_cases] in args]
 
 
 def get_settings() -> Settings:
@@ -261,19 +373,21 @@ def get_settings() -> Settings:
     Returns:
         The new settings object.
     """
-    parsed_args = _get_parser().parse_args()
-    return Settings(
-        config_file_path=parsed_args.config_file,
-        output_dir=parsed_args.output_dir,
-        timeout=parsed_args.timeout,
-        verbose=parsed_args.verbose,
-        skip_setup=parsed_args.skip_setup,
-        dpdk_tarball_path=Path(
-            Path(DPDKGitTarball(parsed_args.tarball, parsed_args.output_dir))
-            if not os.path.exists(parsed_args.tarball)
-            else Path(parsed_args.tarball)
-        ),
-        compile_timeout=parsed_args.compile_timeout,
-        test_suites=_process_test_suites(parsed_args.test_suite),
-        re_run=parsed_args.re_run,
+    parser = _get_parser()
+
+    if len(sys.argv) == 1:
+        parser.print_help()
+        sys.exit(1)
+
+    args = parser.parse_args()
+
+    args.dpdk_tarball_path = Path(
+        Path(DPDKGitTarball(args.dpdk_tarball_path, args.output_dir))
+        if not os.path.exists(args.dpdk_tarball_path)
+        else Path(args.dpdk_tarball_path)
     )
+
+    args.test_suites = _process_test_suites(parser, args.test_suites)
+
+    kwargs = {k: v for k, v in vars(args).items() if hasattr(SETTINGS, k)}
+    return Settings(**kwargs)
-- 
2.34.1

Re: [PATCH v6 1/3] dts: rework arguments framework

[Juraj Linkeš]

On Fri, May 31, 2024 at 1:22 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> The existing argument handling in the code relies on basic argparse
> functionality and a custom argparse action to integrate environment
> variables. This commit improves the current handling by augmenting
> argparse.
>
> This rework implements the following improvements:
> - There are duplicate expressions scattered throughout the code. To
>   improve readability and maintainability, these are refactored
>   into list/dict comprehensions or factory functions.
> - Instead of relying solely on argument flags, error messages now
>   accurately reference environment variables when applicable, enhancing
>   user experience. For instance:
>
>     error: environment variable DTS_DPDK_TARBALL: Invalid file
>
> - Knowing the number of environment variables and arguments set
>   allow for a useful help page display when none are provided.
> - A display of which environment variables have been detected and their
>   corresponding values in the help page, aiding user awareness.
>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>

Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

Re: [PATCH v6 1/3] dts: rework arguments framework

[Jeremy Spewock]

On Fri, May 31, 2024 at 7:22 AM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> The existing argument handling in the code relies on basic argparse
> functionality and a custom argparse action to integrate environment
> variables. This commit improves the current handling by augmenting
> argparse.
>
> This rework implements the following improvements:
> - There are duplicate expressions scattered throughout the code. To
>   improve readability and maintainability, these are refactored
>   into list/dict comprehensions or factory functions.
> - Instead of relying solely on argument flags, error messages now
>   accurately reference environment variables when applicable, enhancing
>   user experience. For instance:
>
>     error: environment variable DTS_DPDK_TARBALL: Invalid file
>
> - Knowing the number of environment variables and arguments set
>   allow for a useful help page display when none are provided.
> - A display of which environment variables have been detected and their
>   corresponding values in the help page, aiding user awareness.
>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>

Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

[PATCH v6 2/3] dts: constrain DPDK source argument

[Luca Vizzarro]

DTS needs an input to gather the DPDK source code from. This is then
built on the remote target. This commit makes sure that this input is
more constrained, separating the Git revision ID – used to create a
tarball using Git – and providing tarballed source code directly, while
retaining mutual exclusion.

This makes the code more readable and easier to handle for input
validation, of which this commit introduces a basic one based on the
pre-existing code.

Moreover it ensures that these flags are explicitly required to be set
by the user, dropping a default value.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 doc/guides/tools/dts.rst  | 38 +++++++++++-------------
 dts/framework/settings.py | 62 +++++++++++++++++++++++++++++----------
 dts/framework/utils.py    | 43 ++++++++++++++++-----------
 3 files changed, 89 insertions(+), 54 deletions(-)

diff --git a/doc/guides/tools/dts.rst b/doc/guides/tools/dts.rst
index 6c350f8667..f877bf2df0 100644
--- a/doc/guides/tools/dts.rst
+++ b/doc/guides/tools/dts.rst
@@ -215,41 +215,37 @@ DTS is run with ``main.py`` located in the ``dts`` directory after entering Poet
 .. code-block:: console
 
    (dts-py3.10) $ ./main.py --help
-   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] [--tarball FILE_PATH]
+   usage: main.py [-h] [--config-file FILE_PATH] [--output-dir DIR_PATH] [-t SECONDS] [-v] [-s] (--tarball FILE_PATH | --revision ID)
                   [--compile-timeout SECONDS] [--test-suite TEST_SUITE [TEST_CASES ...]] [--re-run N_TIMES]
 
-   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command
-   line arguments have higher priority.
+   Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher
+   priority.
 
    options:
      -h, --help            show this help message and exit
      --config-file FILE_PATH
-                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets.
-                           (default: conf.yaml)
+                           [DTS_CFG_FILE] The configuration file that describes the test cases, SUTs and targets. (default: conf.yaml)
      --output-dir DIR_PATH, --output DIR_PATH
                            [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output)
      -t SECONDS, --timeout SECONDS
-                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK.
-                           (default: 15)
-     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console.
-                           (default: False)
+                           [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15)
+     -v, --verbose         [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False)
      -s, --skip-setup      [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: False)
-     --tarball FILE_PATH, --snapshot FILE_PATH, --git-ref FILE_PATH
-                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID,tag ID or tree ID to
-                           test. To test local changes, first commit them, then use the commit ID with this option.
-                           (default: dpdk.tar.xz)
+     --tarball FILE_PATH, --snapshot FILE_PATH
+                           [DTS_DPDK_TARBALL] Path to DPDK source code tarball to test. (default: None)
+     --revision ID, --rev ID, --git-ref ID
+                           [DTS_DPDK_REVISION_ID] Git revision ID to test. Could be commit, tag, tree ID etc. To test local changes, first
+                           commit them, then use their commit ID. (default: None)
      --compile-timeout SECONDS
                            [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200)
      --test-suite TEST_SUITE [TEST_CASES ...]
-                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is
-                           the test suite name, and the rest are test case names, which are optional. May be specified
-                           multiple times. To specify multiple test suites in the environment variable, join the lists
-                           with a comma. Examples: --test-suite SUITE1 CASE1 CASE2 --test-suite SUITE2 CASE1 ... |
-                           DTS_TEST_SUITES='SUITE1 CASE1 CASE2, SUITE2 CASE1, ...' | --test-suite SUITE1 --test-suite SUITE2
-                           CASE1 ... | DTS_TEST_SUITES='SUITE1, SUITE2 CASE1, ...' (default: [])
+                           [DTS_TEST_SUITES] A list containing a test suite with test cases. The first parameter is the test suite name, and
+                           the rest are test case names, which are optional. May be specified multiple times. To specify multiple test suites
+                           in the environment variable, join the lists with a comma. Examples: --test-suite suite case case --test-suite
+                           suite case ... | DTS_TEST_SUITES='suite case case, suite case, ...' | --test-suite suite --test-suite suite case
+                           ... | DTS_TEST_SUITES='suite, suite case, ...' (default: [])
      --re-run N_TIMES, --re_run N_TIMES
-                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs.
-                           (default: 0)
+                           [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs. (default: 0)
 
 
 The brackets contain the names of environment variables that set the same thing.
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index 7ce7ae2466..f95876113f 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -44,10 +44,16 @@
 
     Set to any value to skip building DPDK.
 
-.. option:: --tarball, --snapshot, --git-ref
+.. option:: --tarball, --snapshot
 .. envvar:: DTS_DPDK_TARBALL
 
-    The path to a DPDK tarball, git commit ID, tag ID or tree ID to test.
+    Path to DPDK source code tarball to test.
+
+.. option:: --revision, --rev, --git-ref
+.. envvar:: DTS_DPDK_REVISION_ID
+
+    Git revision ID to test. Could be commit, tag, tree ID etc.
+    To test local changes, first commit them, then use their commit ID.
 
 .. option:: --test-suite
 .. envvar:: DTS_TEST_SUITES
@@ -80,7 +86,8 @@
 from typing import Callable
 
 from .config import TestSuiteConfig
-from .utils import DPDKGitTarball
+from .exception import ConfigurationError
+from .utils import DPDKGitTarball, get_commit_id
 
 
 @dataclass(slots=True)
@@ -101,7 +108,7 @@ class Settings:
     #:
     skip_setup: bool = False
     #:
-    dpdk_tarball_path: Path | str = "dpdk.tar.xz"
+    dpdk_tarball_path: Path | str = ""
     #:
     compile_timeout: float = 1200
     #:
@@ -227,6 +234,22 @@ def _get_help_string(self, action):
         return help
 
 
+def _parse_tarball_path(file_path: str) -> Path:
+    """Validate whether `file_path` is valid and return a Path object."""
+    path = Path(file_path)
+    if not path.exists() or not path.is_file():
+        raise argparse.ArgumentTypeError("The file path provided is not a valid file")
+    return path
+
+
+def _parse_revision_id(rev_id: str) -> str:
+    """Validate revision ID and retrieve corresponding commit ID."""
+    try:
+        return get_commit_id(rev_id)
+    except ConfigurationError:
+        raise argparse.ArgumentTypeError("The Git revision ID supplied is invalid or ambiguous")
+
+
 def _get_parser() -> _DTSArgumentParser:
     """Create the argument parser for DTS.
 
@@ -290,20 +313,30 @@ def _get_parser() -> _DTSArgumentParser:
     )
     _add_env_var_to_action(action)
 
-    action = parser.add_argument(
+    dpdk_source = parser.add_mutually_exclusive_group(required=True)
+
+    action = dpdk_source.add_argument(
         "--tarball",
         "--snapshot",
-        "--git-ref",
-        default=SETTINGS.dpdk_tarball_path,
-        type=Path,
-        help="Path to DPDK source code tarball or a git commit ID, "
-        "tag ID or tree ID to test. To test local changes, first commit them, "
-        "then use the commit ID with this option.",
+        type=_parse_tarball_path,
+        help="Path to DPDK source code tarball to test.",
         metavar="FILE_PATH",
         dest="dpdk_tarball_path",
     )
     _add_env_var_to_action(action, "DPDK_TARBALL")
 
+    action = dpdk_source.add_argument(
+        "--revision",
+        "--rev",
+        "--git-ref",
+        type=_parse_revision_id,
+        help="Git revision ID to test. Could be commit, tag, tree ID etc. "
+        "To test local changes, first commit them, then use their commit ID.",
+        metavar="ID",
+        dest="dpdk_revision_id",
+    )
+    _add_env_var_to_action(action)
+
     action = parser.add_argument(
         "--compile-timeout",
         default=SETTINGS.compile_timeout,
@@ -381,11 +414,8 @@ def get_settings() -> Settings:
 
     args = parser.parse_args()
 
-    args.dpdk_tarball_path = Path(
-        Path(DPDKGitTarball(args.dpdk_tarball_path, args.output_dir))
-        if not os.path.exists(args.dpdk_tarball_path)
-        else Path(args.dpdk_tarball_path)
-    )
+    if args.dpdk_revision_id:
+        args.dpdk_tarball_path = DPDKGitTarball(args.dpdk_revision_id, args.output_dir)
 
     args.test_suites = _process_test_suites(parser, args.test_suites)
 
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index 74a11f1aaf..862bafb46c 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Various utility classes and functions.
 
@@ -70,6 +71,31 @@ def get_packet_summaries(packets: list[Packet]) -> str:
     return f"Packet contents: \n{packet_summaries}"
 
 
+def get_commit_id(rev_id: str) -> str:
+    """Given a Git revision ID, return the corresponding commit ID.
+
+    Args:
+        rev_id: The Git revision ID.
+
+    Raises:
+        ConfigurationError: The ``git rev-parse`` command failed, suggesting
+            an invalid or ambiguous revision ID was supplied.
+    """
+    result = subprocess.run(
+        ["git", "rev-parse", "--verify", rev_id],
+        text=True,
+        capture_output=True,
+    )
+    if result.returncode != 0:
+        raise ConfigurationError(
+            f"{rev_id} is not a valid git reference.\n"
+            f"Command: {result.args}\n"
+            f"Stdout: {result.stdout}\n"
+            f"Stderr: {result.stderr}"
+        )
+    return result.stdout.strip()
+
+
 class StrEnum(Enum):
     """Enum with members stored as strings."""
 
@@ -170,7 +196,6 @@ def __init__(
 
         self._tarball_dir = Path(output_dir, "tarball")
 
-        self._get_commit_id()
         self._create_tarball_dir()
 
         self._tarball_name = (
@@ -180,22 +205,6 @@ def __init__(
         if not self._tarball_path:
             self._create_tarball()
 
-    def _get_commit_id(self) -> None:
-        result = subprocess.run(
-            ["git", "rev-parse", "--verify", self._git_ref],
-            text=True,
-            capture_output=True,
-        )
-        if result.returncode != 0:
-            raise ConfigurationError(
-                f"{self._git_ref} is neither a path to an existing DPDK "
-                "archive nor a valid git reference.\n"
-                f"Command: {result.args}\n"
-                f"Stdout: {result.stdout}\n"
-                f"Stderr: {result.stderr}"
-            )
-        self._git_ref = result.stdout.strip()
-
     def _create_tarball_dir(self) -> None:
         os.makedirs(self._tarball_dir, exist_ok=True)
 
-- 
2.34.1

Re: [PATCH v6 2/3] dts: constrain DPDK source argument

[Juraj Linkeš]

On Fri, May 31, 2024 at 1:22 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> DTS needs an input to gather the DPDK source code from. This is then
> built on the remote target. This commit makes sure that this input is
> more constrained, separating the Git revision ID – used to create a
> tarball using Git – and providing tarballed source code directly, while
> retaining mutual exclusion.
>
> This makes the code more readable and easier to handle for input
> validation, of which this commit introduces a basic one based on the
> pre-existing code.
>
> Moreover it ensures that these flags are explicitly required to be set
> by the user, dropping a default value.
>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>

Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

Re: [PATCH v6 2/3] dts: constrain DPDK source argument

[Jeremy Spewock]

On Fri, May 31, 2024 at 8:51 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> On Fri, May 31, 2024 at 1:22 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
> >
> > DTS needs an input to gather the DPDK source code from. This is then
> > built on the remote target. This commit makes sure that this input is
> > more constrained, separating the Git revision ID – used to create a
> > tarball using Git – and providing tarballed source code directly, while
> > retaining mutual exclusion.
> >
> > This makes the code more readable and easier to handle for input
> > validation, of which this commit introduces a basic one based on the
> > pre-existing code.
> >
> > Moreover it ensures that these flags are explicitly required to be set
> > by the user, dropping a default value.
> >
> > Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> > Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
>
> Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

[PATCH v6 3/3] dts: store stderr in RemoteCommandExecutionError

[Luca Vizzarro]

Store the stderr of an executed command in RemoteCommandExecutionError.
Consequently, when the exception is logged the error message includes
the stderr.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/exception.py                     | 13 ++++++++++---
 dts/framework/remote_session/remote_session.py |  3 ++-
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/dts/framework/exception.py b/dts/framework/exception.py
index cce1e0231a..8502c41779 100644
--- a/dts/framework/exception.py
+++ b/dts/framework/exception.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """DTS exceptions.
 
@@ -129,21 +130,27 @@ class RemoteCommandExecutionError(DTSError):
     severity: ClassVar[ErrorSeverity] = ErrorSeverity.REMOTE_CMD_EXEC_ERR
     #: The executed command.
     command: str
+    _command_stderr: str
     _command_return_code: int
 
-    def __init__(self, command: str, command_return_code: int):
+    def __init__(self, command: str, command_stderr: str, command_return_code: int):
         """Define the meaning of the first two arguments.
 
         Args:
             command: The executed command.
+            command_stderr: The stderr of the executed command.
             command_return_code: The return code of the executed command.
         """
         self.command = command
+        self._command_stderr = command_stderr
         self._command_return_code = command_return_code
 
     def __str__(self) -> str:
-        """Include both the command and return code in the string representation."""
-        return f"Command {self.command} returned a non-zero exit code: {self._command_return_code}"
+        """Include the command, its return code and stderr in the string representation."""
+        return (
+            f"Command '{self.command}' returned a non-zero exit code: "
+            f"{self._command_return_code}\nStderr: {self._command_stderr}"
+        )
 
 
 class InteractiveCommandExecutionError(DTSError):
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
index ad0f53720a..8db26e9efc 100644
--- a/dts/framework/remote_session/remote_session.py
+++ b/dts/framework/remote_session/remote_session.py
@@ -2,6 +2,7 @@
 # Copyright(c) 2010-2014 Intel Corporation
 # Copyright(c) 2022-2023 PANTHEON.tech s.r.o.
 # Copyright(c) 2022-2023 University of New Hampshire
+# Copyright(c) 2024 Arm Limited
 
 """Base remote session.
 
@@ -172,7 +173,7 @@ def send_command(
             )
             self._logger.debug(f"stdout: '{result.stdout}'")
             self._logger.debug(f"stderr: '{result.stderr}'")
-            raise RemoteCommandExecutionError(command, result.return_code)
+            raise RemoteCommandExecutionError(command, result.stderr, result.return_code)
         self._logger.debug(f"Received from '{command}':\n{result}")
         self.history.append(result)
         return result
-- 
2.34.1

Re: [PATCH v6 3/3] dts: store stderr in RemoteCommandExecutionError

[Juraj Linkeš]

On Fri, May 31, 2024 at 1:22 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> Store the stderr of an executed command in RemoteCommandExecutionError.
> Consequently, when the exception is logged the error message includes
> the stderr.
>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>

Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

Re: [PATCH v6 3/3] dts: store stderr in RemoteCommandExecutionError

[Jeremy Spewock]

On Fri, May 31, 2024 at 8:51 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> On Fri, May 31, 2024 at 1:22 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
> >
> > Store the stderr of an executed command in RemoteCommandExecutionError.
> > Consequently, when the exception is logged the error message includes
> > the stderr.
> >
> > Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> > Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
>
> Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

Re: [PATCH v6 0/3] error and usage improvements

[Thomas Monjalon]

> Luca Vizzarro (3):
>   dts: rework arguments framework
>   dts: constrain DPDK source argument
>   dts: store stderr in RemoteCommandExecutionError

Applied, thanks.