DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Juraj Linkeš" <juraj.linkes@pantheon.tech>
To: Luca Vizzarro <luca.vizzarro@arm.com>
Cc: dev@dpdk.org, Jack Bond-Preston <jack.bond-preston@arm.com>,
	 Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Subject: Re: [PATCH 1/6] dts: add parameters data structure
Date: Tue, 9 Apr 2024 14:10:08 +0200	[thread overview]
Message-ID: <CAOb5WZYhMPoHmt7kJVmqkA3o=NX07=KYy6N8XnGqUtABaQ6ETQ@mail.gmail.com> (raw)
In-Reply-To: <20240326190422.577028-2-luca.vizzarro@arm.com>

On Tue, Mar 26, 2024 at 8:04 PM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> This commit introduces a new "params" module, which adds a new way
> to manage command line parameters. The provided Params dataclass
> is able to read the fields of its child class and produce a string
> representation to supply to the command line. Any data structure
> that is intended to represent command line parameters can inherit
> it. The representation can then manipulated by using the dataclass
> field metadata in conjunction with the provided functions:
>
> * value_only, used to supply a value without forming an option/flag
> * options_end, used to prefix with an options end delimiter (`--`)
> * short, used to define a short parameter name, e.g. `-p`
> * long, used to define a long parameter name, e.g. `--parameter`
> * multiple, used to turn a list into repeating parameters
> * field_mixins, used to manipulate the string representation of the
>   value

We shouldn't list what the patch does, but rather explain the choices
made within. It seems to me the patch is here to give devs an API
instead of them having to construct strings - explaining why this
approach improves the old one should be in the commit message.

>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Jack Bond-Preston <jack.bond-preston@arm.com>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
>  dts/framework/params.py | 232 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 232 insertions(+)
>  create mode 100644 dts/framework/params.py
>
> diff --git a/dts/framework/params.py b/dts/framework/params.py
> new file mode 100644
> index 0000000000..6b48c8353e
> --- /dev/null
> +++ b/dts/framework/params.py
> @@ -0,0 +1,232 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 Arm Limited
> +
> +"""Parameter manipulation module.
> +
> +This module provides :class:`~Params` which can be used to model any data structure
> +that is meant to represent any command parameters.
> +"""
> +
> +from dataclasses import dataclass, field, fields
> +from typing import Any, Callable, Literal, Reversible, TypeVar, Iterable
> +from enum import Flag
> +
> +
> +T = TypeVar("T")
> +#: Type for a Mixin.
> +Mixin = Callable[[Any], str]
> +#: Type for an option parameter.
> +Option = Literal[True, None]
> +#: Type for a yes/no option parameter.
> +BooleanOption = Literal[True, False, None]
> +
> +META_VALUE_ONLY = "value_only"
> +META_OPTIONS_END = "options_end"
> +META_SHORT_NAME = "short_name"
> +META_LONG_NAME = "long_name"
> +META_MULTIPLE = "multiple"
> +META_MIXINS = "mixins"

These are only used in the Params class (but not set), so I'd either
move them there or make them private.

> +
> +
> +def value_only(metadata: dict[str, Any] = {}) -> dict[str, Any]:
> +    """Injects the value of the attribute as-is without flag. Metadata modifier for :func:`dataclasses.field`."""
> +    return {**metadata, META_VALUE_ONLY: True}

These methods, on the other hand, are used outside this module, so it
makes sense to have them outside Params. It could be better to have
them inside as static methods, as they're closely related. Looking at
how they're used, we should unite the imports:
1. In testpmd_shell, they're imported directly (from framework.params
import long)
2. In sut_node, just the params module is imported (from framework
import params and then accessed via it: metadata=params.short("l"))

If we move these to Params, we could import Params and use them
similarly to 2. Not sure which one is better.

> +
> +
> +def short(name: str, metadata: dict[str, Any] = {}) -> dict[str, Any]:

It seems to me that having the metadata argument doesn't do anything
in these basic functions.

> +    """Overrides any parameter name with the given short option. Metadata modifier for :func:`dataclasses.field`.
> +
> +    .. code:: python
> +
> +        logical_cores: str | None = field(default="1-4", metadata=short("l"))
> +
> +    will render as ``-l=1-4`` instead of ``--logical-cores=1-4``.
> +    """
> +    return {**metadata, META_SHORT_NAME: name}
> +
> +
> +def long(name: str, metadata: dict[str, Any] = {}) -> dict[str, Any]:
> +    """Overrides the inferred parameter name to the specified one. Metadata modifier for :func:`dataclasses.field`.
> +
> +    .. code:: python
> +
> +        x_name: str | None = field(default="y", metadata=long("x"))
> +
> +    will render as ``--x=y``, but the field is accessed and modified through ``x_name``.
> +    """
> +    return {**metadata, META_LONG_NAME: name}
> +
> +
> +def options_end(metadata: dict[str, Any] = {}) -> dict[str, Any]:
> +    """Precedes the value with an options end delimiter (``--``). Metadata modifier for :func:`dataclasses.field`."""
> +    return {**metadata, META_OPTIONS_END: True}
> +
> +
> +def multiple(metadata: dict[str, Any] = {}) -> dict[str, Any]:
> +    """Specifies that this parameter is set multiple times. Must be a list. Metadata modifier for :func:`dataclasses.field`.
> +
> +    .. code:: python
> +
> +        ports: list[int] | None = field(default_factory=lambda: [0, 1, 2], metadata=multiple(param_name("port")))
> +
> +    will render as ``--port=0 --port=1 --port=2``. Note that modifiers can be chained like in this example.
> +    """
> +    return {**metadata, META_MULTIPLE: True}
> +
> +
> +def field_mixins(*mixins: Mixin, metadata: dict[str, Any] = {}) -> dict[str, Any]:

Any reason why mixins are plural? I've only seen this used with one
argument, do you anticipate we'd need to use more than one? We could
make this singular and if we ever need to do two things, we could just
pass a function that does those two things (or calls two different
functions). Also, I'd just rename the mixin the func or something like
that.

The default of an argument should not be mutable, here's a quick
explanation: https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments

I don't really like the name. The positional arguments are applied to
the value and that should be reflected in the name - I'd like to see
something like convert in the name.

> +    """Takes in a variable number of mixins to manipulate the value's rendering. Metadata modifier for :func:`dataclasses.field`.
> +
> +    The ``metadata`` keyword argument can be used to chain metadata modifiers together.
> +
> +    Mixins can be chained together, executed from right to left in the arguments list order.
> +
> +    Example:
> +
> +    .. code:: python
> +
> +        hex_bitmask: int | None = field(default=0b1101, metadata=field_mixins(hex, metadata=param_name("mask")))
> +
> +    will render as ``--mask=0xd``. The :func:`hex` built-in can be used as a mixin turning a valid integer into a hexadecimal representation.
> +    """
> +    return {**metadata, META_MIXINS: mixins}

metadata | {META_MIXINS: mixins} also creates a new dictionary with
values from both and I think that would be more readable (since it's
mentioned in docs:
https://docs.python.org/3/library/stdtypes.html#mapping-types-dict).

> +
> +
> +def _reduce_mixins(mixins: Reversible[Mixin], value: Any) -> str:
> +    for mixin in reversed(mixins):
> +        value = mixin(value)
> +    return value
> +
> +
> +def str_mixins(*mixins: Mixin):

Again the name, this doesn't strike me as a decorator name. A
decorator modifies the thing it's decorating so it should contain a
verb describing what it's doing.

Maybe we could also do singular mixin here, as I described above. It
may result in cleaner API.

> +    """Decorator which modifies the ``__str__`` method, enabling support for mixins.
> +
> +    Mixins can be chained together, executed from right to left in the arguments list order.
> +
> +    Example:
> +
> +    .. code:: python
> +
> +        @str_mixins(hex_from_flag_value)
> +        class BitMask(enum.Flag):
> +            A = auto()
> +            B = auto()
> +
> +    will allow ``BitMask`` to render as a hexadecimal value.
> +    """
> +
> +    def _class_decorator(original_class):
> +        original_class.__str__ = lambda self: _reduce_mixins(mixins, self)
> +        return original_class
> +
> +    return _class_decorator
> +
> +
> +def comma_separated(values: Iterable[T]) -> str:
> +    """Mixin which renders an iterable in a comma-separated string."""
> +    return ",".join([str(value).strip() for value in values if value is not None])
> +
> +
> +def bracketed(value: str) -> str:
> +    """Mixin which adds round brackets to the input."""
> +    return f"({value})"
> +
> +
> +def str_from_flag_value(flag: Flag) -> str:
> +    """Mixin which returns the value from a :class:`enum.Flag` as a string."""
> +    return str(flag.value)
> +
> +
> +def hex_from_flag_value(flag: Flag) -> str:
> +    """Mixin which turns a :class:`enum.Flag` value into hexadecimal."""
> +    return hex(flag.value)
> +
> +
> +def _make_option(param_name: str, short: bool = False, no: bool = False) -> str:
> +    param_name = param_name.replace("_", "-")
> +    return f"{'-' if short else '--'}{'no-' if no else ''}{param_name}"
> +
> +
> +@dataclass
> +class Params:
> +    """Helper abstract dataclass that renders its fields into command line arguments.

Let's make the abstract class explicitly abstract with
https://docs.python.org/3/library/abc.html#abc.ABC. It won't be a full
abstract class since it won't have any abstract method or properties,
but I think it'll be better this way.

> +
> +    The parameter name is taken from the field name by default. The following:
> +
> +    .. code:: python
> +
> +        name: str | None = "value"
> +
> +    is rendered as ``--name=value``.
> +    Through :func:`dataclasses.field` the resulting parameter can be manipulated by applying
> +    appropriate metadata. This class can be used with the following metadata modifiers:
> +
> +    * :func:`value_only`
> +    * :func:`options_end`
> +    * :func:`short`
> +    * :func:`long`
> +    * :func:`multiple`
> +    * :func:`field_mixins`
> +
> +    To use fields as option switches set the value to ``True`` to enable them. If you

There should be a comma between switches and set.

> +    use a yes/no option switch you can also set ``False`` which would enable an option
> +    prefixed with ``--no-``. Examples:
> +
> +    .. code:: python
> +
> +        interactive: Option = True  # renders --interactive
> +        numa: BooleanOption = False # renders --no-numa

I'm wondering whether these could be simplified, but it's pretty good
this way. I'd change the names though:
Option -> Switch
BooleanOption -> NoSwitch (or YesNoSwitch? NegativeSwitch? Not sure
about this one)

All options (short, long, etc.) are options so that's where it's
confusing. The BooleanOption doesn't really capture what we're doing
with it (prepending no-) so I want a name that captures it.

There could also be some confusion with this being different from the
rest of the options API. This only requires us to set the type, the
rest must be passed in dataclasses.field. It's probably not that big
of a deal though.

> +
> +    Setting ``None`` will disable any option.

I'd reword this to "Setting an option to ``None`` will prevent it from
being rendered." or something similar. Disabling an option is kinda
ambiguous.

The :attr:`~Option` type alias is provided for
> +    regular option switches, whereas :attr:`~BooleanOption` is offered for yes/no ones.
> +
> +    An instance of a dataclass inheriting ``Params`` can also be assigned to an attribute, this helps with grouping parameters
> +    together. The attribute holding the dataclass will be ignored and the latter will just be rendered as expected.
> +    """
> +
> +    def __str__(self) -> str:
> +        arguments: list[str] = []
> +
> +        for field in fields(self):
> +            value = getattr(self, field.name)
> +
> +            if value is None:
> +                continue
> +
> +            options_end = field.metadata.get(META_OPTIONS_END, False)
> +            if options_end:
> +                arguments.append("--")

This is a confusing way to separate the options. The separator itself
is an option (I'd rename it to sep or separator instead of end), so
I'd make a separate field for it. We could pass init=False to field()
in the field definition, but that relies on fields being ordered, so
we'd need to also pass order=True to the dataclass constructor.

> +
> +            value_only = field.metadata.get(META_VALUE_ONLY, False)
> +            if isinstance(value, Params) or value_only or options_end:
> +                arguments.append(str(value))
> +                continue
> +
> +            # take "short_name" metadata, or "long_name" metadata, or infer from field name
> +            option_name = field.metadata.get(
> +                META_SHORT_NAME, field.metadata.get(META_LONG_NAME, field.name)
> +            )
> +            is_short = META_SHORT_NAME in field.metadata
> +
> +            if isinstance(value, bool):
> +                arguments.append(_make_option(option_name, short=is_short, no=(not value)))
> +                continue
> +
> +            option = _make_option(option_name, short=is_short)
> +            separator = " " if is_short else "="
> +            str_mixins = field.metadata.get(META_MIXINS, [])
> +            multiple = field.metadata.get(META_MULTIPLE, False)
> +
> +            values = value if multiple else [value]
> +            for entry_value in values:
> +                entry_value = _reduce_mixins(str_mixins, entry_value)
> +                arguments.append(f"{option}{separator}{entry_value}")
> +
> +        return " ".join(arguments)
> +
> +
> +@dataclass
> +class StrParams(Params):
> +    """A drop-in replacement for parameters passed as a string."""
> +
> +    value: str = field(metadata=value_only())
> --
> 2.34.1
>

  parent reply	other threads:[~2024-04-09 12:10 UTC|newest]

Thread overview: 54+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-03-26 19:04 [PATCH 0/6] dts: add testpmd params and statefulness Luca Vizzarro
2024-03-26 19:04 ` [PATCH 1/6] dts: add parameters data structure Luca Vizzarro
2024-03-28 16:48   ` Jeremy Spewock
2024-04-09 15:52     ` Luca Vizzarro
2024-04-09 12:10   ` Juraj Linkeš [this message]
2024-04-09 16:28     ` Luca Vizzarro
2024-04-10  9:15       ` Juraj Linkeš
2024-04-10  9:51         ` Luca Vizzarro
2024-04-10 10:04           ` Juraj Linkeš
2024-03-26 19:04 ` [PATCH 2/6] dts: use Params for interactive shells Luca Vizzarro
2024-03-28 16:48   ` Jeremy Spewock
2024-04-09 14:56     ` Juraj Linkeš
2024-04-10  9:34       ` Luca Vizzarro
2024-04-10  9:58         ` Juraj Linkeš
2024-03-26 19:04 ` [PATCH 3/6] dts: add testpmd shell params Luca Vizzarro
2024-03-28 16:48   ` Jeremy Spewock
2024-04-09 16:37   ` Juraj Linkeš
2024-04-10 10:49     ` Luca Vizzarro
2024-04-10 13:17       ` Juraj Linkeš
2024-03-26 19:04 ` [PATCH 4/6] dts: use testpmd params for scatter test suite Luca Vizzarro
2024-04-09 19:12   ` Juraj Linkeš
2024-04-10 10:53     ` Luca Vizzarro
2024-04-10 13:18       ` Juraj Linkeš
2024-04-26 18:06         ` Jeremy Spewock
2024-04-29  7:45           ` Juraj Linkeš
2024-03-26 19:04 ` [PATCH 5/6] dts: add statefulness to InteractiveShell Luca Vizzarro
2024-03-28 16:48   ` Jeremy Spewock
2024-04-10  6:53     ` Juraj Linkeš
2024-04-10 11:27       ` Luca Vizzarro
2024-04-10 13:35         ` Juraj Linkeš
2024-04-10 14:07           ` Luca Vizzarro
2024-04-12 12:33             ` Juraj Linkeš
2024-04-29 14:48           ` Jeremy Spewock
2024-03-26 19:04 ` [PATCH 6/6] dts: add statefulness to TestPmdShell Luca Vizzarro
2024-03-28 16:48   ` Jeremy Spewock
2024-04-10  7:41     ` Juraj Linkeš
2024-04-10 11:35       ` Luca Vizzarro
2024-04-11 10:30         ` Juraj Linkeš
2024-04-11 11:47           ` Luca Vizzarro
2024-04-11 12:13             ` Juraj Linkeš
2024-04-11 13:59               ` Luca Vizzarro
2024-04-26 18:06               ` Jeremy Spewock
2024-04-29 12:06                 ` Juraj Linkeš
2024-04-10  7:50   ` Juraj Linkeš
2024-04-10 11:37     ` Luca Vizzarro
2024-05-09 11:20 ` [PATCH v2 0/8] dts: add testpmd params Luca Vizzarro
2024-05-09 11:20   ` [PATCH v2 1/8] dts: add params manipulation module Luca Vizzarro
2024-05-09 11:20   ` [PATCH v2 2/8] dts: use Params for interactive shells Luca Vizzarro
2024-05-09 11:20   ` [PATCH v2 3/8] dts: refactor EalParams Luca Vizzarro
2024-05-09 11:20   ` [PATCH v2 4/8] dts: remove module-wide imports Luca Vizzarro
2024-05-09 11:20   ` [PATCH v2 5/8] dts: add testpmd shell params Luca Vizzarro
2024-05-09 11:20   ` [PATCH v2 6/8] dts: use testpmd params for scatter test suite Luca Vizzarro
2024-05-09 11:20   ` [PATCH v2 7/8] dts: rework interactive shells Luca Vizzarro
2024-05-09 11:20   ` [PATCH v2 8/8] dts: use Unpack for type checking and hinting Luca Vizzarro

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAOb5WZYhMPoHmt7kJVmqkA3o=NX07=KYy6N8XnGqUtABaQ6ETQ@mail.gmail.com' \
    --to=juraj.linkes@pantheon.tech \
    --cc=dev@dpdk.org \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=jack.bond-preston@arm.com \
    --cc=luca.vizzarro@arm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).