From: "Juraj Linkeš" <juraj.linkes@pantheon.tech>
To: thomas@monjalon.net, david.marchand@redhat.com,
jerinjacobk@gmail.com, ronan.randles@intel.com,
Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu,
lijuan.tu@intel.com
Cc: dev@dpdk.org, "Juraj Linkeš" <juraj.linkes@pantheon.tech>
Subject: [PATCH v2 5/8] dts: add config parser module
Date: Mon, 11 Jul 2022 14:51:23 +0000 [thread overview]
Message-ID: <20220711145126.295427-6-juraj.linkes@pantheon.tech> (raw)
In-Reply-To: <20220711145126.295427-1-juraj.linkes@pantheon.tech>
From: Owen Hilyard <ohilyard@iol.unh.edu>
The configuration is split into two parts, one defining the parameters
of the test run and the other defining the topology to be used.
The format of the configuration is YAML. It is validated according to a
json schema which also servers as detailed documentation of the various
configuration fields. This means that the complete set of allowed values
are tied to the schema as a source of truth. This enables making changes
to parts of DTS that interface with config files without a high risk of
breaking someone's configuration.
This configuration system uses immutable objects to represent the
configuration, making IDE/LSP autocomplete work properly.
Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/conf.yaml | 7 ++
dts/framework/config/__init__.py | 116 +++++++++++++++++++++
dts/framework/config/conf_yaml_schema.json | 73 +++++++++++++
dts/framework/exception.py | 15 +++
dts/framework/settings.py | 40 +++++++
5 files changed, 251 insertions(+)
create mode 100644 dts/conf.yaml
create mode 100644 dts/framework/config/__init__.py
create mode 100644 dts/framework/config/conf_yaml_schema.json
create mode 100644 dts/framework/settings.py
diff --git a/dts/conf.yaml b/dts/conf.yaml
new file mode 100644
index 0000000000..11b5f53a66
--- /dev/null
+++ b/dts/conf.yaml
@@ -0,0 +1,7 @@
+executions:
+ - system_under_test: "SUT 1"
+nodes:
+ - name: "SUT 1"
+ hostname: "SUT IP Address or hostname"
+ user: root
+ password: "leave blank to use SSH keys"
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
new file mode 100644
index 0000000000..511e70c9a5
--- /dev/null
+++ b/dts/framework/config/__init__.py
@@ -0,0 +1,116 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2021 Intel Corporation
+# Copyright(c) 2022 University of New Hampshire
+#
+
+"""
+Generic port and topology nodes configuration file load function
+"""
+import json
+import os.path
+import pathlib
+from dataclasses import dataclass
+from enum import Enum, auto, unique
+from typing import Any, Optional
+
+import warlock
+import yaml
+
+from framework.settings import get_config_file_path
+
+
+class StrEnum(Enum):
+ @staticmethod
+ def _generate_next_value_(
+ name: str, start: int, count: int, last_values: object
+ ) -> str:
+ return name
+
+
+@unique
+class NodeType(StrEnum):
+ physical = auto()
+ virtual = auto()
+
+
+# Slots enables some optimizations, by pre-allocating space for the defined
+# attributes in the underlying data structure.
+#
+# Frozen makes the object immutable. This enables further optimizations,
+# and makes it thread safe should we every want to move in that direction.
+@dataclass(slots=True, frozen=True)
+class NodeConfiguration:
+ name: str
+ hostname: str
+ user: str
+ password: Optional[str]
+
+ @staticmethod
+ def from_dict(d: dict) -> "NodeConfiguration":
+ return NodeConfiguration(
+ name=d["name"],
+ hostname=d["hostname"],
+ user=d["user"],
+ password=d.get("password"),
+ )
+
+
+@dataclass(slots=True, frozen=True)
+class ExecutionConfiguration:
+ system_under_test: str
+
+ @staticmethod
+ def from_dict(d: dict) -> "ExecutionConfiguration":
+ return ExecutionConfiguration(
+ system_under_test=d["system_under_test"],
+ )
+
+
+@dataclass(slots=True, frozen=True)
+class Configuration:
+ executions: list[ExecutionConfiguration]
+ nodes: list[NodeConfiguration]
+
+ @staticmethod
+ def from_dict(d: dict) -> "Configuration":
+ executions: list[ExecutionConfiguration] = list(
+ map(ExecutionConfiguration.from_dict, d["executions"])
+ )
+ nodes: list[NodeConfiguration] = list(
+ map(NodeConfiguration.from_dict, d["nodes"])
+ )
+ assert len(nodes) > 0, "There must be a node to test"
+
+ for i, n1 in enumerate(nodes):
+ for j, n2 in enumerate(nodes):
+ if i != j:
+ assert n1.name == n2.name, "Duplicate node names are not allowed"
+
+ node_names = {node.name for node in nodes}
+ for execution in executions:
+ assert (
+ execution.system_under_test in node_names
+ ), f"Unknown SUT {execution.system_under_test} in execution"
+
+ return Configuration(executions=executions, nodes=nodes)
+
+
+def load_config(conf_file_path: str) -> Configuration:
+ """
+ Loads the configuration file and the configuration file schema,
+ validates the configuration file, and creates a configuration object.
+ """
+ conf_file_path: str = get_config_file_path(conf_file_path)
+ with open(conf_file_path, "r") as f:
+ config_data = yaml.safe_load(f)
+ schema_path = os.path.join(
+ pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json"
+ )
+
+ with open(schema_path, "r") as f:
+ schema = json.load(f)
+ config: dict[str, Any] = warlock.model_factory(schema, name="_Config")(
+ config_data
+ )
+ config_obj: Configuration = Configuration.from_dict(dict(config))
+ return config_obj
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
new file mode 100644
index 0000000000..8dea50e285
--- /dev/null
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -0,0 +1,73 @@
+{
+ "$schema": "https://json-schema.org/draft-07/schema",
+ "title": "DPDK DTS Config Schema",
+ "definitions": {
+ "node_name": {
+ "type": "string",
+ "description": "A unique identifier for a node"
+ },
+ "node_role": {
+ "type": "string",
+ "description": "The role a node plays in DTS",
+ "enum": [
+ "system_under_test",
+ "traffic_generator"
+ ]
+ }
+ },
+ "type": "object",
+ "properties": {
+ "nodes": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "A unique identifier for this node"
+ },
+ "hostname": {
+ "type": "string",
+ "description": "A hostname from which the node running DTS can access this node. This can also be an IP address."
+ },
+ "user": {
+ "type": "string",
+ "description": "The user to access this node with."
+ },
+ "password": {
+ "type": "string",
+ "description": "The password to use on this node. SSH keys are preferred."
+ }
+ },
+ "additionalProperties": false,
+ "required": [
+ "name",
+ "hostname",
+ "user"
+ ]
+ },
+ "minimum": 1
+ },
+ "executions": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "system_under_test": {
+ "$ref": "#/definitions/node_name"
+ }
+ },
+ "additionalProperties": false,
+ "required": [
+ "system_under_test"
+ ]
+ },
+ "minimum": 1
+ }
+ },
+ "required": [
+ "executions",
+ "nodes"
+ ],
+ "additionalProperties": false
+}
\ No newline at end of file
diff --git a/dts/framework/exception.py b/dts/framework/exception.py
index 54f9f189a4..252fa47fc4 100644
--- a/dts/framework/exception.py
+++ b/dts/framework/exception.py
@@ -58,3 +58,18 @@ def __init__(self, host: str):
def __str__(self) -> str:
return f"SSH session with {self.host} has died"
+
+
+class ConfigParseException(Exception):
+
+ """
+ Configuration file parse failure exception.
+ """
+
+ config: str
+
+ def __init__(self, conf_file: str):
+ self.config = conf_file
+
+ def __str__(self) -> str:
+ return f"Failed to parse config file [{self.config}]"
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
new file mode 100644
index 0000000000..ca3cbd71b1
--- /dev/null
+++ b/dts/framework/settings.py
@@ -0,0 +1,40 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2021 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+import os
+import re
+
+DEFAULT_CONFIG_FILE_PATH: str = "./conf.yaml"
+
+# DTS global environment variables
+DTS_ENV_PAT: str = r"DTS_*"
+DTS_CFG_FILE: str = "DTS_CFG_FILE"
+
+
+def load_global_setting(key: str) -> str:
+ """
+ Load DTS global setting
+ """
+ if re.match(DTS_ENV_PAT, key):
+ env_key = key
+ else:
+ env_key = "DTS_" + key
+
+ return os.environ.get(env_key, "")
+
+
+def get_config_file_path(conf_file_path: str) -> str:
+ """
+ The root path of framework configs.
+ """
+ if conf_file_path == DEFAULT_CONFIG_FILE_PATH:
+ # if the user didn't specify a path on cmdline, they could've specified
+ # it in the env variable
+ conf_file_path = load_global_setting(DTS_CFG_FILE)
+ if conf_file_path == "":
+ conf_file_path = DEFAULT_CONFIG_FILE_PATH
+
+ return conf_file_path
--
2.30.2
next prev parent reply other threads:[~2022-07-11 14:52 UTC|newest]
Thread overview: 105+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-22 12:14 [PATCH v1 0/8] dts: ssh connection to a node Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 1/8] dts: add ssh pexpect library Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 2/8] dts: add locks for parallel node connections Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 3/8] dts: add ssh connection extension Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 4/8] dts: add basic logging facility Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 5/8] dts: add Node base class Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 6/8] dts: add config parser module Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 7/8] dts: add dts runtime workflow module Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 8/8] dts: add main script for running dts Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 0/8] ssh connection to a node Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 1/8] dts: add basic logging facility Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 2/8] dts: add ssh pexpect library Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 3/8] dts: add locks for parallel node connections Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 4/8] dts: add ssh connection extension Juraj Linkeš
2022-07-11 14:51 ` Juraj Linkeš [this message]
2022-07-11 14:51 ` [PATCH v2 6/8] dts: add Node base class Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 7/8] dts: add dts workflow module Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 8/8] dts: add dts executable script Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 0/9] dts: ssh connection to a node Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 1/9] dts: add project tools config Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 2/9] dts: add developer tools Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 3/9] dts: add basic logging facility Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 4/9] dts: add ssh pexpect library Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 5/9] dts: add ssh connection extension Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 6/9] dts: add config parser module Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 7/9] dts: add Node base class Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 8/9] dts: add dts workflow module Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 9/9] dts: add dts executable script Juraj Linkeš
2022-07-29 10:55 ` [PATCH v4 0/9] dts: ssh connection to a node Juraj Linkeš
2022-07-29 10:55 ` [PATCH v4 1/9] dts: add project tools config Juraj Linkeš
2022-08-10 6:30 ` Tu, Lijuan
2022-09-07 16:16 ` Bruce Richardson
2022-09-09 13:38 ` Juraj Linkeš
2022-09-09 13:52 ` Bruce Richardson
2022-09-09 14:13 ` Juraj Linkeš
2022-09-12 14:06 ` Owen Hilyard
2022-09-12 15:15 ` Bruce Richardson
2022-09-13 12:08 ` Juraj Linkeš
2022-09-13 14:18 ` Bruce Richardson
2022-09-13 19:03 ` Honnappa Nagarahalli
2022-09-13 19:19 ` Honnappa Nagarahalli
2022-09-14 9:37 ` Thomas Monjalon
2022-09-14 12:55 ` Juraj Linkeš
2022-09-14 13:11 ` Bruce Richardson
2022-09-14 14:28 ` Thomas Monjalon
2022-09-21 10:49 ` Juraj Linkeš
2022-09-13 19:11 ` Honnappa Nagarahalli
2022-07-29 10:55 ` [PATCH v4 2/9] dts: add developer tools Juraj Linkeš
2022-08-10 6:30 ` Tu, Lijuan
2022-09-07 16:37 ` Bruce Richardson
2022-09-13 12:38 ` Juraj Linkeš
2022-09-13 20:38 ` Honnappa Nagarahalli
2022-09-14 7:37 ` Bruce Richardson
2022-09-14 12:45 ` Juraj Linkeš
2022-09-14 13:13 ` Bruce Richardson
2022-09-14 14:26 ` Thomas Monjalon
2022-09-14 19:08 ` Honnappa Nagarahalli
2022-09-20 12:14 ` Juraj Linkeš
2022-09-20 12:22 ` Tu, Lijuan
2022-07-29 10:55 ` [PATCH v4 3/9] dts: add basic logging facility Juraj Linkeš
2022-08-10 6:31 ` Tu, Lijuan
2022-09-08 8:31 ` Bruce Richardson
2022-09-13 12:52 ` Juraj Linkeš
2022-09-13 23:31 ` Honnappa Nagarahalli
2022-09-14 12:51 ` Juraj Linkeš
2022-07-29 10:55 ` [PATCH v4 4/9] dts: add ssh pexpect library Juraj Linkeš
2022-08-10 6:31 ` Tu, Lijuan
2022-09-08 9:53 ` Bruce Richardson
2022-09-13 13:36 ` Juraj Linkeš
2022-09-13 14:23 ` Bruce Richardson
2022-09-13 14:59 ` Stanislaw Kardach
2022-09-13 17:23 ` Owen Hilyard
2022-09-14 0:03 ` Honnappa Nagarahalli
2022-09-14 7:42 ` Bruce Richardson
2022-09-14 7:58 ` Stanislaw Kardach
2022-09-14 19:57 ` Honnappa Nagarahalli
2022-09-19 14:21 ` Owen Hilyard
2022-09-20 17:54 ` Honnappa Nagarahalli
2022-09-21 1:01 ` Tu, Lijuan
2022-09-21 5:37 ` Jerin Jacob
2022-09-22 9:03 ` Juraj Linkeš
2022-09-14 9:42 ` Stanislaw Kardach
2022-09-22 9:41 ` Juraj Linkeš
2022-09-22 14:32 ` Stanislaw Kardach
2022-09-23 7:22 ` Juraj Linkeš
2022-09-23 8:15 ` Bruce Richardson
2022-09-23 10:18 ` Stanislaw Kardach
2022-07-29 10:55 ` [PATCH v4 5/9] dts: add ssh connection extension Juraj Linkeš
2022-08-10 6:32 ` Tu, Lijuan
2022-09-13 17:04 ` Bruce Richardson
2022-09-13 17:32 ` Owen Hilyard
2022-09-14 7:46 ` Bruce Richardson
2022-09-14 12:02 ` Owen Hilyard
2022-09-14 13:15 ` Bruce Richardson
2022-07-29 10:55 ` [PATCH v4 6/9] dts: add config parser module Juraj Linkeš
2022-08-10 6:33 ` Tu, Lijuan
2022-09-13 17:19 ` Bruce Richardson
2022-09-13 17:47 ` Owen Hilyard
2022-09-14 7:48 ` Bruce Richardson
2022-07-29 10:55 ` [PATCH v4 7/9] dts: add Node base class Juraj Linkeš
2022-08-10 6:33 ` Tu, Lijuan
2022-07-29 10:55 ` [PATCH v4 8/9] dts: add dts workflow module Juraj Linkeš
2022-08-10 6:34 ` Tu, Lijuan
2022-07-29 10:55 ` [PATCH v4 9/9] dts: add dts executable script Juraj Linkeš
2022-08-10 6:35 ` Tu, Lijuan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220711145126.295427-6-juraj.linkes@pantheon.tech \
--to=juraj.linkes@pantheon.tech \
--cc=Honnappa.Nagarahalli@arm.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=jerinjacobk@gmail.com \
--cc=lijuan.tu@intel.com \
--cc=ohilyard@iol.unh.edu \
--cc=ronan.randles@intel.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).