[PATCH v5 00/10] dts: ssh connection to a node

DPDK patches and discussions
 help / color / mirror / Atom feed

* [PATCH v5 00/10] dts: ssh connection to a node
@ 2022-09-26 14:17 Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 01/10] dts: add project tools config Juraj Linkeš
                   ` (10 more replies)
  0 siblings, 11 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

All the necessary code needed to connect to a node in a topology with
a bit more, such as basic logging and some extra useful methods.

To run the code, modify the config file, conf.yaml and execute ./main.py
from the root dts folder. Here's an example config:
executions:
  - system_under_test: "SUT 1"
nodes:
  - name: "SUT 1"
    hostname: 127.0.0.1
    user: root
    password: mypw.change.me

There are configuration files with a README that help with setting up
the execution/development environment.

The code only connects to a node. You'll see logs emitted to console
saying where DTS connected.

There's only a bit of documentation, as there's not much to document.
We'll add some real docs when there's enough functionality to document,
when the HelloWorld testcases is in (point 4 in our roadmap below). What
will be documented later is runtime dependencies and how to set up the DTS
control node environment.

This is our current roadmap:
1. Review this patchset and do the rest of the items in parallel, if
possible.
2. We have extracted the code needed to run the most basic testcase,
HelloWorld, which runs the DPDK Hello World application. We'll split
this along logical/functional boundaries and send after 1 is done.
3. Once we have 2 applied, we're planning on adding a basic functional
testcase - pf_smoke. This send a bit of traffic, so the big addition is
the software traffic generator, Scapy. There's some work already done on
Traffic generators we'll be sending as a dependence on this patch
series.
4. After 3, we'll add a basic performance testcase which doesn't use
Scapy, but Trex or Ixia instead.
5. This is far in the future, but at this point we should have all of
the core functionality in place. What then remains is adding the rest of
the testcases.

We're already working on items 2-4 and we may send more patches even
before this patch series is accepted if that's beneficial. The new
patches would then depend on this patch.

This patch, as well as all others in the pipeline, are the result of
extensive DTS workgroup review which happens internally. If you'd like
us to make it more public we'd have no problem with that.

v3:
Added project config files and developer tools.
Removed locks for parallel nodes, which are not needed now and will be
implemented much later (in a different patch).

v4:
Minor fixes - added missing Exception and utils function.

v5:
Reordered commits because the dependencies between commits changed.
Added more developer tools.
Added definitions of DTS testbed elements.
Reworked SSH implementation - split it so that the split between an
abstraction and the actual implementation is clearer.
Modified the directory structure to better organize the current and the
future code.

Juraj Linkeš (9):
  dts: add project tools config
  dts: add developer tools
  dts: add basic logging facility
  dts: add remote session abstraction
  dts: add ssh connection module
  dts: add node base class
  dts: add dts workflow module
  dts: add dts executable script
  maintainers: add dts maintainers

Owen Hilyard (1):
  dts: add config parser module

 .editorconfig                                 |   2 +-
 .gitignore                                    |   9 +-
 MAINTAINERS                                   |   5 +
 devtools/python-checkpatch.sh                 |  39 ++
 devtools/python-format.sh                     |  54 +++
 devtools/python-lint.sh                       |  26 ++
 doc/guides/contributing/coding_style.rst      |   4 +-
 dts/.devcontainer/devcontainer.json           |  30 ++
 dts/Dockerfile                                |  39 ++
 dts/README.md                                 | 154 ++++++++
 dts/conf.yaml                                 |   6 +
 dts/framework/__init__.py                     |   4 +
 dts/framework/config/__init__.py              |  99 +++++
 dts/framework/config/conf_yaml_schema.json    |  73 ++++
 dts/framework/dts.py                          |  69 ++++
 dts/framework/exception.py                    |  71 ++++
 dts/framework/logger.py                       | 115 ++++++
 dts/framework/remote_session/__init__.py      |   5 +
 .../remote_session/remote_session.py          | 100 +++++
 .../remote_session/session_factory.py         |  16 +
 dts/framework/remote_session/ssh_session.py   | 189 ++++++++++
 dts/framework/settings.py                     | 108 ++++++
 dts/framework/testbed_model/__init__.py       |   8 +
 dts/framework/testbed_model/node.py           |  83 +++++
 dts/framework/utils.py                        |  31 ++
 dts/main.py                                   |  24 ++
 dts/poetry.lock                               | 351 ++++++++++++++++++
 dts/pyproject.toml                            |  55 +++
 28 files changed, 1765 insertions(+), 4 deletions(-)
 create mode 100755 devtools/python-checkpatch.sh
 create mode 100755 devtools/python-format.sh
 create mode 100755 devtools/python-lint.sh
 create mode 100644 dts/.devcontainer/devcontainer.json
 create mode 100644 dts/Dockerfile
 create mode 100644 dts/README.md
 create mode 100644 dts/conf.yaml
 create mode 100644 dts/framework/__init__.py
 create mode 100644 dts/framework/config/__init__.py
 create mode 100644 dts/framework/config/conf_yaml_schema.json
 create mode 100644 dts/framework/dts.py
 create mode 100644 dts/framework/exception.py
 create mode 100644 dts/framework/logger.py
 create mode 100644 dts/framework/remote_session/__init__.py
 create mode 100644 dts/framework/remote_session/remote_session.py
 create mode 100644 dts/framework/remote_session/session_factory.py
 create mode 100644 dts/framework/remote_session/ssh_session.py
 create mode 100644 dts/framework/settings.py
 create mode 100644 dts/framework/testbed_model/__init__.py
 create mode 100644 dts/framework/testbed_model/node.py
 create mode 100644 dts/framework/utils.py
 create mode 100755 dts/main.py
 create mode 100644 dts/poetry.lock
 create mode 100644 dts/pyproject.toml

-- 
2.30.2

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 01/10] dts: add project tools config
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 02/10] dts: add developer tools Juraj Linkeš
                   ` (9 subsequent siblings)
  10 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

Add configuration for Python tools used in DTS:
Poetry, dependency and package manager
Black, formatter
Pylama, static analysis
Isort, import sorting

Add Python and DTS specifics to .gitignore and .editorconfig. Of note is
the change of maximum line length of Python code to 88, which is a good
compromise between shorter files, readability and other considerations.
More in [0].

[0] https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 .editorconfig                            |   2 +-
 .gitignore                               |   9 +-
 doc/guides/contributing/coding_style.rst |   4 +-
 dts/README.md                            |  13 +
 dts/poetry.lock                          | 351 +++++++++++++++++++++++
 dts/pyproject.toml                       |  55 ++++
 6 files changed, 430 insertions(+), 4 deletions(-)
 create mode 100644 dts/README.md
 create mode 100644 dts/poetry.lock
 create mode 100644 dts/pyproject.toml

diff --git a/.editorconfig b/.editorconfig
index ab41c95085..f20996f329 100644
--- a/.editorconfig
+++ b/.editorconfig
@@ -16,7 +16,7 @@ max_line_length = 100
 [*.py]
 indent_style = space
 indent_size = 4
-max_line_length = 79
+max_line_length = 88 # https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length
 
 [meson.build]
 indent_style = space
diff --git a/.gitignore b/.gitignore
index 212c7aa28e..460c5c6f21 100644
--- a/.gitignore
+++ b/.gitignore
@@ -33,8 +33,13 @@ GRTAGS
 tags
 TAGS
 
-# ignore python bytecode files
-*.pyc
+# python byte-compiled/optimized/dll files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# DTS results
+dts/output
 
 # ignore default build directory, and directories from test-meson-builds.sh
 build
diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index 89db6260cf..c86a03e7cc 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -851,7 +851,9 @@ Python Code
 All Python code should be compliant with
 `PEP8 (Style Guide for Python Code) <https://www.python.org/dev/peps/pep-0008/>`_.
 
-The ``pep8`` tool can be used for testing compliance with the guidelines.
+The ``pep8`` tool can be used for testing compliance with the guidelines. Note that the
+maximum line length is 88, as that is a good compromise between shorter files, usability
+with other tools (side-by-side diffs, docs, presentations) and disability accommodation.
 
 Integrating with the Build System
 ---------------------------------
diff --git a/dts/README.md b/dts/README.md
new file mode 100644
index 0000000000..8a334746a7
--- /dev/null
+++ b/dts/README.md
@@ -0,0 +1,13 @@
+# [Poetry](https://python-poetry.org/docs/)
+The typical style of python dependency management, requirements.txt, has a few issues.
+The advantages of Poetry include specifying what python version is required and forcing
+you to specify versions, enforced by a lockfile, both of which help prevent broken
+dependencies. Another benefit is the use of pyproject.toml, which has become the
+standard config file for python projects, improving project organization.
+
+# Python Version
+The Python Version required by DTS is specified in [DTS python config file](./pyproject.toml)
+in the **[tool.poetry.dependencies]** section. Poetry doesn't install Python, so you may
+need to satisfy this requirement if your Python is not up-to-date. A tool such as
+[Pyenv](https://github.com/pyenv/pyenv) is a good way to get Python, though not the only
+one. However, DTS includes a development environment in the form of a Docker image.
diff --git a/dts/poetry.lock b/dts/poetry.lock
new file mode 100644
index 0000000000..46dd5e8933
--- /dev/null
+++ b/dts/poetry.lock
@@ -0,0 +1,351 @@
+[[package]]
+name = "attrs"
+version = "22.1.0"
+description = "Classes Without Boilerplate"
+category = "main"
+optional = false
+python-versions = ">=3.5"
+
+[package.extras]
+dev = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "mypy (>=0.900,!=0.940)", "pytest-mypy-plugins", "zope.interface", "furo", "sphinx", "sphinx-notfound-page", "pre-commit", "cloudpickle"]
+docs = ["furo", "sphinx", "zope.interface", "sphinx-notfound-page"]
+tests = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "mypy (>=0.900,!=0.940)", "pytest-mypy-plugins", "zope.interface", "cloudpickle"]
+tests_no_zope = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "mypy (>=0.900,!=0.940)", "pytest-mypy-plugins", "cloudpickle"]
+
+[[package]]
+name = "black"
+version = "22.8.0"
+description = "The uncompromising code formatter."
+category = "dev"
+optional = false
+python-versions = ">=3.6.2"
+
+[package.dependencies]
+click = ">=8.0.0"
+mypy-extensions = ">=0.4.3"
+pathspec = ">=0.9.0"
+platformdirs = ">=2"
+tomli = {version = ">=1.1.0", markers = "python_full_version < \"3.11.0a7\""}
+
+[package.extras]
+colorama = ["colorama (>=0.4.3)"]
+d = ["aiohttp (>=3.7.4)"]
+jupyter = ["ipython (>=7.8.0)", "tokenize-rt (>=3.2.0)"]
+uvloop = ["uvloop (>=0.15.2)"]
+
+[[package]]
+name = "click"
+version = "8.1.3"
+description = "Composable command line interface toolkit"
+category = "dev"
+optional = false
+python-versions = ">=3.7"
+
+[package.dependencies]
+colorama = {version = "*", markers = "platform_system == \"Windows\""}
+
+[[package]]
+name = "colorama"
+version = "0.4.5"
+description = "Cross-platform colored terminal text."
+category = "dev"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
+
+[[package]]
+name = "isort"
+version = "5.10.1"
+description = "A Python utility / library to sort Python imports."
+category = "dev"
+optional = false
+python-versions = ">=3.6.1,<4.0"
+
+[package.extras]
+pipfile_deprecated_finder = ["pipreqs", "requirementslib"]
+requirements_deprecated_finder = ["pipreqs", "pip-api"]
+colors = ["colorama (>=0.4.3,<0.5.0)"]
+plugins = ["setuptools"]
+
+[[package]]
+name = "jsonpatch"
+version = "1.32"
+description = "Apply JSON-Patches (RFC 6902)"
+category = "main"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
+
+[package.dependencies]
+jsonpointer = ">=1.9"
+
+[[package]]
+name = "jsonpointer"
+version = "2.3"
+description = "Identify specific nodes in a JSON document (RFC 6901)"
+category = "main"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
+
+[[package]]
+name = "jsonschema"
+version = "4.16.0"
+description = "An implementation of JSON Schema validation for Python"
+category = "main"
+optional = false
+python-versions = ">=3.7"
+
+[package.dependencies]
+attrs = ">=17.4.0"
+pyrsistent = ">=0.14.0,<0.17.0 || >0.17.0,<0.17.1 || >0.17.1,<0.17.2 || >0.17.2"
+
+[package.extras]
+format = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3987", "uri-template", "webcolors (>=1.11)"]
+format-nongpl = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3986-validator (>0.1.0)", "uri-template", "webcolors (>=1.11)"]
+
+[[package]]
+name = "mccabe"
+version = "0.7.0"
+description = "McCabe checker, plugin for flake8"
+category = "dev"
+optional = false
+python-versions = ">=3.6"
+
+[[package]]
+name = "mypy"
+version = "0.961"
+description = "Optional static typing for Python"
+category = "dev"
+optional = false
+python-versions = ">=3.6"
+
+[package.dependencies]
+mypy-extensions = ">=0.4.3"
+tomli = {version = ">=1.1.0", markers = "python_version < \"3.11\""}
+typing-extensions = ">=3.10"
+
+[package.extras]
+dmypy = ["psutil (>=4.0)"]
+python2 = ["typed-ast (>=1.4.0,<2)"]
+reports = ["lxml"]
+
+[[package]]
+name = "mypy-extensions"
+version = "0.4.3"
+description = "Experimental type system extensions for programs checked with the mypy typechecker."
+category = "dev"
+optional = false
+python-versions = "*"
+
+[[package]]
+name = "pathspec"
+version = "0.10.1"
+description = "Utility library for gitignore style pattern matching of file paths."
+category = "dev"
+optional = false
+python-versions = ">=3.7"
+
+[[package]]
+name = "pexpect"
+version = "4.8.0"
+description = "Pexpect allows easy control of interactive console applications."
+category = "main"
+optional = false
+python-versions = "*"
+
+[package.dependencies]
+ptyprocess = ">=0.5"
+
+[[package]]
+name = "platformdirs"
+version = "2.5.2"
+description = "A small Python module for determining appropriate platform-specific dirs, e.g. a \"user data dir\"."
+category = "dev"
+optional = false
+python-versions = ">=3.7"
+
+[package.extras]
+docs = ["furo (>=2021.7.5b38)", "proselint (>=0.10.2)", "sphinx-autodoc-typehints (>=1.12)", "sphinx (>=4)"]
+test = ["appdirs (==1.4.4)", "pytest-cov (>=2.7)", "pytest-mock (>=3.6)", "pytest (>=6)"]
+
+[[package]]
+name = "ptyprocess"
+version = "0.7.0"
+description = "Run a subprocess in a pseudo terminal"
+category = "main"
+optional = false
+python-versions = "*"
+
+[[package]]
+name = "pycodestyle"
+version = "2.9.1"
+description = "Python style guide checker"
+category = "dev"
+optional = false
+python-versions = ">=3.6"
+
+[[package]]
+name = "pydocstyle"
+version = "6.1.1"
+description = "Python docstring style checker"
+category = "dev"
+optional = false
+python-versions = ">=3.6"
+
+[package.dependencies]
+snowballstemmer = "*"
+
+[package.extras]
+toml = ["toml"]
+
+[[package]]
+name = "pyflakes"
+version = "2.5.0"
+description = "passive checker of Python programs"
+category = "dev"
+optional = false
+python-versions = ">=3.6"
+
+[[package]]
+name = "pylama"
+version = "8.4.1"
+description = "Code audit tool for python"
+category = "dev"
+optional = false
+python-versions = ">=3.7"
+
+[package.dependencies]
+mccabe = ">=0.7.0"
+pycodestyle = ">=2.9.1"
+pydocstyle = ">=6.1.1"
+pyflakes = ">=2.5.0"
+
+[package.extras]
+all = ["pylint", "eradicate", "radon", "mypy", "vulture"]
+eradicate = ["eradicate"]
+mypy = ["mypy"]
+pylint = ["pylint"]
+radon = ["radon"]
+tests = ["pytest (>=7.1.2)", "pytest-mypy", "eradicate (>=2.0.0)", "radon (>=5.1.0)", "mypy", "pylint (>=2.11.1)", "pylama-quotes", "toml", "vulture", "types-setuptools", "types-toml"]
+toml = ["toml (>=0.10.2)"]
+vulture = ["vulture"]
+
+[[package]]
+name = "pyrsistent"
+version = "0.18.1"
+description = "Persistent/Functional/Immutable data structures"
+category = "main"
+optional = false
+python-versions = ">=3.7"
+
+[[package]]
+name = "pyyaml"
+version = "6.0"
+description = "YAML parser and emitter for Python"
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
+[[package]]
+name = "scapy"
+version = "2.4.5"
+description = "Scapy: interactive packet manipulation tool"
+category = "main"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4"
+
+[package.extras]
+basic = ["ipython"]
+complete = ["ipython", "pyx", "cryptography (>=2.0)", "matplotlib"]
+docs = ["sphinx (>=3.0.0)", "sphinx_rtd_theme (>=0.4.3)", "tox (>=3.0.0)"]
+
+[[package]]
+name = "snowballstemmer"
+version = "2.2.0"
+description = "This package provides 29 stemmers for 28 languages generated from Snowball algorithms."
+category = "dev"
+optional = false
+python-versions = "*"
+
+[[package]]
+name = "toml"
+version = "0.10.2"
+description = "Python Library for Tom's Obvious, Minimal Language"
+category = "dev"
+optional = false
+python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"
+
+[[package]]
+name = "tomli"
+version = "2.0.1"
+description = "A lil' TOML parser"
+category = "dev"
+optional = false
+python-versions = ">=3.7"
+
+[[package]]
+name = "types-pyyaml"
+version = "6.0.11"
+description = "Typing stubs for PyYAML"
+category = "main"
+optional = false
+python-versions = "*"
+
+[[package]]
+name = "typing-extensions"
+version = "4.3.0"
+description = "Backported and Experimental Type Hints for Python 3.7+"
+category = "dev"
+optional = false
+python-versions = ">=3.7"
+
+[[package]]
+name = "warlock"
+version = "2.0.1"
+description = "Python object model built on JSON schema and JSON patch."
+category = "main"
+optional = false
+python-versions = ">=3.7,<4.0"
+
+[package.dependencies]
+jsonpatch = ">=1,<2"
+jsonschema = ">=4,<5"
+
+[metadata]
+lock-version = "1.1"
+python-versions = "^3.10"
+content-hash = "2d2cea9d5018adf0827dc0b98b9a530e9d7cfc206de0d2b726ecae840404bdd5"
+
+[metadata.files]
+attrs = []
+black = []
+click = []
+colorama = []
+isort = []
+jsonpatch = []
+jsonpointer = []
+jsonschema = []
+mccabe = []
+mypy = []
+mypy-extensions = []
+pathspec = []
+pexpect = [
+    {file = "pexpect-4.8.0-py2.py3-none-any.whl", hash = "sha256:0b48a55dcb3c05f3329815901ea4fc1537514d6ba867a152b581d69ae3710937"},
+    {file = "pexpect-4.8.0.tar.gz", hash = "sha256:fc65a43959d153d0114afe13997d439c22823a27cefceb5ff35c2178c6784c0c"},
+]
+platformdirs = [
+    {file = "platformdirs-2.5.2-py3-none-any.whl", hash = "sha256:027d8e83a2d7de06bbac4e5ef7e023c02b863d7ea5d079477e722bb41ab25788"},
+    {file = "platformdirs-2.5.2.tar.gz", hash = "sha256:58c8abb07dcb441e6ee4b11d8df0ac856038f944ab98b7be6b27b2a3c7feef19"},
+]
+ptyprocess = []
+pycodestyle = []
+pydocstyle = []
+pyflakes = []
+pylama = []
+pyrsistent = []
+pyyaml = []
+scapy = []
+snowballstemmer = []
+toml = []
+tomli = []
+types-pyyaml = []
+typing-extensions = []
+warlock = []
diff --git a/dts/pyproject.toml b/dts/pyproject.toml
new file mode 100644
index 0000000000..13bbf5dc34
--- /dev/null
+++ b/dts/pyproject.toml
@@ -0,0 +1,55 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+#
+
+[tool.poetry]
+name = "dts"
+version = "0.1.0"
+description = ""
+authors = ["Owen Hilyard <ohilyard@iol.unh.edu>", "dts@dpdk.org"]
+
+[tool.poetry.dependencies]
+python = "^3.10"
+pexpect = "^4.8.0"
+warlock = "^2.0.1"
+PyYAML = "^6.0"
+types-PyYAML = "^6.0.8"
+scapy = "^2.4.5"
+
+[tool.poetry.dev-dependencies]
+mypy = "^0.961"
+black = "^22.6.0"
+isort = "^5.10.1"
+pylama = "^8.4.1"
+pyflakes = "2.5.0"
+toml = "^0.10.2"
+
+[tool.poetry.scripts]
+dts = "main:main"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.pylama]
+format = "pylint"
+linters = "pep8,pycodestyle,pylint,mccabe,mypy"
+max_line_length = 88 # https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length
+
+[tool.mypy]
+# Scapy is mostly untyped
+ignore_missing_imports = true
+disallow_untyped_defs = true
+disallow_untyped_calls = true
+python_version = "3.10"
+check_untyped_defs = true
+strict_optional = true
+strict_equality = true
+
+[tool.isort]
+profile = "black"
+
+[tool.black]
+line-length = 88 # https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length
+target-version = ['py310']
+include = '\.pyi?$'
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 02/10] dts: add developer tools
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 01/10] dts: add project tools config Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 03/10] dts: add config parser module Juraj Linkeš
                   ` (8 subsequent siblings)
  10 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

The Dockerfile contains basic image for CI and developers. There's also
an integration of the Dockerfile with Visual Studio.

The devtools that check Python code are Black and Isort to format the
code and Pylama to do static analysis.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 devtools/python-checkpatch.sh       |  39 ++++++++
 devtools/python-format.sh           |  54 +++++++++++
 devtools/python-lint.sh             |  26 +++++
 dts/.devcontainer/devcontainer.json |  30 ++++++
 dts/Dockerfile                      |  39 ++++++++
 dts/README.md                       | 141 ++++++++++++++++++++++++++++
 6 files changed, 329 insertions(+)
 create mode 100755 devtools/python-checkpatch.sh
 create mode 100755 devtools/python-format.sh
 create mode 100755 devtools/python-lint.sh
 create mode 100644 dts/.devcontainer/devcontainer.json
 create mode 100644 dts/Dockerfile

diff --git a/devtools/python-checkpatch.sh b/devtools/python-checkpatch.sh
new file mode 100755
index 0000000000..5d9cc6f52b
--- /dev/null
+++ b/devtools/python-checkpatch.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+#
+
+function get_devtools_dir() {
+    dirname "$0"
+}
+
+function main() {
+    DEVTOOLS_DIR="$(get_devtools_dir)"
+    ERRORS=0
+
+    echo "Formatting:"
+    env "$DEVTOOLS_DIR/python-format.sh" -c
+    ERRORS=$(( ERRORS + $? ))
+
+    echo -ne "\n\n"
+    echo "Linting:"
+    env "$DEVTOOLS_DIR/python-lint.sh"
+    ERRORS=$(( ERRORS + $?))
+
+    exit $ERRORS
+}
+
+function usage() {
+    echo "Runs all of the dts devtools scripts."
+    echo "$0 usage:" && grep -P " \w+\)\ #" "$0"
+    exit 0
+}
+
+# There shouldn't be any arguments
+while getopts "" arg; do
+    case $arg in
+    *)
+    esac
+done
+
+main
diff --git a/devtools/python-format.sh b/devtools/python-format.sh
new file mode 100755
index 0000000000..6daf35bf9c
--- /dev/null
+++ b/devtools/python-format.sh
@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+#
+
+function format() {
+    echo "Formatting code with black."
+    black .
+    echo "Sorting imports with isort."
+    isort .
+}
+
+function main() {
+    format
+    exit 0
+}
+
+function check_formatting() {
+    git update-index --refresh
+    retval=$?
+    if [[ $retval -ne 0 ]]
+    then
+        echo 'The "needs update" files have been reformatted.'\
+	     'Please update your commit.'
+    fi
+    exit $retval
+}
+
+function usage() {
+    echo "Automatically formats dts."
+    echo "$0 usage:" && grep -P " \w+\)\ #" $0
+    exit 0
+}
+
+while getopts "h,c,d:" arg; do
+    case $arg in
+    h) # Display this message
+        usage
+        ;;
+
+# Unlike most of these other scripts, format has an argument to control the
+# non-zero exit code. This is to allow you to set it as your IDE's formatting
+# script, since many IDEs are not compatible with formatting scripts which
+# consider changing anything as a failure condition.
+    c) # Exit with a non-zero exit code if any files were not properly formatted.
+        format
+        check_formatting
+        ;;
+    *)
+    esac
+done
+
+echo "Running formatting"
+main
diff --git a/devtools/python-lint.sh b/devtools/python-lint.sh
new file mode 100755
index 0000000000..e9e17867a7
--- /dev/null
+++ b/devtools/python-lint.sh
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+#
+
+function main() {
+    echo "Running static analysis (linting) using pylama."
+    pylama .
+    exit $?
+}
+
+function usage() {
+    echo "Runs pylama, the linter for DTS."
+    echo "Exits with a non-zero exit code if there were errors."
+    exit 1
+}
+
+# There shouldn't be any arguments
+while getopts "" arg; do
+    case $arg in
+    *)
+        usage
+    esac
+done
+
+main
diff --git a/dts/.devcontainer/devcontainer.json b/dts/.devcontainer/devcontainer.json
new file mode 100644
index 0000000000..41ca28fc17
--- /dev/null
+++ b/dts/.devcontainer/devcontainer.json
@@ -0,0 +1,30 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
+// https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/docker-existing-dockerfile
+{
+	"name": "Existing Dockerfile",
+
+	// Sets the run context to one level up instead of the .devcontainer folder.
+	"context": "..",
+
+	// Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
+	"dockerFile": "../Dockerfile",
+
+	// Use 'forwardPorts' to make a list of ports inside the container available locally.
+	// "forwardPorts": [],
+
+	// Uncomment the next line to run commands after the container is created - for example installing curl.
+	"postCreateCommand": "poetry install",
+
+	"extensions": [
+		"ms-python.vscode-pylance",
+	]
+
+	// Uncomment when using a ptrace-based debugger like C++, Go, and Rust
+	// "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ],
+
+	// Uncomment to use the Docker CLI from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker.
+	// "mounts": [ "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ],
+
+	// Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root.
+	// "remoteUser": "vscode"
+}
diff --git a/dts/Dockerfile b/dts/Dockerfile
new file mode 100644
index 0000000000..f3b6652db0
--- /dev/null
+++ b/dts/Dockerfile
@@ -0,0 +1,39 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+#
+
+# There are two Docker images defined in this Dockerfile.
+# One is to be used in CI for automated testing.
+# The other provides a DTS development environment, simplifying Python dependency management.
+
+FROM ubuntu:22.04 AS base
+
+RUN apt-get -y update && apt-get -y upgrade && \
+    apt-get -y install --no-install-recommends \
+        python3 \
+        python3-pip \
+        python3-pexpect \
+        python3-poetry \
+        python3-cachecontrol \
+        openssh-client
+WORKDIR /dpdk/dts
+
+
+FROM base AS runner
+
+# This image is intended to be used as the base for automated systems.
+# It bakes DTS into the image during the build.
+
+COPY . /dpdk/dts
+RUN poetry install --no-dev
+
+CMD ["poetry", "run", "python", "main.py"]
+
+FROM base AS dev
+
+# This image is intended to be used as DTS development environment. It doesn't need C compilation
+# capabilities, only Python dependencies. Once a container mounting DTS using this image is running,
+# the dependencies should be installed using Poetry.
+
+RUN apt-get -y install --no-install-recommends \
+        vim emacs git
diff --git a/dts/README.md b/dts/README.md
index 8a334746a7..86df6b81f4 100644
--- a/dts/README.md
+++ b/dts/README.md
@@ -1,3 +1,36 @@
+DTS environment
+===============
+This README contains helpful steps for setting up your own DTS development or execution
+environment. As DTS is written purely in Python, we only need to download pre-built
+Python packages, obviating the need for build tools. This in turn means that both the
+DTS development and execution environments are the same. DTS environment, DTS
+development environment and DTS execution environment are used interchangeably, as
+they're referring to the same thing.
+
+# DTS definitions
+Before talking about environment setup itself, it's useful to sort out some basic
+definitions:
+* **DTS node**: A generic description of any element/server DTS connects to.
+* **DTS execution environment**: An environment containing Python with packages needed
+   to run DTS.
+* **DTS execution environment node**: A node where at least one DTS execution
+   environment is present. This is the node where we run DTS and from which DTS connects
+   to other nodes.
+* **System under test**: An SUT is the combination of DPDK and the hardware we're
+   testing in conjunction with DPDK (NICs, crypto and other devices).
+* **System under test node**: A node where at least one SUT is present.
+* **Traffic generator**: A TG is either software or hardware capable of sending packets.
+* **Traffic generator node**: A node where at least one TG is present. In case of
+   hardware traffic generators, the TG and the node are literally the same.
+
+In most cases, referring to an execution environment, SUT, TG or the node they're
+running on interchangeably (e.g. using SUT and SUT node interchangeably) doesn't cause
+confusion. There could theoretically be more than of these running on the same node and
+in that case it's useful to have stricter definitions. An example would be two different
+traffic generators (such as Trex and Scapy) running on the same node. A different
+example would be a node containing both a DTS execution environment and a traffic
+generator, in which case it's both a DTS execution environment node and a TG node.
+
 # [Poetry](https://python-poetry.org/docs/)
 The typical style of python dependency management, requirements.txt, has a few issues.
 The advantages of Poetry include specifying what python version is required and forcing
@@ -11,3 +44,111 @@ in the **[tool.poetry.dependencies]** section. Poetry doesn't install Python, so
 need to satisfy this requirement if your Python is not up-to-date. A tool such as
 [Pyenv](https://github.com/pyenv/pyenv) is a good way to get Python, though not the only
 one. However, DTS includes a development environment in the form of a Docker image.
+
+# DTS Environment
+The execution and development environments for DTS are the same, a
+[Docker](https://docs.docker.com/) container defined by our [Dockerfile](./Dockerfile).
+Using a container for the development environment helps with a few things.
+
+1. It helps enforce the boundary between the DTS environment and the TG/SUT, something
+   which caused issues in the past.
+2. It makes creating containers to run DTS inside automated tooling much easier, since
+   they can be based off of a known-working environment that will be updated as DTS is.
+3. It abstracts DTS from the server it is running on. This means that the bare-metal os
+   can be whatever corporate policy or your personal preferences dictate, and DTS does
+   not have to try to support all distros that are supported by DPDK CI.
+4. It makes automated testing for DTS easier, since new dependencies can be sent in with
+  the patches.
+5. It fixes the issue of undocumented dependencies, where some test suites require
+   python libraries that are not installed.
+6. Allows everyone to use the same python version easily, even if they are using a
+   distribution or Windows with out-of-date packages.
+7. Allows you to run the tester on Windows while developing via Docker for Windows.
+
+## Tips for setting up a development environment
+
+### Getting a docker shell
+These commands will give you a bash shell inside the container with all the python
+dependencies installed. This will place you inside a python virtual environment. DTS is
+mounted via a volume, which is essentially a symlink from the host to the container.
+This enables you to edit and run inside the container and then delete the container when
+you are done, keeping your work.
+
+```shell
+docker build --target dev -t dpdk-dts .
+docker run -v $(pwd)/..:/dpdk -it dpdk-dts bash
+$ poetry install
+$ poetry shell
+```
+
+### Vim/Emacs
+Any editor in the ubuntu repos should be easy to use, with vim and emacs already
+installed. You can add your normal config files as a volume, enabling you to use your
+preferred settings.
+
+```shell
+docker run -v ${HOME}/.vimrc:/root/.vimrc -v $(pwd)/..:/dpdk -it dpdk-dts bash
+```
+
+### Visual Studio Code
+VSCode has first-class support for developing with containers. You may need to run the
+non-docker setup commands in the integrated terminal. DTS contains a .devcontainer
+config, so if you open the folder in vscode it should prompt you to use the dev
+container assuming you have the plugin installed. Please refer to
+[VS Development Containers Docs](https://code.visualstudio.com/docs/remote/containers)
+to set it all up.
+
+### Other
+Searching for '$IDE dev containers' will probably lead you in the right direction.
+
+DTS Devtools
+============
+
+# Running the scripts
+These scripts should be run in the [dts](.) directory. You can install their
+dependencies directly, but all the scripts are designed to run in the DTS container
+(specified by [Dockerfile](./Dockerfile)). The .git directory for dpdk must be present
+inside the Dockerfile, meaning you may need to mount the repository as a volume, as
+outlined earlier.
+
+# Script Descriptions
+
+### [../devtools/python-checkpatch.sh](../devtools/python-checkpatch.sh)
+This script runs all the scripts below that provide information on code quality and
+correctness,  exiting with a non-zero exit code if any of the scripts below found any
+issues.
+
+### [../devtools/python-format.sh](../devtools/python-format.sh)
+By default, this script will format all the python code according to the DTS code style
+standards. It will not change the semantics of any code, but fixes many issues around
+whitespace, comment formatting and line length automatically.
+
+This script uses two tools to accomplish this:
+
+* [isort](https://pycqa.github.io/isort/): which alphabetically sorts python imports
+within blocks.
+* [black](https://github.com/psf/black): This tool does most of the actual formatting,
+and works similarly to clang-format.
+
+### [../devtools/python-lint.sh](../devtools/python-lint.sh)
+This script runs [pylama](https://github.com/klen/pylama), which runs a collection of
+python linters and aggregates output. It will run these tools over the repository:
+
+* pycodestyle
+* pylint
+* mccabe
+* mypy
+
+Some lints are disabled due to conflicts with the automatic formatters.
+
+Mypy is not running in strict mode since scapy, an important dependency for packet
+manipulation, inspection and construction, does not have python type annotations. In
+strict mode, this makes mypy fail even an empty file that imports scapy.
+
+# Adding additional scripts
+The shebang MUST be "#!/usr/bin/env bash". Many developers will be working inside a
+python virtual environment, where environment variables are changed to allow multiple
+python versions to coexist on the same system.
+
+If the script provides feedback on code quality or correctness, or can reasonably be
+made to do so, it should be added to dts-checkpatch.sh.
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 03/10] dts: add config parser module
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 01/10] dts: add project tools config Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 02/10] dts: add developer tools Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 04/10] dts: add basic logging facility Juraj Linkeš
                   ` (7 subsequent siblings)
  10 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

From: Owen Hilyard <ohilyard@iol.unh.edu>

The configuration is split into two parts, one defining the parameters
of the test run and the other defining the topology to be used.

The format of the configuration is YAML. It is validated according to a
json schema which also server as detailed documentation of the various
configuration fields. This means that the complete set of allowed values
are tied to the schema as a source of truth. This enables making changes
to parts of DTS that interface with config files without a high risk of
breaking someone's configuration.

This configuration system uses immutable objects to represent the
configuration, making IDE/LSP autocomplete work properly.

There are two ways to specify the configuration file path, an
environment variable or a command line argument, applied in that order.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 dts/conf.yaml                              |  6 ++
 dts/framework/config/__init__.py           | 99 ++++++++++++++++++++++
 dts/framework/config/conf_yaml_schema.json | 73 ++++++++++++++++
 dts/framework/exception.py                 | 23 +++++
 dts/framework/settings.py                  | 84 ++++++++++++++++++
 5 files changed, 285 insertions(+)
 create mode 100644 dts/conf.yaml
 create mode 100644 dts/framework/config/__init__.py
 create mode 100644 dts/framework/config/conf_yaml_schema.json
 create mode 100644 dts/framework/exception.py
 create mode 100644 dts/framework/settings.py

diff --git a/dts/conf.yaml b/dts/conf.yaml
new file mode 100644
index 0000000000..75947dc234
--- /dev/null
+++ b/dts/conf.yaml
@@ -0,0 +1,6 @@
+executions:
+  - system_under_test: "SUT 1"
+nodes:
+  - name: "SUT 1"
+    hostname: sut1.change.me.localhost
+    user: root
diff --git a/dts/framework/config/__init__.py b/dts/framework/config/__init__.py
new file mode 100644
index 0000000000..a0fdffcd77
--- /dev/null
+++ b/dts/framework/config/__init__.py
@@ -0,0 +1,99 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2021 Intel Corporation
+# Copyright(c) 2022 University of New Hampshire
+#
+
+"""
+Generic port and topology nodes configuration file load function
+"""
+import json
+import os.path
+import pathlib
+from dataclasses import dataclass
+from typing import Any, Optional
+
+import warlock
+import yaml
+
+from framework.settings import SETTINGS
+
+
+# Slots enables some optimizations, by pre-allocating space for the defined
+# attributes in the underlying data structure.
+#
+# Frozen makes the object immutable. This enables further optimizations,
+# and makes it thread safe should we every want to move in that direction.
+@dataclass(slots=True, frozen=True)
+class NodeConfiguration:
+    name: str
+    hostname: str
+    user: str
+    password: Optional[str]
+
+    @staticmethod
+    def from_dict(d: dict) -> "NodeConfiguration":
+        return NodeConfiguration(
+            name=d["name"],
+            hostname=d["hostname"],
+            user=d["user"],
+            password=d.get("password"),
+        )
+
+
+@dataclass(slots=True, frozen=True)
+class ExecutionConfiguration:
+    system_under_test: NodeConfiguration
+
+    @staticmethod
+    def from_dict(d: dict, node_map: dict) -> "ExecutionConfiguration":
+        sut_name = d["system_under_test"]
+        assert sut_name in node_map, f"Unknown SUT {sut_name} in execution {d}"
+
+        return ExecutionConfiguration(
+            system_under_test=node_map[sut_name],
+        )
+
+
+@dataclass(slots=True, frozen=True)
+class Configuration:
+    executions: list[ExecutionConfiguration]
+
+    @staticmethod
+    def from_dict(d: dict) -> "Configuration":
+        nodes: list[NodeConfiguration] = list(
+            map(NodeConfiguration.from_dict, d["nodes"])
+        )
+        assert len(nodes) > 0, "There must be a node to test"
+
+        node_map = {node.name: node for node in nodes}
+        assert len(nodes) == len(node_map), "Duplicate node names are not allowed"
+
+        executions: list[ExecutionConfiguration] = list(
+            map(
+                ExecutionConfiguration.from_dict, d["executions"], [node_map for _ in d]
+            )
+        )
+
+        return Configuration(executions=executions)
+
+
+def load_config() -> Configuration:
+    """
+    Loads the configuration file and the configuration file schema,
+    validates the configuration file, and creates a configuration object.
+    """
+    with open(SETTINGS.config_file_path, "r") as f:
+        config_data = yaml.safe_load(f)
+
+    schema_path = os.path.join(
+        pathlib.Path(__file__).parent.resolve(), "conf_yaml_schema.json"
+    )
+
+    with open(schema_path, "r") as f:
+        schema = json.load(f)
+    config: dict[str, Any] = warlock.model_factory(schema, name="_Config")(config_data)
+    config_obj: Configuration = Configuration.from_dict(dict(config))
+    return config_obj
+
+
+CONFIGURATION = load_config()
diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
new file mode 100644
index 0000000000..53c9058a4c
--- /dev/null
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -0,0 +1,73 @@
+{
+  "$schema": "https://json-schema.org/draft-07/schema",
+  "title": "DPDK DTS Config Schema",
+  "definitions": {
+    "node_name": {
+      "type": "string",
+      "description": "A unique identifier for a node"
+    },
+    "node_role": {
+      "type": "string",
+      "description": "The role a node plays in DTS",
+      "enum": [
+        "system_under_test",
+        "traffic_generator"
+      ]
+    }
+  },
+  "type": "object",
+  "properties": {
+    "nodes": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "name": {
+            "type": "string",
+            "description": "A unique identifier for this node"
+          },
+          "hostname": {
+            "type": "string",
+            "description": "A hostname from which the node running DTS can access this node. This can also be an IP address."
+          },
+          "user": {
+            "type": "string",
+            "description": "The user to access this node with."
+          },
+          "password": {
+            "type": "string",
+            "description": "The password to use on this node. Use only as a last resort. SSH keys are STRONGLY preferred."
+          }
+        },
+        "additionalProperties": false,
+        "required": [
+          "name",
+          "hostname",
+          "user"
+        ]
+      },
+      "minimum": 1
+    },
+    "executions": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "system_under_test": {
+            "$ref": "#/definitions/node_name"
+          }
+        },
+        "additionalProperties": false,
+        "required": [
+          "system_under_test"
+        ]
+      },
+      "minimum": 1
+    }
+  },
+  "required": [
+    "executions",
+    "nodes"
+  ],
+  "additionalProperties": false
+}
diff --git a/dts/framework/exception.py b/dts/framework/exception.py
new file mode 100644
index 0000000000..60fd98c9ca
--- /dev/null
+++ b/dts/framework/exception.py
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+"""
+User-defined exceptions used across the framework.
+"""
+
+
+class ConfigParseException(Exception):
+    """
+    Configuration file parse failure exception.
+    """
+
+    config: str
+
+    def __init__(self, conf_file: str):
+        self.config = conf_file
+
+    def __str__(self) -> str:
+        return f"Failed to parse config file [{self.config}]"
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
new file mode 100644
index 0000000000..d1a955502b
--- /dev/null
+++ b/dts/framework/settings.py
@@ -0,0 +1,84 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2021 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+import argparse
+import os
+from dataclasses import dataclass
+from typing import Any, Callable, Iterable, Optional, Sequence, TypeVar
+
+_T = TypeVar("_T")
+
+
+def _env_arg(env_var: str) -> Any:
+    class _EnvironmentArgument(argparse.Action):
+        def __init__(
+            self,
+            option_strings: Sequence[str],
+            dest: str,
+            nargs: Optional[str | int] = None,
+            const: Optional[str] = None,
+            default: str = None,
+            type: Callable[[str], Optional[_T | argparse.FileType]] = None,
+            choices: Optional[Iterable[_T]] = None,
+            required: bool = True,
+            help: Optional[str] = None,
+            metavar: Optional[str | tuple[str, ...]] = None,
+        ) -> None:
+            env_var_value = os.environ.get(env_var)
+            default = env_var_value or default
+            super(_EnvironmentArgument, self).__init__(
+                option_strings,
+                dest,
+                nargs=nargs,
+                const=const,
+                default=default,
+                type=type,
+                choices=choices,
+                required=required,
+                help=help,
+                metavar=metavar,
+            )
+
+        def __call__(
+            self,
+            parser: argparse.ArgumentParser,
+            namespace: argparse.Namespace,
+            values: Any,
+            option_string: str = None,
+        ) -> None:
+            setattr(namespace, self.dest, values)
+
+    return _EnvironmentArgument
+
+
+@dataclass(slots=True, frozen=True)
+class _Settings:
+    config_file_path: str
+
+
+def _get_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(description="DPDK test framework.")
+
+    parser.add_argument(
+        "--config-file",
+        action=_env_arg("DTS_CFG_FILE"),
+        default="conf.yaml",
+        required=False,
+        help="[DTS_CFG_FILE] configuration file that describes the test cases, SUTs "
+        "and targets.",
+    )
+
+    return parser
+
+
+def _get_settings() -> _Settings:
+    parsed_args = _get_parser().parse_args()
+    return _Settings(
+        config_file_path=parsed_args.config_file,
+    )
+
+
+SETTINGS: _Settings = _get_settings()
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 04/10] dts: add basic logging facility
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
                   ` (2 preceding siblings ...)
  2022-09-26 14:17 ` [PATCH v5 03/10] dts: add config parser module Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 05/10] dts: add remote session abstraction Juraj Linkeš
                   ` (6 subsequent siblings)
  10 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

The logging module provides loggers distinguished by two attributes,
a custom format and a verbosity switch. The loggers log to both console
and more verbosely to files.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 dts/framework/__init__.py |   4 ++
 dts/framework/logger.py   | 115 ++++++++++++++++++++++++++++++++++++++
 dts/framework/settings.py |  12 ++++
 3 files changed, 131 insertions(+)
 create mode 100644 dts/framework/__init__.py
 create mode 100644 dts/framework/logger.py

diff --git a/dts/framework/__init__.py b/dts/framework/__init__.py
new file mode 100644
index 0000000000..6cdc0639db
--- /dev/null
+++ b/dts/framework/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
diff --git a/dts/framework/logger.py b/dts/framework/logger.py
new file mode 100644
index 0000000000..6917006115
--- /dev/null
+++ b/dts/framework/logger.py
@@ -0,0 +1,115 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+import logging
+import os.path
+from typing import TypedDict
+
+from .settings import SETTINGS
+
+"""
+DTS logger module with several log level. DTS framework and TestSuite logs
+are saved in different log files.
+"""
+date_fmt = "%Y/%m/%d %H:%M:%S"
+stream_fmt = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+
+
+class LoggerDictType(TypedDict):
+    logger: "DTSLOG"
+    name: str
+    node: str
+
+
+# List for saving all using loggers
+global Loggers
+Loggers: list[LoggerDictType] = []
+
+
+class DTSLOG(logging.LoggerAdapter):
+    """
+    DTS log class for framework and testsuite.
+    """
+
+    node: str
+    logger: logging.Logger
+    sh: logging.StreamHandler
+    fh: logging.FileHandler
+    verbose_fh: logging.FileHandler
+
+    def __init__(self, logger: logging.Logger, node: str = "suite"):
+        self.logger = logger
+        # 1 means log everything, this will be used by file handlers if their level
+        # is not set
+        self.logger.setLevel(1)
+
+        self.node = node
+
+        # add handler to emit to stdout
+        sh = logging.StreamHandler()
+        sh.setFormatter(logging.Formatter(stream_fmt, date_fmt))
+        sh.setLevel(logging.INFO)  # console handler default level
+
+        if SETTINGS.verbose is True:
+            sh.setLevel(logging.DEBUG)
+
+        self.logger.addHandler(sh)
+        self.sh = sh
+
+        if not os.path.exists("output"):
+            os.mkdir("output")
+
+        fh = logging.FileHandler(f"output/{node}.log")
+        fh.setFormatter(
+            logging.Formatter(
+                fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
+                datefmt=date_fmt,
+            )
+        )
+
+        self.logger.addHandler(fh)
+        self.fh = fh
+
+        # This outputs EVERYTHING, intended for post-mortem debugging
+        # Also optimized for processing via AWK (awk -F '|' ...)
+        verbose_fh = logging.FileHandler(f"output/{node}.verbose.log")
+        verbose_fh.setFormatter(
+            logging.Formatter(
+                fmt="%(asctime)s|%(name)s|%(levelname)s|%(pathname)s|%(lineno)d|"
+                "%(funcName)s|%(process)d|%(thread)d|%(threadName)s|%(message)s",
+                datefmt=date_fmt,
+            )
+        )
+
+        self.logger.addHandler(verbose_fh)
+        self.verbose_fh = verbose_fh
+
+        super(DTSLOG, self).__init__(self.logger, dict(node=self.node))
+
+    def logger_exit(self) -> None:
+        """
+        Remove stream handler and logfile handler.
+        """
+        for handler in (self.sh, self.fh, self.verbose_fh):
+            handler.flush()
+            self.logger.removeHandler(handler)
+
+
+def getLogger(name: str, node: str = "suite") -> DTSLOG:
+    """
+    Get logger handler and if there's no handler for specified Node will create one.
+    """
+    global Loggers
+    # return saved logger
+    logger: LoggerDictType
+    for logger in Loggers:
+        if logger["name"] == name and logger["node"] == node:
+            return logger["logger"]
+
+    # return new logger
+    dts_logger: DTSLOG = DTSLOG(logging.getLogger(name), node)
+    Loggers.append({"logger": dts_logger, "name": name, "node": node})
+    return dts_logger
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index d1a955502b..ed43ba33a0 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -57,6 +57,7 @@ def __call__(
 @dataclass(slots=True, frozen=True)
 class _Settings:
     config_file_path: str
+    verbose: bool
 
 
 def _get_parser() -> argparse.ArgumentParser:
@@ -71,6 +72,16 @@ def _get_parser() -> argparse.ArgumentParser:
         "and targets.",
     )
 
+    parser.add_argument(
+        "-v",
+        "--verbose",
+        action=_env_arg("DTS_VERBOSE"),
+        default="N",
+        required=False,
+        help="[DTS_VERBOSE] Set to 'Y' to enable verbose output, logging all messages "
+        "to the console.",
+    )
+
     return parser
 
 
@@ -78,6 +89,7 @@ def _get_settings() -> _Settings:
     parsed_args = _get_parser().parse_args()
     return _Settings(
         config_file_path=parsed_args.config_file,
+        verbose=(parsed_args.verbose == "Y"),
     )
 
 
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 05/10] dts: add remote session abstraction
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
                   ` (3 preceding siblings ...)
  2022-09-26 14:17 ` [PATCH v5 04/10] dts: add basic logging facility Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 06/10] dts: add ssh connection module Juraj Linkeš
                   ` (5 subsequent siblings)
  10 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

The abstraction allows for easy switching of implementations of remote
connections (ssh, telnet, etc.). It implements some common features,
such as logging of commands and their outputs and history bookkeeping
and defines methods that must be implemented by derived classes.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 dts/framework/remote_session/__init__.py      |   5 +
 .../remote_session/remote_session.py          | 100 ++++++++++++++++++
 dts/framework/settings.py                     |  12 +++
 3 files changed, 117 insertions(+)
 create mode 100644 dts/framework/remote_session/__init__.py
 create mode 100644 dts/framework/remote_session/remote_session.py

diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py
new file mode 100644
index 0000000000..ecf2f5099f
--- /dev/null
+++ b/dts/framework/remote_session/__init__.py
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+#
+
+from .session_factory import RemoteSession, create_remote_session
diff --git a/dts/framework/remote_session/remote_session.py b/dts/framework/remote_session/remote_session.py
new file mode 100644
index 0000000000..8b55c04a95
--- /dev/null
+++ b/dts/framework/remote_session/remote_session.py
@@ -0,0 +1,100 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+import dataclasses
+from abc import ABC, abstractmethod
+from typing import Optional
+
+from framework.config import NodeConfiguration
+from framework.logger import DTSLOG
+from framework.settings import SETTINGS
+
+
+@dataclasses.dataclass(slots=True, frozen=True)
+class HistoryRecord:
+    command: str
+    name: str
+    output: str | int
+
+
+class RemoteSession(ABC):
+    _node_config: NodeConfiguration
+    hostname: str
+    username: str
+    password: str
+    ip: str
+    port: Optional[int]
+    name: str
+    logger: DTSLOG
+    history: list[HistoryRecord]
+
+    def __init__(
+        self,
+        node_config: NodeConfiguration,
+        session_name: str,
+        logger: DTSLOG,
+    ):
+        self._node_config = node_config
+        self.logger = logger
+        self.name = session_name
+
+        self.hostname = node_config.hostname
+        self.ip = self.hostname
+        self.port = None
+        if ":" in self.hostname:
+            self.ip, port = self.hostname.split(":")
+            self.port = int(port)
+
+        self.username = node_config.user
+        self.password = node_config.password or ""
+        self.logger.info(f"Remote {self.username}@{self.hostname}")
+        self.history = []
+
+        self._connect()
+
+    def _history_add(self, command: str, output: str) -> None:
+        self.history.append(
+            HistoryRecord(command=command, name=self.name, output=output)
+        )
+
+    def send_command(self, command: str, timeout: float = SETTINGS.timeout) -> str:
+        self.logger.info(f"Sending: {command}")
+        out = self._send_command(command, timeout)
+        self.logger.debug(f"Received from {command}: {out}")
+        self._history_add(command=command, output=out)
+        return out
+
+    def close(self, force: bool = False) -> None:
+        self.logger.logger_exit()
+        self._close(force)
+
+    @abstractmethod
+    def _connect(self) -> None:
+        """
+        Create connection to assigned node.
+        """
+        pass
+
+    @abstractmethod
+    def _send_command(self, command: str, timeout: float) -> str:
+        """
+        Send a command and return the output.
+        """
+        pass
+
+    @abstractmethod
+    def _close(self, force: bool = False) -> None:
+        """
+        Close the remote session, freeing all used resources.
+        """
+        pass
+
+    @abstractmethod
+    def is_alive(self) -> bool:
+        """
+        Check whether the session is still responding.
+        """
+        pass
diff --git a/dts/framework/settings.py b/dts/framework/settings.py
index ed43ba33a0..39be3808ff 100644
--- a/dts/framework/settings.py
+++ b/dts/framework/settings.py
@@ -58,6 +58,7 @@ def __call__(
 class _Settings:
     config_file_path: str
     verbose: bool
+    timeout: float
 
 
 def _get_parser() -> argparse.ArgumentParser:
@@ -82,6 +83,16 @@ def _get_parser() -> argparse.ArgumentParser:
         "to the console.",
     )
 
+    parser.add_argument(
+        "-t",
+        "--timeout",
+        action=_env_arg("DTS_TIMEOUT"),
+        default=15,
+        required=False,
+        help="[DTS_TIMEOUT] The default timeout for all DTS operations except for "
+        "compiling DPDK.",
+    )
+
     return parser
 
 
@@ -90,6 +101,7 @@ def _get_settings() -> _Settings:
     return _Settings(
         config_file_path=parsed_args.config_file,
         verbose=(parsed_args.verbose == "Y"),
+        timeout=float(parsed_args.timeout),
     )
 
 
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 06/10] dts: add ssh connection module
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
                   ` (4 preceding siblings ...)
  2022-09-26 14:17 ` [PATCH v5 05/10] dts: add remote session abstraction Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-09-27 10:12   ` Stanislaw Kardach
  2022-09-26 14:17 ` [PATCH v5 07/10] dts: add node base class Juraj Linkeš
                   ` (4 subsequent siblings)
  10 siblings, 1 reply; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

The module uses the pexpect python library and implements connection to
a node and two ways to interact with the node:
1. Send a string with specified prompt which will be matched after
   the string has been sent to the node.
2. Send a command to be executed. No prompt is specified here.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 dts/framework/exception.py                    |  48 +++++
 .../remote_session/session_factory.py         |  16 ++
 dts/framework/remote_session/ssh_session.py   | 189 ++++++++++++++++++
 dts/framework/utils.py                        |  13 ++
 4 files changed, 266 insertions(+)
 create mode 100644 dts/framework/remote_session/session_factory.py
 create mode 100644 dts/framework/remote_session/ssh_session.py
 create mode 100644 dts/framework/utils.py

diff --git a/dts/framework/exception.py b/dts/framework/exception.py
index 60fd98c9ca..8466990aa5 100644
--- a/dts/framework/exception.py
+++ b/dts/framework/exception.py
@@ -9,6 +9,54 @@
 """
 
 
+class TimeoutException(Exception):
+    """
+    Command execution timeout.
+    """
+
+    command: str
+    output: str
+
+    def __init__(self, command: str, output: str):
+        self.command = command
+        self.output = output
+
+    def __str__(self) -> str:
+        return f"TIMEOUT on {self.command}"
+
+    def get_output(self) -> str:
+        return self.output
+
+
+class SSHConnectionException(Exception):
+    """
+    SSH connection error.
+    """
+
+    host: str
+
+    def __init__(self, host: str):
+        self.host = host
+
+    def __str__(self) -> str:
+        return f"Error trying to connect with {self.host}"
+
+
+class SSHSessionDeadException(Exception):
+    """
+    SSH session is not alive.
+    It can no longer be used.
+    """
+
+    host: str
+
+    def __init__(self, host: str):
+        self.host = host
+
+    def __str__(self) -> str:
+        return f"SSH session with {self.host} has died"
+
+
 class ConfigParseException(Exception):
     """
     Configuration file parse failure exception.
diff --git a/dts/framework/remote_session/session_factory.py b/dts/framework/remote_session/session_factory.py
new file mode 100644
index 0000000000..ff05df97bf
--- /dev/null
+++ b/dts/framework/remote_session/session_factory.py
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+from framework.config import NodeConfiguration
+from framework.logger import DTSLOG
+
+from .remote_session import RemoteSession
+from .ssh_session import SSHSession
+
+
+def create_remote_session(
+    node_config: NodeConfiguration, name: str, logger: DTSLOG
+) -> RemoteSession:
+    return SSHSession(node_config, name, logger)
diff --git a/dts/framework/remote_session/ssh_session.py b/dts/framework/remote_session/ssh_session.py
new file mode 100644
index 0000000000..e0614e0f90
--- /dev/null
+++ b/dts/framework/remote_session/ssh_session.py
@@ -0,0 +1,189 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+
+import time
+
+from pexpect import pxssh
+
+from framework.config import NodeConfiguration
+from framework.exception import (
+    SSHConnectionException,
+    SSHSessionDeadException,
+    TimeoutException,
+)
+from framework.logger import DTSLOG
+from framework.utils import GREEN, RED
+
+from .remote_session import RemoteSession
+
+
+class SSHSession(RemoteSession):
+    """
+    Module for creating Pexpect SSH sessions to a node.
+    """
+
+    session: pxssh.pxssh
+    magic_prompt: str
+
+    def __init__(
+        self,
+        node_config: NodeConfiguration,
+        session_name: str,
+        logger: DTSLOG,
+    ):
+        self.magic_prompt = "MAGIC PROMPT"
+        super(SSHSession, self).__init__(node_config, session_name, logger)
+
+    def _connect(self) -> None:
+        """
+        Create connection to assigned node.
+        """
+        retry_attempts = 10
+        login_timeout = 20 if self.port else 10
+        password_regex = (
+            r"(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)"
+        )
+        try:
+            for retry_attempt in range(retry_attempts):
+                self.session = pxssh.pxssh(encoding="utf-8")
+                try:
+                    self.session.login(
+                        self.ip,
+                        self.username,
+                        self.password,
+                        original_prompt="[$#>]",
+                        port=self.port,
+                        login_timeout=login_timeout,
+                        password_regex=password_regex,
+                    )
+                    break
+                except Exception as e:
+                    print(e)
+                    time.sleep(2)
+                    print(f"Retrying connection: retry number {retry_attempt + 1}.")
+            else:
+                raise Exception(f"Connection to {self.hostname} failed")
+
+            self.logger.info(f"Connection to {self.hostname} succeeded")
+            self.send_expect("stty -echo", "#")
+            self.send_expect("stty columns 1000", "#")
+        except Exception as e:
+            print(RED(str(e)))
+            if getattr(self, "port", None):
+                suggestion = (
+                    f"\nSuggestion: Check if the firewall on {self.hostname} is "
+                    f"stopped.\n"
+                )
+                print(GREEN(suggestion))
+
+            raise SSHConnectionException(self.hostname)
+
+    def send_expect_base(self, command: str, prompt: str, timeout: float) -> str:
+        self.clean_session()
+        original_prompt = self.session.PROMPT
+        self.session.PROMPT = prompt
+        self.__sendline(command)
+        self.__prompt(command, timeout)
+
+        before = self._get_output()
+        self.session.PROMPT = original_prompt
+        return before
+
+    def send_expect(
+        self, command: str, prompt: str, timeout: float = 15, verify: bool = False
+    ) -> str | int:
+        try:
+            ret = self.send_expect_base(command, prompt, timeout)
+            if verify:
+                ret_status = self.send_expect_base("echo $?", prompt, timeout)
+                try:
+                    retval = int(ret_status)
+                    if not retval:
+                        self.logger.error(f"Command: {command} failure!")
+                        self.logger.error(ret)
+                        return retval
+                    else:
+                        return ret
+                except ValueError:
+                    return ret
+            else:
+                return ret
+        except Exception as e:
+            print(
+                f"Exception happened in [{command}] and output is "
+                f"[{self._get_output()}]"
+            )
+            raise e
+
+    def _send_command(self, command: str, timeout: float = 1) -> str:
+        try:
+            self.clean_session()
+            self.__sendline(command)
+        except Exception as e:
+            raise e
+
+        output = self.get_output(timeout=timeout)
+        self.session.PROMPT = self.session.UNIQUE_PROMPT
+        self.session.prompt(0.1)
+
+        return output
+
+    def clean_session(self) -> None:
+        self.get_output(timeout=0.01)
+
+    def _get_output(self) -> str:
+        if not self.is_alive():
+            raise SSHSessionDeadException(self.hostname)
+        before = self.session.before.rsplit("\r\n", 1)[0]
+        if before == "[PEXPECT]":
+            return ""
+        return before
+
+    def get_output(self, timeout: float = 15) -> str:
+        """
+        Get all output before timeout
+        """
+        self.session.PROMPT = self.magic_prompt
+        try:
+            self.session.prompt(timeout)
+        except Exception:
+            pass
+
+        before = self._get_output()
+        self.__flush()
+
+        self.logger.debug(before)
+        return before
+
+    def __flush(self) -> None:
+        """
+        Clear all session buffer
+        """
+        self.session.buffer = ""
+        self.session.before = ""
+
+    def __prompt(self, command: str, timeout: float) -> None:
+        if not self.session.prompt(timeout):
+            raise TimeoutException(command, self._get_output()) from None
+
+    def __sendline(self, command: str) -> None:
+        if not self.is_alive():
+            raise SSHSessionDeadException(self.hostname)
+        if len(command) == 2 and command.startswith("^"):
+            self.session.sendcontrol(command[1])
+        else:
+            self.session.sendline(command)
+
+    def _close(self, force: bool = False) -> None:
+        if force is True:
+            self.session.close()
+        else:
+            if self.is_alive():
+                self.session.logout()
+
+    def is_alive(self) -> bool:
+        return self.session.isalive()
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
new file mode 100644
index 0000000000..26b784ebb5
--- /dev/null
+++ b/dts/framework/utils.py
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+
+def RED(text: str) -> str:
+    return f"\u001B[31;1m{str(text)}\u001B[0m"
+
+
+def GREEN(text: str) -> str:
+    return f"\u001B[32;1m{str(text)}\u001B[0m"
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 07/10] dts: add node base class
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
                   ` (5 preceding siblings ...)
  2022-09-26 14:17 ` [PATCH v5 06/10] dts: add ssh connection module Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 08/10] dts: add dts workflow module Juraj Linkeš
                   ` (3 subsequent siblings)
  10 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

The base class implements basic node management methods - connect and
execute commands.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 dts/framework/testbed_model/__init__.py |  8 +++
 dts/framework/testbed_model/node.py     | 83 +++++++++++++++++++++++++
 2 files changed, 91 insertions(+)
 create mode 100644 dts/framework/testbed_model/__init__.py
 create mode 100644 dts/framework/testbed_model/node.py

diff --git a/dts/framework/testbed_model/__init__.py b/dts/framework/testbed_model/__init__.py
new file mode 100644
index 0000000000..63b13511a8
--- /dev/null
+++ b/dts/framework/testbed_model/__init__.py
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+#
+
+"""
+This module contains the classes used to model the physical traffic generator,
+sut and any other components that need to be interacted with.
+"""
diff --git a/dts/framework/testbed_model/node.py b/dts/framework/testbed_model/node.py
new file mode 100644
index 0000000000..7bc298d9eb
--- /dev/null
+++ b/dts/framework/testbed_model/node.py
@@ -0,0 +1,83 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+from typing import Optional
+
+from framework.config import NodeConfiguration
+from framework.logger import DTSLOG, getLogger
+from framework.remote_session import RemoteSession, create_remote_session
+from framework.settings import SETTINGS
+
+"""
+A node is a generic host that DTS connects to and manages.
+"""
+
+
+class Node(object):
+    """
+    Basic module for node management. This module implements methods that
+    manage a node, such as information gathering (of CPU/PCI/NIC) and
+    environment setup.
+    """
+
+    main_session: RemoteSession
+    name: str
+    logger: DTSLOG
+    _config: NodeConfiguration
+    _other_sessions: list[RemoteSession]
+
+    def __init__(self, node_config: NodeConfiguration):
+        self._config = node_config
+        self.name = node_config.name
+
+        self.logger = getLogger(self.name)
+        self.logger.info(f"Created node: {self.name}")
+        self.main_session = create_remote_session(self._config, self.name, self.logger)
+        self._other_sessions = []
+
+    def get_ip_address(self) -> str:
+        """
+        Get SUT's ip address.
+        """
+        return self._config.hostname
+
+    def get_password(self) -> Optional[str]:
+        """
+        Get SUT's login password.
+        """
+        return self._config.password
+
+    def get_username(self) -> str:
+        """
+        Get SUT's login username.
+        """
+        return self._config.user
+
+    def send_command(self, cmds: str, timeout: float = SETTINGS.timeout) -> str:
+        """
+        Send commands to node and return string before timeout.
+        """
+
+        return self.main_session.send_command(cmds, timeout)
+
+    def create_session(self, name: str) -> RemoteSession:
+        connection = create_remote_session(
+            self._config,
+            name,
+            getLogger(name, node=self.name),
+        )
+        self._other_sessions.append(connection)
+        return connection
+
+    def node_exit(self) -> None:
+        """
+        Recover all resource before node exit
+        """
+        if self.main_session:
+            self.main_session.close()
+        for session in self._other_sessions:
+            session.close()
+        self.logger.logger_exit()
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 08/10] dts: add dts workflow module
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
                   ` (6 preceding siblings ...)
  2022-09-26 14:17 ` [PATCH v5 07/10] dts: add node base class Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 09/10] dts: add dts executable script Juraj Linkeš
                   ` (2 subsequent siblings)
  10 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

The module implements methods needed to run DTS. It handles the creation
of objects and eventually the whole DTS workflow, such as running node
setups, test gathering, setup and execution and various cleanups.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 dts/framework/dts.py   | 69 ++++++++++++++++++++++++++++++++++++++++++
 dts/framework/utils.py | 18 +++++++++++
 2 files changed, 87 insertions(+)
 create mode 100644 dts/framework/dts.py

diff --git a/dts/framework/dts.py b/dts/framework/dts.py
new file mode 100644
index 0000000000..6b3ec069f6
--- /dev/null
+++ b/dts/framework/dts.py
@@ -0,0 +1,69 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2019 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+import sys
+import traceback
+from typing import Iterable, Optional
+
+from framework import logger
+from framework.testbed_model.node import Node
+
+from .config import CONFIGURATION
+from .logger import getLogger
+from .utils import check_dts_python_version
+
+log_handler: Optional[logger.DTSLOG] = None
+
+
+def run_all() -> None:
+    """
+    Main process of DTS, it will run all test suites in the config file.
+    """
+
+    global log_handler
+
+    # check the python version of the server that run dts
+    check_dts_python_version()
+
+    log_handler = getLogger("dts")
+
+    nodes = {}
+    # This try/finally block means "Run the try block, if there is an exception,
+    # run the finally block before passing it upward. If there is not an exception,
+    # run the finally block after the try block is finished." This helps avoid the
+    # problem of python's interpreter exit context, which essentially prevents you
+    # from making certain system calls. This makes cleaning up resources difficult,
+    # since most of the resources in DTS are network-based, which is restricted.
+    try:
+        # for all Execution sections
+        for execution in CONFIGURATION.executions:
+            sut_config = execution.system_under_test
+            if sut_config.name not in nodes:
+                node = Node(sut_config)
+                nodes[sut_config.name] = node
+                out = node.send_command("echo Hello World")
+
+    except Exception as e:
+        # sys.exit() doesn't produce a stack trace, need to print it explicitly
+        traceback.print_exc()
+        raise e
+
+    finally:
+        quit_execution(nodes.values())
+
+
+def quit_execution(sut_nodes: Iterable[Node]) -> None:
+    """
+    Close session to SUT and TG before quit.
+    Return exit status when failure occurred.
+    """
+    for sut_node in sut_nodes:
+        # close all session
+        sut_node.node_exit()
+
+    if log_handler is not None:
+        log_handler.info("DTS ended")
+    sys.exit(0)
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index 26b784ebb5..2a174831d0 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -4,6 +4,8 @@
 # Copyright(c) 2022 University of New Hampshire
 #
 
+import sys
+
 
 def RED(text: str) -> str:
     return f"\u001B[31;1m{str(text)}\u001B[0m"
@@ -11,3 +13,19 @@ def RED(text: str) -> str:
 
 def GREEN(text: str) -> str:
     return f"\u001B[32;1m{str(text)}\u001B[0m"
+
+
+def check_dts_python_version() -> None:
+    if sys.version_info.major < 3 or (
+        sys.version_info.major == 3 and sys.version_info.minor < 10
+    ):
+        print(
+            RED(
+                (
+                    "WARNING: DTS execution node's python version is lower than"
+                    "python 3.10, is deprecated and will not work in future releases."
+                )
+            ),
+            file=sys.stderr,
+        )
+        print(RED("Please use Python >= 3.10 instead"), file=sys.stderr)
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 09/10] dts: add dts executable script
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
                   ` (7 preceding siblings ...)
  2022-09-26 14:17 ` [PATCH v5 08/10] dts: add dts workflow module Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-09-26 14:17 ` [PATCH v5 10/10] maintainers: add dts maintainers Juraj Linkeš
  2022-10-11 15:40 ` [PATCH v5 00/10] dts: ssh connection to a node Owen Hilyard
  10 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

The script is an interface to run DTS.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 dts/main.py | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
 create mode 100755 dts/main.py

diff --git a/dts/main.py b/dts/main.py
new file mode 100755
index 0000000000..a700707650
--- /dev/null
+++ b/dts/main.py
@@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+# Copyright(c) 2022 University of New Hampshire
+#
+
+"""
+A test framework for testing DPDK.
+"""
+
+import logging
+
+from framework import dts
+
+
+def main() -> None:
+    dts.run_all()
+
+
+# Main program begins here
+if __name__ == "__main__":
+    logging.raiseExceptions = True
+    main()
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH v5 10/10] maintainers: add dts maintainers
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
                   ` (8 preceding siblings ...)
  2022-09-26 14:17 ` [PATCH v5 09/10] dts: add dts executable script Juraj Linkeš
@ 2022-09-26 14:17 ` Juraj Linkeš
  2022-10-11 15:40 ` [PATCH v5 00/10] dts: ssh connection to a node Owen Hilyard
  10 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-26 14:17 UTC (permalink / raw)
  To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, kda, bruce.richardson
  Cc: dev, Juraj Linkeš

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 MAINTAINERS | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 32ffdd1a61..32a6493502 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -114,6 +114,11 @@ F: buildtools/symlink-drivers-solibs.py
 F: devtools/test-meson-builds.sh
 F: devtools/check-meson.py
 
+DTS
+M: Lijuan Tu <lijuan.tu@intel.com>
+M: Owen Hilyard <ohilyard@iol.unh.edu>
+F: dts/
+
 Public CI
 M: Aaron Conole <aconole@redhat.com>
 M: Michael Santana <maicolgabriel@hotmail.com>
-- 
2.30.2


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH v5 06/10] dts: add ssh connection module
  2022-09-26 14:17 ` [PATCH v5 06/10] dts: add ssh connection module Juraj Linkeš
@ 2022-09-27 10:12   ` Stanislaw Kardach
  2022-09-30  8:10     ` Juraj Linkeš
  0 siblings, 1 reply; 14+ messages in thread
From: Stanislaw Kardach @ 2022-09-27 10:12 UTC (permalink / raw)
  To: Juraj Linkeš
  Cc: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, bruce.richardson, dev

On Mon, Sep 26, 2022 at 02:17:09PM +0000, Juraj Linkeš wrote:
> The module uses the pexpect python library and implements connection to
> a node and two ways to interact with the node:
> 1. Send a string with specified prompt which will be matched after
>    the string has been sent to the node.
> 2. Send a command to be executed. No prompt is specified here.
> 
> Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> ---
>  dts/framework/exception.py                    |  48 +++++
>  .../remote_session/session_factory.py         |  16 ++
>  dts/framework/remote_session/ssh_session.py   | 189 ++++++++++++++++++
>  dts/framework/utils.py                        |  13 ++
>  4 files changed, 266 insertions(+)
>  create mode 100644 dts/framework/remote_session/session_factory.py
>  create mode 100644 dts/framework/remote_session/ssh_session.py
>  create mode 100644 dts/framework/utils.py
> 
> diff --git a/dts/framework/exception.py b/dts/framework/exception.py
> index 60fd98c9ca..8466990aa5 100644
> --- a/dts/framework/exception.py
> +++ b/dts/framework/exception.py
> @@ -9,6 +9,54 @@
>  """
>  
>  
> +class TimeoutException(Exception):
> +    """
> +    Command execution timeout.
> +    """
> +
> +    command: str
> +    output: str
> +
> +    def __init__(self, command: str, output: str):
> +        self.command = command
> +        self.output = output
> +
> +    def __str__(self) -> str:
> +        return f"TIMEOUT on {self.command}"
> +
> +    def get_output(self) -> str:
> +        return self.output
> +
> +
> +class SSHConnectionException(Exception):
> +    """
> +    SSH connection error.
> +    """
> +
> +    host: str
> +
> +    def __init__(self, host: str):
> +        self.host = host
> +
> +    def __str__(self) -> str:
> +        return f"Error trying to connect with {self.host}"
> +
> +
> +class SSHSessionDeadException(Exception):
> +    """
> +    SSH session is not alive.
> +    It can no longer be used.
> +    """
> +
> +    host: str
> +
> +    def __init__(self, host: str):
> +        self.host = host
> +
> +    def __str__(self) -> str:
> +        return f"SSH session with {self.host} has died"
> +
> +
>  class ConfigParseException(Exception):
>      """
>      Configuration file parse failure exception.
> diff --git a/dts/framework/remote_session/session_factory.py b/dts/framework/remote_session/session_factory.py
> new file mode 100644
> index 0000000000..ff05df97bf
> --- /dev/null
> +++ b/dts/framework/remote_session/session_factory.py
> @@ -0,0 +1,16 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2022 PANTHEON.tech s.r.o.
> +# Copyright(c) 2022 University of New Hampshire
> +#
> +
> +from framework.config import NodeConfiguration
> +from framework.logger import DTSLOG
> +
> +from .remote_session import RemoteSession
> +from .ssh_session import SSHSession
> +
> +
> +def create_remote_session(
> +    node_config: NodeConfiguration, name: str, logger: DTSLOG
> +) -> RemoteSession:
> +    return SSHSession(node_config, name, logger)
> diff --git a/dts/framework/remote_session/ssh_session.py b/dts/framework/remote_session/ssh_session.py
> new file mode 100644
> index 0000000000..e0614e0f90
> --- /dev/null
> +++ b/dts/framework/remote_session/ssh_session.py
> @@ -0,0 +1,189 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2010-2014 Intel Corporation
> +# Copyright(c) 2022 PANTHEON.tech s.r.o.
> +# Copyright(c) 2022 University of New Hampshire
> +#
> +
> +
> +import time
> +
> +from pexpect import pxssh
> +
> +from framework.config import NodeConfiguration
> +from framework.exception import (
> +    SSHConnectionException,
> +    SSHSessionDeadException,
> +    TimeoutException,
> +)
> +from framework.logger import DTSLOG
> +from framework.utils import GREEN, RED
> +
> +from .remote_session import RemoteSession
> +
> +
> +class SSHSession(RemoteSession):
> +    """
> +    Module for creating Pexpect SSH sessions to a node.
> +    """
> +
> +    session: pxssh.pxssh
> +    magic_prompt: str
> +
> +    def __init__(
> +        self,
> +        node_config: NodeConfiguration,
> +        session_name: str,
> +        logger: DTSLOG,
> +    ):
> +        self.magic_prompt = "MAGIC PROMPT"
> +        super(SSHSession, self).__init__(node_config, session_name, logger)
> +
> +    def _connect(self) -> None:
> +        """
> +        Create connection to assigned node.
> +        """
> +        retry_attempts = 10
> +        login_timeout = 20 if self.port else 10
> +        password_regex = (
> +            r"(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)"
> +        )
> +        try:
> +            for retry_attempt in range(retry_attempts):
> +                self.session = pxssh.pxssh(encoding="utf-8")
> +                try:
> +                    self.session.login(
> +                        self.ip,
> +                        self.username,
> +                        self.password,
> +                        original_prompt="[$#>]",
> +                        port=self.port,
> +                        login_timeout=login_timeout,
> +                        password_regex=password_regex,
> +                    )
> +                    break
> +                except Exception as e:
> +                    print(e)
> +                    time.sleep(2)
> +                    print(f"Retrying connection: retry number {retry_attempt + 1}.")
> +            else:
> +                raise Exception(f"Connection to {self.hostname} failed")
> +
> +            self.logger.info(f"Connection to {self.hostname} succeeded")
> +            self.send_expect("stty -echo", "#")
> +            self.send_expect("stty columns 1000", "#")
> +        except Exception as e:
> +            print(RED(str(e)))
> +            if getattr(self, "port", None):
> +                suggestion = (
> +                    f"\nSuggestion: Check if the firewall on {self.hostname} is "
> +                    f"stopped.\n"
> +                )
> +                print(GREEN(suggestion))
> +
> +            raise SSHConnectionException(self.hostname)
> +
> +    def send_expect_base(self, command: str, prompt: str, timeout: float) -> str:
> +        self.clean_session()
> +        original_prompt = self.session.PROMPT
> +        self.session.PROMPT = prompt
> +        self.__sendline(command)
> +        self.__prompt(command, timeout)
> +
> +        before = self._get_output()
> +        self.session.PROMPT = original_prompt
> +        return before
> +
> +    def send_expect(
> +        self, command: str, prompt: str, timeout: float = 15, verify: bool = False
> +    ) -> str | int:
> +        try:
> +            ret = self.send_expect_base(command, prompt, timeout)
> +            if verify:
> +                ret_status = self.send_expect_base("echo $?", prompt, timeout)
> +                try:
> +                    retval = int(ret_status)
> +                    if not retval:
> +                        self.logger.error(f"Command: {command} failure!")
> +                        self.logger.error(ret)
> +                        return retval
> +                    else:
> +                        return ret
Just a minor nit. Isn't the verify logic reversed in this commit?
In V4 "if not retval" was an OK case (returning the output), now it
reports an error.
> +                except ValueError:
> +                    return ret
> +            else:
> +                return ret
> +        except Exception as e:
> +            print(
> +                f"Exception happened in [{command}] and output is "
> +                f"[{self._get_output()}]"
> +            )
> +            raise e
> +
> +    def _send_command(self, command: str, timeout: float = 1) -> str:
> +        try:
> +            self.clean_session()
> +            self.__sendline(command)
> +        except Exception as e:
> +            raise e
> +
> +        output = self.get_output(timeout=timeout)
> +        self.session.PROMPT = self.session.UNIQUE_PROMPT
> +        self.session.prompt(0.1)
> +
> +        return output
> +
> +    def clean_session(self) -> None:
> +        self.get_output(timeout=0.01)
> +
> +    def _get_output(self) -> str:
> +        if not self.is_alive():
> +            raise SSHSessionDeadException(self.hostname)
> +        before = self.session.before.rsplit("\r\n", 1)[0]
> +        if before == "[PEXPECT]":
> +            return ""
> +        return before
> +
> +    def get_output(self, timeout: float = 15) -> str:
> +        """
> +        Get all output before timeout
> +        """
> +        self.session.PROMPT = self.magic_prompt
> +        try:
> +            self.session.prompt(timeout)
> +        except Exception:
> +            pass
> +
> +        before = self._get_output()
> +        self.__flush()
> +
> +        self.logger.debug(before)
> +        return before
> +
> +    def __flush(self) -> None:
> +        """
> +        Clear all session buffer
> +        """
> +        self.session.buffer = ""
> +        self.session.before = ""
> +
> +    def __prompt(self, command: str, timeout: float) -> None:
> +        if not self.session.prompt(timeout):
> +            raise TimeoutException(command, self._get_output()) from None
> +
> +    def __sendline(self, command: str) -> None:
> +        if not self.is_alive():
> +            raise SSHSessionDeadException(self.hostname)
> +        if len(command) == 2 and command.startswith("^"):
> +            self.session.sendcontrol(command[1])
> +        else:
> +            self.session.sendline(command)
> +
> +    def _close(self, force: bool = False) -> None:
> +        if force is True:
> +            self.session.close()
> +        else:
> +            if self.is_alive():
> +                self.session.logout()
> +
> +    def is_alive(self) -> bool:
> +        return self.session.isalive()
> diff --git a/dts/framework/utils.py b/dts/framework/utils.py
> new file mode 100644
> index 0000000000..26b784ebb5
> --- /dev/null
> +++ b/dts/framework/utils.py
> @@ -0,0 +1,13 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2010-2014 Intel Corporation
> +# Copyright(c) 2022 PANTHEON.tech s.r.o.
> +# Copyright(c) 2022 University of New Hampshire
> +#
> +
> +
> +def RED(text: str) -> str:
> +    return f"\u001B[31;1m{str(text)}\u001B[0m"
> +
> +
> +def GREEN(text: str) -> str:
> +    return f"\u001B[32;1m{str(text)}\u001B[0m"
> -- 
> 2.30.2
> 

Reviewed-by: Stanislaw Kardach <kda@semihalf.com>

-- 
Best Regards,
Stanislaw Kardach

^ permalink raw reply	[flat|nested] 14+ messages in thread

* RE: [PATCH v5 06/10] dts: add ssh connection module
  2022-09-27 10:12   ` Stanislaw Kardach
@ 2022-09-30  8:10     ` Juraj Linkeš
  0 siblings, 0 replies; 14+ messages in thread
From: Juraj Linkeš @ 2022-09-30  8:10 UTC (permalink / raw)
  To: Stanislaw Kardach
  Cc: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard,
	lijuan.tu, bruce.richardson, dev



> -----Original Message-----
> From: Stanislaw Kardach <kda@semihalf.com>
> Sent: Tuesday, September 27, 2022 12:12 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> Honnappa.Nagarahalli@arm.com; ohilyard@iol.unh.edu; lijuan.tu@intel.com;
> bruce.richardson@intel.com; dev@dpdk.org
> Subject: Re: [PATCH v5 06/10] dts: add ssh connection module
> 
> On Mon, Sep 26, 2022 at 02:17:09PM +0000, Juraj Linkeš wrote:
> > The module uses the pexpect python library and implements connection
> > to a node and two ways to interact with the node:
> > 1. Send a string with specified prompt which will be matched after
> >    the string has been sent to the node.
> > 2. Send a command to be executed. No prompt is specified here.
> >
> > Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > ---
> >  dts/framework/exception.py                    |  48 +++++
> >  .../remote_session/session_factory.py         |  16 ++
> >  dts/framework/remote_session/ssh_session.py   | 189 ++++++++++++++++++
> >  dts/framework/utils.py                        |  13 ++
> >  4 files changed, 266 insertions(+)
> >  create mode 100644 dts/framework/remote_session/session_factory.py
> >  create mode 100644 dts/framework/remote_session/ssh_session.py
> >  create mode 100644 dts/framework/utils.py
> >
> > diff --git a/dts/framework/exception.py b/dts/framework/exception.py
> > index 60fd98c9ca..8466990aa5 100644
> > --- a/dts/framework/exception.py
> > +++ b/dts/framework/exception.py
> > @@ -9,6 +9,54 @@
> >  """
> >
> >
> > +class TimeoutException(Exception):
> > +    """
> > +    Command execution timeout.
> > +    """
> > +
> > +    command: str
> > +    output: str
> > +
> > +    def __init__(self, command: str, output: str):
> > +        self.command = command
> > +        self.output = output
> > +
> > +    def __str__(self) -> str:
> > +        return f"TIMEOUT on {self.command}"
> > +
> > +    def get_output(self) -> str:
> > +        return self.output
> > +
> > +
> > +class SSHConnectionException(Exception):
> > +    """
> > +    SSH connection error.
> > +    """
> > +
> > +    host: str
> > +
> > +    def __init__(self, host: str):
> > +        self.host = host
> > +
> > +    def __str__(self) -> str:
> > +        return f"Error trying to connect with {self.host}"
> > +
> > +
> > +class SSHSessionDeadException(Exception):
> > +    """
> > +    SSH session is not alive.
> > +    It can no longer be used.
> > +    """
> > +
> > +    host: str
> > +
> > +    def __init__(self, host: str):
> > +        self.host = host
> > +
> > +    def __str__(self) -> str:
> > +        return f"SSH session with {self.host} has died"
> > +
> > +
> >  class ConfigParseException(Exception):
> >      """
> >      Configuration file parse failure exception.
> > diff --git a/dts/framework/remote_session/session_factory.py
> > b/dts/framework/remote_session/session_factory.py
> > new file mode 100644
> > index 0000000000..ff05df97bf
> > --- /dev/null
> > +++ b/dts/framework/remote_session/session_factory.py
> > @@ -0,0 +1,16 @@
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2022
> > +PANTHEON.tech s.r.o.
> > +# Copyright(c) 2022 University of New Hampshire #
> > +
> > +from framework.config import NodeConfiguration from framework.logger
> > +import DTSLOG
> > +
> > +from .remote_session import RemoteSession from .ssh_session import
> > +SSHSession
> > +
> > +
> > +def create_remote_session(
> > +    node_config: NodeConfiguration, name: str, logger: DTSLOG
> > +) -> RemoteSession:
> > +    return SSHSession(node_config, name, logger)
> > diff --git a/dts/framework/remote_session/ssh_session.py
> > b/dts/framework/remote_session/ssh_session.py
> > new file mode 100644
> > index 0000000000..e0614e0f90
> > --- /dev/null
> > +++ b/dts/framework/remote_session/ssh_session.py
> > @@ -0,0 +1,189 @@
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2010-2014
> > +Intel Corporation # Copyright(c) 2022 PANTHEON.tech s.r.o.
> > +# Copyright(c) 2022 University of New Hampshire #
> > +
> > +
> > +import time
> > +
> > +from pexpect import pxssh
> > +
> > +from framework.config import NodeConfiguration from
> > +framework.exception import (
> > +    SSHConnectionException,
> > +    SSHSessionDeadException,
> > +    TimeoutException,
> > +)
> > +from framework.logger import DTSLOG
> > +from framework.utils import GREEN, RED
> > +
> > +from .remote_session import RemoteSession
> > +
> > +
> > +class SSHSession(RemoteSession):
> > +    """
> > +    Module for creating Pexpect SSH sessions to a node.
> > +    """
> > +
> > +    session: pxssh.pxssh
> > +    magic_prompt: str
> > +
> > +    def __init__(
> > +        self,
> > +        node_config: NodeConfiguration,
> > +        session_name: str,
> > +        logger: DTSLOG,
> > +    ):
> > +        self.magic_prompt = "MAGIC PROMPT"
> > +        super(SSHSession, self).__init__(node_config, session_name,
> > + logger)
> > +
> > +    def _connect(self) -> None:
> > +        """
> > +        Create connection to assigned node.
> > +        """
> > +        retry_attempts = 10
> > +        login_timeout = 20 if self.port else 10
> > +        password_regex = (
> > +            r"(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)"
> > +        )
> > +        try:
> > +            for retry_attempt in range(retry_attempts):
> > +                self.session = pxssh.pxssh(encoding="utf-8")
> > +                try:
> > +                    self.session.login(
> > +                        self.ip,
> > +                        self.username,
> > +                        self.password,
> > +                        original_prompt="[$#>]",
> > +                        port=self.port,
> > +                        login_timeout=login_timeout,
> > +                        password_regex=password_regex,
> > +                    )
> > +                    break
> > +                except Exception as e:
> > +                    print(e)
> > +                    time.sleep(2)
> > +                    print(f"Retrying connection: retry number {retry_attempt + 1}.")
> > +            else:
> > +                raise Exception(f"Connection to {self.hostname}
> > + failed")
> > +
> > +            self.logger.info(f"Connection to {self.hostname} succeeded")
> > +            self.send_expect("stty -echo", "#")
> > +            self.send_expect("stty columns 1000", "#")
> > +        except Exception as e:
> > +            print(RED(str(e)))
> > +            if getattr(self, "port", None):
> > +                suggestion = (
> > +                    f"\nSuggestion: Check if the firewall on {self.hostname} is "
> > +                    f"stopped.\n"
> > +                )
> > +                print(GREEN(suggestion))
> > +
> > +            raise SSHConnectionException(self.hostname)
> > +
> > +    def send_expect_base(self, command: str, prompt: str, timeout: float) ->
> str:
> > +        self.clean_session()
> > +        original_prompt = self.session.PROMPT
> > +        self.session.PROMPT = prompt
> > +        self.__sendline(command)
> > +        self.__prompt(command, timeout)
> > +
> > +        before = self._get_output()
> > +        self.session.PROMPT = original_prompt
> > +        return before
> > +
> > +    def send_expect(
> > +        self, command: str, prompt: str, timeout: float = 15, verify: bool = False
> > +    ) -> str | int:
> > +        try:
> > +            ret = self.send_expect_base(command, prompt, timeout)
> > +            if verify:
> > +                ret_status = self.send_expect_base("echo $?", prompt, timeout)
> > +                try:
> > +                    retval = int(ret_status)
> > +                    if not retval:
> > +                        self.logger.error(f"Command: {command} failure!")
> > +                        self.logger.error(ret)
> > +                        return retval
> > +                    else:
> > +                        return ret
> Just a minor nit. Isn't the verify logic reversed in this commit?
> In V4 "if not retval" was an OK case (returning the output), now it reports an
> error.

Yes, this is a mistake on my part:
0 is False and any other value is True, which is (basically) the opposite of shell. That means not retval is True when retval = 0 and that shouldn't produce an error. Thanks for the catch.

> > +                except ValueError:
> > +                    return ret
> > +            else:
> > +                return ret
> > +        except Exception as e:
> > +            print(
> > +                f"Exception happened in [{command}] and output is "
> > +                f"[{self._get_output()}]"
> > +            )
> > +            raise e
> > +
> > +    def _send_command(self, command: str, timeout: float = 1) -> str:
> > +        try:
> > +            self.clean_session()
> > +            self.__sendline(command)
> > +        except Exception as e:
> > +            raise e
> > +
> > +        output = self.get_output(timeout=timeout)
> > +        self.session.PROMPT = self.session.UNIQUE_PROMPT
> > +        self.session.prompt(0.1)
> > +
> > +        return output
> > +
> > +    def clean_session(self) -> None:
> > +        self.get_output(timeout=0.01)
> > +
> > +    def _get_output(self) -> str:
> > +        if not self.is_alive():
> > +            raise SSHSessionDeadException(self.hostname)
> > +        before = self.session.before.rsplit("\r\n", 1)[0]
> > +        if before == "[PEXPECT]":
> > +            return ""
> > +        return before
> > +
> > +    def get_output(self, timeout: float = 15) -> str:
> > +        """
> > +        Get all output before timeout
> > +        """
> > +        self.session.PROMPT = self.magic_prompt
> > +        try:
> > +            self.session.prompt(timeout)
> > +        except Exception:
> > +            pass
> > +
> > +        before = self._get_output()
> > +        self.__flush()
> > +
> > +        self.logger.debug(before)
> > +        return before
> > +
> > +    def __flush(self) -> None:
> > +        """
> > +        Clear all session buffer
> > +        """
> > +        self.session.buffer = ""
> > +        self.session.before = ""
> > +
> > +    def __prompt(self, command: str, timeout: float) -> None:
> > +        if not self.session.prompt(timeout):
> > +            raise TimeoutException(command, self._get_output()) from
> > + None
> > +
> > +    def __sendline(self, command: str) -> None:
> > +        if not self.is_alive():
> > +            raise SSHSessionDeadException(self.hostname)
> > +        if len(command) == 2 and command.startswith("^"):
> > +            self.session.sendcontrol(command[1])
> > +        else:
> > +            self.session.sendline(command)
> > +
> > +    def _close(self, force: bool = False) -> None:
> > +        if force is True:
> > +            self.session.close()
> > +        else:
> > +            if self.is_alive():
> > +                self.session.logout()
> > +
> > +    def is_alive(self) -> bool:
> > +        return self.session.isalive()
> > diff --git a/dts/framework/utils.py b/dts/framework/utils.py new file
> > mode 100644 index 0000000000..26b784ebb5
> > --- /dev/null
> > +++ b/dts/framework/utils.py
> > @@ -0,0 +1,13 @@
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2010-2014
> > +Intel Corporation # Copyright(c) 2022 PANTHEON.tech s.r.o.
> > +# Copyright(c) 2022 University of New Hampshire #
> > +
> > +
> > +def RED(text: str) -> str:
> > +    return f"\u001B[31;1m{str(text)}\u001B[0m"
> > +
> > +
> > +def GREEN(text: str) -> str:
> > +    return f"\u001B[32;1m{str(text)}\u001B[0m"
> > --
> > 2.30.2
> >
> 
> Reviewed-by: Stanislaw Kardach <kda@semihalf.com>
> 
> --
> Best Regards,
> Stanislaw Kardach


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH v5 00/10] dts: ssh connection to a node
  2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
                   ` (9 preceding siblings ...)
  2022-09-26 14:17 ` [PATCH v5 10/10] maintainers: add dts maintainers Juraj Linkeš
@ 2022-10-11 15:40 ` Owen Hilyard
  10 siblings, 0 replies; 14+ messages in thread
From: Owen Hilyard @ 2022-10-11 15:40 UTC (permalink / raw)
  To: Juraj Linkeš
  Cc: thomas, david.marchand, Honnappa.Nagarahalli, lijuan.tu, kda,
	bruce.richardson, dev

[-- Attachment #1: Type: text/plain, Size: 6491 bytes --]

On Mon, Sep 26, 2022 at 10:17 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:

> All the necessary code needed to connect to a node in a topology with
> a bit more, such as basic logging and some extra useful methods.
>
> To run the code, modify the config file, conf.yaml and execute ./main.py
> from the root dts folder. Here's an example config:
> executions:
>   - system_under_test: "SUT 1"
> nodes:
>   - name: "SUT 1"
>     hostname: 127.0.0.1
>     user: root
>     password: mypw.change.me
>
> There are configuration files with a README that help with setting up
> the execution/development environment.
>
> The code only connects to a node. You'll see logs emitted to console
> saying where DTS connected.
>
> There's only a bit of documentation, as there's not much to document.
> We'll add some real docs when there's enough functionality to document,
> when the HelloWorld testcases is in (point 4 in our roadmap below). What
> will be documented later is runtime dependencies and how to set up the DTS
> control node environment.
>
> This is our current roadmap:
> 1. Review this patchset and do the rest of the items in parallel, if
> possible.
> 2. We have extracted the code needed to run the most basic testcase,
> HelloWorld, which runs the DPDK Hello World application. We'll split
> this along logical/functional boundaries and send after 1 is done.
> 3. Once we have 2 applied, we're planning on adding a basic functional
> testcase - pf_smoke. This send a bit of traffic, so the big addition is
> the software traffic generator, Scapy. There's some work already done on
> Traffic generators we'll be sending as a dependence on this patch
> series.
> 4. After 3, we'll add a basic performance testcase which doesn't use
> Scapy, but Trex or Ixia instead.
> 5. This is far in the future, but at this point we should have all of
> the core functionality in place. What then remains is adding the rest of
> the testcases.
>
> We're already working on items 2-4 and we may send more patches even
> before this patch series is accepted if that's beneficial. The new
> patches would then depend on this patch.
>
> This patch, as well as all others in the pipeline, are the result of
> extensive DTS workgroup review which happens internally. If you'd like
> us to make it more public we'd have no problem with that.
>
> v3:
> Added project config files and developer tools.
> Removed locks for parallel nodes, which are not needed now and will be
> implemented much later (in a different patch).
>
> v4:
> Minor fixes - added missing Exception and utils function.
>
> v5:
> Reordered commits because the dependencies between commits changed.
> Added more developer tools.
> Added definitions of DTS testbed elements.
> Reworked SSH implementation - split it so that the split between an
> abstraction and the actual implementation is clearer.
> Modified the directory structure to better organize the current and the
> future code.
>
> Juraj Linkeš (9):
>   dts: add project tools config
>   dts: add developer tools
>   dts: add basic logging facility
>   dts: add remote session abstraction
>   dts: add ssh connection module
>   dts: add node base class
>   dts: add dts workflow module
>   dts: add dts executable script
>   maintainers: add dts maintainers
>
> Owen Hilyard (1):
>   dts: add config parser module
>
>  .editorconfig                                 |   2 +-
>  .gitignore                                    |   9 +-
>  MAINTAINERS                                   |   5 +
>  devtools/python-checkpatch.sh                 |  39 ++
>  devtools/python-format.sh                     |  54 +++
>  devtools/python-lint.sh                       |  26 ++
>  doc/guides/contributing/coding_style.rst      |   4 +-
>  dts/.devcontainer/devcontainer.json           |  30 ++
>  dts/Dockerfile                                |  39 ++
>  dts/README.md                                 | 154 ++++++++
>  dts/conf.yaml                                 |   6 +
>  dts/framework/__init__.py                     |   4 +
>  dts/framework/config/__init__.py              |  99 +++++
>  dts/framework/config/conf_yaml_schema.json    |  73 ++++
>  dts/framework/dts.py                          |  69 ++++
>  dts/framework/exception.py                    |  71 ++++
>  dts/framework/logger.py                       | 115 ++++++
>  dts/framework/remote_session/__init__.py      |   5 +
>  .../remote_session/remote_session.py          | 100 +++++
>  .../remote_session/session_factory.py         |  16 +
>  dts/framework/remote_session/ssh_session.py   | 189 ++++++++++
>  dts/framework/settings.py                     | 108 ++++++
>  dts/framework/testbed_model/__init__.py       |   8 +
>  dts/framework/testbed_model/node.py           |  83 +++++
>  dts/framework/utils.py                        |  31 ++
>  dts/main.py                                   |  24 ++
>  dts/poetry.lock                               | 351 ++++++++++++++++++
>  dts/pyproject.toml                            |  55 +++
>  28 files changed, 1765 insertions(+), 4 deletions(-)
>  create mode 100755 devtools/python-checkpatch.sh
>  create mode 100755 devtools/python-format.sh
>  create mode 100755 devtools/python-lint.sh
>  create mode 100644 dts/.devcontainer/devcontainer.json
>  create mode 100644 dts/Dockerfile
>  create mode 100644 dts/README.md
>  create mode 100644 dts/conf.yaml
>  create mode 100644 dts/framework/__init__.py
>  create mode 100644 dts/framework/config/__init__.py
>  create mode 100644 dts/framework/config/conf_yaml_schema.json
>  create mode 100644 dts/framework/dts.py
>  create mode 100644 dts/framework/exception.py
>  create mode 100644 dts/framework/logger.py
>  create mode 100644 dts/framework/remote_session/__init__.py
>  create mode 100644 dts/framework/remote_session/remote_session.py
>  create mode 100644 dts/framework/remote_session/session_factory.py
>  create mode 100644 dts/framework/remote_session/ssh_session.py
>  create mode 100644 dts/framework/settings.py
>  create mode 100644 dts/framework/testbed_model/__init__.py
>  create mode 100644 dts/framework/testbed_model/node.py
>  create mode 100644 dts/framework/utils.py
>  create mode 100755 dts/main.py
>  create mode 100644 dts/poetry.lock
>  create mode 100644 dts/pyproject.toml
>
> --
> 2.30.2
>
>
Everything looks good from my perspective.

[-- Attachment #2: Type: text/html, Size: 7609 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2022-10-11 15:41 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-09-26 14:17 [PATCH v5 00/10] dts: ssh connection to a node Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 01/10] dts: add project tools config Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 02/10] dts: add developer tools Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 03/10] dts: add config parser module Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 04/10] dts: add basic logging facility Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 05/10] dts: add remote session abstraction Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 06/10] dts: add ssh connection module Juraj Linkeš
2022-09-27 10:12   ` Stanislaw Kardach
2022-09-30  8:10     ` Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 07/10] dts: add node base class Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 08/10] dts: add dts workflow module Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 09/10] dts: add dts executable script Juraj Linkeš
2022-09-26 14:17 ` [PATCH v5 10/10] maintainers: add dts maintainers Juraj Linkeš
2022-10-11 15:40 ` [PATCH v5 00/10] dts: ssh connection to a node Owen Hilyard

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).