From: "Juraj Linkeš" <juraj.linkes@pantheon.tech>
To: thomas@monjalon.net, david.marchand@redhat.com,
ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com,
ohilyard@iol.unh.edu, lijuan.tu@intel.com
Cc: dev@dpdk.org, "Juraj Linkeš" <juraj.linkes@pantheon.tech>
Subject: [PATCH v4 2/9] dts: add developer tools
Date: Fri, 29 Jul 2022 10:55:43 +0000 [thread overview]
Message-ID: <20220729105550.1382664-3-juraj.linkes@pantheon.tech> (raw)
In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech>
The Dockerfile contains basic image for CI and developers. There's also
an integration of the Dockerfile with Visual Studio.
The formatter script uses Black and Isort to format the Python code.
Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
dts/.devcontainer/devcontainer.json | 30 ++++++++++++
dts/Dockerfile | 38 +++++++++++++++
dts/README.md | 74 ++++++++++++++++++++++++++++-
dts/format.sh | 45 ++++++++++++++++++
4 files changed, 186 insertions(+), 1 deletion(-)
create mode 100644 dts/.devcontainer/devcontainer.json
create mode 100644 dts/Dockerfile
create mode 100755 dts/format.sh
diff --git a/dts/.devcontainer/devcontainer.json b/dts/.devcontainer/devcontainer.json
new file mode 100644
index 0000000000..41ca28fc17
--- /dev/null
+++ b/dts/.devcontainer/devcontainer.json
@@ -0,0 +1,30 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
+// https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/docker-existing-dockerfile
+{
+ "name": "Existing Dockerfile",
+
+ // Sets the run context to one level up instead of the .devcontainer folder.
+ "context": "..",
+
+ // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
+ "dockerFile": "../Dockerfile",
+
+ // Use 'forwardPorts' to make a list of ports inside the container available locally.
+ // "forwardPorts": [],
+
+ // Uncomment the next line to run commands after the container is created - for example installing curl.
+ "postCreateCommand": "poetry install",
+
+ "extensions": [
+ "ms-python.vscode-pylance",
+ ]
+
+ // Uncomment when using a ptrace-based debugger like C++, Go, and Rust
+ // "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ],
+
+ // Uncomment to use the Docker CLI from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker.
+ // "mounts": [ "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ],
+
+ // Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root.
+ // "remoteUser": "vscode"
+}
diff --git a/dts/Dockerfile b/dts/Dockerfile
new file mode 100644
index 0000000000..6700aa45b8
--- /dev/null
+++ b/dts/Dockerfile
@@ -0,0 +1,38 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+#
+
+FROM ubuntu:22.04 AS base
+
+RUN apt-get -y update && apt-get -y upgrade && \
+ apt-get -y install --no-install-recommends \
+ python3 \
+ python3-pip \
+ python3-pexpect \
+ python3-poetry \
+ python3-cachecontrol \
+ openssh-client
+
+
+FROM base AS runner
+
+# This container is intended to be used as the base for automated systems.
+# It bakes DTS into the container during the build.
+
+RUN mkdir /dts
+COPY ./pyproject.toml /dts/pyproject.toml
+COPY ./poetry.lock /dts/poetry.lock
+WORKDIR /dts
+RUN poetry install --no-dev
+COPY . /dts
+
+CMD ["poetry", "run", "python", "main.py"]
+
+FROM base AS dev
+
+# This container is intended to be used as a development environment.
+
+RUN apt-get -y install --no-install-recommends \
+ vim emacs git
+
+WORKDIR /dts
diff --git a/dts/README.md b/dts/README.md
index d8f88f97fe..55a272d767 100644
--- a/dts/README.md
+++ b/dts/README.md
@@ -12,4 +12,76 @@ The Python Version required by DTS is specified in
[DTS python config file](./pyproject.toml) in the **[tool.poetry.dependencies]**
section. Poetry doesn't install Python, so you may need to satisfy this requirement if
your Python is not up to date. A tool such as [Pyenv](https://github.com/pyenv/pyenv)
-is a good way to get Python, though not the only one.
+is a good way to get Python, though not the only one. However, DTS includes a
+development environment in the form of a Docker image.
+
+# Expected Environment
+
+The expected execution and development environments for DTS are the same,
+the container defined by [Dockerfile](./Dockerfile). Using a container for the
+development environment helps with a few things.
+
+1. It helps enforce the boundary between the tester and the traffic
+ generator/sut, something which has experienced issues in the past.
+2. It makes creating containers to run DTS inside automated tooling
+ much easier, since they can be based off of a known-working environment
+ that will be updated as DTS is.
+3. It abstracts DTS from the server it is running on. This means that the
+ bare-metal os can be whatever corporate policy or your personal preferences
+ dictate, and DTS does not have to try to support all 15 distros that
+ are supported by DPDK CI.
+4. It makes automated testing for DTS easier, since new dependencies can be
+ sent in with the patches.
+5. It fixes the issue of undocumented dependencies, where some test suites
+ require python libraries that are not installed.
+6. Allows everyone to use the same python version easily, even if they are
+ using an LTS distro or Windows.
+7. Allows you to run the tester on Windows while developing via Docker for
+ Windows.
+
+## Tips for setting up a development environment
+
+### Getting a docker shell
+
+These commands will give you a bash shell inside the container with all the python
+dependencies installed. This will place you inside a python virtual
+environment. DTS is mounted via a volume, which is essentially a symlink
+from the host to the container. This enables you to edit and run inside the container
+and then delete the container when you are done, keeping your work.
+
+```shell
+docker build --target dev -t dpdk-dts .
+docker run -v $(pwd):/dts -it dpdk-dts bash
+$ poetry install
+$ poetry shell
+```
+
+### Vim/Emacs
+
+Any editor in the ubuntu repos should be easy to use. You can add your normal
+config files as a volume, enabling you to use your preferred settings.
+
+```shell
+apt install vim
+apt install emacs
+```
+
+### Visual Studio Code
+
+VSCode has first-class support for developing with containers. You may need to run the
+non-docker setup commands in the integrated terminal. DTS contains a .devcontainer
+config, so if you open the folder in vscode it should prompt you to use the dev
+container assuming you have the plugin installed. Please refer to
+[VS Development Containers Docs](https://code.visualstudio.com/docs/remote/containers)
+to set it all up.
+
+### Other
+
+Searching for '$IDE dev containers' will probably lead you in the right
+direction.
+
+# Python Formatting
+
+The tools used to format Python code in DTS are Black and Isort. There's a shell
+script, function.sh, which runs the formatters. Poetry will install these tools,
+so once you have that set up, you should run it before submitting patches.
diff --git a/dts/format.sh b/dts/format.sh
new file mode 100755
index 0000000000..7d72335470
--- /dev/null
+++ b/dts/format.sh
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+# Copyright(c) 2022 PANTHEON.tech s.r.o.
+#
+
+function main() {
+ # The directory to work on is either passed in as argument 1,
+ # or is the current working directory
+ DIRECTORY=${1:-$(pwd)}
+ LINE_LENGTH=88
+
+ BLACK_VERSION=$(awk '/\[tool.poetry.dev-dependencies\]/,/$^/' pyproject.toml |\
+ grep black | grep -o '[0-9][^"]*')
+
+ PYTHON_VERSION=$(awk '/\[tool.poetry.dependencies\]/,/$^/' pyproject.toml |\
+ grep python | grep -o '[0-9][^"]*' | tr -d '.')
+
+ isort \
+ --overwrite-in-place \
+ --profile black \
+ -j "$(nproc)" \
+ --line-length $LINE_LENGTH \
+ --python-version auto \
+ "$DIRECTORY"
+
+ black \
+ --line-length $LINE_LENGTH \
+ --required-version "${BLACK_VERSION}" \
+ --target-version "py${PYTHON_VERSION}" \
+ --safe \
+ "$DIRECTORY"
+}
+
+function help() {
+ echo "usage: format.sh <directory>"
+}
+
+if [ "$1" == "-h" ] || [ "$1" == "--help" ]; then
+ help
+ exit 0
+fi
+
+main "$1"
+
--
2.30.2
next prev parent reply other threads:[~2022-07-29 10:56 UTC|newest]
Thread overview: 105+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-22 12:14 [PATCH v1 0/8] dts: ssh connection to a node Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 1/8] dts: add ssh pexpect library Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 2/8] dts: add locks for parallel node connections Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 3/8] dts: add ssh connection extension Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 4/8] dts: add basic logging facility Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 5/8] dts: add Node base class Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 6/8] dts: add config parser module Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 7/8] dts: add dts runtime workflow module Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 8/8] dts: add main script for running dts Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 0/8] ssh connection to a node Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 1/8] dts: add basic logging facility Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 2/8] dts: add ssh pexpect library Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 3/8] dts: add locks for parallel node connections Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 4/8] dts: add ssh connection extension Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 5/8] dts: add config parser module Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 6/8] dts: add Node base class Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 7/8] dts: add dts workflow module Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 8/8] dts: add dts executable script Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 0/9] dts: ssh connection to a node Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 1/9] dts: add project tools config Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 2/9] dts: add developer tools Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 3/9] dts: add basic logging facility Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 4/9] dts: add ssh pexpect library Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 5/9] dts: add ssh connection extension Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 6/9] dts: add config parser module Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 7/9] dts: add Node base class Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 8/9] dts: add dts workflow module Juraj Linkeš
2022-07-28 10:00 ` [PATCH v3 9/9] dts: add dts executable script Juraj Linkeš
2022-07-29 10:55 ` [PATCH v4 0/9] dts: ssh connection to a node Juraj Linkeš
2022-07-29 10:55 ` [PATCH v4 1/9] dts: add project tools config Juraj Linkeš
2022-08-10 6:30 ` Tu, Lijuan
2022-09-07 16:16 ` Bruce Richardson
2022-09-09 13:38 ` Juraj Linkeš
2022-09-09 13:52 ` Bruce Richardson
2022-09-09 14:13 ` Juraj Linkeš
2022-09-12 14:06 ` Owen Hilyard
2022-09-12 15:15 ` Bruce Richardson
2022-09-13 12:08 ` Juraj Linkeš
2022-09-13 14:18 ` Bruce Richardson
2022-09-13 19:03 ` Honnappa Nagarahalli
2022-09-13 19:19 ` Honnappa Nagarahalli
2022-09-14 9:37 ` Thomas Monjalon
2022-09-14 12:55 ` Juraj Linkeš
2022-09-14 13:11 ` Bruce Richardson
2022-09-14 14:28 ` Thomas Monjalon
2022-09-21 10:49 ` Juraj Linkeš
2022-09-13 19:11 ` Honnappa Nagarahalli
2022-07-29 10:55 ` Juraj Linkeš [this message]
2022-08-10 6:30 ` [PATCH v4 2/9] dts: add developer tools Tu, Lijuan
2022-09-07 16:37 ` Bruce Richardson
2022-09-13 12:38 ` Juraj Linkeš
2022-09-13 20:38 ` Honnappa Nagarahalli
2022-09-14 7:37 ` Bruce Richardson
2022-09-14 12:45 ` Juraj Linkeš
2022-09-14 13:13 ` Bruce Richardson
2022-09-14 14:26 ` Thomas Monjalon
2022-09-14 19:08 ` Honnappa Nagarahalli
2022-09-20 12:14 ` Juraj Linkeš
2022-09-20 12:22 ` Tu, Lijuan
2022-07-29 10:55 ` [PATCH v4 3/9] dts: add basic logging facility Juraj Linkeš
2022-08-10 6:31 ` Tu, Lijuan
2022-09-08 8:31 ` Bruce Richardson
2022-09-13 12:52 ` Juraj Linkeš
2022-09-13 23:31 ` Honnappa Nagarahalli
2022-09-14 12:51 ` Juraj Linkeš
2022-07-29 10:55 ` [PATCH v4 4/9] dts: add ssh pexpect library Juraj Linkeš
2022-08-10 6:31 ` Tu, Lijuan
2022-09-08 9:53 ` Bruce Richardson
2022-09-13 13:36 ` Juraj Linkeš
2022-09-13 14:23 ` Bruce Richardson
2022-09-13 14:59 ` Stanislaw Kardach
2022-09-13 17:23 ` Owen Hilyard
2022-09-14 0:03 ` Honnappa Nagarahalli
2022-09-14 7:42 ` Bruce Richardson
2022-09-14 7:58 ` Stanislaw Kardach
2022-09-14 19:57 ` Honnappa Nagarahalli
2022-09-19 14:21 ` Owen Hilyard
2022-09-20 17:54 ` Honnappa Nagarahalli
2022-09-21 1:01 ` Tu, Lijuan
2022-09-21 5:37 ` Jerin Jacob
2022-09-22 9:03 ` Juraj Linkeš
2022-09-14 9:42 ` Stanislaw Kardach
2022-09-22 9:41 ` Juraj Linkeš
2022-09-22 14:32 ` Stanislaw Kardach
2022-09-23 7:22 ` Juraj Linkeš
2022-09-23 8:15 ` Bruce Richardson
2022-09-23 10:18 ` Stanislaw Kardach
2022-07-29 10:55 ` [PATCH v4 5/9] dts: add ssh connection extension Juraj Linkeš
2022-08-10 6:32 ` Tu, Lijuan
2022-09-13 17:04 ` Bruce Richardson
2022-09-13 17:32 ` Owen Hilyard
2022-09-14 7:46 ` Bruce Richardson
2022-09-14 12:02 ` Owen Hilyard
2022-09-14 13:15 ` Bruce Richardson
2022-07-29 10:55 ` [PATCH v4 6/9] dts: add config parser module Juraj Linkeš
2022-08-10 6:33 ` Tu, Lijuan
2022-09-13 17:19 ` Bruce Richardson
2022-09-13 17:47 ` Owen Hilyard
2022-09-14 7:48 ` Bruce Richardson
2022-07-29 10:55 ` [PATCH v4 7/9] dts: add Node base class Juraj Linkeš
2022-08-10 6:33 ` Tu, Lijuan
2022-07-29 10:55 ` [PATCH v4 8/9] dts: add dts workflow module Juraj Linkeš
2022-08-10 6:34 ` Tu, Lijuan
2022-07-29 10:55 ` [PATCH v4 9/9] dts: add dts executable script Juraj Linkeš
2022-08-10 6:35 ` Tu, Lijuan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220729105550.1382664-3-juraj.linkes@pantheon.tech \
--to=juraj.linkes@pantheon.tech \
--cc=Honnappa.Nagarahalli@arm.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=lijuan.tu@intel.com \
--cc=ohilyard@iol.unh.edu \
--cc=ronan.randles@intel.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).