From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 42864A04FD; Fri, 29 Jul 2022 12:56:08 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0BD7542C24; Fri, 29 Jul 2022 12:56:00 +0200 (CEST) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by mails.dpdk.org (Postfix) with ESMTP id A6D4842C15 for ; Fri, 29 Jul 2022 12:55:57 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id A1D141C207A; Fri, 29 Jul 2022 12:55:56 +0200 (CEST) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bpgen4rGPg4p; Fri, 29 Jul 2022 12:55:55 +0200 (CEST) Received: from entguard.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id 5B67C13125E; Fri, 29 Jul 2022 12:55:52 +0200 (CEST) From: =?UTF-8?q?Juraj=20Linke=C5=A1?= To: thomas@monjalon.net, david.marchand@redhat.com, ronan.randles@intel.com, Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu, lijuan.tu@intel.com Cc: dev@dpdk.org, =?UTF-8?q?Juraj=20Linke=C5=A1?= Subject: [PATCH v4 2/9] dts: add developer tools Date: Fri, 29 Jul 2022 10:55:43 +0000 Message-Id: <20220729105550.1382664-3-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220729105550.1382664-1-juraj.linkes@pantheon.tech> References: <20220728100044.1318484-1-juraj.linkes@pantheon.tech> <20220729105550.1382664-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The Dockerfile contains basic image for CI and developers. There's also an integration of the Dockerfile with Visual Studio. The formatter script uses Black and Isort to format the Python code. Signed-off-by: Owen Hilyard Signed-off-by: Juraj Linkeš --- dts/.devcontainer/devcontainer.json | 30 ++++++++++++ dts/Dockerfile | 38 +++++++++++++++ dts/README.md | 74 ++++++++++++++++++++++++++++- dts/format.sh | 45 ++++++++++++++++++ 4 files changed, 186 insertions(+), 1 deletion(-) create mode 100644 dts/.devcontainer/devcontainer.json create mode 100644 dts/Dockerfile create mode 100755 dts/format.sh diff --git a/dts/.devcontainer/devcontainer.json b/dts/.devcontainer/devcontainer.json new file mode 100644 index 0000000000..41ca28fc17 --- /dev/null +++ b/dts/.devcontainer/devcontainer.json @@ -0,0 +1,30 @@ +// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at: +// https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/docker-existing-dockerfile +{ + "name": "Existing Dockerfile", + + // Sets the run context to one level up instead of the .devcontainer folder. + "context": "..", + + // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename. + "dockerFile": "../Dockerfile", + + // Use 'forwardPorts' to make a list of ports inside the container available locally. + // "forwardPorts": [], + + // Uncomment the next line to run commands after the container is created - for example installing curl. + "postCreateCommand": "poetry install", + + "extensions": [ + "ms-python.vscode-pylance", + ] + + // Uncomment when using a ptrace-based debugger like C++, Go, and Rust + // "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ], + + // Uncomment to use the Docker CLI from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker. + // "mounts": [ "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ], + + // Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root. + // "remoteUser": "vscode" +} diff --git a/dts/Dockerfile b/dts/Dockerfile new file mode 100644 index 0000000000..6700aa45b8 --- /dev/null +++ b/dts/Dockerfile @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2022 University of New Hampshire +# + +FROM ubuntu:22.04 AS base + +RUN apt-get -y update && apt-get -y upgrade && \ + apt-get -y install --no-install-recommends \ + python3 \ + python3-pip \ + python3-pexpect \ + python3-poetry \ + python3-cachecontrol \ + openssh-client + + +FROM base AS runner + +# This container is intended to be used as the base for automated systems. +# It bakes DTS into the container during the build. + +RUN mkdir /dts +COPY ./pyproject.toml /dts/pyproject.toml +COPY ./poetry.lock /dts/poetry.lock +WORKDIR /dts +RUN poetry install --no-dev +COPY . /dts + +CMD ["poetry", "run", "python", "main.py"] + +FROM base AS dev + +# This container is intended to be used as a development environment. + +RUN apt-get -y install --no-install-recommends \ + vim emacs git + +WORKDIR /dts diff --git a/dts/README.md b/dts/README.md index d8f88f97fe..55a272d767 100644 --- a/dts/README.md +++ b/dts/README.md @@ -12,4 +12,76 @@ The Python Version required by DTS is specified in [DTS python config file](./pyproject.toml) in the **[tool.poetry.dependencies]** section. Poetry doesn't install Python, so you may need to satisfy this requirement if your Python is not up to date. A tool such as [Pyenv](https://github.com/pyenv/pyenv) -is a good way to get Python, though not the only one. +is a good way to get Python, though not the only one. However, DTS includes a +development environment in the form of a Docker image. + +# Expected Environment + +The expected execution and development environments for DTS are the same, +the container defined by [Dockerfile](./Dockerfile). Using a container for the +development environment helps with a few things. + +1. It helps enforce the boundary between the tester and the traffic + generator/sut, something which has experienced issues in the past. +2. It makes creating containers to run DTS inside automated tooling + much easier, since they can be based off of a known-working environment + that will be updated as DTS is. +3. It abstracts DTS from the server it is running on. This means that the + bare-metal os can be whatever corporate policy or your personal preferences + dictate, and DTS does not have to try to support all 15 distros that + are supported by DPDK CI. +4. It makes automated testing for DTS easier, since new dependencies can be + sent in with the patches. +5. It fixes the issue of undocumented dependencies, where some test suites + require python libraries that are not installed. +6. Allows everyone to use the same python version easily, even if they are + using an LTS distro or Windows. +7. Allows you to run the tester on Windows while developing via Docker for + Windows. + +## Tips for setting up a development environment + +### Getting a docker shell + +These commands will give you a bash shell inside the container with all the python +dependencies installed. This will place you inside a python virtual +environment. DTS is mounted via a volume, which is essentially a symlink +from the host to the container. This enables you to edit and run inside the container +and then delete the container when you are done, keeping your work. + +```shell +docker build --target dev -t dpdk-dts . +docker run -v $(pwd):/dts -it dpdk-dts bash +$ poetry install +$ poetry shell +``` + +### Vim/Emacs + +Any editor in the ubuntu repos should be easy to use. You can add your normal +config files as a volume, enabling you to use your preferred settings. + +```shell +apt install vim +apt install emacs +``` + +### Visual Studio Code + +VSCode has first-class support for developing with containers. You may need to run the +non-docker setup commands in the integrated terminal. DTS contains a .devcontainer +config, so if you open the folder in vscode it should prompt you to use the dev +container assuming you have the plugin installed. Please refer to +[VS Development Containers Docs](https://code.visualstudio.com/docs/remote/containers) +to set it all up. + +### Other + +Searching for '$IDE dev containers' will probably lead you in the right +direction. + +# Python Formatting + +The tools used to format Python code in DTS are Black and Isort. There's a shell +script, function.sh, which runs the formatters. Poetry will install these tools, +so once you have that set up, you should run it before submitting patches. diff --git a/dts/format.sh b/dts/format.sh new file mode 100755 index 0000000000..7d72335470 --- /dev/null +++ b/dts/format.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2022 University of New Hampshire +# Copyright(c) 2022 PANTHEON.tech s.r.o. +# + +function main() { + # The directory to work on is either passed in as argument 1, + # or is the current working directory + DIRECTORY=${1:-$(pwd)} + LINE_LENGTH=88 + + BLACK_VERSION=$(awk '/\[tool.poetry.dev-dependencies\]/,/$^/' pyproject.toml |\ + grep black | grep -o '[0-9][^"]*') + + PYTHON_VERSION=$(awk '/\[tool.poetry.dependencies\]/,/$^/' pyproject.toml |\ + grep python | grep -o '[0-9][^"]*' | tr -d '.') + + isort \ + --overwrite-in-place \ + --profile black \ + -j "$(nproc)" \ + --line-length $LINE_LENGTH \ + --python-version auto \ + "$DIRECTORY" + + black \ + --line-length $LINE_LENGTH \ + --required-version "${BLACK_VERSION}" \ + --target-version "py${PYTHON_VERSION}" \ + --safe \ + "$DIRECTORY" +} + +function help() { + echo "usage: format.sh " +} + +if [ "$1" == "-h" ] || [ "$1" == "--help" ]; then + help + exit 0 +fi + +main "$1" + -- 2.30.2