From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5E54D43C30; Thu, 29 Feb 2024 17:14:32 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4036B42DBE; Thu, 29 Feb 2024 17:14:32 +0100 (CET) Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) by mails.dpdk.org (Postfix) with ESMTP id 1E7C1402E0 for ; Thu, 29 Feb 2024 17:14:31 +0100 (CET) Received: by mail-lj1-f179.google.com with SMTP id 38308e7fff4ca-2d236ae4f45so1082921fa.0 for ; Thu, 29 Feb 2024 08:14:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1709223270; x=1709828070; darn=dpdk.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=fgjpmw8kau5FJyTYsttL8RoFIvvxUrZJc+svKLlmEsc=; b=Lb0TG4IBacBBiFSgZNWg0m79PahTOaOBrSIiYhCkyJfW/CIxUYWk0CaA/Bpy+j7wUz Smqy89I+vtZeSeQTqSkX0/sIjchn3PCaZ812ymdfK6VnV96+ww/R+eqr5FwurayADKTI FrTKpmL/cWyIzfYa7biTMcGcn40b2tdwC+OXw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709223270; x=1709828070; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fgjpmw8kau5FJyTYsttL8RoFIvvxUrZJc+svKLlmEsc=; b=WpTDvv3WnUi3ZKmP9Ei2axFShdOwd+1HPGRrN98oapI/VRWA6skH41ywmxQmRoigQZ OkO0Cd3meSLF/cYmOgaAnqOeir54YCZ/wIgpiJWUpCY40ApRsnvaJNhn6ppydiI+xpJQ /NKIHKKcc5sp5Q8Ruy97sWLBWJb1ZKMs6jyhp7YvE0JT4Rncv8TCMv4RIX/OgP3N9jmJ 9woLYMRuSKEaxo+N/taN3OyGSRBN0tMCaeUEuZJe6rXkW9/Zj8rKnMGbYRImwG2kIQpp s04iQq5x8OuWRopdk/wY44/xYKZJma//CFOF7AYbEzr2gZiSjJAnz68IUkvEj5Vi65Dm VoTQ== X-Gm-Message-State: AOJu0YyS5Pj3DDdXhle8fNseAkJxdr/AB0/EEbEsWdHAlZku9gGgaaeO nCpgzCsc3MVZPiJWQSXIkCbQYPQQ0eSw99days9pWkq4AxBVcQ6R9dQ/FeNMVVZipCSDIwdOMJY 9Q8Xv8S2dLi0YuHRyPDdflIeuV0raYZVdOxCEYApj500s36h5M6s= X-Google-Smtp-Source: AGHT+IEGNMWHkPKKEb5jVALMrmdY9YuhSFD2nWwor+pSO4N3+t6k64OajcRfn+eRT0dnVRSGfhitvl7sFq1vp8fiwUU= X-Received: by 2002:a05:651c:14a:b0:2d2:d9b2:b300 with SMTP id c10-20020a05651c014a00b002d2d9b2b300mr1667835ljd.3.1709223269819; Thu, 29 Feb 2024 08:14:29 -0800 (PST) MIME-Version: 1.0 References: <20240111222608.13239-1-jspewock@iol.unh.edu> <20240116191814.31316-1-jspewock@iol.unh.edu> In-Reply-To: From: Nicholas Pratte Date: Thu, 29 Feb 2024 11:14:18 -0500 Message-ID: Subject: Re: [PATCH v4] dts: add Dockerfile To: dev@dpdk.org, Jeremy Spewock Content-Type: multipart/alternative; boundary="000000000000d709dc0612878db8" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --000000000000d709dc0612878db8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Tested-by: Nicholas Pratte On Thu, Feb 29, 2024 at 10:48=E2=80=AFAM Patrick Robb w= rote: > > > ---------- Forwarded message --------- > From: > Date: Tue, Jan 16, 2024 at 2:18=E2=80=AFPM > Subject: [PATCH v4] dts: add Dockerfile > To: , , < > thomas@monjalon.net>, , , = < > paul.szczepanek@arm.com>, > Cc: , Jeremy Spewock > > > From: Juraj Linke=C5=A1 > > The Dockerfile defines development and CI runner images. > > Signed-off-by: Juraj Linke=C5=A1 > Signed-off-by: Jeremy Spewock > --- > v4: > > Remove an example from and updated a comment in the devcontainer.json > and added the --no-root flag to the README to comply with the warning > message and the discussion on slack. > > v3: > > Remove pexpect. > > v2: > > This verson updates the dockerfile to instead install poetry using pipx > due to the version of poetry installed using the package repositories of > the distro being out of date, and to conform to documentation on > installing poetry. > > This version also adds extra information to the README about the > preference of using SSH keys, and added a way to inject them into the > devcontainer for vscode. > > dts/.devcontainer/devcontainer.json | 30 +++++++++++++ > dts/Dockerfile | 38 ++++++++++++++++ > dts/README.md | 70 +++++++++++++++++++++++++++++ > 3 files changed, 138 insertions(+) > create mode 100644 dts/.devcontainer/devcontainer.json > create mode 100644 dts/Dockerfile > create mode 100644 dts/README.md > > diff --git a/dts/.devcontainer/devcontainer.json > b/dts/.devcontainer/devcontainer.json > new file mode 100644 > index 0000000000..4d737f1b40 > --- /dev/null > +++ b/dts/.devcontainer/devcontainer.json > @@ -0,0 +1,30 @@ > +// For format details, see https://aka.ms/devcontainer.json. For config > options, see the README at: > +// > https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containe= rs/docker-existing-dockerfile > +{ > + "name": "Existing Dockerfile", > + > + // Sets the run context to one level up instead of the > .devcontainer folder. > + "context": "..", > + > + // Update the 'dockerFile' property if you aren't using the > standard 'Dockerfile' filename. > + "dockerFile": "../Dockerfile", > + > + // Use 'forwardPorts' to make a list of ports inside the containe= r > available locally. > + // "forwardPorts": [], > + > + // The next line runs commands after the container is created - i= n > our case, installing dependencies. > + "postCreateCommand": "poetry install --no-root", > + > + "extensions": [ > + "ms-python.vscode-pylance", > + ] > + > + // Uncomment when using a ptrace-based debugger like C++, Go, and > Rust > + // "runArgs": [ "--cap-add=3DSYS_PTRACE", "--security-opt", > "seccomp=3Dunconfined" ], > + > + // Uncomment to use the Docker CLI from inside the container. See > https://aka.ms/vscode-remote/samples/docker-from-docker. > + // "mounts": [ > "source=3D/var/run/docker.sock,target=3D/var/run/docker.sock,type=3Dbind"= ], > + > + // Uncomment to mount your SSH keys into the devcontainer used by > vscode. > + // "mounts": > ["source=3D${localEnv:HOME}/.ssh,destination=3D/root/.ssh,type=3Dbind,rea= donly"] > +} > diff --git a/dts/Dockerfile b/dts/Dockerfile > new file mode 100644 > index 0000000000..fa4c1af10e > --- /dev/null > +++ b/dts/Dockerfile > @@ -0,0 +1,38 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2022 University of New Hampshire > + > +# There are two Docker images defined in this Dockerfile. > +# One is to be used in CI for automated testing. > +# The other provides a DTS development environment, simplifying Python > dependency management. > + > +FROM ubuntu:22.04 AS base > + > +RUN apt-get -y update && apt-get -y upgrade && \ > + apt-get -y install --no-install-recommends \ > + python3 \ > + python3-pip \ > + pipx \ > + python3-cachecontrol \ > + openssh-client && \ > + pipx install poetry>=3D1.5.1 && pipx ensurepath > +WORKDIR /dpdk/dts > + > + > +FROM base AS runner > + > +# This image is intended to be used as the base for automated systems. > +# It bakes DTS into the image during the build. > + > +COPY . /dpdk/dts > +RUN poetry install --no-dev > + > +CMD ["poetry", "run", "python", "main.py"] > + > +FROM base AS dev > + > +# This image is intended to be used as DTS development environment. It > doesn't need C compilation > +# capabilities, only Python dependencies. Once a container mounting DTS > using this image is running, > +# the dependencies should be installed using Poetry. > + > +RUN apt-get -y install --no-install-recommends \ > + vim emacs git > diff --git a/dts/README.md b/dts/README.md > new file mode 100644 > index 0000000000..36c8cc9a0c > --- /dev/null > +++ b/dts/README.md > @@ -0,0 +1,70 @@ > +# DTS Environment > +The execution and development environments for DTS are the same, a > +[Docker](https://docs.docker.com/) container defined by our > [Dockerfile](./Dockerfile). > +Using a container for the development environment helps with a few thing= s. > + > +1. It helps enforce the boundary between the DTS environment and the > TG/SUT, something > + which caused issues in the past. > +2. It makes creating containers to run DTS inside automated tooling much > easier, since > + they can be based off of a known-working environment that will be > updated as DTS is. > +3. It abstracts DTS from the server it is running on. This means that th= e > bare-metal os > + can be whatever corporate policy or your personal preferences dictate= , > and DTS does > + not have to try to support all distros that are supported by DPDK CI. > +4. It makes automated testing for DTS easier, since new dependencies can > be sent in with > + the patches. > +5. It fixes the issue of undocumented dependencies, where some test > suites require > + python libraries that are not installed. > +6. Allows everyone to use the same python version easily, even if they > are using a > + distribution or Windows with out-of-date packages. > +7. Allows you to run the tester on Windows while developing via Docker > for Windows. > + > +## Tips for setting up a development environment > + > +### Getting a docker shell > +These commands will give you a bash shell inside the container with all > the python > +dependencies installed. This will place you inside a python virtual > environment. DTS is > +mounted via a volume, which is essentially a symlink from the host to th= e > container. > +This enables you to edit and run inside the container and then delete th= e > container when > +you are done, keeping your work. It is also strongly recommended that yo= u > mount your SSH > +keys into the container to allow you to connect to hosts without > specifying a password. > + > +#### Start docker container with SSH keys > + > +```shell > +docker build --target dev -t dpdk-dts . > +docker run -v $(pwd)/..:/dpdk -v /home/dtsuser/.ssh:/root/.ssh:ro -it > dpdk-dts bash > +$ poetry install --no-root > +$ poetry shell > +``` > + > +#### Start docker container without SSH keys > + > +```shell > +docker build --target dev -t dpdk-dts . > +docker run -v $(pwd)/..:/dpdk -it dpdk-dts bash > +$ poetry install --no-root > +$ poetry shell > +``` > + > +### Vim/Emacs > +Any editor in the ubuntu repos should be easy to use, with vim and emacs > already > +installed. You can add your normal config files as a volume, enabling yo= u > to use your > +preferred settings. > + > +```shell > +docker run -v ${HOME}/.vimrc:/root/.vimrc -v $(pwd)/..:/dpdk -it dpdk-dt= s > bash > +``` > + > +### Visual Studio Code > +VSCode has first-class support for developing with containers. You may > need to run the > +non-docker setup commands in the integrated terminal. DTS contains a > .devcontainer > +config, so if you open the folder in vscode it should prompt you to use > the dev > +container assuming you have the plugin installed. Please refer to > +[VS Development Containers Docs]( > https://code.visualstudio.com/docs/remote/containers) > +to set it all up. Additionally, there is a line in > `.devcontainer/devcontainer.json` > +that, when included, will mount the SSH keys of the user currently > running vscode into > +the container for you. The `source` on this line can be altered to mount > any SSH keys on > +the local machine into the container at the correct location. > + > +### Other > +Searching for '$IDE dev containers' will probably lead you in the right > direction. > -- > 2.43.0 > > > --000000000000d709dc0612878db8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Tested-by: Nicholas Pratte= <npratte@iol.unh.edu>
=

On Thu, Feb 29, 2024 at 10:48=E2=80=AFAM Patrick Robb <probb@iol.unh.edu> wrote:


---------- F= orwarded message ---------
From: <jspewock@iol.unh.edu>
Date: Tue, Jan 16, 2024 at 2:18=E2=80=AFPM
Subject: [PATCH v4] dts= : add Dockerfile
To: <Honnappa.Nagarahalli@arm.com>, <juraj.linkes= @pantheon.tech>, <thomas@monjalon.net>, <wathsala.vithanage@arm.com>, <probb@iol.unh.edu>= ;, <paul.s= zczepanek@arm.com>, <yoan.picchi@foss.arm.com>
Cc: <dev@dpdk.org>, Jeremy Spewock= <jspewock@iol= .unh.edu>


From: Juraj Linke=C5=A1 <juraj.linkes@= pantheon.tech>

The Dockerfile defines development and CI runner images.

Signed-off-by: Juraj Linke=C5=A1 <juraj.linkes@pantheon.tech>
Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
v4:

Remove an example from and updated a comment in the devcontainer.json
and added the --no-root flag to the README to comply with the warning
message and the discussion on slack.

v3:

Remove pexpect.

v2:

This verson updates the dockerfile to instead install poetry using pipx
due to the version of poetry installed using the package repositories of the distro being out of date, and to conform to documentation on
installing poetry.

This version also adds extra information to the README about the
preference of using SSH keys, and added a way to inject them into the
devcontainer for vscode.

=C2=A0dts/.devcontainer/devcontainer.json | 30 +++++++++++++
=C2=A0dts/Dockerfile=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 | 38 ++++++++++++++++
=C2=A0dts/README.md=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0| 70 +++++++++++++++++++++++++++++
=C2=A03 files changed, 138 insertions(+)
=C2=A0create mode 100644 dts/.devcontainer/devcontainer.json
=C2=A0create mode 100644 dts/Dockerfile
=C2=A0create mode 100644 dts/README.md

diff --git a/dts/.devcontainer/devcontainer.json b/dts/.devcontainer/devcon= tainer.json
new file mode 100644
index 0000000000..4d737f1b40
--- /dev/null
+++ b/dts/.devcontainer/devcontainer.json
@@ -0,0 +1,30 @@
+// For format details, see https://aka.ms/devcontainer.json. Fo= r config options, see the README at:
+// https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/cont= ainers/docker-existing-dockerfile
+{
+=C2=A0 =C2=A0 =C2=A0 =C2=A0"name": "Existing Dockerfile&quo= t;,
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// Sets the run context to one level up instead= of the .devcontainer folder.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0"context": "..",
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// Update the 'dockerFile' property if = you aren't using the standard 'Dockerfile' filename.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0"dockerFile": "../Dockerfile&quo= t;,
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// Use 'forwardPorts' to make a list of= ports inside the container available locally.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// "forwardPorts": [],
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// The next line runs commands after the contai= ner is created - in our case, installing dependencies.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0"postCreateCommand": "poetry ins= tall --no-root",
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0"extensions": [
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"ms-python.vsc= ode-pylance",
+=C2=A0 =C2=A0 =C2=A0 =C2=A0]
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// Uncomment when using a ptrace-based debugger= like C++, Go, and Rust
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// "runArgs": [ "--cap-add=3DSYS= _PTRACE", "--security-opt", "seccomp=3Dunconfined"= ],
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// Uncomment to use the Docker CLI from inside = the container. See https://aka.ms/vscode-remo= te/samples/docker-from-docker.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// "mounts": [ "source=3D/var/ru= n/docker.sock,target=3D/var/run/docker.sock,type=3Dbind" ],
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// Uncomment to mount your SSH keys into the de= vcontainer used by vscode.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0// "mounts": ["source=3D${localE= nv:HOME}/.ssh,destination=3D/root/.ssh,type=3Dbind,readonly"]
+}
diff --git a/dts/Dockerfile b/dts/Dockerfile
new file mode 100644
index 0000000000..fa4c1af10e
--- /dev/null
+++ b/dts/Dockerfile
@@ -0,0 +1,38 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2022 University of New Hampshire
+
+# There are two Docker images defined in this Dockerfile.
+# One is to be used in CI for automated testing.
+# The other provides a DTS development environment, simplifying Python dep= endency management.
+
+FROM ubuntu:22.04 AS base
+
+RUN apt-get -y update && apt-get -y upgrade && \
+=C2=A0 =C2=A0 apt-get -y install --no-install-recommends \
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 python3 \
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 python3-pip \
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 pipx \
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 python3-cachecontrol \
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 openssh-client && \
+=C2=A0 =C2=A0 pipx install poetry>=3D1.5.1 && pipx ensurepath +WORKDIR /dpdk/dts
+
+
+FROM base AS runner
+
+# This image is intended to be used as the base for automated systems.
+# It bakes DTS into the image during the build.
+
+COPY . /dpdk/dts
+RUN poetry install --no-dev
+
+CMD ["poetry", "run", "python", "main.p= y"]
+
+FROM base AS dev
+
+# This image is intended to be used as DTS development environment. It doe= sn't need C compilation
+# capabilities, only Python dependencies. Once a container mounting DTS us= ing this image is running,
+# the dependencies should be installed using Poetry.
+
+RUN apt-get -y install --no-install-recommends \
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 vim emacs git
diff --git a/dts/README.md b/dts/README.md
new file mode 100644
index 0000000000..36c8cc9a0c
--- /dev/null
+++ b/dts/README.md
@@ -0,0 +1,70 @@
+# DTS Environment
+The execution and development environments for DTS are the same, a
+[Docker](https://docs.docker.com/) container defined by our [Dockerfile= ](./Dockerfile).
+Using a container for the development environment helps with a few things.=
+
+1. It helps enforce the boundary between the DTS environment and the TG/SU= T, something
+=C2=A0 =C2=A0which caused issues in the past.
+2. It makes creating containers to run DTS inside automated tooling much e= asier, since
+=C2=A0 =C2=A0they can be based off of a known-working environment that wil= l be updated as DTS is.
+3. It abstracts DTS from the server it is running on. This means that the = bare-metal os
+=C2=A0 =C2=A0can be whatever corporate policy or your personal preferences= dictate, and DTS does
+=C2=A0 =C2=A0not have to try to support all distros that are supported by = DPDK CI.
+4. It makes automated testing for DTS easier, since new dependencies can b= e sent in with
+=C2=A0 the patches.
+5. It fixes the issue of undocumented dependencies, where some test suites= require
+=C2=A0 =C2=A0python libraries that are not installed.
+6. Allows everyone to use the same python version easily, even if they are= using a
+=C2=A0 =C2=A0distribution or Windows with out-of-date packages.
+7. Allows you to run the tester on Windows while developing via Docker for= Windows.
+
+## Tips for setting up a development environment
+
+### Getting a docker shell
+These commands will give you a bash shell inside the container with all th= e python
+dependencies installed. This will place you inside a python virtual enviro= nment. DTS is
+mounted via a volume, which is essentially a symlink from the host to the = container.
+This enables you to edit and run inside the container and then delete the = container when
+you are done, keeping your work. It is also strongly recommended that you = mount your SSH
+keys into the container to allow you to connect to hosts without specifyin= g a password.
+
+#### Start docker container with SSH keys
+
+```shell
+docker build --target dev -t dpdk-dts .
+docker run -v $(pwd)/..:/dpdk -v /home/dtsuser/.ssh:/root/.ssh:ro -it dpdk= -dts bash
+$ poetry install --no-root
+$ poetry shell
+```
+
+#### Start docker container without SSH keys
+
+```shell
+docker build --target dev -t dpdk-dts .
+docker run -v $(pwd)/..:/dpdk -it dpdk-dts bash
+$ poetry install --no-root
+$ poetry shell
+```
+
+### Vim/Emacs
+Any editor in the ubuntu repos should be easy to use, with vim and emacs a= lready
+installed. You can add your normal config files as a volume, enabling you = to use your
+preferred settings.
+
+```shell
+docker run -v ${HOME}/.vimrc:/root/.vimrc -v $(pwd)/..:/dpdk -it dpdk-dts = bash
+```
+
+### Visual Studio Code
+VSCode has first-class support for developing with containers. You may nee= d to run the
+non-docker setup commands in the integrated terminal. DTS contains a .devc= ontainer
+config, so if you open the folder in vscode it should prompt you to use th= e dev
+container assuming you have the plugin installed. Please refer to
+[VS Development Containers Docs](https://code.v= isualstudio.com/docs/remote/containers)
+to set it all up. Additionally, there is a line in `.devcontainer/devconta= iner.json`
+that, when included, will mount the SSH keys of the user currently running= vscode into
+the container for you. The `source` on this line can be altered to mount a= ny SSH keys on
+the local machine into the container at the correct location.
+
+### Other
+Searching for '$IDE dev containers' will probably lead you in the = right direction.
--
2.43.0


=C2=A0
--000000000000d709dc0612878db8--