From: Adam Hassick <ahassick@iol.unh.edu>
To: ci@dpdk.org
Cc: alialnu@nvidia.com, aconole@redhat.com,
Adam Hassick <ahassick@iol.unh.edu>
Subject: [PATCH v7 0/6] Community Lab Containers and Builder Engine
Date: Tue, 11 Jul 2023 16:21:18 -0400 [thread overview]
Message-ID: <20230711202124.1636317-1-ahassick@iol.unh.edu> (raw)
This patch series contains a new version of the DPDK CI containers. The old
version was tied very tightly to the Community Lab Infrastructure, so it was
not suitable for general use. This version is designed to make adding new
OSes or OS versions as easy as possible. The minimum functionality can easily
be built on any system that can compile DPDK. It includes support for
building containers for other non-native architectures (ex: arm containers
on x86) and for baking ABI references into the images.
The inventory file as added in this patch series defines what the community lab
currently supports.
If you want to build these yourself, don't try to do parallel Makefile builds
on non-server hardware. Libabigail is built into the containers, and since it
is not avaliable in all distros it is compiled from source for many targets.
If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the
current settings, DPDK will be compiled twice for every target in the
inventory file. This will become three times when DPDK main has ABI stability
again. Due to how the containers are constructed, building on non-native
architectures is especially expensive because DPDK is compiled with an emulated
compiler in a qemu vm that only has one thread.
---
v2:
* Added poetry.lock
* Remove duplicates in inventory file
* Fix typo in base container template
v3:
* Libabigail is now not built by default unless DPDK_CI_CONTAINERS_BUILD_ABI=y
* Made minimum podman version 4.0.0 due to dependencies on cache mounts
v4:
* Fix whitespace errors
v5:
* Updated container templates and inventory
* Add support for OCI manifests
* Update README to reflect changes
v6:
* Fix spelling errors and whitespace errors
v7:
* Add workaround to Arch template to install Python packages as root
* Compare against v4 and polish the patchset
Owen Hilyard (6):
* Add README file for containers * Add pyproject file with required
dependencies for building containers
Adds a yaml file used to define what containers should be built, what
packages the containers should have, what platforms to build for and
how they should be tagged.
This script will template out all of the Dockerfiles based on the
definitions provided in the inventory using the jinja2 templating
library.
Adds a variety of extensible templates used to create the Dockerfiles
for each target. All templates inherit from base, and most distros
and distro families have their own templates that are used to define
common functionality. Multiple versions of a distro may use a single
template if they are similar enough (ex: fedora).
Adds a container that can be used to run the python scripts to create
the Dockerfiles for the CI containers. This removes a large number
of python environment dependencies from the host requirements.
The Makefile that can be used to build all of the container images
using "make build", and can also be used to push them to a remote
repository (for use in CI).
containers/Makefile | 252 ++++++++++
containers/README.md | 178 +++++++
containers/container_builder.dockerfile | 26 +
containers/template_engine/inventory.yaml | 470 ++++++++++++++++++
.../template_engine/inventory_schema.json | 221 ++++++++
containers/template_engine/make_dockerfile.py | 358 +++++++++++++
containers/template_engine/poetry.lock | 220 ++++++++
containers/template_engine/pyproject.toml | 21 +
.../templates/containers.makefile.j2 | 73 +++
.../templates/containers/alpine.dockerfile.j2 | 3 +
.../templates/containers/arch.dockerfile.j2 | 39 ++
.../templates/containers/base.dockerfile.j2 | 120 +++++
.../containers/centos8.dockerfile.j2 | 21 +
.../containers/centos9.dockerfile.j2 | 17 +
.../templates/containers/debian.dockerfile.j2 | 7 +
.../containers/debian10.dockerfile.j2 | 3 +
.../containers/debian11.dockerfile.j2 | 3 +
.../debian11_arm_ipsec.dockerfile.j2 | 16 +
.../containers/debian_bullseye.dockerfile.j2 | 3 +
.../containers/debian_buster.dockerfile.j2 | 3 +
.../templates/containers/fedora.dockerfile.j2 | 11 +
.../containers/fedora36_clang.dockerfile.j2 | 7 +
.../containers/fedora_clang.dockerfile.j2 | 8 +
.../containers/fedora_coverity.dockerfile.j2 | 10 +
.../containers/opensuse.dockerfile.j2 | 10 +
.../containers/redhat_family.dockerfile.j2 | 5 +
.../templates/containers/rhel.dockerfile.j2 | 16 +
.../templates/containers/rhel7.dockerfile.j2 | 15 +
.../templates/containers/rhel8.dockerfile.j2 | 15 +
.../templates/containers/rhel9.dockerfile.j2 | 19 +
.../templates/containers/rpm.dockerfile.j2 | 3 +
.../templates/containers/ubuntu.dockerfile.j2 | 3 +
.../containers/ubuntu20.04.dockerfile.j2 | 12 +
.../containers/ubuntu22.04.dockerfile.j2 | 3 +
.../containers/ubuntu_cross.dockerfile.j2 | 11 +
.../containers/ubuntu_sve.dockerfile.j2 | 12 +
36 files changed, 2214 insertions(+)
create mode 100644 containers/Makefile
create mode 100644 containers/README.md
create mode 100644 containers/container_builder.dockerfile
create mode 100644 containers/template_engine/inventory.yaml
create mode 100644 containers/template_engine/inventory_schema.json
create mode 100755 containers/template_engine/make_dockerfile.py
create mode 100644 containers/template_engine/poetry.lock
create mode 100644 containers/template_engine/pyproject.toml
create mode 100644 containers/template_engine/templates/containers.makefile.j2
create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
--
2.34.1
next reply other threads:[~2023-07-11 20:22 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-11 20:21 Adam Hassick [this message]
2023-07-11 20:21 ` [PATCH v7 1/6] containers/docs: Add container builder start Adam Hassick
2023-07-12 11:43 ` Ali Alnubani
2023-07-11 20:21 ` [PATCH v7 2/6] containers/inventory: Add inventory for container builder Adam Hassick
2023-07-11 20:21 ` [PATCH v7 3/6] containers/builder: Dockerfile creation script Adam Hassick
2023-07-11 20:21 ` [PATCH v7 4/6] containers/templates: Templates for Dockerfiles Adam Hassick
2023-07-11 20:21 ` [PATCH v7 5/6] containers/container_builder: Container for python scripts Adam Hassick
2023-07-11 20:21 ` [PATCH v7 6/6] containers/Makefile: Makefile to automate builds Adam Hassick
2023-07-12 11:43 ` Ali Alnubani
2023-07-12 11:45 ` [PATCH v7 0/6] Community Lab Containers and Builder Engine Ali Alnubani
2023-07-17 19:07 ` Aaron Conole
2023-07-18 13:40 ` Adam Hassick
2023-07-18 14:17 ` Aaron Conole
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230711202124.1636317-1-ahassick@iol.unh.edu \
--to=ahassick@iol.unh.edu \
--cc=aconole@redhat.com \
--cc=alialnu@nvidia.com \
--cc=ci@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).