* [PATCH v5 00/11] Community Lab Containers and Builder Engine
@ 2023-05-23 17:04 Adam Hassick
2023-05-23 17:04 ` [PATCH v5 01/11] containers/docs: Add container builder start Adam Hassick
` (10 more replies)
0 siblings, 11 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Adam Hassick, aconole, alialnu
This patch series contains a new version of the DPDK CI containers. The old
version was tied very tightly to the Community Lab Infrastructure, so it was
not suitable for general use. This version is designed to make adding new
OSes or OS versions as easy as possible. The minimum functionality can easily
be built on any system that can compile DPDK. It includes support for
building containers for other non-native architectures (ex: arm containers
on x86) and for baking ABI references into the images. Support for
building the Coverity Scan container image has also been added.
The inventory file as added in this patch series defines what the community lab
currently supports.
If you want to build these yourself, don't try to do parallel Makefile builds
on non-server hardware. Libabigail is built into the containers, and since it
is not avaliable in all distros it is compiled from source for many targets.
If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the
current settings, DPDK will be compiled thrice for every target in the
inventory file. Due to how the containers are constructed, building on
non-native architectures is especially expensive because DPDK is compiled with
an emulated compiler in a qemu vm that only has one thread.
Adam Hassick (5):
containers/inventory: Add ABI rev and coverity attribute
containers/builder: Add arguments to templating script
containers/templates: Update templates
containers/Makefile: Add new variables and target changes
containers/docs: Update README
Owen Hilyard (6):
containers/docs: Add container builder start
containers/inventory: Add inventory for container builder
containers/builder: Dockerfile creation script
containers/templates: Templates for Dockerfiles
containers/container_builder: Container for python scripts
containers/Makefile: Makefile to automate builds
containers/Makefile | 252 ++++++++++
containers/README.md | 169 +++++++
containers/container_builder.dockerfile | 26 ++
containers/template_engine/inventory.yaml | 442 ++++++++++++++++++
.../template_engine/inventory_schema.json | 221 +++++++++
containers/template_engine/make_dockerfile.py | 358 ++++++++++++++
containers/template_engine/poetry.lock | 220 +++++++++
containers/template_engine/pyproject.toml | 21 +
.../templates/containers.makefile.j2 | 73 +++
.../templates/containers/alpine.dockerfile.j2 | 4 +
.../templates/containers/arch.dockerfile.j2 | 37 ++
.../templates/containers/base.dockerfile.j2 | 121 +++++
.../containers/centos8.dockerfile.j2 | 21 +
.../containers/centos9.dockerfile.j2 | 17 +
.../templates/containers/debian.dockerfile.j2 | 7 +
.../containers/debian10.dockerfile.j2 | 3 +
.../containers/debian11.dockerfile.j2 | 3 +
.../debian11_arm_ipsec.dockerfile.j2 | 16 +
.../containers/debian_bullseye.dockerfile.j2 | 3 +
.../containers/debian_buster.dockerfile.j2 | 3 +
.../templates/containers/fedora.dockerfile.j2 | 11 +
.../containers/fedora36_clang.dockerfile.j2 | 7 +
.../containers/fedora_clang.dockerfile.j2 | 7 +
.../containers/fedora_coverity.dockerfile.j2 | 10 +
.../containers/opensuse.dockerfile.j2 | 10 +
.../containers/redhat_family.dockerfile.j2 | 5 +
.../templates/containers/rhel.dockerfile.j2 | 16 +
.../templates/containers/rhel7.dockerfile.j2 | 15 +
.../templates/containers/rhel8.dockerfile.j2 | 15 +
.../templates/containers/rhel9.dockerfile.j2 | 19 +
.../templates/containers/rpm.dockerfile.j2 | 3 +
.../templates/containers/ubuntu.dockerfile.j2 | 3 +
.../containers/ubuntu20.04.dockerfile.j2 | 12 +
.../containers/ubuntu22.04.dockerfile.j2 | 3 +
.../containers/ubuntu_cross.dockerfile.j2 | 12 +
.../containers/ubuntu_sve.dockerfile.j2 | 12 +
36 files changed, 2177 insertions(+)
create mode 100644 containers/Makefile
create mode 100644 containers/README.md
create mode 100644 containers/container_builder.dockerfile
create mode 100644 containers/template_engine/inventory.yaml
create mode 100644 containers/template_engine/inventory_schema.json
create mode 100755 containers/template_engine/make_dockerfile.py
create mode 100644 containers/template_engine/poetry.lock
create mode 100644 containers/template_engine/pyproject.toml
create mode 100644 containers/template_engine/templates/containers.makefile.j2
create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 01/11] containers/docs: Add container builder start
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-25 13:48 ` Aaron Conole
2023-05-23 17:04 ` [PATCH v5 02/11] containers/inventory: Add inventory for container builder Adam Hassick
` (9 subsequent siblings)
10 siblings, 1 reply; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Owen Hilyard, aconole, alialnu
From: Owen Hilyard <ohilyard@iol.unh.edu>
* Add README file for containers
* Add pyproject file with required dependencies for building containers
This module allows anyone to build the containers used in DPDK CI, and
allows the community to contribute container definitions back to DPDK
CI. Please read the README for more information, since some
functionality is opt-in due to resource requirements.
Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
containers/README.md | 138 ++++++++++++++++++++++
containers/template_engine/pyproject.toml | 21 ++++
2 files changed, 159 insertions(+)
create mode 100644 containers/README.md
create mode 100644 containers/template_engine/pyproject.toml
diff --git a/containers/README.md b/containers/README.md
new file mode 100644
index 0000000..5d01caf
--- /dev/null
+++ b/containers/README.md
@@ -0,0 +1,138 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2022 University of New Hampshire
+
+DPDK CI Container Build System
+==============================
+
+## Rational
+
+There are a few important factors for why a custom build system was created for
+the containers instead of using an existing one. The first was that podman was
+nearly mandatory for this task.
+
+### Why Podman
+
+1. Licensed RHEL containers need podman
+
+The build system MUST be able to handle creating properly licensed RHEL
+containers, so that the RHEL CI testing is as accurate as possible.
+
+2. "Developer Laptop Friendliness"
+
+Another goal of the build system was to enable anyone to easily build the
+containers. Not all developers are able to use Linux as the main OS on their
+main development machine. Podman runs on MacOS via podman-machine and Windows
+either by podman-machine or WSL.
+
+3. OCI Containers
+
+OCI containers are more portable than some other container solutions. Much of
+the progress on getting containers running on top of FreeBSD jails targets OCI
+containers specifically. The tracking issue for this is
+[https://reviews.freebsd.org/D21570](https://reviews.freebsd.org/D21570).
+Once upstream support happens, there should be a relatively simple path to
+supporting containers in FreeBSD once podman/docker APIs are better supported.
+At the moment, lack up upstream support means no support in this project for
+FreeBSD.
+
+### Python and Makefiles instead of Buildah as a library
+
+The next question someone might have is why a combination of Python and
+makefiles were used instead of using buildah as a library. The largest
+reason is that every DPDK developer is going to need to have some
+level of familiarity with Python due to DTS. Buildah is only available
+as a library via Go, and would tie DPDK to a particular container
+implementation. Go, while not difficult to learn, is a compiled language,
+meaning that the build system would require a build system.
+
+The other reason is that most of the logic that needs to be performed is very
+simple, and python has a few libraries that do most of the work. If it weren't
+for the desire to have an inventory file (inventory.yaml) with a schema
+(inventory_schema.json), this probably could have been an AWK script. After the
+container images are produced, it is very easy to use the same template
+engine to produce a makefile that can be used to both build and push the
+containers. This makefile can be run with multiple jobs for parallel building
+of containers, something not supported by all compose implementations.
+
+Meson was considered instead of Makefiles, however, Meson does not handle new
+Meson being generated during the build very well, and Meson wants most commands
+to have an output file, which is not true of many of the commands. Meson is
+also more difficult to generate using a templating library than Makefile
+targets.
+
+## Building
+
+### Environment Variables
+
+All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any
+issues.
+
+| Variable | Description | Default | Valid Values |
+| -------------------------- | ----------------------------------------------- | ------- | ------------ |
+DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE | Fail during dockerfile generation if any container in the inventory is not buildable. Currently will cause a failure if you are not on RHEL and try to build RHEL containers. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_BUILD_ABI | Whether to bake ABI images into the containers. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL | Whether to build libabigail from source on distros that do not package it. DPDK_CI_CONTAINERS_BUILD_ABI=Y overrides this to 'Y' | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINER_BUILDER_PROGRAM | What container builder program to use. | 'podman' | Any container builder that exposes the same interface and provides the same behavior as podman.
+DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository.
+DPDK_CI_CONTAINERS_DPDK_CLONE_URL | What URL to clone DPDK from. | 'https://dpdk.org/git/dpdk' | Any DPDK mirror.
+DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG | What tag to give to the container which creates the dockerfiles. The default should be fine unless you have issues with collisions. | 'dpdk_ci_container_builder' | Any valid OCI container tag (A valid C function name will work)
+DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS | Extra arguments to add to the push command, can be used for credentials if 'podman login' won't work. | '' | [https://docs.podman.io/en/latest/markdown/podman-push.1.html#options](https://docs.podman.io/en/latest/markdown/podman-push.1.html#options)
+DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME | The hostname of the registry to push to. | 'localhost' | The hostname of any system exposing an OCI container registry or localhost to push to local storage.
+DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH | The path to a directory to copy into all of the containers at /scripts | unset | The path to any local file directory.
+DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers in. All generated files will be placed in this directory or one of it's children | '$(CURDIR)/container_context' | Any absolute directory path
+
+### Builder System Requirements
+
+#### Required Programs
+
+* GNU make (POSIX make may work, but is not supported)
+* git
+* find
+* posix utilities (GNU coreutils will work)
+* bash
+* podman >= 4.0.0 (docker or other container builder programs may work, but are
+unsupported)
+ * podman 4.0.0 allows run mounts, which allow mounting a directory into the build context of a container. This is used to persist ccache directories for each container.
+* qemu-$ARCH-static for any non-native architecture/revision you want to build for.
+
+#### Hardware
+
+| Hardware Type | Requirement | Reason |
+| ------------- | ---------------------------- | ----------------------------------- |
+| Disk space | 5 GB of disk space per image | Some images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references.
+Memory | Either 1.5x or 2x the memory needed to compile DPDK per makefile job | 1.5x is enough for the container overhead and caching when compiling natively, 2x is for builds under emulation (ARM container on x86, etc).
+
+
+#### RHEL containers
+
+RHEL container images must be built on RHEL.
+
+### Build containers locally
+
+```bash
+make build
+```
+
+### Push containers to registry
+
+This will probably involve following prompts in your terminal, but if you have
+other authentication set up, (LDAP, Kerberos, etc), it may not prompt you.
+Logging into a registry is what allows you to upload containers to a remote
+system for others to pull down.
+
+If you are working alone, you probably can ignore this and keep the containers
+locally. If you are in an enterprise setting, ask your DevOps or Systems
+Administration team where the preferred location for hosting containers is.
+
+Since these images take so long to build, it is recommended to use a container
+registry and have any CI systems pull from that registry.
+
+Redhat guide to setting up a podman container registry:
+[https://www.redhat.com/sysadmin/simple-container-registry](https://www.redhat.com/sysadmin/simple-container-registry)
+
+```bash
+$DPDK_CI_CONTAINER_BUILDER_PROGRAM login $DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME
+# < Complete login process >
+make push
+```
\ No newline at end of file
diff --git a/containers/template_engine/pyproject.toml b/containers/template_engine/pyproject.toml
new file mode 100644
index 0000000..f5611ce
--- /dev/null
+++ b/containers/template_engine/pyproject.toml
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2022 University of New Hampshire
+
+[tool.poetry]
+name = "dpdk_ci_containers"
+version = "0.1.0"
+description = ""
+authors = ["Owen Hilyard <ohilyard@iol.unh.edu>"]
+license = "BSD-3-Clause"
+
+[tool.poetry.dependencies]
+python = "^3.8"
+Jinja2 = "^3.1.2"
+jsonschema = "^4.10.0"
+PyYAML = "^6.0"
+
+[tool.poetry.dev-dependencies]
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 02/11] containers/inventory: Add inventory for container builder
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
2023-05-23 17:04 ` [PATCH v5 01/11] containers/docs: Add container builder start Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-23 17:04 ` [PATCH v5 03/11] containers/builder: Dockerfile creation script Adam Hassick
` (8 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Owen Hilyard, aconole, alialnu
From: Owen Hilyard <ohilyard@iol.unh.edu>
Adds a yaml file used to define what containers should be built, what
packages the containers should have, what platforms to build for and how they
should be tagged.
Also adds a JSON schema, which can be used to validate that the
inventory file is well-formed.
Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
containers/template_engine/inventory.yaml | 356 ++++++++++++++++++
.../template_engine/inventory_schema.json | 153 ++++++++
2 files changed, 509 insertions(+)
create mode 100644 containers/template_engine/inventory.yaml
create mode 100644 containers/template_engine/inventory_schema.json
diff --git a/containers/template_engine/inventory.yaml b/containers/template_engine/inventory.yaml
new file mode 100644
index 0000000..eb299c0
--- /dev/null
+++ b/containers/template_engine/inventory.yaml
@@ -0,0 +1,356 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2022 University of New Hampshire
+---
+abi:
+ # versions should also include the abi versions for all stable ABIs.
+ versions:
+ - major: 20 # oldest supported lts
+ minor: 11
+ git_tag: v20.11
+ - major: 21 # most recent lts
+ minor: 11
+ git_tag: v21.11
+ # - major: 22 # current abi
+ # minor: 07
+ # git_tag: v22.07
+
+dockerfiles:
+ groups:
+ all:
+ package_manager_command: "NO_INSTALL_COMMAND_SET"
+ parent: "NONE"
+ packages:
+ - gcc
+ - make
+ - git
+ - sudo
+ - curl
+ - automake
+ - autoconf
+ - flex
+ - bison
+ - bc
+ rpm:
+ parent: "all"
+ package_manager_command: yum install -y
+ packages:
+ - diffutils
+ - pkg-config
+ - python3
+ - python3-pip
+ - librdmacm
+ - rdma-core-devel
+ - libmnl-devel
+ - ccache
+ - zip
+ - autoconf
+ redhat_family:
+ parent: "rpm"
+ packages: []
+ fedora:
+ parent: "redhat_family"
+ packages:
+ - python3-pyelftools
+ - zip
+ - clang
+ - ninja-build
+ - elfutils-libelf-devel
+ - jansson-devel
+ - libatomic
+ - libpcap-devel
+ - numactl-devel
+ - findutils
+ - libtool
+ - elfutils-devel
+ - libxml2-devel
+ - libabigail
+ centos8:
+ parent: "redhat_family"
+ packages:
+ - elfutils-libelf-devel
+ - jansson-devel
+ - libatomic
+ - libpcap-devel
+ - numactl-devel
+ - libabigail
+ centos9:
+ parent: "redhat_family"
+ package_manager_command: "yum install --allowerasing -y"
+ packages:
+ - elfutils-libelf-devel
+ - jansson-devel
+ - libatomic
+ - libpcap-devel
+ - numactl-devel
+ - libtool
+ - g++
+ - elfutils-devel
+ - libxml2-devel
+ rhel:
+ parent: "redhat_family"
+ package_manager_command: "yum --disableplugin=subscription-manager install -y"
+ packages:
+ - elfutils-libelf-devel
+ - jansson-devel
+ - libatomic
+ - libpcap-devel
+ - numactl-devel
+ rhel7:
+ parent: "rhel"
+ packages:
+ - libabigail
+ - ninja-build
+ rhel8:
+ parent: "rhel"
+ package_manager_command: "yum --allowerasing --disableplugin=subscription-manager install -y"
+ packages:
+ - libabigail
+ - ninja-build
+ - libxml2-devel
+ rhel9:
+ parent: "rhel"
+ package_manager_command: "yum --allowerasing --disableplugin=subscription-manager install -y"
+ packages:
+ - libxml2-devel
+ opensuse:
+ parent: "rpm"
+ package_manager_command: "zypper install -y"
+ packages:
+ - python3-pyelftools
+ - ninja
+ - gzip
+ - libelf-devel
+ - libjansson-devel
+ - librdmacm1
+ - libatomic1
+ - libnuma-devel
+ - libabigail-tools
+ - tar
+ alpine:
+ parent: 'all'
+ package_manager_command: 'apk add'
+ packages:
+ - python3
+ - py3-pip
+ - alpine-sdk
+ - ccache
+ - cmake
+ - bash
+ - autoconf
+ - automake
+ - libtool
+ - elfutils-dev
+ - libxml2-dev
+ - fts-dev
+ - fts
+ - numactl-dev
+ - libc-dev
+ - bsd-compat-headers
+ - libexecinfo-dev
+ - linux-headers
+ - libatomic
+ - zip
+ - pkgconfig
+ arch:
+ parent: 'all'
+ package_manager_command: 'pacman -Syu --noconfirm'
+ packages:
+ - glibc
+ - cmake
+ - ninja
+ - pandoc
+ - python
+ - python-docutils
+ - ethtool
+ - file
+ - fakeroot
+ - pkg-config
+ - numactl
+ - elfutils
+ - gawk
+ - diffutils
+ - python-pip
+ - libabigail
+ debian:
+ parent: 'all'
+ package_manager_command: 'apt-get update && apt-get install --no-install-recommends -y'
+ packages:
+ - libnuma-dev
+ - zip
+ - librdmacm1
+ - libcrypto++-dev
+ - python3-pyelftools
+ - libxml2-dev
+ - libelf-dev
+ - libpcap-dev
+ - libjansson-dev
+ - python3-setuptools
+ - autoconf
+ - pkg-config
+ - libtool
+ - ccache
+ - libatomic1
+ - python3-wheel
+ - libdw-dev
+ - rdma-core
+ - ninja-build
+ - automake
+ - librdmacm-dev
+ - libmnl-dev
+ - python3-pip
+ - build-essential
+ - aspell-en
+ - parallel
+ - python3-grpcio
+ - lcov
+ debian10:
+ parent: 'debian'
+ packages: []
+ debian11:
+ parent: 'debian'
+ packages: []
+ ubuntu:
+ parent: 'debian'
+ packages: []
+ ubuntu18.04:
+ parent: 'ubuntu'
+ packages: []
+ ubuntu20.04:
+ parent: 'ubuntu'
+ packages:
+ - python3-grpcio
+ - lcov
+ - jq
+ - default-jre
+ - nodejs
+ ubuntu22.04:
+ parent: 'ubuntu'
+ packages:
+ - python3-grpcio
+ - lcov
+ - abigail-tools
+ - jq
+ ubuntu_cross: # x86 -> ARM
+ parent: 'ubuntu20.04'
+ packages:
+ - clang
+ - llvm
+ - llvm-dev
+ - llvm-runtime
+ - lld
+ - gcc-aarch64-linux-gnu
+ - libgcc-9-dev-arm64-cross
+ - libatomic1-arm64-cross
+ - libc6-dev-arm64-cross
+ - pkg-config-aarch64-linux-gnu
+ - dpkg-dev
+ - gcc-arm-linux-gnueabihf
+ - libc6-dev-armhf-cross
+ - pkg-config-arm-linux-gnueabihf
+ ubuntu_sve:
+ parent: 'ubuntu20.04'
+ packages:
+ - gcc-10
+ targets:
+
+# Alpine
+ - name: alpine_compile
+ group: 'alpine'
+ base_image: alpine
+ platforms: [linux/amd64]
+ extra_information:
+ environment_id: 45
+
+# Arch
+ - name: arch_compile
+ group: 'arch'
+ base_image: archlinux:base
+ platforms: [linux/amd64]
+
+# Fedora
+ - name: fedora35_compile
+ group: "fedora"
+ base_image: fedora:35
+ platforms: [linux/amd64]
+ - name: fedora36_compile
+ group: "fedora"
+ base_image: fedora:36
+ platforms: [linux/amd64]
+ - name: fedora36_clang
+ group: "fedora"
+ base_image: fedora:36
+ platforms: [linux/amd64]
+
+# CentOs
+ - name: centosstream8_compile
+ group: "centos8"
+ base_image: quay.io/centos/centos:stream8
+ platforms: [linux/amd64]
+
+ - name: centosstream9_compile
+ group: "centos9"
+ base_image: quay.io/centos/centos:stream9
+ platforms: [linux/amd64]
+
+# RHEL
+ - name: rhel7_compile
+ group: "rhel7"
+ base_image: registry.access.redhat.com/ubi7/ubi
+ platforms: [linux/amd64]
+
+ - name: rhel8_compile
+ group: "rhel8"
+ base_image: registry.access.redhat.com/ubi8/ubi
+ platforms: [linux/amd64]
+
+ - name: rhel9_compile
+ group: "rhel9"
+ base_image: registry.access.redhat.com/ubi9/ubi
+ platforms: [linux/amd64]
+
+# OpenSuse
+ - name: opensuse-leap15_compile
+ group: "opensuse"
+ base_image: opensuse/leap:15
+ platforms: [linux/amd64]
+
+# Debian
+ - name: debian10_compile # debian 10
+ group: 'debian10'
+ base_image: 'debian:buster'
+ platforms: [linux/amd64]
+
+ - name: debian11_compile # debian 11
+ group: 'debian11'
+ base_image: 'debian:bullseye'
+ platforms: [linux/amd64]
+
+# Ubuntu
+ - name: ubuntu20.04_compile
+ group: 'ubuntu20.04'
+ base_image: 'ubuntu:20.04'
+ platforms: [linux/amd64]
+ extra_tags:
+ - $R/ubuntu20.04_libabigail:latest
+ - $R/ubuntu20.04_libabigail:$T
+ - $R/ubuntu20.04_spell-check:latest
+ - $R/ubuntu20.04_spell-check:$T
+ - $R/ubuntu20.04_sonarscanner:latest
+ - $R/ubuntu20.04_sonarscanner:$T
+
+ # x86 -> ARM cross compile
+ - name: ubuntu20.04_cross
+ group: 'ubuntu_cross'
+ base_image: 'ubuntu:20.04'
+ platforms: [linux/amd64]
+ cross_file: "config/arm/arm64_armv8_linux_gcc"
+
+ - name: ubuntu20.04_sve_compile
+ group: 'ubuntu_sve'
+ base_image: 'arm64v8/ubuntu:20.04'
+ platforms: [linux/arm64]
+
+ - name: ubuntu22.04_compile
+ group: 'ubuntu22.04'
+ base_image: 'ubuntu:22.04'
+ platforms: [linux/amd64]
\ No newline at end of file
diff --git a/containers/template_engine/inventory_schema.json b/containers/template_engine/inventory_schema.json
new file mode 100644
index 0000000..0e641b7
--- /dev/null
+++ b/containers/template_engine/inventory_schema.json
@@ -0,0 +1,153 @@
+{
+ "$schema": "https://json-schema.org/draft-07/schema",
+ "title": "dpdklab-ci container inventory schema",
+ "$comment": "SPDX-License-Identifier: BSD-3-Clause\nCopyright (c) 2022 University of New Hampshire",
+ "definitions": {},
+ "type": "object",
+ "properties": {
+ "abi": {
+ "type": "object",
+ "properties": {
+ "versions": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "major": {
+ "type": "integer",
+ "description": "The major version number (usually the year of release)"
+ },
+ "minor": {
+ "type": "integer",
+ "description": "The minor version number (usually the month of release)"
+ },
+ "git_tag": {
+ "type": "string",
+ "description": "The git tag to check out to get this ABI version"
+ }
+ },
+ "required": [
+ "major",
+ "minor",
+ "git_tag"
+ ],
+ "additionalProperties": false
+ },
+ "minItems": 0
+ }
+ },
+ "required": [
+ "versions"
+ ],
+ "additionalProperties": false
+ },
+ "dockerfiles": {
+ "type": "object",
+ "properties": {
+ "groups": {
+ "type": "object",
+ "patternProperties": {
+ "^[a-z][a-z_0-9\\.]+$": {
+ "type": "object",
+ "properties": {
+ "package_manager_command": {
+ "type": "string",
+ "description": "The command to install packages without user input."
+ },
+ "parent": {
+ "type": "string",
+ "description": "Which group to inherit from, or NONE if no parent exists"
+ },
+ "packages": {
+ "type": "array",
+ "description": "A list of package names to install",
+ "items": {
+ "type": "string"
+ },
+ "uniqueItems": true
+ }
+ },
+ "required": [
+ "parent",
+ "packages"
+ ],
+ "additionalProperties": false
+ }
+ },
+ "minProperties": 1
+ },
+ "targets": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "The name of the docker container"
+ },
+ "group": {
+ "type": "string",
+ "description": "Which group the target is a member of"
+ },
+ "base_image": {
+ "type": "string",
+ "description": "The base docker image."
+ },
+ "platforms": {
+ "type": "array",
+ "description": "What platforms to build for",
+ "items": {
+ "type": "string",
+ "enum": [
+ "linux/amd64",
+ "linux/arm64",
+ "linux/riscv64",
+ "linux/ppc64le",
+ "linux/arm/v7",
+ "linux/arm/v6"
+ ]
+ },
+ "minItems": 1
+ },
+ "cross_file": {
+ "type": "string",
+ "description": "A cross file to compile meson with. If not present, meson's automatic behavior will be used."
+ },
+ "extra_tags": {
+ "type": "array",
+ "description": "Tags to apply to this image",
+ "items": {
+ "type": "string",
+ "description": "$R will be replaced with the 'registry_hostname' variable in this file. $T will be replaced with a timestamp in the form of '%Y-%m-%d'. $N will be replaced by the name of the target.. The tags '$R/$N:latest' and '$R/$N:$T' will always be present"
+ }
+ },
+ "extra_information": {
+ "type": "object",
+ "description": "Extra information to be stored as a JSON file at /container_info.json",
+ "additionalProperties": true
+ }
+ },
+ "required": [
+ "name",
+ "group",
+ "base_image",
+ "platforms"
+ ],
+ "additionalProperties": false
+ },
+ "minItems": 1
+ }
+ },
+ "required": [
+ "groups",
+ "targets"
+ ],
+ "additionalProperties": false
+ }
+ },
+ "required": [
+ "abi",
+ "dockerfiles"
+ ],
+ "additionalProperties": false
+}
\ No newline at end of file
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 03/11] containers/builder: Dockerfile creation script
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
2023-05-23 17:04 ` [PATCH v5 01/11] containers/docs: Add container builder start Adam Hassick
2023-05-23 17:04 ` [PATCH v5 02/11] containers/inventory: Add inventory for container builder Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-23 17:04 ` [PATCH v5 04/11] containers/templates: Templates for Dockerfiles Adam Hassick
` (7 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Owen Hilyard, aconole, alialnu
From: Owen Hilyard <ohilyard@iol.unh.edu>
This script will template out all of the Dockerfiles based on the
definitions provided in the inventory using the jinja2 templating
library.
Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
containers/template_engine/make_dockerfile.py | 240 ++++++++++++++++++
1 file changed, 240 insertions(+)
create mode 100755 containers/template_engine/make_dockerfile.py
diff --git a/containers/template_engine/make_dockerfile.py b/containers/template_engine/make_dockerfile.py
new file mode 100755
index 0000000..9a3c19b
--- /dev/null
+++ b/containers/template_engine/make_dockerfile.py
@@ -0,0 +1,240 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2022 University of New Hampshire
+import argparse
+import json
+import logging
+import os
+from dataclasses import dataclass
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+import jsonschema
+import yaml
+from jinja2 import Environment, FileSystemLoader, select_autoescape
+
+
+@dataclass(frozen=True)
+class Options:
+ on_rhel: bool
+ fail_on_unbuildable: bool
+ build_libabigail: bool
+ build_abi: bool
+ output_dir: str
+ registry_hostname: str
+
+
+def _get_arg_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(description="Makes the dockerfile")
+ parser.add_argument("--output-dir", required=True)
+ parser.add_argument(
+ "--rhel",
+ action="store_true",
+ help="Overwrite the check for running on RHEL",
+ default=False,
+ )
+ parser.add_argument(
+ "--fail-on-unbuildable",
+ action="store_true",
+ help="If any container would not be possible to build, fail and exit with a non-zero exit code.",
+ default=False,
+ )
+ parser.add_argument(
+ "--build-abi",
+ action="store_true",
+ help="Whether to build the ABI references into the image. Disabled by \
+ default due to producing 10+ GB images. \
+ Implies '--build-libabigail'.",
+ )
+ parser.add_argument(
+ "--build-libabigail",
+ action="store_true",
+ help="Whether to build libabigail from source for distros that do not \
+ package it. Implied by '--build-abi'",
+ )
+ return parser
+
+
+def parse_args() -> Options:
+ parser = _get_arg_parser()
+ args = parser.parse_args()
+
+ registry_hostname = (
+ os.environ.get("DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME") or "localhost"
+ )
+
+ # In order to to build the ABIs, libabigail must be built from source on
+ # some platforms
+ build_libabigail: bool = args.build_libabigail or args.build_abi
+
+ opts = Options(
+ on_rhel=args.rhel,
+ fail_on_unbuildable=args.fail_on_unbuildable,
+ build_libabigail=build_libabigail,
+ build_abi=args.build_abi,
+ output_dir=args.output_dir,
+ registry_hostname=registry_hostname,
+ )
+ logging.info(f"make_dockerfile.py options: {opts}")
+ return opts
+
+
+def running_on_RHEL(options: Options) -> bool:
+ """
+ RHEL containers can only be built on RHEL, so disable them and emit a
+ warning if not on RHEL.
+ """
+ redhat_release_path = "/etc/redhat-release"
+
+ if os.path.exists(redhat_release_path):
+ with open(redhat_release_path) as f:
+ first_line = f.readline()
+ on_rhel = "Red Hat Enterprise Linux" in first_line
+ if on_rhel:
+ logging.info("Running on RHEL, allowing RHEL containers")
+ return True
+
+ logging.warning("Not on RHEL, disabling RHEL containers")
+ assert options is not None, "Internal state error, OPTIONS should not be None"
+
+ if options.on_rhel:
+ logging.info("Override enabled, enabling RHEL containers")
+
+ return options.on_rhel
+
+
+def get_path_to_parent_directory() -> str:
+ return os.path.dirname(__file__)
+
+
+def get_raw_inventory():
+ parent_dir = get_path_to_parent_directory()
+
+ schema_path = os.path.join(parent_dir, "inventory_schema.json")
+ inventory_path = os.path.join(parent_dir, "inventory.yaml")
+
+ inventory: Dict[str, Any]
+ with open(inventory_path, "r") as f:
+ inventory = yaml.safe_load(f)
+
+ schema: Dict[str, Any]
+ with open(schema_path, "r") as f:
+ schema = json.load(f)
+
+ jsonschema.validate(instance=inventory, schema=schema)
+ return inventory
+
+
+def apply_group_config_to_target(
+ target: Dict[str, Any],
+ raw_inventory: Dict[str, Any],
+ on_rhel: bool,
+ fail_on_unbuildable: bool,
+) -> Optional[Dict[str, Any]]:
+ groups_for_target: List[Dict[str, Any]] = []
+ groups: List[Dict[str, Any]] = raw_inventory["dockerfiles"]["groups"]
+ group = groups[target["group"]]
+
+ target_primary_group = target["group"]
+
+ assert isinstance(target_primary_group, str), "Target group name was not a string"
+
+ requires_rhel = "rhel" in target_primary_group.lower()
+
+ if requires_rhel and not on_rhel:
+ logging.warning(
+ f"Disabling target {target['name']}, because it must be built on RHEL."
+ )
+ if fail_on_unbuildable:
+ raise AssertionError(
+ f"Not on RHEL and target {target['name']} must be built on RHEL"
+ )
+
+ return None
+
+ while group["parent"] != "NONE":
+ groups_for_target.append(group)
+ group = groups[group["parent"]]
+
+ groups_for_target.append(group) # add the "all" group
+ groups_for_target.reverse() # reverse it so overrides work
+
+ target_packages: List[str] = target.get("packages") or []
+
+ for group in groups_for_target:
+ target_packages = [*target_packages, *(group.get("packages") or [])]
+ target = dict(target, **group)
+
+ target["packages"] = target_packages
+
+ return target
+
+
+def get_processed_inventory(options: Options) -> Dict[str, Any]:
+ raw_inventory: Dict[str, Any] = get_raw_inventory()
+ on_rhel = running_on_RHEL(options)
+ targets = raw_inventory["dockerfiles"]["targets"]
+ targets = [
+ apply_group_config_to_target(
+ target, raw_inventory, on_rhel, options.fail_on_unbuildable
+ )
+ for target in targets
+ ]
+ # remove disabled options
+ targets = [target for target in targets if target is not None]
+ raw_inventory["dockerfiles"]["targets"] = targets
+
+ return raw_inventory
+
+
+def main():
+ options: Options = parse_args()
+
+ env = Environment(
+ loader=FileSystemLoader("templates"),
+ )
+
+ inventory = get_processed_inventory(options)
+
+ timestamp = datetime.now().strftime("%Y-%m-%d")
+
+ for target in inventory["dockerfiles"]["targets"]:
+ template = env.get_template(f"containers/{target['group']}.dockerfile.j2")
+ dockerfile_location = os.path.join(
+ options.output_dir, target["name"] + ".dockerfile"
+ )
+
+ tags: list[str] = target.get("extra_tags") or []
+ tags.insert(0, "$R/$N:latest")
+ tags.insert(1, "$R/$N:$T")
+
+ target["tags"] = tags
+
+ rendered_dockerfile = template.render(
+ timestamp=timestamp,
+ target=target,
+ build_libabigail=options.build_libabigail,
+ build_abi=options.build_abi,
+ registry_hostname=options.registry_hostname,
+ **inventory,
+ )
+ with open(dockerfile_location, "w") as output_file:
+ output_file.write(rendered_dockerfile)
+
+ makefile_template = env.get_template(f"containers.makefile.j2")
+ rendered_makefile = makefile_template.render(
+ timestamp=timestamp,
+ build_libabigail=options.build_libabigail,
+ build_abi=options.build_abi,
+ registry_hostname=options.registry_hostname,
+ **inventory,
+ )
+ makefile_output_path = os.path.join(options.output_dir, "Makefile")
+ with open(makefile_output_path, "w") as f:
+ f.write(rendered_makefile)
+
+
+if __name__ == "__main__":
+ logging.basicConfig()
+ logging.root.setLevel(0) # log everything
+ main()
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 04/11] containers/templates: Templates for Dockerfiles
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
` (2 preceding siblings ...)
2023-05-23 17:04 ` [PATCH v5 03/11] containers/builder: Dockerfile creation script Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-23 17:04 ` [PATCH v5 05/11] containers/container_builder: Container for python scripts Adam Hassick
` (6 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Owen Hilyard, aconole, alialnu
From: Owen Hilyard <ohilyard@iol.unh.edu>
Adds a variety of extensible templates used to create the Dockerfiles
for each target. All templates inherit from base, and most distros and
distro families have their own templates that are used to define common
functionality. Multiple versions of a distro may use a single template
if they are similar enough (ex: fedora).
Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
.../templates/containers.makefile.j2 | 31 +++++++
.../templates/containers/alpine.dockerfile.j2 | 37 ++++++++
.../templates/containers/arch.dockerfile.j2 | 37 ++++++++
.../templates/containers/base.dockerfile.j2 | 85 +++++++++++++++++++
.../containers/centos8.dockerfile.j2 | 21 +++++
.../containers/centos9.dockerfile.j2 | 17 ++++
.../templates/containers/debian.dockerfile.j2 | 7 ++
.../containers/debian10.dockerfile.j2 | 3 +
.../containers/debian11.dockerfile.j2 | 3 +
.../containers/debian_bullseye.dockerfile.j2 | 3 +
.../containers/debian_buster.dockerfile.j2 | 3 +
.../templates/containers/fedora.dockerfile.j2 | 11 +++
.../containers/fedora36_clang.dockerfile.j2 | 7 ++
.../containers/opensuse.dockerfile.j2 | 10 +++
.../containers/redhat_family.dockerfile.j2 | 5 ++
.../templates/containers/rhel.dockerfile.j2 | 12 +++
.../templates/containers/rhel7.dockerfile.j2 | 19 +++++
.../templates/containers/rhel8.dockerfile.j2 | 19 +++++
.../templates/containers/rhel9.dockerfile.j2 | 25 ++++++
.../templates/containers/rpm.dockerfile.j2 | 3 +
.../templates/containers/ubuntu.dockerfile.j2 | 3 +
.../containers/ubuntu20.04.dockerfile.j2 | 12 +++
.../containers/ubuntu22.04.dockerfile.j2 | 7 ++
.../containers/ubuntu_cross.dockerfile.j2 | 11 +++
.../containers/ubuntu_sve.dockerfile.j2 | 10 +++
25 files changed, 401 insertions(+)
create mode 100644 containers/template_engine/templates/containers.makefile.j2
create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
diff --git a/containers/template_engine/templates/containers.makefile.j2 b/containers/template_engine/templates/containers.makefile.j2
new file mode 100644
index 0000000..fb71044
--- /dev/null
+++ b/containers/template_engine/templates/containers.makefile.j2
@@ -0,0 +1,31 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+#################### START VARS #################
+DPDK_CI_CONTAINER_BUILDER_PROGRAM?=podman
+DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?=
+DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?={{ registry_hostname }}
+
+#################### END VARS #################
+
+.PHONY: build_all
+build_all:{% for container in dockerfiles.targets %} build_{{ container.name }}{% endfor %}
+
+.PHONY: push_all
+push_all:{% for container in dockerfiles.targets %} push_{{ container.name }}{% endfor %}
+
+{% for container in dockerfiles.targets %}
+#################### START {{ container.name }} #####################
+ {% set container_id = container.name %}
+.PHONY: build_{{ container_id }}
+build_{{ container_id }}:
+ {% set base_tag = "%s/%s\:latest" | format(registry_hostname, container_id) -%}
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build --target {{container.name}} --platform {{ container.platforms | join(',') }} {% for tag in container.tags -%}-t {{ tag | replace('$R', registry_hostname) | replace('$T', timestamp) | replace('$N', container.name) | replace(':', '\\:') }} {% endfor %} -f {{ container_id }}.dockerfile $(CURDIR)
+
+.PHONY: push_{{ container_id }}
+push_{{ container_id }}: build_{{ container_id }}
+ {%- set base_tag = "%s/%s\:latest" | format(registry_hostname, container_id) -%}
+ {%- for tag in container.tags %}
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) push $(DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS) {{ base_tag }} docker\://{{ tag | replace('$R', registry_hostname) | replace('$T', timestamp) | replace('$N', container.name) | replace(':', '\\:') }}
+ {%- endfor %}
+#################### END {{ container.name }} #####################
+{% endfor %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/alpine.dockerfile.j2 b/containers/template_engine/templates/containers/alpine.dockerfile.j2
new file mode 100644
index 0000000..ab21bed
--- /dev/null
+++ b/containers/template_engine/templates/containers/alpine.dockerfile.j2
@@ -0,0 +1,37 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/base.dockerfile.j2' %}
+
+{% block pip_installs %}
+RUN python3 -m pip install ninja pyelftools
+{% endblock %}
+
+{% block install_libabigail %}
+{% if build_libabigail %}
+COPY libabigail /libabigail
+# libabigail2.0 does not build on alpine, this commit does with the modification done in the next step.
+RUN --mount=type=cache,target=/ccache cd libabigail && git checkout df28c220976f41620b0bd7b22000815e11f66b75 && sed -i 's/fts-standalone/libfts/g' ./configure.ac && mkdir build && autoreconf -i && cd build && ../configure --prefix=/usr/local && (make all install -j $(nproc) || echo 'failure expected') && (make install || echo 'failure expected') && cd / && rm -rf /libabigail
+{% else %}
+# libabigail build is disabled
+{% endif %}
+{% endblock install_libabigail %}
+
+{% block build_abi %}
+{% for abi_version in abi.versions %}
+ {% if abi_version.major >= 22 %}
+{% set abi_version_str = abi_version.git_tag %}
+RUN echo '{{ abi_version_str }}' >> abi_versions
+RUN mkdir {{ abi_version_str }}
+# compile DPDK
+RUN --mount=type=cache,target=/ccache \
+ mkdir -p /dpdk/build && rm -rf /dpdk/build && cd /dpdk && git fetch --tags && git checkout tags/{{ abi_version_str }} && cd /references && \
+ meson /dpdk /dpdk/build {% if "cross_file" in target -%}
+ --cross-file=/dpdk/{{ target.cross_file }}
+ {%- endif %} -Dexamples=all --buildtype=debugoptimized && DESTDIR={{ abi_folder }} ninja -C /dpdk/build install && \
+ bash /dpdk/devtools/gen-abi.sh {{ abi_folder }} && mv {{ abi_folder }}/usr/local/include {{ abi_folder }} && \
+ rm -rf {{ abi_folder }}/usr && rm -rf /dpdk/build
+ {% endif %}
+{% endfor %}
+RUN chmod -R 777 .
+RUN rm -rf /dpdk
+{% endblock build_abi %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/arch.dockerfile.j2 b/containers/template_engine/templates/containers/arch.dockerfile.j2
new file mode 100644
index 0000000..901607f
--- /dev/null
+++ b/containers/template_engine/templates/containers/arch.dockerfile.j2
@@ -0,0 +1,37 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/base.dockerfile.j2' %}
+
+{% block pre_package_manager %}
+RUN pacman-key --init
+RUN pacman -Sy --noconfirm archlinux-keyring && pacman -Su --noconfirm
+{% endblock %}
+
+
+{% block pip_installs %}
+RUN python3 -m pip install ninja pyelftools
+{% endblock %}
+
+{% block before_meson_after_pip %}
+# add user for building aur packages
+RUN useradd builder && \
+ mkdir /home/builder && \
+ chown builder:builder /home/builder
+
+# switch user to make packages from aur
+USER builder
+
+# clone and make rdma-core packages
+RUN cd /home/builder && git clone https://aur.archlinux.org/rdma-core.git
+RUN cd /home/builder/rdma-core && makepkg
+
+USER root
+
+RUN cd /home/builder/rdma-core && pacman --noconfirm -U rdma-core-*.pkg.tar.*
+
+RUN rm -rf /home/builder
+{% endblock %}
+
+{% block install_libabigail %}
+{# Installed via package manager #}
+{% endblock install_libabigail %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/base.dockerfile.j2 b/containers/template_engine/templates/containers/base.dockerfile.j2
new file mode 100644
index 0000000..5ffeeb1
--- /dev/null
+++ b/containers/template_engine/templates/containers/base.dockerfile.j2
@@ -0,0 +1,85 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+# ------------------------ Begin {{ target.name }} ------------------------------------------------
+
+FROM {{ target.base_image }} as {{ target.name }}
+
+ENV CCACHE_DIR=/ccache
+RUN mkdir /ccache
+{% block extra_env_vars -%}{%- endblock extra_env_vars %}
+
+{% block pre_package_manager -%}{%- endblock pre_package_manager %}
+
+# Install packages
+RUN {{ target.package_manager_command }} {{ target.packages | join(" ") }}
+
+# Installs from pip
+{% block pip_installs -%}{%- endblock pip_installs %}
+
+# Extra setup after python libs are installed
+{% block before_meson_after_pip -%}{%- endblock %}
+
+# install libabigail
+{% block install_libabigail %}
+{% if build_libabigail %}
+COPY libabigail /libabigail
+RUN --mount=type=cache,target=/ccache cd /libabigail && mkdir build && autoreconf -i && cd build && ../configure --prefix=/usr/local && make all install && \
+ ldconfig && cd / && rm -rf /libabigail
+{% else %}
+# libabigail build is disabled
+{% endif %}
+{% endblock install_libabigail %}
+
+# cross compilation setup
+{% block cross_compilation_setup %}
+{% endblock cross_compilation_setup %}
+
+# Copy externally pulled DPDK into the container
+RUN mkdir /dpdk
+
+COPY dpdk /dpdk
+
+# Install meson
+{% block install_meson %}
+# Install meson from dpdk main
+# Writes to the mount are discarded once the container is built and is not shared with other containers or the host
+RUN --mount=type=cache,target=/ccache cd /dpdk && chown -R root /dpdk && git checkout main && sh .ci/linux-setup.sh
+{% endblock install_meson %}
+
+ENV RTE_KERNELDIR CHANGE_ME
+
+RUN mkdir /references
+WORKDIR /references
+RUN touch abi_versions
+# build the abi
+{% if build_abi %}
+ {% block build_abi %}
+ {% for abi_version in abi.versions %}
+ {%- set abi_version_str = abi_version.git_tag -%}
+ {%- set abi_folder = "/references/" + abi_version_str + "_reference" %}
+RUN echo '{{ abi_version_str }}' >> abi_versions
+RUN mkdir -p {{ abi_folder }}
+# compile DPDK
+RUN --mount=type=cache,target=/ccache \
+ mkdir -p /dpdk/build && rm -rf /dpdk/build && cd /dpdk && git clean -xfd && git fetch --tags && git checkout tags/{{ abi_version_str }} && cd /references && \
+ meson /dpdk /dpdk/build {% if "cross_file" in target -%}
+ --cross-file=/dpdk/{{ target.cross_file }}
+ {%- endif %} -Dexamples=all --buildtype=debugoptimized && DESTDIR={{ abi_folder }} ninja -C /dpdk/build install && \
+ bash /dpdk/devtools/gen-abi.sh {{ abi_folder }} && mv {{ abi_folder }}/usr/local/include {{ abi_folder }} && \
+ rm -rf {{ abi_folder }}/usr && rm -rf /dpdk/build
+ {% endfor %}
+RUN rm -rf /dpdk
+RUN chmod -R 777 .
+ {% endblock build_abi %}
+{% else %}
+{% endif %}
+
+# Embed extra target information into the container
+{% if 'extra_information' in target %}
+RUN echo '{{ target.extra_information | tojson }}' > /container_info.json
+{% endif %}
+
+COPY ./scripts /scripts
+WORKDIR /
+# ------------------------ End {{ target.name }} ------------------------------------------------
+
diff --git a/containers/template_engine/templates/containers/centos8.dockerfile.j2 b/containers/template_engine/templates/containers/centos8.dockerfile.j2
new file mode 100644
index 0000000..bda024a
--- /dev/null
+++ b/containers/template_engine/templates/containers/centos8.dockerfile.j2
@@ -0,0 +1,21 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/redhat_family.dockerfile.j2' %}
+
+{% block pre_package_manager %}
+# For ninja-build, meson, libmnl-devel, nasm, and CUnit-devel
+RUN dnf install -y dnf-command\(config-manager\) dnf-plugins-core
+# Install EPEL repository for ccache
+RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
+# Enable powertools repository for libpcap-devel
+RUN dnf config-manager --set-enabled powertools
+{% endblock %}
+
+{% block pip_installs %}
+RUN python3 -m pip install --upgrade pip
+RUN python3 -m pip install pyelftools ninja
+{% endblock %}
+
+{% block install_libabigail %}
+{# Install via package manager #}
+{% endblock install_libabigail %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/centos9.dockerfile.j2 b/containers/template_engine/templates/containers/centos9.dockerfile.j2
new file mode 100644
index 0000000..8cacb99
--- /dev/null
+++ b/containers/template_engine/templates/containers/centos9.dockerfile.j2
@@ -0,0 +1,17 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/redhat_family.dockerfile.j2' %}
+
+{% block pre_package_manager %}
+# For ninja-build, meson, libmnl-devel, nasm, and CUnit-devel
+RUN dnf install -y dnf-command\(config-manager\) dnf-plugins-core
+# Install EPEL repository for ccache
+RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
+# Enable powertools repository for libpcap-devel
+RUN dnf config-manager --set-enabled crb
+{% endblock %}
+
+{% block pip_installs %}
+RUN python3 -m pip install --upgrade pip
+RUN python3 -m pip install pyelftools ninja
+{% endblock %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/debian.dockerfile.j2 b/containers/template_engine/templates/containers/debian.dockerfile.j2
new file mode 100644
index 0000000..74b56b6
--- /dev/null
+++ b/containers/template_engine/templates/containers/debian.dockerfile.j2
@@ -0,0 +1,7 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/base.dockerfile.j2' %}
+
+{% block pre_package_manager %}
+ENV DEBIAN_FRONTEND=noninteractive
+{% endblock %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/debian10.dockerfile.j2 b/containers/template_engine/templates/containers/debian10.dockerfile.j2
new file mode 100644
index 0000000..c52f866
--- /dev/null
+++ b/containers/template_engine/templates/containers/debian10.dockerfile.j2
@@ -0,0 +1,3 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/debian.dockerfile.j2' %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/debian11.dockerfile.j2 b/containers/template_engine/templates/containers/debian11.dockerfile.j2
new file mode 100644
index 0000000..0fd8a68
--- /dev/null
+++ b/containers/template_engine/templates/containers/debian11.dockerfile.j2
@@ -0,0 +1,3 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/debian.dockerfile.j2' %}
diff --git a/containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2 b/containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2
new file mode 100644
index 0000000..0fd8a68
--- /dev/null
+++ b/containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2
@@ -0,0 +1,3 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/debian.dockerfile.j2' %}
diff --git a/containers/template_engine/templates/containers/debian_buster.dockerfile.j2 b/containers/template_engine/templates/containers/debian_buster.dockerfile.j2
new file mode 100644
index 0000000..c52f866
--- /dev/null
+++ b/containers/template_engine/templates/containers/debian_buster.dockerfile.j2
@@ -0,0 +1,3 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/debian.dockerfile.j2' %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/fedora.dockerfile.j2 b/containers/template_engine/templates/containers/fedora.dockerfile.j2
new file mode 100644
index 0000000..97e438e
--- /dev/null
+++ b/containers/template_engine/templates/containers/fedora.dockerfile.j2
@@ -0,0 +1,11 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/redhat_family.dockerfile.j2' %}
+
+{%- block pip_installs -%}
+RUN python3 -m pip install ninja
+{%- endblock -%}
+
+{% block install_libabigail %}
+# installed by package manager
+{% endblock install_libabigail %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2 b/containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2
new file mode 100644
index 0000000..bc104ba
--- /dev/null
+++ b/containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2
@@ -0,0 +1,7 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/fedora.dockerfile.j2' %}
+
+{% block extra_env_vars -%}
+ENV CC=clang
+{%- endblock extra_env_vars %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/opensuse.dockerfile.j2 b/containers/template_engine/templates/containers/opensuse.dockerfile.j2
new file mode 100644
index 0000000..48d2502
--- /dev/null
+++ b/containers/template_engine/templates/containers/opensuse.dockerfile.j2
@@ -0,0 +1,10 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/rpm.dockerfile.j2' %}
+
+{% block pip_installs %}
+{% endblock %}
+
+{% block install_libabigail %}
+{# Install via package manager #}
+{% endblock install_libabigail %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/redhat_family.dockerfile.j2 b/containers/template_engine/templates/containers/redhat_family.dockerfile.j2
new file mode 100644
index 0000000..9ae6577
--- /dev/null
+++ b/containers/template_engine/templates/containers/redhat_family.dockerfile.j2
@@ -0,0 +1,5 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/rpm.dockerfile.j2' %}
+
+{# All of the distros in the Redhat Linux Lineage, ex: RHEL, Fedora, CentOS #}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/rhel.dockerfile.j2 b/containers/template_engine/templates/containers/rhel.dockerfile.j2
new file mode 100644
index 0000000..a350e84
--- /dev/null
+++ b/containers/template_engine/templates/containers/rhel.dockerfile.j2
@@ -0,0 +1,12 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/redhat_family.dockerfile.j2' %}
+
+{% block pip_installs %}
+RUN python3 -m pip install pyelftools
+{% endblock %}
+
+
+{% block before_meson_after_pip %}
+RUN yum --disableplugin=subscription-manager clean all
+{% endblock %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/rhel7.dockerfile.j2 b/containers/template_engine/templates/containers/rhel7.dockerfile.j2
new file mode 100644
index 0000000..ccfc0ec
--- /dev/null
+++ b/containers/template_engine/templates/containers/rhel7.dockerfile.j2
@@ -0,0 +1,19 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/rhel.dockerfile.j2' %}
+
+{% block pre_package_manager %}
+# repolist is needed to refresh the RHEL repos from the host this is being built on
+RUN yum repolist
+# Yes this next step is gross, there is no better way to enable the repos in the UBI containers.
+RUN sed -i '/\[rhel-7-server-extras-rpms\]/,/^ *\[/ s/enabled\ =\ 0/enabled\ =\ 1/' /etc/yum.repos.d/redhat.repo \
+ && sed -i '/\[rhel-7-server-optional-rpms\]/,/^ *\[/ s/enabled\ =\ 0/enabled\ =\ 1/' /etc/yum.repos.d/redhat.repo
+RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
+RUN yum --disableplugin=subscription-manager update -y
+
+RUN yum groupinstall -y 'Development Tools'
+{% endblock %}
+
+{% block install_libabigail %}
+{# Install via package manager #}
+{% endblock install_libabigail %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/rhel8.dockerfile.j2 b/containers/template_engine/templates/containers/rhel8.dockerfile.j2
new file mode 100644
index 0000000..dcaffc9
--- /dev/null
+++ b/containers/template_engine/templates/containers/rhel8.dockerfile.j2
@@ -0,0 +1,19 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/rhel.dockerfile.j2' %}
+
+{% block pre_package_manager %}
+# repolist is needed to refresh the RHEL repos from the host this is being built on
+RUN yum repolist
+RUN yum --disableplugin=subscription-manager update -y
+RUN yum --disableplugin=subscription-manager config-manager --set-enable codeready-builder-for-rhel-8-$(uname -i)-rpms
+
+# Required to install ccache
+RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm && yum upgrade -y
+
+RUN yum groupinstall -y 'Development Tools'
+{% endblock %}
+
+{% block install_libabigail %}
+{# Install via package manager #}
+{% endblock install_libabigail %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/rhel9.dockerfile.j2 b/containers/template_engine/templates/containers/rhel9.dockerfile.j2
new file mode 100644
index 0000000..8f0eccc
--- /dev/null
+++ b/containers/template_engine/templates/containers/rhel9.dockerfile.j2
@@ -0,0 +1,25 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/rhel.dockerfile.j2' %}
+
+{% block pre_package_manager %}
+# repolist is needed to refresh the RHEL repos from the host this is being built on
+RUN yum repolist
+RUN yum --disableplugin=subscription-manager config-manager --set-enable codeready-builder-for-rhel-9-$(uname -i)-rpms
+RUN yum --disableplugin=subscription-manager update -y
+
+# Required to install ccache
+RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && yum upgrade -y
+
+RUN yum groupinstall -y 'Development Tools'
+{% endblock %}
+
+{% block pip_installs %}
+RUN python3 -m pip install pyelftools ninja
+{% endblock %}
+
+{% block install_libabigail %}
+COPY libabigail libabigail
+RUN cd libabigail && mkdir build && autoreconf -i && cd build && ../configure --prefix=/usr/local && make all install && \
+ ldconfig && cd .. && rm -rf libabigail
+{% endblock install_libabigail %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/rpm.dockerfile.j2 b/containers/template_engine/templates/containers/rpm.dockerfile.j2
new file mode 100644
index 0000000..f386b42
--- /dev/null
+++ b/containers/template_engine/templates/containers/rpm.dockerfile.j2
@@ -0,0 +1,3 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/base.dockerfile.j2' %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/ubuntu.dockerfile.j2 b/containers/template_engine/templates/containers/ubuntu.dockerfile.j2
new file mode 100644
index 0000000..c52f866
--- /dev/null
+++ b/containers/template_engine/templates/containers/ubuntu.dockerfile.j2
@@ -0,0 +1,3 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/debian.dockerfile.j2' %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2 b/containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
new file mode 100644
index 0000000..59d2c28
--- /dev/null
+++ b/containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
@@ -0,0 +1,12 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/ubuntu.dockerfile.j2' %}
+
+{% block pip_installs -%}
+RUN python3 -m pip install --no-cache-dir requests
+{%- endblock pip_installs %}
+
+{% block before_meson_after_pip -%}
+ADD https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.6.2.2472-linux.zip /
+ADD https://sonarcloud.io/static/cpp/build-wrapper-linux-x86.zip /
+{%- endblock %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2 b/containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
new file mode 100644
index 0000000..7a8b971
--- /dev/null
+++ b/containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
@@ -0,0 +1,7 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/ubuntu.dockerfile.j2' %}
+
+{% block install_libabigail %}
+# installed by package manager
+{% endblock install_libabigail %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2 b/containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
new file mode 100644
index 0000000..236efbc
--- /dev/null
+++ b/containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
@@ -0,0 +1,11 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/ubuntu20.04.dockerfile.j2' %}
+
+{% block cross_compilation_setup %}
+# Link ccache to different compilers
+ # aarch64 -> ccache
+RUN ln -s $(which ccache) /usr/local/bin/aarch64-linux-gnu-gcc && \
+ # aarch32 -> ccache
+ ln -s $(which ccache) /usr/local/bin/arm-linux-gnueabihf-gcc
+{% endblock cross_compilation_setup %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2 b/containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
new file mode 100644
index 0000000..3c3e471
--- /dev/null
+++ b/containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
@@ -0,0 +1,10 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/ubuntu20.04.dockerfile.j2' %}
+{% block cross_compilation_setup %}
+RUN ln -s $(which ccache) /usr/local/bin/gcc-10
+{% endblock cross_compilation_setup %}
+
+{% block build_abi %}
+# ABI is disabled for this container
+{% endblock build_abi %}
\ No newline at end of file
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 05/11] containers/container_builder: Container for python scripts
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
` (3 preceding siblings ...)
2023-05-23 17:04 ` [PATCH v5 04/11] containers/templates: Templates for Dockerfiles Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-23 17:04 ` [PATCH v5 06/11] containers/Makefile: Makefile to automate builds Adam Hassick
` (5 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Owen Hilyard, aconole, alialnu
From: Owen Hilyard <ohilyard@iol.unh.edu>
Adds a container that can be used to run the python scripts to create
the Dockerfiles for the CI containers. This removes a large number of
python environment dependencies from the host requirements.
The lockfile for poetry has also been added to help ensure that the
container is always built in a known-working state.
Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
containers/container_builder.dockerfile | 26 +++
containers/template_engine/poetry.lock | 220 ++++++++++++++++++++++++
2 files changed, 246 insertions(+)
create mode 100644 containers/container_builder.dockerfile
create mode 100644 containers/template_engine/poetry.lock
diff --git a/containers/container_builder.dockerfile b/containers/container_builder.dockerfile
new file mode 100644
index 0000000..dd1c5eb
--- /dev/null
+++ b/containers/container_builder.dockerfile
@@ -0,0 +1,26 @@
+FROM ubuntu:22.04
+
+ENV DPDK_CI_CONTAINERS_COMPOSE_FILE_BUILDER=podman-compose
+ENV DPDK_CI_CONTAINER_BUILDER_PROGRAM=podman
+ENV DPDK_CI_CONTAINERS_PYTHON3_CMD=python3
+ENV DEBIAN_FRONTEND=noninteractive
+
+# If building on a RHEL host, 'yum repolist' will refresh the RHEL repos from the host this is being built on
+RUN apt-get update && apt-get install --no-install-recommends -y \
+ python3 python3-pip \
+ # Makefile deps
+ make git
+RUN pip3 install poetry
+
+RUN git config --global --add safe.directory /container_workspace/container_context/dpdk
+RUN git config --global --add safe.directory /container_workspace/container_context/libabigail
+
+# Globally install required python libraries
+COPY template_engine/pyproject.toml /tmp/pyproject.toml
+COPY template_engine/poetry.lock /tmp/poetry.lock
+WORKDIR /tmp/
+RUN poetry config virtualenvs.create false
+RUN poetry install --no-interaction --no-ansi
+
+RUN mkdir /container_workspace
+WORKDIR /container_workspace
diff --git a/containers/template_engine/poetry.lock b/containers/template_engine/poetry.lock
new file mode 100644
index 0000000..87ba829
--- /dev/null
+++ b/containers/template_engine/poetry.lock
@@ -0,0 +1,220 @@
+[[package]]
+name = "attrs"
+version = "22.1.0"
+description = "Classes Without Boilerplate"
+category = "main"
+optional = false
+python-versions = ">=3.5"
+
+[package.extras]
+dev = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "mypy (>=0.900,!=0.940)", "pytest-mypy-plugins", "zope.interface", "furo", "sphinx", "sphinx-notfound-page", "pre-commit", "cloudpickle"]
+docs = ["furo", "sphinx", "zope.interface", "sphinx-notfound-page"]
+tests = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "mypy (>=0.900,!=0.940)", "pytest-mypy-plugins", "zope.interface", "cloudpickle"]
+tests_no_zope = ["coverage[toml] (>=5.0.2)", "hypothesis", "pympler", "pytest (>=4.3.0)", "mypy (>=0.900,!=0.940)", "pytest-mypy-plugins", "cloudpickle"]
+
+[[package]]
+name = "importlib-resources"
+version = "5.9.0"
+description = "Read resources from Python packages"
+category = "main"
+optional = false
+python-versions = ">=3.7"
+
+[package.dependencies]
+zipp = {version = ">=3.1.0", markers = "python_version < \"3.10\""}
+
+[package.extras]
+docs = ["sphinx", "jaraco.packaging (>=9)", "rst.linker (>=1.9)", "jaraco.tidelift (>=1.4)"]
+testing = ["pytest (>=6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-cov", "pytest-enabler (>=1.3)", "pytest-black (>=0.3.7)", "pytest-mypy (>=0.9.1)"]
+
+[[package]]
+name = "jinja2"
+version = "3.1.2"
+description = "A very fast and expressive template engine."
+category = "main"
+optional = false
+python-versions = ">=3.7"
+
+[package.dependencies]
+MarkupSafe = ">=2.0"
+
+[package.extras]
+i18n = ["Babel (>=2.7)"]
+
+[[package]]
+name = "jsonschema"
+version = "4.10.0"
+description = "An implementation of JSON Schema validation for Python"
+category = "main"
+optional = false
+python-versions = ">=3.7"
+
+[package.dependencies]
+attrs = ">=17.4.0"
+importlib-resources = {version = ">=1.4.0", markers = "python_version < \"3.9\""}
+pkgutil-resolve-name = {version = ">=1.3.10", markers = "python_version < \"3.9\""}
+pyrsistent = ">=0.14.0,<0.17.0 || >0.17.0,<0.17.1 || >0.17.1,<0.17.2 || >0.17.2"
+
+[package.extras]
+format = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3987", "uri-template", "webcolors (>=1.11)"]
+format-nongpl = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3986-validator (>0.1.0)", "uri-template", "webcolors (>=1.11)"]
+
+[[package]]
+name = "markupsafe"
+version = "2.1.1"
+description = "Safely add untrusted strings to HTML/XML markup."
+category = "main"
+optional = false
+python-versions = ">=3.7"
+
+[[package]]
+name = "pkgutil-resolve-name"
+version = "1.3.10"
+description = "Resolve a name to an object."
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
+[[package]]
+name = "pyrsistent"
+version = "0.18.1"
+description = "Persistent/Functional/Immutable data structures"
+category = "main"
+optional = false
+python-versions = ">=3.7"
+
+[[package]]
+name = "pyyaml"
+version = "6.0"
+description = "YAML parser and emitter for Python"
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
+[[package]]
+name = "zipp"
+version = "3.8.1"
+description = "Backport of pathlib-compatible object wrapper for zip files"
+category = "main"
+optional = false
+python-versions = ">=3.7"
+
+[package.extras]
+docs = ["sphinx", "jaraco.packaging (>=9)", "rst.linker (>=1.9)", "jaraco.tidelift (>=1.4)"]
+testing = ["pytest (>=6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-cov", "pytest-enabler (>=1.3)", "jaraco.itertools", "func-timeout", "pytest-black (>=0.3.7)", "pytest-mypy (>=0.9.1)"]
+
+[metadata]
+lock-version = "1.1"
+python-versions = "^3.8"
+content-hash = "6afb29c80b2176acf4af522df8b53c4f5334b7fe66e8360e832a82b7d9fe830a"
+
+[metadata.files]
+attrs = []
+importlib-resources = []
+jinja2 = [
+ {file = "Jinja2-3.1.2-py3-none-any.whl", hash = "sha256:6088930bfe239f0e6710546ab9c19c9ef35e29792895fed6e6e31a023a182a61"},
+ {file = "Jinja2-3.1.2.tar.gz", hash = "sha256:31351a702a408a9e7595a8fc6150fc3f43bb6bf7e319770cbc0db9df9437e852"},
+]
+jsonschema = []
+markupsafe = [
+ {file = "MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:86b1f75c4e7c2ac2ccdaec2b9022845dbb81880ca318bb7a0a01fbf7813e3812"},
+ {file = "MarkupSafe-2.1.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:f121a1420d4e173a5d96e47e9a0c0dcff965afdf1626d28de1460815f7c4ee7a"},
+ {file = "MarkupSafe-2.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a49907dd8420c5685cfa064a1335b6754b74541bbb3706c259c02ed65b644b3e"},
+ {file = "MarkupSafe-2.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:10c1bfff05d95783da83491be968e8fe789263689c02724e0c691933c52994f5"},
+ {file = "MarkupSafe-2.1.1-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b7bd98b796e2b6553da7225aeb61f447f80a1ca64f41d83612e6139ca5213aa4"},
+ {file = "MarkupSafe-2.1.1-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:b09bf97215625a311f669476f44b8b318b075847b49316d3e28c08e41a7a573f"},
+ {file = "MarkupSafe-2.1.1-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:694deca8d702d5db21ec83983ce0bb4b26a578e71fbdbd4fdcd387daa90e4d5e"},
+ {file = "MarkupSafe-2.1.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:efc1913fd2ca4f334418481c7e595c00aad186563bbc1ec76067848c7ca0a933"},
+ {file = "MarkupSafe-2.1.1-cp310-cp310-win32.whl", hash = "sha256:4a33dea2b688b3190ee12bd7cfa29d39c9ed176bda40bfa11099a3ce5d3a7ac6"},
+ {file = "MarkupSafe-2.1.1-cp310-cp310-win_amd64.whl", hash = "sha256:dda30ba7e87fbbb7eab1ec9f58678558fd9a6b8b853530e176eabd064da81417"},
+ {file = "MarkupSafe-2.1.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:671cd1187ed5e62818414afe79ed29da836dde67166a9fac6d435873c44fdd02"},
+ {file = "MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3799351e2336dc91ea70b034983ee71cf2f9533cdff7c14c90ea126bfd95d65a"},
+ {file = "MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e72591e9ecd94d7feb70c1cbd7be7b3ebea3f548870aa91e2732960fa4d57a37"},
+ {file = "MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6fbf47b5d3728c6aea2abb0589b5d30459e369baa772e0f37a0320185e87c980"},
+ {file = "MarkupSafe-2.1.1-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:d5ee4f386140395a2c818d149221149c54849dfcfcb9f1debfe07a8b8bd63f9a"},
+ {file = "MarkupSafe-2.1.1-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:bcb3ed405ed3222f9904899563d6fc492ff75cce56cba05e32eff40e6acbeaa3"},
+ {file = "MarkupSafe-2.1.1-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:e1c0b87e09fa55a220f058d1d49d3fb8df88fbfab58558f1198e08c1e1de842a"},
+ {file = "MarkupSafe-2.1.1-cp37-cp37m-win32.whl", hash = "sha256:8dc1c72a69aa7e082593c4a203dcf94ddb74bb5c8a731e4e1eb68d031e8498ff"},
+ {file = "MarkupSafe-2.1.1-cp37-cp37m-win_amd64.whl", hash = "sha256:97a68e6ada378df82bc9f16b800ab77cbf4b2fada0081794318520138c088e4a"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:e8c843bbcda3a2f1e3c2ab25913c80a3c5376cd00c6e8c4a86a89a28c8dc5452"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0212a68688482dc52b2d45013df70d169f542b7394fc744c02a57374a4207003"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8e576a51ad59e4bfaac456023a78f6b5e6e7651dcd383bcc3e18d06f9b55d6d1"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4b9fe39a2ccc108a4accc2676e77da025ce383c108593d65cc909add5c3bd601"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:96e37a3dc86e80bf81758c152fe66dbf60ed5eca3d26305edf01892257049925"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:6d0072fea50feec76a4c418096652f2c3238eaa014b2f94aeb1d56a66b41403f"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:089cf3dbf0cd6c100f02945abeb18484bd1ee57a079aefd52cffd17fba910b88"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:6a074d34ee7a5ce3effbc526b7083ec9731bb3cbf921bbe1d3005d4d2bdb3a63"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-win32.whl", hash = "sha256:421be9fbf0ffe9ffd7a378aafebbf6f4602d564d34be190fc19a193232fd12b1"},
+ {file = "MarkupSafe-2.1.1-cp38-cp38-win_amd64.whl", hash = "sha256:fc7b548b17d238737688817ab67deebb30e8073c95749d55538ed473130ec0c7"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:e04e26803c9c3851c931eac40c695602c6295b8d432cbe78609649ad9bd2da8a"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:b87db4360013327109564f0e591bd2a3b318547bcef31b468a92ee504d07ae4f"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:99a2a507ed3ac881b975a2976d59f38c19386d128e7a9a18b7df6fff1fd4c1d6"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:56442863ed2b06d19c37f94d999035e15ee982988920e12a5b4ba29b62ad1f77"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3ce11ee3f23f79dbd06fb3d63e2f6af7b12db1d46932fe7bd8afa259a5996603"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:33b74d289bd2f5e527beadcaa3f401e0df0a89927c1559c8566c066fa4248ab7"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:43093fb83d8343aac0b1baa75516da6092f58f41200907ef92448ecab8825135"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:8e3dcf21f367459434c18e71b2a9532d96547aef8a871872a5bd69a715c15f96"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-win32.whl", hash = "sha256:d4306c36ca495956b6d568d276ac11fdd9c30a36f1b6eb928070dc5360b22e1c"},
+ {file = "MarkupSafe-2.1.1-cp39-cp39-win_amd64.whl", hash = "sha256:46d00d6cfecdde84d40e572d63735ef81423ad31184100411e6e3388d405e247"},
+ {file = "MarkupSafe-2.1.1.tar.gz", hash = "sha256:7f91197cc9e48f989d12e4e6fbc46495c446636dfc81b9ccf50bb0ec74b91d4b"},
+]
+pkgutil-resolve-name = []
+pyrsistent = [
+ {file = "pyrsistent-0.18.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:df46c854f490f81210870e509818b729db4488e1f30f2a1ce1698b2295a878d1"},
+ {file = "pyrsistent-0.18.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5d45866ececf4a5fff8742c25722da6d4c9e180daa7b405dc0a2a2790d668c26"},
+ {file = "pyrsistent-0.18.1-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:4ed6784ceac462a7d6fcb7e9b663e93b9a6fb373b7f43594f9ff68875788e01e"},
+ {file = "pyrsistent-0.18.1-cp310-cp310-win32.whl", hash = "sha256:e4f3149fd5eb9b285d6bfb54d2e5173f6a116fe19172686797c056672689daf6"},
+ {file = "pyrsistent-0.18.1-cp310-cp310-win_amd64.whl", hash = "sha256:636ce2dc235046ccd3d8c56a7ad54e99d5c1cd0ef07d9ae847306c91d11b5fec"},
+ {file = "pyrsistent-0.18.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:e92a52c166426efbe0d1ec1332ee9119b6d32fc1f0bbfd55d5c1088070e7fc1b"},
+ {file = "pyrsistent-0.18.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d7a096646eab884bf8bed965bad63ea327e0d0c38989fc83c5ea7b8a87037bfc"},
+ {file = "pyrsistent-0.18.1-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:cdfd2c361b8a8e5d9499b9082b501c452ade8bbf42aef97ea04854f4a3f43b22"},
+ {file = "pyrsistent-0.18.1-cp37-cp37m-win32.whl", hash = "sha256:7ec335fc998faa4febe75cc5268a9eac0478b3f681602c1f27befaf2a1abe1d8"},
+ {file = "pyrsistent-0.18.1-cp37-cp37m-win_amd64.whl", hash = "sha256:6455fc599df93d1f60e1c5c4fe471499f08d190d57eca040c0ea182301321286"},
+ {file = "pyrsistent-0.18.1-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:fd8da6d0124efa2f67d86fa70c851022f87c98e205f0594e1fae044e7119a5a6"},
+ {file = "pyrsistent-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7bfe2388663fd18bd8ce7db2c91c7400bf3e1a9e8bd7d63bf7e77d39051b85ec"},
+ {file = "pyrsistent-0.18.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0e3e1fcc45199df76053026a51cc59ab2ea3fc7c094c6627e93b7b44cdae2c8c"},
+ {file = "pyrsistent-0.18.1-cp38-cp38-win32.whl", hash = "sha256:b568f35ad53a7b07ed9b1b2bae09eb15cdd671a5ba5d2c66caee40dbf91c68ca"},
+ {file = "pyrsistent-0.18.1-cp38-cp38-win_amd64.whl", hash = "sha256:d1b96547410f76078eaf66d282ddca2e4baae8964364abb4f4dcdde855cd123a"},
+ {file = "pyrsistent-0.18.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:f87cc2863ef33c709e237d4b5f4502a62a00fab450c9e020892e8e2ede5847f5"},
+ {file = "pyrsistent-0.18.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6bc66318fb7ee012071b2792024564973ecc80e9522842eb4e17743604b5e045"},
+ {file = "pyrsistent-0.18.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:914474c9f1d93080338ace89cb2acee74f4f666fb0424896fcfb8d86058bf17c"},
+ {file = "pyrsistent-0.18.1-cp39-cp39-win32.whl", hash = "sha256:1b34eedd6812bf4d33814fca1b66005805d3640ce53140ab8bbb1e2651b0d9bc"},
+ {file = "pyrsistent-0.18.1-cp39-cp39-win_amd64.whl", hash = "sha256:e24a828f57e0c337c8d8bb9f6b12f09dfdf0273da25fda9e314f0b684b415a07"},
+ {file = "pyrsistent-0.18.1.tar.gz", hash = "sha256:d4d61f8b993a7255ba714df3aca52700f8125289f84f704cf80916517c46eb96"},
+]
+pyyaml = [
+ {file = "PyYAML-6.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d4db7c7aef085872ef65a8fd7d6d09a14ae91f691dec3e87ee5ee0539d516f53"},
+ {file = "PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9df7ed3b3d2e0ecfe09e14741b857df43adb5a3ddadc919a2d94fbdf78fea53c"},
+ {file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77f396e6ef4c73fdc33a9157446466f1cff553d979bd00ecb64385760c6babdc"},
+ {file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a80a78046a72361de73f8f395f1f1e49f956c6be882eed58505a15f3e430962b"},
+ {file = "PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5"},
+ {file = "PyYAML-6.0-cp310-cp310-win32.whl", hash = "sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513"},
+ {file = "PyYAML-6.0-cp310-cp310-win_amd64.whl", hash = "sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a"},
+ {file = "PyYAML-6.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86"},
+ {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9758427f"},
+ {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92"},
+ {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:98c4d36e99714e55cfbaaee6dd5badbc9a1ec339ebfc3b1f52e293aee6bb71a4"},
+ {file = "PyYAML-6.0-cp36-cp36m-win32.whl", hash = "sha256:0283c35a6a9fbf047493e3a0ce8d79ef5030852c51e9d911a27badfde0605293"},
+ {file = "PyYAML-6.0-cp36-cp36m-win_amd64.whl", hash = "sha256:07751360502caac1c067a8132d150cf3d61339af5691fe9e87803040dbc5db57"},
+ {file = "PyYAML-6.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:819b3830a1543db06c4d4b865e70ded25be52a2e0631ccd2f6a47a2822f2fd7c"},
+ {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:473f9edb243cb1935ab5a084eb238d842fb8f404ed2193a915d1784b5a6b5fc0"},
+ {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0ce82d761c532fe4ec3f87fc45688bdd3a4c1dc5e0b4a19814b9009a29baefd4"},
+ {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:231710d57adfd809ef5d34183b8ed1eeae3f76459c18fb4a0b373ad56bedcdd9"},
+ {file = "PyYAML-6.0-cp37-cp37m-win32.whl", hash = "sha256:c5687b8d43cf58545ade1fe3e055f70eac7a5a1a0bf42824308d868289a95737"},
+ {file = "PyYAML-6.0-cp37-cp37m-win_amd64.whl", hash = "sha256:d15a181d1ecd0d4270dc32edb46f7cb7733c7c508857278d3d378d14d606db2d"},
+ {file = "PyYAML-6.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0b4624f379dab24d3725ffde76559cff63d9ec94e1736b556dacdfebe5ab6d4b"},
+ {file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:213c60cd50106436cc818accf5baa1aba61c0189ff610f64f4a3e8c6726218ba"},
+ {file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9fa600030013c4de8165339db93d182b9431076eb98eb40ee068700c9c813e34"},
+ {file = "PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:277a0ef2981ca40581a47093e9e2d13b3f1fbbeffae064c1d21bfceba2030287"},
+ {file = "PyYAML-6.0-cp38-cp38-win32.whl", hash = "sha256:d4eccecf9adf6fbcc6861a38015c2a64f38b9d94838ac1810a9023a0609e1b78"},
+ {file = "PyYAML-6.0-cp38-cp38-win_amd64.whl", hash = "sha256:1e4747bc279b4f613a09eb64bba2ba602d8a6664c6ce6396a4d0cd413a50ce07"},
+ {file = "PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:055d937d65826939cb044fc8c9b08889e8c743fdc6a32b33e2390f66013e449b"},
+ {file = "PyYAML-6.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e61ceaab6f49fb8bdfaa0f92c4b57bcfbea54c09277b1b4f7ac376bfb7a7c174"},
+ {file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d67d839ede4ed1b28a4e8909735fc992a923cdb84e618544973d7dfc71540803"},
+ {file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cba8c411ef271aa037d7357a2bc8f9ee8b58b9965831d9e51baf703280dc73d3"},
+ {file = "PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:40527857252b61eacd1d9af500c3337ba8deb8fc298940291486c465c8b46ec0"},
+ {file = "PyYAML-6.0-cp39-cp39-win32.whl", hash = "sha256:b5b9eccad747aabaaffbc6064800670f0c297e52c12754eb1d976c57e4f74dcb"},
+ {file = "PyYAML-6.0-cp39-cp39-win_amd64.whl", hash = "sha256:b3d267842bf12586ba6c734f89d1f5b871df0273157918b0ccefa29deb05c21c"},
+ {file = "PyYAML-6.0.tar.gz", hash = "sha256:68fb519c14306fec9720a2a5b45bc9f0c8d1b9c72adf45c37baedfcd949c35a2"},
+]
+zipp = []
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 06/11] containers/Makefile: Makefile to automate builds
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
` (4 preceding siblings ...)
2023-05-23 17:04 ` [PATCH v5 05/11] containers/container_builder: Container for python scripts Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-23 17:04 ` [PATCH v5 07/11] containers/inventory: Add ABI rev and coverity attribute Adam Hassick
` (4 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Owen Hilyard, aconole, alialnu
From: Owen Hilyard <ohilyard@iol.unh.edu>
The Makefile that can be used to build all of the container images using
"make build", and can also be used to push them to a remote repository
(for use in CI).
Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
containers/Makefile | 166 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 166 insertions(+)
create mode 100644 containers/Makefile
diff --git a/containers/Makefile b/containers/Makefile
new file mode 100644
index 0000000..3d56162
--- /dev/null
+++ b/containers/Makefile
@@ -0,0 +1,166 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2022 University of New Hampshire
+
+# Recommended Resources:
+# All:
+# ~100 GB of disk space: These are not minimal containers, they
+# have full distros in them minus the kernel.
+# With ABI images, expect 10 GB per distro per
+# platform. Without, it's closer to 2 GB per distro
+# per platform.
+#
+# No ABI:
+# Serial Build (make -j 1): Should run on any post 2010 system reasonably well.
+# Parallel build (make -j $(nproc)): 1 GB of free ram per job should be good enough.
+# ABI:
+# Time: If you are building more than four distros, or you have one distro that is built under
+# emulation (x86 host + arm container or the inverse), it may be best to run it overnight.
+# Some versions of qemu + podman will try to compile with a single thread under emulation,
+# which can cause what should be an hour-long compile to turn into a 8+ hour compile.
+# Serial Build (make -j 1): 6 GB of free ram should be fine
+# Parallel build (make -j $(nproc)): 6 GB of free ram per job. You may be compiling ~20 versions of DPDK at the same time, with some under emulation.
+#
+
+
+############ Arguments #############
+# Set to 'Y' to override distro detection
+DPDK_CI_CONTAINERS_ON_RHEL?=$(shell (test -f /etc/redhat-release && grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') || echo 'N')
+
+# If set to 'Y' and any container is detected as unbuildable, fail the build.
+DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=N
+
+# If set to 'Y', build the ABIs for specificed DPDK versions and embed them
+# in the container
+DPDK_CI_CONTAINERS_BUILD_ABI?=N
+
+# If set to 'Y', build libabigail from source for distros that do not have it.
+# If DPDK_CI_CONTAINERS_BUILD_ABI='Y', then this is always enabled.
+ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y)
+DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=Y
+else
+DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL?=N
+endif
+
+# Used to set the python interpreter
+DPDK_CI_CONTAINERS_PYTHON3_CMD?=python3.8
+
+# The base program to use to build individual containers.
+DPDK_CI_CONTAINER_BUILDER_PROGRAM?=podman
+
+# The url of the git repository to pull libabigail from
+DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=git://sourceware.org/git/libabigail.git
+
+# The url of the git repository to pull DPDK from
+DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=https://dpdk.org/git/dpdk
+
+# The tag to apply to the built container builder image
+DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder
+
+# Extra arguments to add to the push command, can be used for credentials
+DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?=
+
+DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=localhost
+
+# The path to a directory to be recursively copied to $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY). Not used if unset.
+# DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH
+
+DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context
+############ End Arguments #############
+
+############ Internal Variables #############
+SCRIPT_ARGS=--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)"
+
+ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y)
+ SCRIPT_ARGS +=--rhel
+endif
+
+ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y)
+ SCRIPT_ARGS +=--fail-on-unbuildable
+endif
+
+ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y)
+ SCRIPT_ARGS +=--build-abi
+endif
+
+ifeq ($(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL), Y)
+ SCRIPT_ARGS +=--build-libabigail
+endif
+
+SCRIPT_DIRECTORY=$(CURDIR)/template_engine
+
+LIBABIGAIL_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabigail
+DPDK_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk
+
+DOCKERFILE=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile
+
+MAKE_DOCKERFILE_SCRIPT=$(SCRIPT_DIRECTORY)/make_dockerfile.py
+MAKE_DOCKERFILE_COMMAND=$(DPDK_CI_CONTAINERS_PYTHON3_CMD) $(MAKE_DOCKERFILE_SCRIPT)
+
+TEMPLATE_FILE_DIRECTORY=$(SCRIPT_DIRECTORY)/templates
+DOCKER_CONTAINER_TEMPLATE_DIR=$(TEMPLATE_FILE_DIRECTORY)/containers
+
+INVENTORY_FILE=$(SCRIPT_DIRECTORY)/inventory.yaml
+
+EXTRA_SCRIPTS_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scripts
+
+GIT_FETCH_ARGS=--all --tags
+
+CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile
+
+export
+############ End Internal Variables #############
+
+.PHONY: build push make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk clean_container_files clean
+
+build: make_docker_files_in_container extra_scripts
+ $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) build_all
+
+push: make_docker_files_in_container extra_scripts
+ $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_all
+
+make_docker_files_in_container: build_builder_container $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run -v $(CURDIR):/container_workspace -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context \
+ -e DPDK_CI_CONTAINERS_ON_RHEL=$(DPDK_CI_CONTAINERS_ON_RHEL) \
+ -e DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=$(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL) \
+ -e DPDK_CI_CONTAINERS_BUILD_ABI=$(DPDK_CI_CONTAINERS_BUILD_ABI) \
+ -e DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE=$(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE) \
+ -e DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME='$(DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME)' \
+ $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) make docker_deps
+
+build_builder_container:
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f $(CONTAINER_BUILDER_DOCKERFILE) -t $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) $(CURDIR)
+
+docker_deps: $(DOCKERFILE) deps extra_scripts
+ chmod 666 $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/*.dockerfile
+
+$(DOCKERFILE): $(INVENTORY_FILE) $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) deps $(MAKE_DOCKERFILE_SCRIPT) $(shell find $(DOCKER_CONTAINER_TEMPLATE_DIR) -type f)
+ cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFILE_COMMAND) $(SCRIPT_ARGS)
+
+deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk libabigail
+
+extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(EXTRA_SCRIPTS_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)
+ifdef $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)
+ cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) $(EXTRA_SCRIPTS_DIRECTORY)
+endif
+ mkdir -p $(EXTRA_SCRIPTS_DIRECTORY)
+
+# Clone libabigail source
+libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
+ git -C $(LIBABIGAIL_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL) $(LIBABIGAIL_DIRECTORY)
+ git -C $(LIBABIGAIL_DIRECTORY) describe --tags `git -C $(LIBABIGAIL_DIRECTORY) rev-list --tags --max-count=1` | xargs -n 1 git -C $(LIBABIGAIL_DIRECTORY) checkout
+
+# Clone DPDK source
+dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
+ git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY)
+
+$(EXTRA_SCRIPTS_DIRECTORY):
+ mkdir -p $(EXTRA_SCRIPTS_DIRECTORY)
+
+$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):
+ mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
+
+clean_container_files:
+ rm $(DOCKERFILE)
+
+clean:
+ rm -rf $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 07/11] containers/inventory: Add ABI rev and coverity attribute
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
` (5 preceding siblings ...)
2023-05-23 17:04 ` [PATCH v5 06/11] containers/Makefile: Makefile to automate builds Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-23 17:04 ` [PATCH v5 08/11] containers/builder: Add arguments to templating script Adam Hassick
` (3 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Adam Hassick, aconole, alialnu
Adds a new attribute that allows the version of libabigail to
be checked out to a specific revision. This was needed because
libabigail 2.3.0 does not build on Debian 10 or 11 at the time
of writing.
Adds another ABI "version" to the inventory to represent the
XML comparison reference.
Adds another new attribute to specify if a container requires
Coverity Scan binaries.
Updates the versions of Fedora in the inventory.
Adds a new target for building the Coverity Scan container.
Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
---
containers/template_engine/inventory.yaml | 240 ++++++++++++------
.../template_engine/inventory_schema.json | 100 ++++++--
2 files changed, 247 insertions(+), 93 deletions(-)
diff --git a/containers/template_engine/inventory.yaml b/containers/template_engine/inventory.yaml
index eb299c0..2eb371b 100644
--- a/containers/template_engine/inventory.yaml
+++ b/containers/template_engine/inventory.yaml
@@ -4,15 +4,38 @@
abi:
# versions should also include the abi versions for all stable ABIs.
versions:
- - major: 20 # oldest supported lts
- minor: 11
- git_tag: v20.11
- - major: 21 # most recent lts
- minor: 11
- git_tag: v21.11
- # - major: 22 # current abi
- # minor: 07
- # git_tag: v22.07
+ # - dpdk_version: # LTS - 3
+ # major: 20
+ # minor: 11
+ # revision: 0
+ # git_tag: "20.11"
+ # abi_version:
+ # major: 21
+ # minor: 0
+ - dpdk_version: # LTS - 2
+ major: 21
+ minor: 11
+ revision: 0
+ git_tag: "21.11"
+ abi_version:
+ major: 22
+ minor: 0
+ - dpdk_version:
+ major: 22 # LTS - 1
+ minor: 11
+ revision: 1
+ git_tag: "tags/v22.11.1"
+ abi_version:
+ major: 23
+ minor: 10 # .xml comparison
+ - dpdk_version:
+ major: 22 # LTS
+ minor: 11
+ revision: 1
+ git_tag: "tags/v22.11.1"
+ abi_version:
+ major: 23
+ minor: 11 # .so comparison
dockerfiles:
groups:
@@ -44,6 +67,7 @@ dockerfiles:
- ccache
- zip
- autoconf
+ - libbpf-devel
redhat_family:
parent: "rpm"
packages: []
@@ -64,6 +88,13 @@ dockerfiles:
- elfutils-devel
- libxml2-devel
- libabigail
+ - openssl-devel
+ fedora_coverity:
+ parent: fedora
+ packages: []
+ fedora_clang:
+ parent: fedora
+ packages: []
centos8:
parent: "redhat_family"
packages:
@@ -73,6 +104,7 @@ dockerfiles:
- libpcap-devel
- numactl-devel
- libabigail
+ - openssl-devel
centos9:
parent: "redhat_family"
package_manager_command: "yum install --allowerasing -y"
@@ -86,30 +118,31 @@ dockerfiles:
- g++
- elfutils-devel
- libxml2-devel
+ - openssl-devel
rhel:
parent: "redhat_family"
- package_manager_command: "yum --disableplugin=subscription-manager install -y"
+ package_manager_command: "yum install -y"
packages:
- elfutils-libelf-devel
- jansson-devel
- libatomic
- libpcap-devel
- numactl-devel
+ - openssl-devel
+ - libabigail
rhel7:
parent: "rhel"
packages:
- - libabigail
- ninja-build
rhel8:
parent: "rhel"
- package_manager_command: "yum --allowerasing --disableplugin=subscription-manager install -y"
+ package_manager_command: "yum --allowerasing install -y"
packages:
- - libabigail
- ninja-build
- libxml2-devel
rhel9:
parent: "rhel"
- package_manager_command: "yum --allowerasing --disableplugin=subscription-manager install -y"
+ package_manager_command: "yum --allowerasing install -y"
packages:
- libxml2-devel
opensuse:
@@ -126,34 +159,34 @@ dockerfiles:
- libnuma-devel
- libabigail-tools
- tar
+ - openssl-devel
alpine:
- parent: 'all'
- package_manager_command: 'apk add'
+ parent: "all"
+ package_manager_command: "apk add"
packages:
- - python3
- - py3-pip
- - alpine-sdk
+ - bsd-compat-headers
- ccache
- - cmake
- - bash
- - autoconf
- - automake
- - libtool
- - elfutils-dev
- - libxml2-dev
- - fts-dev
- - fts
- - numactl-dev
+ - jansson-dev
+ - libarchive-dev
+ - libbpf-dev
- libc-dev
- - bsd-compat-headers
- - libexecinfo-dev
+ - libpcap-dev
- linux-headers
- - libatomic
+ - ninja
+ - numactl-dev
+ - openssl-dev
+ - py3-pip
+ - py3-elftools
+ - py3-setuptools
+ - py3-wheel
+ - zlib-dev
+ - python3
+ - bash
- zip
- - pkgconfig
+ - meson
arch:
- parent: 'all'
- package_manager_command: 'pacman -Syu --noconfirm'
+ parent: "all"
+ package_manager_command: "pacman -Syu --noconfirm"
packages:
- glibc
- cmake
@@ -171,9 +204,11 @@ dockerfiles:
- diffutils
- python-pip
- libabigail
+ - openssl
+ - libbpf
debian:
- parent: 'all'
- package_manager_command: 'apt-get update && apt-get install --no-install-recommends -y'
+ parent: "all"
+ package_manager_command: "apt-get update && apt-get install --no-install-recommends -y"
packages:
- libnuma-dev
- zip
@@ -203,20 +238,26 @@ dockerfiles:
- parallel
- python3-grpcio
- lcov
+ - libbpf-dev
+ - libssl-dev
debian10:
- parent: 'debian'
+ parent: "debian"
packages: []
debian11:
- parent: 'debian'
+ parent: "debian"
+ packages:
+ - nasm
+ debian11_arm_ipsec:
+ parent: "debian11"
packages: []
ubuntu:
parent: 'debian'
packages: []
ubuntu18.04:
- parent: 'ubuntu'
+ parent: "ubuntu"
packages: []
ubuntu20.04:
- parent: 'ubuntu'
+ parent: "ubuntu"
packages:
- python3-grpcio
- lcov
@@ -224,14 +265,14 @@ dockerfiles:
- default-jre
- nodejs
ubuntu22.04:
- parent: 'ubuntu'
+ parent: "ubuntu"
packages:
- python3-grpcio
- lcov
- abigail-tools
- jq
ubuntu_cross: # x86 -> ARM
- parent: 'ubuntu20.04'
+ parent: "ubuntu20.04"
packages:
- clang
- llvm
@@ -248,26 +289,30 @@ dockerfiles:
- libc6-dev-armhf-cross
- pkg-config-arm-linux-gnueabihf
ubuntu_sve:
- parent: 'ubuntu20.04'
+ parent: "ubuntu20.04"
packages:
- gcc-10
targets:
-
-# Alpine
+ # Alpine
- name: alpine_compile
- group: 'alpine'
+ group: "alpine"
base_image: alpine
- platforms: [linux/amd64]
+ platforms: [linux/amd64, linux/arm64]
+ force_disable_abi: true
+ minimum_dpdk_version:
+ major: 22
+ minor: 0
+ revision: 0
extra_information:
environment_id: 45
-# Arch
+ # Arch
- name: arch_compile
- group: 'arch'
+ group: "arch"
base_image: archlinux:base
platforms: [linux/amd64]
-# Fedora
+ # Fedora
- name: fedora35_compile
group: "fedora"
base_image: fedora:35
@@ -275,13 +320,36 @@ dockerfiles:
- name: fedora36_compile
group: "fedora"
base_image: fedora:36
- platforms: [linux/amd64]
- - name: fedora36_clang
+ platforms: [linux/amd64, linux/arm64]
+ - name: fedora37_compile
group: "fedora"
- base_image: fedora:36
+ base_image: fedora:37
+ platforms: [linux/amd64, linux/arm64]
+ - name: fedora38_compile
+ group: "fedora"
+ base_image: fedora:38
+ platforms: [linux/amd64, linux/arm64]
+
+ - name: fedora38_clang
+ group: "fedora_clang"
+ base_image: fedora:38
+ platforms: [linux/amd64, linux/arm64]
+
+ - name: fedora37_coverity
+ group: "fedora_coverity"
+ base_image: fedora:37
+ force_disable_abi: true
+ requires_coverity: true
platforms: [linux/amd64]
-# CentOs
+ - name: fedora37_coverity
+ group: "fedora_coverity"
+ base_image: fedora:37
+ force_disable_abi: true
+ requires_coverity: true
+ platforms: [linux/amd64]
+
+ # CentOs
- name: centosstream8_compile
group: "centos8"
base_image: quay.io/centos/centos:stream8
@@ -290,9 +358,9 @@ dockerfiles:
- name: centosstream9_compile
group: "centos9"
base_image: quay.io/centos/centos:stream9
- platforms: [linux/amd64]
+ platforms: [linux/amd64, linux/arm64]
-# RHEL
+ # RHEL
- name: rhel7_compile
group: "rhel7"
base_image: registry.access.redhat.com/ubi7/ubi
@@ -306,30 +374,34 @@ dockerfiles:
- name: rhel9_compile
group: "rhel9"
base_image: registry.access.redhat.com/ubi9/ubi
- platforms: [linux/amd64]
+ platforms: [linux/amd64, linux/arm64]
-# OpenSuse
+ # OpenSuse
- name: opensuse-leap15_compile
group: "opensuse"
base_image: opensuse/leap:15
- platforms: [linux/amd64]
+ platforms: [linux/amd64, linux/arm64]
-# Debian
+ # Debian
- name: debian10_compile # debian 10
- group: 'debian10'
- base_image: 'debian:buster'
+ group: "debian10"
+ base_image: "debian:buster"
+ libabigail_revision: libabigail-2.2
platforms: [linux/amd64]
- name: debian11_compile # debian 11
- group: 'debian11'
- base_image: 'debian:bullseye'
- platforms: [linux/amd64]
+ group: "debian11"
+ base_image: "debian:bullseye"
+ libabigail_revision: libabigail-2.2
+ platforms:
+ - linux/amd64
+ - linux/arm64
-# Ubuntu
+ # Ubuntu
- name: ubuntu20.04_compile
- group: 'ubuntu20.04'
- base_image: 'ubuntu:20.04'
- platforms: [linux/amd64]
+ group: "ubuntu20.04"
+ base_image: "ubuntu:20.04"
+ platforms: [linux/amd64, linux/arm64]
extra_tags:
- $R/ubuntu20.04_libabigail:latest
- $R/ubuntu20.04_libabigail:$T
@@ -340,17 +412,31 @@ dockerfiles:
# x86 -> ARM cross compile
- name: ubuntu20.04_cross
- group: 'ubuntu_cross'
- base_image: 'ubuntu:20.04'
+ group: "ubuntu_cross"
+ base_image: "ubuntu:20.04"
platforms: [linux/amd64]
cross_file: "config/arm/arm64_armv8_linux_gcc"
+ # Also, x86 -> ARM cross compile (for SVE test)
- name: ubuntu20.04_sve_compile
- group: 'ubuntu_sve'
- base_image: 'arm64v8/ubuntu:20.04'
+ group: "ubuntu_sve"
+ base_image: "arm64v8/ubuntu:20.04"
platforms: [linux/arm64]
+ # TEMPORARY: This has the forked IPsec lib developed by ARM. This causes issues on older LTS versions of DPDK.
+ # Once all supported and tested LTS versions no longer break when this lib is installed, recombine with debian11_compile.
+ - name: debian11_arm_ipsec
+ group: "debian11_arm_ipsec"
+ base_image: "debian:bullseye"
+ platforms: [linux/arm64]
+ force_disable_abi: true
+ extra_tags:
+ - $R/debian11_zuc_compile:latest
+ - $R/debian11_snow3g_compile:latest
+ - $R/debian11_zuc_compile:$T
+ - $R/debian11_snow3g_compile:$T
+
- name: ubuntu22.04_compile
- group: 'ubuntu22.04'
- base_image: 'ubuntu:22.04'
- platforms: [linux/amd64]
\ No newline at end of file
+ group: "ubuntu22.04"
+ base_image: "ubuntu:22.04"
+ platforms: [linux/amd64]
diff --git a/containers/template_engine/inventory_schema.json b/containers/template_engine/inventory_schema.json
index 0e641b7..7b79e44 100644
--- a/containers/template_engine/inventory_schema.json
+++ b/containers/template_engine/inventory_schema.json
@@ -2,7 +2,6 @@
"$schema": "https://json-schema.org/draft-07/schema",
"title": "dpdklab-ci container inventory schema",
"$comment": "SPDX-License-Identifier: BSD-3-Clause\nCopyright (c) 2022 University of New Hampshire",
- "definitions": {},
"type": "object",
"properties": {
"abi": {
@@ -13,25 +12,57 @@
"items": {
"type": "object",
"properties": {
- "major": {
- "type": "integer",
- "description": "The major version number (usually the year of release)"
- },
- "minor": {
- "type": "integer",
- "description": "The minor version number (usually the month of release)"
+ "dpdk_version": {
+ "type": "object",
+ "properties": {
+ "major": {
+ "type": "integer",
+ "description": "The major version number (usually the year of release)"
+ },
+ "minor": {
+ "type": "integer",
+ "description": "The minor version number (usually the month of release)"
+ },
+ "revision": {
+ "type": "integer",
+ "description": "The revision version number. Starts at 0."
+ },
+ "git_tag": {
+ "type": "string",
+ "description": "The git tag to check out to get this ABI version"
+ }
+ },
+ "required": [
+ "major",
+ "minor",
+ "revision",
+ "git_tag"
+ ],
+ "additionalProperties": false
},
- "git_tag": {
- "type": "string",
- "description": "The git tag to check out to get this ABI version"
+ "abi_version": {
+ "type": "object",
+ "properties": {
+ "major": {
+ "type": "integer",
+ "description": "The major version number (usually the year of release)"
+ },
+ "minor": {
+ "type": "integer",
+ "description": "The minor version number (usually the month of release)"
+ }
+ },
+ "required": [
+ "major",
+ "minor"
+ ],
+ "additionalProperties": false
}
},
"required": [
- "major",
- "minor",
- "git_tag"
- ],
- "additionalProperties": false
+ "dpdk_version",
+ "abi_version"
+ ]
},
"minItems": 0
}
@@ -113,6 +144,43 @@
"type": "string",
"description": "A cross file to compile meson with. If not present, meson's automatic behavior will be used."
},
+ "libabigail_revision": {
+ "type": "string",
+ "description": "A revision or branch of libabigail to check out before building for a given target."
+ },
+ "force_disable_abi": {
+ "type": "boolean",
+ "description": "Whether to forcibly disable ABI for this target. This should be used as a temporary measure until a fix can be created.",
+ "default": "false"
+ },
+ "requires_coverity": {
+ "type": "boolean",
+ "description": "Whether this target requires the Coverity Scan binaries.",
+ "default": "false"
+ },
+ "minimum_dpdk_version": {
+ "type": "object",
+ "properties": {
+ "major": {
+ "type": "integer",
+ "description": "The major version number (usually the year of release)"
+ },
+ "minor": {
+ "type": "integer",
+ "description": "The minor version number (usually the month of release)"
+ },
+ "revision": {
+ "type": "integer",
+ "description": "The revision version number. Starts at 0."
+ }
+ },
+ "required": [
+ "major",
+ "minor",
+ "revision"
+ ],
+ "additionalProperties": false
+ },
"extra_tags": {
"type": "array",
"description": "Tags to apply to this image",
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 08/11] containers/builder: Add arguments to templating script
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
` (6 preceding siblings ...)
2023-05-23 17:04 ` [PATCH v5 07/11] containers/inventory: Add ABI rev and coverity attribute Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-23 17:04 ` [PATCH v5 09/11] containers/templates: Update templates Adam Hassick
` (2 subsequent siblings)
10 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Adam Hassick, aconole, alialnu
Adds arguments to the script to support builder mode,
overriding the date tag, expecting Coverity Scan, and
limiting the scope of target architectures.
Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
---
containers/template_engine/make_dockerfile.py | 134 ++++++++++++++++--
1 file changed, 126 insertions(+), 8 deletions(-)
diff --git a/containers/template_engine/make_dockerfile.py b/containers/template_engine/make_dockerfile.py
index 9a3c19b..60da2a8 100755
--- a/containers/template_engine/make_dockerfile.py
+++ b/containers/template_engine/make_dockerfile.py
@@ -5,8 +5,10 @@ import argparse
import json
import logging
import os
+import re
from dataclasses import dataclass
from datetime import datetime
+import platform
from typing import Any, Dict, List, Optional
import jsonschema
@@ -18,10 +20,16 @@ from jinja2 import Environment, FileSystemLoader, select_autoescape
class Options:
on_rhel: bool
fail_on_unbuildable: bool
+ has_coverity: bool
build_libabigail: bool
build_abi: bool
output_dir: str
registry_hostname: str
+ host_arch_only: bool
+ omit_latest: bool
+ is_builder: bool
+ date_override: Optional[str]
+ ninja_workers: Optional[int]
def _get_arg_parser() -> argparse.ArgumentParser:
@@ -52,6 +60,39 @@ def _get_arg_parser() -> argparse.ArgumentParser:
help="Whether to build libabigail from source for distros that do not \
package it. Implied by '--build-abi'",
)
+ parser.add_argument(
+ "--host-arch-only",
+ action="store_true",
+ help="Only build containers for the architecture of the host system",
+ )
+ parser.add_argument(
+ "--omit-latest",
+ action="store_true",
+ help="Whether to include the \"latest\" tag in the generated makefile."
+ )
+ parser.add_argument(
+ "--builder-mode",
+ action="store_true",
+ help="Specifies that the makefile is being templated for a builder. \
+ This implicitly sets \"--host-arch-only\" to true and disables making the manifests.",
+ default=False
+ )
+ parser.add_argument(
+ "--date",
+ type=str,
+ help="Overrides generation of the timestamp and uses the provided string instead."
+ )
+ parser.add_argument(
+ "--ninja-workers",
+ type=int,
+ help="Specifies a number of ninja workers to limit builds to. Uses the ninja default when not given."
+ )
+ parser.add_argument(
+ "--coverity",
+ action="store_true",
+ help="Whether the Coverity Scan binaries are available for building the Coverity containers.",
+ default=False
+ )
return parser
@@ -74,7 +115,14 @@ def parse_args() -> Options:
build_abi=args.build_abi,
output_dir=args.output_dir,
registry_hostname=registry_hostname,
+ host_arch_only=args.host_arch_only or args.builder_mode,
+ omit_latest=args.omit_latest,
+ is_builder=args.builder_mode,
+ date_override=args.date,
+ ninja_workers=args.ninja_workers,
+ has_coverity=args.coverity
)
+
logging.info(f"make_dockerfile.py options: {opts}")
return opts
@@ -169,14 +217,71 @@ def apply_group_config_to_target(
return target
+def apply_defaults_to_target(target: Dict[str, Any]) -> Dict[str, Any]:
+ def default_if_unset(target: Dict[str, Any], key: str, value: Any) -> Dict[str, Any]:
+ if key not in target:
+ target[key] = value
-def get_processed_inventory(options: Options) -> Dict[str, Any]:
+ return target
+
+ target = default_if_unset(target, "requires_coverity", False)
+ target = default_if_unset(target, "force_disable_abi", False)
+ target = default_if_unset(target, "minimum_dpdk_version", dict(major=0, minor=0, revision=0))
+ target = default_if_unset(target, "extra_information", {})
+
+ return target
+
+def get_host_arch() -> str:
+ machine: str = platform.machine()
+ match machine:
+ case "aarch64" | "armv8b" | "armv8l":
+ return "linux/arm64"
+ case "ppc64le":
+ return "linux/ppc64le"
+ case "x86_64" | "x64" | "amd64":
+ return "linux/amd64"
+ case arch:
+ raise ValueError(f"Unknown arch {arch}")
+
+def process_target(
+ target: Dict[str, Any],
+ raw_inventory: Dict[str, Any],
+ has_coverity: bool,
+ on_rhel: bool,
+ fail_on_unbuildable: bool,
+ host_arch_only: bool,
+ build_timestamp: str
+) -> Optional[Dict[str, Any]]:
+ target = apply_defaults_to_target(target)
+ # Copy the platforms, for building the manifest list.
+
+ # Write the build timestamp.
+ target["extra_information"].update({
+ "build_timestamp": build_timestamp
+ })
+
+ if (not has_coverity) and target["requires_coverity"]:
+ print(f"Disabling {target['name']}. Target requires Coverity, and it is not enabled.")
+ return None
+
+ if host_arch_only:
+ host_arch = get_host_arch()
+ if host_arch in target["platforms"]:
+ target["platforms"] = [host_arch]
+ else:
+ return None
+
+ return apply_group_config_to_target(
+ target, raw_inventory, on_rhel, fail_on_unbuildable
+ )
+
+def get_processed_inventory(options: Options, build_timestamp: str) -> Dict[str, Any]:
raw_inventory: Dict[str, Any] = get_raw_inventory()
on_rhel = running_on_RHEL(options)
targets = raw_inventory["dockerfiles"]["targets"]
targets = [
- apply_group_config_to_target(
- target, raw_inventory, on_rhel, options.fail_on_unbuildable
+ process_target(
+ target, raw_inventory, options.has_coverity, on_rhel, options.fail_on_unbuildable, options.host_arch_only, build_timestamp
)
for target in targets
]
@@ -194,9 +299,14 @@ def main():
loader=FileSystemLoader("templates"),
)
- inventory = get_processed_inventory(options)
+ build_timestamp = datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
- timestamp = datetime.now().strftime("%Y-%m-%d")
+ inventory = get_processed_inventory(options, build_timestamp)
+
+ if options.date_override:
+ timestamp = options.date_override
+ else:
+ timestamp = datetime.now().strftime("%Y-%m-%d")
for target in inventory["dockerfiles"]["targets"]:
template = env.get_template(f"containers/{target['group']}.dockerfile.j2")
@@ -205,9 +315,13 @@ def main():
)
tags: list[str] = target.get("extra_tags") or []
- tags.insert(0, "$R/$N:latest")
- tags.insert(1, "$R/$N:$T")
-
+
+ tags.insert(0, "$R/$N:$T")
+ if not options.omit_latest:
+ tags.insert(0, "$R/$N:latest")
+ else:
+ tags = list(filter(lambda x: re.match('^.*:latest$', x) is None, tags))
+
target["tags"] = tags
rendered_dockerfile = template.render(
@@ -215,7 +329,9 @@ def main():
target=target,
build_libabigail=options.build_libabigail,
build_abi=options.build_abi,
+ build_timestamp=build_timestamp,
registry_hostname=options.registry_hostname,
+ ninja_workers=options.ninja_workers,
**inventory,
)
with open(dockerfile_location, "w") as output_file:
@@ -226,7 +342,9 @@ def main():
timestamp=timestamp,
build_libabigail=options.build_libabigail,
build_abi=options.build_abi,
+ host_arch_only=options.host_arch_only,
registry_hostname=options.registry_hostname,
+ is_builder=options.is_builder,
**inventory,
)
makefile_output_path = os.path.join(options.output_dir, "Makefile")
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 09/11] containers/templates: Update templates
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
` (7 preceding siblings ...)
2023-05-23 17:04 ` [PATCH v5 08/11] containers/builder: Add arguments to templating script Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-23 17:04 ` [PATCH v5 10/11] containers/Makefile: Add new variables and target changes Adam Hassick
2023-05-23 17:04 ` [PATCH v5 11/11] containers/docs: Update README Adam Hassick
10 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Adam Hassick, aconole, alialnu
* Add OCI manifest support
* Add Docker multi-stage support
* Add Coverity Scan container template
* Add ARM ipsec container for cryptodev testing
Various template updates for added features and use of new arguments.
A check was added to distinguish between which ABI version is to be
generated as a binary reference or an XML reference.
Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
---
.../templates/containers.makefile.j2 | 62 ++++++++++++---
.../templates/containers/alpine.dockerfile.j2 | 33 --------
.../templates/containers/base.dockerfile.j2 | 78 ++++++++++++++-----
.../containers/debian11.dockerfile.j2 | 2 +-
.../debian11_arm_ipsec.dockerfile.j2 | 16 ++++
.../containers/fedora_clang.dockerfile.j2 | 7 ++
.../containers/fedora_coverity.dockerfile.j2 | 10 +++
.../templates/containers/rhel.dockerfile.j2 | 8 +-
.../templates/containers/rhel7.dockerfile.j2 | 8 +-
.../templates/containers/rhel8.dockerfile.j2 | 10 +--
.../templates/containers/rhel9.dockerfile.j2 | 12 +--
.../containers/ubuntu22.04.dockerfile.j2 | 6 +-
.../containers/ubuntu_cross.dockerfile.j2 | 3 +-
.../containers/ubuntu_sve.dockerfile.j2 | 2 +
14 files changed, 162 insertions(+), 95 deletions(-)
create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2
diff --git a/containers/template_engine/templates/containers.makefile.j2 b/containers/template_engine/templates/containers.makefile.j2
index fb71044..900bfc8 100644
--- a/containers/template_engine/templates/containers.makefile.j2
+++ b/containers/template_engine/templates/containers.makefile.j2
@@ -10,22 +10,64 @@ DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?={{ registry_hostname }}
.PHONY: build_all
build_all:{% for container in dockerfiles.targets %} build_{{ container.name }}{% endfor %}
-.PHONY: push_all
-push_all:{% for container in dockerfiles.targets %} push_{{ container.name }}{% endfor %}
+.PHONY: push_image_all
+push_image_all:{% for container in dockerfiles.targets %} push_image_{{ container.name }}{% endfor %}
+
+.PHONY: push_manifest_all
+push_manifest_all:
+{%- if is_builder %}
+ echo "Disabled: Builder mode was enabled."
+{%- else -%}
+ {% for container in dockerfiles.targets %} push_manifest_{{ container.name }}{% endfor %}
+{%- endif %}
+
+{%- macro push_indx(container, is_docker) %}
+ {%- for tag in container.tags -%}
+ {%- set formatted_tag = tag | replace('$R', registry_hostname) | replace('$T', timestamp) | replace('$N', container.name) | replace(':', '\\:') -%}
+ {%- set trunk_tag = formatted_tag.split('/')[1] -%}
+ {%- if is_docker -%}
+ {%- set ref_tag = formatted_tag -%}
+ {%- set push_xargs = '' -%}
+ {%- else -%}
+ {%- set ref_tag = trunk_tag -%}
+ {%- set push_xargs = ' docker\://%s' | format(formatted_tag) -%}
+ {%- endif -%}
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) manifest rm {{ ref_tag }} || echo "Image {{ ref_tag }} did not exist initially."
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) manifest create {{ ref_tag }}
+ {%- for platform in container.platforms -%}
+ {%- set plat_tag = "%s/%s\:image-%s-%s" | format(registry_hostname, container.name, platform.replace('/', ''), timestamp) %} \
+ {{ plat_tag -}} {% endfor %}
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) manifest push $(DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS) {{ ref_tag }} {{- push_xargs }}
+ {% endfor -%}
+{% endmacro %}
{% for container in dockerfiles.targets %}
#################### START {{ container.name }} #####################
{% set container_id = container.name %}
.PHONY: build_{{ container_id }}
+
build_{{ container_id }}:
- {% set base_tag = "%s/%s\:latest" | format(registry_hostname, container_id) -%}
- $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build --target {{container.name}} --platform {{ container.platforms | join(',') }} {% for tag in container.tags -%}-t {{ tag | replace('$R', registry_hostname) | replace('$T', timestamp) | replace('$N', container.name) | replace(':', '\\:') }} {% endfor %} -f {{ container_id }}.dockerfile $(CURDIR)
-
-.PHONY: push_{{ container_id }}
-push_{{ container_id }}: build_{{ container_id }}
- {%- set base_tag = "%s/%s\:latest" | format(registry_hostname, container_id) -%}
- {%- for tag in container.tags %}
- $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) push $(DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS) {{ base_tag }} docker\://{{ tag | replace('$R', registry_hostname) | replace('$T', timestamp) | replace('$N', container.name) | replace(':', '\\:') }}
+ {%- for platform in container.platforms -%}
+ {%- set plat_tag = "%s/%s\:image-%s-%s" | format(registry_hostname, container_id, platform.replace('/', ''), timestamp) %}
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build --target {{container.name}} --platform {{ platform }} \
+ -t {{ plat_tag }} -f {{ container_id }}.dockerfile $(CURDIR) {%- endfor %}
+
+.PHONY: push_image_{{ container_id }}
+push_image_{{ container_id }}: build_{{ container_id }}
+ {%- for platform in container.platforms -%}
+ {%- set plat_tag = "%s/%s\:image-%s-%s" | format(registry_hostname, container_id, platform.replace('/', ''), timestamp) %}
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) push $(DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS) {{ plat_tag }}
{%- endfor %}
+{% if not is_builder %}
+.PHONY: push_manifest_{{ container_id }}
+push_manifest_{{ container_id }}:
+{%- if host_arch_only and not is_builder %} push_image_{{ container_id }}{% endif %}
+ifeq ($(DPDK_CI_CONTAINER_BUILDER_PROGRAM), docker)
+ {{ push_indx(container, true) }}
+else
+ {{ push_indx(container, false) }}
+endif
+{% endif %}
+
#################### END {{ container.name }} #####################
{% endfor %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/alpine.dockerfile.j2 b/containers/template_engine/templates/containers/alpine.dockerfile.j2
index ab21bed..b258d64 100644
--- a/containers/template_engine/templates/containers/alpine.dockerfile.j2
+++ b/containers/template_engine/templates/containers/alpine.dockerfile.j2
@@ -2,36 +2,3 @@
{#- Copyright (c) 2022 University of New Hampshire -#}
{% extends 'containers/base.dockerfile.j2' %}
-{% block pip_installs %}
-RUN python3 -m pip install ninja pyelftools
-{% endblock %}
-
-{% block install_libabigail %}
-{% if build_libabigail %}
-COPY libabigail /libabigail
-# libabigail2.0 does not build on alpine, this commit does with the modification done in the next step.
-RUN --mount=type=cache,target=/ccache cd libabigail && git checkout df28c220976f41620b0bd7b22000815e11f66b75 && sed -i 's/fts-standalone/libfts/g' ./configure.ac && mkdir build && autoreconf -i && cd build && ../configure --prefix=/usr/local && (make all install -j $(nproc) || echo 'failure expected') && (make install || echo 'failure expected') && cd / && rm -rf /libabigail
-{% else %}
-# libabigail build is disabled
-{% endif %}
-{% endblock install_libabigail %}
-
-{% block build_abi %}
-{% for abi_version in abi.versions %}
- {% if abi_version.major >= 22 %}
-{% set abi_version_str = abi_version.git_tag %}
-RUN echo '{{ abi_version_str }}' >> abi_versions
-RUN mkdir {{ abi_version_str }}
-# compile DPDK
-RUN --mount=type=cache,target=/ccache \
- mkdir -p /dpdk/build && rm -rf /dpdk/build && cd /dpdk && git fetch --tags && git checkout tags/{{ abi_version_str }} && cd /references && \
- meson /dpdk /dpdk/build {% if "cross_file" in target -%}
- --cross-file=/dpdk/{{ target.cross_file }}
- {%- endif %} -Dexamples=all --buildtype=debugoptimized && DESTDIR={{ abi_folder }} ninja -C /dpdk/build install && \
- bash /dpdk/devtools/gen-abi.sh {{ abi_folder }} && mv {{ abi_folder }}/usr/local/include {{ abi_folder }} && \
- rm -rf {{ abi_folder }}/usr && rm -rf /dpdk/build
- {% endif %}
-{% endfor %}
-RUN chmod -R 777 .
-RUN rm -rf /dpdk
-{% endblock build_abi %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/base.dockerfile.j2 b/containers/template_engine/templates/containers/base.dockerfile.j2
index 5ffeeb1..576ddb2 100644
--- a/containers/template_engine/templates/containers/base.dockerfile.j2
+++ b/containers/template_engine/templates/containers/base.dockerfile.j2
@@ -2,10 +2,18 @@
{#- Copyright (c) 2022 University of New Hampshire -#}
# ------------------------ Begin {{ target.name }} ------------------------------------------------
-FROM {{ target.base_image }} as {{ target.name }}
+{%- set build_base_img = target.name + "_base" %}
+FROM {{ target.base_image }} as {% if build_abi and not target.force_disable_abi -%}
+{{ build_base_img }}
+{%- else -%}
+{{ target.name }}
+{%- endif %}
+
+ENV CCACHE_DIR=/ccache/
+ENV CCACHE_COMPILERCHECK=content
+ENV CCACHE_NAMESPACE="{{ target.name }}-{{ build_timestamp }}"
+ENV BUILD_TIMESTAMP="{{ build_timestamp }}"
-ENV CCACHE_DIR=/ccache
-RUN mkdir /ccache
{% block extra_env_vars -%}{%- endblock extra_env_vars %}
{% block pre_package_manager -%}{%- endblock pre_package_manager %}
@@ -21,9 +29,10 @@ RUN {{ target.package_manager_command }} {{ target.packages | join(" ") }}
# install libabigail
{% block install_libabigail %}
-{% if build_libabigail %}
+{% if build_libabigail and build_abi and (not "force_disable_abi" in target or not target.force_disable_abi) %}
COPY libabigail /libabigail
-RUN --mount=type=cache,target=/ccache cd /libabigail && mkdir build && autoreconf -i && cd build && ../configure --prefix=/usr/local && make all install && \
+RUN --mount=type=cache,target=/ccache,z cd /libabigail
+{%- if "libabigail_revision" in target %} && git checkout {{ target.libabigail_revision }} {%- endif %} && mkdir build && autoreconf -i && cd build && ../configure --prefix=/usr/local && make all install && \
ldconfig && cd / && rm -rf /libabigail
{% else %}
# libabigail build is disabled
@@ -36,14 +45,15 @@ RUN --mount=type=cache,target=/ccache cd /libabigail && mkdir build && autorecon
# Copy externally pulled DPDK into the container
RUN mkdir /dpdk
-
-COPY dpdk /dpdk
+COPY dpdk-stable /dpdk-stable
# Install meson
{% block install_meson %}
# Install meson from dpdk main
# Writes to the mount are discarded once the container is built and is not shared with other containers or the host
-RUN --mount=type=cache,target=/ccache cd /dpdk && chown -R root /dpdk && git checkout main && sh .ci/linux-setup.sh
+COPY dpdk /dpdk
+RUN --mount=type=cache,target=/ccache,z cd /dpdk && chown -R root /dpdk && git checkout main && sh .ci/linux-setup.sh
+RUN rm -rf /dpdk
{% endblock install_meson %}
ENV RTE_KERNELDIR CHANGE_ME
@@ -52,25 +62,47 @@ RUN mkdir /references
WORKDIR /references
RUN touch abi_versions
# build the abi
-{% if build_abi %}
+{% if build_abi and not target.force_disable_abi %}
{% block build_abi %}
- {% for abi_version in abi.versions %}
- {%- set abi_version_str = abi_version.git_tag -%}
+ {% for version in abi.versions %}
+ {%- set abi_version_str = (version.abi_version.major|string) + "." + (version.abi_version.minor|string) -%}
+ {%- set abi_stage_name = target.name + "_" + abi_version_str -%}
{%- set abi_folder = "/references/" + abi_version_str + "_reference" %}
+ {%- set meets_version_requirements = "minimum_dpdk_version" in target and version.dpdk_version.major >= target.minimum_dpdk_version.major and version.dpdk_version.minor >= target.minimum_dpdk_version.minor and version.dpdk_version.revision >= target.minimum_dpdk_version.revision -%}
+ {% if meets_version_requirements %}
+
+FROM {{ build_base_img }} AS {{ abi_stage_name }}
RUN echo '{{ abi_version_str }}' >> abi_versions
RUN mkdir -p {{ abi_folder }}
# compile DPDK
-RUN --mount=type=cache,target=/ccache \
- mkdir -p /dpdk/build && rm -rf /dpdk/build && cd /dpdk && git clean -xfd && git fetch --tags && git checkout tags/{{ abi_version_str }} && cd /references && \
- meson /dpdk /dpdk/build {% if "cross_file" in target -%}
- --cross-file=/dpdk/{{ target.cross_file }}
- {%- endif %} -Dexamples=all --buildtype=debugoptimized && DESTDIR={{ abi_folder }} ninja -C /dpdk/build install && \
- bash /dpdk/devtools/gen-abi.sh {{ abi_folder }} && mv {{ abi_folder }}/usr/local/include {{ abi_folder }} && \
- rm -rf {{ abi_folder }}/usr && rm -rf /dpdk/build
+RUN --mount=type=cache,target=/ccache,z \
+ mkdir -p /dpdk-stable/build && rm -rf /dpdk-stable/build && cd /dpdk-stable && git clean -xfd && git fetch --tags && git checkout {{ version.dpdk_version.git_tag }} && cd /references && \
+ meson /dpdk-stable /dpdk-stable/build {% if "cross_file" in target -%}
+ --cross-file=/dpdk-stable/{{ target.cross_file }}
+ {%- endif %} -Dexamples=all --buildtype=debugoptimized && DESTDIR={{ abi_folder }} ninja {% if ninja_workers -%}
+ -j {{ ninja_workers }} {% endif -%} -C /dpdk-stable/build install && \
+ {%- if (abi_version_str == '22.0' or abi_version_str == '23.10') %}
+ bash /dpdk-stable/devtools/gen-abi.sh {{ abi_folder }} && mv {{ abi_folder }}/usr/local/include {{ abi_folder }} && \
+ rm -rf {{ abi_folder }}/usr
+ {%- else %}
+ find {{ abi_folder }}/usr/local -name '*.a' -delete && rm -rf {{ abi_folder }}/usr/local/bin && rm -rf {{ abi_folder }}/usr/local/share
+ {%- endif %} && rm -rf /dpdk-stable/build
+ {% endif %}
{% endfor %}
-RUN rm -rf /dpdk
-RUN chmod -R 777 .
+
+FROM {{ build_base_img }} AS {{ target.name }}
+ {% for version in abi.versions %}
+ {%- set abi_version_str = (version.abi_version.major|string) + "." + (version.abi_version.minor|string) -%}
+ {%- set abi_stage_name = target.name + "_" + abi_version_str -%}
+ {%- set abi_folder = "/references/" + abi_version_str + "_reference" %}
+ {%- set meets_version_requirements = "minimum_dpdk_version" in target and version.dpdk_version.major >= target.minimum_dpdk_version.major and version.dpdk_version.minor >= target.minimum_dpdk_version.minor and version.dpdk_version.revision >= target.minimum_dpdk_version.revision -%}
+ {% if meets_version_requirements %}
+RUN mkdir -p {{ abi_folder }}
+COPY --from={{ abi_stage_name }} {{ abi_folder }} {{ abi_folder }}
+ {%- endif %}
+ {%- endfor %}
{% endblock build_abi %}
+RUN chmod -R 777 .
{% else %}
{% endif %}
@@ -79,7 +111,11 @@ RUN chmod -R 777 .
RUN echo '{{ target.extra_information | tojson }}' > /container_info.json
{% endif %}
-COPY ./scripts /scripts
WORKDIR /
+
+{% block copy_extras -%}
+COPY scripts scripts
+{%- endblock copy_extras %}
+
# ------------------------ End {{ target.name }} ------------------------------------------------
diff --git a/containers/template_engine/templates/containers/debian11.dockerfile.j2 b/containers/template_engine/templates/containers/debian11.dockerfile.j2
index 0fd8a68..c52f866 100644
--- a/containers/template_engine/templates/containers/debian11.dockerfile.j2
+++ b/containers/template_engine/templates/containers/debian11.dockerfile.j2
@@ -1,3 +1,3 @@
{#- SPDX-License-Identifier: BSD-3-Clause -#}
{#- Copyright (c) 2022 University of New Hampshire -#}
-{% extends 'containers/debian.dockerfile.j2' %}
+{% extends 'containers/debian.dockerfile.j2' %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2 b/containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2
new file mode 100644
index 0000000..ccef9cb
--- /dev/null
+++ b/containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2
@@ -0,0 +1,16 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/debian11.dockerfile.j2' %}
+
+{% block before_meson_after_pip %}
+WORKDIR /opt
+
+# I would use ADD on a git repo, but the support for this was added so recently it doesn't run on my machine.
+RUN git clone --depth 1 --branch SECLIB-IPSEC-2022.05.25 https://git.gitlab.arm.com/arm-reference-solutions/ipsec-mb.git
+
+WORKDIR /opt/ipsec-mb
+RUN make -j $(nproc) && make install
+
+WORKDIR /
+
+{% endblock before_meson_after_pip %}
diff --git a/containers/template_engine/templates/containers/fedora_clang.dockerfile.j2 b/containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
new file mode 100644
index 0000000..bc104ba
--- /dev/null
+++ b/containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
@@ -0,0 +1,7 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2022 University of New Hampshire -#}
+{% extends 'containers/fedora.dockerfile.j2' %}
+
+{% block extra_env_vars -%}
+ENV CC=clang
+{%- endblock extra_env_vars %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2 b/containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2
new file mode 100644
index 0000000..f17eb7a
--- /dev/null
+++ b/containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2
@@ -0,0 +1,10 @@
+{#- SPDX-License-Identifier: BSD-3-Clause -#}
+{#- Copyright (c) 2023 University of New Hampshire -#}
+{% extends 'containers/fedora.dockerfile.j2' %}
+
+{% block copy_extras -%}
+COPY scripts scripts
+COPY coverity coverity
+
+ENTRYPOINT /scripts/coverity.sh
+{%- endblock copy_extras %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/rhel.dockerfile.j2 b/containers/template_engine/templates/containers/rhel.dockerfile.j2
index a350e84..b50b2e9 100644
--- a/containers/template_engine/templates/containers/rhel.dockerfile.j2
+++ b/containers/template_engine/templates/containers/rhel.dockerfile.j2
@@ -8,5 +8,9 @@ RUN python3 -m pip install pyelftools
{% block before_meson_after_pip %}
-RUN yum --disableplugin=subscription-manager clean all
-{% endblock %}
\ No newline at end of file
+RUN yum clean all
+{% endblock %}
+
+{% block install_libabigail %}
+{# Install via package manager #}
+{% endblock install_libabigail %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/rhel7.dockerfile.j2 b/containers/template_engine/templates/containers/rhel7.dockerfile.j2
index ccfc0ec..7045b99 100644
--- a/containers/template_engine/templates/containers/rhel7.dockerfile.j2
+++ b/containers/template_engine/templates/containers/rhel7.dockerfile.j2
@@ -9,11 +9,7 @@ RUN yum repolist
RUN sed -i '/\[rhel-7-server-extras-rpms\]/,/^ *\[/ s/enabled\ =\ 0/enabled\ =\ 1/' /etc/yum.repos.d/redhat.repo \
&& sed -i '/\[rhel-7-server-optional-rpms\]/,/^ *\[/ s/enabled\ =\ 0/enabled\ =\ 1/' /etc/yum.repos.d/redhat.repo
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
-RUN yum --disableplugin=subscription-manager update -y
+RUN yum update -y
RUN yum groupinstall -y 'Development Tools'
-{% endblock %}
-
-{% block install_libabigail %}
-{# Install via package manager #}
-{% endblock install_libabigail %}
\ No newline at end of file
+{% endblock %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/rhel8.dockerfile.j2 b/containers/template_engine/templates/containers/rhel8.dockerfile.j2
index dcaffc9..2824bfd 100644
--- a/containers/template_engine/templates/containers/rhel8.dockerfile.j2
+++ b/containers/template_engine/templates/containers/rhel8.dockerfile.j2
@@ -5,15 +5,11 @@
{% block pre_package_manager %}
# repolist is needed to refresh the RHEL repos from the host this is being built on
RUN yum repolist
-RUN yum --disableplugin=subscription-manager update -y
-RUN yum --disableplugin=subscription-manager config-manager --set-enable codeready-builder-for-rhel-8-$(uname -i)-rpms
+RUN yum update -y
+RUN yum config-manager --set-enable codeready-builder-for-rhel-8-$(uname -i)-rpms
# Required to install ccache
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm && yum upgrade -y
RUN yum groupinstall -y 'Development Tools'
-{% endblock %}
-
-{% block install_libabigail %}
-{# Install via package manager #}
-{% endblock install_libabigail %}
\ No newline at end of file
+{% endblock %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/rhel9.dockerfile.j2 b/containers/template_engine/templates/containers/rhel9.dockerfile.j2
index 8f0eccc..695f573 100644
--- a/containers/template_engine/templates/containers/rhel9.dockerfile.j2
+++ b/containers/template_engine/templates/containers/rhel9.dockerfile.j2
@@ -5,8 +5,8 @@
{% block pre_package_manager %}
# repolist is needed to refresh the RHEL repos from the host this is being built on
RUN yum repolist
-RUN yum --disableplugin=subscription-manager config-manager --set-enable codeready-builder-for-rhel-9-$(uname -i)-rpms
-RUN yum --disableplugin=subscription-manager update -y
+RUN yum config-manager --set-enable codeready-builder-for-rhel-9-$(uname -i)-rpms
+RUN yum update -y
# Required to install ccache
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && yum upgrade -y
@@ -16,10 +16,4 @@ RUN yum groupinstall -y 'Development Tools'
{% block pip_installs %}
RUN python3 -m pip install pyelftools ninja
-{% endblock %}
-
-{% block install_libabigail %}
-COPY libabigail libabigail
-RUN cd libabigail && mkdir build && autoreconf -i && cd build && ../configure --prefix=/usr/local && make all install && \
- ldconfig && cd .. && rm -rf libabigail
-{% endblock install_libabigail %}
\ No newline at end of file
+{% endblock %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2 b/containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
index 7a8b971..ba22471 100644
--- a/containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
+++ b/containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
@@ -1,7 +1,3 @@
{#- SPDX-License-Identifier: BSD-3-Clause -#}
{#- Copyright (c) 2022 University of New Hampshire -#}
-{% extends 'containers/ubuntu.dockerfile.j2' %}
-
-{% block install_libabigail %}
-# installed by package manager
-{% endblock install_libabigail %}
\ No newline at end of file
+{% extends 'containers/ubuntu.dockerfile.j2' %}
\ No newline at end of file
diff --git a/containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2 b/containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
index 236efbc..fbdd6dc 100644
--- a/containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
+++ b/containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
@@ -8,4 +8,5 @@
RUN ln -s $(which ccache) /usr/local/bin/aarch64-linux-gnu-gcc && \
# aarch32 -> ccache
ln -s $(which ccache) /usr/local/bin/arm-linux-gnueabihf-gcc
-{% endblock cross_compilation_setup %}
\ No newline at end of file
+{% endblock cross_compilation_setup %}
+
diff --git a/containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2 b/containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
index 3c3e471..5e940bf 100644
--- a/containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
+++ b/containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
@@ -7,4 +7,6 @@ RUN ln -s $(which ccache) /usr/local/bin/gcc-10
{% block build_abi %}
# ABI is disabled for this container
+
+FROM {{ build_base_img }} AS {{ target.name }}
{% endblock build_abi %}
\ No newline at end of file
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 10/11] containers/Makefile: Add new variables and target changes
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
` (8 preceding siblings ...)
2023-05-23 17:04 ` [PATCH v5 09/11] containers/templates: Update templates Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-23 17:04 ` [PATCH v5 11/11] containers/docs: Update README Adam Hassick
10 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Adam Hassick, aconole, alialnu
Adds new variables in the Makefile, as well as new defaults.
These take advantage of the new arguments added to the
Dockerfile template script.
New targets were added to push only images, OCI manifests,
or both. The new behavior of the "push" target is to push
everything.
Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
---
containers/Makefile | 116 ++++++++++++++++++++++++++++++++++++++------
1 file changed, 101 insertions(+), 15 deletions(-)
diff --git a/containers/Makefile b/containers/Makefile
index 3d56162..714c326 100644
--- a/containers/Makefile
+++ b/containers/Makefile
@@ -33,6 +33,19 @@ DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=N
# in the container
DPDK_CI_CONTAINERS_BUILD_ABI?=N
+# If set to 'Y', the "latest" tag for all images will be omitted. Intended for testing changes to your inventory.
+DPDK_CI_CONTAINERS_NO_LATEST_TAG?=N
+
+# If set to 'Y', will build containers that rely on the Coverity Scan tool.
+DPDK_CI_CONTAINERS_COVERITY?=N
+
+# The path to Coverity Scan binaries. These will be installed inside the container.
+# This field is only required if the Coverity flag is enabled.
+# DPDK_CI_CONTAINERS_COVERITY_PATH?=/opt/dpdklab/coverity
+
+# If set to a non-empty value, overrides the auto-generated date tag with the value.
+# DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE=70-01-01
+
# If set to 'Y', build libabigail from source for distros that do not have it.
# If DPDK_CI_CONTAINERS_BUILD_ABI='Y', then this is always enabled.
ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y)
@@ -41,6 +54,16 @@ else
DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL?=N
endif
+# If set to an integer value, this will restrict the count of Ninja workers performing the ABI build to the given integer.
+# The argument is benign if DPDK_CI_CONTANERS_BUILD_ABI is 'N'.
+# DPDK_CI_CONTAINERS_NINJA_WORKERS?=16
+
+# If set to 'Y', only build containers matching the host architecture
+DPDK_CI_CONTAINERS_ONLY_HOST_ARCH?=N
+
+# If set to 'Y', assumes only host arch and disables the push_manifests target.
+DPDK_CI_CONTAINERS_IS_BUILDER?=N
+
# Used to set the python interpreter
DPDK_CI_CONTAINERS_PYTHON3_CMD?=python3.8
@@ -51,7 +74,10 @@ DPDK_CI_CONTAINER_BUILDER_PROGRAM?=podman
DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=git://sourceware.org/git/libabigail.git
# The url of the git repository to pull DPDK from
-DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=https://dpdk.org/git/dpdk
+DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=http://dpdk.org/git/dpdk
+
+# The url of the git repository to pull DPDK stable from
+DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL?=http://dpdk.org/git/dpdk-stable
# The tag to apply to the built container builder image
DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder
@@ -59,10 +85,11 @@ DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder
# Extra arguments to add to the push command, can be used for credentials
DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?=
+# Provide the hostname of the registry to push up to
DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=localhost
# The path to a directory to be recursively copied to $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY). Not used if unset.
-# DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH
+#DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH?=
DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context
############ End Arguments #############
@@ -71,25 +98,50 @@ DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context
SCRIPT_ARGS=--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)"
ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y)
- SCRIPT_ARGS +=--rhel
+ SCRIPT_ARGS +=--rhel
endif
ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y)
- SCRIPT_ARGS +=--fail-on-unbuildable
+ SCRIPT_ARGS +=--fail-on-unbuildable
endif
ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y)
- SCRIPT_ARGS +=--build-abi
+ SCRIPT_ARGS +=--build-abi
endif
ifeq ($(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL), Y)
- SCRIPT_ARGS +=--build-libabigail
+ SCRIPT_ARGS +=--build-libabigail
+endif
+
+ifeq ($(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH), Y)
+ SCRIPT_ARGS +=--host-arch-only
+endif
+
+ifeq ($(DPDK_CI_CONTAINERS_NO_LATEST_TAG), Y)
+ SCRIPT_ARGS +=--omit-latest
+endif
+
+ifeq ($(DPDK_CI_CONTAINERS_IS_BUILDER), Y)
+ SCRIPT_ARGS +=--builder-mode
+endif
+
+ifneq ($(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE),)
+ SCRIPT_ARGS +=--date $(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE)
+endif
+
+ifneq ($(DPDK_CI_CONTAINERS_NINJA_WORKERS),)
+ SCRIPT_ARGS +=--ninja-workers $(DPDK_CI_CONTAINERS_NINJA_WORKERS)
+endif
+
+ifeq ($(DPDK_CI_CONTAINERS_COVERITY), Y)
+ SCRIPT_ARGS +=--coverity
endif
SCRIPT_DIRECTORY=$(CURDIR)/template_engine
LIBABIGAIL_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabigail
DPDK_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk
+DPDK_STABLE_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-stable
DOCKERFILE=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile
@@ -103,6 +155,10 @@ INVENTORY_FILE=$(SCRIPT_DIRECTORY)/inventory.yaml
EXTRA_SCRIPTS_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scripts
+EXTRA_CONFIG_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/configs
+
+COVERITY_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/coverity
+
GIT_FETCH_ARGS=--all --tags
CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile
@@ -110,23 +166,36 @@ CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile
export
############ End Internal Variables #############
-.PHONY: build push make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk clean_container_files clean
+.PHONY: build push push_images push_manifests make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk coverity clean_container_files clean
-build: make_docker_files_in_container extra_scripts
+build: make_docker_files_in_container external_files
$(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) build_all
-push: make_docker_files_in_container extra_scripts
- $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_all
+push_images: make_docker_files_in_container external_files
+ $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_image_all
+
+push_manifests: make_docker_files_in_container
+ $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_manifest_all
+
+push: push_images push_manifests
make_docker_files_in_container: build_builder_container $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
- $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run -v $(CURDIR):/container_workspace -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context \
+ $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run --rm -v $(CURDIR):/container_workspace:z -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context:z \
-e DPDK_CI_CONTAINERS_ON_RHEL=$(DPDK_CI_CONTAINERS_ON_RHEL) \
-e DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=$(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL) \
-e DPDK_CI_CONTAINERS_BUILD_ABI=$(DPDK_CI_CONTAINERS_BUILD_ABI) \
-e DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE=$(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE) \
-e DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME='$(DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME)' \
+ -e DPDK_CI_CONTAINERS_ONLY_HOST_ARCH='$(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH)' \
+ -e DPDK_CI_CONTAINERS_NO_LATEST_TAG='$(DPDK_CI_CONTAINERS_NO_LATEST_TAG)' \
+ -e DPDK_CI_CONTAINERS_IS_BUILDER='$(DPDK_CI_CONTAINERS_IS_BUILDER)' \
+ -e DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE='$(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE)' \
+ -e DPDK_CI_CONTAINERS_NINJA_WORKERS='$(DPDK_CI_CONTAINERS_NINJA_WORKERS)' \
+ -e DPDK_CI_CONTAINERS_COVERITY='$(DPDK_CI_CONTAINERS_COVERITY)' \
$(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) make docker_deps
+external_files: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) extra_scripts coverity
+
build_builder_container:
$(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f $(CONTAINER_BUILDER_DOCKERFILE) -t $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) $(CURDIR)
@@ -136,13 +205,12 @@ docker_deps: $(DOCKERFILE) deps extra_scripts
$(DOCKERFILE): $(INVENTORY_FILE) $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) deps $(MAKE_DOCKERFILE_SCRIPT) $(shell find $(DOCKER_CONTAINER_TEMPLATE_DIR) -type f)
cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFILE_COMMAND) $(SCRIPT_ARGS)
-deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk libabigail
+deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk libabigail dpdk-stable
extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(EXTRA_SCRIPTS_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)
-ifdef $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)
- cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) $(EXTRA_SCRIPTS_DIRECTORY)
+ifdef DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH
+ cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)/* $(EXTRA_SCRIPTS_DIRECTORY)
endif
- mkdir -p $(EXTRA_SCRIPTS_DIRECTORY)
# Clone libabigail source
libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
@@ -153,9 +221,27 @@ libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY)
+# Copy the Coveirty Scan binaries from a source into the context directory.
+coverity: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(COVERITY_DIRECTORY)
+ifeq ($(DPDK_CI_CONTAINERS_COVERITY),Y)
+ifndef DPDK_CI_CONTAINERS_COVERITY_PATH
+ exit 1 # The coverity path must be set if building Coverity containers is enabled.
+else
+ cp -pur $(DPDK_CI_CONTAINERS_COVERITY_PATH)/* $(COVERITY_DIRECTORY)
+endif
+else
+ # Coverity is disabled for this run.
+endif
+
+dpdk-stable: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
+ git -C $(DPDK_STABLE_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL) $(DPDK_STABLE_DIRECTORY)
+
$(EXTRA_SCRIPTS_DIRECTORY):
mkdir -p $(EXTRA_SCRIPTS_DIRECTORY)
+$(COVERITY_DIRECTORY):
+ mkdir -p $(COVERITY_DIRECTORY)
+
$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):
mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 11/11] containers/docs: Update README
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
` (9 preceding siblings ...)
2023-05-23 17:04 ` [PATCH v5 10/11] containers/Makefile: Add new variables and target changes Adam Hassick
@ 2023-05-23 17:04 ` Adam Hassick
2023-05-25 13:45 ` Aaron Conole
10 siblings, 1 reply; 15+ messages in thread
From: Adam Hassick @ 2023-05-23 17:04 UTC (permalink / raw)
To: ci; +Cc: Adam Hassick, aconole, alialnu
Updates the README with information on the added arguments
and support for building OCI manifests.
Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
---
containers/README.md | 123 +++++++++++++++++++++++++++----------------
1 file changed, 77 insertions(+), 46 deletions(-)
diff --git a/containers/README.md b/containers/README.md
index 5d01caf..59edfcc 100644
--- a/containers/README.md
+++ b/containers/README.md
@@ -14,73 +14,80 @@ nearly mandatory for this task.
1. Licensed RHEL containers need podman
-The build system MUST be able to handle creating properly licensed RHEL
-containers, so that the RHEL CI testing is as accurate as possible.
+The build system MUST be able to handle creating properly licensed RHEL
+containers, so that the RHEL CI testing is as accurate as possbile.
2. "Developer Laptop Friendliness"
-Another goal of the build system was to enable anyone to easily build the
+Another goal of the build system was to enable anyone to easily build the
containers. Not all developers are able to use Linux as the main OS on their
main development machine. Podman runs on MacOS via podman-machine and Windows
-either by podman-machine or WSL.
+either by podman-machine or WSL.
3. OCI Containers
-OCI containers are more portable than some other container solutions. Much of
+OCI containers are more portable than some other container solutions. Much of
the progress on getting containers running on top of FreeBSD jails targets OCI
-containers specifically. The tracking issue for this is
+containers specifically. The tracking issue for this is
[https://reviews.freebsd.org/D21570](https://reviews.freebsd.org/D21570).
-Once upstream support happens, there should be a relatively simple path to
+Once upstream support happens, there should be a relatively simple path to
supporting containers in FreeBSD once podman/docker APIs are better supported.
-At the moment, lack up upstream support means no support in this project for
-FreeBSD.
+At the moment, lack up upstream support means no support in this project for
+FreeBSD.
### Python and Makefiles instead of Buildah as a library
-The next question someone might have is why a combination of Python and
+The next question someone might have is why a combination of Python and
makefiles were used instead of using buildah as a library. The largest
reason is that every DPDK developer is going to need to have some
-level of familiarity with Python due to DTS. Buildah is only available
-as a library via Go, and would tie DPDK to a particular container
+level of familarity with Python due to DTS. Buildah is only avaliable
+as a library via Go, and would tie DPDK to a particular container
implementation. Go, while not difficult to learn, is a compiled language,
meaning that the build system would require a build system.
-The other reason is that most of the logic that needs to be performed is very
-simple, and python has a few libraries that do most of the work. If it weren't
-for the desire to have an inventory file (inventory.yaml) with a schema
-(inventory_schema.json), this probably could have been an AWK script. After the
-container images are produced, it is very easy to use the same template
-engine to produce a makefile that can be used to both build and push the
-containers. This makefile can be run with multiple jobs for parallel building
-of containers, something not supported by all compose implementations.
-
-Meson was considered instead of Makefiles, however, Meson does not handle new
-Meson being generated during the build very well, and Meson wants most commands
-to have an output file, which is not true of many of the commands. Meson is
-also more difficult to generate using a templating library than Makefile
-targets.
+The other reason is that most of the logic that needs to be performed is very
+simple, and python has a few libraries that do most of the work. If it weren't
+for the desire to have an inventory file (inventory.yaml) with a schema
+(inventory_schema.json), this probably could have been an AWK script. After the
+container images are produced, it is very easy to use the same templating
+engine to produce a makefile that can be used to both build and push the
+containers. This makefile can be run with multiple jobs for parallel building
+of containers, something not supported by all compose implementations.
+
+Meson was considered instead of Makefiles, however, Meson does not handle new
+Meson being generated during the build very well, and Meson wants most commands
+to have an output file, which is not true of many of the commands. Meson is
+also more difficult to generate using a templating library than Makefile
+targets.
## Building
### Environment Variables
-All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any
-issues.
+All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any
+issues.
| Variable | Description | Default | Valid Values |
| -------------------------- | ----------------------------------------------- | ------- | ------------ |
-DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N'
+| DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N'
DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE | Fail during dockerfile generation if any container in the inventory is not buildable. Currently will cause a failure if you are not on RHEL and try to build RHEL containers. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_ONLY_HOST_ARCH | If set to 'Y', only images for the local system architecture will be built. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_IS_BUILDER | If set to 'Y', disables the manifest features, and only builds images for the local system architecture. Intended to be set when used inside another orchestration sofware. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_NINJA_WORKERS | The number of Ninja workers to use to build ABI images. Variable setting is benign if ABI is disabled. | unset | Any positive integer greater than zero.
DPDK_CI_CONTAINERS_BUILD_ABI | Whether to bake ABI images into the containers. | 'N' | 'Y' or 'N'
-DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL | Whether to build libabigail from source on distros that do not package it. DPDK_CI_CONTAINERS_BUILD_ABI=Y overrides this to 'Y' | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_NO_LATEST_TAG | Disables tagging the final manifests as "latest" in the local store and remote registry. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_COVERITY | Enable building Coverity images. Setting this flag will make the Coverity binaries required. | 'N' | 'Y' or 'N'
DPDK_CI_CONTAINER_BUILDER_PROGRAM | What container builder program to use. | 'podman' | Any container builder that exposes the same interface and provides the same behavior as podman.
-DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository.
+DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository.
DPDK_CI_CONTAINERS_DPDK_CLONE_URL | What URL to clone DPDK from. | 'https://dpdk.org/git/dpdk' | Any DPDK mirror.
+DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL | What URL to clone DPDK stable form. | http://dpdk.org/git/dpdk-stable | Any DPDK stable mirror.
DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG | What tag to give to the container which creates the dockerfiles. The default should be fine unless you have issues with collisions. | 'dpdk_ci_container_builder' | Any valid OCI container tag (A valid C function name will work)
DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS | Extra arguments to add to the push command, can be used for credentials if 'podman login' won't work. | '' | [https://docs.podman.io/en/latest/markdown/podman-push.1.html#options](https://docs.podman.io/en/latest/markdown/podman-push.1.html#options)
DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME | The hostname of the registry to push to. | 'localhost' | The hostname of any system exposing an OCI container registry or localhost to push to local storage.
DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH | The path to a directory to copy into all of the containers at /scripts | unset | The path to any local file directory.
+DPDK_CI_CONTAINERS_COVERITY_PATH | The path to Coverity Scan binaries. Only required of the Coverity flag is set. | unset | The path to any local file directory.
DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers in. All generated files will be placed in this directory or one of it's children | '$(CURDIR)/container_context' | Any absolute directory path
+DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE | Uses a provided string instead of generating a new date tag. Intended for development use. | unset | Any string that is a valid OCI manifest tag.
### Builder System Requirements
@@ -91,16 +98,15 @@ DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers
* find
* posix utilities (GNU coreutils will work)
* bash
-* podman >= 4.0.0 (docker or other container builder programs may work, but are
-unsupported)
- * podman 4.0.0 allows run mounts, which allow mounting a directory into the build context of a container. This is used to persist ccache directories for each container.
-* qemu-$ARCH-static for any non-native architecture/revision you want to build for.
+* podman (docker is unsupported, and will NOT work for making manifests)
+* qemu-$ARCH-static for any non-native architecture/revision you want to build
+for.
#### Hardware
-| Hardware Type | Requirement | Reason |
-| ------------- | ---------------------------- | ----------------------------------- |
-| Disk space | 5 GB of disk space per image | Some images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references.
+| Hardware Type | Requirement | Reason |
+| ------------- | ----------------------------- | ----------------------------------- |
+| Disk space | 10 GB of disk space per image | Many of the final images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references. Intermediate images generated by the builds will consume some additional space that is recoverable after the build.
Memory | Either 1.5x or 2x the memory needed to compile DPDK per makefile job | 1.5x is enough for the container overhead and caching when compiling natively, 2x is for builds under emulation (ARM container on x86, etc).
@@ -111,28 +117,53 @@ RHEL container images must be built on RHEL.
### Build containers locally
```bash
+# Build using the default arguments
make build
```
+The resulting images will be tagged based on the date tag and platform.
+Image generated tags follow this format: `image-{{ platform }}-{{ date_tag }}`
+Where `platform` denotes the platform of the image, and `date_tag` is the generated date tag or the override string provided
+through the environment variable.
+
+They should appear in the local image store on your system.
+
### Push containers to registry
-This will probably involve following prompts in your terminal, but if you have
-other authentication set up, (LDAP, Kerberos, etc), it may not prompt you.
+This will probably involve following prompts in your terminal, but if you have
+other authentication set up, (LDAP, Kerberos, etc), it may not prompt you.
Logging into a registry is what allows you to upload containers to a remote
-system for others to pull down.
+system for others to pull down.
If you are working alone, you probably can ignore this and keep the containers
-locally. If you are in an enterprise setting, ask your DevOps or Systems
-Administration team where the preferred location for hosting containers is.
+locally. If you are in an enterprise setting, ask your DevOps or Systems
+Administration team where the preferred location for hosting containers is.
-Since these images take so long to build, it is recommended to use a container
+Since these images take so long to build, it is recommended to use a container
registry and have any CI systems pull from that registry.
-Redhat guide to setting up a podman container registry:
+Redhat guide to setting up a podman container registry:
[https://www.redhat.com/sysadmin/simple-container-registry](https://www.redhat.com/sysadmin/simple-container-registry)
```bash
$DPDK_CI_CONTAINER_BUILDER_PROGRAM login $DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME
# < Complete login process >
make push
-```
\ No newline at end of file
+```
+
+#### Manifests
+
+OCI manifests allow the grouping of images for different platforms under the same tag in a repository on a registry.
+The use of OCI manifests over tagged images reduces the amount of system platform related branching in CI scripting.
+
+The Makefile provides the option to push only the images, only the manifests, or push the images and make manifests.
+The default "push" target will perform the last case.
+If you choose to create the manifests, then these will be created with the "final" tags like "latest" and the date timestamp.
+
+Manifest creation is known to not be compatible with Docker.
+This feature is known to work when using Podman to post content to a Docker v2 registry.
+
+The manifests may be created on the registry independently of the image builds using the `push_manifests` target in place of the `push` target.
+In contrast, the `push_images` target will only push the images and not create the manifests.
+
+If the `DPDK_CI_CONTAINERS_IS_BUILDER` variable is set to 'Y', then the `push_manifests` target will be disabled.
\ No newline at end of file
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v5 11/11] containers/docs: Update README
2023-05-23 17:04 ` [PATCH v5 11/11] containers/docs: Update README Adam Hassick
@ 2023-05-25 13:45 ` Aaron Conole
2023-06-05 20:09 ` Adam Hassick
0 siblings, 1 reply; 15+ messages in thread
From: Aaron Conole @ 2023-05-25 13:45 UTC (permalink / raw)
To: Adam Hassick; +Cc: ci, alialnu
Adam Hassick <ahassick@iol.unh.edu> writes:
> Updates the README with information on the added arguments
> and support for building OCI manifests.
>
> Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
> ---
I stopped reviewing. The changes here introduce spelling mistakes and
spaces at the end of lines. Please proofread it before making changes.
> containers/README.md | 123 +++++++++++++++++++++++++++----------------
> 1 file changed, 77 insertions(+), 46 deletions(-)
>
> diff --git a/containers/README.md b/containers/README.md
> index 5d01caf..59edfcc 100644
> --- a/containers/README.md
> +++ b/containers/README.md
> @@ -14,73 +14,80 @@ nearly mandatory for this task.
>
> 1. Licensed RHEL containers need podman
>
> -The build system MUST be able to handle creating properly licensed RHEL
> -containers, so that the RHEL CI testing is as accurate as possible.
> +The build system MUST be able to handle creating properly licensed RHEL
> +containers, so that the RHEL CI testing is as accurate as possbile.
Why these lines changed? Rather, I see that a space was added at the
end, and a spelling mistake was introduced s/possbile/possible/. That
can't be intentional - please undo this change.
> 2. "Developer Laptop Friendliness"
>
> -Another goal of the build system was to enable anyone to easily build the
> +Another goal of the build system was to enable anyone to easily build the
> containers. Not all developers are able to use Linux as the main OS on their
> main development machine. Podman runs on MacOS via podman-machine and Windows
> -either by podman-machine or WSL.
> +either by podman-machine or WSL.
Why were spaces added at the end in this section? Please don't do that.
> 3. OCI Containers
>
> -OCI containers are more portable than some other container solutions. Much of
> +OCI containers are more portable than some other container solutions. Much of
> the progress on getting containers running on top of FreeBSD jails targets OCI
> -containers specifically. The tracking issue for this is
> +containers specifically. The tracking issue for this is
> [https://reviews.freebsd.org/D21570](https://reviews.freebsd.org/D21570).
> -Once upstream support happens, there should be a relatively simple path to
> +Once upstream support happens, there should be a relatively simple path to
> supporting containers in FreeBSD once podman/docker APIs are better supported.
> -At the moment, lack up upstream support means no support in this project for
> -FreeBSD.
> +At the moment, lack up upstream support means no support in this project for
> +FreeBSD.
>
> ### Python and Makefiles instead of Buildah as a library
>
> -The next question someone might have is why a combination of Python and
> +The next question someone might have is why a combination of Python and
> makefiles were used instead of using buildah as a library. The largest
> reason is that every DPDK developer is going to need to have some
> -level of familiarity with Python due to DTS. Buildah is only available
> -as a library via Go, and would tie DPDK to a particular container
> +level of familarity with Python due to DTS. Buildah is only avaliable
familiarity was correct. available was correct.
> +as a library via Go, and would tie DPDK to a particular container
> implementation. Go, while not difficult to learn, is a compiled language,
> meaning that the build system would require a build system.
>
> -The other reason is that most of the logic that needs to be performed is very
> -simple, and python has a few libraries that do most of the work. If it weren't
> -for the desire to have an inventory file (inventory.yaml) with a schema
> -(inventory_schema.json), this probably could have been an AWK script. After the
> -container images are produced, it is very easy to use the same template
> -engine to produce a makefile that can be used to both build and push the
> -containers. This makefile can be run with multiple jobs for parallel building
> -of containers, something not supported by all compose implementations.
> -
> -Meson was considered instead of Makefiles, however, Meson does not handle new
> -Meson being generated during the build very well, and Meson wants most commands
> -to have an output file, which is not true of many of the commands. Meson is
> -also more difficult to generate using a templating library than Makefile
> -targets.
> +The other reason is that most of the logic that needs to be performed is very
> +simple, and python has a few libraries that do most of the work. If it weren't
> +for the desire to have an inventory file (inventory.yaml) with a schema
> +(inventory_schema.json), this probably could have been an AWK script. After the
> +container images are produced, it is very easy to use the same templating
> +engine to produce a makefile that can be used to both build and push the
> +containers. This makefile can be run with multiple jobs for parallel building
> +of containers, something not supported by all compose implementations.
> +
> +Meson was considered instead of Makefiles, however, Meson does not handle new
> +Meson being generated during the build very well, and Meson wants most commands
> +to have an output file, which is not true of many of the commands. Meson is
> +also more difficult to generate using a templating library than Makefile
> +targets.
>
> ## Building
>
> ### Environment Variables
>
> -All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any
> -issues.
> +All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any
> +issues.
>
> | Variable | Description | Default | Valid Values |
> | -------------------------- | ----------------------------------------------- | ------- | ------------ |
> -DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N'
> +| DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N'
> DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE | Fail during dockerfile generation if any container in the inventory is not buildable. Currently will cause a failure if you are not on RHEL and try to build RHEL containers. | 'N' | 'Y' or 'N'
> +DPDK_CI_CONTAINERS_ONLY_HOST_ARCH | If set to 'Y', only images for the local system architecture will be built. | 'N' | 'Y' or 'N'
> +DPDK_CI_CONTAINERS_IS_BUILDER | If set to 'Y', disables the manifest features, and only builds images for the local system architecture. Intended to be set when used inside another orchestration sofware. | 'N' | 'Y' or 'N'
> +DPDK_CI_CONTAINERS_NINJA_WORKERS | The number of Ninja workers to use to build ABI images. Variable setting is benign if ABI is disabled. | unset | Any positive integer greater than zero.
> DPDK_CI_CONTAINERS_BUILD_ABI | Whether to bake ABI images into the containers. | 'N' | 'Y' or 'N'
> -DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL | Whether to build libabigail from source on distros that do not package it. DPDK_CI_CONTAINERS_BUILD_ABI=Y overrides this to 'Y' | 'N' | 'Y' or 'N'
> +DPDK_CI_CONTAINERS_NO_LATEST_TAG | Disables tagging the final manifests as "latest" in the local store and remote registry. | 'N' | 'Y' or 'N'
> +DPDK_CI_CONTAINERS_COVERITY | Enable building Coverity images. Setting this flag will make the Coverity binaries required. | 'N' | 'Y' or 'N'
> DPDK_CI_CONTAINER_BUILDER_PROGRAM | What container builder program to use. | 'podman' | Any container builder that exposes the same interface and provides the same behavior as podman.
> -DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository.
> +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository.
> DPDK_CI_CONTAINERS_DPDK_CLONE_URL | What URL to clone DPDK from. | 'https://dpdk.org/git/dpdk' | Any DPDK mirror.
> +DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL | What URL to clone DPDK stable form. | http://dpdk.org/git/dpdk-stable | Any DPDK stable mirror.
> DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG | What tag to give to the container which creates the dockerfiles. The default should be fine unless you have issues with collisions. | 'dpdk_ci_container_builder' | Any valid OCI container tag (A valid C function name will work)
> DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS | Extra arguments to add to the push command, can be used for credentials if 'podman login' won't work. | '' | [https://docs.podman.io/en/latest/markdown/podman-push.1.html#options](https://docs.podman.io/en/latest/markdown/podman-push.1.html#options)
> DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME | The hostname of the registry to push to. | 'localhost' | The hostname of any system exposing an OCI container registry or localhost to push to local storage.
> DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH | The path to a directory to copy into all of the containers at /scripts | unset | The path to any local file directory.
> +DPDK_CI_CONTAINERS_COVERITY_PATH | The path to Coverity Scan binaries. Only required of the Coverity flag is set. | unset | The path to any local file directory.
> DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers in. All generated files will be placed in this directory or one of it's children | '$(CURDIR)/container_context' | Any absolute directory path
> +DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE | Uses a provided string instead of generating a new date tag. Intended for development use. | unset | Any string that is a valid OCI manifest tag.
>
> ### Builder System Requirements
>
> @@ -91,16 +98,15 @@ DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers
> * find
> * posix utilities (GNU coreutils will work)
> * bash
> -* podman >= 4.0.0 (docker or other container builder programs may work, but are
> -unsupported)
> - * podman 4.0.0 allows run mounts, which allow mounting a directory into the build context of a container. This is used to persist ccache directories for each container.
> -* qemu-$ARCH-static for any non-native architecture/revision you want to build for.
> +* podman (docker is unsupported, and will NOT work for making manifests)
> +* qemu-$ARCH-static for any non-native architecture/revision you want to build
> +for.
>
> #### Hardware
>
> -| Hardware Type | Requirement | Reason |
> -| ------------- | ---------------------------- | ----------------------------------- |
> -| Disk space | 5 GB of disk space per image | Some images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references.
> +| Hardware Type | Requirement | Reason |
> +| ------------- | ----------------------------- | ----------------------------------- |
> +| Disk space | 10 GB of disk space per image | Many of the final images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references. Intermediate images generated by the builds will consume some additional space that is recoverable after the build.
> Memory | Either 1.5x or 2x the memory needed to compile DPDK per makefile job | 1.5x is enough for the container overhead and caching when compiling natively, 2x is for builds under emulation (ARM container on x86, etc).
>
>
> @@ -111,28 +117,53 @@ RHEL container images must be built on RHEL.
> ### Build containers locally
>
> ```bash
> +# Build using the default arguments
> make build
> ```
>
> +The resulting images will be tagged based on the date tag and platform.
> +Image generated tags follow this format: `image-{{ platform }}-{{ date_tag }}`
> +Where `platform` denotes the platform of the image, and `date_tag` is the generated date tag or the override string provided
> +through the environment variable.
> +
> +They should appear in the local image store on your system.
> +
> ### Push containers to registry
>
> -This will probably involve following prompts in your terminal, but if you have
> -other authentication set up, (LDAP, Kerberos, etc), it may not prompt you.
> +This will probably involve following prompts in your terminal, but if you have
> +other authentication set up, (LDAP, Kerberos, etc), it may not prompt you.
> Logging into a registry is what allows you to upload containers to a remote
> -system for others to pull down.
> +system for others to pull down.
>
> If you are working alone, you probably can ignore this and keep the containers
> -locally. If you are in an enterprise setting, ask your DevOps or Systems
> -Administration team where the preferred location for hosting containers is.
> +locally. If you are in an enterprise setting, ask your DevOps or Systems
> +Administration team where the preferred location for hosting containers is.
>
> -Since these images take so long to build, it is recommended to use a container
> +Since these images take so long to build, it is recommended to use a container
> registry and have any CI systems pull from that registry.
>
> -Redhat guide to setting up a podman container registry:
> +Redhat guide to setting up a podman container registry:
> [https://www.redhat.com/sysadmin/simple-container-registry](https://www.redhat.com/sysadmin/simple-container-registry)
>
> ```bash
> $DPDK_CI_CONTAINER_BUILDER_PROGRAM login $DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME
> # < Complete login process >
> make push
> -```
> \ No newline at end of file
> +```
> +
> +#### Manifests
> +
> +OCI manifests allow the grouping of images for different platforms under the same tag in a repository on a registry.
> +The use of OCI manifests over tagged images reduces the amount of system platform related branching in CI scripting.
> +
> +The Makefile provides the option to push only the images, only the manifests, or push the images and make manifests.
> +The default "push" target will perform the last case.
> +If you choose to create the manifests, then these will be created with the "final" tags like "latest" and the date timestamp.
> +
> +Manifest creation is known to not be compatible with Docker.
> +This feature is known to work when using Podman to post content to a Docker v2 registry.
> +
> +The manifests may be created on the registry independently of the image builds using the `push_manifests` target in place of the `push` target.
> +In contrast, the `push_images` target will only push the images and not create the manifests.
> +
> +If the `DPDK_CI_CONTAINERS_IS_BUILDER` variable is set to 'Y', then the `push_manifests` target will be disabled.
> \ No newline at end of file
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v5 01/11] containers/docs: Add container builder start
2023-05-23 17:04 ` [PATCH v5 01/11] containers/docs: Add container builder start Adam Hassick
@ 2023-05-25 13:48 ` Aaron Conole
0 siblings, 0 replies; 15+ messages in thread
From: Aaron Conole @ 2023-05-25 13:48 UTC (permalink / raw)
To: Adam Hassick; +Cc: ci, Owen Hilyard, alialnu
Adam Hassick <ahassick@iol.unh.edu> writes:
> From: Owen Hilyard <ohilyard@iol.unh.edu>
>
> * Add README file for containers
> * Add pyproject file with required dependencies for building containers
>
> This module allows anyone to build the containers used in DPDK CI, and
> allows the community to contribute container definitions back to DPDK
> CI. Please read the README for more information, since some
> functionality is opt-in due to resource requirements.
>
> Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
> ---
I don't like the structure of this series.
For example, you introduce README.md in this patch, and then later on
rewrite sections of it. Please don't do that. There is no reason to
introduce wrong information in one patch and remove it in another.
In general, make an introduced change to be the complete change. The
exception is when you are introducing features which modify existing
functionality, and therefore need to split the final form of a change
over a few discreet steps.
> containers/README.md | 138 ++++++++++++++++++++++
> containers/template_engine/pyproject.toml | 21 ++++
> 2 files changed, 159 insertions(+)
> create mode 100644 containers/README.md
> create mode 100644 containers/template_engine/pyproject.toml
>
> diff --git a/containers/README.md b/containers/README.md
> new file mode 100644
> index 0000000..5d01caf
> --- /dev/null
> +++ b/containers/README.md
> @@ -0,0 +1,138 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2022 University of New Hampshire
> +
> +DPDK CI Container Build System
> +==============================
> +
> +## Rational
> +
> +There are a few important factors for why a custom build system was created for
> +the containers instead of using an existing one. The first was that podman was
> +nearly mandatory for this task.
> +
> +### Why Podman
> +
> +1. Licensed RHEL containers need podman
> +
> +The build system MUST be able to handle creating properly licensed RHEL
> +containers, so that the RHEL CI testing is as accurate as possible.
> +
> +2. "Developer Laptop Friendliness"
> +
> +Another goal of the build system was to enable anyone to easily build the
> +containers. Not all developers are able to use Linux as the main OS on their
> +main development machine. Podman runs on MacOS via podman-machine and Windows
> +either by podman-machine or WSL.
> +
> +3. OCI Containers
> +
> +OCI containers are more portable than some other container solutions. Much of
> +the progress on getting containers running on top of FreeBSD jails targets OCI
> +containers specifically. The tracking issue for this is
> +[https://reviews.freebsd.org/D21570](https://reviews.freebsd.org/D21570).
> +Once upstream support happens, there should be a relatively simple path to
> +supporting containers in FreeBSD once podman/docker APIs are better supported.
> +At the moment, lack up upstream support means no support in this project for
> +FreeBSD.
> +
> +### Python and Makefiles instead of Buildah as a library
> +
> +The next question someone might have is why a combination of Python and
> +makefiles were used instead of using buildah as a library. The largest
> +reason is that every DPDK developer is going to need to have some
> +level of familiarity with Python due to DTS. Buildah is only available
> +as a library via Go, and would tie DPDK to a particular container
> +implementation. Go, while not difficult to learn, is a compiled language,
> +meaning that the build system would require a build system.
> +
> +The other reason is that most of the logic that needs to be performed is very
> +simple, and python has a few libraries that do most of the work. If it weren't
> +for the desire to have an inventory file (inventory.yaml) with a schema
> +(inventory_schema.json), this probably could have been an AWK script. After the
> +container images are produced, it is very easy to use the same template
> +engine to produce a makefile that can be used to both build and push the
> +containers. This makefile can be run with multiple jobs for parallel building
> +of containers, something not supported by all compose implementations.
> +
> +Meson was considered instead of Makefiles, however, Meson does not handle new
> +Meson being generated during the build very well, and Meson wants most commands
> +to have an output file, which is not true of many of the commands. Meson is
> +also more difficult to generate using a templating library than Makefile
> +targets.
> +
> +## Building
> +
> +### Environment Variables
> +
> +All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any
> +issues.
> +
> +| Variable | Description | Default | Valid Values |
> +| -------------------------- | ----------------------------------------------- | ------- | ------------ |
> +DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N'
> +DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE | Fail during dockerfile generation if any container in the inventory is not buildable. Currently will cause a failure if you are not on RHEL and try to build RHEL containers. | 'N' | 'Y' or 'N'
> +DPDK_CI_CONTAINERS_BUILD_ABI | Whether to bake ABI images into the containers. | 'N' | 'Y' or 'N'
> +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL | Whether to build libabigail from source on distros that do not package it. DPDK_CI_CONTAINERS_BUILD_ABI=Y overrides this to 'Y' | 'N' | 'Y' or 'N'
> +DPDK_CI_CONTAINER_BUILDER_PROGRAM | What container builder program to use. | 'podman' | Any container builder that exposes the same interface and provides the same behavior as podman.
> +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository.
> +DPDK_CI_CONTAINERS_DPDK_CLONE_URL | What URL to clone DPDK from. | 'https://dpdk.org/git/dpdk' | Any DPDK mirror.
> +DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG | What tag to give to the container which creates the dockerfiles. The default should be fine unless you have issues with collisions. | 'dpdk_ci_container_builder' | Any valid OCI container tag (A valid C function name will work)
> +DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS | Extra arguments to add to the push command, can be used for credentials if 'podman login' won't work. | '' | [https://docs.podman.io/en/latest/markdown/podman-push.1.html#options](https://docs.podman.io/en/latest/markdown/podman-push.1.html#options)
> +DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME | The hostname of the registry to push to. | 'localhost' | The hostname of any system exposing an OCI container registry or localhost to push to local storage.
> +DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH | The path to a directory to copy into all of the containers at /scripts | unset | The path to any local file directory.
> +DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers in. All generated files will be placed in this directory or one of it's children | '$(CURDIR)/container_context' | Any absolute directory path
> +
> +### Builder System Requirements
> +
> +#### Required Programs
> +
> +* GNU make (POSIX make may work, but is not supported)
> +* git
> +* find
> +* posix utilities (GNU coreutils will work)
> +* bash
> +* podman >= 4.0.0 (docker or other container builder programs may work, but are
> +unsupported)
> + * podman 4.0.0 allows run mounts, which allow mounting a directory into the build context of a container. This is used to persist ccache directories for each container.
> +* qemu-$ARCH-static for any non-native architecture/revision you want to build for.
> +
> +#### Hardware
> +
> +| Hardware Type | Requirement | Reason |
> +| ------------- | ---------------------------- | ----------------------------------- |
> +| Disk space | 5 GB of disk space per image | Some images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references.
> +Memory | Either 1.5x or 2x the memory needed to compile DPDK per makefile job | 1.5x is enough for the container overhead and caching when compiling natively, 2x is for builds under emulation (ARM container on x86, etc).
> +
> +
> +#### RHEL containers
> +
> +RHEL container images must be built on RHEL.
> +
> +### Build containers locally
> +
> +```bash
> +make build
> +```
> +
> +### Push containers to registry
> +
> +This will probably involve following prompts in your terminal, but if you have
> +other authentication set up, (LDAP, Kerberos, etc), it may not prompt you.
> +Logging into a registry is what allows you to upload containers to a remote
> +system for others to pull down.
> +
> +If you are working alone, you probably can ignore this and keep the containers
> +locally. If you are in an enterprise setting, ask your DevOps or Systems
> +Administration team where the preferred location for hosting containers is.
> +
> +Since these images take so long to build, it is recommended to use a container
> +registry and have any CI systems pull from that registry.
> +
> +Redhat guide to setting up a podman container registry:
> +[https://www.redhat.com/sysadmin/simple-container-registry](https://www.redhat.com/sysadmin/simple-container-registry)
> +
> +```bash
> +$DPDK_CI_CONTAINER_BUILDER_PROGRAM login $DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME
> +# < Complete login process >
> +make push
> +```
> \ No newline at end of file
> diff --git a/containers/template_engine/pyproject.toml b/containers/template_engine/pyproject.toml
> new file mode 100644
> index 0000000..f5611ce
> --- /dev/null
> +++ b/containers/template_engine/pyproject.toml
> @@ -0,0 +1,21 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2022 University of New Hampshire
> +
> +[tool.poetry]
> +name = "dpdk_ci_containers"
> +version = "0.1.0"
> +description = ""
> +authors = ["Owen Hilyard <ohilyard@iol.unh.edu>"]
> +license = "BSD-3-Clause"
> +
> +[tool.poetry.dependencies]
> +python = "^3.8"
> +Jinja2 = "^3.1.2"
> +jsonschema = "^4.10.0"
> +PyYAML = "^6.0"
> +
> +[tool.poetry.dev-dependencies]
> +
> +[build-system]
> +requires = ["poetry-core>=1.0.0"]
> +build-backend = "poetry.core.masonry.api"
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v5 11/11] containers/docs: Update README
2023-05-25 13:45 ` Aaron Conole
@ 2023-06-05 20:09 ` Adam Hassick
0 siblings, 0 replies; 15+ messages in thread
From: Adam Hassick @ 2023-06-05 20:09 UTC (permalink / raw)
To: Aaron Conole; +Cc: ci, alialnu
[-- Attachment #1: Type: text/plain, Size: 16768 bytes --]
Those changes were not intentional. I mistakenly updated an internally kept
copy of this README that was a draft, and added to that.
I apologize for not proofreading the changes, lesson learned. I've used a
linter to check for trailing whitespaces and spelling errors in the v6
patch.
On Thu, May 25, 2023 at 9:45 AM Aaron Conole <aconole@redhat.com> wrote:
> Adam Hassick <ahassick@iol.unh.edu> writes:
>
> > Updates the README with information on the added arguments
> > and support for building OCI manifests.
> >
> > Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
> > ---
>
> I stopped reviewing. The changes here introduce spelling mistakes and
> spaces at the end of lines. Please proofread it before making changes.
>
> > containers/README.md | 123 +++++++++++++++++++++++++++----------------
> > 1 file changed, 77 insertions(+), 46 deletions(-)
> >
> > diff --git a/containers/README.md b/containers/README.md
> > index 5d01caf..59edfcc 100644
> > --- a/containers/README.md
> > +++ b/containers/README.md
> > @@ -14,73 +14,80 @@ nearly mandatory for this task.
> >
> > 1. Licensed RHEL containers need podman
> >
> > -The build system MUST be able to handle creating properly licensed RHEL
> > -containers, so that the RHEL CI testing is as accurate as possible.
> > +The build system MUST be able to handle creating properly licensed RHEL
> > +containers, so that the RHEL CI testing is as accurate as possbile.
>
> Why these lines changed? Rather, I see that a space was added at the
> end, and a spelling mistake was introduced s/possbile/possible/. That
> can't be intentional - please undo this change.
>
> > 2. "Developer Laptop Friendliness"
> >
> > -Another goal of the build system was to enable anyone to easily build
> the
> > +Another goal of the build system was to enable anyone to easily build
> the
> > containers. Not all developers are able to use Linux as the main OS on
> their
> > main development machine. Podman runs on MacOS via podman-machine and
> Windows
> > -either by podman-machine or WSL.
> > +either by podman-machine or WSL.
>
> Why were spaces added at the end in this section? Please don't do that.
>
> > 3. OCI Containers
> >
> > -OCI containers are more portable than some other container solutions.
> Much of
> > +OCI containers are more portable than some other container solutions.
> Much of
> > the progress on getting containers running on top of FreeBSD jails
> targets OCI
> > -containers specifically. The tracking issue for this is
> > +containers specifically. The tracking issue for this is
> > [
> https://reviews.freebsd.org/D21570](https://reviews.freebsd.org/D21570).
> > -Once upstream support happens, there should be a relatively simple path
> to
> > +Once upstream support happens, there should be a relatively simple path
> to
> > supporting containers in FreeBSD once podman/docker APIs are better
> supported.
> > -At the moment, lack up upstream support means no support in this
> project for
> > -FreeBSD.
> > +At the moment, lack up upstream support means no support in this
> project for
> > +FreeBSD.
> >
> > ### Python and Makefiles instead of Buildah as a library
> >
> > -The next question someone might have is why a combination of Python and
> > +The next question someone might have is why a combination of Python and
> > makefiles were used instead of using buildah as a library. The largest
> > reason is that every DPDK developer is going to need to have some
> > -level of familiarity with Python due to DTS. Buildah is only available
> > -as a library via Go, and would tie DPDK to a particular container
> > +level of familarity with Python due to DTS. Buildah is only avaliable
>
> familiarity was correct. available was correct.
>
> > +as a library via Go, and would tie DPDK to a particular container
> > implementation. Go, while not difficult to learn, is a compiled
> language,
> > meaning that the build system would require a build system.
> >
> > -The other reason is that most of the logic that needs to be performed
> is very
> > -simple, and python has a few libraries that do most of the work. If it
> weren't
> > -for the desire to have an inventory file (inventory.yaml) with a schema
> > -(inventory_schema.json), this probably could have been an AWK script.
> After the
> > -container images are produced, it is very easy to use the same template
> > -engine to produce a makefile that can be used to both build and push the
> > -containers. This makefile can be run with multiple jobs for parallel
> building
> > -of containers, something not supported by all compose implementations.
> > -
> > -Meson was considered instead of Makefiles, however, Meson does not
> handle new
> > -Meson being generated during the build very well, and Meson wants most
> commands
> > -to have an output file, which is not true of many of the commands.
> Meson is
> > -also more difficult to generate using a templating library than Makefile
> > -targets.
> > +The other reason is that most of the logic that needs to be performed
> is very
> > +simple, and python has a few libraries that do most of the work. If it
> weren't
> > +for the desire to have an inventory file (inventory.yaml) with a schema
> > +(inventory_schema.json), this probably could have been an AWK script.
> After the
> > +container images are produced, it is very easy to use the same
> templating
> > +engine to produce a makefile that can be used to both build and push
> the
> > +containers. This makefile can be run with multiple jobs for parallel
> building
> > +of containers, something not supported by all compose implementations.
> > +
> > +Meson was considered instead of Makefiles, however, Meson does not
> handle new
> > +Meson being generated during the build very well, and Meson wants most
> commands
> > +to have an output file, which is not true of many of the commands.
> Meson is
> > +also more difficult to generate using a templating library than
> Makefile
> > +targets.
> >
> > ## Building
> >
> > ### Environment Variables
> >
> > -All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid
> any
> > -issues.
> > +All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid
> any
> > +issues.
> >
> > | Variable | Description
> | Default | Valid Values |
> > | -------------------------- |
> ----------------------------------------------- | ------- | ------------ |
> > -DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL.
> RHEL containers must be built on licensed RHEL, this can be used to
> forcibly enable/disable RHEL containers if automatic detection fails. |
> (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\|
> echo 'N' | 'Y' or 'N'
> > +| DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed
> RHEL. RHEL containers must be built on licensed RHEL, this can be used to
> forcibly enable/disable RHEL containers if automatic detection fails. |
> (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\|
> echo 'N' | 'Y' or 'N'
> > DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE | Fail during dockerfile
> generation if any container in the inventory is not buildable. Currently
> will cause a failure if you are not on RHEL and try to build RHEL
> containers. | 'N' | 'Y' or 'N'
> > +DPDK_CI_CONTAINERS_ONLY_HOST_ARCH | If set to 'Y', only images for the
> local system architecture will be built. | 'N' | 'Y' or 'N'
> > +DPDK_CI_CONTAINERS_IS_BUILDER | If set to 'Y', disables the manifest
> features, and only builds images for the local system architecture.
> Intended to be set when used inside another orchestration sofware. | 'N' |
> 'Y' or 'N'
> > +DPDK_CI_CONTAINERS_NINJA_WORKERS | The number of Ninja workers to use
> to build ABI images. Variable setting is benign if ABI is disabled. | unset
> | Any positive integer greater than zero.
> > DPDK_CI_CONTAINERS_BUILD_ABI | Whether to bake ABI images into the
> containers. | 'N' | 'Y' or 'N'
> > -DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL | Whether to build libabigail from
> source on distros that do not package it. DPDK_CI_CONTAINERS_BUILD_ABI=Y
> overrides this to 'Y' | 'N' | 'Y' or 'N'
> > +DPDK_CI_CONTAINERS_NO_LATEST_TAG | Disables tagging the final manifests
> as "latest" in the local store and remote registry. | 'N' | 'Y' or 'N'
> > +DPDK_CI_CONTAINERS_COVERITY | Enable building Coverity images. Setting
> this flag will make the Coverity binaries required. | 'N' | 'Y' or 'N'
> > DPDK_CI_CONTAINER_BUILDER_PROGRAM | What container builder program to
> use. | 'podman' | Any container builder that exposes the same interface and
> provides the same behavior as podman.
> > -DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail
> from, since some distros need to compile it from source. | 'git://
> sourceware.org/git/libabigail.git' | A repository containing libabigail
> which shares history with the main repository.
> > +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail
> from, since some distros need to compile it from source. | 'git://
> sourceware.org/git/libabigail.git' | A repository containing libabigail
> which shares history with the main repository.
> > DPDK_CI_CONTAINERS_DPDK_CLONE_URL | What URL to clone DPDK from. | '
> https://dpdk.org/git/dpdk' | Any DPDK mirror.
> > +DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL | What URL to clone DPDK
> stable form. | http://dpdk.org/git/dpdk-stable | Any DPDK stable mirror.
> > DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG | What tag to give to the
> container which creates the dockerfiles. The default should be fine unless
> you have issues with collisions. | 'dpdk_ci_container_builder' | Any valid
> OCI container tag (A valid C function name will work)
> > DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS | Extra arguments to add to the push
> command, can be used for credentials if 'podman login' won't work. | '' | [
> https://docs.podman.io/en/latest/markdown/podman-push.1.html#options](https://docs.podman.io/en/latest/markdown/podman-push.1.html#options)
> > DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME | The hostname of the registry to
> push to. | 'localhost' | The hostname of any system exposing an OCI
> container registry or localhost to push to local storage.
> > DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH | The path to a directory to copy
> into all of the containers at /scripts | unset | The path to any local file
> directory.
> > +DPDK_CI_CONTAINERS_COVERITY_PATH | The path to Coverity Scan binaries.
> Only required of the Coverity flag is set. | unset | The path to any local
> file directory.
> > DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the
> containers in. All generated files will be placed in this directory or one
> of it's children | '$(CURDIR)/container_context' | Any absolute directory
> path
> > +DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE | Uses a provided string instead
> of generating a new date tag. Intended for development use. | unset | Any
> string that is a valid OCI manifest tag.
> >
> > ### Builder System Requirements
> >
> > @@ -91,16 +98,15 @@ DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the
> directory to build the containers
> > * find
> > * posix utilities (GNU coreutils will work)
> > * bash
> > -* podman >= 4.0.0 (docker or other container builder programs may work,
> but are
> > -unsupported)
> > - * podman 4.0.0 allows run mounts, which allow mounting a directory
> into the build context of a container. This is used to persist ccache
> directories for each container.
> > -* qemu-$ARCH-static for any non-native architecture/revision you want
> to build for.
> > +* podman (docker is unsupported, and will NOT work for making manifests)
> > +* qemu-$ARCH-static for any non-native architecture/revision you want
> to build
> > +for.
> >
> > #### Hardware
> >
> > -| Hardware Type | Requirement | Reason |
> > -| ------------- | ---------------------------- |
> ----------------------------------- |
> > -| Disk space | 5 GB of disk space per image | Some images are 4 GB
> at the moment, and as DPDK's API grows, so will the ABI references.
> > +| Hardware Type | Requirement | Reason |
> > +| ------------- | ----------------------------- |
> ----------------------------------- |
> > +| Disk space | 10 GB of disk space per image | Many of the final
> images are 4 GB at the moment, and as DPDK's API grows, so will the ABI
> references. Intermediate images generated by the builds will consume some
> additional space that is recoverable after the build.
> > Memory | Either 1.5x or 2x the memory needed to compile DPDK per
> makefile job | 1.5x is enough for the container overhead and caching when
> compiling natively, 2x is for builds under emulation (ARM container on x86,
> etc).
> >
> >
> > @@ -111,28 +117,53 @@ RHEL container images must be built on RHEL.
> > ### Build containers locally
> >
> > ```bash
> > +# Build using the default arguments
> > make build
> > ```
> >
> > +The resulting images will be tagged based on the date tag and platform.
> > +Image generated tags follow this format: `image-{{ platform }}-{{
> date_tag }}`
> > +Where `platform` denotes the platform of the image, and `date_tag` is
> the generated date tag or the override string provided
> > +through the environment variable.
> > +
> > +They should appear in the local image store on your system.
> > +
> > ### Push containers to registry
> >
> > -This will probably involve following prompts in your terminal, but if
> you have
> > -other authentication set up, (LDAP, Kerberos, etc), it may not prompt
> you.
> > +This will probably involve following prompts in your terminal, but if
> you have
> > +other authentication set up, (LDAP, Kerberos, etc), it may not prompt
> you.
> > Logging into a registry is what allows you to upload containers to a
> remote
> > -system for others to pull down.
> > +system for others to pull down.
> >
> > If you are working alone, you probably can ignore this and keep the
> containers
> > -locally. If you are in an enterprise setting, ask your DevOps or Systems
> > -Administration team where the preferred location for hosting containers
> is.
> > +locally. If you are in an enterprise setting, ask your DevOps or
> Systems
> > +Administration team where the preferred location for hosting containers
> is.
> >
> > -Since these images take so long to build, it is recommended to use a
> container
> > +Since these images take so long to build, it is recommended to use a
> container
> > registry and have any CI systems pull from that registry.
> >
> > -Redhat guide to setting up a podman container registry:
> > +Redhat guide to setting up a podman container registry:
> > [
> https://www.redhat.com/sysadmin/simple-container-registry](https://www.redhat.com/sysadmin/simple-container-registry)
> >
> > ```bash
> > $DPDK_CI_CONTAINER_BUILDER_PROGRAM login
> $DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME
> > # < Complete login process >
> > make push
> > -```
> > \ No newline at end of file
> > +```
> > +
> > +#### Manifests
> > +
> > +OCI manifests allow the grouping of images for different platforms
> under the same tag in a repository on a registry.
> > +The use of OCI manifests over tagged images reduces the amount of
> system platform related branching in CI scripting.
> > +
> > +The Makefile provides the option to push only the images, only the
> manifests, or push the images and make manifests.
> > +The default "push" target will perform the last case.
> > +If you choose to create the manifests, then these will be created with
> the "final" tags like "latest" and the date timestamp.
> > +
> > +Manifest creation is known to not be compatible with Docker.
> > +This feature is known to work when using Podman to post content to a
> Docker v2 registry.
> > +
> > +The manifests may be created on the registry independently of the image
> builds using the `push_manifests` target in place of the `push` target.
> > +In contrast, the `push_images` target will only push the images and not
> create the manifests.
> > +
> > +If the `DPDK_CI_CONTAINERS_IS_BUILDER` variable is set to 'Y', then the
> `push_manifests` target will be disabled.
> > \ No newline at end of file
>
>
--
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248
[-- Attachment #2: Type: text/html, Size: 20168 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2023-06-05 20:09 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-23 17:04 [PATCH v5 00/11] Community Lab Containers and Builder Engine Adam Hassick
2023-05-23 17:04 ` [PATCH v5 01/11] containers/docs: Add container builder start Adam Hassick
2023-05-25 13:48 ` Aaron Conole
2023-05-23 17:04 ` [PATCH v5 02/11] containers/inventory: Add inventory for container builder Adam Hassick
2023-05-23 17:04 ` [PATCH v5 03/11] containers/builder: Dockerfile creation script Adam Hassick
2023-05-23 17:04 ` [PATCH v5 04/11] containers/templates: Templates for Dockerfiles Adam Hassick
2023-05-23 17:04 ` [PATCH v5 05/11] containers/container_builder: Container for python scripts Adam Hassick
2023-05-23 17:04 ` [PATCH v5 06/11] containers/Makefile: Makefile to automate builds Adam Hassick
2023-05-23 17:04 ` [PATCH v5 07/11] containers/inventory: Add ABI rev and coverity attribute Adam Hassick
2023-05-23 17:04 ` [PATCH v5 08/11] containers/builder: Add arguments to templating script Adam Hassick
2023-05-23 17:04 ` [PATCH v5 09/11] containers/templates: Update templates Adam Hassick
2023-05-23 17:04 ` [PATCH v5 10/11] containers/Makefile: Add new variables and target changes Adam Hassick
2023-05-23 17:04 ` [PATCH v5 11/11] containers/docs: Update README Adam Hassick
2023-05-25 13:45 ` Aaron Conole
2023-06-05 20:09 ` Adam Hassick
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).