From: Adam Hassick <ahassick@iol.unh.edu>
To: ci@dpdk.org
Cc: alialnu@nvidia.com, aconole@redhat.com,
Adam Hassick <ahassick@iol.unh.edu>
Subject: [PATCH v7 2/6] containers/inventory: Add inventory for container builder
Date: Tue, 11 Jul 2023 16:21:20 -0400 [thread overview]
Message-ID: <20230711202124.1636317-3-ahassick@iol.unh.edu> (raw)
In-Reply-To: <20230711202124.1636317-1-ahassick@iol.unh.edu>
From: Owen Hilyard <ohilyard at iol.unh.edu>
Adds a yaml file used to define what containers should be built, what
packages the containers should have, what platforms to build for and how they
should be tagged.
Also adds a JSON schema, which can be used to validate that the
inventory file is well-formed.
Signed-off-by: Owen Hilyard <ohilyard at iol.unh.edu>
Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
---
containers/template_engine/inventory.yaml | 470 ++++++++++++++++++
.../template_engine/inventory_schema.json | 221 ++++++++
2 files changed, 691 insertions(+)
create mode 100644 containers/template_engine/inventory.yaml
create mode 100644 containers/template_engine/inventory_schema.json
diff --git a/containers/template_engine/inventory.yaml b/containers/template_engine/inventory.yaml
new file mode 100644
index 0000000..31dbdb6
--- /dev/null
+++ b/containers/template_engine/inventory.yaml
@@ -0,0 +1,470 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2022 University of New Hampshire
+---
+abi:
+ # versions should also include the abi versions for all stable ABIs.
+ versions:
+ # - dpdk_version: # LTS - 3
+ # major: 20
+ # minor: 11
+ # revision: 0
+ # git_tag: "20.11"
+ # abi_version:
+ # major: 21
+ # minor: 0
+ - dpdk_version: # LTS - 2
+ major: 21
+ minor: 11
+ revision: 0
+ git_tag: "21.11"
+ abi_version:
+ major: 22
+ minor: 0
+ - dpdk_version:
+ major: 22 # LTS - 1
+ minor: 11
+ revision: 1
+ git_tag: "tags/v22.11.1"
+ abi_version:
+ major: 23
+ minor: 10 # .xml comparison
+ - dpdk_version:
+ major: 22 # LTS
+ minor: 11
+ revision: 1
+ git_tag: "tags/v22.11.1"
+ abi_version:
+ major: 23
+ minor: 11 # .so comparison
+
+dockerfiles:
+ groups:
+ all:
+ package_manager_command: "NO_INSTALL_COMMAND_SET"
+ parent: "NONE"
+ packages:
+ - gcc
+ - make
+ - git
+ - sudo
+ - curl
+ - automake
+ - autoconf
+ - flex
+ - bison
+ - bc
+ rpm:
+ parent: "all"
+ package_manager_command: yum install -y
+ packages:
+ - diffutils
+ - pkg-config
+ - python3
+ - python3-pip
+ - librdmacm
+ - rdma-core-devel
+ - libmnl-devel
+ - ccache
+ - zip
+ - autoconf
+ - libbpf-devel
+ redhat_family:
+ parent: "rpm"
+ packages: []
+ fedora:
+ parent: "redhat_family"
+ packages:
+ - python3-pyelftools
+ - zip
+ - clang
+ - ninja-build
+ - elfutils-libelf-devel
+ - jansson-devel
+ - libatomic
+ - libpcap-devel
+ - numactl-devel
+ - findutils
+ - libtool
+ - elfutils-devel
+ - libxml2-devel
+ - libabigail
+ - openssl-devel
+ fedora_coverity:
+ parent: fedora
+ packages: []
+ fedora_clang:
+ parent: fedora
+ packages: []
+ centos8:
+ parent: "redhat_family"
+ packages:
+ - elfutils-libelf-devel
+ - jansson-devel
+ - libatomic
+ - libpcap-devel
+ - numactl-devel
+ - libabigail
+ - openssl-devel
+ centos9:
+ parent: "redhat_family"
+ package_manager_command: "yum install --allowerasing -y"
+ packages:
+ - elfutils-libelf-devel
+ - jansson-devel
+ - libatomic
+ - libpcap-devel
+ - numactl-devel
+ - libtool
+ - g++
+ - elfutils-devel
+ - libxml2-devel
+ - openssl-devel
+ rhel:
+ parent: "redhat_family"
+ package_manager_command: "yum install -y"
+ packages:
+ - elfutils-libelf-devel
+ - jansson-devel
+ - libatomic
+ - libpcap-devel
+ - numactl-devel
+ - openssl-devel
+ - libabigail
+ rhel7:
+ parent: "rhel"
+ packages:
+ - ninja-build
+ rhel8:
+ parent: "rhel"
+ package_manager_command: "yum --allowerasing install -y"
+ packages:
+ - ninja-build
+ - libxml2-devel
+ rhel9:
+ parent: "rhel"
+ package_manager_command: "yum --allowerasing install -y"
+ packages:
+ - libxml2-devel
+ opensuse:
+ parent: "rpm"
+ package_manager_command: "zypper install -y"
+ packages:
+ - python3-pyelftools
+ - ninja
+ - gzip
+ - libelf-devel
+ - libjansson-devel
+ - librdmacm1
+ - libatomic1
+ - libnuma-devel
+ - libabigail-tools
+ - tar
+ - openssl-devel
+ alpine:
+ parent: "all"
+ package_manager_command: "apk add"
+ packages:
+ - bsd-compat-headers
+ - ccache
+ - jansson-dev
+ - libarchive-dev
+ - libbpf-dev
+ - libc-dev
+ - libpcap-dev
+ - linux-headers
+ - ninja
+ - numactl-dev
+ - openssl-dev
+ - py3-pip
+ - py3-elftools
+ - py3-setuptools
+ - py3-wheel
+ - zlib-dev
+ - python3
+ - bash
+ - zip
+ - meson
+ arch:
+ parent: "all"
+ package_manager_command: "pacman -Syu --noconfirm"
+ packages:
+ - glibc
+ - cmake
+ - ninja
+ - pandoc
+ - python
+ - python-docutils
+ - ethtool
+ - file
+ - fakeroot
+ - pkg-config
+ - numactl
+ - elfutils
+ - gawk
+ - diffutils
+ - python-pip
+ - libabigail
+ - openssl
+ - libbpf
+ debian:
+ parent: "all"
+ package_manager_command: "apt-get update && apt-get install --no-install-recommends -y"
+ packages:
+ - libnuma-dev
+ - zip
+ - librdmacm1
+ - libcrypto++-dev
+ - python3-pyelftools
+ - libxml2-dev
+ - libelf-dev
+ - libpcap-dev
+ - libjansson-dev
+ - python3-setuptools
+ - autoconf
+ - pkg-config
+ - libtool
+ - ccache
+ - libatomic1
+ - python3-wheel
+ - libdw-dev
+ - rdma-core
+ - ninja-build
+ - automake
+ - librdmacm-dev
+ - libmnl-dev
+ - python3-pip
+ - build-essential
+ - aspell-en
+ - parallel
+ - python3-grpcio
+ - lcov
+ - libbpf-dev
+ - libssl-dev
+ debian10:
+ parent: "debian"
+ packages: []
+ debian11:
+ parent: "debian"
+ packages:
+ - nasm
+ debian11_arm_ipsec:
+ parent: "debian11"
+ packages: []
+ ubuntu:
+ parent: 'debian'
+ packages: []
+ ubuntu18.04:
+ parent: "ubuntu"
+ packages: []
+ ubuntu20.04:
+ parent: "ubuntu"
+ packages:
+ - python3-grpcio
+ - lcov
+ - jq
+ - default-jre
+ - nodejs
+ ubuntu22.04:
+ parent: "ubuntu"
+ packages:
+ - python3-grpcio
+ - lcov
+ - abigail-tools
+ - jq
+ ubuntu_cross: # x86 -> ARM
+ parent: "ubuntu20.04"
+ packages:
+ - clang
+ - llvm
+ - llvm-dev
+ - llvm-runtime
+ - lld
+ - gcc-aarch64-linux-gnu
+ - libgcc-9-dev-arm64-cross
+ - libatomic1-arm64-cross
+ - libc6-dev-arm64-cross
+ - pkg-config-aarch64-linux-gnu
+ - dpkg-dev
+ - gcc-arm-linux-gnueabihf
+ - libc6-dev-armhf-cross
+ - pkg-config-arm-linux-gnueabihf
+ ubuntu_sve:
+ parent: "ubuntu20.04"
+ packages:
+ - gcc-10
+ targets:
+ # Alpine
+ - name: alpine_compile
+ group: "alpine"
+ base_image: alpine
+ platforms: [linux/amd64, linux/arm64]
+ force_disable_abi: true
+ minimum_dpdk_version:
+ major: 22
+ minor: 0
+ revision: 0
+ extra_information:
+ amd64_environment_id: 45
+ arm64_environment_id: 75
+
+ # Arch
+ - name: arch_compile
+ group: "arch"
+ base_image: archlinux:base
+ platforms: [linux/amd64]
+ extra_information:
+ amd64_environment_id: 26
+
+ - name: fedora37_compile
+ group: "fedora"
+ base_image: fedora:37
+ platforms: [linux/amd64, linux/arm64]
+ extra_information:
+ amd64_environment_id: 69
+ arm64_environment_id: 70
+
+ - name: fedora38_compile
+ group: "fedora"
+ base_image: fedora:38
+ platforms: [linux/amd64, linux/arm64]
+ extra_information:
+ amd64_environment_id: 71
+ arm64_environment_id: 72
+
+ - name: fedora38_clang
+ group: "fedora_clang"
+ base_image: fedora:38
+ platforms: [linux/amd64, linux/arm64]
+ extra_information:
+ amd64_environment_id: 73
+ arm64_environment_id: 74
+
+ - name: fedora37_coverity
+ group: "fedora_coverity"
+ base_image: fedora:37
+ force_disable_abi: true
+ requires_coverity: true
+ platforms: [linux/amd64]
+
+ # CentOs
+ - name: centosstream8_compile
+ group: "centos8"
+ base_image: quay.io/centos/centos:stream8
+ platforms: [linux/amd64]
+ extra_information:
+ amd64_environment_id: 39
+
+ - name: centosstream9_compile
+ group: "centos9"
+ base_image: quay.io/centos/centos:stream9
+ platforms: [linux/amd64, linux/arm64]
+ extra_information:
+ amd64_environment_id: 53
+ arm64_environment_id: 63
+
+ # RHEL
+ - name: rhel7_compile
+ group: "rhel7"
+ base_image: registry.access.redhat.com/ubi7/ubi
+ platforms: [linux/amd64]
+ extra_information:
+ amd64_environment_id: 36
+
+ - name: rhel8_compile
+ group: "rhel8"
+ base_image: registry.access.redhat.com/ubi8/ubi
+ platforms: [linux/amd64]
+ extra_information:
+ amd64_environment_id: 37
+
+ - name: rhel9_compile
+ group: "rhel9"
+ base_image: registry.access.redhat.com/ubi9/ubi
+ platforms: [linux/amd64, linux/arm64]
+ extra_information:
+ amd64_environment_id: 76
+ arm64_environment_id: 77
+
+ # OpenSuse
+ - name: opensuse-leap15_compile
+ group: "opensuse"
+ base_image: opensuse/leap:15
+ platforms: [linux/amd64, linux/arm64]
+ extra_information:
+ amd64_environment_id: 25
+ arm64_environment_id: 67
+
+ # Debian
+ - name: debian10_compile # debian 10
+ group: "debian10"
+ base_image: "debian:buster"
+ libabigail_revision: libabigail-2.2
+ platforms: [linux/amd64]
+ extra_information:
+ amd64_environment_id: 52
+
+ - name: debian11_compile # debian 11
+ group: "debian11"
+ base_image: "debian:bullseye"
+ libabigail_revision: libabigail-2.2
+ platforms:
+ - linux/amd64
+ - linux/arm64
+ extra_information:
+ amd64_environment_id: 51
+ arm64_environment_id: 64
+
+ # Ubuntu
+ - name: ubuntu20.04_compile
+ group: "ubuntu20.04"
+ base_image: "ubuntu:20.04"
+ platforms: [linux/amd64, linux/arm64]
+ extra_tags:
+ - $R/ubuntu20.04_libabigail:latest
+ - $R/ubuntu20.04_libabigail:$T
+ - $R/ubuntu20.04_spell-check:latest
+ - $R/ubuntu20.04_spell-check:$T
+ - $R/ubuntu20.04_sonarscanner:latest
+ - $R/ubuntu20.04_sonarscanner:$T
+ extra_information:
+ amd64_environment_id: 38
+ arm64_environment_id: 62
+
+ # x86 -> ARM cross compile
+ - name: ubuntu20.04_cross
+ group: "ubuntu_cross"
+ base_image: "ubuntu:20.04"
+ platforms: [linux/amd64]
+ cross_file: "config/arm/arm64_armv8_linux_gcc"
+ extra_information:
+ amd64_environment_id: 47
+
+ # Also, x86 -> ARM cross compile (for SVE test)
+ - name: ubuntu20.04_sve_compile
+ group: "ubuntu_sve"
+ base_image: "arm64v8/ubuntu:20.04"
+ platforms: [linux/arm64]
+ extra_information:
+ arm64_environment_id: 46
+
+ # TEMPORARY: This has the forked IPsec lib developed by ARM. This causes issues on older LTS versions of DPDK.
+ # Once all supported and tested LTS versions no longer break when this lib is installed, recombine with debian11_compile.
+ - name: debian11_arm_ipsec
+ group: "debian11_arm_ipsec"
+ base_image: "debian:bullseye"
+ platforms: [linux/arm64]
+ force_disable_abi: true
+ extra_tags:
+ - $R/debian11_zuc_compile:latest
+ - $R/debian11_snow3g_compile:latest
+ - $R/debian11_zuc_compile:$T
+ - $R/debian11_snow3g_compile:$T
+ extra_information:
+ arm64_environment_id: 58
+
+ - name: ubuntu22.04_compile
+ group: "ubuntu22.04"
+ base_image: "ubuntu:22.04"
+ platforms: [linux/amd64]
+ extra_information:
+ amd64_environment_id: 54
diff --git a/containers/template_engine/inventory_schema.json b/containers/template_engine/inventory_schema.json
new file mode 100644
index 0000000..7b79e44
--- /dev/null
+++ b/containers/template_engine/inventory_schema.json
@@ -0,0 +1,221 @@
+{
+ "$schema": "https://json-schema.org/draft-07/schema",
+ "title": "dpdklab-ci container inventory schema",
+ "$comment": "SPDX-License-Identifier: BSD-3-Clause\nCopyright (c) 2022 University of New Hampshire",
+ "type": "object",
+ "properties": {
+ "abi": {
+ "type": "object",
+ "properties": {
+ "versions": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "dpdk_version": {
+ "type": "object",
+ "properties": {
+ "major": {
+ "type": "integer",
+ "description": "The major version number (usually the year of release)"
+ },
+ "minor": {
+ "type": "integer",
+ "description": "The minor version number (usually the month of release)"
+ },
+ "revision": {
+ "type": "integer",
+ "description": "The revision version number. Starts at 0."
+ },
+ "git_tag": {
+ "type": "string",
+ "description": "The git tag to check out to get this ABI version"
+ }
+ },
+ "required": [
+ "major",
+ "minor",
+ "revision",
+ "git_tag"
+ ],
+ "additionalProperties": false
+ },
+ "abi_version": {
+ "type": "object",
+ "properties": {
+ "major": {
+ "type": "integer",
+ "description": "The major version number (usually the year of release)"
+ },
+ "minor": {
+ "type": "integer",
+ "description": "The minor version number (usually the month of release)"
+ }
+ },
+ "required": [
+ "major",
+ "minor"
+ ],
+ "additionalProperties": false
+ }
+ },
+ "required": [
+ "dpdk_version",
+ "abi_version"
+ ]
+ },
+ "minItems": 0
+ }
+ },
+ "required": [
+ "versions"
+ ],
+ "additionalProperties": false
+ },
+ "dockerfiles": {
+ "type": "object",
+ "properties": {
+ "groups": {
+ "type": "object",
+ "patternProperties": {
+ "^[a-z][a-z_0-9\\.]+$": {
+ "type": "object",
+ "properties": {
+ "package_manager_command": {
+ "type": "string",
+ "description": "The command to install packages without user input."
+ },
+ "parent": {
+ "type": "string",
+ "description": "Which group to inherit from, or NONE if no parent exists"
+ },
+ "packages": {
+ "type": "array",
+ "description": "A list of package names to install",
+ "items": {
+ "type": "string"
+ },
+ "uniqueItems": true
+ }
+ },
+ "required": [
+ "parent",
+ "packages"
+ ],
+ "additionalProperties": false
+ }
+ },
+ "minProperties": 1
+ },
+ "targets": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "The name of the docker container"
+ },
+ "group": {
+ "type": "string",
+ "description": "Which group the target is a member of"
+ },
+ "base_image": {
+ "type": "string",
+ "description": "The base docker image."
+ },
+ "platforms": {
+ "type": "array",
+ "description": "What platforms to build for",
+ "items": {
+ "type": "string",
+ "enum": [
+ "linux/amd64",
+ "linux/arm64",
+ "linux/riscv64",
+ "linux/ppc64le",
+ "linux/arm/v7",
+ "linux/arm/v6"
+ ]
+ },
+ "minItems": 1
+ },
+ "cross_file": {
+ "type": "string",
+ "description": "A cross file to compile meson with. If not present, meson's automatic behavior will be used."
+ },
+ "libabigail_revision": {
+ "type": "string",
+ "description": "A revision or branch of libabigail to check out before building for a given target."
+ },
+ "force_disable_abi": {
+ "type": "boolean",
+ "description": "Whether to forcibly disable ABI for this target. This should be used as a temporary measure until a fix can be created.",
+ "default": "false"
+ },
+ "requires_coverity": {
+ "type": "boolean",
+ "description": "Whether this target requires the Coverity Scan binaries.",
+ "default": "false"
+ },
+ "minimum_dpdk_version": {
+ "type": "object",
+ "properties": {
+ "major": {
+ "type": "integer",
+ "description": "The major version number (usually the year of release)"
+ },
+ "minor": {
+ "type": "integer",
+ "description": "The minor version number (usually the month of release)"
+ },
+ "revision": {
+ "type": "integer",
+ "description": "The revision version number. Starts at 0."
+ }
+ },
+ "required": [
+ "major",
+ "minor",
+ "revision"
+ ],
+ "additionalProperties": false
+ },
+ "extra_tags": {
+ "type": "array",
+ "description": "Tags to apply to this image",
+ "items": {
+ "type": "string",
+ "description": "$R will be replaced with the 'registry_hostname' variable in this file. $T will be replaced with a timestamp in the form of '%Y-%m-%d'. $N will be replaced by the name of the target.. The tags '$R/$N:latest' and '$R/$N:$T' will always be present"
+ }
+ },
+ "extra_information": {
+ "type": "object",
+ "description": "Extra information to be stored as a JSON file at /container_info.json",
+ "additionalProperties": true
+ }
+ },
+ "required": [
+ "name",
+ "group",
+ "base_image",
+ "platforms"
+ ],
+ "additionalProperties": false
+ },
+ "minItems": 1
+ }
+ },
+ "required": [
+ "groups",
+ "targets"
+ ],
+ "additionalProperties": false
+ }
+ },
+ "required": [
+ "abi",
+ "dockerfiles"
+ ],
+ "additionalProperties": false
+}
\ No newline at end of file
--
2.34.1
next prev parent reply other threads:[~2023-07-11 20:22 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-11 20:21 [PATCH v7 0/6] Community Lab Containers and Builder Engine Adam Hassick
2023-07-11 20:21 ` [PATCH v7 1/6] containers/docs: Add container builder start Adam Hassick
2023-07-12 11:43 ` Ali Alnubani
2023-07-11 20:21 ` Adam Hassick [this message]
2023-07-11 20:21 ` [PATCH v7 3/6] containers/builder: Dockerfile creation script Adam Hassick
2023-07-11 20:21 ` [PATCH v7 4/6] containers/templates: Templates for Dockerfiles Adam Hassick
2023-07-11 20:21 ` [PATCH v7 5/6] containers/container_builder: Container for python scripts Adam Hassick
2023-07-11 20:21 ` [PATCH v7 6/6] containers/Makefile: Makefile to automate builds Adam Hassick
2023-07-12 11:43 ` Ali Alnubani
2023-07-12 11:45 ` [PATCH v7 0/6] Community Lab Containers and Builder Engine Ali Alnubani
2023-07-17 19:07 ` Aaron Conole
2023-07-18 13:40 ` Adam Hassick
2023-07-18 14:17 ` Aaron Conole
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230711202124.1636317-3-ahassick@iol.unh.edu \
--to=ahassick@iol.unh.edu \
--cc=aconole@redhat.com \
--cc=alialnu@nvidia.com \
--cc=ci@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).