From: Aaron Conole <aconole@redhat.com>
To: Adam Hassick <ahassick@iol.unh.edu>
Cc: ci@dpdk.org, alialnu@nvidia.com, Owen Hilyard <ohilyard@iol.unh.edu>
Subject: Re: [PATCH v8 2/6] containers/inventory: Add inventory for container builder
Date: Fri, 04 Aug 2023 09:58:10 -0400 [thread overview]
Message-ID: <f7ty1iq6hal.fsf@redhat.com> (raw)
In-Reply-To: <20230717210815.29737-3-ahassick@iol.unh.edu> (Adam Hassick's message of "Mon, 17 Jul 2023 17:08:11 -0400")
Adam Hassick <ahassick@iol.unh.edu> writes:
> From: Owen Hilyard <ohilyard@iol.unh.edu>
>
> Adds a yaml file used to define what containers should be built, what
> packages the containers should have, what platforms to build for and how they
> should be tagged.
>
> Also adds a JSON schema, which can be used to validate that the
> inventory file is well-formed.
>
> Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
> Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
> ---
> containers/template_engine/inventory.yaml | 470 ++++++++++++++++++
> .../template_engine/inventory_schema.json | 221 ++++++++
> 2 files changed, 691 insertions(+)
> create mode 100644 containers/template_engine/inventory.yaml
> create mode 100644 containers/template_engine/inventory_schema.json
>
> diff --git a/containers/template_engine/inventory.yaml b/containers/template_engine/inventory.yaml
> new file mode 100644
> index 0000000..31dbdb6
> --- /dev/null
> +++ b/containers/template_engine/inventory.yaml
> @@ -0,0 +1,470 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2022 University of New Hampshire
> +---
> +abi:
> + # versions should also include the abi versions for all stable ABIs.
> + versions:
> + # - dpdk_version: # LTS - 3
> + # major: 20
> + # minor: 11
> + # revision: 0
> + # git_tag: "20.11"
> + # abi_version:
> + # major: 21
> + # minor: 0
Why is this commented out?
> + - dpdk_version: # LTS - 2
> + major: 21
> + minor: 11
> + revision: 0
> + git_tag: "21.11"
> + abi_version:
> + major: 22
> + minor: 0
> + - dpdk_version:
> + major: 22 # LTS - 1
> + minor: 11
> + revision: 1
> + git_tag: "tags/v22.11.1"
> + abi_version:
> + major: 23
> + minor: 10 # .xml comparison
> + - dpdk_version:
> + major: 22 # LTS
> + minor: 11
> + revision: 1
> + git_tag: "tags/v22.11.1"
> + abi_version:
> + major: 23
> + minor: 11 # .so comparison
> +
> +dockerfiles:
> + groups:
> + all:
> + package_manager_command: "NO_INSTALL_COMMAND_SET"
> + parent: "NONE"
> + packages:
> + - gcc
> + - make
> + - git
> + - sudo
> + - curl
> + - automake
> + - autoconf
> + - flex
> + - bison
> + - bc
> + rpm:
> + parent: "all"
> + package_manager_command: yum install -y
> + packages:
> + - diffutils
> + - pkg-config
> + - python3
> + - python3-pip
> + - librdmacm
> + - rdma-core-devel
> + - libmnl-devel
> + - ccache
> + - zip
> + - autoconf
> + - libbpf-devel
> + redhat_family:
> + parent: "rpm"
> + packages: []
> + fedora:
> + parent: "redhat_family"
> + packages:
> + - python3-pyelftools
> + - zip
> + - clang
> + - ninja-build
> + - elfutils-libelf-devel
> + - jansson-devel
> + - libatomic
> + - libpcap-devel
> + - numactl-devel
> + - findutils
> + - libtool
> + - elfutils-devel
> + - libxml2-devel
> + - libabigail
> + - openssl-devel
> + fedora_coverity:
> + parent: fedora
> + packages: []
> + fedora_clang:
> + parent: fedora
> + packages: []
> + centos8:
> + parent: "redhat_family"
> + packages:
> + - elfutils-libelf-devel
> + - jansson-devel
> + - libatomic
> + - libpcap-devel
> + - numactl-devel
> + - libabigail
> + - openssl-devel
> + centos9:
> + parent: "redhat_family"
> + package_manager_command: "yum install --allowerasing -y"
> + packages:
> + - elfutils-libelf-devel
> + - jansson-devel
> + - libatomic
> + - libpcap-devel
> + - numactl-devel
> + - libtool
> + - g++
> + - elfutils-devel
> + - libxml2-devel
> + - openssl-devel
> + rhel:
> + parent: "redhat_family"
> + package_manager_command: "yum install -y"
> + packages:
> + - elfutils-libelf-devel
> + - jansson-devel
> + - libatomic
> + - libpcap-devel
> + - numactl-devel
> + - openssl-devel
> + - libabigail
> + rhel7:
> + parent: "rhel"
> + packages:
> + - ninja-build
> + rhel8:
> + parent: "rhel"
> + package_manager_command: "yum --allowerasing install -y"
> + packages:
> + - ninja-build
> + - libxml2-devel
> + rhel9:
> + parent: "rhel"
> + package_manager_command: "yum --allowerasing install -y"
> + packages:
> + - libxml2-devel
> + opensuse:
> + parent: "rpm"
> + package_manager_command: "zypper install -y"
> + packages:
> + - python3-pyelftools
> + - ninja
> + - gzip
> + - libelf-devel
> + - libjansson-devel
> + - librdmacm1
> + - libatomic1
> + - libnuma-devel
> + - libabigail-tools
> + - tar
> + - openssl-devel
> + alpine:
> + parent: "all"
> + package_manager_command: "apk add"
> + packages:
> + - bsd-compat-headers
> + - ccache
> + - jansson-dev
> + - libarchive-dev
> + - libbpf-dev
> + - libc-dev
> + - libpcap-dev
> + - linux-headers
> + - ninja
> + - numactl-dev
> + - openssl-dev
> + - py3-pip
> + - py3-elftools
> + - py3-setuptools
> + - py3-wheel
> + - zlib-dev
> + - python3
> + - bash
> + - zip
> + - meson
> + arch:
> + parent: "all"
> + package_manager_command: "pacman -Syu --noconfirm"
> + packages:
> + - glibc
> + - cmake
> + - ninja
> + - pandoc
> + - python
> + - python-docutils
> + - ethtool
> + - file
> + - fakeroot
> + - pkg-config
> + - numactl
> + - elfutils
> + - gawk
> + - diffutils
> + - python-pip
> + - libabigail
> + - openssl
> + - libbpf
> + debian:
> + parent: "all"
> + package_manager_command: "apt-get update && apt-get install --no-install-recommends -y"
> + packages:
> + - libnuma-dev
> + - zip
> + - librdmacm1
> + - libcrypto++-dev
> + - python3-pyelftools
> + - libxml2-dev
> + - libelf-dev
> + - libpcap-dev
> + - libjansson-dev
> + - python3-setuptools
> + - autoconf
> + - pkg-config
> + - libtool
> + - ccache
> + - libatomic1
> + - python3-wheel
> + - libdw-dev
> + - rdma-core
> + - ninja-build
> + - automake
> + - librdmacm-dev
> + - libmnl-dev
> + - python3-pip
> + - build-essential
> + - aspell-en
> + - parallel
> + - python3-grpcio
> + - lcov
> + - libbpf-dev
> + - libssl-dev
> + debian10:
> + parent: "debian"
> + packages: []
> + debian11:
> + parent: "debian"
> + packages:
> + - nasm
> + debian11_arm_ipsec:
> + parent: "debian11"
> + packages: []
> + ubuntu:
> + parent: 'debian'
> + packages: []
> + ubuntu18.04:
> + parent: "ubuntu"
> + packages: []
> + ubuntu20.04:
> + parent: "ubuntu"
> + packages:
> + - python3-grpcio
> + - lcov
> + - jq
> + - default-jre
> + - nodejs
> + ubuntu22.04:
> + parent: "ubuntu"
> + packages:
> + - python3-grpcio
> + - lcov
> + - abigail-tools
> + - jq
> + ubuntu_cross: # x86 -> ARM
> + parent: "ubuntu20.04"
> + packages:
> + - clang
> + - llvm
> + - llvm-dev
> + - llvm-runtime
> + - lld
> + - gcc-aarch64-linux-gnu
> + - libgcc-9-dev-arm64-cross
> + - libatomic1-arm64-cross
> + - libc6-dev-arm64-cross
> + - pkg-config-aarch64-linux-gnu
> + - dpkg-dev
> + - gcc-arm-linux-gnueabihf
> + - libc6-dev-armhf-cross
> + - pkg-config-arm-linux-gnueabihf
> + ubuntu_sve:
> + parent: "ubuntu20.04"
> + packages:
> + - gcc-10
> + targets:
> + # Alpine
> + - name: alpine_compile
> + group: "alpine"
> + base_image: alpine
> + platforms: [linux/amd64, linux/arm64]
> + force_disable_abi: true
> + minimum_dpdk_version:
> + major: 22
> + minor: 0
> + revision: 0
> + extra_information:
> + amd64_environment_id: 45
> + arm64_environment_id: 75
> +
> + # Arch
> + - name: arch_compile
> + group: "arch"
> + base_image: archlinux:base
> + platforms: [linux/amd64]
> + extra_information:
> + amd64_environment_id: 26
> +
> + - name: fedora37_compile
> + group: "fedora"
> + base_image: fedora:37
> + platforms: [linux/amd64, linux/arm64]
> + extra_information:
> + amd64_environment_id: 69
> + arm64_environment_id: 70
> +
> + - name: fedora38_compile
> + group: "fedora"
> + base_image: fedora:38
> + platforms: [linux/amd64, linux/arm64]
> + extra_information:
> + amd64_environment_id: 71
> + arm64_environment_id: 72
> +
> + - name: fedora38_clang
> + group: "fedora_clang"
> + base_image: fedora:38
> + platforms: [linux/amd64, linux/arm64]
> + extra_information:
> + amd64_environment_id: 73
> + arm64_environment_id: 74
> +
> + - name: fedora37_coverity
> + group: "fedora_coverity"
> + base_image: fedora:37
> + force_disable_abi: true
> + requires_coverity: true
> + platforms: [linux/amd64]
> +
> + # CentOs
> + - name: centosstream8_compile
> + group: "centos8"
> + base_image: quay.io/centos/centos:stream8
> + platforms: [linux/amd64]
> + extra_information:
> + amd64_environment_id: 39
> +
> + - name: centosstream9_compile
> + group: "centos9"
> + base_image: quay.io/centos/centos:stream9
> + platforms: [linux/amd64, linux/arm64]
> + extra_information:
> + amd64_environment_id: 53
> + arm64_environment_id: 63
> +
> + # RHEL
> + - name: rhel7_compile
> + group: "rhel7"
> + base_image: registry.access.redhat.com/ubi7/ubi
> + platforms: [linux/amd64]
> + extra_information:
> + amd64_environment_id: 36
> +
> + - name: rhel8_compile
> + group: "rhel8"
> + base_image: registry.access.redhat.com/ubi8/ubi
> + platforms: [linux/amd64]
> + extra_information:
> + amd64_environment_id: 37
> +
> + - name: rhel9_compile
> + group: "rhel9"
> + base_image: registry.access.redhat.com/ubi9/ubi
> + platforms: [linux/amd64, linux/arm64]
> + extra_information:
> + amd64_environment_id: 76
> + arm64_environment_id: 77
> +
> + # OpenSuse
> + - name: opensuse-leap15_compile
> + group: "opensuse"
> + base_image: opensuse/leap:15
> + platforms: [linux/amd64, linux/arm64]
> + extra_information:
> + amd64_environment_id: 25
> + arm64_environment_id: 67
> +
> + # Debian
> + - name: debian10_compile # debian 10
> + group: "debian10"
> + base_image: "debian:buster"
> + libabigail_revision: libabigail-2.2
> + platforms: [linux/amd64]
> + extra_information:
> + amd64_environment_id: 52
> +
> + - name: debian11_compile # debian 11
> + group: "debian11"
> + base_image: "debian:bullseye"
> + libabigail_revision: libabigail-2.2
> + platforms:
> + - linux/amd64
> + - linux/arm64
> + extra_information:
> + amd64_environment_id: 51
> + arm64_environment_id: 64
> +
> + # Ubuntu
> + - name: ubuntu20.04_compile
> + group: "ubuntu20.04"
> + base_image: "ubuntu:20.04"
> + platforms: [linux/amd64, linux/arm64]
> + extra_tags:
> + - $R/ubuntu20.04_libabigail:latest
> + - $R/ubuntu20.04_libabigail:$T
> + - $R/ubuntu20.04_spell-check:latest
> + - $R/ubuntu20.04_spell-check:$T
> + - $R/ubuntu20.04_sonarscanner:latest
> + - $R/ubuntu20.04_sonarscanner:$T
> + extra_information:
> + amd64_environment_id: 38
> + arm64_environment_id: 62
> +
> + # x86 -> ARM cross compile
> + - name: ubuntu20.04_cross
> + group: "ubuntu_cross"
> + base_image: "ubuntu:20.04"
> + platforms: [linux/amd64]
> + cross_file: "config/arm/arm64_armv8_linux_gcc"
> + extra_information:
> + amd64_environment_id: 47
> +
> + # Also, x86 -> ARM cross compile (for SVE test)
> + - name: ubuntu20.04_sve_compile
> + group: "ubuntu_sve"
> + base_image: "arm64v8/ubuntu:20.04"
> + platforms: [linux/arm64]
> + extra_information:
> + arm64_environment_id: 46
> +
> + # TEMPORARY: This has the forked IPsec lib developed by ARM. This causes issues on older LTS versions of DPDK.
> + # Once all supported and tested LTS versions no longer break when this lib is installed, recombine with debian11_compile.
> + - name: debian11_arm_ipsec
> + group: "debian11_arm_ipsec"
> + base_image: "debian:bullseye"
> + platforms: [linux/arm64]
> + force_disable_abi: true
> + extra_tags:
> + - $R/debian11_zuc_compile:latest
> + - $R/debian11_snow3g_compile:latest
> + - $R/debian11_zuc_compile:$T
> + - $R/debian11_snow3g_compile:$T
> + extra_information:
> + arm64_environment_id: 58
> +
> + - name: ubuntu22.04_compile
> + group: "ubuntu22.04"
> + base_image: "ubuntu:22.04"
> + platforms: [linux/amd64]
> + extra_information:
> + amd64_environment_id: 54
> diff --git a/containers/template_engine/inventory_schema.json b/containers/template_engine/inventory_schema.json
> new file mode 100644
> index 0000000..7b79e44
> --- /dev/null
> +++ b/containers/template_engine/inventory_schema.json
> @@ -0,0 +1,221 @@
> +{
> + "$schema": "https://json-schema.org/draft-07/schema",
> + "title": "dpdklab-ci container inventory schema",
> + "$comment": "SPDX-License-Identifier: BSD-3-Clause\nCopyright (c) 2022 University of New Hampshire",
> + "type": "object",
> + "properties": {
> + "abi": {
> + "type": "object",
> + "properties": {
> + "versions": {
> + "type": "array",
> + "items": {
> + "type": "object",
> + "properties": {
> + "dpdk_version": {
> + "type": "object",
> + "properties": {
> + "major": {
> + "type": "integer",
> + "description": "The major version number (usually the year of release)"
> + },
> + "minor": {
> + "type": "integer",
> + "description": "The minor version number (usually the month of release)"
> + },
> + "revision": {
> + "type": "integer",
> + "description": "The revision version number. Starts at 0."
> + },
> + "git_tag": {
> + "type": "string",
> + "description": "The git tag to check out to get this ABI version"
> + }
> + },
> + "required": [
> + "major",
> + "minor",
> + "revision",
> + "git_tag"
> + ],
> + "additionalProperties": false
> + },
> + "abi_version": {
> + "type": "object",
> + "properties": {
> + "major": {
> + "type": "integer",
> + "description": "The major version number (usually the year of release)"
> + },
> + "minor": {
> + "type": "integer",
> + "description": "The minor version number (usually the month of release)"
> + }
> + },
> + "required": [
> + "major",
> + "minor"
> + ],
> + "additionalProperties": false
> + }
> + },
> + "required": [
> + "dpdk_version",
> + "abi_version"
> + ]
> + },
> + "minItems": 0
> + }
> + },
> + "required": [
> + "versions"
> + ],
> + "additionalProperties": false
> + },
> + "dockerfiles": {
> + "type": "object",
> + "properties": {
> + "groups": {
> + "type": "object",
> + "patternProperties": {
> + "^[a-z][a-z_0-9\\.]+$": {
> + "type": "object",
> + "properties": {
> + "package_manager_command": {
> + "type": "string",
> + "description": "The command to install packages without user input."
> + },
> + "parent": {
> + "type": "string",
> + "description": "Which group to inherit from, or NONE if no parent exists"
> + },
> + "packages": {
> + "type": "array",
> + "description": "A list of package names to install",
> + "items": {
> + "type": "string"
> + },
> + "uniqueItems": true
> + }
> + },
> + "required": [
> + "parent",
> + "packages"
> + ],
> + "additionalProperties": false
> + }
> + },
> + "minProperties": 1
> + },
> + "targets": {
> + "type": "array",
> + "items": {
> + "type": "object",
> + "properties": {
> + "name": {
> + "type": "string",
> + "description": "The name of the docker container"
> + },
> + "group": {
> + "type": "string",
> + "description": "Which group the target is a member of"
> + },
> + "base_image": {
> + "type": "string",
> + "description": "The base docker image."
> + },
> + "platforms": {
> + "type": "array",
> + "description": "What platforms to build for",
> + "items": {
> + "type": "string",
> + "enum": [
> + "linux/amd64",
> + "linux/arm64",
> + "linux/riscv64",
> + "linux/ppc64le",
> + "linux/arm/v7",
> + "linux/arm/v6"
> + ]
> + },
> + "minItems": 1
> + },
> + "cross_file": {
> + "type": "string",
> + "description": "A cross file to compile meson with. If not present, meson's automatic behavior will be used."
> + },
> + "libabigail_revision": {
> + "type": "string",
> + "description": "A revision or branch of libabigail to check out before building for a given target."
> + },
> + "force_disable_abi": {
> + "type": "boolean",
> + "description": "Whether to forcibly disable ABI for this target. This should be used as a temporary measure until a fix can be created.",
> + "default": "false"
> + },
> + "requires_coverity": {
> + "type": "boolean",
> + "description": "Whether this target requires the Coverity Scan binaries.",
> + "default": "false"
> + },
> + "minimum_dpdk_version": {
> + "type": "object",
> + "properties": {
> + "major": {
> + "type": "integer",
> + "description": "The major version number (usually the year of release)"
> + },
> + "minor": {
> + "type": "integer",
> + "description": "The minor version number (usually the month of release)"
> + },
> + "revision": {
> + "type": "integer",
> + "description": "The revision version number. Starts at 0."
> + }
> + },
> + "required": [
> + "major",
> + "minor",
> + "revision"
> + ],
> + "additionalProperties": false
> + },
> + "extra_tags": {
> + "type": "array",
> + "description": "Tags to apply to this image",
> + "items": {
> + "type": "string",
> + "description": "$R will be replaced with the 'registry_hostname' variable in this file. $T will be replaced with a timestamp in the form of '%Y-%m-%d'. $N will be replaced by the name of the target.. The tags '$R/$N:latest' and '$R/$N:$T' will always be present"
> + }
> + },
> + "extra_information": {
> + "type": "object",
> + "description": "Extra information to be stored as a JSON file at /container_info.json",
> + "additionalProperties": true
> + }
> + },
> + "required": [
> + "name",
> + "group",
> + "base_image",
> + "platforms"
> + ],
> + "additionalProperties": false
> + },
> + "minItems": 1
> + }
> + },
> + "required": [
> + "groups",
> + "targets"
> + ],
> + "additionalProperties": false
> + }
> + },
> + "required": [
> + "abi",
> + "dockerfiles"
> + ],
> + "additionalProperties": false
> +}
> \ No newline at end of file
next prev parent reply other threads:[~2023-08-04 13:58 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-17 21:08 [PATCH v8 0/6] Community Lab Containers and Builder Engine Adam Hassick
2023-07-17 21:08 ` [PATCH v8 1/6] containers/docs: Add container builder start Adam Hassick
2023-07-17 21:08 ` [PATCH v8 2/6] containers/inventory: Add inventory for container builder Adam Hassick
2023-08-04 13:58 ` Aaron Conole [this message]
2023-08-04 19:30 ` Adam Hassick
2023-07-17 21:08 ` [PATCH v8 3/6] containers/builder: Dockerfile creation script Adam Hassick
2023-08-04 13:59 ` Aaron Conole
2023-07-17 21:08 ` [PATCH v8 4/6] containers/templates: Templates for Dockerfiles Adam Hassick
2023-08-04 14:02 ` Aaron Conole
2023-08-04 19:34 ` Adam Hassick
2023-07-17 21:08 ` [PATCH v8 5/6] containers/container_builder: Container for python scripts Adam Hassick
2023-08-04 14:03 ` Aaron Conole
2023-07-17 21:08 ` [PATCH v8 6/6] containers/Makefile: Makefile to automate builds Adam Hassick
2023-08-04 14:06 ` Aaron Conole
2023-07-28 20:38 ` [PATCH v8 0/6] Community Lab Containers and Builder Engine Aaron Conole
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f7ty1iq6hal.fsf@redhat.com \
--to=aconole@redhat.com \
--cc=ahassick@iol.unh.edu \
--cc=alialnu@nvidia.com \
--cc=ci@dpdk.org \
--cc=ohilyard@iol.unh.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).