From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F198C42FD3; Fri, 4 Aug 2023 15:58:18 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C876E40EE1; Fri, 4 Aug 2023 15:58:18 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 020AC40E2D for ; Fri, 4 Aug 2023 15:58:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691157496; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=3nfWNjJswGm2Cpv/mPJ17tQ5KY3Axak+SDPcZ3VRkFY=; b=WTYagtjbEtNuR7+MTjQNkzQZDwnO46AOf+22i4HWeS5syu6WPlh0JZ7PslSbTG8GBn3S2U TWSCgHuQd7+rYhml2C2V+grgO3yvb4oQGhaSOdDAnw0C715CUySvwWBjjixE2E5mDlk38S GDs0LBnsjONtZsUKjiNF0Tc6V7ZlJ3c= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-18-O4bqnYI5NIidQTGJADA5FA-1; Fri, 04 Aug 2023 09:58:13 -0400 X-MC-Unique: O4bqnYI5NIidQTGJADA5FA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 97F3538210A4; Fri, 4 Aug 2023 13:58:12 +0000 (UTC) Received: from RHTPC1VM0NT (unknown [10.22.33.16]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D0BE940C206F; Fri, 4 Aug 2023 13:58:11 +0000 (UTC) From: Aaron Conole To: Adam Hassick Cc: ci@dpdk.org, alialnu@nvidia.com, Owen Hilyard Subject: Re: [PATCH v8 2/6] containers/inventory: Add inventory for container builder References: <20230717210815.29737-1-ahassick@iol.unh.edu> <20230717210815.29737-3-ahassick@iol.unh.edu> Date: Fri, 04 Aug 2023 09:58:10 -0400 In-Reply-To: <20230717210815.29737-3-ahassick@iol.unh.edu> (Adam Hassick's message of "Mon, 17 Jul 2023 17:08:11 -0400") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Adam Hassick writes: > From: Owen Hilyard > > Adds a yaml file used to define what containers should be built, what > packages the containers should have, what platforms to build for and how they > should be tagged. > > Also adds a JSON schema, which can be used to validate that the > inventory file is well-formed. > > Signed-off-by: Owen Hilyard > Signed-off-by: Adam Hassick > --- > containers/template_engine/inventory.yaml | 470 ++++++++++++++++++ > .../template_engine/inventory_schema.json | 221 ++++++++ > 2 files changed, 691 insertions(+) > create mode 100644 containers/template_engine/inventory.yaml > create mode 100644 containers/template_engine/inventory_schema.json > > diff --git a/containers/template_engine/inventory.yaml b/containers/template_engine/inventory.yaml > new file mode 100644 > index 0000000..31dbdb6 > --- /dev/null > +++ b/containers/template_engine/inventory.yaml > @@ -0,0 +1,470 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright (c) 2022 University of New Hampshire > +--- > +abi: > + # versions should also include the abi versions for all stable ABIs. > + versions: > + # - dpdk_version: # LTS - 3 > + # major: 20 > + # minor: 11 > + # revision: 0 > + # git_tag: "20.11" > + # abi_version: > + # major: 21 > + # minor: 0 Why is this commented out? > + - dpdk_version: # LTS - 2 > + major: 21 > + minor: 11 > + revision: 0 > + git_tag: "21.11" > + abi_version: > + major: 22 > + minor: 0 > + - dpdk_version: > + major: 22 # LTS - 1 > + minor: 11 > + revision: 1 > + git_tag: "tags/v22.11.1" > + abi_version: > + major: 23 > + minor: 10 # .xml comparison > + - dpdk_version: > + major: 22 # LTS > + minor: 11 > + revision: 1 > + git_tag: "tags/v22.11.1" > + abi_version: > + major: 23 > + minor: 11 # .so comparison > + > +dockerfiles: > + groups: > + all: > + package_manager_command: "NO_INSTALL_COMMAND_SET" > + parent: "NONE" > + packages: > + - gcc > + - make > + - git > + - sudo > + - curl > + - automake > + - autoconf > + - flex > + - bison > + - bc > + rpm: > + parent: "all" > + package_manager_command: yum install -y > + packages: > + - diffutils > + - pkg-config > + - python3 > + - python3-pip > + - librdmacm > + - rdma-core-devel > + - libmnl-devel > + - ccache > + - zip > + - autoconf > + - libbpf-devel > + redhat_family: > + parent: "rpm" > + packages: [] > + fedora: > + parent: "redhat_family" > + packages: > + - python3-pyelftools > + - zip > + - clang > + - ninja-build > + - elfutils-libelf-devel > + - jansson-devel > + - libatomic > + - libpcap-devel > + - numactl-devel > + - findutils > + - libtool > + - elfutils-devel > + - libxml2-devel > + - libabigail > + - openssl-devel > + fedora_coverity: > + parent: fedora > + packages: [] > + fedora_clang: > + parent: fedora > + packages: [] > + centos8: > + parent: "redhat_family" > + packages: > + - elfutils-libelf-devel > + - jansson-devel > + - libatomic > + - libpcap-devel > + - numactl-devel > + - libabigail > + - openssl-devel > + centos9: > + parent: "redhat_family" > + package_manager_command: "yum install --allowerasing -y" > + packages: > + - elfutils-libelf-devel > + - jansson-devel > + - libatomic > + - libpcap-devel > + - numactl-devel > + - libtool > + - g++ > + - elfutils-devel > + - libxml2-devel > + - openssl-devel > + rhel: > + parent: "redhat_family" > + package_manager_command: "yum install -y" > + packages: > + - elfutils-libelf-devel > + - jansson-devel > + - libatomic > + - libpcap-devel > + - numactl-devel > + - openssl-devel > + - libabigail > + rhel7: > + parent: "rhel" > + packages: > + - ninja-build > + rhel8: > + parent: "rhel" > + package_manager_command: "yum --allowerasing install -y" > + packages: > + - ninja-build > + - libxml2-devel > + rhel9: > + parent: "rhel" > + package_manager_command: "yum --allowerasing install -y" > + packages: > + - libxml2-devel > + opensuse: > + parent: "rpm" > + package_manager_command: "zypper install -y" > + packages: > + - python3-pyelftools > + - ninja > + - gzip > + - libelf-devel > + - libjansson-devel > + - librdmacm1 > + - libatomic1 > + - libnuma-devel > + - libabigail-tools > + - tar > + - openssl-devel > + alpine: > + parent: "all" > + package_manager_command: "apk add" > + packages: > + - bsd-compat-headers > + - ccache > + - jansson-dev > + - libarchive-dev > + - libbpf-dev > + - libc-dev > + - libpcap-dev > + - linux-headers > + - ninja > + - numactl-dev > + - openssl-dev > + - py3-pip > + - py3-elftools > + - py3-setuptools > + - py3-wheel > + - zlib-dev > + - python3 > + - bash > + - zip > + - meson > + arch: > + parent: "all" > + package_manager_command: "pacman -Syu --noconfirm" > + packages: > + - glibc > + - cmake > + - ninja > + - pandoc > + - python > + - python-docutils > + - ethtool > + - file > + - fakeroot > + - pkg-config > + - numactl > + - elfutils > + - gawk > + - diffutils > + - python-pip > + - libabigail > + - openssl > + - libbpf > + debian: > + parent: "all" > + package_manager_command: "apt-get update && apt-get install --no-install-recommends -y" > + packages: > + - libnuma-dev > + - zip > + - librdmacm1 > + - libcrypto++-dev > + - python3-pyelftools > + - libxml2-dev > + - libelf-dev > + - libpcap-dev > + - libjansson-dev > + - python3-setuptools > + - autoconf > + - pkg-config > + - libtool > + - ccache > + - libatomic1 > + - python3-wheel > + - libdw-dev > + - rdma-core > + - ninja-build > + - automake > + - librdmacm-dev > + - libmnl-dev > + - python3-pip > + - build-essential > + - aspell-en > + - parallel > + - python3-grpcio > + - lcov > + - libbpf-dev > + - libssl-dev > + debian10: > + parent: "debian" > + packages: [] > + debian11: > + parent: "debian" > + packages: > + - nasm > + debian11_arm_ipsec: > + parent: "debian11" > + packages: [] > + ubuntu: > + parent: 'debian' > + packages: [] > + ubuntu18.04: > + parent: "ubuntu" > + packages: [] > + ubuntu20.04: > + parent: "ubuntu" > + packages: > + - python3-grpcio > + - lcov > + - jq > + - default-jre > + - nodejs > + ubuntu22.04: > + parent: "ubuntu" > + packages: > + - python3-grpcio > + - lcov > + - abigail-tools > + - jq > + ubuntu_cross: # x86 -> ARM > + parent: "ubuntu20.04" > + packages: > + - clang > + - llvm > + - llvm-dev > + - llvm-runtime > + - lld > + - gcc-aarch64-linux-gnu > + - libgcc-9-dev-arm64-cross > + - libatomic1-arm64-cross > + - libc6-dev-arm64-cross > + - pkg-config-aarch64-linux-gnu > + - dpkg-dev > + - gcc-arm-linux-gnueabihf > + - libc6-dev-armhf-cross > + - pkg-config-arm-linux-gnueabihf > + ubuntu_sve: > + parent: "ubuntu20.04" > + packages: > + - gcc-10 > + targets: > + # Alpine > + - name: alpine_compile > + group: "alpine" > + base_image: alpine > + platforms: [linux/amd64, linux/arm64] > + force_disable_abi: true > + minimum_dpdk_version: > + major: 22 > + minor: 0 > + revision: 0 > + extra_information: > + amd64_environment_id: 45 > + arm64_environment_id: 75 > + > + # Arch > + - name: arch_compile > + group: "arch" > + base_image: archlinux:base > + platforms: [linux/amd64] > + extra_information: > + amd64_environment_id: 26 > + > + - name: fedora37_compile > + group: "fedora" > + base_image: fedora:37 > + platforms: [linux/amd64, linux/arm64] > + extra_information: > + amd64_environment_id: 69 > + arm64_environment_id: 70 > + > + - name: fedora38_compile > + group: "fedora" > + base_image: fedora:38 > + platforms: [linux/amd64, linux/arm64] > + extra_information: > + amd64_environment_id: 71 > + arm64_environment_id: 72 > + > + - name: fedora38_clang > + group: "fedora_clang" > + base_image: fedora:38 > + platforms: [linux/amd64, linux/arm64] > + extra_information: > + amd64_environment_id: 73 > + arm64_environment_id: 74 > + > + - name: fedora37_coverity > + group: "fedora_coverity" > + base_image: fedora:37 > + force_disable_abi: true > + requires_coverity: true > + platforms: [linux/amd64] > + > + # CentOs > + - name: centosstream8_compile > + group: "centos8" > + base_image: quay.io/centos/centos:stream8 > + platforms: [linux/amd64] > + extra_information: > + amd64_environment_id: 39 > + > + - name: centosstream9_compile > + group: "centos9" > + base_image: quay.io/centos/centos:stream9 > + platforms: [linux/amd64, linux/arm64] > + extra_information: > + amd64_environment_id: 53 > + arm64_environment_id: 63 > + > + # RHEL > + - name: rhel7_compile > + group: "rhel7" > + base_image: registry.access.redhat.com/ubi7/ubi > + platforms: [linux/amd64] > + extra_information: > + amd64_environment_id: 36 > + > + - name: rhel8_compile > + group: "rhel8" > + base_image: registry.access.redhat.com/ubi8/ubi > + platforms: [linux/amd64] > + extra_information: > + amd64_environment_id: 37 > + > + - name: rhel9_compile > + group: "rhel9" > + base_image: registry.access.redhat.com/ubi9/ubi > + platforms: [linux/amd64, linux/arm64] > + extra_information: > + amd64_environment_id: 76 > + arm64_environment_id: 77 > + > + # OpenSuse > + - name: opensuse-leap15_compile > + group: "opensuse" > + base_image: opensuse/leap:15 > + platforms: [linux/amd64, linux/arm64] > + extra_information: > + amd64_environment_id: 25 > + arm64_environment_id: 67 > + > + # Debian > + - name: debian10_compile # debian 10 > + group: "debian10" > + base_image: "debian:buster" > + libabigail_revision: libabigail-2.2 > + platforms: [linux/amd64] > + extra_information: > + amd64_environment_id: 52 > + > + - name: debian11_compile # debian 11 > + group: "debian11" > + base_image: "debian:bullseye" > + libabigail_revision: libabigail-2.2 > + platforms: > + - linux/amd64 > + - linux/arm64 > + extra_information: > + amd64_environment_id: 51 > + arm64_environment_id: 64 > + > + # Ubuntu > + - name: ubuntu20.04_compile > + group: "ubuntu20.04" > + base_image: "ubuntu:20.04" > + platforms: [linux/amd64, linux/arm64] > + extra_tags: > + - $R/ubuntu20.04_libabigail:latest > + - $R/ubuntu20.04_libabigail:$T > + - $R/ubuntu20.04_spell-check:latest > + - $R/ubuntu20.04_spell-check:$T > + - $R/ubuntu20.04_sonarscanner:latest > + - $R/ubuntu20.04_sonarscanner:$T > + extra_information: > + amd64_environment_id: 38 > + arm64_environment_id: 62 > + > + # x86 -> ARM cross compile > + - name: ubuntu20.04_cross > + group: "ubuntu_cross" > + base_image: "ubuntu:20.04" > + platforms: [linux/amd64] > + cross_file: "config/arm/arm64_armv8_linux_gcc" > + extra_information: > + amd64_environment_id: 47 > + > + # Also, x86 -> ARM cross compile (for SVE test) > + - name: ubuntu20.04_sve_compile > + group: "ubuntu_sve" > + base_image: "arm64v8/ubuntu:20.04" > + platforms: [linux/arm64] > + extra_information: > + arm64_environment_id: 46 > + > + # TEMPORARY: This has the forked IPsec lib developed by ARM. This causes issues on older LTS versions of DPDK. > + # Once all supported and tested LTS versions no longer break when this lib is installed, recombine with debian11_compile. > + - name: debian11_arm_ipsec > + group: "debian11_arm_ipsec" > + base_image: "debian:bullseye" > + platforms: [linux/arm64] > + force_disable_abi: true > + extra_tags: > + - $R/debian11_zuc_compile:latest > + - $R/debian11_snow3g_compile:latest > + - $R/debian11_zuc_compile:$T > + - $R/debian11_snow3g_compile:$T > + extra_information: > + arm64_environment_id: 58 > + > + - name: ubuntu22.04_compile > + group: "ubuntu22.04" > + base_image: "ubuntu:22.04" > + platforms: [linux/amd64] > + extra_information: > + amd64_environment_id: 54 > diff --git a/containers/template_engine/inventory_schema.json b/containers/template_engine/inventory_schema.json > new file mode 100644 > index 0000000..7b79e44 > --- /dev/null > +++ b/containers/template_engine/inventory_schema.json > @@ -0,0 +1,221 @@ > +{ > + "$schema": "https://json-schema.org/draft-07/schema", > + "title": "dpdklab-ci container inventory schema", > + "$comment": "SPDX-License-Identifier: BSD-3-Clause\nCopyright (c) 2022 University of New Hampshire", > + "type": "object", > + "properties": { > + "abi": { > + "type": "object", > + "properties": { > + "versions": { > + "type": "array", > + "items": { > + "type": "object", > + "properties": { > + "dpdk_version": { > + "type": "object", > + "properties": { > + "major": { > + "type": "integer", > + "description": "The major version number (usually the year of release)" > + }, > + "minor": { > + "type": "integer", > + "description": "The minor version number (usually the month of release)" > + }, > + "revision": { > + "type": "integer", > + "description": "The revision version number. Starts at 0." > + }, > + "git_tag": { > + "type": "string", > + "description": "The git tag to check out to get this ABI version" > + } > + }, > + "required": [ > + "major", > + "minor", > + "revision", > + "git_tag" > + ], > + "additionalProperties": false > + }, > + "abi_version": { > + "type": "object", > + "properties": { > + "major": { > + "type": "integer", > + "description": "The major version number (usually the year of release)" > + }, > + "minor": { > + "type": "integer", > + "description": "The minor version number (usually the month of release)" > + } > + }, > + "required": [ > + "major", > + "minor" > + ], > + "additionalProperties": false > + } > + }, > + "required": [ > + "dpdk_version", > + "abi_version" > + ] > + }, > + "minItems": 0 > + } > + }, > + "required": [ > + "versions" > + ], > + "additionalProperties": false > + }, > + "dockerfiles": { > + "type": "object", > + "properties": { > + "groups": { > + "type": "object", > + "patternProperties": { > + "^[a-z][a-z_0-9\\.]+$": { > + "type": "object", > + "properties": { > + "package_manager_command": { > + "type": "string", > + "description": "The command to install packages without user input." > + }, > + "parent": { > + "type": "string", > + "description": "Which group to inherit from, or NONE if no parent exists" > + }, > + "packages": { > + "type": "array", > + "description": "A list of package names to install", > + "items": { > + "type": "string" > + }, > + "uniqueItems": true > + } > + }, > + "required": [ > + "parent", > + "packages" > + ], > + "additionalProperties": false > + } > + }, > + "minProperties": 1 > + }, > + "targets": { > + "type": "array", > + "items": { > + "type": "object", > + "properties": { > + "name": { > + "type": "string", > + "description": "The name of the docker container" > + }, > + "group": { > + "type": "string", > + "description": "Which group the target is a member of" > + }, > + "base_image": { > + "type": "string", > + "description": "The base docker image." > + }, > + "platforms": { > + "type": "array", > + "description": "What platforms to build for", > + "items": { > + "type": "string", > + "enum": [ > + "linux/amd64", > + "linux/arm64", > + "linux/riscv64", > + "linux/ppc64le", > + "linux/arm/v7", > + "linux/arm/v6" > + ] > + }, > + "minItems": 1 > + }, > + "cross_file": { > + "type": "string", > + "description": "A cross file to compile meson with. If not present, meson's automatic behavior will be used." > + }, > + "libabigail_revision": { > + "type": "string", > + "description": "A revision or branch of libabigail to check out before building for a given target." > + }, > + "force_disable_abi": { > + "type": "boolean", > + "description": "Whether to forcibly disable ABI for this target. This should be used as a temporary measure until a fix can be created.", > + "default": "false" > + }, > + "requires_coverity": { > + "type": "boolean", > + "description": "Whether this target requires the Coverity Scan binaries.", > + "default": "false" > + }, > + "minimum_dpdk_version": { > + "type": "object", > + "properties": { > + "major": { > + "type": "integer", > + "description": "The major version number (usually the year of release)" > + }, > + "minor": { > + "type": "integer", > + "description": "The minor version number (usually the month of release)" > + }, > + "revision": { > + "type": "integer", > + "description": "The revision version number. Starts at 0." > + } > + }, > + "required": [ > + "major", > + "minor", > + "revision" > + ], > + "additionalProperties": false > + }, > + "extra_tags": { > + "type": "array", > + "description": "Tags to apply to this image", > + "items": { > + "type": "string", > + "description": "$R will be replaced with the 'registry_hostname' variable in this file. $T will be replaced with a timestamp in the form of '%Y-%m-%d'. $N will be replaced by the name of the target.. The tags '$R/$N:latest' and '$R/$N:$T' will always be present" > + } > + }, > + "extra_information": { > + "type": "object", > + "description": "Extra information to be stored as a JSON file at /container_info.json", > + "additionalProperties": true > + } > + }, > + "required": [ > + "name", > + "group", > + "base_image", > + "platforms" > + ], > + "additionalProperties": false > + }, > + "minItems": 1 > + } > + }, > + "required": [ > + "groups", > + "targets" > + ], > + "additionalProperties": false > + } > + }, > + "required": [ > + "abi", > + "dockerfiles" > + ], > + "additionalProperties": false > +} > \ No newline at end of file