DPDK CI discussions
 help / color / mirror / Atom feed
From: Adam Hassick <ahassick@iol.unh.edu>
To: ci@dpdk.org
Cc: aconole@redhat.com, alialnu@nvidia.com,
	Owen Hilyard <ohilyard@iol.unh.edu>,
	Adam Hassick <ahassick@iol.unh.edu>
Subject: [PATCH v8 2/6] containers/inventory: Add inventory for container builder
Date: Mon, 17 Jul 2023 17:08:11 -0400	[thread overview]
Message-ID: <20230717210815.29737-3-ahassick@iol.unh.edu> (raw)
In-Reply-To: <20230717210815.29737-1-ahassick@iol.unh.edu>

From: Owen Hilyard <ohilyard@iol.unh.edu>

Adds a yaml file used to define what containers should be built, what
packages the containers should have, what platforms to build for and how they
should be tagged.

Also adds a JSON schema, which can be used to validate that the
inventory file is well-formed.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
---
 containers/template_engine/inventory.yaml     | 470 ++++++++++++++++++
 .../template_engine/inventory_schema.json     | 221 ++++++++
 2 files changed, 691 insertions(+)
 create mode 100644 containers/template_engine/inventory.yaml
 create mode 100644 containers/template_engine/inventory_schema.json

diff --git a/containers/template_engine/inventory.yaml b/containers/template_engine/inventory.yaml
new file mode 100644
index 0000000..31dbdb6
--- /dev/null
+++ b/containers/template_engine/inventory.yaml
@@ -0,0 +1,470 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2022 University of New Hampshire
+---
+abi:
+  # versions should also include the abi versions for all stable ABIs.
+  versions:
+    # - dpdk_version: # LTS - 3
+    #     major: 20
+    #     minor: 11
+    #     revision: 0
+    #     git_tag: "20.11"
+    #   abi_version:
+    #     major: 21
+    #     minor: 0
+    - dpdk_version: # LTS - 2
+        major: 21
+        minor: 11
+        revision: 0
+        git_tag: "21.11"
+      abi_version:
+        major: 22
+        minor: 0
+    - dpdk_version:
+        major: 22 # LTS - 1
+        minor: 11
+        revision: 1
+        git_tag: "tags/v22.11.1"
+      abi_version:
+        major: 23
+        minor: 10 # .xml comparison
+    - dpdk_version:
+        major: 22 # LTS
+        minor: 11
+        revision: 1
+        git_tag: "tags/v22.11.1"
+      abi_version:
+        major: 23
+        minor: 11 # .so comparison
+
+dockerfiles:
+  groups:
+    all:
+      package_manager_command: "NO_INSTALL_COMMAND_SET"
+      parent: "NONE"
+      packages:
+        - gcc
+        - make
+        - git
+        - sudo
+        - curl
+        - automake
+        - autoconf
+        - flex
+        - bison
+        - bc
+    rpm:
+      parent: "all"
+      package_manager_command: yum install -y
+      packages:
+        - diffutils
+        - pkg-config
+        - python3
+        - python3-pip
+        - librdmacm
+        - rdma-core-devel
+        - libmnl-devel
+        - ccache
+        - zip
+        - autoconf
+        - libbpf-devel
+    redhat_family:
+      parent: "rpm"
+      packages: []
+    fedora:
+      parent: "redhat_family"
+      packages:
+        - python3-pyelftools
+        - zip
+        - clang
+        - ninja-build
+        - elfutils-libelf-devel
+        - jansson-devel
+        - libatomic
+        - libpcap-devel
+        - numactl-devel
+        - findutils
+        - libtool
+        - elfutils-devel
+        - libxml2-devel
+        - libabigail
+        - openssl-devel
+    fedora_coverity:
+      parent: fedora
+      packages: []
+    fedora_clang:
+      parent: fedora
+      packages: []
+    centos8:
+      parent: "redhat_family"
+      packages:
+        - elfutils-libelf-devel
+        - jansson-devel
+        - libatomic
+        - libpcap-devel
+        - numactl-devel
+        - libabigail
+        - openssl-devel
+    centos9:
+      parent: "redhat_family"
+      package_manager_command: "yum install --allowerasing -y"
+      packages:
+        - elfutils-libelf-devel
+        - jansson-devel
+        - libatomic
+        - libpcap-devel
+        - numactl-devel
+        - libtool
+        - g++
+        - elfutils-devel
+        - libxml2-devel
+        - openssl-devel
+    rhel:
+      parent: "redhat_family"
+      package_manager_command: "yum install -y"
+      packages:
+        - elfutils-libelf-devel
+        - jansson-devel
+        - libatomic
+        - libpcap-devel
+        - numactl-devel
+        - openssl-devel
+        - libabigail
+    rhel7:
+      parent: "rhel"
+      packages:
+        - ninja-build
+    rhel8:
+      parent: "rhel"
+      package_manager_command: "yum --allowerasing install -y"
+      packages:
+        - ninja-build
+        - libxml2-devel
+    rhel9:
+      parent: "rhel"
+      package_manager_command: "yum --allowerasing install -y"
+      packages:
+        - libxml2-devel
+    opensuse:
+      parent: "rpm"
+      package_manager_command: "zypper install -y"
+      packages:
+        - python3-pyelftools
+        - ninja
+        - gzip
+        - libelf-devel
+        - libjansson-devel
+        - librdmacm1
+        - libatomic1
+        - libnuma-devel
+        - libabigail-tools
+        - tar
+        - openssl-devel
+    alpine:
+      parent: "all"
+      package_manager_command: "apk add"
+      packages:
+        - bsd-compat-headers
+        - ccache
+        - jansson-dev
+        - libarchive-dev
+        - libbpf-dev
+        - libc-dev
+        - libpcap-dev
+        - linux-headers
+        - ninja
+        - numactl-dev
+        - openssl-dev
+        - py3-pip
+        - py3-elftools
+        - py3-setuptools
+        - py3-wheel
+        - zlib-dev
+        - python3
+        - bash
+        - zip
+        - meson
+    arch:
+      parent: "all"
+      package_manager_command: "pacman -Syu --noconfirm"
+      packages:
+        - glibc
+        - cmake
+        - ninja
+        - pandoc
+        - python
+        - python-docutils
+        - ethtool
+        - file
+        - fakeroot
+        - pkg-config
+        - numactl
+        - elfutils
+        - gawk
+        - diffutils
+        - python-pip
+        - libabigail
+        - openssl
+        - libbpf
+    debian:
+      parent: "all"
+      package_manager_command: "apt-get update && apt-get install --no-install-recommends -y"
+      packages:
+        - libnuma-dev
+        - zip
+        - librdmacm1
+        - libcrypto++-dev
+        - python3-pyelftools
+        - libxml2-dev
+        - libelf-dev
+        - libpcap-dev
+        - libjansson-dev
+        - python3-setuptools
+        - autoconf
+        - pkg-config
+        - libtool
+        - ccache
+        - libatomic1
+        - python3-wheel
+        - libdw-dev
+        - rdma-core
+        - ninja-build
+        - automake
+        - librdmacm-dev
+        - libmnl-dev
+        - python3-pip
+        - build-essential
+        - aspell-en
+        - parallel
+        - python3-grpcio
+        - lcov
+        - libbpf-dev
+        - libssl-dev
+    debian10:
+      parent: "debian"
+      packages: []
+    debian11:
+      parent: "debian"
+      packages:
+        - nasm
+    debian11_arm_ipsec:
+      parent: "debian11"
+      packages: []
+    ubuntu:
+      parent: 'debian'
+      packages: []
+    ubuntu18.04:
+      parent: "ubuntu"
+      packages: []
+    ubuntu20.04:
+      parent: "ubuntu"
+      packages:
+        - python3-grpcio
+        - lcov
+        - jq
+        - default-jre
+        - nodejs
+    ubuntu22.04:
+      parent: "ubuntu"
+      packages:
+        - python3-grpcio
+        - lcov
+        - abigail-tools
+        - jq
+    ubuntu_cross: # x86 -> ARM
+      parent: "ubuntu20.04"
+      packages:
+        - clang
+        - llvm
+        - llvm-dev
+        - llvm-runtime
+        - lld
+        - gcc-aarch64-linux-gnu
+        - libgcc-9-dev-arm64-cross
+        - libatomic1-arm64-cross
+        - libc6-dev-arm64-cross
+        - pkg-config-aarch64-linux-gnu
+        - dpkg-dev
+        - gcc-arm-linux-gnueabihf
+        - libc6-dev-armhf-cross
+        - pkg-config-arm-linux-gnueabihf
+    ubuntu_sve:
+      parent: "ubuntu20.04"
+      packages:
+        - gcc-10
+  targets:
+    # Alpine
+    - name: alpine_compile
+      group: "alpine"
+      base_image: alpine
+      platforms: [linux/amd64, linux/arm64]
+      force_disable_abi: true
+      minimum_dpdk_version:
+        major: 22
+        minor: 0
+        revision: 0
+      extra_information:
+        amd64_environment_id: 45
+        arm64_environment_id: 75
+
+    # Arch
+    - name: arch_compile
+      group: "arch"
+      base_image: archlinux:base
+      platforms: [linux/amd64]
+      extra_information:
+        amd64_environment_id: 26
+
+    - name: fedora37_compile
+      group: "fedora"
+      base_image: fedora:37
+      platforms: [linux/amd64, linux/arm64]
+      extra_information:
+        amd64_environment_id: 69
+        arm64_environment_id: 70
+
+    - name: fedora38_compile
+      group: "fedora"
+      base_image: fedora:38
+      platforms: [linux/amd64, linux/arm64]
+      extra_information:
+        amd64_environment_id: 71
+        arm64_environment_id: 72
+
+    - name: fedora38_clang
+      group: "fedora_clang"
+      base_image: fedora:38
+      platforms: [linux/amd64, linux/arm64]
+      extra_information:
+        amd64_environment_id: 73
+        arm64_environment_id: 74
+
+    - name: fedora37_coverity
+      group: "fedora_coverity"
+      base_image: fedora:37
+      force_disable_abi: true
+      requires_coverity: true
+      platforms: [linux/amd64]
+
+    # CentOs
+    - name: centosstream8_compile
+      group: "centos8"
+      base_image: quay.io/centos/centos:stream8
+      platforms: [linux/amd64]
+      extra_information:
+        amd64_environment_id: 39
+
+    - name: centosstream9_compile
+      group: "centos9"
+      base_image: quay.io/centos/centos:stream9
+      platforms: [linux/amd64, linux/arm64]
+      extra_information:
+        amd64_environment_id: 53
+        arm64_environment_id: 63
+
+    # RHEL
+    - name: rhel7_compile
+      group: "rhel7"
+      base_image: registry.access.redhat.com/ubi7/ubi
+      platforms: [linux/amd64]
+      extra_information:
+        amd64_environment_id: 36
+
+    - name: rhel8_compile
+      group: "rhel8"
+      base_image: registry.access.redhat.com/ubi8/ubi
+      platforms: [linux/amd64]
+      extra_information:
+        amd64_environment_id: 37
+
+    - name: rhel9_compile
+      group: "rhel9"
+      base_image: registry.access.redhat.com/ubi9/ubi
+      platforms: [linux/amd64, linux/arm64]
+      extra_information:
+        amd64_environment_id: 76
+        arm64_environment_id: 77
+
+    # OpenSuse
+    - name: opensuse-leap15_compile
+      group: "opensuse"
+      base_image: opensuse/leap:15
+      platforms: [linux/amd64, linux/arm64]
+      extra_information:
+        amd64_environment_id: 25
+        arm64_environment_id: 67
+
+    # Debian
+    - name: debian10_compile # debian 10
+      group: "debian10"
+      base_image: "debian:buster"
+      libabigail_revision: libabigail-2.2
+      platforms: [linux/amd64]
+      extra_information:
+        amd64_environment_id: 52
+
+    - name: debian11_compile # debian 11
+      group: "debian11"
+      base_image: "debian:bullseye"
+      libabigail_revision: libabigail-2.2
+      platforms:
+        - linux/amd64
+        - linux/arm64
+      extra_information:
+        amd64_environment_id: 51
+        arm64_environment_id: 64
+
+    # Ubuntu
+    - name: ubuntu20.04_compile
+      group: "ubuntu20.04"
+      base_image: "ubuntu:20.04"
+      platforms: [linux/amd64, linux/arm64]
+      extra_tags:
+        - $R/ubuntu20.04_libabigail:latest
+        - $R/ubuntu20.04_libabigail:$T
+        - $R/ubuntu20.04_spell-check:latest
+        - $R/ubuntu20.04_spell-check:$T
+        - $R/ubuntu20.04_sonarscanner:latest
+        - $R/ubuntu20.04_sonarscanner:$T
+      extra_information:
+        amd64_environment_id: 38
+        arm64_environment_id: 62
+
+    # x86 -> ARM cross compile
+    - name: ubuntu20.04_cross
+      group: "ubuntu_cross"
+      base_image: "ubuntu:20.04"
+      platforms: [linux/amd64]
+      cross_file: "config/arm/arm64_armv8_linux_gcc"
+      extra_information:
+        amd64_environment_id: 47
+
+    # Also, x86 -> ARM cross compile (for SVE test)
+    - name: ubuntu20.04_sve_compile
+      group: "ubuntu_sve"
+      base_image: "arm64v8/ubuntu:20.04"
+      platforms: [linux/arm64]
+      extra_information:
+        arm64_environment_id: 46
+
+    # TEMPORARY: This has the forked IPsec lib developed by ARM. This causes issues on older LTS versions of DPDK.
+    # Once all supported and tested LTS versions no longer break when this lib is installed, recombine with debian11_compile.
+    - name: debian11_arm_ipsec
+      group: "debian11_arm_ipsec"
+      base_image: "debian:bullseye"
+      platforms: [linux/arm64]
+      force_disable_abi: true
+      extra_tags:
+        - $R/debian11_zuc_compile:latest
+        - $R/debian11_snow3g_compile:latest
+        - $R/debian11_zuc_compile:$T
+        - $R/debian11_snow3g_compile:$T
+      extra_information:
+        arm64_environment_id: 58
+
+    - name: ubuntu22.04_compile
+      group: "ubuntu22.04"
+      base_image: "ubuntu:22.04"
+      platforms: [linux/amd64]
+      extra_information:
+        amd64_environment_id: 54
diff --git a/containers/template_engine/inventory_schema.json b/containers/template_engine/inventory_schema.json
new file mode 100644
index 0000000..7b79e44
--- /dev/null
+++ b/containers/template_engine/inventory_schema.json
@@ -0,0 +1,221 @@
+{
+    "$schema": "https://json-schema.org/draft-07/schema",
+    "title": "dpdklab-ci container inventory schema",
+    "$comment": "SPDX-License-Identifier: BSD-3-Clause\nCopyright (c) 2022 University of New Hampshire",
+    "type": "object",
+    "properties": {
+        "abi": {
+            "type": "object",
+            "properties": {
+                "versions": {
+                    "type": "array",
+                    "items": {
+                        "type": "object",
+                        "properties": {
+                            "dpdk_version": {
+                                "type": "object",
+                                "properties": {
+                                    "major": {
+                                        "type": "integer",
+                                        "description": "The major version number (usually the year of release)"
+                                    },
+                                    "minor": {
+                                        "type": "integer",
+                                        "description": "The minor version number (usually the month of release)"
+                                    },
+                                    "revision": {
+                                        "type": "integer",
+                                        "description": "The revision version number. Starts at 0."
+                                    },
+                                    "git_tag": {
+                                        "type": "string",
+                                        "description": "The git tag to check out to get this ABI version"
+                                    }
+                                },
+                                "required": [
+                                    "major",
+                                    "minor",
+                                    "revision",
+                                    "git_tag"
+                                ],
+                                "additionalProperties": false
+                            },
+                            "abi_version": {
+                                "type": "object",
+                                "properties": {
+                                    "major": {
+                                        "type": "integer",
+                                        "description": "The major version number (usually the year of release)"
+                                    },
+                                    "minor": {
+                                        "type": "integer",
+                                        "description": "The minor version number (usually the month of release)"
+                                    }
+                                },
+                                "required": [
+                                    "major",
+                                    "minor"
+                                ],
+                                "additionalProperties": false
+                            }
+                        },
+                        "required": [
+                            "dpdk_version",
+                            "abi_version"
+                        ]
+                    },
+                    "minItems": 0
+                }
+            },
+            "required": [
+                "versions"
+            ],
+            "additionalProperties": false
+        },
+        "dockerfiles": {
+            "type": "object",
+            "properties": {
+                "groups": {
+                    "type": "object",
+                    "patternProperties": {
+                        "^[a-z][a-z_0-9\\.]+$": {
+                            "type": "object",
+                            "properties": {
+                                "package_manager_command": {
+                                    "type": "string",
+                                    "description": "The command to install packages without user input."
+                                },
+                                "parent": {
+                                    "type": "string",
+                                    "description": "Which group to inherit from, or NONE if no parent exists"
+                                },
+                                "packages": {
+                                    "type": "array",
+                                    "description": "A list of package names to install",
+                                    "items": {
+                                        "type": "string"
+                                    },
+                                    "uniqueItems": true
+                                }
+                            },
+                            "required": [
+                                "parent",
+                                "packages"
+                            ],
+                            "additionalProperties": false
+                        }
+                    },
+                    "minProperties": 1
+                },
+                "targets": {
+                    "type": "array",
+                    "items": {
+                        "type": "object",
+                        "properties": {
+                            "name": {
+                                "type": "string",
+                                "description": "The name of the docker container"
+                            },
+                            "group": {
+                                "type": "string",
+                                "description": "Which group the target is a member of"
+                            },
+                            "base_image": {
+                                "type": "string",
+                                "description": "The base docker image."
+                            },
+                            "platforms": {
+                                "type": "array",
+                                "description": "What platforms to build for",
+                                "items": {
+                                    "type": "string",
+                                    "enum": [
+                                        "linux/amd64",
+                                        "linux/arm64",
+                                        "linux/riscv64",
+                                        "linux/ppc64le",
+                                        "linux/arm/v7",
+                                        "linux/arm/v6"
+                                    ]
+                                },
+                                "minItems": 1
+                            },
+                            "cross_file": {
+                                "type": "string",
+                                "description": "A cross file to compile meson with. If not present, meson's automatic behavior will be used."
+                            },
+                            "libabigail_revision": {
+                                "type": "string",
+                                "description": "A revision or branch of libabigail to check out before building for a given target."
+                            },
+                            "force_disable_abi": {
+                                "type": "boolean",
+                                "description": "Whether to forcibly disable ABI for this target. This should be used as a temporary measure until a fix can be created.",
+                                "default": "false"
+                            },
+                            "requires_coverity": {
+                                "type": "boolean",
+                                "description": "Whether this target requires the Coverity Scan binaries.",
+                                "default": "false"
+                            },
+                            "minimum_dpdk_version": {
+                                "type": "object",
+                                "properties": {
+                                    "major": {
+                                        "type": "integer",
+                                        "description": "The major version number (usually the year of release)"
+                                    },
+                                    "minor": {
+                                        "type": "integer",
+                                        "description": "The minor version number (usually the month of release)"
+                                    },
+                                    "revision": {
+                                        "type": "integer",
+                                        "description": "The revision version number. Starts at 0."
+                                    }
+                                },
+                                "required": [
+                                    "major",
+                                    "minor",
+                                    "revision"
+                                ],
+                                "additionalProperties": false
+                            },
+                            "extra_tags": {
+                                "type": "array",
+                                "description": "Tags to apply to this image",
+                                "items": {
+                                    "type": "string",
+                                    "description": "$R will be replaced with the 'registry_hostname' variable in this file. $T will be replaced with a timestamp in the form of '%Y-%m-%d'. $N will be replaced by the name of the target.. The tags '$R/$N:latest' and '$R/$N:$T' will always be present"
+                                }
+                            },
+                            "extra_information": {
+                                "type": "object",
+                                "description": "Extra information to be stored as a JSON file at /container_info.json",
+                                "additionalProperties": true
+                            }
+                        },
+                        "required": [
+                            "name",
+                            "group",
+                            "base_image",
+                            "platforms"
+                        ],
+                        "additionalProperties": false
+                    },
+                    "minItems": 1
+                }
+            },
+            "required": [
+                "groups",
+                "targets"
+            ],
+            "additionalProperties": false
+        }
+    },
+    "required": [
+        "abi",
+        "dockerfiles"
+    ],
+    "additionalProperties": false
+}
\ No newline at end of file
-- 
2.41.0


  parent reply	other threads:[~2023-07-17 21:11 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-07-17 21:08 [PATCH v8 0/6] Community Lab Containers and Builder Engine Adam Hassick
2023-07-17 21:08 ` [PATCH v8 1/6] containers/docs: Add container builder start Adam Hassick
2023-07-17 21:08 ` Adam Hassick [this message]
2023-08-04 13:58   ` [PATCH v8 2/6] containers/inventory: Add inventory for container builder Aaron Conole
2023-08-04 19:30     ` Adam Hassick
2023-07-17 21:08 ` [PATCH v8 3/6] containers/builder: Dockerfile creation script Adam Hassick
2023-08-04 13:59   ` Aaron Conole
2023-07-17 21:08 ` [PATCH v8 4/6] containers/templates: Templates for Dockerfiles Adam Hassick
2023-08-04 14:02   ` Aaron Conole
2023-08-04 19:34     ` Adam Hassick
2023-07-17 21:08 ` [PATCH v8 5/6] containers/container_builder: Container for python scripts Adam Hassick
2023-08-04 14:03   ` Aaron Conole
2023-07-17 21:08 ` [PATCH v8 6/6] containers/Makefile: Makefile to automate builds Adam Hassick
2023-08-04 14:06   ` Aaron Conole
2023-07-28 20:38 ` [PATCH v8 0/6] Community Lab Containers and Builder Engine Aaron Conole

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230717210815.29737-3-ahassick@iol.unh.edu \
    --to=ahassick@iol.unh.edu \
    --cc=aconole@redhat.com \
    --cc=alialnu@nvidia.com \
    --cc=ci@dpdk.org \
    --cc=ohilyard@iol.unh.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).