From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3E52FA00C5 for ; Mon, 31 Oct 2022 15:16:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BF54C40E5A; Mon, 31 Oct 2022 15:16:25 +0100 (CET) Received: from mail-pf1-f227.google.com (mail-pf1-f227.google.com [209.85.210.227]) by mails.dpdk.org (Postfix) with ESMTP id 68D894069B for ; Mon, 31 Oct 2022 15:16:24 +0100 (CET) Received: by mail-pf1-f227.google.com with SMTP id b185so10788823pfb.9 for ; Mon, 31 Oct 2022 07:16:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ywwzIyeYzYumJyxPXZNjs+Zo2hcaa01HzY+ZBoNp5Bo=; b=BqyQZX1l8p/W/R0K0NdnbMqL6q+N4MytroINmWx6TpriWfyay3rBFwkJbbYoOw0r9Y NayJAph0GtvMWocUaOkPmkXVuoeg0WfKvNQiW8PYueFuI8uTeorfAyjxli7/2z8m6WrQ 6j4DhLR5ZBa1YL3sLVN6bqMFpYPCajQXGurZg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ywwzIyeYzYumJyxPXZNjs+Zo2hcaa01HzY+ZBoNp5Bo=; b=4C7FBPo/9XhsgOk+FDoHWMwOPWL7abFPIE7wCuuFD167WJmXOcTa4RHfxRYz8fFKwL d5UPt6Wlg9feCEf3x6uVCw9GGA0J8ulITFVIM5Y53OMIy+6eyM/I22ZAQvqWtMaN2f4E 33jOmKG+scbgL3YvBfDyy39Dd0v2v/WoIvcbVm0xwQyBro0Fmv9/wyTcHc67F6J0XwQK sttk/BdddYAvc0/ieY+R4gioX+6MRKLBgeTl4GDiKVY3+C/8/6S393XHb+FxtQtt5CQg HBEhwu1mHZcN4YOlxYi3Mrw/vxn4fMd4FyJoH9D94SRIIXl3X1anK8J7mYFZfhn7evh3 VNaA== X-Gm-Message-State: ACrzQf2Ilp0ngcQmk+8yT7ZIv72/g0jOWFleYywXHkxcm9weTGosm7ff Y9VYkwCIzvekLndfnPcDWHUlIWw5YXpWK6frulrDTtvwtIpJh5nqoqwFrjEtb5LU5ZXoGCqVCUe z7FwTznCrluXKiHCSvlJVYeRlH22JaR2mYU0MOURjHz+Z+M68EX0R9ifthnyOSLlsJiz0pRmY6U vTaHhO2Naaa9En X-Google-Smtp-Source: AMsMyM55uSbaAUOhWpP+CoYq0/Bo01Fa8eBnyCY2fgasvBAlcWHkUA8olflyZfRTaO24hLr6D9FUar57e1hK X-Received: by 2002:a63:44b:0:b0:46f:1082:32ac with SMTP id 72-20020a63044b000000b0046f108232acmr13344390pge.136.1667225783468; Mon, 31 Oct 2022 07:16:23 -0700 (PDT) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [2606:4100:3880:1234::84]) by smtp-relay.gmail.com with ESMTPS id nh4-20020a17090b364400b0020a82491ec7sm419465pjb.5.2022.10.31.07.16.23 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 31 Oct 2022 07:16:23 -0700 (PDT) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1257::105d]) by postal.iol.unh.edu (Postfix) with ESMTP id 86CB5605C3FA; Mon, 31 Oct 2022 10:16:22 -0400 (EDT) From: ohilyard@iol.unh.edu To: ci@dpdk.org Cc: aconole@redhat.com, alialnu@nvidia.com, Owen Hilyard Subject: [PATCH v4 0/6] Community Lab Container Definitions Date: Mon, 31 Oct 2022 10:16:14 -0400 Message-Id: <20221031141620.54952-1-ohilyard@iol.unh.edu> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org From: Owen Hilyard This patch series contains a new version of the DPDK CI containers. The old version was tied very tightly to the Community Lab Infrastructure, so it was not suitable for general use. This version is designed to make adding new OSes or OS versions as easy as possible. The minimum functionality can easily be built on any system that can compile DPDK. It includes support for building containers for other non-native architectures (ex: arm containers on x86) and for baking ABI references into the images. The inventory file as added in this patch series defines what the community lab currently supports. If you want to build these yourself, don't try to do parallel Makefile builds on non-server hardware. Libabigail is built into the containers, and since it is not avaliable in all distros it is compiled from source for many targets. If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the current settings, DPDK will be compiled twice for every target in the inventory file. This will become three times when DPDK main has ABI stability again. Due to how the containers are constructed, building on non-native architectures is especially expensive because DPDK is compiled with an emulated compiler in a qemu vm that only has one thread. --- v2: * Added poetry.lock * Remove duplicates in inventory file * Fix typo in base container template v3: * Libabigail is now not built by default unless DPDK_CI_CONTAINERS_BUILD_ABI=y * Made minimum podman version 4.0.0 due to dependencies on cache mounts v4: * Fix whitespace errors Owen Hilyard (6): containers/docs: Add container builder start containers/inventory: Add inventory for container builder containers/builder: Dockerfile creation script containers/templates: Templates for Dockerfiles containers/container_builder: Container for python scripts containers/Makefile: Makefile to automate builds containers/Makefile | 166 ++++++++ containers/README.md | 138 +++++++ containers/container_builder.dockerfile | 26 ++ containers/template_engine/inventory.yaml | 356 ++++++++++++++++++ .../template_engine/inventory_schema.json | 153 ++++++++ containers/template_engine/make_dockerfile.py | 240 ++++++++++++ containers/template_engine/poetry.lock | 220 +++++++++++ containers/template_engine/pyproject.toml | 21 ++ .../templates/containers.makefile.j2 | 31 ++ .../templates/containers/alpine.dockerfile.j2 | 37 ++ .../templates/containers/arch.dockerfile.j2 | 37 ++ .../templates/containers/base.dockerfile.j2 | 85 +++++ .../containers/centos8.dockerfile.j2 | 21 ++ .../containers/centos9.dockerfile.j2 | 17 + .../templates/containers/debian.dockerfile.j2 | 7 + .../containers/debian10.dockerfile.j2 | 3 + .../containers/debian11.dockerfile.j2 | 3 + .../containers/debian_bullseye.dockerfile.j2 | 3 + .../containers/debian_buster.dockerfile.j2 | 3 + .../templates/containers/fedora.dockerfile.j2 | 11 + .../containers/fedora36_clang.dockerfile.j2 | 7 + .../containers/opensuse.dockerfile.j2 | 10 + .../containers/redhat_family.dockerfile.j2 | 5 + .../templates/containers/rhel.dockerfile.j2 | 12 + .../templates/containers/rhel7.dockerfile.j2 | 19 + .../templates/containers/rhel8.dockerfile.j2 | 19 + .../templates/containers/rhel9.dockerfile.j2 | 25 ++ .../templates/containers/rpm.dockerfile.j2 | 3 + .../templates/containers/ubuntu.dockerfile.j2 | 3 + .../containers/ubuntu20.04.dockerfile.j2 | 12 + .../containers/ubuntu22.04.dockerfile.j2 | 7 + .../containers/ubuntu_cross.dockerfile.j2 | 11 + .../containers/ubuntu_sve.dockerfile.j2 | 10 + 33 files changed, 1721 insertions(+) create mode 100644 containers/Makefile create mode 100644 containers/README.md create mode 100644 containers/container_builder.dockerfile create mode 100644 containers/template_engine/inventory.yaml create mode 100644 containers/template_engine/inventory_schema.json create mode 100755 containers/template_engine/make_dockerfile.py create mode 100644 containers/template_engine/poetry.lock create mode 100644 containers/template_engine/pyproject.toml create mode 100644 containers/template_engine/templates/containers.makefile.j2 create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2 -- 2.34.1