From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C1C0242B83 for ; Tue, 23 May 2023 19:04:23 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BF72840A80; Tue, 23 May 2023 19:04:23 +0200 (CEST) Received: from mail-qv1-f51.google.com (mail-qv1-f51.google.com [209.85.219.51]) by mails.dpdk.org (Postfix) with ESMTP id 1CBD14014F for ; Tue, 23 May 2023 19:04:22 +0200 (CEST) Received: by mail-qv1-f51.google.com with SMTP id 6a1803df08f44-623a6c15aacso36662396d6.0 for ; Tue, 23 May 2023 10:04:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1684861461; x=1687453461; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=YvpTgVsG9+gmLtCtJl+4uT/GYwynhiASfpvTYFicEQY=; b=ICMWHlDGCxyFj0iwJIQE8I2ts4gM2L8fPDx8DuU2cVlCPtaTvi7RjzEfi9rUNPEbuC JOjPrA7zlTNCyCNyoCIoAIe5t++GrbWOqbHro1JuK2yE7HZSx8X9DH/+wBPfyOnkKGSo GUKypN+tqS8GhGOPtdrxxwkhU7kvOR8DY/byU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684861461; x=1687453461; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YvpTgVsG9+gmLtCtJl+4uT/GYwynhiASfpvTYFicEQY=; b=WSvklWFhw4gu9hm4XStycRRhqOlYnz6qqt1wAnlBAFT0rgIN5If3TguGj1N7aMewgA ZQAZ/Wm9CepuMFtRGnO8cdxatBqLzdNkcjO1BrebSVe7dkO0g72NWplOzWboI9uJ3Oqa GqFcIyYvUVH8G4pWJA/8brOubtIJMYxAzzStjC3rZck9+uX5CMr8KXrXBPnXWgMTQtzS sG1MVZwTMWBhpe41FNaqS1pOP62bIXwYBvNkEAD3/C2dfx2to+k3xHCVi2+5DJd505hS oTE/W5SshKMeTptqChXbpqVTmCqyToQzmuCysTBMUlYnYsnJluSS2tsZCZisMqYJAx6Y OUkQ== X-Gm-Message-State: AC+VfDyQw/BwkTuX//xN2eKp6jntvZW9rucnqv+5QblxCgP9sso6LE65 2djuxecNiTaC9YH+heEwUyvsTzgBSdhyelSeMJ8AZ7gBSvuix/jQ1rQ+pFDyl3peJET5Dw+7Jzk tieSbJ629s4ttVQvMgDRHnNB4pj4f3rFa+IXDY8qUqVyOQKCQPg2ULb8jT/ELkQ== X-Google-Smtp-Source: ACHHUZ5Z98JUkJK/ekD7R09LvbhGQ3MJtHUGiYycUOT4AwZv9f5SDvNZifDxjnoAv+bltxDNaWnPzQ== X-Received: by 2002:a05:6214:21ef:b0:623:a60e:e49 with SMTP id p15-20020a05621421ef00b00623a60e0e49mr17316373qvj.19.1684861461277; Tue, 23 May 2023 10:04:21 -0700 (PDT) Received: from ahassick-Desktop.iol.unh.edu ([2606:4100:3880:1220:7704:829d:adaa:3294]) by smtp.gmail.com with ESMTPSA id g14-20020ac8774e000000b003e38c9a2a22sm2996553qtu.92.2023.05.23.10.04.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 10:04:20 -0700 (PDT) From: Adam Hassick To: ci@dpdk.org Cc: Adam Hassick , aconole@redhat.com, alialnu@nvidia.com Subject: [PATCH v5 00/11] Community Lab Containers and Builder Engine Date: Tue, 23 May 2023 13:04:02 -0400 Message-Id: <20230523170413.812922-1-ahassick@iol.unh.edu> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org This patch series contains a new version of the DPDK CI containers. The old version was tied very tightly to the Community Lab Infrastructure, so it was not suitable for general use. This version is designed to make adding new OSes or OS versions as easy as possible. The minimum functionality can easily be built on any system that can compile DPDK. It includes support for building containers for other non-native architectures (ex: arm containers on x86) and for baking ABI references into the images. Support for building the Coverity Scan container image has also been added. The inventory file as added in this patch series defines what the community lab currently supports. If you want to build these yourself, don't try to do parallel Makefile builds on non-server hardware. Libabigail is built into the containers, and since it is not avaliable in all distros it is compiled from source for many targets. If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the current settings, DPDK will be compiled thrice for every target in the inventory file. Due to how the containers are constructed, building on non-native architectures is especially expensive because DPDK is compiled with an emulated compiler in a qemu vm that only has one thread. Adam Hassick (5): containers/inventory: Add ABI rev and coverity attribute containers/builder: Add arguments to templating script containers/templates: Update templates containers/Makefile: Add new variables and target changes containers/docs: Update README Owen Hilyard (6): containers/docs: Add container builder start containers/inventory: Add inventory for container builder containers/builder: Dockerfile creation script containers/templates: Templates for Dockerfiles containers/container_builder: Container for python scripts containers/Makefile: Makefile to automate builds containers/Makefile | 252 ++++++++++ containers/README.md | 169 +++++++ containers/container_builder.dockerfile | 26 ++ containers/template_engine/inventory.yaml | 442 ++++++++++++++++++ .../template_engine/inventory_schema.json | 221 +++++++++ containers/template_engine/make_dockerfile.py | 358 ++++++++++++++ containers/template_engine/poetry.lock | 220 +++++++++ containers/template_engine/pyproject.toml | 21 + .../templates/containers.makefile.j2 | 73 +++ .../templates/containers/alpine.dockerfile.j2 | 4 + .../templates/containers/arch.dockerfile.j2 | 37 ++ .../templates/containers/base.dockerfile.j2 | 121 +++++ .../containers/centos8.dockerfile.j2 | 21 + .../containers/centos9.dockerfile.j2 | 17 + .../templates/containers/debian.dockerfile.j2 | 7 + .../containers/debian10.dockerfile.j2 | 3 + .../containers/debian11.dockerfile.j2 | 3 + .../debian11_arm_ipsec.dockerfile.j2 | 16 + .../containers/debian_bullseye.dockerfile.j2 | 3 + .../containers/debian_buster.dockerfile.j2 | 3 + .../templates/containers/fedora.dockerfile.j2 | 11 + .../containers/fedora36_clang.dockerfile.j2 | 7 + .../containers/fedora_clang.dockerfile.j2 | 7 + .../containers/fedora_coverity.dockerfile.j2 | 10 + .../containers/opensuse.dockerfile.j2 | 10 + .../containers/redhat_family.dockerfile.j2 | 5 + .../templates/containers/rhel.dockerfile.j2 | 16 + .../templates/containers/rhel7.dockerfile.j2 | 15 + .../templates/containers/rhel8.dockerfile.j2 | 15 + .../templates/containers/rhel9.dockerfile.j2 | 19 + .../templates/containers/rpm.dockerfile.j2 | 3 + .../templates/containers/ubuntu.dockerfile.j2 | 3 + .../containers/ubuntu20.04.dockerfile.j2 | 12 + .../containers/ubuntu22.04.dockerfile.j2 | 3 + .../containers/ubuntu_cross.dockerfile.j2 | 12 + .../containers/ubuntu_sve.dockerfile.j2 | 12 + 36 files changed, 2177 insertions(+) create mode 100644 containers/Makefile create mode 100644 containers/README.md create mode 100644 containers/container_builder.dockerfile create mode 100644 containers/template_engine/inventory.yaml create mode 100644 containers/template_engine/inventory_schema.json create mode 100755 containers/template_engine/make_dockerfile.py create mode 100644 containers/template_engine/poetry.lock create mode 100644 containers/template_engine/pyproject.toml create mode 100644 containers/template_engine/templates/containers.makefile.j2 create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2 -- 2.34.1