From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 184C342B9D for ; Thu, 25 May 2023 19:15:18 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ED86340E6E; Thu, 25 May 2023 19:15:17 +0200 (CEST) Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by mails.dpdk.org (Postfix) with ESMTP id E549940DDB for ; Thu, 25 May 2023 19:15:16 +0200 (CEST) Received: by mail-qt1-f174.google.com with SMTP id d75a77b69052e-3f805551a19so588711cf.3 for ; Thu, 25 May 2023 10:15:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1685034916; x=1687626916; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=RQdL+0j/lGtITnpucj0zYBfuZhOMnOEj7g6Im3AYejQ=; b=eoHNlOzl6V68+vXeXHX/iA2KxfXVW3OCKnQUtFmQ7XQel8z5nyqem33Z/UeZD24Jz+ NHMkc8zjcInHiDvQdKSv7BdetJ60TTktpQBN3qsmGw33UTaGlW321BxJVHb/wSSU3iQg 4R74gZRgebgZCdB+fZotLGmg2iGZNqYceyxCA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685034916; x=1687626916; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=RQdL+0j/lGtITnpucj0zYBfuZhOMnOEj7g6Im3AYejQ=; b=XmAd3rkL1L07M2aX/K4YCCXA5b7nSY28OlwOE5El00rLCQaZvDVODhWDEkIYAUFPnJ vw1s+icFr4DNNaf5XpuX6rrDXpF38tKYogu2dyS1gZ8SRX1eSgEeTE3Ah1bm9sG2iUhh 1M0ocvW8hXcmINWpJldhn1p59MU1Hz0Y1Jv/gfAlttDpZKHGjagIYyZ/ndl74GyhNTWR eTtIXOkim2bW5AnwKfHKTFQFyoMyLUk9iJ2EUw6eoItl/tGpNI3HBAV5RKIQ5E78PXI6 Odokf8LLiqC3If4HdfRE8+GrY5yvds8M5mf6ycntj7QSN2lYUJBLFMOITqGulRFsx6yA LUIw== X-Gm-Message-State: AC+VfDwa9MjZRT8kELA7ZYPXjBfLjMj3aFAnIQEizg7nGFsh47zN/OSv HmcSzzW05zy1kfkh8bUzYVrTW0WIVI8kNnf4eNjIdxA0BTqNwsTQQC+laBB2g9AOCNqqhOsuyzH 6SXRrpmYzxIatL6aKMDmYRG5ZIJlv4AodE/s5Gu/+pbt/xRmQNI2W2etQYOYkAg== X-Google-Smtp-Source: ACHHUZ5v5V4bZqBLtR3lBL9IOls8BkyJIe2tBGlyFHdwbxlHoh7C/+bHIhzjwn9dhehGplmc5DnCXw== X-Received: by 2002:a05:622a:1aa6:b0:3f5:543:4c46 with SMTP id s38-20020a05622a1aa600b003f505434c46mr255387qtc.53.1685034916058; Thu, 25 May 2023 10:15:16 -0700 (PDT) Received: from ahassick-Desktop.iol.unh.edu ([2606:4100:3880:1220:1742:973:cf40:525f]) by smtp.gmail.com with ESMTPSA id ff27-20020a05622a4d9b00b003f6bbd7863csm545609qtb.86.2023.05.25.10.15.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 May 2023 10:15:15 -0700 (PDT) From: Adam Hassick To: ci@dpdk.org Cc: aconole@redhat.com, alialnu@nvidia.com, Adam Hassick Subject: [PATCH v6 0/6] Community Lab Containers and Builder Engine Date: Thu, 25 May 2023 13:14:59 -0400 Message-Id: <20230525171505.1276457-1-ahassick@iol.unh.edu> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org This patch series contains a new version of the DPDK CI containers. The old version was tied very tightly to the Community Lab Infrastructure, so it was not suitable for general use. This version is designed to make adding new OSes or OS versions as easy as possible. The minimum functionality can easily be built on any system that can compile DPDK. It includes support for building containers for other non-native architectures (ex: arm containers on x86) and for baking ABI references into the images. Support for building the Coverity Scan container image has also been added. The inventory file as added in this patch series defines what the community lab currently supports. If you want to build these yourself, don't try to do parallel Makefile builds on non-server hardware. Libabigail is built into the containers, and since it is not avaliable in all distros it is compiled from source for many targets. If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the current settings, DPDK will be compiled thrice for every target in the inventory file. Due to how the containers are constructed, building on non-native architectures is especially expensive because DPDK is compiled with an emulated compiler in a qemu vm that only has one thread. Owen Hilyard (6): containers/docs: Add container builder start containers/inventory: Add inventory for container builder containers/builder: Dockerfile creation script containers/templates: Templates for Dockerfiles containers/container_builder: Container for python scripts containers/Makefile: Makefile to automate builds containers/Makefile | 253 ++++++++++ containers/README.md | 175 +++++++ containers/container_builder.dockerfile | 26 ++ containers/template_engine/inventory.yaml | 442 ++++++++++++++++++ .../template_engine/inventory_schema.json | 221 +++++++++ containers/template_engine/make_dockerfile.py | 358 ++++++++++++++ containers/template_engine/poetry.lock | 220 +++++++++ containers/template_engine/pyproject.toml | 21 + .../templates/containers.makefile.j2 | 73 +++ .../templates/containers/alpine.dockerfile.j2 | 3 + .../templates/containers/arch.dockerfile.j2 | 37 ++ .../templates/containers/base.dockerfile.j2 | 120 +++++ .../containers/centos8.dockerfile.j2 | 21 + .../containers/centos9.dockerfile.j2 | 17 + .../templates/containers/debian.dockerfile.j2 | 7 + .../containers/debian10.dockerfile.j2 | 3 + .../containers/debian11.dockerfile.j2 | 3 + .../debian11_arm_ipsec.dockerfile.j2 | 16 + .../containers/debian_bullseye.dockerfile.j2 | 3 + .../containers/debian_buster.dockerfile.j2 | 3 + .../templates/containers/fedora.dockerfile.j2 | 11 + .../containers/fedora36_clang.dockerfile.j2 | 7 + .../containers/fedora_clang.dockerfile.j2 | 7 + .../containers/fedora_coverity.dockerfile.j2 | 10 + .../containers/opensuse.dockerfile.j2 | 10 + .../containers/redhat_family.dockerfile.j2 | 5 + .../templates/containers/rhel.dockerfile.j2 | 16 + .../templates/containers/rhel7.dockerfile.j2 | 15 + .../templates/containers/rhel8.dockerfile.j2 | 15 + .../templates/containers/rhel9.dockerfile.j2 | 19 + .../templates/containers/rpm.dockerfile.j2 | 3 + .../templates/containers/ubuntu.dockerfile.j2 | 3 + .../containers/ubuntu20.04.dockerfile.j2 | 12 + .../containers/ubuntu22.04.dockerfile.j2 | 3 + .../containers/ubuntu_cross.dockerfile.j2 | 11 + .../containers/ubuntu_sve.dockerfile.j2 | 12 + 36 files changed, 2181 insertions(+) create mode 100644 containers/Makefile create mode 100644 containers/README.md create mode 100644 containers/container_builder.dockerfile create mode 100644 containers/template_engine/inventory.yaml create mode 100644 containers/template_engine/inventory_schema.json create mode 100755 containers/template_engine/make_dockerfile.py create mode 100644 containers/template_engine/poetry.lock create mode 100644 containers/template_engine/pyproject.toml create mode 100644 containers/template_engine/templates/containers.makefile.j2 create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2 -- 2.34.1