From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 70BF9A0542 for ; Mon, 24 Oct 2022 19:47:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6B89841143; Mon, 24 Oct 2022 19:47:29 +0200 (CEST) Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by mails.dpdk.org (Postfix) with ESMTP id 0446B40A8B for ; Mon, 24 Oct 2022 19:47:28 +0200 (CEST) Received: by mail-pf1-f177.google.com with SMTP id d10so9585389pfh.6 for ; Mon, 24 Oct 2022 10:47:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=VzDwU2dOLxBw2cSEdURCbQLIkRUsS7yEONvZSCWKZuM=; b=PHez+iEGClQ+jv7OC0QhLLCJDWIpc9+Hma5CjgGEFq4tbpbGQ2sNlP4YmHo4Ag48yV F+wgjS+/oc5VuFdHONCqtScxjIGMhhaGGBADltFXLoXut9teNKxrf1rSzPpNuh2IHUnb TO42q0hUVY+O5JOX0r83rosh6GsSLgynEAvQ4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=VzDwU2dOLxBw2cSEdURCbQLIkRUsS7yEONvZSCWKZuM=; b=vjm9R807luHKBOTabsAM3B2IwYN1ACNo1CmR/BkWzS85aRFa2Ogr72ckUwd/T7uYIM Gz3uaaf+95ymDeLwNyPPX8httHjZt/YX6RXjX73xWjJaF7+v3CImGJtTnG4iT6ntkUC/ lAvWoB5SE60KDdgwwTzSA1ZzHx6mQPmrvXXYN4BNK5NlV/dJHg+YRJPcqfs3yn6ath2i UGfIvoNPzWmKe+2+dDXcfoE8ZxLFFHBEVovbzQ2otr60pVKwUT1+B73lYxgiVSoBYWn8 0CSa2yNr7Crhh4v3hdjnw0C7zRTMhfn9BklMsx5isAsEwdbNEw/PgdOMha8Gyk7Yrsyj BZSg== X-Gm-Message-State: ACrzQf3wadE1itEq4xIbFTJLvCe5UbELHQ168LKmRPWavggvokEVTFTW /HtrejgpzG6VNR+TEEANwyW1OXF5c3n3D0I7foy95/nJjpbreA== X-Google-Smtp-Source: AMsMyM4r3Yft05SPSICD1CmV6XULcHMSSGKbHgXJ7ScBBuAibQjjfQNWyptKP8arkT/Hci6EvNbAdpmnxITMns1jY+M= X-Received: by 2002:aa7:91c5:0:b0:56b:e2db:5b75 with SMTP id z5-20020aa791c5000000b0056be2db5b75mr4883486pfa.73.1666633647095; Mon, 24 Oct 2022 10:47:27 -0700 (PDT) MIME-Version: 1.0 References: <20221024174345.15402-1-ohilyard@iol.unh.edu> In-Reply-To: <20221024174345.15402-1-ohilyard@iol.unh.edu> From: Owen Hilyard Date: Mon, 24 Oct 2022 13:46:51 -0400 Message-ID: Subject: Re: [PATCH v3 0/6] Community Lab Container Definitions To: ci@dpdk.org Cc: aconole@redhat.com, alialnu@nvidia.com Content-Type: multipart/alternative; boundary="000000000000819bd805ebcb6297" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --000000000000819bd805ebcb6297 Content-Type: text/plain; charset="UTF-8" On Mon, Oct 24, 2022 at 1:43 PM wrote: > From: Owen Hilyard > > This patch series contains a new version of the DPDK CI containers. The old > version was tied very tightly to the Community Lab Infrastructure, so it > was > not suitable for general use. This version is designed to make adding new > OSes or OS versions as easy as possible. The minimum functionality can > easily > be built on any system that can compile DPDK. It includes support for > building containers for other non-native architectures (ex: arm containers > on x86) and for baking ABI references into the images. > > The inventory file as added in this patch series defines what the > community lab > currently supports. > > If you want to build these yourself, don't try to do parallel Makefile > builds > on non-server hardware. Libabigail is built into the containers, and since > it > is not avaliable in all distros it is compiled from source for many > targets. > If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the > current settings, DPDK will be compiled twice for every target in the > inventory file. This will become three times when DPDK main has ABI > stability > again. Due to how the containers are constructed, building on non-native > architectures is especially expensive because DPDK is compiled with an > emulated > compiler in a qemu vm that only has one thread. > > Owen Hilyard (6): > containers/docs: Add container builder start > containers/inventory: Add inventory for container builder > containers/builder: Dockerfile creation script > containers/templates: Templates for Dockerfiles > containers/container_builder: Container for python scripts > containers/Makefile: Makefile to automate builds > > containers/Makefile | 166 ++++++++ > containers/README.md | 138 +++++++ > containers/container_builder.dockerfile | 26 ++ > containers/template_engine/inventory.yaml | 356 ++++++++++++++++++ > .../template_engine/inventory_schema.json | 153 ++++++++ > containers/template_engine/make_dockerfile.py | 240 ++++++++++++ > containers/template_engine/poetry.lock | 220 +++++++++++ > containers/template_engine/pyproject.toml | 21 ++ > .../templates/containers.makefile.j2 | 31 ++ > .../templates/containers/alpine.dockerfile.j2 | 37 ++ > .../templates/containers/arch.dockerfile.j2 | 37 ++ > .../templates/containers/base.dockerfile.j2 | 85 +++++ > .../containers/centos8.dockerfile.j2 | 21 ++ > .../containers/centos9.dockerfile.j2 | 17 + > .../templates/containers/debian.dockerfile.j2 | 7 + > .../containers/debian10.dockerfile.j2 | 3 + > .../containers/debian11.dockerfile.j2 | 3 + > .../containers/debian_bullseye.dockerfile.j2 | 3 + > .../containers/debian_buster.dockerfile.j2 | 3 + > .../templates/containers/fedora.dockerfile.j2 | 11 + > .../containers/fedora36_clang.dockerfile.j2 | 7 + > .../containers/opensuse.dockerfile.j2 | 10 + > .../containers/redhat_family.dockerfile.j2 | 5 + > .../templates/containers/rhel.dockerfile.j2 | 12 + > .../templates/containers/rhel7.dockerfile.j2 | 19 + > .../templates/containers/rhel8.dockerfile.j2 | 19 + > .../templates/containers/rhel9.dockerfile.j2 | 25 ++ > .../templates/containers/rpm.dockerfile.j2 | 3 + > .../templates/containers/ubuntu.dockerfile.j2 | 3 + > .../containers/ubuntu20.04.dockerfile.j2 | 12 + > .../containers/ubuntu22.04.dockerfile.j2 | 7 + > .../containers/ubuntu_cross.dockerfile.j2 | 11 + > .../containers/ubuntu_sve.dockerfile.j2 | 10 + > 33 files changed, 1721 insertions(+) > create mode 100644 containers/Makefile > create mode 100644 containers/README.md > create mode 100644 containers/container_builder.dockerfile > create mode 100644 containers/template_engine/inventory.yaml > create mode 100644 containers/template_engine/inventory_schema.json > create mode 100755 containers/template_engine/make_dockerfile.py > create mode 100644 containers/template_engine/poetry.lock > create mode 100644 containers/template_engine/pyproject.toml > create mode 100644 > containers/template_engine/templates/containers.makefile.j2 > create mode 100644 > containers/template_engine/templates/containers/alpine.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/arch.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/base.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/centos8.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/centos9.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/debian.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/debian10.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/debian11.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/debian_buster.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/fedora.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/opensuse.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/redhat_family.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/rhel.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/rhel7.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/rhel8.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/rhel9.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/rpm.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/ubuntu.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2 > create mode 100644 > containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2 > > -- > 2.34.1 > Ali, since I've made some functional modifications I would appreciate a retest. It should be the same as before, except that by default libabigail will not be built from source if DPDK_CI_CONTAINERS_BUILD_ABI=N. Owen --000000000000819bd805ebcb6297 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Mon, Oct 24, 2022 at 1:43 PM <ohilyard@iol.unh.edu> wrote:
<= /div>
From: Owen Hilyard <ohilyard@iol.unh.edu>

This patch series contains a new version of the DPDK CI containers. The old=
version was tied very tightly to the Community Lab Infrastructure, so it wa= s
not suitable for general use. This version is designed to make adding new OSes or OS versions as easy as possible. The minimum functionality can easi= ly
be built on any system that can compile DPDK. It includes support for
building containers for other non-native architectures (ex: arm containers<= br> on x86) and for baking ABI references into the images.

The inventory file as added in this patch series defines what the community= lab
currently supports.

If you want to build these yourself, don't try to do parallel Makefile = builds
on non-server hardware. Libabigail is built into the containers, and since = it
is not avaliable in all distros it is compiled from source for many targets= .
If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=3DY), with th= e
current settings, DPDK will be compiled twice for every target in the
inventory file. This will become three times when DPDK main has ABI stabili= ty
again. Due to how the containers are constructed, building on non-native architectures is especially expensive because DPDK is compiled with an emul= ated
compiler in a qemu vm that only has one thread.

Owen Hilyard (6):
=C2=A0 containers/docs: Add container builder start
=C2=A0 containers/inventory: Add inventory for container builder
=C2=A0 containers/builder: Dockerfile creation script
=C2=A0 containers/templates: Templates for Dockerfiles
=C2=A0 containers/container_builder: Container for python scripts
=C2=A0 containers/Makefile: Makefile to automate builds

=C2=A0containers/Makefile=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| 166 ++++++++
=C2=A0containers/README.md=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | 138 +++++++
=C2=A0containers/container_builder.dockerfile=C2=A0 =C2=A0 =C2=A0 =C2=A0|= =C2=A0 26 ++
=C2=A0containers/template_engine/inventory.yaml=C2=A0 =C2=A0 =C2=A0| 356 ++= ++++++++++++++++
=C2=A0.../template_engine/inventory_schema.json=C2=A0 =C2=A0 =C2=A0| 153 ++= ++++++
=C2=A0containers/template_engine/make_dockerfile.py | 240 ++++++++++++
=C2=A0containers/template_engine/poetry.lock=C2=A0 =C2=A0 =C2=A0 =C2=A0 | 2= 20 +++++++++++
=C2=A0containers/template_engine/pyproject.toml=C2=A0 =C2=A0 =C2=A0|=C2=A0 = 21 ++
=C2=A0.../templates/containers.makefile.j2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 31 ++
=C2=A0.../templates/containers/alpine.dockerfile.j2 |=C2=A0 37 ++
=C2=A0.../templates/containers/arch.dockerfile.j2=C2=A0 =C2=A0|=C2=A0 37 ++=
=C2=A0.../templates/containers/base.dockerfile.j2=C2=A0 =C2=A0|=C2=A0 85 ++= +++
=C2=A0.../containers/centos8.dockerfile.j2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 21 ++
=C2=A0.../containers/centos9.dockerfile.j2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 17 +
=C2=A0.../templates/containers/debian.dockerfile.j2 |=C2=A0 =C2=A07 +
=C2=A0.../containers/debian10.dockerfile.j2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0|=C2=A0 =C2=A03 +
=C2=A0.../containers/debian11.dockerfile.j2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0|=C2=A0 =C2=A03 +
=C2=A0.../containers/debian_bullseye.dockerfile.j2=C2=A0 |=C2=A0 =C2=A03 +<= br> =C2=A0.../containers/debian_buster.dockerfile.j2=C2=A0 =C2=A0 |=C2=A0 =C2= =A03 +
=C2=A0.../templates/containers/fedora.dockerfile.j2 |=C2=A0 11 +
=C2=A0.../containers/fedora36_clang.dockerfile.j2=C2=A0 =C2=A0|=C2=A0 =C2= =A07 +
=C2=A0.../containers/opensuse.dockerfile.j2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0|=C2=A0 10 +
=C2=A0.../containers/redhat_family.dockerfile.j2=C2=A0 =C2=A0 |=C2=A0 =C2= =A05 +
=C2=A0.../templates/containers/rhel.dockerfile.j2=C2=A0 =C2=A0|=C2=A0 12 +<= br> =C2=A0.../templates/containers/rhel7.dockerfile.j2=C2=A0 |=C2=A0 19 +
=C2=A0.../templates/containers/rhel8.dockerfile.j2=C2=A0 |=C2=A0 19 +
=C2=A0.../templates/containers/rhel9.dockerfile.j2=C2=A0 |=C2=A0 25 ++
=C2=A0.../templates/containers/rpm.dockerfile.j2=C2=A0 =C2=A0 |=C2=A0 =C2= =A03 +
=C2=A0.../templates/containers/ubuntu.dockerfile.j2 |=C2=A0 =C2=A03 +
=C2=A0.../containers/ubuntu20.04.dockerfile.j2=C2=A0 =C2=A0 =C2=A0 |=C2=A0 = 12 +
=C2=A0.../containers/ubuntu22.04.dockerfile.j2=C2=A0 =C2=A0 =C2=A0 |=C2=A0 = =C2=A07 +
=C2=A0.../containers/ubuntu_cross.dockerfile.j2=C2=A0 =C2=A0 =C2=A0|=C2=A0 = 11 +
=C2=A0.../containers/ubuntu_sve.dockerfile.j2=C2=A0 =C2=A0 =C2=A0 =C2=A0|= =C2=A0 10 +
=C2=A033 files changed, 1721 insertions(+)
=C2=A0create mode 100644 containers/Makefile
=C2=A0create mode 100644 containers/README.md
=C2=A0create mode 100644 containers/container_builder.dockerfile
=C2=A0create mode 100644 containers/template_engine/inventory.yaml
=C2=A0create mode 100644 containers/template_engine/inventory_schema.json =C2=A0create mode 100755 containers/template_engine/make_dockerfile.py
=C2=A0create mode 100644 containers/template_engine/poetry.lock
=C2=A0create mode 100644 containers/template_engine/pyproject.toml
=C2=A0create mode 100644 containers/template_engine/templates/containers.ma= kefile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/al= pine.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/ar= ch.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/ba= se.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/ce= ntos8.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/ce= ntos9.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/de= bian.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/de= bian10.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/de= bian11.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/de= bian_bullseye.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/de= bian_buster.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/fe= dora.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/fe= dora36_clang.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/op= ensuse.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/re= dhat_family.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/rh= el.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/rh= el7.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/rh= el8.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/rh= el9.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/rp= m.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/ub= untu.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/ub= untu20.04.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/ub= untu22.04.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/ub= untu_cross.dockerfile.j2
=C2=A0create mode 100644 containers/template_engine/templates/containers/ub= untu_sve.dockerfile.j2

--
2.34.1

Ali, since I've made some functional mo= difications I would appreciate a retest. It should be the same as before, e= xcept that by default libabigail will not be built from source if DPDK_CI_C= ONTAINERS_BUILD_ABI=3DN.

Owen
--000000000000819bd805ebcb6297--