From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 09B7442354; Tue, 10 Oct 2023 22:07:03 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DAA7E402AE; Tue, 10 Oct 2023 22:07:02 +0200 (CEST) Received: from mail-yb1-f172.google.com (mail-yb1-f172.google.com [209.85.219.172]) by mails.dpdk.org (Postfix) with ESMTP id 46E434028A for ; Tue, 10 Oct 2023 22:07:01 +0200 (CEST) Received: by mail-yb1-f172.google.com with SMTP id 3f1490d57ef6-d818d65f23cso6614252276.3 for ; Tue, 10 Oct 2023 13:07:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1696968420; x=1697573220; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=P6EaMG6Qi8UUWmg11b4YXR3pcBoWo/JOra9Wt/cBO7s=; b=FK4qJrEgJ14qcIelU40cFQXvp7r8DkctiXfsOmbYTr8UzrmxGh1QakD+bW2pCnpWa5 OaDDKp9zvRecRCMplo93fNA5okeVIJ59QSK3sV3FRT7H3N1hopvNm87WVFOqNvF2gQlF HFbuv4ZmJ/flo4Cj9pgLGcAyeaVdWJIicCJBs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696968420; x=1697573220; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=P6EaMG6Qi8UUWmg11b4YXR3pcBoWo/JOra9Wt/cBO7s=; b=nBB8yAGBWhPR+fa8dtQlfQXkE1n539RryGMB1nSFt8hpcutXj8q7kxXGVzzdgox6tH rlhaASRBnT9+R3Wz10vlDMrHUdoeKnWdBH5g+6tY42kiHJ0OjGQVBLHMLDPS7j5GZgT8 HCHnGmgbWTR+Pnj2zMbAPJlUw/NFJbmu9GxiVjuLqcI+Omzr4pKpFDUpXHgflDnn7Iwt 5N9gjEK0p60yJRY+0FcsaeF53AmIJFYbKGvy4LGZJnZRedILV1uiMuf8MFNMeQOLMpFy Vps8MZ3BbWmhODxotHuAMOM15v0HWgYQa0F3bk8ucy+dS+SLO0WvO9ig1tzZBWCL6Tdl wOcw== X-Gm-Message-State: AOJu0YxKUIn3CJORwJk1a4uTlqwhb0t9DakyMmGRi1/n87rkoWKDpMQy gT3sItwphEbbOM6DLk0ZbrTcvDBEODApD/cXF/NLWg== X-Google-Smtp-Source: AGHT+IEeDJXsVbSnJM+EtDdDmie9ZMOX9SnW5vPIX9Cf7zXr8DKFW8SojaxKSRH7DlHNp+sPWAI6EJqBrulfA6UhTH8= X-Received: by 2002:a5b:343:0:b0:d42:d029:ff99 with SMTP id q3-20020a5b0343000000b00d42d029ff99mr16343942ybp.55.1696968420443; Tue, 10 Oct 2023 13:07:00 -0700 (PDT) MIME-Version: 1.0 References: <20230811200018.5650-1-ahassick@iol.unh.edu> <20230811200018.5650-7-ahassick@iol.unh.edu> In-Reply-To: From: Adam Hassick Date: Tue, 10 Oct 2023 16:08:26 -0400 Message-ID: Subject: Re: [PATCH v9 6/6] containers/Makefile: Makefile to automate builds To: Aaron Conole Cc: ci@dpdk.org, alialnu@nvidia.com, Owen Hilyard Content-Type: multipart/alternative; boundary="000000000000e565f40607623faa" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --000000000000e565f40607623faa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Yes, I agree that "specificed" should be corrected to "specified". Thank you for catching this. On Tue, Oct 10, 2023 at 3:40=E2=80=AFPM Aaron Conole w= rote: > Adam Hassick writes: > > > From: Owen Hilyard > > > > The Makefile that can be used to build all of the container images usin= g > > "make build", and can also be used to push them to a remote repository > > (for use in CI). > > > > Signed-off-by: Owen Hilyard > > Signed-off-by: Adam Hassick > > --- > > containers/Makefile | 258 ++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 258 insertions(+) > > create mode 100644 containers/Makefile > > > > diff --git a/containers/Makefile b/containers/Makefile > > new file mode 100644 > > index 0000000..047a208 > > --- /dev/null > > +++ b/containers/Makefile > > @@ -0,0 +1,258 @@ > > +# SPDX-License-Identifier: BSD-3-Clause > > +# Copyright (c) 2023 University of New Hampshire > > + > > +# Recommended Resources: > > +# All: > > +# ~100 GB of disk space: These are not minimal containers, > they > > +# have full > distros in them minus the kernel. > > +# With ABI > images, expect 10 GB per distro per > > +# platform. > Without, it's closer to 2 GB per distro > > +# per platform. > > +# > > +# No ABI: > > +# Serial Build (make -j 1): Should run on a laptop. > > +# Parallel build (make -j $(nproc)): 1 GB of free ram per > job should be good enough. > > +# ABI: > > +# Time: If you are building more than four distros, or you > have one distro that is built under > > +# emulation (x86 host + arm container or the > inverse), it may be best to run it overnight. > > +# Some versions of qemu + podman will try to > compile with a single thread under emulation, > > +# which can cause what should be an hour-long > compile to turn into a 8+ hour compile. > > +# Serial Build (make -j 1): 6 GB of free ram should be fine > > +# Parallel build (make -j $(nproc)): 6 GB of free ram per job. You > may be compiling ~20 versions of DPDK at the same time, with some under > emulation. > > +# > > + > > + > > +############ Arguments ############# > > +# Set to 'Y' to override distro detection > > +DPDK_CI_CONTAINERS_ON_RHEL?=3D$(shell (test -f /etc/redhat-release && > grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') || ec= ho > 'N') > > + > > +# If set to 'Y' and any container is detected as unbuildable, fail the > build. > > +DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=3DN > > + > > +# If set to 'Y', build the ABIs for specificed DPDK versions and embed > them > > If you agree, I can fix this to specified when committing to the repo. > > > +# in the container > > +DPDK_CI_CONTAINERS_BUILD_ABI?=3DN > > + > > +# If set to 'Y', the "latest" tag for all images will be omitted. > Intended for testing changes to your inventory. > > +DPDK_CI_CONTAINERS_NO_LATEST_TAG?=3DN > > + > > +# If set to 'Y', will build containers that rely on the Coverity Scan > tool. > > +DPDK_CI_CONTAINERS_COVERITY?=3DN > > + > > +# The path to Coverity Scan binaries. These will be installed inside > the container. > > +# This field is only required if the Coverity flag is enabled. > > +# DPDK_CI_CONTAINERS_COVERITY_PATH?=3D/opt/dpdklab/coverity > > + > > +# If set to a non-empty value, overrides the auto-generated date tag > with the value. > > +# DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE=3D70-01-01 > > + > > +# If set to 'Y', build libabigail from source for distros that do not > have it. > > +# If DPDK_CI_CONTAINERS_BUILD_ABI=3D'Y', then this is always enabled. > > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) > > +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=3DY > > +else > > +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL?=3DN > > +endif > > + > > +# If set to an integer value, this will restrict the count of Ninja > workers performing the ABI build to the given integer. > > +# The argument is benign if DPDK_CI_CONTANERS_BUILD_ABI is 'N'. > > +# DPDK_CI_CONTAINERS_NINJA_WORKERS?=3D16 > > + > > +# If set to 'Y', only build containers matching the host architecture > > +DPDK_CI_CONTAINERS_ONLY_HOST_ARCH?=3DN > > + > > +# If set to 'Y', assumes only host arch and disables the push_manifest= s > target. > > +DPDK_CI_CONTAINERS_IS_BUILDER?=3DN > > + > > +# Used to set the python interpreter > > +DPDK_CI_CONTAINERS_PYTHON3_CMD?=3Dpython3.8 > > + > > +# The base program to use to build individual containers. > > +DPDK_CI_CONTAINER_BUILDER_PROGRAM?=3Dpodman > > + > > +# The url of the git repository to pull libabigail from > > +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=3Dgit:// > sourceware.org/git/libabigail.git > > + > > +# The url of the git repository to pull DPDK from > > +DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=3Dhttps://dpdk.org/git/dpdk > > + > > +# The url of the git repository to pull the DPDK CI scripts from. > > +DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL?=3D > https://dpdk.org/git/tools/dpdk-ci > > + > > +# The url of the git repository to pull DPDK stable from > > +DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL?=3D > https://dpdk.org/git/dpdk-stable > > + > > +# The tag to apply to the built container builder image > > +DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=3Ddpdk_ci_container_builder > > + > > +# Extra arguments to add to the push command, can be used for > credentials > > +DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?=3D > > + > > +# Provide the hostname of the registry to push up to > > +DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=3Dlocalhost > > + > > +# The path to a directory to be recursively copied to > $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY). Not used if unset. > > +#DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH?=3D > > + > > +DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=3D$(CURDIR)/container_context > > +############ End Arguments ############# > > + > > +############ Internal Variables ############# > > +SCRIPT_ARGS=3D--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)" > > + > > +ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y) > > + SCRIPT_ARGS +=3D--rhel > > +endif > > + > > +ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y) > > + SCRIPT_ARGS +=3D--fail-on-unbuildable > > +endif > > + > > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) > > + SCRIPT_ARGS +=3D--build-abi > > +endif > > + > > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL), Y) > > + SCRIPT_ARGS +=3D--build-libabigail > > +endif > > + > > +ifeq ($(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH), Y) > > + SCRIPT_ARGS +=3D--host-arch-only > > +endif > > + > > +ifeq ($(DPDK_CI_CONTAINERS_NO_LATEST_TAG), Y) > > + SCRIPT_ARGS +=3D--omit-latest > > +endif > > + > > +ifeq ($(DPDK_CI_CONTAINERS_IS_BUILDER), Y) > > + SCRIPT_ARGS +=3D--builder-mode > > +endif > > + > > +ifneq ($(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE),) > > + SCRIPT_ARGS +=3D--date $(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE) > > +endif > > + > > +ifneq ($(DPDK_CI_CONTAINERS_NINJA_WORKERS),) > > + SCRIPT_ARGS +=3D--ninja-workers $(DPDK_CI_CONTAINERS_NINJA_WORKER= S) > > +endif > > + > > +ifeq ($(DPDK_CI_CONTAINERS_COVERITY), Y) > > + SCRIPT_ARGS +=3D--coverity > > +endif > > + > > +SCRIPT_DIRECTORY=3D$(CURDIR)/template_engine > > + > > +LIBABIGAIL_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabig= ail > > +DPDK_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk > > +DPDK_CI_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-ci > > > +DPDK_STABLE_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-sta= ble > > + > > +DOCKERFILE=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile > > + > > +MAKE_DOCKERFILE_SCRIPT=3D$(SCRIPT_DIRECTORY)/make_dockerfile.py > > +MAKE_DOCKERFILE_COMMAND=3D$(DPDK_CI_CONTAINERS_PYTHON3_CMD) > $(MAKE_DOCKERFILE_SCRIPT) > > + > > +TEMPLATE_FILE_DIRECTORY=3D$(SCRIPT_DIRECTORY)/templates > > +DOCKER_CONTAINER_TEMPLATE_DIR=3D$(TEMPLATE_FILE_DIRECTORY)/containers > > + > > +INVENTORY_FILE=3D$(SCRIPT_DIRECTORY)/inventory.yaml > > + > > +EXTRA_SCRIPTS_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scri= pts > > + > > +COVERITY_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/coverity > > + > > +GIT_FETCH_ARGS=3D--all --tags > > + > > +CONTAINER_BUILDER_DOCKERFILE=3D$(CURDIR)/container_builder.dockerfile > > + > > +export > > +############ End Internal Variables ############# > > + > > +.PHONY: build push push_images push_manifests > make_docker_files_in_container build_builder_container docker_deps > extra_scripts libabigail dpdk dpdk-ci coverity clean_container_files clea= n > > + > > +build: make_docker_files_in_container external_files > > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) build_all > > + > > +push_images: make_docker_files_in_container external_files > > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_image_all > > + > > +push_manifests: make_docker_files_in_container > > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > push_manifest_all > > + > > +push: push_images push_manifests > > + > > +make_docker_files_in_container: build_builder_container > $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > > + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run --rm -v > $(CURDIR):/container_workspace:z -v > $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_co= ntext:z > \ > > + -e > DPDK_CI_CONTAINERS_ON_RHEL=3D$(DPDK_CI_CONTAINERS_ON_RHEL) \ > > + -e > DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=3D$(DPDK_CI_CONTAINERS_BUILD_LIBABIGA= IL) \ > > + -e > DPDK_CI_CONTAINERS_BUILD_ABI=3D$(DPDK_CI_CONTAINERS_BUILD_ABI) \ > > + -e > DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE=3D$(DPDK_CI_CONTAINERS_FAIL_ON_UNB= UILDABLE) > \ > > + -e > DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME=3D'$(DPDK_CI_CONTAINERS_REGISTRY_HOS= TNAME)' > \ > > + -e > DPDK_CI_CONTAINERS_ONLY_HOST_ARCH=3D'$(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH)= ' \ > > + -e > DPDK_CI_CONTAINERS_NO_LATEST_TAG=3D'$(DPDK_CI_CONTAINERS_NO_LATEST_TAG)' = \ > > + -e > DPDK_CI_CONTAINERS_IS_BUILDER=3D'$(DPDK_CI_CONTAINERS_IS_BUILDER)' \ > > + -e > DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE=3D'$(DPDK_CI_CONTAINERS_DATE_TAG_OVE= RRIDE)' > \ > > + -e > DPDK_CI_CONTAINERS_NINJA_WORKERS=3D'$(DPDK_CI_CONTAINERS_NINJA_WORKERS)' = \ > > + -e > DPDK_CI_CONTAINERS_COVERITY=3D'$(DPDK_CI_CONTAINERS_COVERITY)' \ > > + $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) make > docker_deps > > + > > +external_files: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) extra_scripts > coverity > > + > > +build_builder_container: > > + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f > $(CONTAINER_BUILDER_DOCKERFILE) -t > $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) $(CURDIR) > > + > > +docker_deps: $(DOCKERFILE) deps extra_scripts > > + chmod 666 $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/*.dockerfile > > + > > +$(DOCKERFILE): $(INVENTORY_FILE) > $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) deps $(MAKE_DOCKERFILE_SCRIPT) > $(shell find $(DOCKER_CONTAINER_TEMPLATE_DIR) -type f) > > + cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFILE_COMMAND) $(SCRIPT_ARG= S) > > + > > +deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk dpdk-ci libabigail > dpdk-stable > > + > > +extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > $(EXTRA_SCRIPTS_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) > > +ifdef DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH > > + cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)/* > $(EXTRA_SCRIPTS_DIRECTORY) > > +endif > > + > > +# Clone libabigail source > > +libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > > + git -C $(LIBABIGAIL_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git > clone $(DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL) $(LIBABIGAIL_DIRECTORY) > > + git -C $(LIBABIGAIL_DIRECTORY) describe --tags `git -C > $(LIBABIGAIL_DIRECTORY) rev-list --tags --max-count=3D1` | xargs -n 1 git= -C > $(LIBABIGAIL_DIRECTORY) checkout > > + > > +# Clone DPDK source > > +dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > > + git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone > $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY) > > + > > +# Clone DPDK CI tools source. > > +dpdk-ci: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > > + git -C $(DPDK_CI_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone > $(DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL) $(DPDK_CI_DIRECTORY) > > + > > +# Copy the Coverity Scan binaries from a source into the context > directory. > > +coverity: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(COVERITY_DIRECTORY= ) > > +ifeq ($(DPDK_CI_CONTAINERS_COVERITY),Y) > > +ifndef DPDK_CI_CONTAINERS_COVERITY_PATH > > + exit 1 # The coverity path must be set if building Coverity > containers is enabled. > > +else > > + cp -pur $(DPDK_CI_CONTAINERS_COVERITY_PATH)/* $(COVERITY_DIRECTOR= Y) > > +endif > > +else > > + # Coverity is disabled for this run. > > +endif > > + > > +dpdk-stable: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > > + git -C $(DPDK_STABLE_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git > clone $(DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL) $(DPDK_STABLE_DIRECTORY= ) > > + > > +$(EXTRA_SCRIPTS_DIRECTORY): > > + mkdir -p $(EXTRA_SCRIPTS_DIRECTORY) > > + > > +$(COVERITY_DIRECTORY): > > + mkdir -p $(COVERITY_DIRECTORY) > > + > > +$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY): > > + mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > > + > > +clean_container_files: > > + rm $(DOCKERFILE) > > + > > +clean: > > + rm -rf $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > > --=20 *Adam Hassick* Senior Developer UNH InterOperability Lab ahassick@iol.unh.edu iol.unh.edu +1 (603) 475-8248 --000000000000e565f40607623faa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Yes, I agree that "specificed" should be correct= ed to "specified". Thank you for catching this.

On Tue, Oct 10= , 2023 at 3:40=E2=80=AFPM Aaron Conole <aconole@redhat.com> wrote:
Adam Hassick <ahassick@iol.unh.edu> writes:

> From: Owen Hilyard <ohilyard@iol.unh.edu>
>
> The Makefile that can be used to build all of the container images usi= ng
> "make build", and can also be used to push them to a remote = repository
> (for use in CI).
>
> Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
> Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
> ---
>=C2=A0 containers/Makefile | 258 ++++++++++++++++++++++++++++++++++++++= ++++++
>=C2=A0 1 file changed, 258 insertions(+)
>=C2=A0 create mode 100644 containers/Makefile
>
> diff --git a/containers/Makefile b/containers/Makefile
> new file mode 100644
> index 0000000..047a208
> --- /dev/null
> +++ b/containers/Makefile
> @@ -0,0 +1,258 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2023 University of New Hampshire
> +
> +# Recommended Resources:
> +#=C2=A0 =C2=A0 All:
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ~100 GB of disk space: The= se are not minimal containers, they
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0have full distros in= them minus the kernel.
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0With ABI images, exp= ect 10 GB per distro per
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0platform. Without, i= t's closer to 2 GB per distro
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0per platform.
> +#
> +#=C2=A0 =C2=A0 No ABI:
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Serial Build (make -j 1): = Should run on a laptop.
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Parallel build (make -j $(= nproc)): 1 GB of free ram per job should be good enough.
> +#=C2=A0 =C2=A0 ABI:
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Time: If you are building = more than four distros, or you have one distro that is built under
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 emulation (x86 host + arm container or the inverse), it may be b= est to run it overnight.
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 Some versions of qemu + podman will try to compile with a single= thread under emulation,
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 which can cause what should be an hour-long compile to turn into= a 8+ hour compile.
> +#=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Serial Build (make -j 1): = 6 GB of free ram should be fine
> +#=C2=A0 =C2=A0 Parallel build (make -j $(nproc)): 6 GB of free ram pe= r job. You may be compiling ~20 versions of DPDK at the same time, with som= e under emulation.
> +#
> +
> +
> +############ Arguments #############
> +# Set to 'Y' to override distro detection
> +DPDK_CI_CONTAINERS_ON_RHEL?=3D$(shell (test -f /etc/redhat-release &a= mp;& grep -q 'Red Hat Enterprise Linux' /etc/redhat-release &am= p;& echo 'Y') || echo 'N')
> +
> +# If set to 'Y' and any container is detected as unbuildable,= fail the build.
> +DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=3DN
> +
> +# If set to 'Y', build the ABIs for specificed DPDK versions = and embed them

If you agree, I can fix this to specified when committing to the repo.

> +# in the container
> +DPDK_CI_CONTAINERS_BUILD_ABI?=3DN
> +
> +# If set to 'Y', the "latest" tag for all images wi= ll be omitted. Intended for testing changes to your inventory.
> +DPDK_CI_CONTAINERS_NO_LATEST_TAG?=3DN
> +
> +# If set to 'Y', will build containers that rely on the Cover= ity Scan tool.
> +DPDK_CI_CONTAINERS_COVERITY?=3DN
> +
> +# The path to Coverity Scan binaries. These will be installed inside = the container.
> +# This field is only required if the Coverity flag is enabled.
> +# DPDK_CI_CONTAINERS_COVERITY_PATH?=3D/opt/dpdklab/coverity
> +
> +# If set to a non-empty value, overrides the auto-generated date tag = with the value.
> +# DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE=3D70-01-01
> +
> +# If set to 'Y', build libabigail from source for distros tha= t do not have it.
> +# If DPDK_CI_CONTAINERS_BUILD_ABI=3D'Y', then this is always = enabled.
> +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y)
> +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=3DY
> +else
> +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL?=3DN
> +endif
> +
> +# If set to an integer value, this will restrict the count of Ninja w= orkers performing the ABI build to the given integer.
> +# The argument is benign if DPDK_CI_CONTANERS_BUILD_ABI is 'N'= ;.
> +# DPDK_CI_CONTAINERS_NINJA_WORKERS?=3D16
> +
> +# If set to 'Y', only build containers matching the host arch= itecture
> +DPDK_CI_CONTAINERS_ONLY_HOST_ARCH?=3DN
> +
> +# If set to 'Y', assumes only host arch and disables the push= _manifests target.
> +DPDK_CI_CONTAINERS_IS_BUILDER?=3DN
> +
> +# Used to set the python interpreter
> +DPDK_CI_CONTAINERS_PYTHON3_CMD?=3Dpython3.8
> +
> +# The base program to use to build individual containers.
> +DPDK_CI_CONTAINER_BUILDER_PROGRAM?=3Dpodman
> +
> +# The url of the git repository to pull libabigail from
> +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=3Dgit://sourc= eware.org/git/libabigail.git
> +
> +# The url of the git repository to pull DPDK from
> +DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=3Dhttps://dpdk.org/git/dpdk > +
> +# The url of the git repository to pull the DPDK CI scripts from.
> +DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL?=3Dhttps://dpdk.org/git= /tools/dpdk-ci
> +
> +# The url of the git repository to pull DPDK stable from
> +DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL?=3Dhttps://dpdk.org/g= it/dpdk-stable
> +
> +# The tag to apply to the built container builder image
> +DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=3Ddpdk_ci_container_builder=
> +
> +# Extra arguments to add to the push command, can be used for credent= ials
> +DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?=3D
> +
> +# Provide the hostname of the registry to push up to
> +DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=3Dlocalhost
> +
> +# The path to a directory to be recursively copied to $(DPDK_CI_CONTA= INERS_CONTEXT_DIRECTORY). Not used if unset.
> +#DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH?=3D
> +
> +DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=3D$(CURDIR)/container_context > +############ End Arguments #############
> +
> +############ Internal Variables #############
> +SCRIPT_ARGS=3D--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTO= RY)"
> +
> +ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--rhel
> +endif
> +
> +ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--fail-on-unbuildable
> +endif
> +
> +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--build-abi
> +endif
> +
> +ifeq ($(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL), Y)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--build-libabigail
> +endif
> +
> +ifeq ($(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH), Y)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--host-arch-only
> +endif
> +
> +ifeq ($(DPDK_CI_CONTAINERS_NO_LATEST_TAG), Y)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--omit-latest
> +endif
> +
> +ifeq ($(DPDK_CI_CONTAINERS_IS_BUILDER), Y)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--builder-mode
> +endif
> +
> +ifneq ($(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE),)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--date $(DPDK_CI_CONTAINERS_DATE_= TAG_OVERRIDE)
> +endif
> +
> +ifneq ($(DPDK_CI_CONTAINERS_NINJA_WORKERS),)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--ninja-workers $(DPDK_CI_CONTAIN= ERS_NINJA_WORKERS)
> +endif
> +
> +ifeq ($(DPDK_CI_CONTAINERS_COVERITY), Y)
> +=C2=A0 =C2=A0 =C2=A0SCRIPT_ARGS +=3D--coverity
> +endif
> +
> +SCRIPT_DIRECTORY=3D$(CURDIR)/template_engine
> +
> +LIBABIGAIL_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabi= gail
> +DPDK_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk
> +DPDK_CI_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-ci > +DPDK_STABLE_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-= stable
> +
> +DOCKERFILE=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile
> +
> +MAKE_DOCKERFILE_SCRIPT=3D$(SCRIPT_DIRECTORY)/make_dockerfile.py
> +MAKE_DOCKERFILE_COMMAND=3D$(DPDK_CI_CONTAINERS_PYTHON3_CMD) $(MAKE_DO= CKERFILE_SCRIPT)
> +
> +TEMPLATE_FILE_DIRECTORY=3D$(SCRIPT_DIRECTORY)/templates
> +DOCKER_CONTAINER_TEMPLATE_DIR=3D$(TEMPLATE_FILE_DIRECTORY)/containers=
> +
> +INVENTORY_FILE=3D$(SCRIPT_DIRECTORY)/inventory.yaml
> +
> +EXTRA_SCRIPTS_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scr= ipts
> +
> +COVERITY_DIRECTORY=3D$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/coverity=
> +
> +GIT_FETCH_ARGS=3D--all --tags
> +
> +CONTAINER_BUILDER_DOCKERFILE=3D$(CURDIR)/container_builder.dockerfile=
> +
> +export
> +############ End Internal Variables #############
> +
> +.PHONY: build push push_images push_manifests make_docker_files_in_co= ntainer build_builder_container docker_deps extra_scripts libabigail dpdk d= pdk-ci coverity clean_container_files clean
> +
> +build: make_docker_files_in_container external_files
> +=C2=A0 =C2=A0 =C2=A0$(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY= ) build_all
> +
> +push_images: make_docker_files_in_container external_files
> +=C2=A0 =C2=A0 =C2=A0$(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY= ) push_image_all
> +
> +push_manifests: make_docker_files_in_container
> +=C2=A0 =C2=A0 =C2=A0$(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY= ) push_manifest_all
> +
> +push: push_images push_manifests
> +
> +make_docker_files_in_container: build_builder_container $(DPDK_CI_CON= TAINERS_CONTEXT_DIRECTORY)
> +=C2=A0 =C2=A0 =C2=A0$(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run --rm -v = $(CURDIR):/container_workspace:z -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)= :/container_workspace/container_context:z \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _ON_RHEL=3D$(DPDK_CI_CONTAINERS_ON_RHEL) \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _BUILD_LIBABIGAIL=3D$(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL) \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _BUILD_ABI=3D$(DPDK_CI_CONTAINERS_BUILD_ABI) \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _FAIL_ON_UNBUILDABLE=3D$(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE) \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _REGISTRY_HOSTNAME=3D'$(DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME)' \ > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _ONLY_HOST_ARCH=3D'$(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH)' \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _NO_LATEST_TAG=3D'$(DPDK_CI_CONTAINERS_NO_LATEST_TAG)' \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _IS_BUILDER=3D'$(DPDK_CI_CONTAINERS_IS_BUILDER)' \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _DATE_TAG_OVERRIDE=3D'$(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE)' \ > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _NINJA_WORKERS=3D'$(DPDK_CI_CONTAINERS_NINJA_WORKERS)' \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-e DPDK_CI_CONTAINERS= _COVERITY=3D'$(DPDK_CI_CONTAINERS_COVERITY)' \
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0$(DPDK_CI_CONTAINERS_= CONTAINER_BUILDER_TAG) make docker_deps
> +
> +external_files: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) extra_scripts= coverity
> +
> +build_builder_container:
> +=C2=A0 =C2=A0 =C2=A0$(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f $(C= ONTAINER_BUILDER_DOCKERFILE) -t $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG)= $(CURDIR)
> +
> +docker_deps: $(DOCKERFILE) deps extra_scripts
> +=C2=A0 =C2=A0 =C2=A0chmod 666 $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)= /*.dockerfile
> +
> +$(DOCKERFILE): $(INVENTORY_FILE) $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTO= RY) deps $(MAKE_DOCKERFILE_SCRIPT) $(shell find $(DOCKER_CONTAINER_TEMPLATE= _DIR) -type f)
> +=C2=A0 =C2=A0 =C2=A0cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFI= LE_COMMAND) $(SCRIPT_ARGS)
> +
> +deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk dpdk-ci libabigail= dpdk-stable
> +
> +extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(EXTRA_SCRIPT= S_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)
> +ifdef DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH
> +=C2=A0 =C2=A0 =C2=A0cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)/= * $(EXTRA_SCRIPTS_DIRECTORY)
> +endif
> +
> +# Clone libabigail source
> +libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
> +=C2=A0 =C2=A0 =C2=A0git -C $(LIBABIGAIL_DIRECTORY) fetch $(GIT_FETCH_= ARGS) || git clone $(DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL) $(LIBABIGAIL_= DIRECTORY)
> +=C2=A0 =C2=A0 =C2=A0git -C $(LIBABIGAIL_DIRECTORY) describe --tags `g= it -C $(LIBABIGAIL_DIRECTORY) rev-list --tags --max-count=3D1` | xargs -n 1= git -C $(LIBABIGAIL_DIRECTORY) checkout
> +
> +# Clone DPDK source
> +dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
> +=C2=A0 =C2=A0 =C2=A0git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) = || git clone $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY)
> +
> +# Clone DPDK CI tools source.
> +dpdk-ci: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
> +=C2=A0 =C2=A0 =C2=A0git -C $(DPDK_CI_DIRECTORY) fetch $(GIT_FETCH_ARG= S) || git clone $(DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL) $(DPDK_CI_DIRECTORY= )
> +
> +# Copy the Coverity Scan binaries from a source into the context dire= ctory.
> +coverity: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(COVERITY_DIRECTOR= Y)
> +ifeq ($(DPDK_CI_CONTAINERS_COVERITY),Y)
> +ifndef DPDK_CI_CONTAINERS_COVERITY_PATH
> +=C2=A0 =C2=A0 =C2=A0exit 1 # The coverity path must be set if buildin= g Coverity containers is enabled.
> +else
> +=C2=A0 =C2=A0 =C2=A0cp -pur $(DPDK_CI_CONTAINERS_COVERITY_PATH)/* $(C= OVERITY_DIRECTORY)
> +endif
> +else
> +=C2=A0 =C2=A0 =C2=A0# Coverity is disabled for this run.
> +endif
> +
> +dpdk-stable: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)
> +=C2=A0 =C2=A0 =C2=A0git -C $(DPDK_STABLE_DIRECTORY) fetch $(GIT_FETCH= _ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL) $(DPDK_STAB= LE_DIRECTORY)
> +
> +$(EXTRA_SCRIPTS_DIRECTORY):
> +=C2=A0 =C2=A0 =C2=A0mkdir -p $(EXTRA_SCRIPTS_DIRECTORY)
> +
> +$(COVERITY_DIRECTORY):
> +=C2=A0 =C2=A0 =C2=A0mkdir -p $(COVERITY_DIRECTORY)
> +
> +$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):
> +=C2=A0 =C2=A0 =C2=A0mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)<= br> > +
> +clean_container_files:
> +=C2=A0 =C2=A0 =C2=A0rm $(DOCKERFILE)
> +
> +clean:
> +=C2=A0 =C2=A0 =C2=A0rm -rf $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)


--
Adam Hassick
Senior Developer
UNH InterOperability Lab
+1 (603) 475-8248
--000000000000e565f40607623faa--