From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 146D342FD5; Fri, 4 Aug 2023 16:06:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E6A6440EE1; Fri, 4 Aug 2023 16:06:34 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 33F5440E2D for ; Fri, 4 Aug 2023 16:06:34 +0200 (CEST) Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-258-d9Ip7HR5OaiGO6shCIWRIQ-1; Fri, 04 Aug 2023 10:06:30 -0400 X-MC-Unique: d9Ip7HR5OaiGO6shCIWRIQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 45F4838210B1; Fri, 4 Aug 2023 14:06:30 +0000 (UTC) Received: from RHTPC1VM0NT (unknown [10.22.33.16]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B3333492CAC; Fri, 4 Aug 2023 14:06:29 +0000 (UTC) From: Aaron Conole To: Adam Hassick Cc: ci@dpdk.org, alialnu@nvidia.com, Owen Hilyard Subject: Re: [PATCH v8 6/6] containers/Makefile: Makefile to automate builds References: <20230717210815.29737-1-ahassick@iol.unh.edu> <20230717210815.29737-7-ahassick@iol.unh.edu> Date: Fri, 04 Aug 2023 10:06:29 -0400 In-Reply-To: <20230717210815.29737-7-ahassick@iol.unh.edu> (Adam Hassick's message of "Mon, 17 Jul 2023 17:08:15 -0400") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Adam Hassick writes: > From: Owen Hilyard > > The Makefile that can be used to build all of the container images using > "make build", and can also be used to push them to a remote repository > (for use in CI). > > Signed-off-by: Owen Hilyard > Signed-off-by: Adam Hassick > --- > containers/Makefile | 250 ++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 250 insertions(+) > create mode 100644 containers/Makefile > > diff --git a/containers/Makefile b/containers/Makefile > new file mode 100644 > index 0000000..1eec6a0 > --- /dev/null > +++ b/containers/Makefile > @@ -0,0 +1,250 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright (c) 2022 University of New Hampshire 2023? > + > +# Recommended Resources: > +# All: > +# ~100 GB of disk space: These are not minimal containers, they > +# have full distros in them minus the kernel. > +# With ABI images, expect 10 GB per distro per > +# platform. Without, it's closer to 2 GB per distro > +# per platform. > +# > +# No ABI: > +# Serial Build (make -j 1): Should run on any post 2010 system reasonably well. 2010 was 13 years ago, and long before DPDK used IOL. We can probably eliminate this part of the comment. I don't think it makes sense here. > +# Parallel build (make -j $(nproc)): 1 GB of free ram per job should be good enough. > +# ABI: > +# Time: If you are building more than four distros, or you have one distro that is built under > +# emulation (x86 host + arm container or the inverse), it may be best to run it overnight. > +# Some versions of qemu + podman will try to compile with a single thread under emulation, > +# which can cause what should be an hour-long compile to turn into a 8+ hour compile. > +# Serial Build (make -j 1): 6 GB of free ram should be fine > +# Parallel build (make -j $(nproc)): 6 GB of free ram per job. You may be compiling ~20 versions of DPDK at the same time, with some under emulation. > +# > + > + > +############ Arguments ############# > +# Set to 'Y' to override distro detection > +DPDK_CI_CONTAINERS_ON_RHEL?=$(shell (test -f /etc/redhat-release && grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') || echo 'N') > + > +# If set to 'Y' and any container is detected as unbuildable, fail the build. > +DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=N > + > +# If set to 'Y', build the ABIs for specificed DPDK versions and embed them > +# in the container > +DPDK_CI_CONTAINERS_BUILD_ABI?=N > + > +# If set to 'Y', the "latest" tag for all images will be omitted. Intended for testing changes to your inventory. > +DPDK_CI_CONTAINERS_NO_LATEST_TAG?=N > + > +# If set to 'Y', will build containers that rely on the Coverity Scan tool. > +DPDK_CI_CONTAINERS_COVERITY?=N > + > +# The path to Coverity Scan binaries. These will be installed inside the container. > +# This field is only required if the Coverity flag is enabled. > +# DPDK_CI_CONTAINERS_COVERITY_PATH?=/opt/dpdklab/coverity > + > +# If set to a non-empty value, overrides the auto-generated date tag with the value. > +# DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE=70-01-01 > + > +# If set to 'Y', build libabigail from source for distros that do not have it. > +# If DPDK_CI_CONTAINERS_BUILD_ABI='Y', then this is always enabled. > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) > +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=Y > +else > +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL?=N > +endif > + > +# If set to an integer value, this will restrict the count of Ninja workers performing the ABI build to the given integer. > +# The argument is benign if DPDK_CI_CONTANERS_BUILD_ABI is 'N'. > +# DPDK_CI_CONTAINERS_NINJA_WORKERS?=16 > + > +# If set to 'Y', only build containers matching the host architecture > +DPDK_CI_CONTAINERS_ONLY_HOST_ARCH?=N > + > +# If set to 'Y', assumes only host arch and disables the push_manifests target. > +DPDK_CI_CONTAINERS_IS_BUILDER?=N > + > +# Used to set the python interpreter > +DPDK_CI_CONTAINERS_PYTHON3_CMD?=python3.8 > + > +# The base program to use to build individual containers. > +DPDK_CI_CONTAINER_BUILDER_PROGRAM?=podman > + > +# The url of the git repository to pull libabigail from > +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=git://sourceware.org/git/libabigail.git > + > +# The url of the git repository to pull DPDK from > +DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=https://dpdk.org/git/dpdk > + > +# The url of the git repository to pull DPDK stable from > +DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL?=https://dpdk.org/git/dpdk-stable > + > +# The tag to apply to the built container builder image > +DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder > + > +# Extra arguments to add to the push command, can be used for credentials > +DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?= > + > +# Provide the hostname of the registry to push up to > +DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=localhost > + > +# The path to a directory to be recursively copied to $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY). Not used if unset. > +#DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH?= > + > +DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context > +############ End Arguments ############# > + > +############ Internal Variables ############# > +SCRIPT_ARGS=--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)" > + > +ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y) > + SCRIPT_ARGS +=--rhel > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y) > + SCRIPT_ARGS +=--fail-on-unbuildable > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) > + SCRIPT_ARGS +=--build-abi > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL), Y) > + SCRIPT_ARGS +=--build-libabigail > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH), Y) > + SCRIPT_ARGS +=--host-arch-only > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_NO_LATEST_TAG), Y) > + SCRIPT_ARGS +=--omit-latest > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_IS_BUILDER), Y) > + SCRIPT_ARGS +=--builder-mode > +endif > + > +ifneq ($(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE),) > + SCRIPT_ARGS +=--date $(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE) > +endif > + > +ifneq ($(DPDK_CI_CONTAINERS_NINJA_WORKERS),) > + SCRIPT_ARGS +=--ninja-workers $(DPDK_CI_CONTAINERS_NINJA_WORKERS) > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_COVERITY), Y) > + SCRIPT_ARGS +=--coverity > +endif > + > +SCRIPT_DIRECTORY=$(CURDIR)/template_engine > + > +LIBABIGAIL_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabigail > +DPDK_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk > +DPDK_STABLE_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-stable > + > +DOCKERFILE=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile > + > +MAKE_DOCKERFILE_SCRIPT=$(SCRIPT_DIRECTORY)/make_dockerfile.py > +MAKE_DOCKERFILE_COMMAND=$(DPDK_CI_CONTAINERS_PYTHON3_CMD) $(MAKE_DOCKERFILE_SCRIPT) > + > +TEMPLATE_FILE_DIRECTORY=$(SCRIPT_DIRECTORY)/templates > +DOCKER_CONTAINER_TEMPLATE_DIR=$(TEMPLATE_FILE_DIRECTORY)/containers > + > +INVENTORY_FILE=$(SCRIPT_DIRECTORY)/inventory.yaml > + > +EXTRA_SCRIPTS_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scripts > + > +COVERITY_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/coverity > + > +GIT_FETCH_ARGS=--all --tags > + > +CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile > + > +export > +############ End Internal Variables ############# > + > +.PHONY: build push push_images push_manifests make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk coverity clean_container_files clean > + > +build: make_docker_files_in_container external_files > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) build_all > + > +push_images: make_docker_files_in_container external_files > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_image_all > + > +push_manifests: make_docker_files_in_container > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_manifest_all > + > +push: push_images push_manifests > + > +make_docker_files_in_container: build_builder_container $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run --rm -v $(CURDIR):/container_workspace:z -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context:z \ > + -e DPDK_CI_CONTAINERS_ON_RHEL=$(DPDK_CI_CONTAINERS_ON_RHEL) \ > + -e DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=$(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL) \ > + -e DPDK_CI_CONTAINERS_BUILD_ABI=$(DPDK_CI_CONTAINERS_BUILD_ABI) \ > + -e DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE=$(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE) \ > + -e DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME='$(DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME)' \ > + -e DPDK_CI_CONTAINERS_ONLY_HOST_ARCH='$(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH)' \ > + -e DPDK_CI_CONTAINERS_NO_LATEST_TAG='$(DPDK_CI_CONTAINERS_NO_LATEST_TAG)' \ > + -e DPDK_CI_CONTAINERS_IS_BUILDER='$(DPDK_CI_CONTAINERS_IS_BUILDER)' \ > + -e DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE='$(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE)' \ > + -e DPDK_CI_CONTAINERS_NINJA_WORKERS='$(DPDK_CI_CONTAINERS_NINJA_WORKERS)' \ > + -e DPDK_CI_CONTAINERS_COVERITY='$(DPDK_CI_CONTAINERS_COVERITY)' \ > + $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) make docker_deps > + > +external_files: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) extra_scripts coverity > + > +build_builder_container: > + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f $(CONTAINER_BUILDER_DOCKERFILE) -t $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) $(CURDIR) > + > +docker_deps: $(DOCKERFILE) deps extra_scripts > + chmod 666 $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/*.dockerfile > + > +$(DOCKERFILE): $(INVENTORY_FILE) $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) deps $(MAKE_DOCKERFILE_SCRIPT) $(shell find $(DOCKER_CONTAINER_TEMPLATE_DIR) -type f) > + cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFILE_COMMAND) $(SCRIPT_ARGS) > + > +deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk libabigail dpdk-stable > + > +extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(EXTRA_SCRIPTS_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) > +ifdef DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH > + cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)/* $(EXTRA_SCRIPTS_DIRECTORY) > +endif > + > +# Clone libabigail source > +libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(LIBABIGAIL_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL) $(LIBABIGAIL_DIRECTORY) > + git -C $(LIBABIGAIL_DIRECTORY) describe --tags `git -C $(LIBABIGAIL_DIRECTORY) rev-list --tags --max-count=1` | xargs -n 1 git -C $(LIBABIGAIL_DIRECTORY) checkout > + > +# Clone DPDK source > +dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY) > + > +# Copy the Coverity Scan binaries from a source into the context directory. > +coverity: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(COVERITY_DIRECTORY) > +ifeq ($(DPDK_CI_CONTAINERS_COVERITY),Y) > +ifndef DPDK_CI_CONTAINERS_COVERITY_PATH > + exit 1 # The coverity path must be set if building Coverity containers is enabled. > +else > + cp -pur $(DPDK_CI_CONTAINERS_COVERITY_PATH)/* $(COVERITY_DIRECTORY) > +endif > +else > + # Coverity is disabled for this run. > +endif > + > +dpdk-stable: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(DPDK_STABLE_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL) $(DPDK_STABLE_DIRECTORY) > + > +$(EXTRA_SCRIPTS_DIRECTORY): > + mkdir -p $(EXTRA_SCRIPTS_DIRECTORY) > + > +$(COVERITY_DIRECTORY): > + mkdir -p $(COVERITY_DIRECTORY) > + > +$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY): > + mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + > +clean_container_files: > + rm $(DOCKERFILE) > + > +clean: > + rm -rf $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)