From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8C28A42353; Tue, 10 Oct 2023 21:40:41 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 85010402AE; Tue, 10 Oct 2023 21:40:41 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id F338E4028A for ; Tue, 10 Oct 2023 21:40:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1696966839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xC1YkV8wdUaIrGQ4bBcLYzDr3R6GBdbIMJbQs9ymmXM=; b=VzVq2QVLzACRUoS2KE3rmBmLXI+OyAbmjp7AkFi82cdix/x4LdoqZV/uanNQaf0GazCHv0 lOWw7AE66WOgVNtACbn92sA0Nx6Abhi798KN48Y0HvuZKfwpsDtIDcI2st7ko8Jk2RbnSf QXcy05LfZz/grI6wHe4wRqFbfT0MPX8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-149-5xaYhawkO_yeMYclJreTMQ-1; Tue, 10 Oct 2023 15:40:38 -0400 X-MC-Unique: 5xaYhawkO_yeMYclJreTMQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AC6C1858281; Tue, 10 Oct 2023 19:40:37 +0000 (UTC) Received: from RHTPC1VM0NT (unknown [10.22.34.140]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1C3D5C0F788; Tue, 10 Oct 2023 19:40:37 +0000 (UTC) From: Aaron Conole To: Adam Hassick Cc: ci@dpdk.org, alialnu@nvidia.com, Owen Hilyard Subject: Re: [PATCH v9 6/6] containers/Makefile: Makefile to automate builds References: <20230811200018.5650-1-ahassick@iol.unh.edu> <20230811200018.5650-7-ahassick@iol.unh.edu> Date: Tue, 10 Oct 2023 15:40:36 -0400 In-Reply-To: <20230811200018.5650-7-ahassick@iol.unh.edu> (Adam Hassick's message of "Fri, 11 Aug 2023 16:00:18 -0400") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.3 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Adam Hassick writes: > From: Owen Hilyard > > The Makefile that can be used to build all of the container images using > "make build", and can also be used to push them to a remote repository > (for use in CI). > > Signed-off-by: Owen Hilyard > Signed-off-by: Adam Hassick > --- > containers/Makefile | 258 ++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 258 insertions(+) > create mode 100644 containers/Makefile > > diff --git a/containers/Makefile b/containers/Makefile > new file mode 100644 > index 0000000..047a208 > --- /dev/null > +++ b/containers/Makefile > @@ -0,0 +1,258 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright (c) 2023 University of New Hampshire > + > +# Recommended Resources: > +# All: > +# ~100 GB of disk space: These are not minimal containers, they > +# have full distros in them minus the kernel. > +# With ABI images, expect 10 GB per distro per > +# platform. Without, it's closer to 2 GB per distro > +# per platform. > +# > +# No ABI: > +# Serial Build (make -j 1): Should run on a laptop. > +# Parallel build (make -j $(nproc)): 1 GB of free ram per job should be good enough. > +# ABI: > +# Time: If you are building more than four distros, or you have one distro that is built under > +# emulation (x86 host + arm container or the inverse), it may be best to run it overnight. > +# Some versions of qemu + podman will try to compile with a single thread under emulation, > +# which can cause what should be an hour-long compile to turn into a 8+ hour compile. > +# Serial Build (make -j 1): 6 GB of free ram should be fine > +# Parallel build (make -j $(nproc)): 6 GB of free ram per job. You may be compiling ~20 versions of DPDK at the same time, with some under emulation. > +# > + > + > +############ Arguments ############# > +# Set to 'Y' to override distro detection > +DPDK_CI_CONTAINERS_ON_RHEL?=$(shell (test -f /etc/redhat-release && grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') || echo 'N') > + > +# If set to 'Y' and any container is detected as unbuildable, fail the build. > +DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=N > + > +# If set to 'Y', build the ABIs for specificed DPDK versions and embed them If you agree, I can fix this to specified when committing to the repo. > +# in the container > +DPDK_CI_CONTAINERS_BUILD_ABI?=N > + > +# If set to 'Y', the "latest" tag for all images will be omitted. Intended for testing changes to your inventory. > +DPDK_CI_CONTAINERS_NO_LATEST_TAG?=N > + > +# If set to 'Y', will build containers that rely on the Coverity Scan tool. > +DPDK_CI_CONTAINERS_COVERITY?=N > + > +# The path to Coverity Scan binaries. These will be installed inside the container. > +# This field is only required if the Coverity flag is enabled. > +# DPDK_CI_CONTAINERS_COVERITY_PATH?=/opt/dpdklab/coverity > + > +# If set to a non-empty value, overrides the auto-generated date tag with the value. > +# DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE=70-01-01 > + > +# If set to 'Y', build libabigail from source for distros that do not have it. > +# If DPDK_CI_CONTAINERS_BUILD_ABI='Y', then this is always enabled. > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) > +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=Y > +else > +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL?=N > +endif > + > +# If set to an integer value, this will restrict the count of Ninja workers performing the ABI build to the given integer. > +# The argument is benign if DPDK_CI_CONTANERS_BUILD_ABI is 'N'. > +# DPDK_CI_CONTAINERS_NINJA_WORKERS?=16 > + > +# If set to 'Y', only build containers matching the host architecture > +DPDK_CI_CONTAINERS_ONLY_HOST_ARCH?=N > + > +# If set to 'Y', assumes only host arch and disables the push_manifests target. > +DPDK_CI_CONTAINERS_IS_BUILDER?=N > + > +# Used to set the python interpreter > +DPDK_CI_CONTAINERS_PYTHON3_CMD?=python3.8 > + > +# The base program to use to build individual containers. > +DPDK_CI_CONTAINER_BUILDER_PROGRAM?=podman > + > +# The url of the git repository to pull libabigail from > +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=git://sourceware.org/git/libabigail.git > + > +# The url of the git repository to pull DPDK from > +DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=https://dpdk.org/git/dpdk > + > +# The url of the git repository to pull the DPDK CI scripts from. > +DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL?=https://dpdk.org/git/tools/dpdk-ci > + > +# The url of the git repository to pull DPDK stable from > +DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL?=https://dpdk.org/git/dpdk-stable > + > +# The tag to apply to the built container builder image > +DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder > + > +# Extra arguments to add to the push command, can be used for credentials > +DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?= > + > +# Provide the hostname of the registry to push up to > +DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=localhost > + > +# The path to a directory to be recursively copied to $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY). Not used if unset. > +#DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH?= > + > +DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context > +############ End Arguments ############# > + > +############ Internal Variables ############# > +SCRIPT_ARGS=--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)" > + > +ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y) > + SCRIPT_ARGS +=--rhel > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y) > + SCRIPT_ARGS +=--fail-on-unbuildable > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) > + SCRIPT_ARGS +=--build-abi > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL), Y) > + SCRIPT_ARGS +=--build-libabigail > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH), Y) > + SCRIPT_ARGS +=--host-arch-only > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_NO_LATEST_TAG), Y) > + SCRIPT_ARGS +=--omit-latest > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_IS_BUILDER), Y) > + SCRIPT_ARGS +=--builder-mode > +endif > + > +ifneq ($(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE),) > + SCRIPT_ARGS +=--date $(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE) > +endif > + > +ifneq ($(DPDK_CI_CONTAINERS_NINJA_WORKERS),) > + SCRIPT_ARGS +=--ninja-workers $(DPDK_CI_CONTAINERS_NINJA_WORKERS) > +endif > + > +ifeq ($(DPDK_CI_CONTAINERS_COVERITY), Y) > + SCRIPT_ARGS +=--coverity > +endif > + > +SCRIPT_DIRECTORY=$(CURDIR)/template_engine > + > +LIBABIGAIL_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabigail > +DPDK_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk > +DPDK_CI_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-ci > +DPDK_STABLE_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-stable > + > +DOCKERFILE=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile > + > +MAKE_DOCKERFILE_SCRIPT=$(SCRIPT_DIRECTORY)/make_dockerfile.py > +MAKE_DOCKERFILE_COMMAND=$(DPDK_CI_CONTAINERS_PYTHON3_CMD) $(MAKE_DOCKERFILE_SCRIPT) > + > +TEMPLATE_FILE_DIRECTORY=$(SCRIPT_DIRECTORY)/templates > +DOCKER_CONTAINER_TEMPLATE_DIR=$(TEMPLATE_FILE_DIRECTORY)/containers > + > +INVENTORY_FILE=$(SCRIPT_DIRECTORY)/inventory.yaml > + > +EXTRA_SCRIPTS_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scripts > + > +COVERITY_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/coverity > + > +GIT_FETCH_ARGS=--all --tags > + > +CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile > + > +export > +############ End Internal Variables ############# > + > +.PHONY: build push push_images push_manifests make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk dpdk-ci coverity clean_container_files clean > + > +build: make_docker_files_in_container external_files > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) build_all > + > +push_images: make_docker_files_in_container external_files > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_image_all > + > +push_manifests: make_docker_files_in_container > + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_manifest_all > + > +push: push_images push_manifests > + > +make_docker_files_in_container: build_builder_container $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run --rm -v $(CURDIR):/container_workspace:z -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context:z \ > + -e DPDK_CI_CONTAINERS_ON_RHEL=$(DPDK_CI_CONTAINERS_ON_RHEL) \ > + -e DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=$(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL) \ > + -e DPDK_CI_CONTAINERS_BUILD_ABI=$(DPDK_CI_CONTAINERS_BUILD_ABI) \ > + -e DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE=$(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE) \ > + -e DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME='$(DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME)' \ > + -e DPDK_CI_CONTAINERS_ONLY_HOST_ARCH='$(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH)' \ > + -e DPDK_CI_CONTAINERS_NO_LATEST_TAG='$(DPDK_CI_CONTAINERS_NO_LATEST_TAG)' \ > + -e DPDK_CI_CONTAINERS_IS_BUILDER='$(DPDK_CI_CONTAINERS_IS_BUILDER)' \ > + -e DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE='$(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE)' \ > + -e DPDK_CI_CONTAINERS_NINJA_WORKERS='$(DPDK_CI_CONTAINERS_NINJA_WORKERS)' \ > + -e DPDK_CI_CONTAINERS_COVERITY='$(DPDK_CI_CONTAINERS_COVERITY)' \ > + $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) make docker_deps > + > +external_files: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) extra_scripts coverity > + > +build_builder_container: > + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f $(CONTAINER_BUILDER_DOCKERFILE) -t $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) $(CURDIR) > + > +docker_deps: $(DOCKERFILE) deps extra_scripts > + chmod 666 $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/*.dockerfile > + > +$(DOCKERFILE): $(INVENTORY_FILE) $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) deps $(MAKE_DOCKERFILE_SCRIPT) $(shell find $(DOCKER_CONTAINER_TEMPLATE_DIR) -type f) > + cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFILE_COMMAND) $(SCRIPT_ARGS) > + > +deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk dpdk-ci libabigail dpdk-stable > + > +extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(EXTRA_SCRIPTS_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) > +ifdef DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH > + cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)/* $(EXTRA_SCRIPTS_DIRECTORY) > +endif > + > +# Clone libabigail source > +libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(LIBABIGAIL_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL) $(LIBABIGAIL_DIRECTORY) > + git -C $(LIBABIGAIL_DIRECTORY) describe --tags `git -C $(LIBABIGAIL_DIRECTORY) rev-list --tags --max-count=1` | xargs -n 1 git -C $(LIBABIGAIL_DIRECTORY) checkout > + > +# Clone DPDK source > +dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY) > + > +# Clone DPDK CI tools source. > +dpdk-ci: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(DPDK_CI_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL) $(DPDK_CI_DIRECTORY) > + > +# Copy the Coverity Scan binaries from a source into the context directory. > +coverity: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(COVERITY_DIRECTORY) > +ifeq ($(DPDK_CI_CONTAINERS_COVERITY),Y) > +ifndef DPDK_CI_CONTAINERS_COVERITY_PATH > + exit 1 # The coverity path must be set if building Coverity containers is enabled. > +else > + cp -pur $(DPDK_CI_CONTAINERS_COVERITY_PATH)/* $(COVERITY_DIRECTORY) > +endif > +else > + # Coverity is disabled for this run. > +endif > + > +dpdk-stable: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + git -C $(DPDK_STABLE_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL) $(DPDK_STABLE_DIRECTORY) > + > +$(EXTRA_SCRIPTS_DIRECTORY): > + mkdir -p $(EXTRA_SCRIPTS_DIRECTORY) > + > +$(COVERITY_DIRECTORY): > + mkdir -p $(COVERITY_DIRECTORY) > + > +$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY): > + mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) > + > +clean_container_files: > + rm $(DOCKERFILE) > + > +clean: > + rm -rf $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)