From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 13D86A0548 for ; Tue, 11 Oct 2022 18:53:14 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0F09941181; Tue, 11 Oct 2022 18:53:14 +0200 (CEST) Received: from mail-ua1-f97.google.com (mail-ua1-f97.google.com [209.85.222.97]) by mails.dpdk.org (Postfix) with ESMTP id 36415427EE for ; Tue, 11 Oct 2022 18:53:12 +0200 (CEST) Received: by mail-ua1-f97.google.com with SMTP id z6so5246312uaq.2 for ; Tue, 11 Oct 2022 09:53:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=73crBkpbFrGoIpV9hpAYO35h7mbdqX0SXMkHFOp/8os=; b=bNKyhoCDMPyIEj2Km06MEay6bR8PgMPPck8go1QdTTgs8b8VGRm5YFsaMf/s6oIPBw dgngUjVZD4msBSMx8bb8qIyC1rbBhiDRk1Et+9w0OmcehAI5rbfO8Xe2EoBsh3sFk+fB 7SAzKN94jhv5RsJmyIvUB/ogkkzvDtQvXMSGY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=73crBkpbFrGoIpV9hpAYO35h7mbdqX0SXMkHFOp/8os=; b=GL10V4/CDImhDzOBmqPId1PX4wmAFoGrTDSpk9FaJq06MY180AO/PWtunWW8AO7XKm w1jBMdr7zvdFdTywWOzaO3jUVYVNMRNLv9mnl4IbYtkaLmP6zhnuArGM0Y99czn7mDKH ImaApV5m9jaJHV+Yeezh2vRRxG/pQYcuEdco0rfW0Hu4Cx2zsVjJuIBl4FAskl6bWLdI SbwKl7U2sygeu2LTkjJWKNRTx8RZkCdxi9dtU+YKm/YBWI5aeTxTfySYpICsBBGztHm/ cNDLuX60TB96ElxGCgShopD212EZ9+njgRpSeX14+EzPJ3Bck9KA60c3MwwkSKP2J7uQ XQcQ== X-Gm-Message-State: ACrzQf2bb5oUE3h3J2KquLvEN651yNt1XmKFbQelGZ9JBLL3RJTIfJhj 1UMlS3lbwVzKxsfDW6+97mzHOy2aqeunNjAYwsuKAMeRMvNOycr2+WtIOPvWcbl7njQPd0Z7u0o Y7bX6Vfv2QvEzDX0yVJdsl/BxkdhXu12EB5so4i9snG2Y2wS5ra8mvODREOlhC5AQY7cJ8MfyYM wusGPVT2cIZkDi X-Google-Smtp-Source: AMsMyM7SrXay2A8avRTsPUk0tQSBOu7EYmJ2CiMWQ9oO0aUFMVbJ+V0oY+bTAtyKWKIXyWyaj/5JAc1i5n7F X-Received: by 2002:ab0:6027:0:b0:3d9:4b58:1ea4 with SMTP id n7-20020ab06027000000b003d94b581ea4mr13162856ual.26.1665507191517; Tue, 11 Oct 2022 09:53:11 -0700 (PDT) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [2606:4100:3880:1234::84]) by smtp-relay.gmail.com with ESMTPS id k23-20020ab024d7000000b003dd15aa5abdsm1418929uan.0.2022.10.11.09.53.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Oct 2022 09:53:11 -0700 (PDT) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1257::105d]) by postal.iol.unh.edu (Postfix) with ESMTP id 2C5C46052530; Tue, 11 Oct 2022 12:53:05 -0400 (EDT) From: ohilyard@iol.unh.edu To: ci@dpdk.org Cc: aconole@redhat.com, Owen Hilyard Subject: [PATCH 6/6] containers/Makefile: Makefile to automate builds Date: Tue, 11 Oct 2022 12:52:55 -0400 Message-Id: <20221011165255.506428-7-ohilyard@iol.unh.edu> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221011165255.506428-1-ohilyard@iol.unh.edu> References: <20221011165255.506428-1-ohilyard@iol.unh.edu> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org From: Owen Hilyard The Makefile that can be used to build all of the container images using "make build", and can also be used to push them to a remote repository (for use in CI). Signed-off-by: Owen Hilyard --- containers/Makefile | 152 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 containers/Makefile diff --git a/containers/Makefile b/containers/Makefile new file mode 100644 index 0000000..301687e --- /dev/null +++ b/containers/Makefile @@ -0,0 +1,152 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2022 University of New Hampshire + +# Recommended Resources: +# All: +# ~100 GB of disk space: These are not minimal containers, they +# have full distros in them minus the kernel. +# With ABI images, expect 10 GB per distro per +# platform. Without, it's closer to 2 GB per distro +# per platform. +# +# No ABI: +# Serial Build (make -j 1): Should run on any post 2010 system reasonably well. +# Parallel build (make -j $(nproc)): 1 GB of free ram per job should be good enough. +# ABI: +# Time: If you are building more than four distros, or you have one distro that is built under +# emulation (x86 host + arm container or the inverse), it may be best to run it overnight. +# Some versions of qemu + podman will try to compile with a single thread under emulation, +# which can cause what should be an hour-long compile to turn into a 8+ hour compile. +# Serial Build (make -j 1): 6 GB of free ram should be fine +# Parallel build (make -j $(nproc)): 6 GB of free ram per job. You may be compiling ~20 versions of DPDK at the same time, with some under emulation. +# + + +############ Arguments ############# +# Set to 'Y' to override distro detection +DPDK_CI_CONTAINERS_ON_RHEL?=$(shell (test -f /etc/redhat-release && grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') || echo 'N') + +# If set to 'Y' and any container is detected as unbuildable, fail the build. +DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=N + +# If set to 'Y', build +DPDK_CI_CONTAINERS_BUILD_ABI?=N + +# Used to set the python interpreter +DPDK_CI_CONTAINERS_PYTHON3_CMD?=python3.8 + +# The base program to use to build individual containers. +DPDK_CI_CONTAINER_BUILDER_PROGRAM?=podman + +# The url of the git repository to pull libabigail from +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=git://sourceware.org/git/libabigail.git + +# The url of the git repository to pull DPDK from +DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=https://dpdk.org/git/dpdk + +# The tag to apply to the built container builder image +DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder + +# Extra arguments to add to the push command, can be used for credentials +DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?= + +DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=localhost + +# The path to a directory to be recursively copied to $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY). Not used if unset. +# DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH + +DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context +############ End Arguments ############# + +############ Internal Variables ############# +SCRIPT_ARGS=--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)" + +ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y) + SCRIPT_ARGS +=--rhel +endif + +ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y) + SCRIPT_ARGS +=--fail-on-unbuildable +endif + +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) + SCRIPT_ARGS +=--build-abi +endif + +SCRIPT_DIRECTORY=$(CURDIR)/template_engine + +LIBABIGAIL_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabigail +DPDK_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk + +DOCKERFILE=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile + +MAKE_DOCKERFILE_SCRIPT=$(SCRIPT_DIRECTORY)/make_dockerfile.py +MAKE_DOCKERFILE_COMMAND=$(DPDK_CI_CONTAINERS_PYTHON3_CMD) $(MAKE_DOCKERFILE_SCRIPT) + +TEMPLATE_FILE_DIRECTORY=$(SCRIPT_DIRECTORY)/templates +DOCKER_CONTAINER_TEMPLATE_DIR=$(TEMPLATE_FILE_DIRECTORY)/containers + +INVENTORY_FILE=$(SCRIPT_DIRECTORY)/inventory.yaml + +EXTRA_SCRIPTS_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scripts + +GIT_FETCH_ARGS=--all --tags + +CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile + +export +############ End Internal Variables ############# + +.PHONY: build push make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk clean_container_files clean + +build: make_docker_files_in_container extra_scripts + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) build_all + +push: make_docker_files_in_container extra_scripts + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_all + +make_docker_files_in_container: build_builder_container $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run -v $(CURDIR):/container_workspace -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context \ + -e DPDK_CI_CONTAINERS_ON_RHEL=$(DPDK_CI_CONTAINERS_ON_RHEL) \ + -e DPDK_CI_CONTAINERS_BUILD_ABI=$(DPDK_CI_CONTAINERS_BUILD_ABI) \ + -e DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE=$(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE) \ + -e DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME='$(DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME)' \ + $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) make docker_deps + +build_builder_container: + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f $(CONTAINER_BUILDER_DOCKERFILE) -t $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) $(CURDIR) + +docker_deps: $(DOCKERFILE) deps extra_scripts + chmod 666 $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/*.dockerfile + +$(DOCKERFILE): $(INVENTORY_FILE) $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) deps $(MAKE_DOCKERFILE_SCRIPT) $(shell find $(DOCKER_CONTAINER_TEMPLATE_DIR) -type f) + cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFILE_COMMAND) $(SCRIPT_ARGS) + +deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk libabigail + +extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(EXTRA_SCRIPTS_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) +ifdef $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) + cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) $(EXTRA_SCRIPTS_DIRECTORY) +endif + mkdir -p $(EXTRA_SCRIPTS_DIRECTORY) + +# Clone libabigail source +libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + git -C $(LIBABIGAIL_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL) $(LIBABIGAIL_DIRECTORY) + git -C $(LIBABIGAIL_DIRECTORY) describe --tags `git -C $(LIBABIGAIL_DIRECTORY) rev-list --tags --max-count=1` | xargs -n 1 git -C $(LIBABIGAIL_DIRECTORY) checkout + +# Clone DPDK source +dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY) + +$(EXTRA_SCRIPTS_DIRECTORY): + mkdir -p $(EXTRA_SCRIPTS_DIRECTORY) + +$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY): + mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + +clean_container_files: + rm $(DOCKERFILE) + +clean: + rm -rf $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) -- 2.34.1