From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D060042B83 for ; Tue, 23 May 2023 19:04:45 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CDA4442D41; Tue, 23 May 2023 19:04:45 +0200 (CEST) Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by mails.dpdk.org (Postfix) with ESMTP id EECAB42D3E for ; Tue, 23 May 2023 19:04:44 +0200 (CEST) Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-3f6b34d2fdcso11205441cf.1 for ; Tue, 23 May 2023 10:04:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1684861484; x=1687453484; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OkFu8Uvspn2tAN2XAafqZG39pgZ8O3xiZc2IT3sYWB8=; b=C3/zbXIunYIf744ILhEpybWhZgQkK/BVggrKpxX0tuDWaOoqDGuQw35KEUquz2bJUo Iu3lz2aasIh//r3Wne/TRe2ML9/us0htRB5oQWJI7EQ1Acqy5m/PYDNfQU0deuZGQx5x 3I4lExiWG5IEY+7yGmDaXGLKr9gEdvvxn9WvI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684861484; x=1687453484; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OkFu8Uvspn2tAN2XAafqZG39pgZ8O3xiZc2IT3sYWB8=; b=LrKMYuNFlgsYbq9EEan38UaTm7imRvAMEeTvQXQO/R4vYpv0SsR5WIiTnmrGh17z4/ x+2xDZimEvUZfbq4QDBI5/1soY5i5PN2VPh2kTcwnrNI+m06NdV15mMlciAvTgpslHbG scFSEHwPr3vaXc1gI0Ln4WoHGxQ3GhK265cn3J0Lpe5MkhZLc4ZUHOGaFF3BHN8nEYQm /IsNSzjW74hdMF0wDHee8rQFrpz2fxiSp+CTPr+KGL8cU9Q+0Ea+P+pcJFuhvbR7md0x amcBFa7mR5hxqxZ/yFDqPwWh8AXQ52jchccLiZd58gEaZ6tkRq54XCgXG9ypi82CE6nu /u1g== X-Gm-Message-State: AC+VfDwbUVxC02Txzd06i/vteteIU5FaIjwvpR/7EwnuvnCgwH4V0JgF NBRp0K8HgyfVlGruwA9x8Cn0Lkv/rjKb6iCzgRNge/heilIKi3Ke2umEd9H6JfkJBm5n8Cv/waE ViKEsDQbNbbH3FLB0JgsLgfA8bIUzNCI+bXzFKX1RxdEb35r3jEQTbjajyLouIg== X-Google-Smtp-Source: ACHHUZ43OiGeyXjyHsH5jGEKj0UqR60V1tGXzr4AWWZgT16rg4eYnktP5gaHP/+mriIPq+yH14wsuQ== X-Received: by 2002:ac8:4e52:0:b0:3f5:4900:eb6b with SMTP id e18-20020ac84e52000000b003f54900eb6bmr23086567qtw.28.1684861484043; Tue, 23 May 2023 10:04:44 -0700 (PDT) Received: from ahassick-Desktop.iol.unh.edu ([2606:4100:3880:1220:7704:829d:adaa:3294]) by smtp.gmail.com with ESMTPSA id g14-20020ac8774e000000b003e38c9a2a22sm2996553qtu.92.2023.05.23.10.04.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 10:04:43 -0700 (PDT) From: Adam Hassick To: ci@dpdk.org Cc: Adam Hassick , aconole@redhat.com, alialnu@nvidia.com Subject: [PATCH v5 10/11] containers/Makefile: Add new variables and target changes Date: Tue, 23 May 2023 13:04:12 -0400 Message-Id: <20230523170413.812922-11-ahassick@iol.unh.edu> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230523170413.812922-1-ahassick@iol.unh.edu> References: <20230523170413.812922-1-ahassick@iol.unh.edu> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Adds new variables in the Makefile, as well as new defaults. These take advantage of the new arguments added to the Dockerfile template script. New targets were added to push only images, OCI manifests, or both. The new behavior of the "push" target is to push everything. Signed-off-by: Adam Hassick --- containers/Makefile | 116 ++++++++++++++++++++++++++++++++++++++------ 1 file changed, 101 insertions(+), 15 deletions(-) diff --git a/containers/Makefile b/containers/Makefile index 3d56162..714c326 100644 --- a/containers/Makefile +++ b/containers/Makefile @@ -33,6 +33,19 @@ DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=N # in the container DPDK_CI_CONTAINERS_BUILD_ABI?=N +# If set to 'Y', the "latest" tag for all images will be omitted. Intended for testing changes to your inventory. +DPDK_CI_CONTAINERS_NO_LATEST_TAG?=N + +# If set to 'Y', will build containers that rely on the Coverity Scan tool. +DPDK_CI_CONTAINERS_COVERITY?=N + +# The path to Coverity Scan binaries. These will be installed inside the container. +# This field is only required if the Coverity flag is enabled. +# DPDK_CI_CONTAINERS_COVERITY_PATH?=/opt/dpdklab/coverity + +# If set to a non-empty value, overrides the auto-generated date tag with the value. +# DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE=70-01-01 + # If set to 'Y', build libabigail from source for distros that do not have it. # If DPDK_CI_CONTAINERS_BUILD_ABI='Y', then this is always enabled. ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) @@ -41,6 +54,16 @@ else DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL?=N endif +# If set to an integer value, this will restrict the count of Ninja workers performing the ABI build to the given integer. +# The argument is benign if DPDK_CI_CONTANERS_BUILD_ABI is 'N'. +# DPDK_CI_CONTAINERS_NINJA_WORKERS?=16 + +# If set to 'Y', only build containers matching the host architecture +DPDK_CI_CONTAINERS_ONLY_HOST_ARCH?=N + +# If set to 'Y', assumes only host arch and disables the push_manifests target. +DPDK_CI_CONTAINERS_IS_BUILDER?=N + # Used to set the python interpreter DPDK_CI_CONTAINERS_PYTHON3_CMD?=python3.8 @@ -51,7 +74,10 @@ DPDK_CI_CONTAINER_BUILDER_PROGRAM?=podman DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=git://sourceware.org/git/libabigail.git # The url of the git repository to pull DPDK from -DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=https://dpdk.org/git/dpdk +DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=http://dpdk.org/git/dpdk + +# The url of the git repository to pull DPDK stable from +DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL?=http://dpdk.org/git/dpdk-stable # The tag to apply to the built container builder image DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder @@ -59,10 +85,11 @@ DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder # Extra arguments to add to the push command, can be used for credentials DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?= +# Provide the hostname of the registry to push up to DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=localhost # The path to a directory to be recursively copied to $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY). Not used if unset. -# DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH +#DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH?= DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context ############ End Arguments ############# @@ -71,25 +98,50 @@ DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context SCRIPT_ARGS=--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)" ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y) - SCRIPT_ARGS +=--rhel + SCRIPT_ARGS +=--rhel endif ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y) - SCRIPT_ARGS +=--fail-on-unbuildable + SCRIPT_ARGS +=--fail-on-unbuildable endif ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) - SCRIPT_ARGS +=--build-abi + SCRIPT_ARGS +=--build-abi endif ifeq ($(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL), Y) - SCRIPT_ARGS +=--build-libabigail + SCRIPT_ARGS +=--build-libabigail +endif + +ifeq ($(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH), Y) + SCRIPT_ARGS +=--host-arch-only +endif + +ifeq ($(DPDK_CI_CONTAINERS_NO_LATEST_TAG), Y) + SCRIPT_ARGS +=--omit-latest +endif + +ifeq ($(DPDK_CI_CONTAINERS_IS_BUILDER), Y) + SCRIPT_ARGS +=--builder-mode +endif + +ifneq ($(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE),) + SCRIPT_ARGS +=--date $(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE) +endif + +ifneq ($(DPDK_CI_CONTAINERS_NINJA_WORKERS),) + SCRIPT_ARGS +=--ninja-workers $(DPDK_CI_CONTAINERS_NINJA_WORKERS) +endif + +ifeq ($(DPDK_CI_CONTAINERS_COVERITY), Y) + SCRIPT_ARGS +=--coverity endif SCRIPT_DIRECTORY=$(CURDIR)/template_engine LIBABIGAIL_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabigail DPDK_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk +DPDK_STABLE_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-stable DOCKERFILE=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile @@ -103,6 +155,10 @@ INVENTORY_FILE=$(SCRIPT_DIRECTORY)/inventory.yaml EXTRA_SCRIPTS_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scripts +EXTRA_CONFIG_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/configs + +COVERITY_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/coverity + GIT_FETCH_ARGS=--all --tags CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile @@ -110,23 +166,36 @@ CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile export ############ End Internal Variables ############# -.PHONY: build push make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk clean_container_files clean +.PHONY: build push push_images push_manifests make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk coverity clean_container_files clean -build: make_docker_files_in_container extra_scripts +build: make_docker_files_in_container external_files $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) build_all -push: make_docker_files_in_container extra_scripts - $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_all +push_images: make_docker_files_in_container external_files + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_image_all + +push_manifests: make_docker_files_in_container + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_manifest_all + +push: push_images push_manifests make_docker_files_in_container: build_builder_container $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) - $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run -v $(CURDIR):/container_workspace -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context \ + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run --rm -v $(CURDIR):/container_workspace:z -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context:z \ -e DPDK_CI_CONTAINERS_ON_RHEL=$(DPDK_CI_CONTAINERS_ON_RHEL) \ -e DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=$(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL) \ -e DPDK_CI_CONTAINERS_BUILD_ABI=$(DPDK_CI_CONTAINERS_BUILD_ABI) \ -e DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE=$(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE) \ -e DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME='$(DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME)' \ + -e DPDK_CI_CONTAINERS_ONLY_HOST_ARCH='$(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH)' \ + -e DPDK_CI_CONTAINERS_NO_LATEST_TAG='$(DPDK_CI_CONTAINERS_NO_LATEST_TAG)' \ + -e DPDK_CI_CONTAINERS_IS_BUILDER='$(DPDK_CI_CONTAINERS_IS_BUILDER)' \ + -e DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE='$(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE)' \ + -e DPDK_CI_CONTAINERS_NINJA_WORKERS='$(DPDK_CI_CONTAINERS_NINJA_WORKERS)' \ + -e DPDK_CI_CONTAINERS_COVERITY='$(DPDK_CI_CONTAINERS_COVERITY)' \ $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) make docker_deps +external_files: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) extra_scripts coverity + build_builder_container: $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f $(CONTAINER_BUILDER_DOCKERFILE) -t $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) $(CURDIR) @@ -136,13 +205,12 @@ docker_deps: $(DOCKERFILE) deps extra_scripts $(DOCKERFILE): $(INVENTORY_FILE) $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) deps $(MAKE_DOCKERFILE_SCRIPT) $(shell find $(DOCKER_CONTAINER_TEMPLATE_DIR) -type f) cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFILE_COMMAND) $(SCRIPT_ARGS) -deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk libabigail +deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk libabigail dpdk-stable extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(EXTRA_SCRIPTS_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) -ifdef $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) - cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) $(EXTRA_SCRIPTS_DIRECTORY) +ifdef DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH + cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)/* $(EXTRA_SCRIPTS_DIRECTORY) endif - mkdir -p $(EXTRA_SCRIPTS_DIRECTORY) # Clone libabigail source libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) @@ -153,9 +221,27 @@ libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY) +# Copy the Coveirty Scan binaries from a source into the context directory. +coverity: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(COVERITY_DIRECTORY) +ifeq ($(DPDK_CI_CONTAINERS_COVERITY),Y) +ifndef DPDK_CI_CONTAINERS_COVERITY_PATH + exit 1 # The coverity path must be set if building Coverity containers is enabled. +else + cp -pur $(DPDK_CI_CONTAINERS_COVERITY_PATH)/* $(COVERITY_DIRECTORY) +endif +else + # Coverity is disabled for this run. +endif + +dpdk-stable: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + git -C $(DPDK_STABLE_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL) $(DPDK_STABLE_DIRECTORY) + $(EXTRA_SCRIPTS_DIRECTORY): mkdir -p $(EXTRA_SCRIPTS_DIRECTORY) +$(COVERITY_DIRECTORY): + mkdir -p $(COVERITY_DIRECTORY) + $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY): mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) -- 2.34.1