From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 082FCA00C5 for ; Mon, 31 Oct 2022 15:16:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AE14140DDD; Mon, 31 Oct 2022 15:16:25 +0100 (CET) Received: from mail-io1-f97.google.com (mail-io1-f97.google.com [209.85.166.97]) by mails.dpdk.org (Postfix) with ESMTP id 2F30F40223 for ; Mon, 31 Oct 2022 15:16:24 +0100 (CET) Received: by mail-io1-f97.google.com with SMTP id p184so9813381iof.11 for ; Mon, 31 Oct 2022 07:16:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rk8GebkC8a70P516iiQpHcAqGvTMIFweBdQpeBcHeKQ=; b=D97LwJrx4eqI2CHOQfvTzdAwQw2wxU/SSaKtGtZIX23PUxjc5eCacMmtBa/cp2AZEv D3a3O114iWXrCQumaWBrsM2DFjiicWPRU1T4IGmTBqBW3NmvdW16H88YueUMynjsQFmt hF377Nxz17TxKXkFeHtFuTmNcA08DPWXUCMfk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rk8GebkC8a70P516iiQpHcAqGvTMIFweBdQpeBcHeKQ=; b=nYCaOStx3EgPmvJFFREzryuKZbMPu2ByRGQeDGT/76B61ZvrGvHeTb0/2pc7TGp01E R0vWD/FS5iVhJr/64SBq8gvtoqe07cRjeO9hUZuyFiKJQQzmdV8HM0s8X87StUGDLV+n ShB8djdVyjkCA5qY5pa8AJIOKaPfdI6uffP8oRgVTYJ92sWztyRHA/F2wL7Tesz1hXS4 GINQ+PGptojY+aQcn3zryNCc5X7GKOW6kDSqrZjC+ZvNiD/xMWJ9IKLoL62y8TbXWMlN TQbihR2ApUJ3Vm0X+AmfnSNzX2o9N6oSJY0X7QJd1FWpb4jOkJjcL8hYvdHFCrItMRJb Dw6g== X-Gm-Message-State: ACrzQf08HAw9gVKg5iJt4Ia6Scw2TtNsDKEfRoRDexJYK1EBI5gxSgLZ QcPASI6WUBc9eVP98zcg5hmNi35ENZtyO7t2MdIdptALzA1cj1S1Q+bLf4oLeV7J+LXQfAAYGzQ NQTuaj3gU1bzgMP31G38VUGLofhYFm5fcy3ygC1OR2pa4NKdyHIyo/MArpLSTmvXpzHQXgxnMka i3GoIs1bZo+M9C X-Google-Smtp-Source: AMsMyM5+tQfPDux0Q5hUVv9Q9S1zdsHber3TNciMiErcECIVnl0eCwL+XEunk8M0SgKL/SsFh/Rloe7XANVj X-Received: by 2002:a02:735b:0:b0:375:62da:a8a5 with SMTP id a27-20020a02735b000000b0037562daa8a5mr1731188jae.31.1667225783380; Mon, 31 Oct 2022 07:16:23 -0700 (PDT) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [2606:4100:3880:1234::84]) by smtp-relay.gmail.com with ESMTPS id cn11-20020a0566383a0b00b00365cd033acfsm329825jab.63.2022.10.31.07.16.23 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 31 Oct 2022 07:16:23 -0700 (PDT) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1257::105d]) by postal.iol.unh.edu (Postfix) with ESMTP id B2AA3605C3FB; Mon, 31 Oct 2022 10:16:22 -0400 (EDT) From: ohilyard@iol.unh.edu To: ci@dpdk.org Cc: aconole@redhat.com, alialnu@nvidia.com, Owen Hilyard Subject: [PATCH v4 1/6] containers/docs: Add container builder start Date: Mon, 31 Oct 2022 10:16:15 -0400 Message-Id: <20221031141620.54952-2-ohilyard@iol.unh.edu> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221031141620.54952-1-ohilyard@iol.unh.edu> References: <20221031141620.54952-1-ohilyard@iol.unh.edu> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org From: Owen Hilyard * Add README file for containers * Add pyproject file with required dependencies for building containers This module allows anyone to build the containers used in DPDK CI, and allows the community to contribute container definitions back to DPDK CI. Please read the README for more information, since some functionality is opt-in due to resource requirements. Signed-off-by: Owen Hilyard --- containers/README.md | 138 ++++++++++++++++++++++ containers/template_engine/pyproject.toml | 21 ++++ 2 files changed, 159 insertions(+) create mode 100644 containers/README.md create mode 100644 containers/template_engine/pyproject.toml diff --git a/containers/README.md b/containers/README.md new file mode 100644 index 0000000..5d01caf --- /dev/null +++ b/containers/README.md @@ -0,0 +1,138 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2022 University of New Hampshire + +DPDK CI Container Build System +============================== + +## Rational + +There are a few important factors for why a custom build system was created for +the containers instead of using an existing one. The first was that podman was +nearly mandatory for this task. + +### Why Podman + +1. Licensed RHEL containers need podman + +The build system MUST be able to handle creating properly licensed RHEL +containers, so that the RHEL CI testing is as accurate as possible. + +2. "Developer Laptop Friendliness" + +Another goal of the build system was to enable anyone to easily build the +containers. Not all developers are able to use Linux as the main OS on their +main development machine. Podman runs on MacOS via podman-machine and Windows +either by podman-machine or WSL. + +3. OCI Containers + +OCI containers are more portable than some other container solutions. Much of +the progress on getting containers running on top of FreeBSD jails targets OCI +containers specifically. The tracking issue for this is +[https://reviews.freebsd.org/D21570](https://reviews.freebsd.org/D21570). +Once upstream support happens, there should be a relatively simple path to +supporting containers in FreeBSD once podman/docker APIs are better supported. +At the moment, lack up upstream support means no support in this project for +FreeBSD. + +### Python and Makefiles instead of Buildah as a library + +The next question someone might have is why a combination of Python and +makefiles were used instead of using buildah as a library. The largest +reason is that every DPDK developer is going to need to have some +level of familiarity with Python due to DTS. Buildah is only available +as a library via Go, and would tie DPDK to a particular container +implementation. Go, while not difficult to learn, is a compiled language, +meaning that the build system would require a build system. + +The other reason is that most of the logic that needs to be performed is very +simple, and python has a few libraries that do most of the work. If it weren't +for the desire to have an inventory file (inventory.yaml) with a schema +(inventory_schema.json), this probably could have been an AWK script. After the +container images are produced, it is very easy to use the same template +engine to produce a makefile that can be used to both build and push the +containers. This makefile can be run with multiple jobs for parallel building +of containers, something not supported by all compose implementations. + +Meson was considered instead of Makefiles, however, Meson does not handle new +Meson being generated during the build very well, and Meson wants most commands +to have an output file, which is not true of many of the commands. Meson is +also more difficult to generate using a templating library than Makefile +targets. + +## Building + +### Environment Variables + +All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any +issues. + +| Variable | Description | Default | Valid Values | +| -------------------------- | ----------------------------------------------- | ------- | ------------ | +DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N' +DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE | Fail during dockerfile generation if any container in the inventory is not buildable. Currently will cause a failure if you are not on RHEL and try to build RHEL containers. | 'N' | 'Y' or 'N' +DPDK_CI_CONTAINERS_BUILD_ABI | Whether to bake ABI images into the containers. | 'N' | 'Y' or 'N' +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL | Whether to build libabigail from source on distros that do not package it. DPDK_CI_CONTAINERS_BUILD_ABI=Y overrides this to 'Y' | 'N' | 'Y' or 'N' +DPDK_CI_CONTAINER_BUILDER_PROGRAM | What container builder program to use. | 'podman' | Any container builder that exposes the same interface and provides the same behavior as podman. +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository. +DPDK_CI_CONTAINERS_DPDK_CLONE_URL | What URL to clone DPDK from. | 'https://dpdk.org/git/dpdk' | Any DPDK mirror. +DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG | What tag to give to the container which creates the dockerfiles. The default should be fine unless you have issues with collisions. | 'dpdk_ci_container_builder' | Any valid OCI container tag (A valid C function name will work) +DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS | Extra arguments to add to the push command, can be used for credentials if 'podman login' won't work. | '' | [https://docs.podman.io/en/latest/markdown/podman-push.1.html#options](https://docs.podman.io/en/latest/markdown/podman-push.1.html#options) +DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME | The hostname of the registry to push to. | 'localhost' | The hostname of any system exposing an OCI container registry or localhost to push to local storage. +DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH | The path to a directory to copy into all of the containers at /scripts | unset | The path to any local file directory. +DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers in. All generated files will be placed in this directory or one of it's children | '$(CURDIR)/container_context' | Any absolute directory path + +### Builder System Requirements + +#### Required Programs + +* GNU make (POSIX make may work, but is not supported) +* git +* find +* posix utilities (GNU coreutils will work) +* bash +* podman >= 4.0.0 (docker or other container builder programs may work, but are +unsupported) + * podman 4.0.0 allows run mounts, which allow mounting a directory into the build context of a container. This is used to persist ccache directories for each container. +* qemu-$ARCH-static for any non-native architecture/revision you want to build for. + +#### Hardware + +| Hardware Type | Requirement | Reason | +| ------------- | ---------------------------- | ----------------------------------- | +| Disk space | 5 GB of disk space per image | Some images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references. +Memory | Either 1.5x or 2x the memory needed to compile DPDK per makefile job | 1.5x is enough for the container overhead and caching when compiling natively, 2x is for builds under emulation (ARM container on x86, etc). + + +#### RHEL containers + +RHEL container images must be built on RHEL. + +### Build containers locally + +```bash +make build +``` + +### Push containers to registry + +This will probably involve following prompts in your terminal, but if you have +other authentication set up, (LDAP, Kerberos, etc), it may not prompt you. +Logging into a registry is what allows you to upload containers to a remote +system for others to pull down. + +If you are working alone, you probably can ignore this and keep the containers +locally. If you are in an enterprise setting, ask your DevOps or Systems +Administration team where the preferred location for hosting containers is. + +Since these images take so long to build, it is recommended to use a container +registry and have any CI systems pull from that registry. + +Redhat guide to setting up a podman container registry: +[https://www.redhat.com/sysadmin/simple-container-registry](https://www.redhat.com/sysadmin/simple-container-registry) + +```bash +$DPDK_CI_CONTAINER_BUILDER_PROGRAM login $DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME +# < Complete login process > +make push +``` \ No newline at end of file diff --git a/containers/template_engine/pyproject.toml b/containers/template_engine/pyproject.toml new file mode 100644 index 0000000..f5611ce --- /dev/null +++ b/containers/template_engine/pyproject.toml @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2022 University of New Hampshire + +[tool.poetry] +name = "dpdk_ci_containers" +version = "0.1.0" +description = "" +authors = ["Owen Hilyard "] +license = "BSD-3-Clause" + +[tool.poetry.dependencies] +python = "^3.8" +Jinja2 = "^3.1.2" +jsonschema = "^4.10.0" +PyYAML = "^6.0" + +[tool.poetry.dev-dependencies] + +[build-system] +requires = ["poetry-core>=1.0.0"] +build-backend = "poetry.core.masonry.api" -- 2.34.1