[PATCH v2 1/6] containers/docs: Add container builder start

[ohilyard@iol.unh.edu]

From: Owen Hilyard <ohilyard@iol.unh.edu>

* Add README file for containers
* Add pyproject file with required dependencies for building containers

This module allows anyone to build the containers used in DPDK CI, and
allows the community to contribute container definitions back to DPDK
CI. Please read the README for more information, since some
functionality is opt-in due to resource requirements.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
 containers/README.md                      | 137 ++++++++++++++++++++++
 containers/template_engine/pyproject.toml |  21 ++++
 2 files changed, 158 insertions(+)
 create mode 100644 containers/README.md
 create mode 100644 containers/template_engine/pyproject.toml

diff --git a/containers/README.md b/containers/README.md
new file mode 100644
index 0000000..815fab3
--- /dev/null
+++ b/containers/README.md
@@ -0,0 +1,137 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2022 University of New Hampshire
+
+DPDK CI Container Build System
+==============================
+
+## Rational
+
+There are a few important factors for why a custom build system was created for
+the containers instead of using an existing one. The first was that podman was
+nearly mandatory for this task.
+
+### Why Podman
+
+1. Licensed RHEL containers need podman
+
+The build system MUST be able to handle creating properly licensed RHEL 
+containers, so that the RHEL CI testing is as accurate as possbile. 
+
+2. "Developer Laptop Friendliness"
+
+Another goal of the build system was to enable anyone to easily build the 
+containers. Not all developers are able to use Linux as the main OS on their
+main development machine. Podman runs on MacOS via podman-machine and Windows
+either by podman-machine or WSL. 
+
+3. OCI Containers
+
+OCI containers are more portable than some other container solutions. Much of 
+the progress on getting containers running on top of FreeBSD jails targets OCI
+containers specifically. The tracking issue for this is 
+[https://reviews.freebsd.org/D21570](https://reviews.freebsd.org/D21570).
+Once upstream support happens, there should be a relatively simple path to 
+supporting containers in FreeBSD once podman/docker APIs are better supported.
+At the moment, lack up upstream support means no support in this project for 
+FreeBSD.  
+
+### Python and Makefiles instead of Buildah as a library
+
+The next question someone might have is why a combination of Python and 
+makefiles were used instead of using buildah as a library. The largest
+reason is that every DPDK developer is going to need to have some
+level of familarity with Python due to DTS. Buildah is only avaliable
+as a library via Go, and would tie DPDK to a particular container 
+implementation. Go, while not difficult to learn, is a compiled language,
+meaning that the build system would require a build system.
+
+The other reason is that most of the logic that needs to be performed is very 
+simple, and python has a few libraries that do most of the work. If it weren't 
+for the desire to have an inventory file (inventory.yaml) with a schema 
+(inventory_schema.json), this probably could have been an AWK script. After the 
+container images are produced, it is very easy to use the same templating 
+engine to produce a makefile that can be used to both build and push the 
+containers. This makefile can be run with multiple jobs for parallel building 
+of containers, something not supported by all compose implementations. 
+
+Meson was considered instead of Makefiles, however, Meson does not handle new 
+Meson being generated during the build very well, and Meson wants most commands 
+to have an output file, which is not true of many of the commands. Meson is 
+also more difficult to generate using a templating library than Makefile 
+targets. 
+
+## Building
+
+### Environment Variables
+
+All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any 
+issues. 
+
+| Variable                   | Description                                     | Default | Valid Values |
+| -------------------------- | ----------------------------------------------- | ------- | ------------ |
+| DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE | Fail during dockerfile generation if any container in the inventory is not buildable. Currently will cause a failure if you are not on RHEL and try to build RHEL containers. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_BUILD_ABI | Whether to bake ABI images into the containers. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINER_BUILDER_PROGRAM | What container builder program to use. | 'podman' | Any container builder that exposes the same interface and provides the same behavior as podman.
+DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository. 
+DPDK_CI_CONTAINERS_DPDK_CLONE_URL | What URL to clone DPDK from. | 'https://dpdk.org/git/dpdk' | Any DPDK mirror.
+DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG | What tag to give to the container which creates the dockerfiles. The default should be fine unless you have issues with collisions. | 'dpdk_ci_container_builder' | Any valid OCI container tag (A valid C function name will work)
+DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS | Extra arguments to add to the push command, can be used for credentials if 'podman login' won't work. | '' | [https://docs.podman.io/en/latest/markdown/podman-push.1.html#options](https://docs.podman.io/en/latest/markdown/podman-push.1.html#options)
+DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME | The hostname of the registry to push to. | 'localhost' | The hostname of any system exposing an OCI container registry or localhost to push to local storage.
+DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH | The path to a directory to copy into all of the containers at /scripts | unset | The path to any local file directory.
+DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers in. All generated files will be placed in this directory or one of it's children | '$(CURDIR)/container_context' | Any absolute directory path
+
+### Builder System Requirements
+
+#### Required Programs
+
+* GNU make (POSIX make may work, but is not supported)
+* git
+* find
+* posix utilities (GNU coreutils will work)
+* bash
+* podman (docker or other container builder programs may work, but are 
+unsupported)
+* qemu-$ARCH-static for any non-native architecture/revision you want to build 
+for.
+
+#### Hardware
+
+| Hardware Type | Requirement                  | Reason |
+| ------------- | ---------------------------- | ----------------------------------- |
+| Disk space    | 5 GB of disk space per image | Some images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references.
+Memory | Either 1.5x or 2x the memory needed to compile DPDK per makefile job | 1.5x is enough for the container overhead and caching when compiling natively, 2x is for builds under emulation (ARM container on x86, etc).
+
+
+#### RHEL containers
+
+RHEL container images must be built on RHEL.
+
+### Build containers locally
+
+```bash
+make build
+```
+
+### Push containers to registry
+
+This will probably involve following prompts in your terminal, but if you have 
+other authentication set up, (LDAP, Kerberos, etc), it may not prompt you. 
+Logging into a registry is what allows you to upload containers to a remote
+system for others to pull down. 
+
+If you are working alone, you probably can ignore this and keep the containers
+locally. If you are in an enterprise setting, ask your DevOps or Systems 
+Administration team where the preferred location for hosting containers is. 
+
+Since these images take so long to build, it is recommended to use a container 
+registry and have any CI systems pull from that registry.
+
+Redhat guide to setting up a podman container registry: 
+[https://www.redhat.com/sysadmin/simple-container-registry](https://www.redhat.com/sysadmin/simple-container-registry)
+
+```bash
+$DPDK_CI_CONTAINER_BUILDER_PROGRAM login $DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME
+# < Complete login process >
+make push
+```
\ No newline at end of file
diff --git a/containers/template_engine/pyproject.toml b/containers/template_engine/pyproject.toml
new file mode 100644
index 0000000..f5611ce
--- /dev/null
+++ b/containers/template_engine/pyproject.toml
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2022 University of New Hampshire
+
+[tool.poetry]
+name = "dpdk_ci_containers"
+version = "0.1.0"
+description = ""
+authors = ["Owen Hilyard <ohilyard@iol.unh.edu>"]
+license = "BSD-3-Clause"
+
+[tool.poetry.dependencies]
+python = "^3.8"
+Jinja2 = "^3.1.2"
+jsonschema = "^4.10.0"
+PyYAML = "^6.0"
+
+[tool.poetry.dev-dependencies]
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
-- 
2.34.1