[PATCH v5 11/11] containers/docs: Update README

[Adam Hassick <ahassick@iol.unh.edu>]

Updates the README with information on the added arguments
and support for building OCI manifests.

Signed-off-by: Adam Hassick <ahassick@iol.unh.edu>
---
 containers/README.md | 123 +++++++++++++++++++++++++++----------------
 1 file changed, 77 insertions(+), 46 deletions(-)

diff --git a/containers/README.md b/containers/README.md
index 5d01caf..59edfcc 100644
--- a/containers/README.md
+++ b/containers/README.md
@@ -14,73 +14,80 @@ nearly mandatory for this task.
 
 1. Licensed RHEL containers need podman
 
-The build system MUST be able to handle creating properly licensed RHEL
-containers, so that the RHEL CI testing is as accurate as possible.
+The build system MUST be able to handle creating properly licensed RHEL 
+containers, so that the RHEL CI testing is as accurate as possbile. 
 
 2. "Developer Laptop Friendliness"
 
-Another goal of the build system was to enable anyone to easily build the
+Another goal of the build system was to enable anyone to easily build the 
 containers. Not all developers are able to use Linux as the main OS on their
 main development machine. Podman runs on MacOS via podman-machine and Windows
-either by podman-machine or WSL.
+either by podman-machine or WSL. 
 
 3. OCI Containers
 
-OCI containers are more portable than some other container solutions. Much of
+OCI containers are more portable than some other container solutions. Much of 
 the progress on getting containers running on top of FreeBSD jails targets OCI
-containers specifically. The tracking issue for this is
+containers specifically. The tracking issue for this is 
 [https://reviews.freebsd.org/D21570](https://reviews.freebsd.org/D21570).
-Once upstream support happens, there should be a relatively simple path to
+Once upstream support happens, there should be a relatively simple path to 
 supporting containers in FreeBSD once podman/docker APIs are better supported.
-At the moment, lack up upstream support means no support in this project for
-FreeBSD.
+At the moment, lack up upstream support means no support in this project for 
+FreeBSD.  
 
 ### Python and Makefiles instead of Buildah as a library
 
-The next question someone might have is why a combination of Python and
+The next question someone might have is why a combination of Python and 
 makefiles were used instead of using buildah as a library. The largest
 reason is that every DPDK developer is going to need to have some
-level of familiarity with Python due to DTS. Buildah is only available
-as a library via Go, and would tie DPDK to a particular container
+level of familarity with Python due to DTS. Buildah is only avaliable
+as a library via Go, and would tie DPDK to a particular container 
 implementation. Go, while not difficult to learn, is a compiled language,
 meaning that the build system would require a build system.
 
-The other reason is that most of the logic that needs to be performed is very
-simple, and python has a few libraries that do most of the work. If it weren't
-for the desire to have an inventory file (inventory.yaml) with a schema
-(inventory_schema.json), this probably could have been an AWK script. After the
-container images are produced, it is very easy to use the same template
-engine to produce a makefile that can be used to both build and push the
-containers. This makefile can be run with multiple jobs for parallel building
-of containers, something not supported by all compose implementations.
-
-Meson was considered instead of Makefiles, however, Meson does not handle new
-Meson being generated during the build very well, and Meson wants most commands
-to have an output file, which is not true of many of the commands. Meson is
-also more difficult to generate using a templating library than Makefile
-targets.
+The other reason is that most of the logic that needs to be performed is very 
+simple, and python has a few libraries that do most of the work. If it weren't 
+for the desire to have an inventory file (inventory.yaml) with a schema 
+(inventory_schema.json), this probably could have been an AWK script. After the 
+container images are produced, it is very easy to use the same templating 
+engine to produce a makefile that can be used to both build and push the 
+containers. This makefile can be run with multiple jobs for parallel building 
+of containers, something not supported by all compose implementations. 
+
+Meson was considered instead of Makefiles, however, Meson does not handle new 
+Meson being generated during the build very well, and Meson wants most commands 
+to have an output file, which is not true of many of the commands. Meson is 
+also more difficult to generate using a templating library than Makefile 
+targets. 
 
 ## Building
 
 ### Environment Variables
 
-All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any
-issues.
+All environment variables are namespaced to DPDK_CI_CONTAINERS to avoid any 
+issues. 
 
 | Variable                   | Description                                     | Default | Valid Values |
 | -------------------------- | ----------------------------------------------- | ------- | ------------ |
-DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N'
+| DPDK_CI_CONTAINERS_ON_RHEL | Whether you are building on licensed RHEL. RHEL containers must be built on licensed RHEL, this can be used to forcibly enable/disable RHEL containers if automatic detection fails. | (grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') \|\| echo 'N' | 'Y' or 'N'
 DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE | Fail during dockerfile generation if any container in the inventory is not buildable. Currently will cause a failure if you are not on RHEL and try to build RHEL containers. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_ONLY_HOST_ARCH | If set to 'Y', only images for the local system architecture will be built. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_IS_BUILDER | If set to 'Y', disables the manifest features, and only builds images for the local system architecture. Intended to be set when used inside another orchestration sofware. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_NINJA_WORKERS | The number of Ninja workers to use to build ABI images. Variable setting is benign if ABI is disabled. | unset | Any positive integer greater than zero.
 DPDK_CI_CONTAINERS_BUILD_ABI | Whether to bake ABI images into the containers. | 'N' | 'Y' or 'N'
-DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL | Whether to build libabigail from source on distros that do not package it. DPDK_CI_CONTAINERS_BUILD_ABI=Y overrides this to 'Y' | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_NO_LATEST_TAG | Disables tagging the final manifests as "latest" in the local store and remote registry. | 'N' | 'Y' or 'N'
+DPDK_CI_CONTAINERS_COVERITY | Enable building Coverity images. Setting this flag will make the Coverity binaries required. | 'N' | 'Y' or 'N'
 DPDK_CI_CONTAINER_BUILDER_PROGRAM | What container builder program to use. | 'podman' | Any container builder that exposes the same interface and provides the same behavior as podman.
-DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository.
+DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL | What URL to clone libabigail from, since some distros need to compile it from source. | 'git://sourceware.org/git/libabigail.git' | A repository containing libabigail which shares history with the main repository. 
 DPDK_CI_CONTAINERS_DPDK_CLONE_URL | What URL to clone DPDK from. | 'https://dpdk.org/git/dpdk' | Any DPDK mirror.
+DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL | What URL to clone DPDK stable form. | http://dpdk.org/git/dpdk-stable | Any DPDK stable mirror.
 DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG | What tag to give to the container which creates the dockerfiles. The default should be fine unless you have issues with collisions. | 'dpdk_ci_container_builder' | Any valid OCI container tag (A valid C function name will work)
 DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS | Extra arguments to add to the push command, can be used for credentials if 'podman login' won't work. | '' | [https://docs.podman.io/en/latest/markdown/podman-push.1.html#options](https://docs.podman.io/en/latest/markdown/podman-push.1.html#options)
 DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME | The hostname of the registry to push to. | 'localhost' | The hostname of any system exposing an OCI container registry or localhost to push to local storage.
 DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH | The path to a directory to copy into all of the containers at /scripts | unset | The path to any local file directory.
+DPDK_CI_CONTAINERS_COVERITY_PATH | The path to Coverity Scan binaries. Only required of the Coverity flag is set. | unset | The path to any local file directory.
 DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers in. All generated files will be placed in this directory or one of it's children | '$(CURDIR)/container_context' | Any absolute directory path
+DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE | Uses a provided string instead of generating a new date tag. Intended for development use. | unset | Any string that is a valid OCI manifest tag.
 
 ### Builder System Requirements
 
@@ -91,16 +98,15 @@ DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY | Set the directory to build the containers
 * find
 * posix utilities (GNU coreutils will work)
 * bash
-* podman >= 4.0.0 (docker or other container builder programs may work, but are
-unsupported)
-    * podman 4.0.0 allows run mounts, which allow mounting a directory into the build context of a container. This is used to  persist ccache directories for each container.
-* qemu-$ARCH-static for any non-native architecture/revision you want to build for.
+* podman (docker is unsupported, and will NOT work for making manifests)
+* qemu-$ARCH-static for any non-native architecture/revision you want to build 
+for.
 
 #### Hardware
 
-| Hardware Type | Requirement                  | Reason |
-| ------------- | ---------------------------- | ----------------------------------- |
-| Disk space    | 5 GB of disk space per image | Some images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references.
+| Hardware Type | Requirement                   | Reason |
+| ------------- | ----------------------------- | ----------------------------------- |
+| Disk space    | 10 GB of disk space per image | Many of the final images are 4 GB at the moment, and as DPDK's API grows, so will the ABI references. Intermediate images generated by the builds will consume some additional space that is recoverable after the build.
 Memory | Either 1.5x or 2x the memory needed to compile DPDK per makefile job | 1.5x is enough for the container overhead and caching when compiling natively, 2x is for builds under emulation (ARM container on x86, etc).
 
 
@@ -111,28 +117,53 @@ RHEL container images must be built on RHEL.
 ### Build containers locally
 
 ```bash
+# Build using the default arguments
 make build
 ```
 
+The resulting images will be tagged based on the date tag and platform.
+Image generated tags follow this format: `image-{{ platform }}-{{ date_tag }}`
+Where `platform` denotes the platform of the image, and `date_tag` is the generated date tag or the override string provided
+through the environment variable.
+
+They should appear in the local image store on your system.
+
 ### Push containers to registry
 
-This will probably involve following prompts in your terminal, but if you have
-other authentication set up, (LDAP, Kerberos, etc), it may not prompt you.
+This will probably involve following prompts in your terminal, but if you have 
+other authentication set up, (LDAP, Kerberos, etc), it may not prompt you. 
 Logging into a registry is what allows you to upload containers to a remote
-system for others to pull down.
+system for others to pull down. 
 
 If you are working alone, you probably can ignore this and keep the containers
-locally. If you are in an enterprise setting, ask your DevOps or Systems
-Administration team where the preferred location for hosting containers is.
+locally. If you are in an enterprise setting, ask your DevOps or Systems 
+Administration team where the preferred location for hosting containers is. 
 
-Since these images take so long to build, it is recommended to use a container
+Since these images take so long to build, it is recommended to use a container 
 registry and have any CI systems pull from that registry.
 
-Redhat guide to setting up a podman container registry:
+Redhat guide to setting up a podman container registry: 
 [https://www.redhat.com/sysadmin/simple-container-registry](https://www.redhat.com/sysadmin/simple-container-registry)
 
 ```bash
 $DPDK_CI_CONTAINER_BUILDER_PROGRAM login $DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME
 # < Complete login process >
 make push
-```
\ No newline at end of file
+```
+
+#### Manifests
+
+OCI manifests allow the grouping of images for different platforms under the same tag in a repository on a registry.
+The use of OCI manifests over tagged images reduces the amount of system platform related branching in CI scripting.
+
+The Makefile provides the option to push only the images, only the manifests, or push the images and make manifests.
+The default "push" target will perform the last case.
+If you choose to create the manifests, then these will be created with the "final" tags like "latest" and the date timestamp.
+
+Manifest creation is known to not be compatible with Docker.
+This feature is known to work when using Podman to post content to a Docker v2 registry.
+
+The manifests may be created on the registry independently of the image builds using the `push_manifests` target in place of the `push` target.
+In contrast, the `push_images` target will only push the images and not create the manifests.
+
+If the `DPDK_CI_CONTAINERS_IS_BUILDER` variable is set to 'Y', then the `push_manifests` target will be disabled.
\ No newline at end of file
-- 
2.34.1