* [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK
@ 2022-04-06 15:31 Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 1/8] dts: merge DTS format.sh " Juraj Linkeš
` (7 more replies)
0 siblings, 8 replies; 9+ messages in thread
From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw)
To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu
Cc: dev, Juraj Linkeš
Various tools and repo config.
Juraj Linkeš (8):
dts: merge DTS format.sh to DPDK
dts: merge DTS pyproject.toml to DPDK
dts: merge DTS requirements.txt to DPDK
dts: merge DTS vm_images/Dockerfile to DPDK
dts: merge DTS vm_images/README.md to DPDK
dts: merge DTS vm_images/create_vm_image.py to DPDK
dts: merge DTS vm_images/make_build_container.sh to DPDK
dts: merge DTS vm_images/network-init.sh to DPDK
dts/format.sh | 35 ++
dts/pyproject.toml | 3 +
dts/requirements.txt | 42 +++
dts/vm_images/Dockerfile | 9 +
dts/vm_images/README.md | 64 ++++
dts/vm_images/create_vm_image.py | 474 ++++++++++++++++++++++++++
dts/vm_images/make_build_container.sh | 16 +
dts/vm_images/network-init.sh | 6 +
8 files changed, 649 insertions(+)
create mode 100755 dts/format.sh
create mode 100644 dts/pyproject.toml
create mode 100644 dts/requirements.txt
create mode 100644 dts/vm_images/Dockerfile
create mode 100644 dts/vm_images/README.md
create mode 100755 dts/vm_images/create_vm_image.py
create mode 100755 dts/vm_images/make_build_container.sh
create mode 100755 dts/vm_images/network-init.sh
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 1/8] dts: merge DTS format.sh to DPDK
2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš
@ 2022-04-06 15:31 ` Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 2/8] dts: merge DTS pyproject.toml " Juraj Linkeš
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw)
To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu
Cc: dev, Juraj Linkeš
---
dts/format.sh | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
create mode 100755 dts/format.sh
diff --git a/dts/format.sh b/dts/format.sh
new file mode 100755
index 0000000000..d35c0bf1da
--- /dev/null
+++ b/dts/format.sh
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+function main() {
+ # The directory to work on is either passed in as argument 1,
+ # or is the current working directory
+ DIRECTORY=${1:-`pwd`}
+ LINE_LENGTH=88
+
+ isort \
+ --overwrite-in-place \
+ --profile black \
+ -j `nproc` \
+ --line-length $LINE_LENGTH \
+ --python-version auto \
+ $DIRECTORY
+
+ black \
+ --line-length $LINE_LENGTH \
+ --required-version 22.1.0 \
+ --target-version py38 \
+ --safe \
+ $DIRECTORY
+}
+
+function help() {
+ echo "usage: format.sh <directory>"
+}
+
+if [ "$1" == "-h" ] || [ "$1" == "--help" ]; then
+ help
+ exit 0
+fi
+
+main "$1"
+
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 2/8] dts: merge DTS pyproject.toml to DPDK
2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 1/8] dts: merge DTS format.sh " Juraj Linkeš
@ 2022-04-06 15:31 ` Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 3/8] dts: merge DTS requirements.txt " Juraj Linkeš
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw)
To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu
Cc: dev, Juraj Linkeš
---
dts/pyproject.toml | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 dts/pyproject.toml
diff --git a/dts/pyproject.toml b/dts/pyproject.toml
new file mode 100644
index 0000000000..e5befba31b
--- /dev/null
+++ b/dts/pyproject.toml
@@ -0,0 +1,3 @@
+[tool.isort]
+profile = "black"
+
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 3/8] dts: merge DTS requirements.txt to DPDK
2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 1/8] dts: merge DTS format.sh " Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 2/8] dts: merge DTS pyproject.toml " Juraj Linkeš
@ 2022-04-06 15:31 ` Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 4/8] dts: merge DTS vm_images/Dockerfile " Juraj Linkeš
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw)
To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu
Cc: dev, Juraj Linkeš
---
dts/requirements.txt | 42 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
create mode 100644 dts/requirements.txt
diff --git a/dts/requirements.txt b/dts/requirements.txt
new file mode 100644
index 0000000000..f58a64defa
--- /dev/null
+++ b/dts/requirements.txt
@@ -0,0 +1,42 @@
+# BSD LICENSE
+#
+# Copyright(c) 2019 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+xlwt==1.3.0
+pexpect==4.7.0
+numpy==1.18.5
+docutils
+pcapy
+xlrd
+scapy==2.4.4
+threadpool
+isort
+texttable
\ No newline at end of file
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 4/8] dts: merge DTS vm_images/Dockerfile to DPDK
2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš
` (2 preceding siblings ...)
2022-04-06 15:31 ` [RFC PATCH v1 3/8] dts: merge DTS requirements.txt " Juraj Linkeš
@ 2022-04-06 15:31 ` Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 5/8] dts: merge DTS vm_images/README.md " Juraj Linkeš
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw)
To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu
Cc: dev, Juraj Linkeš
---
dts/vm_images/Dockerfile | 9 +++++++++
1 file changed, 9 insertions(+)
create mode 100644 dts/vm_images/Dockerfile
diff --git a/dts/vm_images/Dockerfile b/dts/vm_images/Dockerfile
new file mode 100644
index 0000000000..ce4dbca44a
--- /dev/null
+++ b/dts/vm_images/Dockerfile
@@ -0,0 +1,9 @@
+FROM ubuntu:20.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && apt-get upgrade -y
+
+RUN apt-get install --no-install-recommends -y libguestfs-tools \
+ qemu linux-image-generic qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils \
+ linux-image-unsigned-5.11.0-46-generic qemu-system-x86
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 5/8] dts: merge DTS vm_images/README.md to DPDK
2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš
` (3 preceding siblings ...)
2022-04-06 15:31 ` [RFC PATCH v1 4/8] dts: merge DTS vm_images/Dockerfile " Juraj Linkeš
@ 2022-04-06 15:31 ` Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 6/8] dts: merge DTS vm_images/create_vm_image.py " Juraj Linkeš
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw)
To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu
Cc: dev, Juraj Linkeš
---
dts/vm_images/README.md | 64 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 64 insertions(+)
create mode 100644 dts/vm_images/README.md
diff --git a/dts/vm_images/README.md b/dts/vm_images/README.md
new file mode 100644
index 0000000000..1a59810d43
--- /dev/null
+++ b/dts/vm_images/README.md
@@ -0,0 +1,64 @@
+# DTS VM Images
+
+This folder contains utilities to create VM
+images for use in virtio testing.
+
+## Host Requirements
+
+The host MUST have qemu/kvm with libvirtd installed
+and set up.
+
+The host MUST be the same architecture as the VM
+you are building.
+
+The host MUST have podman and either docker or have podman
+aliased as docker (running "docker" calls podman).
+
+## Creating a VM
+
+Use the "create_vm_image.py" script to create the vm image.
+If you do not have the required containers on your system,
+it will build them.
+
+The root password it asks for is what to set the VM's
+root password to, not the root password of the system
+you run the script on.
+
+``` --debug ``` will enable debug output from guestfs
+tools. This produces a lot of output and you shouldn't
+use it unless something is going wrong.
+
+The base image MUST be a "cloud ready" or "prebuilt"
+image, meaning you cannot use an installer ISO. It also
+must be in the qcow2 format, (use qemu-img to convert it).
+Most distros will have a "cloud image" which is in the
+correct format. This base image will not be modified
+by the build script.
+
+The output image is where all of the modifications go and
+it is the image that you should use with DTS.
+
+## Supported Distros
+
+Currently, only RHEL 8 family distros and Ubuntu 20.04 are
+supported. Debian might work, but it is untested. Most
+testing has gone to Ubuntu 20.04.
+
+## Architectures
+
+Due to the way that guestfs tools work, they must run
+under kvm, but the host needs to have a kernel image
+that can be used to boot the VM. It may be possible
+to work around this issue using containers, but
+several days of experimentation kept running into
+more and more complex issues with the interactions
+between libguestfs and docker/podman. As such,
+your best bet is to build your VMs on either a
+bare-metal system of your desired architecture
+or inside a VM already being emulated as your desired
+architecture. This second approach may run into
+issues with the hypervisor, since not all hypervisors
+support nested virtualization by default. Since you need
+an appropriate kernel image installed as well, it may
+be easiest to build VMs using whatever distro you already
+use for most of your servers.
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 6/8] dts: merge DTS vm_images/create_vm_image.py to DPDK
2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš
` (4 preceding siblings ...)
2022-04-06 15:31 ` [RFC PATCH v1 5/8] dts: merge DTS vm_images/README.md " Juraj Linkeš
@ 2022-04-06 15:31 ` Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 7/8] dts: merge DTS vm_images/make_build_container.sh " Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 8/8] dts: merge DTS vm_images/network-init.sh " Juraj Linkeš
7 siblings, 0 replies; 9+ messages in thread
From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw)
To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu
Cc: dev, Juraj Linkeš
---
dts/vm_images/create_vm_image.py | 474 +++++++++++++++++++++++++++++++
1 file changed, 474 insertions(+)
create mode 100755 dts/vm_images/create_vm_image.py
diff --git a/dts/vm_images/create_vm_image.py b/dts/vm_images/create_vm_image.py
new file mode 100755
index 0000000000..8a010c99ac
--- /dev/null
+++ b/dts/vm_images/create_vm_image.py
@@ -0,0 +1,474 @@
+#!/usr/bin/python3
+
+import argparse
+import enum
+import os
+import subprocess
+from sys import stderr
+from typing import List, Optional, Set, Tuple
+import xml.etree.ElementTree as ET
+import platform
+
+DOCKER_IMAGE_NAME = "dts_vm_builder"
+
+
+# From https://libguestfs.org/guestfs.3.html#guestfs_inspect_get_distro
+class OsFamily(enum.Enum):
+ ALPINE = "alpinelinux"
+ ALT = "altlinux"
+ ARCH = "archlinux"
+ BUILDROOT_DERIVED = "buildroot"
+ CENTOS = "centos"
+ CIRROS = "cirros"
+ COREOS = "coreos"
+ DEBIAN = "debian"
+ FEDORA = "fedora"
+ FREEBSD = "freebsd"
+ FREEDOS = "freedos"
+ FRUNGALWARE = "frugalware"
+ GENTOO = "gentoo"
+ KALI = "kalilinux"
+ KYLIN = "kylin"
+ MINT = "linuxmint"
+ MAGEIA = "mageia"
+ MANDRIVA = "mandriva"
+ MEEGO = "meego"
+ MSDOS = "msdos"
+ NEOKYLIN = "neokylin"
+ NETBSD = "netbsd"
+ OPENBSD = "openbsd"
+ OPENMANDRIVA = "openmandriva"
+ OPENSUSE = "opensuse"
+ ORACLE = "oraclelinux"
+ PARDUS = "pardus"
+ PLD = "pldlinux"
+ RHEL_BASED = "redhat-based"
+ RHEL = "rhel"
+ ROCKY = "rocky"
+ SCIENTIFIC_LINUX = "scientificlinux"
+ SLACKWARE = "slackware"
+ SLES = "sles"
+ SUSE_BASED = "suse-based"
+ TTY_LINUX = "ttylinux"
+ UBUNTU = "ubuntu"
+ VOID = "voidlinux"
+ WINDOWS = "windows"
+
+ UNKNOWN = "unknown"
+
+ def __str__(self):
+ return self.value
+
+
+# The Os Families that are supported
+SUPPORTED_OS_FAMILIES = {
+ OsFamily.CENTOS,
+ OsFamily.DEBIAN,
+ OsFamily.FEDORA,
+ OsFamily.RHEL_BASED,
+ OsFamily.RHEL,
+ OsFamily.UBUNTU,
+}
+
+
+# From https://libguestfs.org/guestfs.3.html#guestfs_file_architecture
+class Arch(enum.Enum):
+ aarch64 = "aarch64"
+ i386 = "i386"
+ ia64 = "ia64"
+ ppc = "ppc"
+ ppc64 = "ppc64"
+ ppc64le = "ppc64le"
+ riscv32 = "riscv32"
+ riscv64 = "riscv64"
+ riscv128 = "riscv128"
+ s390 = "s390"
+ s390x = "s390x"
+ sparc = "sparc"
+ sparc64 = "sparc64"
+ x86_64 = "x86_64"
+
+ def __str__(self):
+ return self.value
+
+
+# The supported architectures
+SUPPORTED_ARCHITECTURES = {Arch.x86_64, Arch.aarch64, Arch.ppc64}
+
+
+def validate_filepath(parser: argparse.ArgumentParser, filepath: str) -> str:
+ if not os.path.isabs(filepath):
+ filepath = os.path.abspath(filepath)
+
+ if os.path.exists(filepath):
+ return filepath
+ else:
+ parser.error(f"Path {filepath} not found")
+
+
+def parse_arguments() -> argparse.Namespace:
+ parser = argparse.ArgumentParser()
+
+ # Base image file
+ parser.add_argument("base_image", type=lambda f: validate_filepath(parser, f))
+
+ # Where to write the vm image to
+ parser.add_argument("output_path")
+
+ # What to set the root password to
+ parser.add_argument(
+ "--root_password", help="The new root password for the vm", default="dts"
+ )
+
+ # Whether to run virt-customize in debug mode
+ parser.add_argument("--debug", action="store_true", default=False)
+
+ return parser.parse_args()
+
+
+def run_subprocess(
+ os_family_tags: Set[OsFamily],
+ base_image_path: str,
+ output_path: str,
+ root_password: str,
+ debug_mode: bool,
+ arch: Arch,
+):
+ copy_base_image_to_output_path(base_image_path, output_path)
+
+ print("Building under emulation")
+
+ # Check if the docker container already exists
+ docker_process = subprocess.run(
+ f"docker image ls {DOCKER_IMAGE_NAME}", capture_output=True, shell=True
+ )
+
+ if docker_process.returncode != 0:
+ error("Unable to check for presence of docker image")
+
+ if not len(docker_process.stdout.splitlines()) >= 2: # image does not exist
+ subprocess.run(f"./make_build_container.sh")
+
+ docker_command = [
+ "docker",
+ "run",
+ # The container needs to access QEMU/KVM
+ "--privileged",
+ "-d",
+ "--platform",
+ ]
+
+ if arch == Arch.x86_64:
+ docker_command += ("linux/amd64",)
+ elif arch == Arch.ppc64le:
+ docker_command += ("linux/ppc64le",)
+ elif arch == Arch.aarch64:
+ docker_command += ("linux/arm64",)
+ else:
+ error(f"Please add {arch} to the if chain selecting the docker platform")
+
+ docker_command += ("-v $(pwd):/vm_folder",)
+
+ if debug_mode:
+ docker_command += (
+ "-e",
+ "LIBGUESTFS_DEBUG=1",
+ "-e",
+ "LIBGUESTFS_TRACE=1",
+ )
+
+ # Run cat so it doesn't terminate until we stop it
+ docker_command += f"-it {DOCKER_IMAGE_NAME}:{arch}", "cat"
+
+ # if debug_mode:
+ print("Running:")
+ print(" ".join(docker_command))
+ print("\n\n")
+
+ docker_process = subprocess.run(
+ " ".join(docker_command), shell=True, capture_output=True
+ )
+
+ if docker_process.returncode != 0:
+ print(docker_process.stderr)
+ print(docker_process.stdout)
+ error("Unable to run docker container, try --debug")
+
+ container_id = docker_process.stdout.strip().decode()
+
+ if debug_mode:
+ print(f"Docker container is {container_id}")
+
+ virt_customize_command = get_virt_customize_command(
+ os_family_tags, output_path, root_password
+ )
+
+ vm_build_command = ["docker", "exec", "-w", "/vm_folder"]
+
+ if debug_mode:
+ vm_build_command += (
+ "-e",
+ "LIBGUESTFS_DEBUG=1",
+ "-e",
+ "LIBGUESTFS_TRACE=1",
+ )
+
+ vm_build_command += (
+ "-it",
+ container_id,
+ )
+
+ vm_build_command += (virt_customize_command,)
+
+ # if debug_mode:
+ print(" ".join(vm_build_command))
+
+ vm_build_process = subprocess.run(" ".join(vm_build_command), shell=True)
+
+ if vm_build_process.returncode == 0:
+ # Shut down the build container
+ subprocess.run(f"docker kill {container_id}", shell=True)
+
+ print(vm_build_process.returncode)
+
+
+def run_command_in_docker_container(
+ container_id: str, command: str, debug_mode: bool, **kwargs
+) -> subprocess.CompletedProcess:
+ docker_command = "docker exec "
+
+ if debug_mode:
+ docker_command += f"-e LIBGUESTFS_DEBUG=1 -e LIBGUESTFS_TRACE=1"
+
+ docker_command += f"-w /vm_folder -t {container_id} {command}"
+ return subprocess.run(docker_command, **kwargs)
+
+
+def copy_base_image_to_output_path(base_image_path: str, output_path: str):
+ real_base_image_path: str = os.path.realpath(base_image_path)
+ real_output_path: str = os.path.realpath(output_path)
+
+ if (
+ real_base_image_path != real_output_path
+ ): # do not copy if they are the same path
+ subprocess.run(
+ ["cp", real_base_image_path, real_output_path],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ )
+
+
+def get_virt_customize_command(
+ os_family_tags: Set[OsFamily], output_path: str, root_password: str
+) -> str:
+ commands = [
+ f"virt-customize -a {output_path} --root-password password:{root_password} --update"
+ ]
+
+ commands = commands + get_enable_additional_repos_commands(os_family_tags)
+
+ packages = get_packages_for_os_family(os_family_tags)
+ packagelist = ",".join(packages)
+ commands += (f"--install {packagelist}",)
+ commands += (f"--run-command {get_install_meson_command(os_family_tags)}",)
+ commands += (f"--run-command {get_setup_hugepages_command(os_family_tags)}",)
+ commands += (f"--run-command {get_hugepage_mount_command(os_family_tags)}",)
+ commands = commands + get_security_enforcement_disable_command(
+ os_family_tags, output_path
+ )
+ return " ".join(commands)
+
+
+def get_enable_additional_repos_commands(os_family_tags: Set[OsFamily]):
+ if OsFamily.RHEL in os_family_tags and OsFamily.FEDORA not in os_family_tags:
+ packages = [
+ "yum-utils",
+ "epel-release",
+ ]
+
+ packagelist = ",".join(packages)
+
+ return [
+ f"--install {packagelist}",
+ f"--run-command 'yum-config-manager --enable powertools'",
+ ]
+ elif OsFamily.DEBIAN in os_family_tags:
+ return []
+
+
+def get_packages_for_os_family(os_family_tags: Set[OsFamily]) -> List[str]:
+ if OsFamily.DEBIAN in os_family_tags:
+ return [
+ "make",
+ "gcc",
+ "g++",
+ "libc-dev",
+ "libc6-dev",
+ "ninja-build",
+ "pkg-config",
+ "libnuma-dev",
+ "python3-pyelftools",
+ "abigail-tools",
+ "git",
+ "librdmacm-dev",
+ "librdmacm1",
+ "rdma-core",
+ "libelf-dev",
+ "libmnl-dev",
+ "libpcap-dev",
+ "libcrypto++-dev",
+ "libjansson-dev",
+ "libatomic1",
+ "python3-pip",
+ "python3-setuptools",
+ "python3-wheel",
+ "iperf",
+ "chrony",
+ ]
+ elif OsFamily.RHEL in os_family_tags:
+ return [
+ "make",
+ "gcc",
+ "pkg-config",
+ "ninja-build",
+ "numactl-libs",
+ "python3-pyelftools",
+ "libabigail-devel",
+ "git",
+ "librdmacm",
+ "librdmacm-utils",
+ "rdma-core",
+ "elfutils-libelf-devel",
+ "libmnl-devel",
+ "libpcap-devel",
+ "cryptopp-devel",
+ "jansson-devel",
+ "libatomic",
+ "python3-pip",
+ "python3-setuptools",
+ "python3-wheel",
+ ]
+ else:
+ error(f"Unable to get packages for {os_family_tags} OS family.")
+
+
+def get_install_meson_command(os_family_tags: Set[OsFamily]) -> str:
+ if OsFamily.DEBIAN in os_family_tags or OsFamily.RHEL in os_family_tags:
+ # the "--trusted-host" flags are included because the date on the system will be Jan 1, 1970 due to the way
+ # guestfs-tools starts the vm. This breaks pip's ssl, so making these hosts trusted fixes that.
+ return '"python3 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org meson"'
+ else:
+ error(f"Unknown command to install meson for {os_family_tags}")
+
+
+def get_setup_hugepages_command(os_family_tags: Set[OsFamily]) -> str:
+ if OsFamily.DEBIAN in os_family_tags or OsFamily.RHEL in os_family_tags:
+ return (
+ '"sed -i -r \'s/GRUB_CMDLINE_LINUX_DEFAULT=\\"([^\\"]+)\\"/'
+ 'GRUB_CMDLINE_LINUX_DEFAULT=\\"\\1 default_hugepagesz=2M hugepagesz=2M'
+ ' hugepages=1375 hugepagesz=1G hugepages=8\\"/\' /etc/default/grub"'
+ )
+ else:
+ error(f"Unknown command to setup hugepages for {os_family_tags}")
+
+
+def get_hugepage_mount_command(os_family_tags: Set[OsFamily]) -> str:
+ if OsFamily.DEBIAN in os_family_tags or OsFamily.RHEL in os_family_tags:
+ return '"mkdir -p /dev/huge && mount nodev -t hugetlbfs -o rw,pagesize=2M /dev/huge/ && umount /dev/huge"'
+ else:
+ error(f"Unknown hugepage mount command for {os_family_tags}")
+
+
+def get_security_enforcement_disable_command(
+ os_family_tags: Set[OsFamily], output_path: str
+) -> List[str]:
+ if OsFamily.RHEL in os_family_tags:
+ return [f"--run-command 'echo \"SELINUX=disabled\" > /etc/selinux/config'"]
+ else:
+ return []
+
+
+def get_os_family_tags(distribution: OsFamily) -> Set[OsFamily]:
+ tags: Set[OsFamily] = {distribution}
+
+ # This is not an if-elif-else chain to reduce duplicate code. This way,
+ # for example, a specialized ubuntu distribution may first be tagged
+ # ubuntu, then all the ubuntu tags will be applied to it. The most
+ # specific distros should be placed first.
+
+ if OsFamily.UBUNTU in tags:
+ tags.add(OsFamily.DEBIAN)
+
+ if OsFamily.FEDORA in tags:
+ tags.add(OsFamily.CENTOS)
+
+ if OsFamily.CENTOS in tags:
+ tags.add(OsFamily.RHEL)
+
+ if OsFamily.RHEL in tags:
+ tags.add(OsFamily.RHEL)
+
+ return tags
+
+
+def check_being_run_as_root():
+ proc = subprocess.run(["whoami"], capture_output=True)
+ if "root".encode() not in proc.stdout:
+ error("This program must be run as root.")
+
+
+def get_image_info(base_image_path: str) -> (OsFamily, Arch):
+ command = [
+ "virt-inspector",
+ # Otherwise it will show everything installed via the package manager
+ "--no-applications",
+ # We don't need to icon for the distro
+ "--no-icon",
+ "-a",
+ base_image_path,
+ ]
+
+ print(" ".join(command))
+
+ proc = subprocess.run(command, capture_output=True)
+ if proc.returncode != 0:
+ print(proc.stdout)
+ print(proc.stderr)
+ error("Unable to inspect base image")
+
+ tree = ET.fromstring(proc.stdout)
+ distro = OsFamily(tree.findtext("operatingsystem/distro"))
+ arch = Arch(tree.findtext("operatingsystem/arch"))
+
+ return distro, arch
+
+
+def main():
+ args = parse_arguments()
+ check_being_run_as_root()
+ distro, arch = get_image_info(args.base_image)
+
+ if distro not in SUPPORTED_OS_FAMILIES:
+ error(f"Unsupported distro {distro}")
+
+ if arch not in SUPPORTED_ARCHITECTURES:
+ error(f"Unsupported architecture {arch}")
+
+ os_family_tags = get_os_family_tags(distro)
+ run_subprocess(
+ os_family_tags,
+ args.base_image,
+ args.output_path,
+ args.root_password,
+ args.debug,
+ arch,
+ )
+
+
+def error(message: str):
+ print(message, file=stderr)
+ exit(1)
+
+
+if __name__ == "__main__":
+ main()
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 7/8] dts: merge DTS vm_images/make_build_container.sh to DPDK
2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš
` (5 preceding siblings ...)
2022-04-06 15:31 ` [RFC PATCH v1 6/8] dts: merge DTS vm_images/create_vm_image.py " Juraj Linkeš
@ 2022-04-06 15:31 ` Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 8/8] dts: merge DTS vm_images/network-init.sh " Juraj Linkeš
7 siblings, 0 replies; 9+ messages in thread
From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw)
To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu
Cc: dev, Juraj Linkeš
---
dts/vm_images/make_build_container.sh | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
create mode 100755 dts/vm_images/make_build_container.sh
diff --git a/dts/vm_images/make_build_container.sh b/dts/vm_images/make_build_container.sh
new file mode 100755
index 0000000000..fb4472436f
--- /dev/null
+++ b/dts/vm_images/make_build_container.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# Podman is used here because Docker does very odd things when
+# building for another architecture. Docker's solution to this,
+# buildx, is still unstable.
+
+podman build --arch arm64 -t dts_vm_builder:aarch64 . &
+DTS_AARCH64_BUILD_PID=$!
+podman build --arch amd64 -t dts_vm_builder:x86_64 . &
+DTS_x86_64_BUILD_PID=$!
+podman build --arch ppc64le -t dts_vm_builder:ppc64le . &
+DTS_PPC64LE_BUILD_PID=$!
+
+wait $DTS_AARCH64_BUILD_PID
+wait $DTS_PPC64LE_BUILD_PID
+wait $DTS_x86_64_BUILD_PID
\ No newline at end of file
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 8/8] dts: merge DTS vm_images/network-init.sh to DPDK
2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš
` (6 preceding siblings ...)
2022-04-06 15:31 ` [RFC PATCH v1 7/8] dts: merge DTS vm_images/make_build_container.sh " Juraj Linkeš
@ 2022-04-06 15:31 ` Juraj Linkeš
7 siblings, 0 replies; 9+ messages in thread
From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw)
To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu
Cc: dev, Juraj Linkeš
---
dts/vm_images/network-init.sh | 6 ++++++
1 file changed, 6 insertions(+)
create mode 100755 dts/vm_images/network-init.sh
diff --git a/dts/vm_images/network-init.sh b/dts/vm_images/network-init.sh
new file mode 100755
index 0000000000..ad0190a5af
--- /dev/null
+++ b/dts/vm_images/network-init.sh
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+# Used to ensure networking is up for all images
+# This is a brute-force approach to try to ensure it always works
+
+ifconfig | grep -Po "^[^:\s]+:" | tr -d ':' | xargs -I % ip link set % up
\ No newline at end of file
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-04-06 15:32 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 1/8] dts: merge DTS format.sh " Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 2/8] dts: merge DTS pyproject.toml " Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 3/8] dts: merge DTS requirements.txt " Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 4/8] dts: merge DTS vm_images/Dockerfile " Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 5/8] dts: merge DTS vm_images/README.md " Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 6/8] dts: merge DTS vm_images/create_vm_image.py " Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 7/8] dts: merge DTS vm_images/make_build_container.sh " Juraj Linkeš
2022-04-06 15:31 ` [RFC PATCH v1 8/8] dts: merge DTS vm_images/network-init.sh " Juraj Linkeš
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).