* [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK @ 2022-04-06 15:31 Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 1/8] dts: merge DTS format.sh " Juraj Linkeš ` (7 more replies) 0 siblings, 8 replies; 9+ messages in thread From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw) To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu Cc: dev, Juraj Linkeš Various tools and repo config. Juraj Linkeš (8): dts: merge DTS format.sh to DPDK dts: merge DTS pyproject.toml to DPDK dts: merge DTS requirements.txt to DPDK dts: merge DTS vm_images/Dockerfile to DPDK dts: merge DTS vm_images/README.md to DPDK dts: merge DTS vm_images/create_vm_image.py to DPDK dts: merge DTS vm_images/make_build_container.sh to DPDK dts: merge DTS vm_images/network-init.sh to DPDK dts/format.sh | 35 ++ dts/pyproject.toml | 3 + dts/requirements.txt | 42 +++ dts/vm_images/Dockerfile | 9 + dts/vm_images/README.md | 64 ++++ dts/vm_images/create_vm_image.py | 474 ++++++++++++++++++++++++++ dts/vm_images/make_build_container.sh | 16 + dts/vm_images/network-init.sh | 6 + 8 files changed, 649 insertions(+) create mode 100755 dts/format.sh create mode 100644 dts/pyproject.toml create mode 100644 dts/requirements.txt create mode 100644 dts/vm_images/Dockerfile create mode 100644 dts/vm_images/README.md create mode 100755 dts/vm_images/create_vm_image.py create mode 100755 dts/vm_images/make_build_container.sh create mode 100755 dts/vm_images/network-init.sh -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 1/8] dts: merge DTS format.sh to DPDK 2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš @ 2022-04-06 15:31 ` Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 2/8] dts: merge DTS pyproject.toml " Juraj Linkeš ` (6 subsequent siblings) 7 siblings, 0 replies; 9+ messages in thread From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw) To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu Cc: dev, Juraj Linkeš --- dts/format.sh | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100755 dts/format.sh diff --git a/dts/format.sh b/dts/format.sh new file mode 100755 index 0000000000..d35c0bf1da --- /dev/null +++ b/dts/format.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash + +function main() { + # The directory to work on is either passed in as argument 1, + # or is the current working directory + DIRECTORY=${1:-`pwd`} + LINE_LENGTH=88 + + isort \ + --overwrite-in-place \ + --profile black \ + -j `nproc` \ + --line-length $LINE_LENGTH \ + --python-version auto \ + $DIRECTORY + + black \ + --line-length $LINE_LENGTH \ + --required-version 22.1.0 \ + --target-version py38 \ + --safe \ + $DIRECTORY +} + +function help() { + echo "usage: format.sh <directory>" +} + +if [ "$1" == "-h" ] || [ "$1" == "--help" ]; then + help + exit 0 +fi + +main "$1" + -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 2/8] dts: merge DTS pyproject.toml to DPDK 2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 1/8] dts: merge DTS format.sh " Juraj Linkeš @ 2022-04-06 15:31 ` Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 3/8] dts: merge DTS requirements.txt " Juraj Linkeš ` (5 subsequent siblings) 7 siblings, 0 replies; 9+ messages in thread From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw) To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu Cc: dev, Juraj Linkeš --- dts/pyproject.toml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 dts/pyproject.toml diff --git a/dts/pyproject.toml b/dts/pyproject.toml new file mode 100644 index 0000000000..e5befba31b --- /dev/null +++ b/dts/pyproject.toml @@ -0,0 +1,3 @@ +[tool.isort] +profile = "black" + -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 3/8] dts: merge DTS requirements.txt to DPDK 2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 1/8] dts: merge DTS format.sh " Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 2/8] dts: merge DTS pyproject.toml " Juraj Linkeš @ 2022-04-06 15:31 ` Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 4/8] dts: merge DTS vm_images/Dockerfile " Juraj Linkeš ` (4 subsequent siblings) 7 siblings, 0 replies; 9+ messages in thread From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw) To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu Cc: dev, Juraj Linkeš --- dts/requirements.txt | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 dts/requirements.txt diff --git a/dts/requirements.txt b/dts/requirements.txt new file mode 100644 index 0000000000..f58a64defa --- /dev/null +++ b/dts/requirements.txt @@ -0,0 +1,42 @@ +# BSD LICENSE +# +# Copyright(c) 2019 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +xlwt==1.3.0 +pexpect==4.7.0 +numpy==1.18.5 +docutils +pcapy +xlrd +scapy==2.4.4 +threadpool +isort +texttable \ No newline at end of file -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 4/8] dts: merge DTS vm_images/Dockerfile to DPDK 2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš ` (2 preceding siblings ...) 2022-04-06 15:31 ` [RFC PATCH v1 3/8] dts: merge DTS requirements.txt " Juraj Linkeš @ 2022-04-06 15:31 ` Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 5/8] dts: merge DTS vm_images/README.md " Juraj Linkeš ` (3 subsequent siblings) 7 siblings, 0 replies; 9+ messages in thread From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw) To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu Cc: dev, Juraj Linkeš --- dts/vm_images/Dockerfile | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 dts/vm_images/Dockerfile diff --git a/dts/vm_images/Dockerfile b/dts/vm_images/Dockerfile new file mode 100644 index 0000000000..ce4dbca44a --- /dev/null +++ b/dts/vm_images/Dockerfile @@ -0,0 +1,9 @@ +FROM ubuntu:20.04 + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && apt-get upgrade -y + +RUN apt-get install --no-install-recommends -y libguestfs-tools \ + qemu linux-image-generic qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils \ + linux-image-unsigned-5.11.0-46-generic qemu-system-x86 -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 5/8] dts: merge DTS vm_images/README.md to DPDK 2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš ` (3 preceding siblings ...) 2022-04-06 15:31 ` [RFC PATCH v1 4/8] dts: merge DTS vm_images/Dockerfile " Juraj Linkeš @ 2022-04-06 15:31 ` Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 6/8] dts: merge DTS vm_images/create_vm_image.py " Juraj Linkeš ` (2 subsequent siblings) 7 siblings, 0 replies; 9+ messages in thread From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw) To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu Cc: dev, Juraj Linkeš --- dts/vm_images/README.md | 64 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 dts/vm_images/README.md diff --git a/dts/vm_images/README.md b/dts/vm_images/README.md new file mode 100644 index 0000000000..1a59810d43 --- /dev/null +++ b/dts/vm_images/README.md @@ -0,0 +1,64 @@ +# DTS VM Images + +This folder contains utilities to create VM +images for use in virtio testing. + +## Host Requirements + +The host MUST have qemu/kvm with libvirtd installed +and set up. + +The host MUST be the same architecture as the VM +you are building. + +The host MUST have podman and either docker or have podman +aliased as docker (running "docker" calls podman). + +## Creating a VM + +Use the "create_vm_image.py" script to create the vm image. +If you do not have the required containers on your system, +it will build them. + +The root password it asks for is what to set the VM's +root password to, not the root password of the system +you run the script on. + +``` --debug ``` will enable debug output from guestfs +tools. This produces a lot of output and you shouldn't +use it unless something is going wrong. + +The base image MUST be a "cloud ready" or "prebuilt" +image, meaning you cannot use an installer ISO. It also +must be in the qcow2 format, (use qemu-img to convert it). +Most distros will have a "cloud image" which is in the +correct format. This base image will not be modified +by the build script. + +The output image is where all of the modifications go and +it is the image that you should use with DTS. + +## Supported Distros + +Currently, only RHEL 8 family distros and Ubuntu 20.04 are +supported. Debian might work, but it is untested. Most +testing has gone to Ubuntu 20.04. + +## Architectures + +Due to the way that guestfs tools work, they must run +under kvm, but the host needs to have a kernel image +that can be used to boot the VM. It may be possible +to work around this issue using containers, but +several days of experimentation kept running into +more and more complex issues with the interactions +between libguestfs and docker/podman. As such, +your best bet is to build your VMs on either a +bare-metal system of your desired architecture +or inside a VM already being emulated as your desired +architecture. This second approach may run into +issues with the hypervisor, since not all hypervisors +support nested virtualization by default. Since you need +an appropriate kernel image installed as well, it may +be easiest to build VMs using whatever distro you already +use for most of your servers. -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 6/8] dts: merge DTS vm_images/create_vm_image.py to DPDK 2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš ` (4 preceding siblings ...) 2022-04-06 15:31 ` [RFC PATCH v1 5/8] dts: merge DTS vm_images/README.md " Juraj Linkeš @ 2022-04-06 15:31 ` Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 7/8] dts: merge DTS vm_images/make_build_container.sh " Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 8/8] dts: merge DTS vm_images/network-init.sh " Juraj Linkeš 7 siblings, 0 replies; 9+ messages in thread From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw) To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu Cc: dev, Juraj Linkeš --- dts/vm_images/create_vm_image.py | 474 +++++++++++++++++++++++++++++++ 1 file changed, 474 insertions(+) create mode 100755 dts/vm_images/create_vm_image.py diff --git a/dts/vm_images/create_vm_image.py b/dts/vm_images/create_vm_image.py new file mode 100755 index 0000000000..8a010c99ac --- /dev/null +++ b/dts/vm_images/create_vm_image.py @@ -0,0 +1,474 @@ +#!/usr/bin/python3 + +import argparse +import enum +import os +import subprocess +from sys import stderr +from typing import List, Optional, Set, Tuple +import xml.etree.ElementTree as ET +import platform + +DOCKER_IMAGE_NAME = "dts_vm_builder" + + +# From https://libguestfs.org/guestfs.3.html#guestfs_inspect_get_distro +class OsFamily(enum.Enum): + ALPINE = "alpinelinux" + ALT = "altlinux" + ARCH = "archlinux" + BUILDROOT_DERIVED = "buildroot" + CENTOS = "centos" + CIRROS = "cirros" + COREOS = "coreos" + DEBIAN = "debian" + FEDORA = "fedora" + FREEBSD = "freebsd" + FREEDOS = "freedos" + FRUNGALWARE = "frugalware" + GENTOO = "gentoo" + KALI = "kalilinux" + KYLIN = "kylin" + MINT = "linuxmint" + MAGEIA = "mageia" + MANDRIVA = "mandriva" + MEEGO = "meego" + MSDOS = "msdos" + NEOKYLIN = "neokylin" + NETBSD = "netbsd" + OPENBSD = "openbsd" + OPENMANDRIVA = "openmandriva" + OPENSUSE = "opensuse" + ORACLE = "oraclelinux" + PARDUS = "pardus" + PLD = "pldlinux" + RHEL_BASED = "redhat-based" + RHEL = "rhel" + ROCKY = "rocky" + SCIENTIFIC_LINUX = "scientificlinux" + SLACKWARE = "slackware" + SLES = "sles" + SUSE_BASED = "suse-based" + TTY_LINUX = "ttylinux" + UBUNTU = "ubuntu" + VOID = "voidlinux" + WINDOWS = "windows" + + UNKNOWN = "unknown" + + def __str__(self): + return self.value + + +# The Os Families that are supported +SUPPORTED_OS_FAMILIES = { + OsFamily.CENTOS, + OsFamily.DEBIAN, + OsFamily.FEDORA, + OsFamily.RHEL_BASED, + OsFamily.RHEL, + OsFamily.UBUNTU, +} + + +# From https://libguestfs.org/guestfs.3.html#guestfs_file_architecture +class Arch(enum.Enum): + aarch64 = "aarch64" + i386 = "i386" + ia64 = "ia64" + ppc = "ppc" + ppc64 = "ppc64" + ppc64le = "ppc64le" + riscv32 = "riscv32" + riscv64 = "riscv64" + riscv128 = "riscv128" + s390 = "s390" + s390x = "s390x" + sparc = "sparc" + sparc64 = "sparc64" + x86_64 = "x86_64" + + def __str__(self): + return self.value + + +# The supported architectures +SUPPORTED_ARCHITECTURES = {Arch.x86_64, Arch.aarch64, Arch.ppc64} + + +def validate_filepath(parser: argparse.ArgumentParser, filepath: str) -> str: + if not os.path.isabs(filepath): + filepath = os.path.abspath(filepath) + + if os.path.exists(filepath): + return filepath + else: + parser.error(f"Path {filepath} not found") + + +def parse_arguments() -> argparse.Namespace: + parser = argparse.ArgumentParser() + + # Base image file + parser.add_argument("base_image", type=lambda f: validate_filepath(parser, f)) + + # Where to write the vm image to + parser.add_argument("output_path") + + # What to set the root password to + parser.add_argument( + "--root_password", help="The new root password for the vm", default="dts" + ) + + # Whether to run virt-customize in debug mode + parser.add_argument("--debug", action="store_true", default=False) + + return parser.parse_args() + + +def run_subprocess( + os_family_tags: Set[OsFamily], + base_image_path: str, + output_path: str, + root_password: str, + debug_mode: bool, + arch: Arch, +): + copy_base_image_to_output_path(base_image_path, output_path) + + print("Building under emulation") + + # Check if the docker container already exists + docker_process = subprocess.run( + f"docker image ls {DOCKER_IMAGE_NAME}", capture_output=True, shell=True + ) + + if docker_process.returncode != 0: + error("Unable to check for presence of docker image") + + if not len(docker_process.stdout.splitlines()) >= 2: # image does not exist + subprocess.run(f"./make_build_container.sh") + + docker_command = [ + "docker", + "run", + # The container needs to access QEMU/KVM + "--privileged", + "-d", + "--platform", + ] + + if arch == Arch.x86_64: + docker_command += ("linux/amd64",) + elif arch == Arch.ppc64le: + docker_command += ("linux/ppc64le",) + elif arch == Arch.aarch64: + docker_command += ("linux/arm64",) + else: + error(f"Please add {arch} to the if chain selecting the docker platform") + + docker_command += ("-v $(pwd):/vm_folder",) + + if debug_mode: + docker_command += ( + "-e", + "LIBGUESTFS_DEBUG=1", + "-e", + "LIBGUESTFS_TRACE=1", + ) + + # Run cat so it doesn't terminate until we stop it + docker_command += f"-it {DOCKER_IMAGE_NAME}:{arch}", "cat" + + # if debug_mode: + print("Running:") + print(" ".join(docker_command)) + print("\n\n") + + docker_process = subprocess.run( + " ".join(docker_command), shell=True, capture_output=True + ) + + if docker_process.returncode != 0: + print(docker_process.stderr) + print(docker_process.stdout) + error("Unable to run docker container, try --debug") + + container_id = docker_process.stdout.strip().decode() + + if debug_mode: + print(f"Docker container is {container_id}") + + virt_customize_command = get_virt_customize_command( + os_family_tags, output_path, root_password + ) + + vm_build_command = ["docker", "exec", "-w", "/vm_folder"] + + if debug_mode: + vm_build_command += ( + "-e", + "LIBGUESTFS_DEBUG=1", + "-e", + "LIBGUESTFS_TRACE=1", + ) + + vm_build_command += ( + "-it", + container_id, + ) + + vm_build_command += (virt_customize_command,) + + # if debug_mode: + print(" ".join(vm_build_command)) + + vm_build_process = subprocess.run(" ".join(vm_build_command), shell=True) + + if vm_build_process.returncode == 0: + # Shut down the build container + subprocess.run(f"docker kill {container_id}", shell=True) + + print(vm_build_process.returncode) + + +def run_command_in_docker_container( + container_id: str, command: str, debug_mode: bool, **kwargs +) -> subprocess.CompletedProcess: + docker_command = "docker exec " + + if debug_mode: + docker_command += f"-e LIBGUESTFS_DEBUG=1 -e LIBGUESTFS_TRACE=1" + + docker_command += f"-w /vm_folder -t {container_id} {command}" + return subprocess.run(docker_command, **kwargs) + + +def copy_base_image_to_output_path(base_image_path: str, output_path: str): + real_base_image_path: str = os.path.realpath(base_image_path) + real_output_path: str = os.path.realpath(output_path) + + if ( + real_base_image_path != real_output_path + ): # do not copy if they are the same path + subprocess.run( + ["cp", real_base_image_path, real_output_path], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + + +def get_virt_customize_command( + os_family_tags: Set[OsFamily], output_path: str, root_password: str +) -> str: + commands = [ + f"virt-customize -a {output_path} --root-password password:{root_password} --update" + ] + + commands = commands + get_enable_additional_repos_commands(os_family_tags) + + packages = get_packages_for_os_family(os_family_tags) + packagelist = ",".join(packages) + commands += (f"--install {packagelist}",) + commands += (f"--run-command {get_install_meson_command(os_family_tags)}",) + commands += (f"--run-command {get_setup_hugepages_command(os_family_tags)}",) + commands += (f"--run-command {get_hugepage_mount_command(os_family_tags)}",) + commands = commands + get_security_enforcement_disable_command( + os_family_tags, output_path + ) + return " ".join(commands) + + +def get_enable_additional_repos_commands(os_family_tags: Set[OsFamily]): + if OsFamily.RHEL in os_family_tags and OsFamily.FEDORA not in os_family_tags: + packages = [ + "yum-utils", + "epel-release", + ] + + packagelist = ",".join(packages) + + return [ + f"--install {packagelist}", + f"--run-command 'yum-config-manager --enable powertools'", + ] + elif OsFamily.DEBIAN in os_family_tags: + return [] + + +def get_packages_for_os_family(os_family_tags: Set[OsFamily]) -> List[str]: + if OsFamily.DEBIAN in os_family_tags: + return [ + "make", + "gcc", + "g++", + "libc-dev", + "libc6-dev", + "ninja-build", + "pkg-config", + "libnuma-dev", + "python3-pyelftools", + "abigail-tools", + "git", + "librdmacm-dev", + "librdmacm1", + "rdma-core", + "libelf-dev", + "libmnl-dev", + "libpcap-dev", + "libcrypto++-dev", + "libjansson-dev", + "libatomic1", + "python3-pip", + "python3-setuptools", + "python3-wheel", + "iperf", + "chrony", + ] + elif OsFamily.RHEL in os_family_tags: + return [ + "make", + "gcc", + "pkg-config", + "ninja-build", + "numactl-libs", + "python3-pyelftools", + "libabigail-devel", + "git", + "librdmacm", + "librdmacm-utils", + "rdma-core", + "elfutils-libelf-devel", + "libmnl-devel", + "libpcap-devel", + "cryptopp-devel", + "jansson-devel", + "libatomic", + "python3-pip", + "python3-setuptools", + "python3-wheel", + ] + else: + error(f"Unable to get packages for {os_family_tags} OS family.") + + +def get_install_meson_command(os_family_tags: Set[OsFamily]) -> str: + if OsFamily.DEBIAN in os_family_tags or OsFamily.RHEL in os_family_tags: + # the "--trusted-host" flags are included because the date on the system will be Jan 1, 1970 due to the way + # guestfs-tools starts the vm. This breaks pip's ssl, so making these hosts trusted fixes that. + return '"python3 -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org meson"' + else: + error(f"Unknown command to install meson for {os_family_tags}") + + +def get_setup_hugepages_command(os_family_tags: Set[OsFamily]) -> str: + if OsFamily.DEBIAN in os_family_tags or OsFamily.RHEL in os_family_tags: + return ( + '"sed -i -r \'s/GRUB_CMDLINE_LINUX_DEFAULT=\\"([^\\"]+)\\"/' + 'GRUB_CMDLINE_LINUX_DEFAULT=\\"\\1 default_hugepagesz=2M hugepagesz=2M' + ' hugepages=1375 hugepagesz=1G hugepages=8\\"/\' /etc/default/grub"' + ) + else: + error(f"Unknown command to setup hugepages for {os_family_tags}") + + +def get_hugepage_mount_command(os_family_tags: Set[OsFamily]) -> str: + if OsFamily.DEBIAN in os_family_tags or OsFamily.RHEL in os_family_tags: + return '"mkdir -p /dev/huge && mount nodev -t hugetlbfs -o rw,pagesize=2M /dev/huge/ && umount /dev/huge"' + else: + error(f"Unknown hugepage mount command for {os_family_tags}") + + +def get_security_enforcement_disable_command( + os_family_tags: Set[OsFamily], output_path: str +) -> List[str]: + if OsFamily.RHEL in os_family_tags: + return [f"--run-command 'echo \"SELINUX=disabled\" > /etc/selinux/config'"] + else: + return [] + + +def get_os_family_tags(distribution: OsFamily) -> Set[OsFamily]: + tags: Set[OsFamily] = {distribution} + + # This is not an if-elif-else chain to reduce duplicate code. This way, + # for example, a specialized ubuntu distribution may first be tagged + # ubuntu, then all the ubuntu tags will be applied to it. The most + # specific distros should be placed first. + + if OsFamily.UBUNTU in tags: + tags.add(OsFamily.DEBIAN) + + if OsFamily.FEDORA in tags: + tags.add(OsFamily.CENTOS) + + if OsFamily.CENTOS in tags: + tags.add(OsFamily.RHEL) + + if OsFamily.RHEL in tags: + tags.add(OsFamily.RHEL) + + return tags + + +def check_being_run_as_root(): + proc = subprocess.run(["whoami"], capture_output=True) + if "root".encode() not in proc.stdout: + error("This program must be run as root.") + + +def get_image_info(base_image_path: str) -> (OsFamily, Arch): + command = [ + "virt-inspector", + # Otherwise it will show everything installed via the package manager + "--no-applications", + # We don't need to icon for the distro + "--no-icon", + "-a", + base_image_path, + ] + + print(" ".join(command)) + + proc = subprocess.run(command, capture_output=True) + if proc.returncode != 0: + print(proc.stdout) + print(proc.stderr) + error("Unable to inspect base image") + + tree = ET.fromstring(proc.stdout) + distro = OsFamily(tree.findtext("operatingsystem/distro")) + arch = Arch(tree.findtext("operatingsystem/arch")) + + return distro, arch + + +def main(): + args = parse_arguments() + check_being_run_as_root() + distro, arch = get_image_info(args.base_image) + + if distro not in SUPPORTED_OS_FAMILIES: + error(f"Unsupported distro {distro}") + + if arch not in SUPPORTED_ARCHITECTURES: + error(f"Unsupported architecture {arch}") + + os_family_tags = get_os_family_tags(distro) + run_subprocess( + os_family_tags, + args.base_image, + args.output_path, + args.root_password, + args.debug, + arch, + ) + + +def error(message: str): + print(message, file=stderr) + exit(1) + + +if __name__ == "__main__": + main() -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 7/8] dts: merge DTS vm_images/make_build_container.sh to DPDK 2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš ` (5 preceding siblings ...) 2022-04-06 15:31 ` [RFC PATCH v1 6/8] dts: merge DTS vm_images/create_vm_image.py " Juraj Linkeš @ 2022-04-06 15:31 ` Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 8/8] dts: merge DTS vm_images/network-init.sh " Juraj Linkeš 7 siblings, 0 replies; 9+ messages in thread From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw) To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu Cc: dev, Juraj Linkeš --- dts/vm_images/make_build_container.sh | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100755 dts/vm_images/make_build_container.sh diff --git a/dts/vm_images/make_build_container.sh b/dts/vm_images/make_build_container.sh new file mode 100755 index 0000000000..fb4472436f --- /dev/null +++ b/dts/vm_images/make_build_container.sh @@ -0,0 +1,16 @@ +#!/bin/bash + +# Podman is used here because Docker does very odd things when +# building for another architecture. Docker's solution to this, +# buildx, is still unstable. + +podman build --arch arm64 -t dts_vm_builder:aarch64 . & +DTS_AARCH64_BUILD_PID=$! +podman build --arch amd64 -t dts_vm_builder:x86_64 . & +DTS_x86_64_BUILD_PID=$! +podman build --arch ppc64le -t dts_vm_builder:ppc64le . & +DTS_PPC64LE_BUILD_PID=$! + +wait $DTS_AARCH64_BUILD_PID +wait $DTS_PPC64LE_BUILD_PID +wait $DTS_x86_64_BUILD_PID \ No newline at end of file -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v1 8/8] dts: merge DTS vm_images/network-init.sh to DPDK 2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš ` (6 preceding siblings ...) 2022-04-06 15:31 ` [RFC PATCH v1 7/8] dts: merge DTS vm_images/make_build_container.sh " Juraj Linkeš @ 2022-04-06 15:31 ` Juraj Linkeš 7 siblings, 0 replies; 9+ messages in thread From: Juraj Linkeš @ 2022-04-06 15:31 UTC (permalink / raw) To: thomas, david.marchand, Honnappa.Nagarahalli, ohilyard, lijuan.tu Cc: dev, Juraj Linkeš --- dts/vm_images/network-init.sh | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 dts/vm_images/network-init.sh diff --git a/dts/vm_images/network-init.sh b/dts/vm_images/network-init.sh new file mode 100755 index 0000000000..ad0190a5af --- /dev/null +++ b/dts/vm_images/network-init.sh @@ -0,0 +1,6 @@ +#!/bin/bash + +# Used to ensure networking is up for all images +# This is a brute-force approach to try to ensure it always works + +ifconfig | grep -Po "^[^:\s]+:" | tr -d ':' | xargs -I % ip link set % up \ No newline at end of file -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-04-06 15:32 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-04-06 15:31 [RFC PATCH v1 0/8] merge DTS repo config and tools to DPDK Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 1/8] dts: merge DTS format.sh " Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 2/8] dts: merge DTS pyproject.toml " Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 3/8] dts: merge DTS requirements.txt " Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 4/8] dts: merge DTS vm_images/Dockerfile " Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 5/8] dts: merge DTS vm_images/README.md " Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 6/8] dts: merge DTS vm_images/create_vm_image.py " Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 7/8] dts: merge DTS vm_images/make_build_container.sh " Juraj Linkeš 2022-04-06 15:31 ` [RFC PATCH v1 8/8] dts: merge DTS vm_images/network-init.sh " Juraj Linkeš
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).