From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id BA671A04AB for ; Fri, 8 Nov 2019 20:42:29 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8EE521D14F; Fri, 8 Nov 2019 20:42:29 +0100 (CET) Received: from mail-pl1-f230.google.com (mail-pl1-f230.google.com [209.85.214.230]) by dpdk.org (Postfix) with ESMTP id 46A8E1D14D for ; Fri, 8 Nov 2019 20:42:28 +0100 (CET) Received: by mail-pl1-f230.google.com with SMTP id ay6so4565239plb.0 for ; Fri, 08 Nov 2019 11:42:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=c53LRbGPkEK6V0eu2FeiU/jYqGEzL0JNQInwODnwu6s=; b=bXdlBiOItfwMaauyrBtUa4/0nBVx4m/x4n44gpYQwcqY/QKtUQ6fVOJrpzzCzfwPEW x/Ah8vvLOz5ENSOmcSeGzuAxViai4J+9/5OCKo8WiC533FMiE5X7ZTT+tuq0XoLVNYgn r0QGisnR+NzTD63VB9QnEUsxu4Z+ZmH1gRag0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=c53LRbGPkEK6V0eu2FeiU/jYqGEzL0JNQInwODnwu6s=; b=UWXLQnBw4O7Joo6KuZJdxVVQwF2fCO/xc8NOSCvnVMVlS1uf4iKpvcSbuKSlXUvigX dWST9mnC63lCSk7a4HcJSTUnh73GHkJIxuP/k/JMOSgqV/J26eMmMoHrUJWhMGUL+Dfv G9D9+Y4L0gllhhUlFsf5sjQqzPLjnc7mDVKehxe9/p1xVt/uUh4Otv7VsSoaIahXiavK Xh3XaEu7rgbHFGMX6m8998L63k8W8C/3/MAnaVElRZeKetf2/JEafmeKzl4RoWVl1lhs xPbWfZ0CvDAl4GJR4ENttbDbRBN9DGmM3rngygu/YShxTDJOGloH+ssm6RFsF9SaYZhn 49Og== X-Gm-Message-State: APjAAAV/LuLe0CjRpj6m2lOKkv5oQ9waVAgsWYrjwm+vPeYa9FfjK5c8 6GAEs4fNm4sTweKu7XVbT/HjCoOcaFDcRYr1Tu5Mt7KFlp3svqlHBHxDf4eWC2O/2EqFAnviVQs 4hO8lTaY6KfbLM2kFxMXxyWHtddHF+ZUPCZXXsYzhuBwIYqHrSYv9pCzKzWydtN755kO9YA== X-Google-Smtp-Source: APXvYqza5y5vAEzKYdsps2X2aBh+xDAUSp+dmfOeMQR8ZPmRjviu5sU93htxPHx1PXOndLI9DklsbtbRsu5W X-Received: by 2002:a17:902:a58c:: with SMTP id az12mr12166868plb.140.1573242147203; Fri, 08 Nov 2019 11:42:27 -0800 (PST) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [132.177.123.84]) by smtp-relay.gmail.com with ESMTPS id v4sm571172pjh.2.2019.11.08.11.42.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 08 Nov 2019 11:42:27 -0800 (PST) X-Relaying-Domain: iol.unh.edu Received: from JP-TPX1C6.iol.unh.edu (unknown [IPv6:2606:4100:3880:1270:e4c1:ded3:b8dd:eb4a]) by postal.iol.unh.edu (Postfix) with ESMTP id D4C3F605C204; Fri, 8 Nov 2019 14:42:25 -0500 (EST) From: Jeremy Plsek To: ci@dpdk.org Cc: Jeremy Plsek Date: Fri, 8 Nov 2019 14:40:53 -0500 Message-Id: <20191108194053.22151-1-jplsek@iol.unh.edu> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-ci] [PATCH] Add smoke testing to dpdk-ci X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" This add the following testing to dpdk-ci: - DPDK compiling - DPDK unit testing - OVS with DPDK compiling - SPDK with DPDK compiling Compiling with downstream projects is meant to quickly check for API/ABI breakage. There are two methods to run this testing: 1. On their own (compile.py, compile_ovs.py, compile_spdk.py) 2. Within a container (*_with_podman.py, *_with_docker.py) The preference is to run testing with a container using podman. Since running rootless podman can still have some hiccups, docker is used in place of it. At the moment, that is just for running unit tests. Once unit tests can run with podman, the "*_with_docker.py" can be removed. Separate images are used to compile different projects. This may one day change to just have one image to compile all projects. There are scripts to compile either with the "make" system or the "meson" system. In the future, once the downstream projects can compile with the "meson" system, the "make" scripts may be removed. --- smoke-testing/README.md | 70 +++++ smoke-testing/ci.py | 257 ++++++++++++++++++ smoke-testing/ci_logging.py | 47 ++++ smoke-testing/compile.py | 119 ++++++++ smoke-testing/compile_meson.py | 53 ++++ smoke-testing/compile_meson_with_docker.py | 32 +++ smoke-testing/compile_meson_with_podman.py | 42 +++ smoke-testing/compile_ovs.py | 89 ++++++ smoke-testing/compile_ovs_with_podman.py | 34 +++ smoke-testing/compile_spdk.py | 32 +++ smoke-testing/compile_spdk_with_podman.py | 35 +++ smoke-testing/compile_with_podman.py | 44 +++ smoke-testing/containers/README.md | 17 ++ .../centos8/dpdk_compile.Dockerfile | 13 + .../centos8/dpdk_compile_ovs.Dockerfile | 17 ++ smoke-testing/containers/compile.sh | 7 + smoke-testing/containers/compile_meson.sh | 8 + smoke-testing/containers/compile_ovs.sh | 10 + smoke-testing/containers/compile_spdk.sh | 7 + .../ubuntu18.04/dpdk_compile.Dockerfile | 15 + .../ubuntu18.04/dpdk_compile_meson.Dockerfile | 19 ++ .../ubuntu18.04/dpdk_compile_ovs.Dockerfile | 15 + .../ubuntu18.04/dpdk_compile_spdk.Dockerfile | 15 + .../ubuntu18.04/dpdk_unit_test.Dockerfile | 19 ++ smoke-testing/containers/unit_test.sh | 6 + smoke-testing/dry_run.sh | 20 ++ smoke-testing/unit_test.py | 47 ++++ smoke-testing/unit_test_with_docker.py | 60 ++++ 28 files changed, 1149 insertions(+) create mode 100644 smoke-testing/README.md create mode 100644 smoke-testing/ci.py create mode 100644 smoke-testing/ci_logging.py create mode 100755 smoke-testing/compile.py create mode 100755 smoke-testing/compile_meson.py create mode 100755 smoke-testing/compile_meson_with_docker.py create mode 100755 smoke-testing/compile_meson_with_podman.py create mode 100755 smoke-testing/compile_ovs.py create mode 100755 smoke-testing/compile_ovs_with_podman.py create mode 100755 smoke-testing/compile_spdk.py create mode 100755 smoke-testing/compile_spdk_with_podman.py create mode 100755 smoke-testing/compile_with_podman.py create mode 100644 smoke-testing/containers/README.md create mode 100644 smoke-testing/containers/centos8/dpdk_compile.Dockerfile create mode 100644 smoke-testing/containers/centos8/dpdk_compile_ovs.Dockerfile create mode 100755 smoke-testing/containers/compile.sh create mode 100755 smoke-testing/containers/compile_meson.sh create mode 100755 smoke-testing/containers/compile_ovs.sh create mode 100755 smoke-testing/containers/compile_spdk.sh create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_compile.Dockerfile create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_compile_meson.Dockerfile create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_compile_ovs.Dockerfile create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_compile_spdk.Dockerfile create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_unit_test.Dockerfile create mode 100755 smoke-testing/containers/unit_test.sh create mode 100755 smoke-testing/dry_run.sh create mode 100755 smoke-testing/unit_test.py create mode 100755 smoke-testing/unit_test_with_docker.py diff --git a/smoke-testing/README.md b/smoke-testing/README.md new file mode 100644 index 0000000..9fd77d1 --- /dev/null +++ b/smoke-testing/README.md @@ -0,0 +1,70 @@ +# Smoke Testing + +This assumes Python3.6 as a minimum. +This is because we currently test with FreeBSD 11.2, CentOS 7, +and Ubuntu 18.04, which now all use Python3.6+. + +Since this is meant to run in a CI environment, logs are are removed on start up. +They are expected to be saved separately by the CI server when the scripts exit. + +## Dependencies + +``` +python3 python3-requests python3-paramiko +``` + +See the container images for build dependencies if you want to compile without utilizing the containers. + +## Running + +An example layout of running this: + +``` +$HOME +├─ +└─workspace + ├─dpdk.tar.gz + │ # Below will be generated by the scripts + ├─dpdk # Extracted tarball to build + ├─out # The output directory, containing the logs + └─out.zip # The output directory, but zipped when the scripts exit +``` + +```bash +cd ~/workspace +..//compile.py dpdk.tar.gz +``` + +The context is the current directory, +so the tarball will be extracted in the current directory, +and logs will be generated in the `out` directory. + +## DPDK unit testing + +Testing is attempted to run as a non-root user. In order to do this, there is some setup involved. + +```bash +sudo groupadd hugetlbfs +sudo usermod -a -G hugetlbfs $USER # $USER is the user running the unit tests +sudo chown -R :hugetlbfs /dev/hugepages +sudo chmod g+w /dev/hugepages -R +sudo sh -c "echo 1024 > /proc/sys/vm/nr_hugepages" +``` + +## Running OVS Performance testing + +This assumes the environment has already been set up. See: +https://github.com/chaudron/ovs_perf + +## Some development thoughts/goals + +Currently, the zip is the only artifact generated that is meant for public use. +Terminal output is mostly meant for debugging. +You must use the logger for the output to be saved to the artifact log. + +Use `ci.run` to log commands and output. Otherwise, `subprocess` can be used. +If you want to see the command generated, but not log them, +use the `ci` variant of `subprocess` if it exists, like `ci.check_call`. +It should be used over `subprocess` anyways, since it allows options like `--print-shell` +to avoid actually running the commands. +That way, setup processes do not get logged, but it still gets printed to the terminal for debugging. diff --git a/smoke-testing/ci.py b/smoke-testing/ci.py new file mode 100644 index 0000000..1afa0f4 --- /dev/null +++ b/smoke-testing/ci.py @@ -0,0 +1,257 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import argparse +import atexit +import os +import shutil +import subprocess +import sys +from os import path +from zipfile import ZIP_DEFLATED, ZipFile + +import paramiko + +from ci_logging import logging, OUTPUT_FOLDER, OUTPUT_ZIP + + +class Password: + """Mask the variable""" + def __init__(self, var): + self.var = var + + def get(self, mask=True): + if mask: + return '***' + else: + return self.var + + def __str__(self): + return '***' + + +class CIArgParse: + + def __init__(self, description): + self.parser = argparse.ArgumentParser(description=description) + self.parser.add_argument( + '--print-shell', action='store_true', + help='Print what would be done in a shell. ' + 'This *may* not be a dry run. ' + 'Useful for sending emails.') + + def parse_args(self): + return self.parser.parse_args() + + +class OS: + CHOICES = ['ubuntu18.04', 'centos8'] + + +class CI: + + ssh_cache = {} + + def __init__(self, print_shell): + self.print_shell = print_shell + atexit.register(self.at_exit) + + def at_exit(self): + """ + When overriding, call super() at the end, to properly save output. + """ + self._close_ssh_cache() + self._save_out() + + def _save_out(self): + # zip output folder + file_paths = self._get_all_file_paths(OUTPUT_FOLDER) + with ZipFile(OUTPUT_ZIP, 'w', ZIP_DEFLATED) as zip_file: + for file_path in file_paths: + zip_file.write(file_path) + + def _get_all_file_paths(self, directory): + file_paths = [] + # crawling through directory and subdirectories + for root, directories, files in os.walk(directory): + for filename in files: + filepath = os.path.join(root, filename) + file_paths.append(filepath) + return file_paths + + def parse_command(self, command, mask=True): + new_command = [] + for word in command: + # not sure of a cleaner way + if isinstance(word, Password): + new_command.append(word.get(mask)) + else: + new_command.append(word) + return new_command + + def _print_cd(self, print_cmd, **kwargs): + if 'cwd' in kwargs: + logging.info('pushd ' + kwargs['cwd']) + + logging.info(print_cmd) + + if 'cwd' in kwargs: + logging.info('popd') + + def run(self, command, exit_on_error=True, on_error=None, **kwargs): + """Prints to output, and log to file, exit on fail.""" + print_cmd = ' '.join(self.parse_command(command)) + + if self.print_shell: + self._print_cd(print_cmd, **kwargs) + return + + logging.info('+ ' + print_cmd) + + try: + proc = subprocess.Popen( + self.parse_command(command, False), + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + **kwargs) + + # live output + for line in iter(proc.stdout.readline, b''): + logging.info(line.decode('utf-8').strip('\n')) + + proc.communicate() + + if proc.returncode: + logging.info(f'[{proc.returncode}] Error running command.') + if on_error: + on_error() + if exit_on_error: + sys.exit(proc.returncode) + return proc.returncode + + except FileNotFoundError: + + logging.info(f'Command not found: {command[0]}') + if on_error: + on_error() + if exit_on_error: + sys.exit(1) + + except OSError as e: + + logging.info('Running command failed.') + logging.info(str(e)) + if on_error: + on_error() + if exit_on_error: + sys.exit(1) + + def ssh(self, host, user, password, command, + exit_on_error=True, on_error=None): + if self.print_shell: + logging.info(command) + return + + key = host + ':' + user + + if key in self.ssh_cache: + client = self.ssh_cache[key] + else: + client = paramiko.SSHClient() + client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + client.connect(host, username=user, password=password) + self.ssh_cache[key] = client + + logging.info('+ ' + command) + + stdin, stdout, stderr = client.exec_command(command) + + for line in stdout: + logging.info(line.strip('\n')) + for line in stderr: + logging.info(line.strip('\n')) + + return_code = stdout.channel.recv_exit_status() + if return_code: + logging.info(f'[{return_code}] Error running command.') + if on_error: + on_error() + if exit_on_error: + sys.exit(return_code) + + def check_call(self, command, **kwargs): + """Prints the command to stdout and log command on print_shell""" + print_cmd = ' '.join(command) + + if self.print_shell: + self._print_cd(print_cmd, **kwargs) + return + + print('+ ' + print_cmd) + + subprocess.check_call(command, **kwargs) + + def _close_ssh_cache(self): + for client in self.ssh_cache: + self.ssh_cache[client].close() + + def delete(self, tree): + if path.exists(tree): + shutil.rmtree(tree) + + +class ContainerMixin: + + def image_build(self, image, args=(), manager='podman'): + script_dir = os.path.dirname(os.path.realpath(__file__)) + + image = self.os + '/' + image + + self.check_call((manager, 'build', '-t', image, + '-f', image + '.Dockerfile', '.') + args, + cwd=script_dir + '/containers') + + def container_run(self, image, args, on_error=None, manager='podman'): + cmd = [manager, 'run', '--rm'] + + if sys.stdout.isatty(): + cmd += ['-it'] + + image = self.os + '/' + image + + self.run(cmd + args + [image], on_error=on_error) + + def kernel(self): + version = (subprocess.check_output(['uname', '-r']) + .decode('utf-8') + .strip('\n')) + + if 'debian' in self.id_like: + return 'linux-headers-' + version + + return version + + def uid(self): + return (subprocess.check_output(['id', '-u']) + .decode('utf-8') + .strip('\n')) + + def gid(self): + return (subprocess.check_output(['id', '-g']) + .decode('utf-8') + .strip('\n')) + + +class Utils: + @staticmethod + def parse_properties(f): + # return a dictionary from a property-like file + # (key values separated by equals) + d = {} + for line in f: + line = line.rstrip() + if not line: + continue + k, v = line.split('=') + d[k] = v + return d diff --git a/smoke-testing/ci_logging.py b/smoke-testing/ci_logging.py new file mode 100644 index 0000000..0c61efb --- /dev/null +++ b/smoke-testing/ci_logging.py @@ -0,0 +1,47 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import logging.config +import os +import shutil + +OUTPUT_FOLDER = 'out' +OUTPUT_ZIP = 'out.zip' + +# make output folder +if os.path.exists(OUTPUT_FOLDER): + shutil.rmtree(OUTPUT_FOLDER) +os.mkdir(OUTPUT_FOLDER) + +# and remove old zip +if os.path.exists(OUTPUT_ZIP): + os.remove(OUTPUT_ZIP) + +logging.config.dictConfig({ + 'version': 1, + 'disable_existing_loggers': False, + + 'formatters': { + 'basic': { + 'format': '%(message)s', + }, + }, + 'handlers': { + 'console': { + 'class': 'logging.StreamHandler', + 'formatter': 'basic', + 'level': 'INFO', + }, + 'file': { + 'class': 'logging.FileHandler', + 'filename': 'out/log.txt', + 'mode': 'w', + 'formatter': 'basic', + 'level': 'INFO', + }, + }, + 'root': { + 'handlers': ['console', 'file'], + 'level': 'NOTSET', + }, +}) diff --git a/smoke-testing/compile.py b/smoke-testing/compile.py new file mode 100755 index 0000000..b27990e --- /dev/null +++ b/smoke-testing/compile.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import multiprocessing +import re +import subprocess + +from ci import CI, CIArgParse, Utils +from ci_logging import logging + + +class CompileArgParse(CIArgParse): + def __init__(self, description='DPDK compile testing'): + super().__init__(description) + self.parser.add_argument('tarball', help='DPDK tarball') + + +class CompileMixin: + + def init_dpdk(self): + # delete previously extracted tarball + self.delete('dpdk') + + # untar tarball (creates dpdk folder) + self.check_call(['tar', 'xzf', self.tarball]) + + +class Compile(CompileMixin, CI): + + def __init__(self, print_shell, tarball): + super().__init__(print_shell) + + self.tarball = tarball + + out = (subprocess.check_output(['uname', '-s']) + .decode('utf-8') + .strip('\n')) + + if out == 'FreeBSD': + self.make = 'gmake' + self.execenv = 'freebsd' + else: + self.make = 'make' + self.execenv = 'linux' + self.kernels = '/usr/src' + + with open('/etc/os-release') as f: + properties = Utils.parse_properties(f) + + self.id_like = properties['ID_LIKE'] + + # use 'in' instead of equality because there can be quotes and + # multiple values in ID_LIKE + if 'rhel' in self.id_like: + self.kernels += '/kernels' + + # meant to be used in string concatenation + self.cpu_count = str(multiprocessing.cpu_count()) + + def get_build_options(self): + options = [ + 'CONFIG_RTE_ENABLE_ASSERT', + 'CONFIG_RTE_EAL_ALLOW_INV_SOCKET_ID', + 'CONFIG_RTE_EAL_ALWAYS_PANIC_ON_ERROR', + 'CONFIG_RTE_ENABLE_AVX512', + # 'CONFIG_RTE_ETHDEV_PROFILE_WITH_VTUNE', # breaks + 'CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP', + ] + + if self.execenv == 'freebsd': + options += [ + 'CONFIG_RTE_USE_LIBBSD', + ] + else: + # These do not work in freebsd + options += [ + 'CONFIG_RTE_LIBRTE_MLX4_PMD', + 'CONFIG_RTE_LIBRTE_MLX5_PMD', + 'CONFIG_RTE_LIBEAL_USE_HPET', + ] + + return options + + def apply_build_options(self): + options = self.get_build_options() + + logging.info('Enabling compile options:\n' + '\n'.join(options)) + + if self.print_shell: + return + + with open('dpdk/config/common_base') as f: + content = f.read() + + for option in options: + content = re.sub(r'(' + option + r'=)n', r'\1y', content) + + with open('dpdk/config/common_base', 'w') as f: + f.write(content) + + def build(self): + self.init_dpdk() + + self.apply_build_options() + + # configure dpdk + self.run([self.make, 'config', f'T=x86_64-native-{self.execenv}-gcc'], + cwd='dpdk') + + # compile dpdk + self.run([self.make, '-j' + self.cpu_count], cwd='dpdk') + + +if __name__ == '__main__': + args = CompileArgParse().parse_args() + + compile_dpdk = Compile(args.print_shell, args.tarball) + compile_dpdk.build() diff --git a/smoke-testing/compile_meson.py b/smoke-testing/compile_meson.py new file mode 100755 index 0000000..e685e5a --- /dev/null +++ b/smoke-testing/compile_meson.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import os +import shutil + +from ci import CI +from ci_logging import OUTPUT_FOLDER +from compile import CompileArgParse, CompileMixin + + +class CompileMesonArgParse(CompileArgParse): + def __init__(self, description='Compile DPDK with meson'): + super().__init__(description) + + +class CompileMeson(CompileMixin, CI): + + def __init__(self, print_shell, tarball): + super().__init__(print_shell) + self.tarball = tarball + + def save_logs(self): + # save build logs (only needed on error) + location = 'dpdk/build/' + for name in ['meson-logs/testlog.txt', '.ninja_log', + 'meson-logs/meson-log.txt']: + full_name = location + name + if os.path.exists(full_name): + shutil.copy(full_name, OUTPUT_FOLDER) + + def build(self): + self.init_dpdk() + + # build dpdk + self.run(['meson', 'build', '--werror', '-Dexamples=all', + '--default-library', 'shared'], + on_error=self.save_logs, cwd='dpdk') + + self.run(['ninja', '-C', 'build'], + on_error=self.save_logs, cwd='dpdk') + + # build docs +# self.run(['ninja', '-C', 'build', 'doc'], +# on_error=self.save_logs, cwd='dpdk') + + +if __name__ == '__main__': + args = CompileMesonArgParse().parse_args() + + compile_dpdk = CompileMeson(args.print_shell, args.tarball) + compile_dpdk.build() diff --git a/smoke-testing/compile_meson_with_docker.py b/smoke-testing/compile_meson_with_docker.py new file mode 100755 index 0000000..1b6f8f6 --- /dev/null +++ b/smoke-testing/compile_meson_with_docker.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import os + +from compile_meson_with_podman import CompileMesonWithPodman, \ + CompileWithPodmanArgParse + + +class CompileMesonWithDocker(CompileMesonWithPodman): + + def build(self): + self.init_dpdk() + + image = 'dpdk_compile_meson' + + self.image_build(image, manager='docker') + + self.container_run(image, [ + '--user', f'{self.uid()}:{self.gid()}', + '-v', os.getcwd() + '/dpdk:/dpdk'], + on_error=self.save_logs, + manager='docker') + + +if __name__ == '__main__': + args = CompileWithPodmanArgParse().parse_args() + + compile_dpdk = CompileMesonWithDocker( + args.print_shell, args.tarball, args.os) + compile_dpdk.build() diff --git a/smoke-testing/compile_meson_with_podman.py b/smoke-testing/compile_meson_with_podman.py new file mode 100755 index 0000000..ef5fb9c --- /dev/null +++ b/smoke-testing/compile_meson_with_podman.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import os + +from ci import ContainerMixin, OS +from compile_meson import CompileMeson, CompileMesonArgParse + + +class CompileWithPodmanArgParse(CompileMesonArgParse): + def __init__(self, description='Compile DPDK with meson in a container'): + super().__init__(description) + self.parser.add_argument( + '--os', choices=OS.CHOICES, default=OS.CHOICES[0], + help='OS container to use') + + +class CompileMesonWithPodman(ContainerMixin, CompileMeson): + + def __init__(self, print_shell, tarball, os): + super().__init__(print_shell, tarball) + self.os = os + + def build(self): + self.init_dpdk() + + image = 'dpdk_compile_meson' + + self.image_build(image) + + self.container_run(image, [ + '-v', os.getcwd() + '/dpdk:/dpdk:Z'], + on_error=self.save_logs) + + +if __name__ == '__main__': + args = CompileWithPodmanArgParse().parse_args() + + compile_dpdk = CompileMesonWithPodman( + args.print_shell, args.tarball, args.os) + compile_dpdk.build() diff --git a/smoke-testing/compile_ovs.py b/smoke-testing/compile_ovs.py new file mode 100755 index 0000000..aa360c2 --- /dev/null +++ b/smoke-testing/compile_ovs.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import os +from subprocess import check_call + +from compile import Compile, CompileArgParse + + +class OVSCompileArgParse(CompileArgParse): + def __init__(self): + super().__init__('OVS Compile testing. This assumes an up-to-date ovs ' + 'git folder exists in the current directory.') + self.parser.add_argument('--install', action='store_true', + help='Whether to run "make install"') + + +class CompileOVS(Compile): + def __init__(self, print_shell, tarball, install): + super().__init__(print_shell, tarball) + + self.install = install + + def build(self): + super().build() + + # create build directory + build = os.path.join('ovs', 'build') + + if self.install: + # running "sudo make install" changes permissions to root + # so use root to remove + check_call(['sudo', 'rm', '-rf', build]) + else: + self.delete(build) + + os.mkdir(build) + + # run boot script + self.run(['./boot.sh'], cwd='ovs') + + # run configure script + cmd = [ + '../configure', + '--enable-static', + '--disable-shared', + '--enable-ssl', + '--with-dpdk=../../dpdk/build', + ] + + if self.execenv != 'freebsd': + cmd += [ + '--enable-libcapng', + ] + + if self.install: + cmd += [ + '--program-prefix=', + '--disable-dependency-tracking', + '--prefix=/usr', + '--exec-prefix=/usr', + '--bindir=/usr/bin', + '--sbindir=/usr/sbin', + '--sysconfdir=/etc', + '--datadir=/usr/share', + '--includedir=/usr/include', + '--libdir=/usr/lib64', + '--libexecdir=/usr/libexec', + '--localstatedir=/var', + '--sharedstatedir=/var/lib', + '--mandir=/usr/share/man', + '--infodir=/usr/share/info' + ] + + self.run(cmd, cwd=build) + + # compile ovs + self.run([self.make, '-j' + self.cpu_count], cwd=build) + + if self.install: + self.run(['sudo', self.make, 'install'], cwd=build) + + +if __name__ == '__main__': + args = OVSCompileArgParse().parse_args() + + compile_ovs = CompileOVS(args.print_shell, args.tarball, args.install) + compile_ovs.build() diff --git a/smoke-testing/compile_ovs_with_podman.py b/smoke-testing/compile_ovs_with_podman.py new file mode 100755 index 0000000..2d633d5 --- /dev/null +++ b/smoke-testing/compile_ovs_with_podman.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import os + +from compile_with_podman import CompileWithPodman, CompileWithPodmanArgParse + + +class OVSCompileArgParse(CompileWithPodmanArgParse): + def __init__(self): + super().__init__('OVS Compile testing. This assumes an up-to-date ovs ' + 'git folder exists in the current directory.') + + +class CompileOVSWithPodman(CompileWithPodman): + + def build(self): + super().build() + + image = 'dpdk_compile_ovs' + + self.image_build(image) + + self.container_run(image, [ + '-v', os.getcwd() + '/dpdk:/dpdk:Z', + '-v', os.getcwd() + '/ovs:/ovs:Z']) + + +if __name__ == '__main__': + args = OVSCompileArgParse().parse_args() + + compile_ovs = CompileOVSWithPodman(args.print_shell, args.tarball, args.os) + compile_ovs.build() diff --git a/smoke-testing/compile_spdk.py b/smoke-testing/compile_spdk.py new file mode 100755 index 0000000..9600fc1 --- /dev/null +++ b/smoke-testing/compile_spdk.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +from compile import Compile, CompileArgParse + + +class SPDKCompileArgParse(CompileArgParse): + def __init__(self): + super().__init__('SPDK Compile testing. This assumes an up-to-date ' + 'spdk git folder exists in the current directory.') + + +class CompileSPDK(Compile): + def build(self): + super().build() + + # run configure script + self.run([ + './configure', + '--with-dpdk=../dpdk/build', + ], cwd='spdk') + + # compile spdk + self.run([self.make, '-j' + self.cpu_count], cwd='spdk') + + +if __name__ == '__main__': + args = SPDKCompileArgParse().parse_args() + + compile_spdk = CompileSPDK(args.print_shell, args.tarball) + compile_spdk.build() diff --git a/smoke-testing/compile_spdk_with_podman.py b/smoke-testing/compile_spdk_with_podman.py new file mode 100755 index 0000000..9adad7f --- /dev/null +++ b/smoke-testing/compile_spdk_with_podman.py @@ -0,0 +1,35 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import os + +from compile_with_podman import CompileWithPodman, CompileWithPodmanArgParse + + +class SPDKCompileArgParse(CompileWithPodmanArgParse): + def __init__(self): + super().__init__('SPDK Compile testing. This assumes an up-to-date ' + 'spdk git folder exists in the current directory.') + + +class CompileSPDKWithPodman(CompileWithPodman): + + def build(self): + super().build() + + image = 'dpdk_compile_spdk' + + self.image_build(image) + + self.container_run(image, [ + '-v', os.getcwd() + '/dpdk:/dpdk:Z', + '-v', os.getcwd() + '/spdk:/spdk:Z']) + + +if __name__ == '__main__': + args = SPDKCompileArgParse().parse_args() + + compile_spdk = CompileSPDKWithPodman( + args.print_shell, args.tarball, args.os) + compile_spdk.build() diff --git a/smoke-testing/compile_with_podman.py b/smoke-testing/compile_with_podman.py new file mode 100755 index 0000000..1d68de9 --- /dev/null +++ b/smoke-testing/compile_with_podman.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import os + +from ci import ContainerMixin, OS +from compile import Compile, CompileArgParse + + +class CompileWithPodmanArgParse(CompileArgParse): + def __init__(self, description='Compile dpdk in a container'): + super().__init__(description) + self.parser.add_argument( + '--os', choices=OS.CHOICES, default=OS.CHOICES[0], + help='OS container to use') + + +class CompileWithPodman(ContainerMixin, Compile): + + def __init__(self, print_shell, tarball, os): + super().__init__(print_shell, tarball) + self.os = os + + def build(self): + self.init_dpdk() + + self.apply_build_options() + + image = 'dpdk_compile' + + self.image_build(image) + + self.container_run(image, [ + '-e', f'RTE_KERNELDIR={self.kernels}/{self.kernel()}', + '-v', os.getcwd() + '/dpdk:/dpdk:Z', + '-v', f'{self.kernels}:{self.kernels}:ro']) + + +if __name__ == '__main__': + args = CompileWithPodmanArgParse().parse_args() + + compile_dpdk = CompileWithPodman(args.print_shell, args.tarball, args.os) + compile_dpdk.build() diff --git a/smoke-testing/containers/README.md b/smoke-testing/containers/README.md new file mode 100644 index 0000000..a4a260c --- /dev/null +++ b/smoke-testing/containers/README.md @@ -0,0 +1,17 @@ +# Containers + +This is used with the `*_with_podman.py` scripts, +but they can be ran on their own too for testing. + +Run as: + +```bash +export IMAGE= # image to use (like "ubuntu18.04/dpdk_compile") +export DPDK= # dpdk directory (like "$(pwd)/dpdk") +podman build -t ${IMAGE} -f ${IMAGE}.Dockerfile . +podman run --rm -it -e "RTE_KERNELDIR=/usr/src/linux-headers-$(uname -r)" -v ${DPDK}:/dpdk -v /usr/src:/usr/src:ro ${IMAGE} +``` + +Since symlinks may be used, the whole /usr/src folder is mounted. + +Docker can also be used. diff --git a/smoke-testing/containers/centos8/dpdk_compile.Dockerfile b/smoke-testing/containers/centos8/dpdk_compile.Dockerfile new file mode 100644 index 0000000..74d3c5d --- /dev/null +++ b/smoke-testing/containers/centos8/dpdk_compile.Dockerfile @@ -0,0 +1,13 @@ +FROM centos:8 + +RUN dnf install -y \ + diffutils \ + gcc make pkg-config numactl-devel librdmacm rdma-core-devel elfutils-libelf-devel + +WORKDIR /dpdk + +COPY compile.sh /compile.sh + +ENV RTE_KERNELDIR CHANGE_ME + +ENTRYPOINT /compile.sh diff --git a/smoke-testing/containers/centos8/dpdk_compile_ovs.Dockerfile b/smoke-testing/containers/centos8/dpdk_compile_ovs.Dockerfile new file mode 100644 index 0000000..017b2ea --- /dev/null +++ b/smoke-testing/containers/centos8/dpdk_compile_ovs.Dockerfile @@ -0,0 +1,17 @@ +FROM centos:8 + +# For libmnl-devel +RUN dnf install -y dnf-command\(config-manager\) +RUN dnf config-manager --set-enabled PowerTools + +RUN dnf install -y \ + diffutils \ + gcc make numactl-devel librdmacm rdma-core-devel \ + openssl-devel libcap-ng-devel autoconf automake libtool libmnl-devel \ + python3 python3-six + +WORKDIR /ovs + +COPY compile_ovs.sh /compile.sh + +ENTRYPOINT /compile.sh diff --git a/smoke-testing/containers/compile.sh b/smoke-testing/containers/compile.sh new file mode 100755 index 0000000..f10f45e --- /dev/null +++ b/smoke-testing/containers/compile.sh @@ -0,0 +1,7 @@ +#!/bin/bash -ex +# SPDX-License-Identifier: BSD-3-Clause + +# This script is only used to run inside a container as the ENTRYPOINT. + +make config T=x86_64-native-linux-gcc +make -j$(nproc) diff --git a/smoke-testing/containers/compile_meson.sh b/smoke-testing/containers/compile_meson.sh new file mode 100755 index 0000000..bb56083 --- /dev/null +++ b/smoke-testing/containers/compile_meson.sh @@ -0,0 +1,8 @@ +#!/bin/bash -ex +# SPDX-License-Identifier: BSD-3-Clause + +# This script is only used to run inside a container as the ENTRYPOINT. + +# build dpdk +meson build --werror -Dexamples=all --default-library shared +ninja -C build diff --git a/smoke-testing/containers/compile_ovs.sh b/smoke-testing/containers/compile_ovs.sh new file mode 100755 index 0000000..6859c24 --- /dev/null +++ b/smoke-testing/containers/compile_ovs.sh @@ -0,0 +1,10 @@ +#!/bin/bash -ex +# SPDX-License-Identifier: BSD-3-Clause + +# This script is only used to run inside a container as the ENTRYPOINT. + +./boot.sh +mkdir -p build +cd build +../configure --enable-static --disable-shared --enable-ssl --enable-libcapng --with-dpdk=/dpdk/build +make -j$(nproc) diff --git a/smoke-testing/containers/compile_spdk.sh b/smoke-testing/containers/compile_spdk.sh new file mode 100755 index 0000000..46af67c --- /dev/null +++ b/smoke-testing/containers/compile_spdk.sh @@ -0,0 +1,7 @@ +#!/bin/bash -ex +# SPDX-License-Identifier: BSD-3-Clause + +# This script is only used to run inside a container as the ENTRYPOINT. + +./configure --with-dpdk=/dpdk/build +make -j$(nproc) diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_compile.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_compile.Dockerfile new file mode 100644 index 0000000..a63b59b --- /dev/null +++ b/smoke-testing/containers/ubuntu18.04/dpdk_compile.Dockerfile @@ -0,0 +1,15 @@ +FROM ubuntu:18.04 + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && \ + apt-get install --no-install-recommends -y \ + gcc pkg-config libnuma-dev librdmacm-dev librdmacm1 rdma-core libelf-dev + +WORKDIR /dpdk + +COPY compile.sh /compile.sh + +ENV RTE_KERNELDIR CHANGE_ME + +ENTRYPOINT /compile.sh diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_compile_meson.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_compile_meson.Dockerfile new file mode 100644 index 0000000..244e085 --- /dev/null +++ b/smoke-testing/containers/ubuntu18.04/dpdk_compile_meson.Dockerfile @@ -0,0 +1,19 @@ +FROM ubuntu:18.04 + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && \ + apt-get install --no-install-recommends -y \ + gcc ninja-build pkg-config libnuma-dev librdmacm-dev librdmacm1 rdma-core libelf-dev \ + python3-pip python3-setuptools python3-wheel + +# Need the latest version of meson, which current repos do not have. +RUN python3 -m pip install meson + +RUN mkdir /build + +WORKDIR /dpdk + +COPY compile_meson.sh /compile.sh + +ENTRYPOINT /compile.sh diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_compile_ovs.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_compile_ovs.Dockerfile new file mode 100644 index 0000000..8cf9e8e --- /dev/null +++ b/smoke-testing/containers/ubuntu18.04/dpdk_compile_ovs.Dockerfile @@ -0,0 +1,15 @@ +FROM ubuntu:18.04 + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && \ + apt-get install --no-install-recommends -y \ + gcc libnuma-dev librdmacm-dev librdmacm1 rdma-core \ + dh-autoreconf libssl-dev libcap-ng-dev libmnl-dev \ + python3 python3-six + +WORKDIR /ovs + +COPY compile_ovs.sh /compile.sh + +ENTRYPOINT /compile.sh diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_compile_spdk.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_compile_spdk.Dockerfile new file mode 100644 index 0000000..6571d57 --- /dev/null +++ b/smoke-testing/containers/ubuntu18.04/dpdk_compile_spdk.Dockerfile @@ -0,0 +1,15 @@ +FROM ubuntu:18.04 + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && \ + apt-get install --no-install-recommends -y \ + make libnuma-dev \ + g++ nasm uuid-dev libaio-dev libcunit1-dev libssl-dev \ + python + +WORKDIR /spdk + +COPY compile_spdk.sh /compile.sh + +ENTRYPOINT /compile.sh diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_unit_test.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_unit_test.Dockerfile new file mode 100644 index 0000000..7ba8247 --- /dev/null +++ b/smoke-testing/containers/ubuntu18.04/dpdk_unit_test.Dockerfile @@ -0,0 +1,19 @@ +FROM ubuntu:18.04 + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && \ + apt-get install --no-install-recommends -y \ + gcc ninja-build pkg-config libnuma-dev librdmacm-dev librdmacm1 rdma-core libelf-dev \ + python3-pip python3-setuptools python3-wheel + +# Need the latest version of meson, which current repos do not have. +RUN python3 -m pip install meson + +RUN mkdir /build + +WORKDIR /dpdk + +COPY unit_test.sh /unit_test.sh + +ENTRYPOINT /unit_test.sh diff --git a/smoke-testing/containers/unit_test.sh b/smoke-testing/containers/unit_test.sh new file mode 100755 index 0000000..1c53196 --- /dev/null +++ b/smoke-testing/containers/unit_test.sh @@ -0,0 +1,6 @@ +#!/bin/bash -ex +# SPDX-License-Identifier: BSD-3-Clause + +# This script is only used to run inside a container as the ENTRYPOINT. + +meson test -C build --suite fast-tests -t 3 diff --git a/smoke-testing/dry_run.sh b/smoke-testing/dry_run.sh new file mode 100755 index 0000000..d4834a7 --- /dev/null +++ b/smoke-testing/dry_run.sh @@ -0,0 +1,20 @@ +#!/bin/bash -e +# This script is mostly used as a sanity check just to make sure tests don't +# at least die on start up. + +BASEDIR=$(dirname "$0") +ARGS="dpdk.tar.gz --print-shell" + +set -x + +$BASEDIR/compile.py $ARGS +$BASEDIR/compile_with_podman.py $ARGS +$BASEDIR/compile_ovs.py $ARGS +$BASEDIR/compile_ovs_with_podman.py $ARGS +$BASEDIR/compile_spdk.py $ARGS +$BASEDIR/compile_spdk_with_podman.py $ARGS +$BASEDIR/compile_meson.py $ARGS +$BASEDIR/compile_meson_with_podman.py $ARGS +$BASEDIR/compile_meson_with_docker.py $ARGS +$BASEDIR/unit_test.py $ARGS +$BASEDIR/unit_test_with_docker.py $ARGS diff --git a/smoke-testing/unit_test.py b/smoke-testing/unit_test.py new file mode 100755 index 0000000..7c246e5 --- /dev/null +++ b/smoke-testing/unit_test.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import os +import sys + +from compile import CompileArgParse +from compile_meson import CompileMeson + + +class RunUnitTestingArgParse(CompileArgParse): + def __init__(self): + super().__init__('Run DPDK unit testing') + + +class RunUnitTestingMixin: + + def can_use_hugetlbfs(self): + # sanity check to make sure the user can write to /dev/hugepages + # before trying to run the unit tests + if not os.access('/dev/hugepages', os.W_OK): + print('!!! Cannot write to /dev/hugepages. ' + 'See README for more information.') + return False + return True + + +class RunUnitTesting(RunUnitTestingMixin, CompileMeson): + + def run_unit_tests(self): + if not self.can_use_hugetlbfs(): + return 1 + + self.build() + + # run unit tests + self.run(['meson', 'test', '-C', 'build', + '--suite', 'fast-tests', '-t', '3'], + on_error=self.save_logs, cwd='dpdk') + + +if __name__ == '__main__': + args = RunUnitTestingArgParse().parse_args() + + unit_testing = RunUnitTesting(args.print_shell, args.tarball) + sys.exit(unit_testing.run_unit_tests()) diff --git a/smoke-testing/unit_test_with_docker.py b/smoke-testing/unit_test_with_docker.py new file mode 100755 index 0000000..50960b4 --- /dev/null +++ b/smoke-testing/unit_test_with_docker.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu. + +import os +import subprocess +import sys + +from ci import OS +from compile_meson_with_docker import CompileMesonWithDocker +from unit_test import RunUnitTestingArgParse, RunUnitTestingMixin + + +class RunUnitTestingWithDockerArgParse(RunUnitTestingArgParse): + def __init__(self): + super().__init__() + self.parser.add_argument( + '--os', choices=OS.CHOICES, default=OS.CHOICES[0], + help='OS container to use') + + +# TODO replace with podman +# Could not figure out how to run container user with +# group permissions (hugetlbfs) +# If this can run with podman, then CompileMesonWithDocker can be removed. +class RunUnitTestingWithDocker(RunUnitTestingMixin, CompileMesonWithDocker): + + def run_unit_tests(self): + if not self.can_use_hugetlbfs(): + return 1 + + self.build() + + image = 'dpdk_unit_test' + + self.image_build(image, manager='docker') + + hugetlbfs_gid = (subprocess.check_output( + ['getent', 'group', 'hugetlbfs']) + .decode('utf-8') + .split(':')[2]) + + self.container_run(image, [ + '--cap-add', 'IPC_LOCK', + '--cap-add', 'NET_RAW', + '--cap-add', 'NET_ADMIN', + '--cap-add', 'SYS_NICE', + '--user', f'{self.uid()}:{hugetlbfs_gid}', + '-v', os.getcwd() + '/dpdk:/dpdk', + '-v', '/dev/hugepages:/dev/hugepages'], + on_error=self.save_logs, + manager='docker') + + +if __name__ == '__main__': + args = RunUnitTestingWithDockerArgParse().parse_args() + + unit_testing = RunUnitTestingWithDocker( + args.print_shell, args.tarball, args.os) + sys.exit(unit_testing.run_unit_tests()) -- 2.20.1