DPDK CI discussions
 help / color / Atom feed
* [dpdk-ci] [PATCH] Add smoke testing to dpdk-ci
@ 2019-11-08 19:40 Jeremy Plsek
  0 siblings, 0 replies; only message in thread
From: Jeremy Plsek @ 2019-11-08 19:40 UTC (permalink / raw)
  To: ci; +Cc: Jeremy Plsek

This add the following testing to dpdk-ci:
- DPDK compiling
- DPDK unit testing
- OVS with DPDK compiling
- SPDK with DPDK compiling

Compiling with downstream projects is meant to quickly check for API/ABI
breakage.

There are two methods to run this testing:
1. On their own (compile.py, compile_ovs.py, compile_spdk.py)
2. Within a container (*_with_podman.py, *_with_docker.py)

The preference is to run testing with a container using podman.

Since running rootless podman can still have some hiccups, docker is
used in place of it. At the moment, that is just for running unit tests.
Once unit tests can run with podman, the "*_with_docker.py" can be
removed.

Separate images are used to compile different projects. This may one day
change to just have one image to compile all projects.

There are scripts to compile either with the "make" system or the
"meson" system. In the future, once the downstream projects can compile
with the "meson" system, the "make" scripts may be removed.
---
 smoke-testing/README.md                       |  70 +++++
 smoke-testing/ci.py                           | 257 ++++++++++++++++++
 smoke-testing/ci_logging.py                   |  47 ++++
 smoke-testing/compile.py                      | 119 ++++++++
 smoke-testing/compile_meson.py                |  53 ++++
 smoke-testing/compile_meson_with_docker.py    |  32 +++
 smoke-testing/compile_meson_with_podman.py    |  42 +++
 smoke-testing/compile_ovs.py                  |  89 ++++++
 smoke-testing/compile_ovs_with_podman.py      |  34 +++
 smoke-testing/compile_spdk.py                 |  32 +++
 smoke-testing/compile_spdk_with_podman.py     |  35 +++
 smoke-testing/compile_with_podman.py          |  44 +++
 smoke-testing/containers/README.md            |  17 ++
 .../centos8/dpdk_compile.Dockerfile           |  13 +
 .../centos8/dpdk_compile_ovs.Dockerfile       |  17 ++
 smoke-testing/containers/compile.sh           |   7 +
 smoke-testing/containers/compile_meson.sh     |   8 +
 smoke-testing/containers/compile_ovs.sh       |  10 +
 smoke-testing/containers/compile_spdk.sh      |   7 +
 .../ubuntu18.04/dpdk_compile.Dockerfile       |  15 +
 .../ubuntu18.04/dpdk_compile_meson.Dockerfile |  19 ++
 .../ubuntu18.04/dpdk_compile_ovs.Dockerfile   |  15 +
 .../ubuntu18.04/dpdk_compile_spdk.Dockerfile  |  15 +
 .../ubuntu18.04/dpdk_unit_test.Dockerfile     |  19 ++
 smoke-testing/containers/unit_test.sh         |   6 +
 smoke-testing/dry_run.sh                      |  20 ++
 smoke-testing/unit_test.py                    |  47 ++++
 smoke-testing/unit_test_with_docker.py        |  60 ++++
 28 files changed, 1149 insertions(+)
 create mode 100644 smoke-testing/README.md
 create mode 100644 smoke-testing/ci.py
 create mode 100644 smoke-testing/ci_logging.py
 create mode 100755 smoke-testing/compile.py
 create mode 100755 smoke-testing/compile_meson.py
 create mode 100755 smoke-testing/compile_meson_with_docker.py
 create mode 100755 smoke-testing/compile_meson_with_podman.py
 create mode 100755 smoke-testing/compile_ovs.py
 create mode 100755 smoke-testing/compile_ovs_with_podman.py
 create mode 100755 smoke-testing/compile_spdk.py
 create mode 100755 smoke-testing/compile_spdk_with_podman.py
 create mode 100755 smoke-testing/compile_with_podman.py
 create mode 100644 smoke-testing/containers/README.md
 create mode 100644 smoke-testing/containers/centos8/dpdk_compile.Dockerfile
 create mode 100644 smoke-testing/containers/centos8/dpdk_compile_ovs.Dockerfile
 create mode 100755 smoke-testing/containers/compile.sh
 create mode 100755 smoke-testing/containers/compile_meson.sh
 create mode 100755 smoke-testing/containers/compile_ovs.sh
 create mode 100755 smoke-testing/containers/compile_spdk.sh
 create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_compile.Dockerfile
 create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_compile_meson.Dockerfile
 create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_compile_ovs.Dockerfile
 create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_compile_spdk.Dockerfile
 create mode 100644 smoke-testing/containers/ubuntu18.04/dpdk_unit_test.Dockerfile
 create mode 100755 smoke-testing/containers/unit_test.sh
 create mode 100755 smoke-testing/dry_run.sh
 create mode 100755 smoke-testing/unit_test.py
 create mode 100755 smoke-testing/unit_test_with_docker.py

diff --git a/smoke-testing/README.md b/smoke-testing/README.md
new file mode 100644
index 0000000..9fd77d1
--- /dev/null
+++ b/smoke-testing/README.md
@@ -0,0 +1,70 @@
+# Smoke Testing
+
+This assumes Python3.6 as a minimum.
+This is because we currently test with FreeBSD 11.2, CentOS 7,
+and Ubuntu 18.04, which now all use Python3.6+.
+
+Since this is meant to run in a CI environment, logs are are removed on start up.
+They are expected to be saved separately by the CI server when the scripts exit.
+
+## Dependencies
+
+```
+python3 python3-requests python3-paramiko
+```
+
+See the container images for build dependencies if you want to compile without utilizing the containers.
+
+## Running
+
+An example layout of running this:
+
+```
+$HOME
+├─<this directory>
+└─workspace
+  ├─dpdk.tar.gz
+  │ # Below will be generated by the scripts
+  ├─dpdk     # Extracted tarball to build
+  ├─out      # The output directory, containing the logs
+  └─out.zip  # The output directory, but zipped when the scripts exit
+```
+
+```bash
+cd ~/workspace
+../<this directory>/compile.py dpdk.tar.gz
+```
+
+The context is the current directory,
+so the tarball will be extracted in the current directory,
+and logs will be generated in the `out` directory.
+
+## DPDK unit testing
+
+Testing is attempted to run as a non-root user. In order to do this, there is some setup involved.
+
+```bash
+sudo groupadd hugetlbfs
+sudo usermod -a -G hugetlbfs $USER # $USER is the user running the unit tests
+sudo chown -R :hugetlbfs /dev/hugepages
+sudo chmod g+w /dev/hugepages -R
+sudo sh -c "echo 1024 > /proc/sys/vm/nr_hugepages"
+```
+
+## Running OVS Performance testing
+
+This assumes the environment has already been set up. See:
+https://github.com/chaudron/ovs_perf
+
+## Some development thoughts/goals
+
+Currently, the zip is the only artifact generated that is meant for public use.
+Terminal output is mostly meant for debugging.
+You must use the logger for the output to be saved to the artifact log.
+
+Use `ci.run` to log commands and output. Otherwise, `subprocess` can be used. 
+If you want to see the command generated, but not log them,
+use the `ci` variant of `subprocess` if it exists, like `ci.check_call`.
+It should be used over `subprocess` anyways, since it allows options like `--print-shell`
+to avoid actually running the commands.
+That way, setup processes do not get logged, but it still gets printed to the terminal for debugging.
diff --git a/smoke-testing/ci.py b/smoke-testing/ci.py
new file mode 100644
index 0000000..1afa0f4
--- /dev/null
+++ b/smoke-testing/ci.py
@@ -0,0 +1,257 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import argparse
+import atexit
+import os
+import shutil
+import subprocess
+import sys
+from os import path
+from zipfile import ZIP_DEFLATED, ZipFile
+
+import paramiko
+
+from ci_logging import logging, OUTPUT_FOLDER, OUTPUT_ZIP
+
+
+class Password:
+    """Mask the variable"""
+    def __init__(self, var):
+        self.var = var
+
+    def get(self, mask=True):
+        if mask:
+            return '***'
+        else:
+            return self.var
+
+    def __str__(self):
+        return '***'
+
+
+class CIArgParse:
+
+    def __init__(self, description):
+        self.parser = argparse.ArgumentParser(description=description)
+        self.parser.add_argument(
+            '--print-shell', action='store_true',
+            help='Print what would be done in a shell. '
+                 'This *may* not be a dry run. '
+                 'Useful for sending emails.')
+
+    def parse_args(self):
+        return self.parser.parse_args()
+
+
+class OS:
+    CHOICES = ['ubuntu18.04', 'centos8']
+
+
+class CI:
+
+    ssh_cache = {}
+
+    def __init__(self, print_shell):
+        self.print_shell = print_shell
+        atexit.register(self.at_exit)
+
+    def at_exit(self):
+        """
+        When overriding, call super() at the end, to properly save output.
+        """
+        self._close_ssh_cache()
+        self._save_out()
+
+    def _save_out(self):
+        # zip output folder
+        file_paths = self._get_all_file_paths(OUTPUT_FOLDER)
+        with ZipFile(OUTPUT_ZIP, 'w', ZIP_DEFLATED) as zip_file:
+            for file_path in file_paths:
+                zip_file.write(file_path)
+
+    def _get_all_file_paths(self, directory):
+        file_paths = []
+        # crawling through directory and subdirectories
+        for root, directories, files in os.walk(directory):
+            for filename in files:
+                filepath = os.path.join(root, filename)
+                file_paths.append(filepath)
+        return file_paths
+
+    def parse_command(self, command, mask=True):
+        new_command = []
+        for word in command:
+            # not sure of a cleaner way
+            if isinstance(word, Password):
+                new_command.append(word.get(mask))
+            else:
+                new_command.append(word)
+        return new_command
+
+    def _print_cd(self, print_cmd, **kwargs):
+        if 'cwd' in kwargs:
+            logging.info('pushd ' + kwargs['cwd'])
+
+        logging.info(print_cmd)
+
+        if 'cwd' in kwargs:
+            logging.info('popd')
+
+    def run(self, command, exit_on_error=True, on_error=None, **kwargs):
+        """Prints to output, and log to file, exit on fail."""
+        print_cmd = ' '.join(self.parse_command(command))
+
+        if self.print_shell:
+            self._print_cd(print_cmd, **kwargs)
+            return
+
+        logging.info('+ ' + print_cmd)
+
+        try:
+            proc = subprocess.Popen(
+                self.parse_command(command, False),
+                stdout=subprocess.PIPE,
+                stderr=subprocess.STDOUT,
+                **kwargs)
+
+            # live output
+            for line in iter(proc.stdout.readline, b''):
+                logging.info(line.decode('utf-8').strip('\n'))
+
+            proc.communicate()
+
+            if proc.returncode:
+                logging.info(f'[{proc.returncode}] Error running command.')
+                if on_error:
+                    on_error()
+                if exit_on_error:
+                    sys.exit(proc.returncode)
+                return proc.returncode
+
+        except FileNotFoundError:
+
+            logging.info(f'Command not found: {command[0]}')
+            if on_error:
+                on_error()
+            if exit_on_error:
+                sys.exit(1)
+
+        except OSError as e:
+
+            logging.info('Running command failed.')
+            logging.info(str(e))
+            if on_error:
+                on_error()
+            if exit_on_error:
+                sys.exit(1)
+
+    def ssh(self, host, user, password, command,
+            exit_on_error=True, on_error=None):
+        if self.print_shell:
+            logging.info(command)
+            return
+
+        key = host + ':' + user
+
+        if key in self.ssh_cache:
+            client = self.ssh_cache[key]
+        else:
+            client = paramiko.SSHClient()
+            client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+            client.connect(host, username=user, password=password)
+            self.ssh_cache[key] = client
+
+        logging.info('+ ' + command)
+
+        stdin, stdout, stderr = client.exec_command(command)
+
+        for line in stdout:
+            logging.info(line.strip('\n'))
+        for line in stderr:
+            logging.info(line.strip('\n'))
+
+        return_code = stdout.channel.recv_exit_status()
+        if return_code:
+            logging.info(f'[{return_code}] Error running command.')
+            if on_error:
+                on_error()
+            if exit_on_error:
+                sys.exit(return_code)
+
+    def check_call(self, command, **kwargs):
+        """Prints the command to stdout and log command on print_shell"""
+        print_cmd = ' '.join(command)
+
+        if self.print_shell:
+            self._print_cd(print_cmd, **kwargs)
+            return
+
+        print('+ ' + print_cmd)
+
+        subprocess.check_call(command, **kwargs)
+
+    def _close_ssh_cache(self):
+        for client in self.ssh_cache:
+            self.ssh_cache[client].close()
+
+    def delete(self, tree):
+        if path.exists(tree):
+            shutil.rmtree(tree)
+
+
+class ContainerMixin:
+
+    def image_build(self, image, args=(), manager='podman'):
+        script_dir = os.path.dirname(os.path.realpath(__file__))
+
+        image = self.os + '/' + image
+
+        self.check_call((manager, 'build', '-t', image,
+                         '-f', image + '.Dockerfile', '.') + args,
+                        cwd=script_dir + '/containers')
+
+    def container_run(self, image, args, on_error=None, manager='podman'):
+        cmd = [manager, 'run', '--rm']
+
+        if sys.stdout.isatty():
+            cmd += ['-it']
+
+        image = self.os + '/' + image
+
+        self.run(cmd + args + [image], on_error=on_error)
+
+    def kernel(self):
+        version = (subprocess.check_output(['uname', '-r'])
+                   .decode('utf-8')
+                   .strip('\n'))
+
+        if 'debian' in self.id_like:
+            return 'linux-headers-' + version
+
+        return version
+
+    def uid(self):
+        return (subprocess.check_output(['id', '-u'])
+                .decode('utf-8')
+                .strip('\n'))
+
+    def gid(self):
+        return (subprocess.check_output(['id', '-g'])
+                .decode('utf-8')
+                .strip('\n'))
+
+
+class Utils:
+    @staticmethod
+    def parse_properties(f):
+        # return a dictionary from a property-like file
+        # (key values separated by equals)
+        d = {}
+        for line in f:
+            line = line.rstrip()
+            if not line:
+                continue
+            k, v = line.split('=')
+            d[k] = v
+        return d
diff --git a/smoke-testing/ci_logging.py b/smoke-testing/ci_logging.py
new file mode 100644
index 0000000..0c61efb
--- /dev/null
+++ b/smoke-testing/ci_logging.py
@@ -0,0 +1,47 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import logging.config
+import os
+import shutil
+
+OUTPUT_FOLDER = 'out'
+OUTPUT_ZIP = 'out.zip'
+
+# make output folder
+if os.path.exists(OUTPUT_FOLDER):
+    shutil.rmtree(OUTPUT_FOLDER)
+os.mkdir(OUTPUT_FOLDER)
+
+# and remove old zip
+if os.path.exists(OUTPUT_ZIP):
+    os.remove(OUTPUT_ZIP)
+
+logging.config.dictConfig({
+    'version': 1,
+    'disable_existing_loggers': False,
+
+    'formatters': {
+        'basic': {
+            'format': '%(message)s',
+        },
+    },
+    'handlers': {
+        'console': {
+            'class': 'logging.StreamHandler',
+            'formatter': 'basic',
+            'level': 'INFO',
+        },
+        'file': {
+            'class': 'logging.FileHandler',
+            'filename': 'out/log.txt',
+            'mode': 'w',
+            'formatter': 'basic',
+            'level': 'INFO',
+        },
+    },
+    'root': {
+        'handlers': ['console', 'file'],
+        'level': 'NOTSET',
+    },
+})
diff --git a/smoke-testing/compile.py b/smoke-testing/compile.py
new file mode 100755
index 0000000..b27990e
--- /dev/null
+++ b/smoke-testing/compile.py
@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import multiprocessing
+import re
+import subprocess
+
+from ci import CI, CIArgParse, Utils
+from ci_logging import logging
+
+
+class CompileArgParse(CIArgParse):
+    def __init__(self, description='DPDK compile testing'):
+        super().__init__(description)
+        self.parser.add_argument('tarball', help='DPDK tarball')
+
+
+class CompileMixin:
+
+    def init_dpdk(self):
+        # delete previously extracted tarball
+        self.delete('dpdk')
+
+        # untar tarball (creates dpdk folder)
+        self.check_call(['tar', 'xzf', self.tarball])
+
+
+class Compile(CompileMixin, CI):
+
+    def __init__(self, print_shell, tarball):
+        super().__init__(print_shell)
+
+        self.tarball = tarball
+
+        out = (subprocess.check_output(['uname', '-s'])
+               .decode('utf-8')
+               .strip('\n'))
+
+        if out == 'FreeBSD':
+            self.make = 'gmake'
+            self.execenv = 'freebsd'
+        else:
+            self.make = 'make'
+            self.execenv = 'linux'
+            self.kernels = '/usr/src'
+
+            with open('/etc/os-release') as f:
+                properties = Utils.parse_properties(f)
+
+            self.id_like = properties['ID_LIKE']
+
+            # use 'in' instead of equality because there can be quotes and
+            # multiple values in ID_LIKE
+            if 'rhel' in self.id_like:
+                self.kernels += '/kernels'
+
+        # meant to be used in string concatenation
+        self.cpu_count = str(multiprocessing.cpu_count())
+
+    def get_build_options(self):
+        options = [
+            'CONFIG_RTE_ENABLE_ASSERT',
+            'CONFIG_RTE_EAL_ALLOW_INV_SOCKET_ID',
+            'CONFIG_RTE_EAL_ALWAYS_PANIC_ON_ERROR',
+            'CONFIG_RTE_ENABLE_AVX512',
+            # 'CONFIG_RTE_ETHDEV_PROFILE_WITH_VTUNE', # breaks
+            'CONFIG_RTE_ETHDEV_TX_PREPARE_NOOP',
+        ]
+
+        if self.execenv == 'freebsd':
+            options += [
+                'CONFIG_RTE_USE_LIBBSD',
+            ]
+        else:
+            # These do not work in freebsd
+            options += [
+                'CONFIG_RTE_LIBRTE_MLX4_PMD',
+                'CONFIG_RTE_LIBRTE_MLX5_PMD',
+                'CONFIG_RTE_LIBEAL_USE_HPET',
+            ]
+
+        return options
+
+    def apply_build_options(self):
+        options = self.get_build_options()
+
+        logging.info('Enabling compile options:\n' + '\n'.join(options))
+
+        if self.print_shell:
+            return
+
+        with open('dpdk/config/common_base') as f:
+            content = f.read()
+
+        for option in options:
+            content = re.sub(r'(' + option + r'=)n', r'\1y', content)
+
+        with open('dpdk/config/common_base', 'w') as f:
+            f.write(content)
+
+    def build(self):
+        self.init_dpdk()
+
+        self.apply_build_options()
+
+        # configure dpdk
+        self.run([self.make, 'config', f'T=x86_64-native-{self.execenv}-gcc'],
+                 cwd='dpdk')
+
+        # compile dpdk
+        self.run([self.make, '-j' + self.cpu_count], cwd='dpdk')
+
+
+if __name__ == '__main__':
+    args = CompileArgParse().parse_args()
+
+    compile_dpdk = Compile(args.print_shell, args.tarball)
+    compile_dpdk.build()
diff --git a/smoke-testing/compile_meson.py b/smoke-testing/compile_meson.py
new file mode 100755
index 0000000..e685e5a
--- /dev/null
+++ b/smoke-testing/compile_meson.py
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import os
+import shutil
+
+from ci import CI
+from ci_logging import OUTPUT_FOLDER
+from compile import CompileArgParse, CompileMixin
+
+
+class CompileMesonArgParse(CompileArgParse):
+    def __init__(self, description='Compile DPDK with meson'):
+        super().__init__(description)
+
+
+class CompileMeson(CompileMixin, CI):
+
+    def __init__(self, print_shell, tarball):
+        super().__init__(print_shell)
+        self.tarball = tarball
+
+    def save_logs(self):
+        # save build logs (only needed on error)
+        location = 'dpdk/build/'
+        for name in ['meson-logs/testlog.txt', '.ninja_log',
+                     'meson-logs/meson-log.txt']:
+            full_name = location + name
+            if os.path.exists(full_name):
+                shutil.copy(full_name, OUTPUT_FOLDER)
+
+    def build(self):
+        self.init_dpdk()
+
+        # build dpdk
+        self.run(['meson', 'build', '--werror', '-Dexamples=all',
+                  '--default-library', 'shared'],
+                 on_error=self.save_logs, cwd='dpdk')
+
+        self.run(['ninja', '-C', 'build'],
+                 on_error=self.save_logs, cwd='dpdk')
+
+        # build docs
+#        self.run(['ninja', '-C', 'build', 'doc'],
+#                 on_error=self.save_logs, cwd='dpdk')
+
+
+if __name__ == '__main__':
+    args = CompileMesonArgParse().parse_args()
+
+    compile_dpdk = CompileMeson(args.print_shell, args.tarball)
+    compile_dpdk.build()
diff --git a/smoke-testing/compile_meson_with_docker.py b/smoke-testing/compile_meson_with_docker.py
new file mode 100755
index 0000000..1b6f8f6
--- /dev/null
+++ b/smoke-testing/compile_meson_with_docker.py
@@ -0,0 +1,32 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import os
+
+from compile_meson_with_podman import CompileMesonWithPodman, \
+    CompileWithPodmanArgParse
+
+
+class CompileMesonWithDocker(CompileMesonWithPodman):
+
+    def build(self):
+        self.init_dpdk()
+
+        image = 'dpdk_compile_meson'
+
+        self.image_build(image, manager='docker')
+
+        self.container_run(image, [
+            '--user', f'{self.uid()}:{self.gid()}',
+            '-v', os.getcwd() + '/dpdk:/dpdk'],
+                           on_error=self.save_logs,
+                           manager='docker')
+
+
+if __name__ == '__main__':
+    args = CompileWithPodmanArgParse().parse_args()
+
+    compile_dpdk = CompileMesonWithDocker(
+        args.print_shell, args.tarball, args.os)
+    compile_dpdk.build()
diff --git a/smoke-testing/compile_meson_with_podman.py b/smoke-testing/compile_meson_with_podman.py
new file mode 100755
index 0000000..ef5fb9c
--- /dev/null
+++ b/smoke-testing/compile_meson_with_podman.py
@@ -0,0 +1,42 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import os
+
+from ci import ContainerMixin, OS
+from compile_meson import CompileMeson, CompileMesonArgParse
+
+
+class CompileWithPodmanArgParse(CompileMesonArgParse):
+    def __init__(self, description='Compile DPDK with meson in a container'):
+        super().__init__(description)
+        self.parser.add_argument(
+            '--os', choices=OS.CHOICES, default=OS.CHOICES[0],
+            help='OS container to use')
+
+
+class CompileMesonWithPodman(ContainerMixin, CompileMeson):
+
+    def __init__(self, print_shell, tarball, os):
+        super().__init__(print_shell, tarball)
+        self.os = os
+
+    def build(self):
+        self.init_dpdk()
+
+        image = 'dpdk_compile_meson'
+
+        self.image_build(image)
+
+        self.container_run(image, [
+            '-v', os.getcwd() + '/dpdk:/dpdk:Z'],
+                           on_error=self.save_logs)
+
+
+if __name__ == '__main__':
+    args = CompileWithPodmanArgParse().parse_args()
+
+    compile_dpdk = CompileMesonWithPodman(
+        args.print_shell, args.tarball, args.os)
+    compile_dpdk.build()
diff --git a/smoke-testing/compile_ovs.py b/smoke-testing/compile_ovs.py
new file mode 100755
index 0000000..aa360c2
--- /dev/null
+++ b/smoke-testing/compile_ovs.py
@@ -0,0 +1,89 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import os
+from subprocess import check_call
+
+from compile import Compile, CompileArgParse
+
+
+class OVSCompileArgParse(CompileArgParse):
+    def __init__(self):
+        super().__init__('OVS Compile testing. This assumes an up-to-date ovs '
+                         'git folder exists in the current directory.')
+        self.parser.add_argument('--install', action='store_true',
+                                 help='Whether to run "make install"')
+
+
+class CompileOVS(Compile):
+    def __init__(self, print_shell, tarball, install):
+        super().__init__(print_shell, tarball)
+
+        self.install = install
+
+    def build(self):
+        super().build()
+
+        # create build directory
+        build = os.path.join('ovs', 'build')
+
+        if self.install:
+            # running "sudo make install" changes permissions to root
+            # so use root to remove
+            check_call(['sudo', 'rm', '-rf', build])
+        else:
+            self.delete(build)
+
+        os.mkdir(build)
+
+        # run boot script
+        self.run(['./boot.sh'], cwd='ovs')
+
+        # run configure script
+        cmd = [
+            '../configure',
+            '--enable-static',
+            '--disable-shared',
+            '--enable-ssl',
+            '--with-dpdk=../../dpdk/build',
+        ]
+
+        if self.execenv != 'freebsd':
+            cmd += [
+                '--enable-libcapng',
+            ]
+
+        if self.install:
+            cmd += [
+                '--program-prefix=',
+                '--disable-dependency-tracking',
+                '--prefix=/usr',
+                '--exec-prefix=/usr',
+                '--bindir=/usr/bin',
+                '--sbindir=/usr/sbin',
+                '--sysconfdir=/etc',
+                '--datadir=/usr/share',
+                '--includedir=/usr/include',
+                '--libdir=/usr/lib64',
+                '--libexecdir=/usr/libexec',
+                '--localstatedir=/var',
+                '--sharedstatedir=/var/lib',
+                '--mandir=/usr/share/man',
+                '--infodir=/usr/share/info'
+            ]
+
+        self.run(cmd, cwd=build)
+
+        # compile ovs
+        self.run([self.make, '-j' + self.cpu_count], cwd=build)
+
+        if self.install:
+            self.run(['sudo', self.make, 'install'], cwd=build)
+
+
+if __name__ == '__main__':
+    args = OVSCompileArgParse().parse_args()
+
+    compile_ovs = CompileOVS(args.print_shell, args.tarball, args.install)
+    compile_ovs.build()
diff --git a/smoke-testing/compile_ovs_with_podman.py b/smoke-testing/compile_ovs_with_podman.py
new file mode 100755
index 0000000..2d633d5
--- /dev/null
+++ b/smoke-testing/compile_ovs_with_podman.py
@@ -0,0 +1,34 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import os
+
+from compile_with_podman import CompileWithPodman, CompileWithPodmanArgParse
+
+
+class OVSCompileArgParse(CompileWithPodmanArgParse):
+    def __init__(self):
+        super().__init__('OVS Compile testing. This assumes an up-to-date ovs '
+                         'git folder exists in the current directory.')
+
+
+class CompileOVSWithPodman(CompileWithPodman):
+
+    def build(self):
+        super().build()
+
+        image = 'dpdk_compile_ovs'
+
+        self.image_build(image)
+
+        self.container_run(image, [
+            '-v', os.getcwd() + '/dpdk:/dpdk:Z',
+            '-v', os.getcwd() + '/ovs:/ovs:Z'])
+
+
+if __name__ == '__main__':
+    args = OVSCompileArgParse().parse_args()
+
+    compile_ovs = CompileOVSWithPodman(args.print_shell, args.tarball, args.os)
+    compile_ovs.build()
diff --git a/smoke-testing/compile_spdk.py b/smoke-testing/compile_spdk.py
new file mode 100755
index 0000000..9600fc1
--- /dev/null
+++ b/smoke-testing/compile_spdk.py
@@ -0,0 +1,32 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+from compile import Compile, CompileArgParse
+
+
+class SPDKCompileArgParse(CompileArgParse):
+    def __init__(self):
+        super().__init__('SPDK Compile testing. This assumes an up-to-date '
+                         'spdk git folder exists in the current directory.')
+
+
+class CompileSPDK(Compile):
+    def build(self):
+        super().build()
+
+        # run configure script
+        self.run([
+            './configure',
+            '--with-dpdk=../dpdk/build',
+        ], cwd='spdk')
+
+        # compile spdk
+        self.run([self.make, '-j' + self.cpu_count], cwd='spdk')
+
+
+if __name__ == '__main__':
+    args = SPDKCompileArgParse().parse_args()
+
+    compile_spdk = CompileSPDK(args.print_shell, args.tarball)
+    compile_spdk.build()
diff --git a/smoke-testing/compile_spdk_with_podman.py b/smoke-testing/compile_spdk_with_podman.py
new file mode 100755
index 0000000..9adad7f
--- /dev/null
+++ b/smoke-testing/compile_spdk_with_podman.py
@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import os
+
+from compile_with_podman import CompileWithPodman, CompileWithPodmanArgParse
+
+
+class SPDKCompileArgParse(CompileWithPodmanArgParse):
+    def __init__(self):
+        super().__init__('SPDK Compile testing. This assumes an up-to-date '
+                         'spdk git folder exists in the current directory.')
+
+
+class CompileSPDKWithPodman(CompileWithPodman):
+
+    def build(self):
+        super().build()
+
+        image = 'dpdk_compile_spdk'
+
+        self.image_build(image)
+
+        self.container_run(image, [
+            '-v', os.getcwd() + '/dpdk:/dpdk:Z',
+            '-v', os.getcwd() + '/spdk:/spdk:Z'])
+
+
+if __name__ == '__main__':
+    args = SPDKCompileArgParse().parse_args()
+
+    compile_spdk = CompileSPDKWithPodman(
+        args.print_shell, args.tarball, args.os)
+    compile_spdk.build()
diff --git a/smoke-testing/compile_with_podman.py b/smoke-testing/compile_with_podman.py
new file mode 100755
index 0000000..1d68de9
--- /dev/null
+++ b/smoke-testing/compile_with_podman.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import os
+
+from ci import ContainerMixin, OS
+from compile import Compile, CompileArgParse
+
+
+class CompileWithPodmanArgParse(CompileArgParse):
+    def __init__(self, description='Compile dpdk in a container'):
+        super().__init__(description)
+        self.parser.add_argument(
+            '--os', choices=OS.CHOICES, default=OS.CHOICES[0],
+            help='OS container to use')
+
+
+class CompileWithPodman(ContainerMixin, Compile):
+
+    def __init__(self, print_shell, tarball, os):
+        super().__init__(print_shell, tarball)
+        self.os = os
+
+    def build(self):
+        self.init_dpdk()
+
+        self.apply_build_options()
+
+        image = 'dpdk_compile'
+
+        self.image_build(image)
+
+        self.container_run(image, [
+            '-e', f'RTE_KERNELDIR={self.kernels}/{self.kernel()}',
+            '-v', os.getcwd() + '/dpdk:/dpdk:Z',
+            '-v', f'{self.kernels}:{self.kernels}:ro'])
+
+
+if __name__ == '__main__':
+    args = CompileWithPodmanArgParse().parse_args()
+
+    compile_dpdk = CompileWithPodman(args.print_shell, args.tarball, args.os)
+    compile_dpdk.build()
diff --git a/smoke-testing/containers/README.md b/smoke-testing/containers/README.md
new file mode 100644
index 0000000..a4a260c
--- /dev/null
+++ b/smoke-testing/containers/README.md
@@ -0,0 +1,17 @@
+# Containers
+
+This is used with the `*_with_podman.py` scripts,
+but they can be ran on their own too for testing.
+
+Run as:
+
+```bash
+export IMAGE= # image to use (like "ubuntu18.04/dpdk_compile")
+export DPDK= # dpdk directory (like "$(pwd)/dpdk")
+podman build -t ${IMAGE} -f ${IMAGE}.Dockerfile .
+podman run --rm -it -e "RTE_KERNELDIR=/usr/src/linux-headers-$(uname -r)" -v ${DPDK}:/dpdk -v /usr/src:/usr/src:ro ${IMAGE}
+```
+
+Since symlinks may be used, the whole /usr/src folder is mounted.
+
+Docker can also be used.
diff --git a/smoke-testing/containers/centos8/dpdk_compile.Dockerfile b/smoke-testing/containers/centos8/dpdk_compile.Dockerfile
new file mode 100644
index 0000000..74d3c5d
--- /dev/null
+++ b/smoke-testing/containers/centos8/dpdk_compile.Dockerfile
@@ -0,0 +1,13 @@
+FROM centos:8
+
+RUN dnf install -y \
+    diffutils \
+    gcc make pkg-config numactl-devel librdmacm rdma-core-devel elfutils-libelf-devel
+
+WORKDIR /dpdk
+
+COPY compile.sh /compile.sh
+
+ENV RTE_KERNELDIR CHANGE_ME
+
+ENTRYPOINT /compile.sh
diff --git a/smoke-testing/containers/centos8/dpdk_compile_ovs.Dockerfile b/smoke-testing/containers/centos8/dpdk_compile_ovs.Dockerfile
new file mode 100644
index 0000000..017b2ea
--- /dev/null
+++ b/smoke-testing/containers/centos8/dpdk_compile_ovs.Dockerfile
@@ -0,0 +1,17 @@
+FROM centos:8
+
+# For libmnl-devel
+RUN dnf install -y dnf-command\(config-manager\)
+RUN dnf config-manager --set-enabled PowerTools
+
+RUN dnf install -y \
+    diffutils \
+    gcc make numactl-devel librdmacm rdma-core-devel \
+    openssl-devel libcap-ng-devel autoconf automake libtool libmnl-devel \
+    python3 python3-six
+
+WORKDIR /ovs
+
+COPY compile_ovs.sh /compile.sh
+
+ENTRYPOINT /compile.sh
diff --git a/smoke-testing/containers/compile.sh b/smoke-testing/containers/compile.sh
new file mode 100755
index 0000000..f10f45e
--- /dev/null
+++ b/smoke-testing/containers/compile.sh
@@ -0,0 +1,7 @@
+#!/bin/bash -ex
+# SPDX-License-Identifier: BSD-3-Clause
+
+# This script is only used to run inside a container as the ENTRYPOINT.
+
+make config T=x86_64-native-linux-gcc
+make -j$(nproc)
diff --git a/smoke-testing/containers/compile_meson.sh b/smoke-testing/containers/compile_meson.sh
new file mode 100755
index 0000000..bb56083
--- /dev/null
+++ b/smoke-testing/containers/compile_meson.sh
@@ -0,0 +1,8 @@
+#!/bin/bash -ex
+# SPDX-License-Identifier: BSD-3-Clause
+
+# This script is only used to run inside a container as the ENTRYPOINT.
+
+# build dpdk
+meson build --werror -Dexamples=all --default-library shared
+ninja -C build
diff --git a/smoke-testing/containers/compile_ovs.sh b/smoke-testing/containers/compile_ovs.sh
new file mode 100755
index 0000000..6859c24
--- /dev/null
+++ b/smoke-testing/containers/compile_ovs.sh
@@ -0,0 +1,10 @@
+#!/bin/bash -ex
+# SPDX-License-Identifier: BSD-3-Clause
+
+# This script is only used to run inside a container as the ENTRYPOINT.
+
+./boot.sh
+mkdir -p build
+cd build
+../configure --enable-static --disable-shared --enable-ssl --enable-libcapng --with-dpdk=/dpdk/build
+make -j$(nproc)
diff --git a/smoke-testing/containers/compile_spdk.sh b/smoke-testing/containers/compile_spdk.sh
new file mode 100755
index 0000000..46af67c
--- /dev/null
+++ b/smoke-testing/containers/compile_spdk.sh
@@ -0,0 +1,7 @@
+#!/bin/bash -ex
+# SPDX-License-Identifier: BSD-3-Clause
+
+# This script is only used to run inside a container as the ENTRYPOINT.
+
+./configure --with-dpdk=/dpdk/build
+make -j$(nproc)
diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_compile.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_compile.Dockerfile
new file mode 100644
index 0000000..a63b59b
--- /dev/null
+++ b/smoke-testing/containers/ubuntu18.04/dpdk_compile.Dockerfile
@@ -0,0 +1,15 @@
+FROM ubuntu:18.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y \
+    gcc pkg-config libnuma-dev librdmacm-dev librdmacm1 rdma-core libelf-dev
+
+WORKDIR /dpdk
+
+COPY compile.sh /compile.sh
+
+ENV RTE_KERNELDIR CHANGE_ME
+
+ENTRYPOINT /compile.sh
diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_compile_meson.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_compile_meson.Dockerfile
new file mode 100644
index 0000000..244e085
--- /dev/null
+++ b/smoke-testing/containers/ubuntu18.04/dpdk_compile_meson.Dockerfile
@@ -0,0 +1,19 @@
+FROM ubuntu:18.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y \
+    gcc ninja-build pkg-config libnuma-dev librdmacm-dev librdmacm1 rdma-core libelf-dev \
+    python3-pip python3-setuptools python3-wheel
+
+# Need the latest version of meson, which current repos do not have.
+RUN python3 -m pip install meson
+
+RUN mkdir /build
+
+WORKDIR /dpdk
+
+COPY compile_meson.sh /compile.sh
+
+ENTRYPOINT /compile.sh
diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_compile_ovs.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_compile_ovs.Dockerfile
new file mode 100644
index 0000000..8cf9e8e
--- /dev/null
+++ b/smoke-testing/containers/ubuntu18.04/dpdk_compile_ovs.Dockerfile
@@ -0,0 +1,15 @@
+FROM ubuntu:18.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y \
+    gcc libnuma-dev librdmacm-dev librdmacm1 rdma-core \
+    dh-autoreconf libssl-dev libcap-ng-dev libmnl-dev \
+    python3 python3-six
+
+WORKDIR /ovs
+
+COPY compile_ovs.sh /compile.sh
+
+ENTRYPOINT /compile.sh
diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_compile_spdk.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_compile_spdk.Dockerfile
new file mode 100644
index 0000000..6571d57
--- /dev/null
+++ b/smoke-testing/containers/ubuntu18.04/dpdk_compile_spdk.Dockerfile
@@ -0,0 +1,15 @@
+FROM ubuntu:18.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y \
+    make libnuma-dev \
+    g++ nasm uuid-dev libaio-dev libcunit1-dev libssl-dev \
+    python
+
+WORKDIR /spdk
+
+COPY compile_spdk.sh /compile.sh
+
+ENTRYPOINT /compile.sh
diff --git a/smoke-testing/containers/ubuntu18.04/dpdk_unit_test.Dockerfile b/smoke-testing/containers/ubuntu18.04/dpdk_unit_test.Dockerfile
new file mode 100644
index 0000000..7ba8247
--- /dev/null
+++ b/smoke-testing/containers/ubuntu18.04/dpdk_unit_test.Dockerfile
@@ -0,0 +1,19 @@
+FROM ubuntu:18.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y \
+    gcc ninja-build pkg-config libnuma-dev librdmacm-dev librdmacm1 rdma-core libelf-dev \
+    python3-pip python3-setuptools python3-wheel
+
+# Need the latest version of meson, which current repos do not have.
+RUN python3 -m pip install meson
+
+RUN mkdir /build
+
+WORKDIR /dpdk
+
+COPY unit_test.sh /unit_test.sh
+
+ENTRYPOINT /unit_test.sh
diff --git a/smoke-testing/containers/unit_test.sh b/smoke-testing/containers/unit_test.sh
new file mode 100755
index 0000000..1c53196
--- /dev/null
+++ b/smoke-testing/containers/unit_test.sh
@@ -0,0 +1,6 @@
+#!/bin/bash -ex
+# SPDX-License-Identifier: BSD-3-Clause
+
+# This script is only used to run inside a container as the ENTRYPOINT.
+
+meson test -C build --suite fast-tests -t 3
diff --git a/smoke-testing/dry_run.sh b/smoke-testing/dry_run.sh
new file mode 100755
index 0000000..d4834a7
--- /dev/null
+++ b/smoke-testing/dry_run.sh
@@ -0,0 +1,20 @@
+#!/bin/bash -e
+# This script is mostly used as a sanity check just to make sure tests don't
+# at least die on start up.
+
+BASEDIR=$(dirname "$0")
+ARGS="dpdk.tar.gz --print-shell"
+
+set -x
+
+$BASEDIR/compile.py $ARGS
+$BASEDIR/compile_with_podman.py $ARGS
+$BASEDIR/compile_ovs.py $ARGS
+$BASEDIR/compile_ovs_with_podman.py $ARGS
+$BASEDIR/compile_spdk.py $ARGS
+$BASEDIR/compile_spdk_with_podman.py $ARGS
+$BASEDIR/compile_meson.py $ARGS
+$BASEDIR/compile_meson_with_podman.py $ARGS
+$BASEDIR/compile_meson_with_docker.py $ARGS
+$BASEDIR/unit_test.py $ARGS
+$BASEDIR/unit_test_with_docker.py $ARGS
diff --git a/smoke-testing/unit_test.py b/smoke-testing/unit_test.py
new file mode 100755
index 0000000..7c246e5
--- /dev/null
+++ b/smoke-testing/unit_test.py
@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import os
+import sys
+
+from compile import CompileArgParse
+from compile_meson import CompileMeson
+
+
+class RunUnitTestingArgParse(CompileArgParse):
+    def __init__(self):
+        super().__init__('Run DPDK unit testing')
+
+
+class RunUnitTestingMixin:
+
+    def can_use_hugetlbfs(self):
+        # sanity check to make sure the user can write to /dev/hugepages
+        # before trying to run the unit tests
+        if not os.access('/dev/hugepages', os.W_OK):
+            print('!!! Cannot write to /dev/hugepages. '
+                  'See README for more information.')
+            return False
+        return True
+
+
+class RunUnitTesting(RunUnitTestingMixin, CompileMeson):
+
+    def run_unit_tests(self):
+        if not self.can_use_hugetlbfs():
+            return 1
+
+        self.build()
+
+        # run unit tests
+        self.run(['meson', 'test', '-C', 'build',
+                  '--suite', 'fast-tests', '-t', '3'],
+                 on_error=self.save_logs, cwd='dpdk')
+
+
+if __name__ == '__main__':
+    args = RunUnitTestingArgParse().parse_args()
+
+    unit_testing = RunUnitTesting(args.print_shell, args.tarball)
+    sys.exit(unit_testing.run_unit_tests())
diff --git a/smoke-testing/unit_test_with_docker.py b/smoke-testing/unit_test_with_docker.py
new file mode 100755
index 0000000..50960b4
--- /dev/null
+++ b/smoke-testing/unit_test_with_docker.py
@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2019 DPDK Project. Developed by UNH-IOL dpdklab@iol.unh.edu.
+
+import os
+import subprocess
+import sys
+
+from ci import OS
+from compile_meson_with_docker import CompileMesonWithDocker
+from unit_test import RunUnitTestingArgParse, RunUnitTestingMixin
+
+
+class RunUnitTestingWithDockerArgParse(RunUnitTestingArgParse):
+    def __init__(self):
+        super().__init__()
+        self.parser.add_argument(
+            '--os', choices=OS.CHOICES, default=OS.CHOICES[0],
+            help='OS container to use')
+
+
+# TODO replace with podman
+# Could not figure out how to run container user with
+# group permissions (hugetlbfs)
+# If this can run with podman, then CompileMesonWithDocker can be removed.
+class RunUnitTestingWithDocker(RunUnitTestingMixin, CompileMesonWithDocker):
+
+    def run_unit_tests(self):
+        if not self.can_use_hugetlbfs():
+            return 1
+
+        self.build()
+
+        image = 'dpdk_unit_test'
+
+        self.image_build(image, manager='docker')
+
+        hugetlbfs_gid = (subprocess.check_output(
+            ['getent', 'group', 'hugetlbfs'])
+            .decode('utf-8')
+            .split(':')[2])
+
+        self.container_run(image, [
+            '--cap-add', 'IPC_LOCK',
+            '--cap-add', 'NET_RAW',
+            '--cap-add', 'NET_ADMIN',
+            '--cap-add', 'SYS_NICE',
+            '--user', f'{self.uid()}:{hugetlbfs_gid}',
+            '-v', os.getcwd() + '/dpdk:/dpdk',
+            '-v', '/dev/hugepages:/dev/hugepages'],
+                           on_error=self.save_logs,
+                           manager='docker')
+
+
+if __name__ == '__main__':
+    args = RunUnitTestingWithDockerArgParse().parse_args()
+
+    unit_testing = RunUnitTestingWithDocker(
+        args.print_shell, args.tarball, args.os)
+    sys.exit(unit_testing.run_unit_tests())
-- 
2.20.1


^ permalink raw reply	[flat|nested] only message in thread

only message in thread, back to index

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-08 19:40 [dpdk-ci] [PATCH] Add smoke testing to dpdk-ci Jeremy Plsek

DPDK CI discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/ci/0 ci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ci ci/ http://inbox.dpdk.org/ci \
		ci@dpdk.org
	public-inbox-index ci


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.ci


AGPL code for this site: git clone https://public-inbox.org/ public-inbox