test suite reviews and discussions
 help / color / mirror / Atom feed
* [dts] [PATCH v2] ci/initial: Added script to get the tests for a patchset
@ 2021-08-12 15:36 ohilyard
  2021-09-03  9:42 ` Juraj Linkeš
  0 siblings, 1 reply; 4+ messages in thread
From: ohilyard @ 2021-08-12 15:36 UTC (permalink / raw)
  To: dts; +Cc: lijuan.tu, Owen Hilyard

From: Owen Hilyard <ohilyard@iol.unh.edu>

This script should be run after the patchset has been applied.
It will check all files that have a diff to the git ref in
DTS_MAIN_BRANCH_REF (currently origin/master).

It will also issue warnings to standard error if a "protected path" is
changed. This is currently configured to only by the ci scripts folder,
since under most circumstances a patch should not need to change anything
in there. This warning will be in the format:
"WARNING: {file_name} is protected"

The script will also issue a warning if a config file is changed. This
warning is also sent to standard error and takes the form of:
"WARNING: {file_name} is a config file and was changed"

The script will output a list of the test suites to run to standard out,
with each entry having one line.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
 ci/Dockerfile                |  18 ++++
 ci/README.txt                |  37 +++++++
 ci/build_image.sh            |   0
 ci/get_tests_for_patchset.py | 197 +++++++++++++++++++++++++++++++++++
 ci/requirements.txt          |  32 ++++++
 5 files changed, 284 insertions(+)
 create mode 100644 ci/Dockerfile
 create mode 100644 ci/README.txt
 create mode 100644 ci/build_image.sh
 create mode 100644 ci/get_tests_for_patchset.py
 create mode 100644 ci/requirements.txt

diff --git a/ci/Dockerfile b/ci/Dockerfile
new file mode 100644
index 00000000..89645b36
--- /dev/null
+++ b/ci/Dockerfile
@@ -0,0 +1,18 @@
+# This container should be build in the ci directory, and then the
+# DTS directory should be mounted as a volume at /dts/
+FROM python:3.9-slim-buster
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+COPY requirements.txt .
+COPY dts_requirements.txt dts_requirements.txt
+
+RUN apt-get update && apt-get install --no-install-recommends -y \
+    # Add a C compiler for all of the c modules in DTS
+    build-essential make gcc git libpcap-dev\
+    python3-pip
+
+RUN pip3 install -r requirements.txt
+RUN pip3 install -r dts_requirements.txt
+# install formatter
+RUN pip3 install black
\ No newline at end of file
diff --git a/ci/README.txt b/ci/README.txt
new file mode 100644
index 00000000..281329f7
--- /dev/null
+++ b/ci/README.txt
@@ -0,0 +1,37 @@
+# BSD LICENSE
+#
+# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+This directory contains scripts used for ci in DTS. Nothing in this directory
+should be modified during the course of submitting a patch to DPDK.
+
+Additional python requirements only needed for running DTS in ci are present in the
+requirements.txt file in this directory.
+
diff --git a/ci/build_image.sh b/ci/build_image.sh
new file mode 100644
index 00000000..e69de29b
diff --git a/ci/get_tests_for_patchset.py b/ci/get_tests_for_patchset.py
new file mode 100644
index 00000000..40e37bce
--- /dev/null
+++ b/ci/get_tests_for_patchset.py
@@ -0,0 +1,197 @@
+# BSD LICENSE
+#
+# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+import argparse
+import os
+import pkgutil
+import re
+import subprocess
+import sys
+from pkgutil import walk_packages
+from types import ModuleType
+from typing import List, Iterable
+
+DTS_MAIN_BRANCH_REF: str = "origin/master"
+
+DTS_TEST_MODULE_PATH: str = "tests"
+# Will be unnecessary once DTS moves to a normal module structure
+DTS_MODULE_PATHS: List[str] = ["framework", "nic", "dep", DTS_TEST_MODULE_PATH]
+
+# This is primarily intended for folders which contain CI scripts, as these
+# should not be modified in the course of normal CI. A file in any of these
+# folders being modified will cause a warning to be emitted to standard error
+DTS_PROTECTED_PATHS: List[str] = [
+    "ci",
+]
+
+# This is intended to help detect when a config files have been changed.
+# The intention behind this detection is to enable additional regression
+# tests related to ensuring stable config file formats
+DTS_CONFIG_PATHS: List[str] = [
+    "conf",
+    "execution.cfg",
+]
+
+
+def get_args() -> str:
+    parser: argparse.ArgumentParser = argparse.ArgumentParser(
+        description="After a patchset is applied, run this script"
+                    "It will then output a list "
+                    "of applicable test suites to standard output, with 1 "
+                    "test suite per line. All other output, such as warnings,"
+                    " errors or informational messages will be sent to "
+                    "standard error. This script may produce no output at all,"
+                    "in which case it should be assumed that there are no "
+                    "applicable test suites.\n\n "
+                    "Exit Codes:\n"
+                    "value | meaning\n"
+                    "----- | -------\n"
+                    "  0   | OK   \n"
+                    "  1   | Error, check standard error",
+    )
+
+    dts_directory: str = os.path.dirname(os.path.dirname(os.path.join(os.getcwd(), __file__)))
+    parser.add_argument("-d", "--dts-directory", type=str, default=dts_directory, required=False)
+    args = parser.parse_args()
+    if not os.path.isdir(args.dts_directory):
+        print(f"{args.dts_directory} is not a directory.", file=sys.stderr)
+        exit(1)
+
+    return args.dts_directory
+
+
+def get_modified_files() -> List[str]:
+    cmd = ['git', 'diff', '--name-only', DTS_MAIN_BRANCH_REF, 'HEAD']
+    process: subprocess.CompletedProcess = subprocess.run(cmd, capture_output=True)
+    if process.returncode != 0:
+        print(f"{' '.join(cmd)} returned {process.returncode}")
+        exit(1)
+
+    # This explicit conversion to utf8 will catch encoding errors
+    stdout: bytes = process.stdout
+    stdout_str: str = stdout.decode("utf-8")
+
+    return stdout_str.splitlines()
+
+
+def get_names_of_modified_python_files(files: List[str]) -> List[str]:
+    return list(
+        map(
+            lambda f: str(re.sub("\\.py", "", f)),
+            map(
+                lambda f: os.path.basename(f),
+                filter(
+                    lambda f: f.endswith(".py"), files
+                )
+            )
+        )
+    )
+
+
+def get_modules() -> List[ModuleType]:
+    return list(map(lambda m: pkgutil.resolve_name(m.name), walk_packages(DTS_MODULE_PATHS)))
+
+
+def get_module_imports(mod: ModuleType) -> Iterable[ModuleType]:
+    imports = []
+    for attribute_label in dir(mod):
+        try:
+            attribute = getattr(mod, attribute_label)
+        except ModuleNotFoundError as _:  # some standard library modules don't like being directly imported
+            continue
+        if isinstance(attribute, type(mod)):
+            imports.append(attribute)
+    return imports
+
+
+def get_modules_in_tree(mod: ModuleType) -> Iterable[ModuleType]:
+    yield from get_module_imports(mod)
+
+
+def get_only_test_suites(modules: Iterable[ModuleType]) -> Iterable[ModuleType]:
+    test_package_path = os.path.join(os.getcwd(), DTS_TEST_MODULE_PATH)
+    mod: ModuleType
+    return filter(lambda mod: mod.__name__.startswith("TestSuite_"),
+                  filter(lambda mod: mod.__file__.startswith(test_package_path),
+                         filter(lambda mod: "__file__" in dir(mod), modules)))
+
+
+def get_test_suite_names(modules: Iterable[ModuleType]) -> Iterable[str]:
+    # Moving this into a set is there to ensure that there are no duplicates
+    return set(list(map(lambda mod: re.sub("TestSuite_", "", mod.__name__), modules)))
+
+
+def get_tests_to_run() -> List[str]:
+    dts_directory: str
+    dts_directory = get_args()
+
+    # This all needs to be done at the top level, so I have to do it here.
+
+    # chdir to the DTS directory, since we want that to be
+    # the context for any commands that are run.
+    os.chdir(dts_directory)
+
+    for path in DTS_MODULE_PATHS:
+        sys.path.append(os.path.join(dts_directory, path))
+
+    files: List[str] = get_modified_files()
+    changed_module_name = get_names_of_modified_python_files(files)
+
+    for protected_folder in DTS_PROTECTED_PATHS:
+        for file_name in files:
+            if file_name.startswith(protected_folder):
+                print(f"WARNING: {file_name} is protected", file=sys.stderr)
+
+    for config_file_path in DTS_CONFIG_PATHS:
+        for file_name in files:
+            if file_name.startswith(config_file_path):
+                print(f"WARNING: {file_name} is a config file and was changed", file=sys.stderr)
+
+    # Each index is 1 level of the tree
+    module_list: List[ModuleType] = [pkgutil.resolve_name(name) for name in changed_module_name]
+    current_index: int = 0
+    while current_index < len(module_list) and len(module_list) > 0:
+        mod = module_list[current_index]
+        if module_list.count(mod) == 1:
+            module_list = module_list + list(get_modules_in_tree(mod))
+        current_index += 1
+
+    test_suites_to_run: List[str] = list(get_test_suite_names(get_only_test_suites(module_list)))
+    test_suites_to_run.sort()
+    return test_suites_to_run
+
+
+def main():
+    test_suites_to_run = get_tests_to_run()
+    print("\n".join(test_suites_to_run))
+
+
+if __name__ == "__main__":
+    main()
diff --git a/ci/requirements.txt b/ci/requirements.txt
new file mode 100644
index 00000000..0abd6716
--- /dev/null
+++ b/ci/requirements.txt
@@ -0,0 +1,32 @@
+# BSD LICENSE
+#
+# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+argparse==1.4.0
\ No newline at end of file
-- 
2.30.2


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dts] [PATCH v2] ci/initial: Added script to get the tests for a patchset
  2021-08-12 15:36 [dts] [PATCH v2] ci/initial: Added script to get the tests for a patchset ohilyard
@ 2021-09-03  9:42 ` Juraj Linkeš
  2021-09-03 16:49   ` Owen Hilyard
  0 siblings, 1 reply; 4+ messages in thread
From: Juraj Linkeš @ 2021-09-03  9:42 UTC (permalink / raw)
  To: ohilyard, dts; +Cc: lijuan.tu



> -----Original Message-----
> From: dts <dts-bounces@dpdk.org> On Behalf Of ohilyard@iol.unh.edu
> Sent: Thursday, August 12, 2021 5:37 PM
> To: dts@dpdk.org
> Cc: lijuan.tu@intel.com; Owen Hilyard <ohilyard@iol.unh.edu>
> Subject: [dts] [PATCH v2] ci/initial: Added script to get the tests for a patchset

I'd strongly suggest using these guidelines (and also adding them to the DTS WG requirements) for writing commit messages: https://chris.beams.io/posts/git-commit/

I think DPDK also uses those (or very similar) guidelines. We'll be consistent and the just rules make sense.

In this commit message I think only the form in the subject should be changed to imperative.

> 
> From: Owen Hilyard <ohilyard@iol.unh.edu>
> 
> This script should be run after the patchset has been applied.
> It will check all files that have a diff to the git ref in DTS_MAIN_BRANCH_REF
> (currently origin/master).
> 
> It will also issue warnings to standard error if a "protected path" is changed. This
> is currently configured to only by the ci scripts folder, since under most
> circumstances a patch should not need to change anything in there. This warning
> will be in the format:
> "WARNING: {file_name} is protected"
> 
> The script will also issue a warning if a config file is changed. This warning is also
> sent to standard error and takes the form of:
> "WARNING: {file_name} is a config file and was changed"
> 
> The script will output a list of the test suites to run to standard out, with each
> entry having one line.
> 
> Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
> ---
>  ci/Dockerfile                |  18 ++++
>  ci/README.txt                |  37 +++++++
>  ci/build_image.sh            |   0
>  ci/get_tests_for_patchset.py | 197 +++++++++++++++++++++++++++++++++++
>  ci/requirements.txt          |  32 ++++++
>  5 files changed, 284 insertions(+)
>  create mode 100644 ci/Dockerfile
>  create mode 100644 ci/README.txt
>  create mode 100644 ci/build_image.sh
>  create mode 100644 ci/get_tests_for_patchset.py  create mode 100644
> ci/requirements.txt
> 
> diff --git a/ci/Dockerfile b/ci/Dockerfile new file mode 100644 index
> 00000000..89645b36
> --- /dev/null
> +++ b/ci/Dockerfile
> @@ -0,0 +1,18 @@
> +# This container should be build in the ci directory, and then the #
> +DTS directory should be mounted as a volume at /dts/ FROM
> +python:3.9-slim-buster
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +
> +COPY requirements.txt .
> +COPY dts_requirements.txt dts_requirements.txt
> +
> +RUN apt-get update && apt-get install --no-install-recommends -y \
> +    # Add a C compiler for all of the c modules in DTS
> +    build-essential make gcc git libpcap-dev\
> +    python3-pip
> +
> +RUN pip3 install -r requirements.txt
> +RUN pip3 install -r dts_requirements.txt
> +# install formatter
> +RUN pip3
> +install black
> \ No newline at end of file

Are these missing newlines intentional?

> diff --git a/ci/README.txt b/ci/README.txt new file mode 100644 index
> 00000000..281329f7
> --- /dev/null
> +++ b/ci/README.txt
> @@ -0,0 +1,37 @@
> +# BSD LICENSE
> +#
> +# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory.
> All rights reserved.
> +# All rights reserved.
> +#
> +# Redistribution and use in source and binary forms, with or without #
> +modification, are permitted provided that the following conditions #
> +are met:
> +#
> +#   * Redistributions of source code must retain the above copyright
> +#     notice, this list of conditions and the following disclaimer.
> +#   * Redistributions in binary form must reproduce the above copyright
> +#     notice, this list of conditions and the following disclaimer in
> +#     the documentation and/or other materials provided with the
> +#     distribution.
> +#   * Neither the name of Intel Corporation nor the names of its
> +#     contributors may be used to endorse or promote products derived
> +#     from this software without specific prior written permission.
> +#

Should Intel be still mentioned here?

In general, how should the licence look like? Was there a discussion about this?

<snip>

> +argparse==1.4.0
> \ No newline at end of file

The other newline and the reason for plural earlier. Pointing this out in case we'll change it.

> --
> 2.30.2
> 



^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dts] [PATCH v2] ci/initial: Added script to get the tests for a patchset
  2021-09-03  9:42 ` Juraj Linkeš
@ 2021-09-03 16:49   ` Owen Hilyard
  2021-10-13 14:19     ` ohilyard
  0 siblings, 1 reply; 4+ messages in thread
From: Owen Hilyard @ 2021-09-03 16:49 UTC (permalink / raw)
  To: Juraj Linkeš; +Cc: dts, lijuan.tu

[-- Attachment #1: Type: text/plain, Size: 169 bytes --]

I can fix the newlines, but I don't remember any discussion about how DTS
will handle licenses. That is probably a topic for either the working group
or the tech board.

[-- Attachment #2: Type: text/html, Size: 194 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [dts] [PATCH v2] ci/initial: Added script to get the tests for a patchset
  2021-09-03 16:49   ` Owen Hilyard
@ 2021-10-13 14:19     ` ohilyard
  0 siblings, 0 replies; 4+ messages in thread
From: ohilyard @ 2021-10-13 14:19 UTC (permalink / raw)
  To: dts; +Cc: juraj.linkes, lijuan.tu, Owen Hilyard

From: Owen Hilyard <ohilyard@iol.unh.edu>

Added a script to ci to allow getting the changed tests for a patchset
to DTS.

Instructions for use are in the module doc comment of the script.

Added a dockerfile that should be updated over time to allow it to run
all DTS CI scripts.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
 ci/Dockerfile                |  50 ++++++++
 ci/README.txt                |  38 ++++++
 ci/dts_requirements.txt      |  43 +++++++
 ci/get_tests_for_patchset.py | 218 +++++++++++++++++++++++++++++++++++
 ci/requirements.txt          |  33 ++++++
 5 files changed, 382 insertions(+)
 create mode 100644 ci/Dockerfile
 create mode 100644 ci/README.txt
 create mode 100644 ci/dts_requirements.txt
 create mode 100644 ci/get_tests_for_patchset.py
 create mode 100644 ci/requirements.txt

diff --git a/ci/Dockerfile b/ci/Dockerfile
new file mode 100644
index 00000000..8ee53786
--- /dev/null
+++ b/ci/Dockerfile
@@ -0,0 +1,50 @@
+# BSD LICENSE
+#
+# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved.
+# Copyright(c) 2021 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+# This container should be build in the ci directory, and then the
+# DTS directory should be mounted as a volume at /dts/
+FROM python:3.9-slim-buster
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+COPY requirements.txt .
+COPY dts_requirements.txt dts_requirements.txt
+
+RUN apt-get update && apt-get install --no-install-recommends -y \
+    # Add a C compiler for all of the c modules in DTS
+    build-essential make gcc git libpcap-dev\
+    python3-pip
+
+RUN pip3 install -r requirements.txt
+RUN pip3 install -r dts_requirements.txt
+# install formatter
+RUN pip3 install black
\ No newline at end of file
diff --git a/ci/README.txt b/ci/README.txt
new file mode 100644
index 00000000..a833e255
--- /dev/null
+++ b/ci/README.txt
@@ -0,0 +1,38 @@
+# BSD LICENSE
+#
+# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved.
+# Copyright(c) 2021 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+This directory contains scripts used for ci in DTS. Nothing in this directory
+should be modified during the course of submitting a patch to DPDK.
+
+Additional python requirements only needed for running DTS in ci are present in the
+requirements.txt file in this directory.
+
diff --git a/ci/dts_requirements.txt b/ci/dts_requirements.txt
new file mode 100644
index 00000000..d88ccd26
--- /dev/null
+++ b/ci/dts_requirements.txt
@@ -0,0 +1,43 @@
+# BSD LICENSE
+#
+# Copyright(c) 2019 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+xlwt==1.3.0
+pexpect==4.7.0
+numpy==1.18.5
+docutils
+pcapy
+xlrd
+scapy==2.4.4
+threadpool
+matplotlib
+gitpython
+pydes
\ No newline at end of file
diff --git a/ci/get_tests_for_patchset.py b/ci/get_tests_for_patchset.py
new file mode 100644
index 00000000..80f99045
--- /dev/null
+++ b/ci/get_tests_for_patchset.py
@@ -0,0 +1,218 @@
+# BSD LICENSE
+#
+# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved.
+# Copyright(c) 2021 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
+This script should be run after a patchset has been applied to DTS.
+It will iterate through the import tree in python, looking for any
+test that could possibly have had behavior change due to the patchset.
+It will then output all of those tests to standard out in a line-delimited
+list.
+
+The script will also issue warning to standard error if any of the
+"protected paths" are changed. The warning will be in the format:
+"WARNING: {file_name} is protected"
+
+The script will also issue a warning if a config file is changed. This
+warning is also sent to standard error and takes the form of:
+"WARNING: {file_name} is a config file and was changed"
+
+This script can either be run natively or with the provided dockerfile.
+"""
+
+
+import argparse
+import os
+import pkgutil
+import re
+import subprocess
+import sys
+from pkgutil import walk_packages
+from types import ModuleType
+from typing import List, Iterable
+
+DTS_MAIN_BRANCH_REF: str = "origin/master"
+
+DTS_TEST_MODULE_PATH: str = "tests"
+# Will be unnecessary once DTS moves to a normal module structure
+DTS_MODULE_PATHS: List[str] = ["framework", "nic", "dep", DTS_TEST_MODULE_PATH]
+
+# This is primarily intended for folders which contain CI scripts, as these
+# should not be modified in the course of normal CI. A file in any of these
+# folders being modified will cause a warning to be emitted to standard error
+DTS_PROTECTED_PATHS: List[str] = [
+    "ci",
+]
+
+# This is intended to help detect when a config files have been changed.
+# The intention behind this detection is to enable additional regression
+# tests related to ensuring stable config file formats
+DTS_CONFIG_PATHS: List[str] = [
+    "conf",
+    "execution.cfg",
+]
+
+
+def get_args() -> str:
+    parser: argparse.ArgumentParser = argparse.ArgumentParser(
+        description="After a patchset is applied, run this script"
+                    "It will then output a list "
+                    "of applicable test suites to standard output, with 1 "
+                    "test suite per line. All other output, such as warnings,"
+                    " errors or informational messages will be sent to "
+                    "standard error. This script may produce no output at all,"
+                    "in which case it should be assumed that there are no "
+                    "applicable test suites.\n\n "
+                    "Exit Codes:\n"
+                    "value | meaning\n"
+                    "----- | -------\n"
+                    "  0   | OK   \n"
+                    "  1   | Error, check standard error",
+    )
+
+    dts_directory: str = os.path.dirname(os.path.dirname(os.path.join(os.getcwd(), __file__)))
+    parser.add_argument("-d", "--dts-directory", type=str, default=dts_directory, required=False)
+    args = parser.parse_args()
+    if not os.path.isdir(args.dts_directory):
+        print(f"{args.dts_directory} is not a directory.", file=sys.stderr)
+        exit(1)
+
+    return args.dts_directory
+
+
+def get_modified_files() -> List[str]:
+    cmd = ['git', 'diff', '--name-only', DTS_MAIN_BRANCH_REF, 'HEAD']
+    process: subprocess.CompletedProcess = subprocess.run(cmd, capture_output=True)
+    if process.returncode != 0:
+        print(f"{' '.join(cmd)} returned {process.returncode}")
+        exit(1)
+
+    # This explicit conversion to utf8 will catch encoding errors
+    stdout: bytes = process.stdout
+    stdout_str: str = stdout.decode("utf-8")
+
+    return stdout_str.splitlines()
+
+
+def get_names_of_modified_python_files(files: List[str]) -> List[str]:
+    return list(
+        map(
+            lambda f: str(re.sub("\\.py", "", f)),
+            map(
+                lambda f: os.path.basename(f),
+                filter(
+                    lambda f: f.endswith(".py"), files
+                )
+            )
+        )
+    )
+
+
+def get_modules() -> List[ModuleType]:
+    return list(map(lambda m: pkgutil.resolve_name(m.name), walk_packages(DTS_MODULE_PATHS)))
+
+
+def get_module_imports(mod: ModuleType) -> Iterable[ModuleType]:
+    imports = []
+    for attribute_label in dir(mod):
+        try:
+            attribute = getattr(mod, attribute_label)
+        except ModuleNotFoundError as _:  # some standard library modules don't like being directly imported
+            continue
+        if isinstance(attribute, type(mod)):
+            imports.append(attribute)
+    return imports
+
+
+def get_modules_in_tree(mod: ModuleType) -> Iterable[ModuleType]:
+    yield from get_module_imports(mod)
+
+
+def get_only_test_suites(modules: Iterable[ModuleType]) -> Iterable[ModuleType]:
+    test_package_path = os.path.join(os.getcwd(), DTS_TEST_MODULE_PATH)
+    mod: ModuleType
+    return filter(lambda mod: mod.__name__.startswith("TestSuite_"),
+                  filter(lambda mod: mod.__file__.startswith(test_package_path),
+                         filter(lambda mod: "__file__" in dir(mod), modules)))
+
+
+def get_test_suite_names(modules: Iterable[ModuleType]) -> Iterable[str]:
+    # Moving this into a set is there to ensure that there are no duplicates
+    return set(list(map(lambda mod: re.sub("TestSuite_", "", mod.__name__), modules)))
+
+
+def get_tests_to_run() -> List[str]:
+    dts_directory: str
+    dts_directory = get_args()
+
+    # This all needs to be done at the top level, so I have to do it here.
+
+    # chdir to the DTS directory, since we want that to be
+    # the context for any commands that are run.
+    os.chdir(dts_directory)
+
+    for path in DTS_MODULE_PATHS:
+        sys.path.append(os.path.join(dts_directory, path))
+
+    files: List[str] = get_modified_files()
+    changed_module_name = get_names_of_modified_python_files(files)
+
+    for protected_folder in DTS_PROTECTED_PATHS:
+        for file_name in files:
+            if file_name.startswith(protected_folder):
+                print(f"WARNING: {file_name} is protected", file=sys.stderr)
+
+    for config_file_path in DTS_CONFIG_PATHS:
+        for file_name in files:
+            if file_name.startswith(config_file_path):
+                print(f"WARNING: {file_name} is a config file and was changed", file=sys.stderr)
+
+    # Each index is 1 level of the tree
+    module_list: List[ModuleType] = [pkgutil.resolve_name(name) for name in changed_module_name]
+    current_index: int = 0
+    while current_index < len(module_list) and len(module_list) > 0:
+        mod = module_list[current_index]
+        if module_list.count(mod) == 1:
+            module_list = module_list + list(get_modules_in_tree(mod))
+        current_index += 1
+
+    test_suites_to_run: List[str] = list(get_test_suite_names(get_only_test_suites(module_list)))
+    test_suites_to_run.sort()
+    return test_suites_to_run
+
+
+def main():
+    test_suites_to_run = get_tests_to_run()
+    print("\n".join(test_suites_to_run))
+
+
+if __name__ == "__main__":
+    main()
diff --git a/ci/requirements.txt b/ci/requirements.txt
new file mode 100644
index 00000000..74d51fc3
--- /dev/null
+++ b/ci/requirements.txt
@@ -0,0 +1,33 @@
+# BSD LICENSE
+#
+# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved.
+# Copyright(c) 2021 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+argparse==1.4.0
\ No newline at end of file
-- 
2.30.2


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-10-13 14:19 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-12 15:36 [dts] [PATCH v2] ci/initial: Added script to get the tests for a patchset ohilyard
2021-09-03  9:42 ` Juraj Linkeš
2021-09-03 16:49   ` Owen Hilyard
2021-10-13 14:19     ` ohilyard

test suite reviews and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/dts/0 dts/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dts dts/ http://inbox.dpdk.org/dts \
		dts@dpdk.org
	public-inbox-index dts

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dts


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git