[dpdk-dev] [PATCH 0/7] Make unit tests great again

[dpdk-dev] [PATCH 0/7] Make unit tests great again

[Anatoly Burakov]

Previously, unit tests were running in groups. There were
technical reasons why that was the case (mostly having to do
with limiting memory), but it was hard to maintain and update
the autotest script.

In 18.05, limiting of memory at DPDK startup was no longer
necessary, as DPDK allocates memory at runtime as needed. This
has the implication that the old test grouping can now be
retired and replaced with a more sensible way of running unit
tests (using multiprocessing pool of workers and a queue of
tests). This patchset accomplishes exactly that.

This patchset conflicts with some of the earlier work on
autotests [1] [2] [3], but i think it presents a cleaner
solution for some of the problems highlighted by those patch
series. I can integrate those patches into this series if
need be.

[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://dpdk.org/dev/patchwork/patch/40371/
[3] http://dpdk.org/dev/patchwork/patch/40372/

Anatoly Burakov (7):
  autotest: fix printing
  autotest: fix invalid code on reports
  autotest: make autotest runner python 2/3 compliant
  autotest: visually separate different test categories
  autotest: improve filtering
  autotest: remove autotest grouping
  autotest: properly parallelize unit tests

 test/test/autotest.py        |  13 +-
 test/test/autotest_data.py   | 749 ++++++++++++++---------------------
 test/test/autotest_runner.py | 519 ++++++++++++------------
 3 files changed, 586 insertions(+), 695 deletions(-)

-- 
2.17.1

[dpdk-dev] [PATCH 1/7] autotest: fix printing

[Anatoly Burakov]

Previously, printing was done using tuple syntax, which caused
output to appear as a tuple as opposed to being one string. Fix
this by using addition operator instead.

Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index a692f0697..b09b57876 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -247,7 +247,7 @@ def __process_results(self, results):
 
             # don't print out total time every line, it's the same anyway
             if i == len(results) - 1:
-                print(result,
+                print(result +
                       "[%02dm %02ds]" % (total_time / 60, total_time % 60))
             else:
                 print(result)
@@ -332,8 +332,8 @@ def run_all_tests(self):
 
             # create table header
             print("")
-            print("Test name".ljust(30), "Test result".ljust(29),
-                  "Test".center(9), "Total".center(9))
+            print("Test name".ljust(30) + "Test result".ljust(29) +
+                  "Test".center(9) + "Total".center(9))
             print("=" * 80)
 
             # make a note of tests start time
-- 
2.17.1

[dpdk-dev] [PATCH 2/7] autotest: fix invalid code on reports

[Anatoly Burakov]

There are no reports defined for any test, so this codepath was
never triggered, but it's still wrong because it's referencing
variables that aren't there. Fix it by passing target into the
test function, and reference correct log variable.

Fixes: e2cc79b75d9f ("app: rework autotest.py")
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index b09b57876..bdc32da5d 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -41,7 +41,7 @@ def wait_prompt(child):
 # quite a bit of effort to make it work).
 
 
-def run_test_group(cmdline, test_group):
+def run_test_group(cmdline, target, test_group):
     results = []
     child = None
     start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group):
             # make a note when the test was finished
             end_time = time.time()
 
+            log = logfile.getvalue()
+
             # append test data to the result tuple
-            result += (test["Name"], end_time - start_time,
-                       logfile.getvalue())
+            result += (test["Name"], end_time - start_time, log)
 
             # call report function, if any defined, and supply it with
             # target and complete log for test run
             if test["Report"]:
-                report = test["Report"](self.target, log)
+                report = test["Report"](target, log)
 
                 # append report to results tuple
                 result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self):
             for test_group in self.parallel_test_groups:
                 result = pool.apply_async(run_test_group,
                                           [self.__get_cmdline(test_group),
+                                           self.target,
                                            test_group])
                 results.append(result)
 
@@ -367,7 +369,7 @@ def run_all_tests(self):
             # run non_parallel tests. they are run one by one, synchronously
             for test_group in self.non_parallel_test_groups:
                 group_result = run_test_group(
-                    self.__get_cmdline(test_group), test_group)
+                    self.__get_cmdline(test_group), self.target, test_group)
 
                 self.__process_results(group_result)
 
-- 
2.17.1

[dpdk-dev] [PATCH 3/7] autotest: make autotest runner python 2/3 compliant

[Anatoly Burakov]

Autotest runner was still using python 2-style print syntax. Fix
it by importing print function from the future, and fix the calls
to be python-3 style.

Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index bdc32da5d..f6b669a2e 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -3,6 +3,7 @@
 
 # The main logic behind running autotests in parallel
 
+from __future__ import print_function
 import StringIO
 import csv
 import multiprocessing
@@ -52,8 +53,8 @@ def run_test_group(cmdline, target, test_group):
         # prepare logging of init
         startuplog = StringIO.StringIO()
 
-        print >>startuplog, "\n%s %s\n" % ("=" * 20, test_group["Prefix"])
-        print >>startuplog, "\ncmdline=%s" % cmdline
+        print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog)
+        print("\ncmdline=%s" % cmdline, file=startuplog)
 
         child = pexpect.spawn(cmdline, logfile=startuplog)
 
@@ -117,7 +118,7 @@ def run_test_group(cmdline, target, test_group):
 
         try:
             # print test name to log buffer
-            print >>logfile, "\n%s %s\n" % ("-" * 20, test["Name"])
+            print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
             # run test function associated with the test
             if stripped or test["Command"] in avail_cmds:
-- 
2.17.1

[dpdk-dev] [PATCH 4/7] autotest: visually separate different test categories

[Anatoly Burakov]

Help visually identify parallel vs. non-parallel autotests.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index f6b669a2e..d9d5f7a97 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -341,6 +341,7 @@ def run_all_tests(self):
             # make a note of tests start time
             self.start = time.time()
 
+            print("Parallel autotests:")
             # assign worker threads to run test groups
             for test_group in self.parallel_test_groups:
                 result = pool.apply_async(run_test_group,
@@ -367,6 +368,7 @@ def run_all_tests(self):
                     # remove result from results list once we're done with it
                     results.remove(group_result)
 
+            print("Non-parallel autotests:")
             # run non_parallel tests. they are run one by one, synchronously
             for test_group in self.non_parallel_test_groups:
                 group_result = run_test_group(
-- 
2.17.1

[dpdk-dev] [PATCH 5/7] autotest: improve filtering

[Anatoly Burakov]

Improve code for filtering test groups. Also, move reading binary
symbols into filtering stage, so that tests that are meant to be
skipped are never attempted to be executed in the first place.
Before running tests, print out any tests that were skipped because
they weren't compiled.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 118 ++++++++++++++++++++---------------
 1 file changed, 66 insertions(+), 52 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index d9d5f7a97..c98ec2a57 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -95,13 +95,6 @@ def run_test_group(cmdline, target, test_group):
     results.append((0, "Success", "Start %s" % test_group["Prefix"],
                     time.time() - start_time, startuplog.getvalue(), None))
 
-    # parse the binary for available test commands
-    binary = cmdline.split()[0]
-    stripped = 'not stripped' not in subprocess.check_output(['file', binary])
-    if not stripped:
-        symbols = subprocess.check_output(['nm', binary]).decode('utf-8')
-        avail_cmds = re.findall('test_register_(\w+)', symbols)
-
     # run all tests in test group
     for test in test_group["Tests"]:
 
@@ -121,10 +114,7 @@ def run_test_group(cmdline, target, test_group):
             print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
             # run test function associated with the test
-            if stripped or test["Command"] in avail_cmds:
-                result = test["Func"](child, test["Command"])
-            else:
-                result = (0, "Skipped [Not Available]")
+            result = test["Func"](child, test["Command"])
 
             # make a note when the test was finished
             end_time = time.time()
@@ -186,8 +176,10 @@ class AutotestRunner:
     def __init__(self, cmdline, target, blacklist, whitelist):
         self.cmdline = cmdline
         self.target = target
+        self.binary = cmdline.split()[0]
         self.blacklist = blacklist
         self.whitelist = whitelist
+        self.skipped = []
 
         # log file filename
         logfile = "%s.log" % target
@@ -276,53 +268,58 @@ def __process_results(self, results):
             if i != 0:
                 self.csvwriter.writerow([test_name, test_result, result_str])
 
-    # this function iterates over test groups and removes each
-    # test that is not in whitelist/blacklist
-    def __filter_groups(self, test_groups):
-        groups_to_remove = []
-
-        # filter out tests from parallel test groups
-        for i, test_group in enumerate(test_groups):
-
-            # iterate over a copy so that we could safely delete individual
-            # tests
-            for test in test_group["Tests"][:]:
-                test_id = test["Command"]
-
-                # dump tests are specified in full e.g. "Dump_mempool"
-                if "_autotest" in test_id:
-                    test_id = test_id[:-len("_autotest")]
-
-                # filter out blacklisted/whitelisted tests
-                if self.blacklist and test_id in self.blacklist:
-                    test_group["Tests"].remove(test)
-                    continue
-                if self.whitelist and test_id not in self.whitelist:
-                    test_group["Tests"].remove(test)
-                    continue
-
-            # modify or remove original group
-            if len(test_group["Tests"]) > 0:
-                test_groups[i] = test_group
-            else:
-                # remember which groups should be deleted
-                # put the numbers backwards so that we start
-                # deleting from the end, not from the beginning
-                groups_to_remove.insert(0, i)
+    # this function checks individual test and decides if this test should be in
+    # the group by comparing it against  whitelist/blacklist. it also checks if
+    # the test is compiled into the binary, and marks it as skipped if necessary
+    def __filter_test(self, test):
+        test_cmd = test["Command"]
+        test_id = test_cmd
+
+        # dump tests are specified in full e.g. "Dump_mempool"
+        if "_autotest" in test_id:
+            test_id = test_id[:-len("_autotest")]
+
+        # filter out blacklisted/whitelisted tests
+        if self.blacklist and test_id in self.blacklist:
+            return False
+        if self.whitelist and test_id not in self.whitelist:
+            return False
+
+        # if test wasn't compiled in, remove it as well
+
+        # parse the binary for available test commands
+        stripped = 'not stripped' not in \
+                   subprocess.check_output(['file', self.binary])
+        if not stripped:
+            symbols = subprocess.check_output(['nm',
+                                               self.binary]).decode('utf-8')
+            avail_cmds = re.findall('test_register_(\w+)', symbols)
+
+            if test_cmd not in avail_cmds:
+                # notify user
+                result = 0, "Skipped [Not compiled]", test_id, 0, "", None
+                self.skipped.append(tuple(result))
+                return False
 
-        # remove test groups that need to be removed
-        for i in groups_to_remove:
-            del test_groups[i]
+        return True
 
-        return test_groups
+    def __filter_group(self, group):
+        group["Tests"] = list(filter(self.__filter_test, group["Tests"]))
+        return len(group["Tests"]) > 0
 
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
-        self.parallel_test_groups = \
-            self.__filter_groups(self.parallel_test_groups)
-        self.non_parallel_test_groups = \
-            self.__filter_groups(self.non_parallel_test_groups)
+        # for each test group, check all tests against the filter, then remove
+        # all groups that don't have any tests
+        self.parallel_test_groups = list(
+            filter(self.__filter_group,
+                   self.parallel_test_groups)
+        )
+        self.non_parallel_test_groups = list(
+            filter(self.__filter_group,
+                   self.non_parallel_test_groups)
+        )
 
         # create a pool of worker threads
         pool = multiprocessing.Pool(processes=1)
@@ -338,6 +335,23 @@ def run_all_tests(self):
                   "Test".center(9) + "Total".center(9))
             print("=" * 80)
 
+            # print out skipped autotests if there were any
+            if len(self.skipped):
+                print("Skipped autotests:")
+
+                # print out any skipped tests
+                for result in self.skipped:
+                    # unpack result tuple
+                    test_result, result_str, test_name, _, _, _ = result
+                    self.csvwriter.writerow([test_name, test_result,
+                                             result_str])
+
+                    t = ("%s:" % test_name).ljust(30)
+                    t += result_str.ljust(29)
+                    t += "[00m 00s]"
+
+                    print(t)
+
             # make a note of tests start time
             self.start = time.time()
 
-- 
2.17.1

[dpdk-dev] [PATCH 6/7] autotest: remove autotest grouping

[Anatoly Burakov]

Previously, all autotests were grouped into (seemingly arbitrary)
groups. The goal was to run all tests in parallel (so that autotest
finishes faster), but we couldn't just do it willy-nilly because
DPDK couldn't allocate and free hugepages on-demand, so we had to
find autotest groupings that could work memory-wise and still be
fast enough to not hold up shorter tests. The inflexibility of
memory subsystem has now been fixed for 18.05, so grouping
autotests is no longer necessary.

Thus, this commit moves all autotests into two groups -
parallel(izable) autotests, and non-arallel(izable) autotests
(typically performance tests). Note that this particular commit
makes running autotests dog slow because while the tests are now
in a single group, the test function itself hasn't changed much,
so all autotests are now run one-by-one, starting and stopping
the DPDK test application.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest.py        |   7 +-
 test/test/autotest_data.py   | 749 ++++++++++++++---------------------
 test/test/autotest_runner.py | 271 +++++--------
 3 files changed, 408 insertions(+), 619 deletions(-)

diff --git a/test/test/autotest.py b/test/test/autotest.py
index 1cfd8cf22..ae27daef7 100644
--- a/test/test/autotest.py
+++ b/test/test/autotest.py
@@ -39,11 +39,8 @@ def usage():
 runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist,
                                         test_whitelist)
 
-for test_group in autotest_data.parallel_test_group_list:
-    runner.add_parallel_test_group(test_group)
-
-for test_group in autotest_data.non_parallel_test_group_list:
-    runner.add_non_parallel_test_group(test_group)
+runner.parallel_tests = autotest_data.parallel_test_list[:]
+runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
 
 num_fails = runner.run_all_tests()
 
diff --git a/test/test/autotest_data.py b/test/test/autotest_data.py
index aacfe0a66..c24e7bc25 100644
--- a/test/test/autotest_data.py
+++ b/test/test/autotest_data.py
@@ -3,465 +3,322 @@
 
 # Test data for autotests
 
-from glob import glob
 from autotest_test_funcs import *
 
-
-# quick and dirty function to find out number of sockets
-def num_sockets():
-    result = len(glob("/sys/devices/system/node/node*"))
-    if result == 0:
-        return 1
-    return result
-
-
-# Assign given number to each socket
-# e.g. 32 becomes 32,32 or 32,32,32,32
-def per_sockets(num):
-    return ",".join([str(num)] * num_sockets())
-
 # groups of tests that can be run in parallel
 # the grouping has been found largely empirically
-parallel_test_group_list = [
-    {
-        "Prefix":    "group_1",
-        "Memory":    per_sockets(8),
-        "Tests":
-        [
-            {
-                "Name":    "Cycles autotest",
-                "Command": "cycles_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Timer autotest",
-                "Command": "timer_autotest",
-                "Func":    timer_autotest,
-                "Report":   None,
-            },
-            {
-                "Name":    "Debug autotest",
-                "Command": "debug_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Errno autotest",
-                "Command": "errno_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Meter autotest",
-                "Command": "meter_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Common autotest",
-                "Command": "common_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Resource autotest",
-                "Command": "resource_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_2",
-        "Memory":    "16",
-        "Tests":
-        [
-            {
-                "Name":    "Memory autotest",
-                "Command": "memory_autotest",
-                "Func":    memory_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Read/write lock autotest",
-                "Command": "rwlock_autotest",
-                "Func":    rwlock_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Logs autotest",
-                "Command": "logs_autotest",
-                "Func":    logs_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "CPU flags autotest",
-                "Command": "cpuflags_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Version autotest",
-                "Command": "version_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "EAL filesystem autotest",
-                "Command": "eal_fs_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "EAL flags autotest",
-                "Command": "eal_flags_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Hash autotest",
-                "Command": "hash_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ],
-    },
-    {
-        "Prefix":    "group_3",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "LPM autotest",
-                "Command": "lpm_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "LPM6 autotest",
-                "Command": "lpm6_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Memcpy autotest",
-                "Command": "memcpy_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Memzone autotest",
-                "Command": "memzone_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "String autotest",
-                "Command": "string_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Alarm autotest",
-                "Command": "alarm_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_4",
-        "Memory":    per_sockets(128),
-        "Tests":
-        [
-            {
-                "Name":    "PCI autotest",
-                "Command": "pci_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Malloc autotest",
-                "Command": "malloc_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Multi-process autotest",
-                "Command": "multiprocess_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Mbuf autotest",
-                "Command": "mbuf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Per-lcore autotest",
-                "Command": "per_lcore_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Ring autotest",
-                "Command": "ring_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_5",
-        "Memory":    "32",
-        "Tests":
-        [
-            {
-                "Name":    "Spinlock autotest",
-                "Command": "spinlock_autotest",
-                "Func":    spinlock_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Byte order autotest",
-                "Command": "byteorder_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "TAILQ autotest",
-                "Command": "tailq_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Command-line autotest",
-                "Command": "cmdline_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Interrupts autotest",
-                "Command": "interrupt_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_6",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Function reentrancy autotest",
-                "Command": "func_reentrancy_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Mempool autotest",
-                "Command": "mempool_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Atomics autotest",
-                "Command": "atomic_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Prefetch autotest",
-                "Command": "prefetch_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Red autotest",
-                "Command": "red_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_7",
-        "Memory":    "64",
-        "Tests":
-        [
-            {
-                "Name":    "PMD ring autotest",
-                "Command": "ring_pmd_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Access list control autotest",
-                "Command": "acl_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Sched autotest",
-                "Command": "sched_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+parallel_test_list = [
+    {
+        "Name":    "Cycles autotest",
+        "Command": "cycles_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Timer autotest",
+        "Command": "timer_autotest",
+        "Func":    timer_autotest,
+        "Report":   None,
+    },
+    {
+        "Name":    "Debug autotest",
+        "Command": "debug_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Errno autotest",
+        "Command": "errno_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Meter autotest",
+        "Command": "meter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Common autotest",
+        "Command": "common_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Resource autotest",
+        "Command": "resource_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memory autotest",
+        "Command": "memory_autotest",
+        "Func":    memory_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Read/write lock autotest",
+        "Command": "rwlock_autotest",
+        "Func":    rwlock_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Logs autotest",
+        "Command": "logs_autotest",
+        "Func":    logs_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "CPU flags autotest",
+        "Command": "cpuflags_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Version autotest",
+        "Command": "version_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "EAL filesystem autotest",
+        "Command": "eal_fs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "EAL flags autotest",
+        "Command": "eal_flags_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash autotest",
+        "Command": "hash_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "LPM autotest",
+        "Command": "lpm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "LPM6 autotest",
+        "Command": "lpm6_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memcpy autotest",
+        "Command": "memcpy_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memzone autotest",
+        "Command": "memzone_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "String autotest",
+        "Command": "string_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Alarm autotest",
+        "Command": "alarm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "PCI autotest",
+        "Command": "pci_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Malloc autotest",
+        "Command": "malloc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Multi-process autotest",
+        "Command": "multiprocess_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mbuf autotest",
+        "Command": "mbuf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Per-lcore autotest",
+        "Command": "per_lcore_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Ring autotest",
+        "Command": "ring_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Spinlock autotest",
+        "Command": "spinlock_autotest",
+        "Func":    spinlock_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Byte order autotest",
+        "Command": "byteorder_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "TAILQ autotest",
+        "Command": "tailq_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Command-line autotest",
+        "Command": "cmdline_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Interrupts autotest",
+        "Command": "interrupt_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Function reentrancy autotest",
+        "Command": "func_reentrancy_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mempool autotest",
+        "Command": "mempool_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Atomics autotest",
+        "Command": "atomic_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Prefetch autotest",
+        "Command": "prefetch_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red autotest",
+        "Command": "red_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "PMD ring autotest",
+        "Command": "ring_pmd_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Access list control autotest",
+        "Command": "acl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Sched autotest",
+        "Command": "sched_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
 ]
 
 # tests that should not be run when any other tests are running
-non_parallel_test_group_list = [
-
+non_parallel_test_list = [
     {
-        "Prefix":    "eventdev",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "Eventdev common autotest",
-                "Command": "eventdev_common_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "eventdev_sw",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "Eventdev sw autotest",
-                "Command": "eventdev_sw_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "kni",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "KNI autotest",
-                "Command": "kni_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "mempool_perf",
-        "Memory":    per_sockets(256),
-        "Tests":
-        [
-            {
-                "Name":    "Mempool performance autotest",
-                "Command": "mempool_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "memcpy_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Memcpy performance autotest",
-                "Command": "memcpy_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "hash_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Hash performance autotest",
-                "Command": "hash_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power autotest",
-                "Command":    "power_autotest",
-                "Func":       default_autotest,
-                "Report":      None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power_acpi_cpufreq",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power ACPI cpufreq autotest",
-                "Command":    "power_acpi_cpufreq_autotest",
-                "Func":       default_autotest,
-                "Report":     None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power_kvm_vm",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power KVM VM  autotest",
-                "Command":    "power_kvm_vm_autotest",
-                "Func":       default_autotest,
-                "Report":     None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "timer_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Timer performance autotest",
-                "Command": "timer_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+        "Name":    "Eventdev common autotest",
+        "Command": "eventdev_common_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Eventdev sw autotest",
+        "Command": "eventdev_sw_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "KNI autotest",
+        "Command": "kni_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mempool performance autotest",
+        "Command": "mempool_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memcpy performance autotest",
+        "Command": "memcpy_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash performance autotest",
+        "Command": "hash_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":       "Power autotest",
+        "Command":    "power_autotest",
+        "Func":       default_autotest,
+        "Report":      None,
+    },
+    {
+        "Name":       "Power ACPI cpufreq autotest",
+        "Command":    "power_acpi_cpufreq_autotest",
+        "Func":       default_autotest,
+        "Report":     None,
+    },
+    {
+        "Name":       "Power KVM VM  autotest",
+        "Command":    "power_kvm_vm_autotest",
+        "Func":       default_autotest,
+        "Report":     None,
+    },
+    {
+        "Name":    "Timer performance autotest",
+        "Command": "timer_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
-
     #
     # Please always make sure that ring_perf is the last test!
     #
     {
-        "Prefix":    "ring_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Ring performance autotest",
-                "Command": "ring_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+        "Name":    "Ring performance autotest",
+        "Command": "ring_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
 ]
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index c98ec2a57..d6ae57e76 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -42,18 +42,16 @@ def wait_prompt(child):
 # quite a bit of effort to make it work).
 
 
-def run_test_group(cmdline, target, test_group):
-    results = []
-    child = None
+def run_test_group(cmdline, prefix, target, test):
     start_time = time.time()
-    startuplog = None
+
+    # prepare logging of init
+    startuplog = StringIO.StringIO()
 
     # run test app
     try:
-        # prepare logging of init
-        startuplog = StringIO.StringIO()
 
-        print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog)
+        print("\n%s %s\n" % ("=" * 20, prefix), file=startuplog)
         print("\ncmdline=%s" % cmdline, file=startuplog)
 
         child = pexpect.spawn(cmdline, logfile=startuplog)
@@ -62,88 +60,54 @@ def run_test_group(cmdline, target, test_group):
         if not wait_prompt(child):
             child.close()
 
-            results.append((-1,
-                            "Fail [No prompt]",
-                            "Start %s" % test_group["Prefix"],
-                            time.time() - start_time,
-                            startuplog.getvalue(),
-                            None))
-
-            # mark all tests as failed
-            for test in test_group["Tests"]:
-                results.append((-1, "Fail [No prompt]", test["Name"],
-                                time.time() - start_time, "", None))
-            # exit test
-            return results
+            return -1, "Fail [No prompt]", "Start %s" % prefix,\
+                   time.time() - start_time, startuplog.getvalue(), None
 
     except:
-        results.append((-1,
-                        "Fail [Can't run]",
-                        "Start %s" % test_group["Prefix"],
-                        time.time() - start_time,
-                        startuplog.getvalue(),
-                        None))
-
-        # mark all tests as failed
-        for t in test_group["Tests"]:
-            results.append((-1, "Fail [Can't run]", t["Name"],
-                            time.time() - start_time, "", None))
-        # exit test
-        return results
-
-    # startup was successful
-    results.append((0, "Success", "Start %s" % test_group["Prefix"],
-                    time.time() - start_time, startuplog.getvalue(), None))
-
-    # run all tests in test group
-    for test in test_group["Tests"]:
-
-        # create log buffer for each test
-        # in multiprocessing environment, the logging would be
-        # interleaved and will create a mess, hence the buffering
-        logfile = StringIO.StringIO()
-        child.logfile = logfile
-
-        result = ()
-
-        # make a note when the test started
-        start_time = time.time()
+        return -1, "Fail [Can't run]", "Start %s" % prefix,\
+               time.time() - start_time, startuplog.getvalue(), None
 
-        try:
-            # print test name to log buffer
-            print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
+    # create log buffer for each test
+    # in multiprocessing environment, the logging would be
+    # interleaved and will create a mess, hence the buffering
+    logfile = StringIO.StringIO()
+    child.logfile = logfile
 
-            # run test function associated with the test
-            result = test["Func"](child, test["Command"])
+    # make a note when the test started
+    start_time = time.time()
 
-            # make a note when the test was finished
-            end_time = time.time()
+    try:
+        # print test name to log buffer
+        print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
-            log = logfile.getvalue()
+        # run test function associated with the test
+        result = test["Func"](child, test["Command"])
 
-            # append test data to the result tuple
-            result += (test["Name"], end_time - start_time, log)
+        # make a note when the test was finished
+        end_time = time.time()
 
-            # call report function, if any defined, and supply it with
-            # target and complete log for test run
-            if test["Report"]:
-                report = test["Report"](target, log)
+        log = logfile.getvalue()
 
-                # append report to results tuple
-                result += (report,)
-            else:
-                # report is None
-                result += (None,)
-        except:
-            # make a note when the test crashed
-            end_time = time.time()
+        # append test data to the result tuple
+        result += (test["Name"], end_time - start_time, log)
+
+        # call report function, if any defined, and supply it with
+        # target and complete log for test run
+        if test["Report"]:
+            report = test["Report"](target, log)
+
+            # append report to results tuple
+            result += (report,)
+        else:
+            # report is None
+            result += (None,)
+    except:
+        # make a note when the test crashed
+        end_time = time.time()
 
-            # mark test as failed
-            result = (-1, "Fail [Crash]", test["Name"],
-                      end_time - start_time, logfile.getvalue(), None)
-        finally:
-            # append the results to the results list
-            results.append(result)
+        # mark test as failed
+        result = (-1, "Fail [Crash]", test["Name"],
+                  end_time - start_time, logfile.getvalue(), None)
 
     # regardless of whether test has crashed, try quitting it
     try:
@@ -155,7 +119,7 @@ def run_test_group(cmdline, target, test_group):
         pass
 
     # return test results
-    return results
+    return result
 
 
 # class representing an instance of autotests run
@@ -180,6 +144,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.blacklist = blacklist
         self.whitelist = whitelist
         self.skipped = []
+        self.parallel_tests = []
+        self.non_parallel_tests = []
 
         # log file filename
         logfile = "%s.log" % target
@@ -193,80 +159,52 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.csvwriter.writerow(["test_name", "test_result", "result_str"])
 
     # set up cmdline string
-    def __get_cmdline(self, test):
+    def __get_cmdline(self):
         cmdline = self.cmdline
 
-        # append memory limitations for each test
-        # otherwise tests won't run in parallel
-        if "i686" not in self.target:
-            cmdline += " --socket-mem=%s" % test["Memory"]
-        else:
-            # affinitize startup so that tests don't fail on i686
-            cmdline = "taskset 1 " + cmdline
-            cmdline += " -m " + str(sum(map(int, test["Memory"].split(","))))
-
-        # set group prefix for autotest group
-        # otherwise they won't run in parallel
-        cmdline += " --file-prefix=%s" % test["Prefix"]
+        # affinitize startup so that tests don't fail on i686
+        cmdline = "taskset 1 " + cmdline
 
         return cmdline
 
-    def add_parallel_test_group(self, test_group):
-        self.parallel_test_groups.append(test_group)
+    def __process_result(self, result):
 
-    def add_non_parallel_test_group(self, test_group):
-        self.non_parallel_test_groups.append(test_group)
+        # unpack result tuple
+        test_result, result_str, test_name, \
+            test_time, log, report = result
 
-    def __process_results(self, results):
-        # this iterates over individual test results
-        for i, result in enumerate(results):
+        # get total run time
+        cur_time = time.time()
+        total_time = int(cur_time - self.start)
 
-            # increase total number of tests that were run
-            # do not include "start" test
-            if i > 0:
-                self.n_tests += 1
+        # print results, test run time and total time since start
+        result = ("%s:" % test_name).ljust(30)
+        result += result_str.ljust(29)
+        result += "[%02dm %02ds]" % (test_time / 60, test_time % 60)
 
-            # unpack result tuple
-            test_result, result_str, test_name, \
-                test_time, log, report = result
+        # don't print out total time every line, it's the same anyway
+        print(result + "[%02dm %02ds]" % (total_time / 60, total_time % 60))
 
-            # get total run time
-            cur_time = time.time()
-            total_time = int(cur_time - self.start)
+        # if test failed and it wasn't a "start" test
+        if test_result < 0:
+            self.fails += 1
 
-            # print results, test run time and total time since start
-            result = ("%s:" % test_name).ljust(30)
-            result += result_str.ljust(29)
-            result += "[%02dm %02ds]" % (test_time / 60, test_time % 60)
+        # collect logs
+        self.log_buffers.append(log)
 
-            # don't print out total time every line, it's the same anyway
-            if i == len(results) - 1:
-                print(result +
-                      "[%02dm %02ds]" % (total_time / 60, total_time % 60))
+        # create report if it exists
+        if report:
+            try:
+                f = open("%s_%s_report.rst" %
+                         (self.target, test_name), "w")
+            except IOError:
+                print("Report for %s could not be created!" % test_name)
             else:
-                print(result)
-
-            # if test failed and it wasn't a "start" test
-            if test_result < 0 and not i == 0:
-                self.fails += 1
-
-            # collect logs
-            self.log_buffers.append(log)
-
-            # create report if it exists
-            if report:
-                try:
-                    f = open("%s_%s_report.rst" %
-                             (self.target, test_name), "w")
-                except IOError:
-                    print("Report for %s could not be created!" % test_name)
-                else:
-                    with f:
-                        f.write(report)
-
-            # write test result to CSV file
-            if i != 0:
-                self.csvwriter.writerow([test_name, test_result, result_str])
+                with f:
+                    f.write(report)
+
+        # write test result to CSV file
+        self.csvwriter.writerow([test_name, test_result, result_str])
 
     # this function checks individual test and decides if this test should be in
     # the group by comparing it against  whitelist/blacklist. it also checks if
@@ -303,22 +241,16 @@ def __filter_test(self, test):
 
         return True
 
-    def __filter_group(self, group):
-        group["Tests"] = list(filter(self.__filter_test, group["Tests"]))
-        return len(group["Tests"]) > 0
-
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
-        # for each test group, check all tests against the filter, then remove
-        # all groups that don't have any tests
-        self.parallel_test_groups = list(
-            filter(self.__filter_group,
-                   self.parallel_test_groups)
+        self.parallel_tests = list(
+            filter(self.__filter_test,
+                   self.parallel_tests)
         )
-        self.non_parallel_test_groups = list(
-            filter(self.__filter_group,
-                   self.non_parallel_test_groups)
+        self.non_parallel_tests = list(
+            filter(self.__filter_test,
+                   self.non_parallel_tests)
         )
 
         # create a pool of worker threads
@@ -355,14 +287,16 @@ def run_all_tests(self):
             # make a note of tests start time
             self.start = time.time()
 
-            print("Parallel autotests:")
-            # assign worker threads to run test groups
-            for test_group in self.parallel_test_groups:
-                result = pool.apply_async(run_test_group,
-                                          [self.__get_cmdline(test_group),
-                                           self.target,
-                                           test_group])
-                results.append(result)
+            if len(self.parallel_tests) > 0:
+                print("Parallel autotests:")
+                # assign worker threads to run test groups
+                for test_group in self.parallel_tests:
+                    result = pool.apply_async(run_test_group,
+                                              [self.__get_cmdline(),
+                                               "",
+                                               self.target,
+                                               test_group])
+                    results.append(result)
 
             # iterate while we have group execution results to get
             while len(results) > 0:
@@ -377,18 +311,19 @@ def run_all_tests(self):
 
                     res = group_result.get()
 
-                    self.__process_results(res)
+                    self.__process_result(res)
 
                     # remove result from results list once we're done with it
                     results.remove(group_result)
 
-            print("Non-parallel autotests:")
-            # run non_parallel tests. they are run one by one, synchronously
-            for test_group in self.non_parallel_test_groups:
-                group_result = run_test_group(
-                    self.__get_cmdline(test_group), self.target, test_group)
+            if len(self.non_parallel_tests) > 0:
+                print("Non-parallel autotests:")
+                # run non_parallel tests. they are run one by one, synchronously
+                for test_group in self.non_parallel_tests:
+                    group_result = run_test_group(
+                        self.__get_cmdline(), "", self.target, test_group)
 
-                self.__process_results(group_result)
+                    self.__process_result(group_result)
 
             # get total run time
             cur_time = time.time()
-- 
2.17.1

[dpdk-dev] [PATCH 7/7] autotest: properly parallelize unit tests

[Anatoly Burakov]

Now that everything else is in place, we can run unit tests in a
different fashion to what they were running as before. Previously,
we had all autotests as part of groups (largely obtained through
trial and error) to ensure parallel execution while still limiting
amounts of memory used by those tests.

This is no longer necessary, and as of previous commit, all tests
are now in the same group (still broken into two categories). They
still run one-by-one though. Fix this by initializing child
processes in multiprocessing Pool initialization, and putting all
tests on the queue, so that tests are executed by the first idle
worker. Tests are also affinitized to different NUMA nodes using
taskset in a round-robin fashion, to prevent over-exhausting
memory on any given NUMA node.

Non-parallel tests are executed in similar fashion, but on a
separate queue which will have only one pool worker, ensuring
non-parallel execution.

Support for FreeBSD is also added to ensure that on FreeBSD, all
tests are run sequentially even for the parallel section.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest.py        |   6 +-
 test/test/autotest_runner.py | 277 ++++++++++++++++++++++-------------
 2 files changed, 183 insertions(+), 100 deletions(-)

diff --git a/test/test/autotest.py b/test/test/autotest.py
index ae27daef7..12997fdf0 100644
--- a/test/test/autotest.py
+++ b/test/test/autotest.py
@@ -36,8 +36,12 @@ def usage():
 
 print(cmdline)
 
+# how many workers to run tests with. FreeBSD doesn't support multiple primary
+# processes, so make it 1, otherwise make it 4. ignored for non-parallel tests
+n_processes = 1 if "bsdapp" in target else 4
+
 runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist,
-                                        test_whitelist)
+                                        test_whitelist, n_processes)
 
 runner.parallel_tests = autotest_data.parallel_test_list[:]
 runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index d6ae57e76..36941a40a 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -6,16 +6,16 @@
 from __future__ import print_function
 import StringIO
 import csv
-import multiprocessing
+from multiprocessing import Pool, Queue
 import pexpect
 import re
 import subprocess
 import sys
 import time
+import glob
+import os
 
 # wait for prompt
-
-
 def wait_prompt(child):
     try:
         child.sendline()
@@ -28,22 +28,47 @@ def wait_prompt(child):
     else:
         return False
 
-# run a test group
-# each result tuple in results list consists of:
-#   result value (0 or -1)
-#   result string
-#   test name
-#   total test run time (double)
-#   raw test log
-#   test report (if not available, should be None)
-#
-# this function needs to be outside AutotestRunner class
-# because otherwise Pool won't work (or rather it will require
-# quite a bit of effort to make it work).
+
+# get all valid NUMA nodes
+def get_numa_nodes():
+    return [
+        int(
+            re.match(r"node(\d+)", os.path.basename(node))
+            .group(1)
+        )
+        for node in glob.glob("/sys/devices/system/node/node*")
+    ]
+
+
+# find first (or any, really) CPU on a particular node, will be used to spread
+# processes around NUMA nodes to avoid exhausting memory on particular node
+def first_cpu_on_node(node_nr):
+    cpu_path = glob.glob("/sys/devices/system/node/node%d/cpu*" % node_nr)[0]
+    cpu_name = os.path.basename(cpu_path)
+    m = re.match(r"cpu(\d+)", cpu_name)
+    return int(m.group(1))
+
+
+pool_child = None  # per-process child
 
 
-def run_test_group(cmdline, prefix, target, test):
+# we initialize each worker with a queue because we need per-pool unique
+# command-line arguments, but we cannot do different arguments in an initializer
+# because the API doesn't allow per-worker initializer arguments. so, instead,
+# we will initialize with a shared queue, and dequeue command-line arguments
+# from this queue
+def pool_init(queue, result_queue):
+    global pool_child
+
+    cmdline, prefix = queue.get()
     start_time = time.time()
+    name = ("Start %s" % prefix) if prefix != "" else "Start"
+
+    # use default prefix if no prefix was specified
+    prefix_cmdline = "--file-prefix=%s" % prefix if prefix != "" else ""
+
+    # append prefix to cmdline
+    cmdline = "%s %s" % (cmdline, prefix_cmdline)
 
     # prepare logging of init
     startuplog = StringIO.StringIO()
@@ -54,24 +79,60 @@ def run_test_group(cmdline, prefix, target, test):
         print("\n%s %s\n" % ("=" * 20, prefix), file=startuplog)
         print("\ncmdline=%s" % cmdline, file=startuplog)
 
-        child = pexpect.spawn(cmdline, logfile=startuplog)
+        pool_child = pexpect.spawn(cmdline, logfile=startuplog)
 
         # wait for target to boot
-        if not wait_prompt(child):
-            child.close()
+        if not wait_prompt(pool_child):
+            pool_child.close()
+
+            result = tuple((-1,
+                            "Fail [No prompt]",
+                            name,
+                            time.time() - start_time,
+                            startuplog.getvalue(),
+                            None))
+            pool_child = None
+        else:
+            result = tuple((0,
+                            "Success",
+                            name,
+                            time.time() - start_time,
+                            startuplog.getvalue(),
+                            None))
+    except:
+        result = tuple((-1,
+                        "Fail [Can't run]",
+                        name,
+                        time.time() - start_time,
+                        startuplog.getvalue(),
+                        None))
+        pool_child = None
 
-            return -1, "Fail [No prompt]", "Start %s" % prefix,\
-                   time.time() - start_time, startuplog.getvalue(), None
+    result_queue.put(result)
 
-    except:
-        return -1, "Fail [Can't run]", "Start %s" % prefix,\
-               time.time() - start_time, startuplog.getvalue(), None
+
+# run a test
+# each result tuple in results list consists of:
+#   result value (0 or -1)
+#   result string
+#   test name
+#   total test run time (double)
+#   raw test log
+#   test report (if not available, should be None)
+#
+# this function needs to be outside AutotestRunner class because otherwise Pool
+# won't work (or rather it will require quite a bit of effort to make it work).
+def run_test(target, test):
+    global pool_child
+
+    if pool_child is None:
+        return -1, "Fail [No test process]", test["Name"], 0, "", None
 
     # create log buffer for each test
     # in multiprocessing environment, the logging would be
     # interleaved and will create a mess, hence the buffering
     logfile = StringIO.StringIO()
-    child.logfile = logfile
+    pool_child.logfile = logfile
 
     # make a note when the test started
     start_time = time.time()
@@ -81,7 +142,7 @@ def run_test_group(cmdline, prefix, target, test):
         print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
         # run test function associated with the test
-        result = test["Func"](child, test["Command"])
+        result = test["Func"](pool_child, test["Command"])
 
         # make a note when the test was finished
         end_time = time.time()
@@ -109,15 +170,6 @@ def run_test_group(cmdline, prefix, target, test):
         result = (-1, "Fail [Crash]", test["Name"],
                   end_time - start_time, logfile.getvalue(), None)
 
-    # regardless of whether test has crashed, try quitting it
-    try:
-        child.sendline("quit")
-        child.close()
-    # if the test crashed, just do nothing instead
-    except:
-        # nop
-        pass
-
     # return test results
     return result
 
@@ -137,7 +189,7 @@ class AutotestRunner:
     blacklist = []
     whitelist = []
 
-    def __init__(self, cmdline, target, blacklist, whitelist):
+    def __init__(self, cmdline, target, blacklist, whitelist, n_processes):
         self.cmdline = cmdline
         self.target = target
         self.binary = cmdline.split()[0]
@@ -146,6 +198,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.skipped = []
         self.parallel_tests = []
         self.non_parallel_tests = []
+        self.n_processes = n_processes
+        self.active_processes = 0
 
         # log file filename
         logfile = "%s.log" % target
@@ -159,11 +213,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.csvwriter.writerow(["test_name", "test_result", "result_str"])
 
     # set up cmdline string
-    def __get_cmdline(self):
-        cmdline = self.cmdline
-
-        # affinitize startup so that tests don't fail on i686
-        cmdline = "taskset 1 " + cmdline
+    def __get_cmdline(self, cpu_nr):
+        cmdline = ("taskset -c %i " % cpu_nr) + self.cmdline
 
         return cmdline
 
@@ -241,6 +292,51 @@ def __filter_test(self, test):
 
         return True
 
+    def __run_test_group(self, test_group, worker_cmdlines):
+        group_queue = Queue()
+        init_result_queue = Queue()
+        for proc, cmdline in enumerate(worker_cmdlines):
+            prefix = "test%i" % proc if len(worker_cmdlines) > 1 else ""
+            group_queue.put(tuple((cmdline, prefix)))
+
+        # create a pool of worker threads
+        # we will initialize child in the initializer, and we don't need to
+        # close the child because when the pool worker gets destroyed, child
+        # closes the process
+        pool = Pool(processes=len(worker_cmdlines),
+                    initializer=pool_init,
+                    initargs=(group_queue, init_result_queue))
+
+        results = []
+
+        # process all initialization results
+        for _ in range(len(worker_cmdlines)):
+            self.__process_result(init_result_queue.get())
+
+        # run all tests asynchronously
+        for test in test_group:
+            result = pool.apply_async(run_test, (self.target, test))
+            results.append(result)
+
+        # tell the pool to stop all processes once done
+        pool.close()
+
+        # iterate while we have group execution results to get
+        while len(results) > 0:
+            # iterate over a copy to be able to safely delete results
+            # this iterates over a list of group results
+            for async_result in results[:]:
+                # if the thread hasn't finished yet, continue
+                if not async_result.ready():
+                    continue
+
+                res = async_result.get()
+
+                self.__process_result(res)
+
+                # remove result from results list once we're done with it
+                results.remove(async_result)
+
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
@@ -253,77 +349,60 @@ def run_all_tests(self):
                    self.non_parallel_tests)
         )
 
-        # create a pool of worker threads
-        pool = multiprocessing.Pool(processes=1)
-
-        results = []
+        parallel_cmdlines = []
+        # FreeBSD doesn't have NUMA support
+        numa_nodes = get_numa_nodes()
+        if len(numa_nodes) > 0:
+            for proc in range(self.n_processes):
+                # spread cpu affinity between NUMA nodes to have less chance of
+                # running out of memory while running multiple test apps in
+                # parallel. to do that, alternate between NUMA nodes in a round
+                # robin fashion, and pick an arbitrary CPU from that node to
+                # taskset our execution to
+                numa_node = numa_nodes[self.active_processes % len(numa_nodes)]
+                cpu_nr = first_cpu_on_node(numa_node)
+                parallel_cmdlines += [self.__get_cmdline(cpu_nr)]
+                # increase number of active processes so that the next cmdline
+                # gets a different NUMA node
+                self.active_processes += 1
+        else:
+            parallel_cmdlines = [self.cmdline] * self.n_processes
 
-        # whatever happens, try to save as much logs as possible
-        try:
+        print("Running tests with %d workers" % self.n_processes)
 
-            # create table header
-            print("")
-            print("Test name".ljust(30) + "Test result".ljust(29) +
-                  "Test".center(9) + "Total".center(9))
-            print("=" * 80)
+        # create table header
+        print("")
+        print("Test name".ljust(30) + "Test result".ljust(29) +
+              "Test".center(9) + "Total".center(9))
+        print("=" * 80)
 
-            # print out skipped autotests if there were any
-            if len(self.skipped):
-                print("Skipped autotests:")
+        if len(self.skipped):
+            print("Skipped autotests:")
 
-                # print out any skipped tests
-                for result in self.skipped:
-                    # unpack result tuple
-                    test_result, result_str, test_name, _, _, _ = result
-                    self.csvwriter.writerow([test_name, test_result,
-                                             result_str])
+            # print out any skipped tests
+            for result in self.skipped:
+                # unpack result tuple
+                test_result, result_str, test_name, _, _, _ = result
+                self.csvwriter.writerow([test_name, test_result, result_str])
 
-                    t = ("%s:" % test_name).ljust(30)
-                    t += result_str.ljust(29)
-                    t += "[00m 00s]"
+                t = ("%s:" % test_name).ljust(30)
+                t += result_str.ljust(29)
+                t += "[00m 00s]"
 
-                    print(t)
+                print(t)
 
-            # make a note of tests start time
-            self.start = time.time()
+        # make a note of tests start time
+        self.start = time.time()
 
+        # whatever happens, try to save as much logs as possible
+        try:
             if len(self.parallel_tests) > 0:
                 print("Parallel autotests:")
-                # assign worker threads to run test groups
-                for test_group in self.parallel_tests:
-                    result = pool.apply_async(run_test_group,
-                                              [self.__get_cmdline(),
-                                               "",
-                                               self.target,
-                                               test_group])
-                    results.append(result)
-
-            # iterate while we have group execution results to get
-            while len(results) > 0:
-
-                # iterate over a copy to be able to safely delete results
-                # this iterates over a list of group results
-                for group_result in results[:]:
-
-                    # if the thread hasn't finished yet, continue
-                    if not group_result.ready():
-                        continue
-
-                    res = group_result.get()
-
-                    self.__process_result(res)
-
-                    # remove result from results list once we're done with it
-                    results.remove(group_result)
+                self.__run_test_group(self.parallel_tests, parallel_cmdlines)
 
             if len(self.non_parallel_tests) > 0:
                 print("Non-parallel autotests:")
-                # run non_parallel tests. they are run one by one, synchronously
-                for test_group in self.non_parallel_tests:
-                    group_result = run_test_group(
-                        self.__get_cmdline(), "", self.target, test_group)
-
-                    self.__process_result(group_result)
+                self.__run_test_group(self.non_parallel_tests, [self.cmdline])
 
             # get total run time
             cur_time = time.time()
-- 
2.17.1

Re: [dpdk-dev] [PATCH 0/7] Make unit tests great again

[Thomas Monjalon]

+Cc Jananee

07/06/2018 23:01, Anatoly Burakov:
> Previously, unit tests were running in groups. There were
> technical reasons why that was the case (mostly having to do
> with limiting memory), but it was hard to maintain and update
> the autotest script.
> 
> In 18.05, limiting of memory at DPDK startup was no longer
> necessary, as DPDK allocates memory at runtime as needed. This
> has the implication that the old test grouping can now be
> retired and replaced with a more sensible way of running unit
> tests (using multiprocessing pool of workers and a queue of
> tests). This patchset accomplishes exactly that.
> 
> This patchset conflicts with some of the earlier work on
> autotests [1] [2] [3], but i think it presents a cleaner
> solution for some of the problems highlighted by those patch
> series. I can integrate those patches into this series if
> need be.
> 
> [1] http://dpdk.org/dev/patchwork/patch/40370/
> [2] http://dpdk.org/dev/patchwork/patch/40371/
> [3] http://dpdk.org/dev/patchwork/patch/40372/

It may be interesting to work on lists of tests as done
in the following patch:
	http://dpdk.org/dev/patchwork/patch/40373/

The idea is to split tests in several categories:
	- basic and short test
	- longer and lower priority
	- performance test
As a long term solution, we can think about making category an attribute
inside the test itself?

Re: [dpdk-dev] [PATCH 0/7] Make unit tests great again

[Burakov, Anatoly]

On 12-Jun-18 2:07 PM, Thomas Monjalon wrote:
> +Cc Jananee
> 
> 07/06/2018 23:01, Anatoly Burakov:
>> Previously, unit tests were running in groups. There were
>> technical reasons why that was the case (mostly having to do
>> with limiting memory), but it was hard to maintain and update
>> the autotest script.
>>
>> In 18.05, limiting of memory at DPDK startup was no longer
>> necessary, as DPDK allocates memory at runtime as needed. This
>> has the implication that the old test grouping can now be
>> retired and replaced with a more sensible way of running unit
>> tests (using multiprocessing pool of workers and a queue of
>> tests). This patchset accomplishes exactly that.
>>
>> This patchset conflicts with some of the earlier work on
>> autotests [1] [2] [3], but i think it presents a cleaner
>> solution for some of the problems highlighted by those patch
>> series. I can integrate those patches into this series if
>> need be.
>>
>> [1] http://dpdk.org/dev/patchwork/patch/40370/
>> [2] http://dpdk.org/dev/patchwork/patch/40371/
>> [3] http://dpdk.org/dev/patchwork/patch/40372/
> 
> It may be interesting to work on lists of tests as done
> in the following patch:
> 	http://dpdk.org/dev/patchwork/patch/40373/
> 
> The idea is to split tests in several categories:
> 	- basic and short test
> 	- longer and lower priority
> 	- performance test
> As a long term solution, we can think about making category an attribute
> inside the test itself?
> 

These test categories do not conflict with my patchset as they 
ultimately rely on white/blacklisting, which will continue to work as 
before.

In my view, it really boils down to two things - either tests can be run 
in parallel with others (i.e. their result won't be affected by another 
independent DPDK test app instance), or not. On top of that, we can use 
blacklisting or whitelisting to define which tests will actually be run 
(i.e. define any "categories" we want), but their (non-)parallelism must 
always be respected to get good test results.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 0/7] Make unit tests great again

[Bruce Richardson]

On Wed, Jun 13, 2018 at 09:38:32AM +0100, Burakov, Anatoly wrote:
> On 12-Jun-18 2:07 PM, Thomas Monjalon wrote:
> > +Cc Jananee
> > 
> > 07/06/2018 23:01, Anatoly Burakov:
> > > Previously, unit tests were running in groups. There were
> > > technical reasons why that was the case (mostly having to do
> > > with limiting memory), but it was hard to maintain and update
> > > the autotest script.
> > > 
> > > In 18.05, limiting of memory at DPDK startup was no longer
> > > necessary, as DPDK allocates memory at runtime as needed. This
> > > has the implication that the old test grouping can now be
> > > retired and replaced with a more sensible way of running unit
> > > tests (using multiprocessing pool of workers and a queue of
> > > tests). This patchset accomplishes exactly that.
> > > 
> > > This patchset conflicts with some of the earlier work on
> > > autotests [1] [2] [3], but i think it presents a cleaner
> > > solution for some of the problems highlighted by those patch
> > > series. I can integrate those patches into this series if
> > > need be.
> > > 
> > > [1] http://dpdk.org/dev/patchwork/patch/40370/
> > > [2] http://dpdk.org/dev/patchwork/patch/40371/
> > > [3] http://dpdk.org/dev/patchwork/patch/40372/
> > 
> > It may be interesting to work on lists of tests as done
> > in the following patch:
> > 	http://dpdk.org/dev/patchwork/patch/40373/
> > 
> > The idea is to split tests in several categories:
> > 	- basic and short test
> > 	- longer and lower priority
> > 	- performance test
> > As a long term solution, we can think about making category an attribute
> > inside the test itself?
> > 
> 
> These test categories do not conflict with my patchset as they ultimately
> rely on white/blacklisting, which will continue to work as before.
> 
> In my view, it really boils down to two things - either tests can be run in
> parallel with others (i.e. their result won't be affected by another
> independent DPDK test app instance), or not. On top of that, we can use
> blacklisting or whitelisting to define which tests will actually be run
> (i.e. define any "categories" we want), but their (non-)parallelism must
> always be respected to get good test results.
> 
Have you looked at: http://mesonbuild.com/Unit-tests.html, at it would be
good to transition away from our own custom script infrastructure for the
tests.

There is already some support for running the unit tests using "meson
test", but it could do with some more cleanup e.g. to move the tests into
suitable suites (corresponding to the categories Thomas has suggested). We
could also do with specifying properly what tests are parallel-safe and
what aren't.

[dpdk-dev] [PATCH v2 00/10] Make unit tests great again

[Reshma Pattan]

Previously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.

In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.

This patchset merges changes done in [1], [2] and [3]

[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://dpdk.org/dev/patchwork/patch/40371/
[3] http://patches.dpdk.org/patch/40373/

v2: merged changes done in patch set [1],[2] and [3]
and created patches 08/10, 09/10, 10/10

Anatoly Burakov (7):
  autotest: fix printing
  autotest: fix invalid code on reports
  autotest: make autotest runner python 2/3 compliant
  autotest: visually separate different test categories
  autotest: improve filtering
  autotest: remove autotest grouping
  autotest: properly parallelize unit tests

Jananee Parthasarathy (1):
  mk: update make targets for classified testcases

Reshma Pattan (2):
  autotest: add new test cases to autotest
  autotest: update result for skipped test cases

 mk/rte.sdkroot.mk                |    4 +-
 mk/rte.sdktest.mk                |   33 +-
 test/test/autotest.py            |   13 +-
 test/test/autotest_data.py       | 1101 +++++++++++++++++++++++---------------
 test/test/autotest_runner.py     |  519 +++++++++---------
 test/test/autotest_test_funcs.py |    6 +-
 6 files changed, 972 insertions(+), 704 deletions(-)

-- 
2.14.4

[dpdk-dev] [PATCH v2 01/10] autotest: fix printing

[Reshma Pattan]

Previously, printing was done using tuple syntax, which caused
output to appear as a tuple as opposed to being one string. Fix
this by using addition operator instead.

Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index a692f0697..b09b57876 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -247,7 +247,7 @@ def __process_results(self, results):
 
             # don't print out total time every line, it's the same anyway
             if i == len(results) - 1:
-                print(result,
+                print(result +
                       "[%02dm %02ds]" % (total_time / 60, total_time % 60))
             else:
                 print(result)
@@ -332,8 +332,8 @@ def run_all_tests(self):
 
             # create table header
             print("")
-            print("Test name".ljust(30), "Test result".ljust(29),
-                  "Test".center(9), "Total".center(9))
+            print("Test name".ljust(30) + "Test result".ljust(29) +
+                  "Test".center(9) + "Total".center(9))
             print("=" * 80)
 
             # make a note of tests start time
-- 
2.14.4

[dpdk-dev] [PATCH v2 02/10] autotest: fix invalid code on reports

[Reshma Pattan]

There are no reports defined for any test, so this codepath was
never triggered, but it's still wrong because it's referencing
variables that aren't there. Fix it by passing target into the
test function, and reference correct log variable.

Fixes: e2cc79b75d9f ("app: rework autotest.py")
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index b09b57876..bdc32da5d 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -41,7 +41,7 @@ def wait_prompt(child):
 # quite a bit of effort to make it work).
 
 
-def run_test_group(cmdline, test_group):
+def run_test_group(cmdline, target, test_group):
     results = []
     child = None
     start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group):
             # make a note when the test was finished
             end_time = time.time()
 
+            log = logfile.getvalue()
+
             # append test data to the result tuple
-            result += (test["Name"], end_time - start_time,
-                       logfile.getvalue())
+            result += (test["Name"], end_time - start_time, log)
 
             # call report function, if any defined, and supply it with
             # target and complete log for test run
             if test["Report"]:
-                report = test["Report"](self.target, log)
+                report = test["Report"](target, log)
 
                 # append report to results tuple
                 result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self):
             for test_group in self.parallel_test_groups:
                 result = pool.apply_async(run_test_group,
                                           [self.__get_cmdline(test_group),
+                                           self.target,
                                            test_group])
                 results.append(result)
 
@@ -367,7 +369,7 @@ def run_all_tests(self):
             # run non_parallel tests. they are run one by one, synchronously
             for test_group in self.non_parallel_test_groups:
                 group_result = run_test_group(
-                    self.__get_cmdline(test_group), test_group)
+                    self.__get_cmdline(test_group), self.target, test_group)
 
                 self.__process_results(group_result)
 
-- 
2.14.4

[dpdk-dev] [PATCH v2 03/10] autotest: make autotest runner python 2/3 compliant

[Reshma Pattan]

Autotest runner was still using python 2-style print syntax. Fix
it by importing print function from the future, and fix the calls
to be python-3 style.

Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index bdc32da5d..f6b669a2e 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -3,6 +3,7 @@
 
 # The main logic behind running autotests in parallel
 
+from __future__ import print_function
 import StringIO
 import csv
 import multiprocessing
@@ -52,8 +53,8 @@ def run_test_group(cmdline, target, test_group):
         # prepare logging of init
         startuplog = StringIO.StringIO()
 
-        print >>startuplog, "\n%s %s\n" % ("=" * 20, test_group["Prefix"])
-        print >>startuplog, "\ncmdline=%s" % cmdline
+        print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog)
+        print("\ncmdline=%s" % cmdline, file=startuplog)
 
         child = pexpect.spawn(cmdline, logfile=startuplog)
 
@@ -117,7 +118,7 @@ def run_test_group(cmdline, target, test_group):
 
         try:
             # print test name to log buffer
-            print >>logfile, "\n%s %s\n" % ("-" * 20, test["Name"])
+            print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
             # run test function associated with the test
             if stripped or test["Command"] in avail_cmds:
-- 
2.14.4

[dpdk-dev] [PATCH v2 04/10] autotest: visually separate different test categories

[Reshma Pattan]

Help visually identify parallel vs. non-parallel autotests.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index f6b669a2e..d9d5f7a97 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -341,6 +341,7 @@ def run_all_tests(self):
             # make a note of tests start time
             self.start = time.time()
 
+            print("Parallel autotests:")
             # assign worker threads to run test groups
             for test_group in self.parallel_test_groups:
                 result = pool.apply_async(run_test_group,
@@ -367,6 +368,7 @@ def run_all_tests(self):
                     # remove result from results list once we're done with it
                     results.remove(group_result)
 
+            print("Non-parallel autotests:")
             # run non_parallel tests. they are run one by one, synchronously
             for test_group in self.non_parallel_test_groups:
                 group_result = run_test_group(
-- 
2.14.4

[dpdk-dev] [PATCH v2 05/10] autotest: improve filtering

[Reshma Pattan]

Improve code for filtering test groups. Also, move reading binary
symbols into filtering stage, so that tests that are meant to be
skipped are never attempted to be executed in the first place.
Before running tests, print out any tests that were skipped because
they weren't compiled.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 118 ++++++++++++++++++++++++-------------------
 1 file changed, 66 insertions(+), 52 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index d9d5f7a97..c98ec2a57 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -95,13 +95,6 @@ def run_test_group(cmdline, target, test_group):
     results.append((0, "Success", "Start %s" % test_group["Prefix"],
                     time.time() - start_time, startuplog.getvalue(), None))
 
-    # parse the binary for available test commands
-    binary = cmdline.split()[0]
-    stripped = 'not stripped' not in subprocess.check_output(['file', binary])
-    if not stripped:
-        symbols = subprocess.check_output(['nm', binary]).decode('utf-8')
-        avail_cmds = re.findall('test_register_(\w+)', symbols)
-
     # run all tests in test group
     for test in test_group["Tests"]:
 
@@ -121,10 +114,7 @@ def run_test_group(cmdline, target, test_group):
             print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
             # run test function associated with the test
-            if stripped or test["Command"] in avail_cmds:
-                result = test["Func"](child, test["Command"])
-            else:
-                result = (0, "Skipped [Not Available]")
+            result = test["Func"](child, test["Command"])
 
             # make a note when the test was finished
             end_time = time.time()
@@ -186,8 +176,10 @@ class AutotestRunner:
     def __init__(self, cmdline, target, blacklist, whitelist):
         self.cmdline = cmdline
         self.target = target
+        self.binary = cmdline.split()[0]
         self.blacklist = blacklist
         self.whitelist = whitelist
+        self.skipped = []
 
         # log file filename
         logfile = "%s.log" % target
@@ -276,53 +268,58 @@ def __process_results(self, results):
             if i != 0:
                 self.csvwriter.writerow([test_name, test_result, result_str])
 
-    # this function iterates over test groups and removes each
-    # test that is not in whitelist/blacklist
-    def __filter_groups(self, test_groups):
-        groups_to_remove = []
-
-        # filter out tests from parallel test groups
-        for i, test_group in enumerate(test_groups):
-
-            # iterate over a copy so that we could safely delete individual
-            # tests
-            for test in test_group["Tests"][:]:
-                test_id = test["Command"]
-
-                # dump tests are specified in full e.g. "Dump_mempool"
-                if "_autotest" in test_id:
-                    test_id = test_id[:-len("_autotest")]
-
-                # filter out blacklisted/whitelisted tests
-                if self.blacklist and test_id in self.blacklist:
-                    test_group["Tests"].remove(test)
-                    continue
-                if self.whitelist and test_id not in self.whitelist:
-                    test_group["Tests"].remove(test)
-                    continue
-
-            # modify or remove original group
-            if len(test_group["Tests"]) > 0:
-                test_groups[i] = test_group
-            else:
-                # remember which groups should be deleted
-                # put the numbers backwards so that we start
-                # deleting from the end, not from the beginning
-                groups_to_remove.insert(0, i)
+    # this function checks individual test and decides if this test should be in
+    # the group by comparing it against  whitelist/blacklist. it also checks if
+    # the test is compiled into the binary, and marks it as skipped if necessary
+    def __filter_test(self, test):
+        test_cmd = test["Command"]
+        test_id = test_cmd
+
+        # dump tests are specified in full e.g. "Dump_mempool"
+        if "_autotest" in test_id:
+            test_id = test_id[:-len("_autotest")]
+
+        # filter out blacklisted/whitelisted tests
+        if self.blacklist and test_id in self.blacklist:
+            return False
+        if self.whitelist and test_id not in self.whitelist:
+            return False
+
+        # if test wasn't compiled in, remove it as well
+
+        # parse the binary for available test commands
+        stripped = 'not stripped' not in \
+                   subprocess.check_output(['file', self.binary])
+        if not stripped:
+            symbols = subprocess.check_output(['nm',
+                                               self.binary]).decode('utf-8')
+            avail_cmds = re.findall('test_register_(\w+)', symbols)
+
+            if test_cmd not in avail_cmds:
+                # notify user
+                result = 0, "Skipped [Not compiled]", test_id, 0, "", None
+                self.skipped.append(tuple(result))
+                return False
 
-        # remove test groups that need to be removed
-        for i in groups_to_remove:
-            del test_groups[i]
+        return True
 
-        return test_groups
+    def __filter_group(self, group):
+        group["Tests"] = list(filter(self.__filter_test, group["Tests"]))
+        return len(group["Tests"]) > 0
 
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
-        self.parallel_test_groups = \
-            self.__filter_groups(self.parallel_test_groups)
-        self.non_parallel_test_groups = \
-            self.__filter_groups(self.non_parallel_test_groups)
+        # for each test group, check all tests against the filter, then remove
+        # all groups that don't have any tests
+        self.parallel_test_groups = list(
+            filter(self.__filter_group,
+                   self.parallel_test_groups)
+        )
+        self.non_parallel_test_groups = list(
+            filter(self.__filter_group,
+                   self.non_parallel_test_groups)
+        )
 
         # create a pool of worker threads
         pool = multiprocessing.Pool(processes=1)
@@ -338,6 +335,23 @@ def run_all_tests(self):
                   "Test".center(9) + "Total".center(9))
             print("=" * 80)
 
+            # print out skipped autotests if there were any
+            if len(self.skipped):
+                print("Skipped autotests:")
+
+                # print out any skipped tests
+                for result in self.skipped:
+                    # unpack result tuple
+                    test_result, result_str, test_name, _, _, _ = result
+                    self.csvwriter.writerow([test_name, test_result,
+                                             result_str])
+
+                    t = ("%s:" % test_name).ljust(30)
+                    t += result_str.ljust(29)
+                    t += "[00m 00s]"
+
+                    print(t)
+
             # make a note of tests start time
             self.start = time.time()
 
-- 
2.14.4

[dpdk-dev] [PATCH v2 06/10] autotest: remove autotest grouping

[Reshma Pattan]

Previously, all autotests were grouped into (seemingly arbitrary)
groups. The goal was to run all tests in parallel (so that autotest
finishes faster), but we couldn't just do it willy-nilly because
DPDK couldn't allocate and free hugepages on-demand, so we had to
find autotest groupings that could work memory-wise and still be
fast enough to not hold up shorter tests. The inflexibility of
memory subsystem has now been fixed for 18.05, so grouping
autotests is no longer necessary.

Thus, this commit moves all autotests into two groups -
parallel(izable) autotests, and non-arallel(izable) autotests
(typically performance tests). Note that this particular commit
makes running autotests dog slow because while the tests are now
in a single group, the test function itself hasn't changed much,
so all autotests are now run one-by-one, starting and stopping
the DPDK test application.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest.py        |   7 +-
 test/test/autotest_data.py   | 749 +++++++++++++++++--------------------------
 test/test/autotest_runner.py | 271 ++++++----------
 3 files changed, 408 insertions(+), 619 deletions(-)

diff --git a/test/test/autotest.py b/test/test/autotest.py
index 1cfd8cf22..ae27daef7 100644
--- a/test/test/autotest.py
+++ b/test/test/autotest.py
@@ -39,11 +39,8 @@ def usage():
 runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist,
                                         test_whitelist)
 
-for test_group in autotest_data.parallel_test_group_list:
-    runner.add_parallel_test_group(test_group)
-
-for test_group in autotest_data.non_parallel_test_group_list:
-    runner.add_non_parallel_test_group(test_group)
+runner.parallel_tests = autotest_data.parallel_test_list[:]
+runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
 
 num_fails = runner.run_all_tests()
 
diff --git a/test/test/autotest_data.py b/test/test/autotest_data.py
index aacfe0a66..c24e7bc25 100644
--- a/test/test/autotest_data.py
+++ b/test/test/autotest_data.py
@@ -3,465 +3,322 @@
 
 # Test data for autotests
 
-from glob import glob
 from autotest_test_funcs import *
 
-
-# quick and dirty function to find out number of sockets
-def num_sockets():
-    result = len(glob("/sys/devices/system/node/node*"))
-    if result == 0:
-        return 1
-    return result
-
-
-# Assign given number to each socket
-# e.g. 32 becomes 32,32 or 32,32,32,32
-def per_sockets(num):
-    return ",".join([str(num)] * num_sockets())
-
 # groups of tests that can be run in parallel
 # the grouping has been found largely empirically
-parallel_test_group_list = [
-    {
-        "Prefix":    "group_1",
-        "Memory":    per_sockets(8),
-        "Tests":
-        [
-            {
-                "Name":    "Cycles autotest",
-                "Command": "cycles_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Timer autotest",
-                "Command": "timer_autotest",
-                "Func":    timer_autotest,
-                "Report":   None,
-            },
-            {
-                "Name":    "Debug autotest",
-                "Command": "debug_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Errno autotest",
-                "Command": "errno_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Meter autotest",
-                "Command": "meter_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Common autotest",
-                "Command": "common_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Resource autotest",
-                "Command": "resource_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_2",
-        "Memory":    "16",
-        "Tests":
-        [
-            {
-                "Name":    "Memory autotest",
-                "Command": "memory_autotest",
-                "Func":    memory_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Read/write lock autotest",
-                "Command": "rwlock_autotest",
-                "Func":    rwlock_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Logs autotest",
-                "Command": "logs_autotest",
-                "Func":    logs_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "CPU flags autotest",
-                "Command": "cpuflags_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Version autotest",
-                "Command": "version_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "EAL filesystem autotest",
-                "Command": "eal_fs_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "EAL flags autotest",
-                "Command": "eal_flags_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Hash autotest",
-                "Command": "hash_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ],
-    },
-    {
-        "Prefix":    "group_3",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "LPM autotest",
-                "Command": "lpm_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "LPM6 autotest",
-                "Command": "lpm6_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Memcpy autotest",
-                "Command": "memcpy_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Memzone autotest",
-                "Command": "memzone_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "String autotest",
-                "Command": "string_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Alarm autotest",
-                "Command": "alarm_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_4",
-        "Memory":    per_sockets(128),
-        "Tests":
-        [
-            {
-                "Name":    "PCI autotest",
-                "Command": "pci_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Malloc autotest",
-                "Command": "malloc_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Multi-process autotest",
-                "Command": "multiprocess_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Mbuf autotest",
-                "Command": "mbuf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Per-lcore autotest",
-                "Command": "per_lcore_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Ring autotest",
-                "Command": "ring_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_5",
-        "Memory":    "32",
-        "Tests":
-        [
-            {
-                "Name":    "Spinlock autotest",
-                "Command": "spinlock_autotest",
-                "Func":    spinlock_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Byte order autotest",
-                "Command": "byteorder_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "TAILQ autotest",
-                "Command": "tailq_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Command-line autotest",
-                "Command": "cmdline_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Interrupts autotest",
-                "Command": "interrupt_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_6",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Function reentrancy autotest",
-                "Command": "func_reentrancy_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Mempool autotest",
-                "Command": "mempool_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Atomics autotest",
-                "Command": "atomic_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Prefetch autotest",
-                "Command": "prefetch_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Red autotest",
-                "Command": "red_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_7",
-        "Memory":    "64",
-        "Tests":
-        [
-            {
-                "Name":    "PMD ring autotest",
-                "Command": "ring_pmd_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Access list control autotest",
-                "Command": "acl_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Sched autotest",
-                "Command": "sched_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+parallel_test_list = [
+    {
+        "Name":    "Cycles autotest",
+        "Command": "cycles_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Timer autotest",
+        "Command": "timer_autotest",
+        "Func":    timer_autotest,
+        "Report":   None,
+    },
+    {
+        "Name":    "Debug autotest",
+        "Command": "debug_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Errno autotest",
+        "Command": "errno_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Meter autotest",
+        "Command": "meter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Common autotest",
+        "Command": "common_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Resource autotest",
+        "Command": "resource_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memory autotest",
+        "Command": "memory_autotest",
+        "Func":    memory_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Read/write lock autotest",
+        "Command": "rwlock_autotest",
+        "Func":    rwlock_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Logs autotest",
+        "Command": "logs_autotest",
+        "Func":    logs_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "CPU flags autotest",
+        "Command": "cpuflags_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Version autotest",
+        "Command": "version_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "EAL filesystem autotest",
+        "Command": "eal_fs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "EAL flags autotest",
+        "Command": "eal_flags_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash autotest",
+        "Command": "hash_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "LPM autotest",
+        "Command": "lpm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "LPM6 autotest",
+        "Command": "lpm6_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memcpy autotest",
+        "Command": "memcpy_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memzone autotest",
+        "Command": "memzone_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "String autotest",
+        "Command": "string_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Alarm autotest",
+        "Command": "alarm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "PCI autotest",
+        "Command": "pci_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Malloc autotest",
+        "Command": "malloc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Multi-process autotest",
+        "Command": "multiprocess_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mbuf autotest",
+        "Command": "mbuf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Per-lcore autotest",
+        "Command": "per_lcore_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Ring autotest",
+        "Command": "ring_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Spinlock autotest",
+        "Command": "spinlock_autotest",
+        "Func":    spinlock_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Byte order autotest",
+        "Command": "byteorder_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "TAILQ autotest",
+        "Command": "tailq_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Command-line autotest",
+        "Command": "cmdline_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Interrupts autotest",
+        "Command": "interrupt_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Function reentrancy autotest",
+        "Command": "func_reentrancy_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mempool autotest",
+        "Command": "mempool_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Atomics autotest",
+        "Command": "atomic_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Prefetch autotest",
+        "Command": "prefetch_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red autotest",
+        "Command": "red_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "PMD ring autotest",
+        "Command": "ring_pmd_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Access list control autotest",
+        "Command": "acl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Sched autotest",
+        "Command": "sched_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
 ]
 
 # tests that should not be run when any other tests are running
-non_parallel_test_group_list = [
-
+non_parallel_test_list = [
     {
-        "Prefix":    "eventdev",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "Eventdev common autotest",
-                "Command": "eventdev_common_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "eventdev_sw",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "Eventdev sw autotest",
-                "Command": "eventdev_sw_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "kni",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "KNI autotest",
-                "Command": "kni_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "mempool_perf",
-        "Memory":    per_sockets(256),
-        "Tests":
-        [
-            {
-                "Name":    "Mempool performance autotest",
-                "Command": "mempool_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "memcpy_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Memcpy performance autotest",
-                "Command": "memcpy_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "hash_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Hash performance autotest",
-                "Command": "hash_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power autotest",
-                "Command":    "power_autotest",
-                "Func":       default_autotest,
-                "Report":      None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power_acpi_cpufreq",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power ACPI cpufreq autotest",
-                "Command":    "power_acpi_cpufreq_autotest",
-                "Func":       default_autotest,
-                "Report":     None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power_kvm_vm",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power KVM VM  autotest",
-                "Command":    "power_kvm_vm_autotest",
-                "Func":       default_autotest,
-                "Report":     None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "timer_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Timer performance autotest",
-                "Command": "timer_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+        "Name":    "Eventdev common autotest",
+        "Command": "eventdev_common_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Eventdev sw autotest",
+        "Command": "eventdev_sw_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "KNI autotest",
+        "Command": "kni_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mempool performance autotest",
+        "Command": "mempool_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memcpy performance autotest",
+        "Command": "memcpy_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash performance autotest",
+        "Command": "hash_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":       "Power autotest",
+        "Command":    "power_autotest",
+        "Func":       default_autotest,
+        "Report":      None,
+    },
+    {
+        "Name":       "Power ACPI cpufreq autotest",
+        "Command":    "power_acpi_cpufreq_autotest",
+        "Func":       default_autotest,
+        "Report":     None,
+    },
+    {
+        "Name":       "Power KVM VM  autotest",
+        "Command":    "power_kvm_vm_autotest",
+        "Func":       default_autotest,
+        "Report":     None,
+    },
+    {
+        "Name":    "Timer performance autotest",
+        "Command": "timer_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
-
     #
     # Please always make sure that ring_perf is the last test!
     #
     {
-        "Prefix":    "ring_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Ring performance autotest",
-                "Command": "ring_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+        "Name":    "Ring performance autotest",
+        "Command": "ring_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
 ]
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index c98ec2a57..d6ae57e76 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -42,18 +42,16 @@ def wait_prompt(child):
 # quite a bit of effort to make it work).
 
 
-def run_test_group(cmdline, target, test_group):
-    results = []
-    child = None
+def run_test_group(cmdline, prefix, target, test):
     start_time = time.time()
-    startuplog = None
+
+    # prepare logging of init
+    startuplog = StringIO.StringIO()
 
     # run test app
     try:
-        # prepare logging of init
-        startuplog = StringIO.StringIO()
 
-        print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog)
+        print("\n%s %s\n" % ("=" * 20, prefix), file=startuplog)
         print("\ncmdline=%s" % cmdline, file=startuplog)
 
         child = pexpect.spawn(cmdline, logfile=startuplog)
@@ -62,88 +60,54 @@ def run_test_group(cmdline, target, test_group):
         if not wait_prompt(child):
             child.close()
 
-            results.append((-1,
-                            "Fail [No prompt]",
-                            "Start %s" % test_group["Prefix"],
-                            time.time() - start_time,
-                            startuplog.getvalue(),
-                            None))
-
-            # mark all tests as failed
-            for test in test_group["Tests"]:
-                results.append((-1, "Fail [No prompt]", test["Name"],
-                                time.time() - start_time, "", None))
-            # exit test
-            return results
+            return -1, "Fail [No prompt]", "Start %s" % prefix,\
+                   time.time() - start_time, startuplog.getvalue(), None
 
     except:
-        results.append((-1,
-                        "Fail [Can't run]",
-                        "Start %s" % test_group["Prefix"],
-                        time.time() - start_time,
-                        startuplog.getvalue(),
-                        None))
-
-        # mark all tests as failed
-        for t in test_group["Tests"]:
-            results.append((-1, "Fail [Can't run]", t["Name"],
-                            time.time() - start_time, "", None))
-        # exit test
-        return results
-
-    # startup was successful
-    results.append((0, "Success", "Start %s" % test_group["Prefix"],
-                    time.time() - start_time, startuplog.getvalue(), None))
-
-    # run all tests in test group
-    for test in test_group["Tests"]:
-
-        # create log buffer for each test
-        # in multiprocessing environment, the logging would be
-        # interleaved and will create a mess, hence the buffering
-        logfile = StringIO.StringIO()
-        child.logfile = logfile
-
-        result = ()
-
-        # make a note when the test started
-        start_time = time.time()
+        return -1, "Fail [Can't run]", "Start %s" % prefix,\
+               time.time() - start_time, startuplog.getvalue(), None
 
-        try:
-            # print test name to log buffer
-            print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
+    # create log buffer for each test
+    # in multiprocessing environment, the logging would be
+    # interleaved and will create a mess, hence the buffering
+    logfile = StringIO.StringIO()
+    child.logfile = logfile
 
-            # run test function associated with the test
-            result = test["Func"](child, test["Command"])
+    # make a note when the test started
+    start_time = time.time()
 
-            # make a note when the test was finished
-            end_time = time.time()
+    try:
+        # print test name to log buffer
+        print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
-            log = logfile.getvalue()
+        # run test function associated with the test
+        result = test["Func"](child, test["Command"])
 
-            # append test data to the result tuple
-            result += (test["Name"], end_time - start_time, log)
+        # make a note when the test was finished
+        end_time = time.time()
 
-            # call report function, if any defined, and supply it with
-            # target and complete log for test run
-            if test["Report"]:
-                report = test["Report"](target, log)
+        log = logfile.getvalue()
 
-                # append report to results tuple
-                result += (report,)
-            else:
-                # report is None
-                result += (None,)
-        except:
-            # make a note when the test crashed
-            end_time = time.time()
+        # append test data to the result tuple
+        result += (test["Name"], end_time - start_time, log)
+
+        # call report function, if any defined, and supply it with
+        # target and complete log for test run
+        if test["Report"]:
+            report = test["Report"](target, log)
+
+            # append report to results tuple
+            result += (report,)
+        else:
+            # report is None
+            result += (None,)
+    except:
+        # make a note when the test crashed
+        end_time = time.time()
 
-            # mark test as failed
-            result = (-1, "Fail [Crash]", test["Name"],
-                      end_time - start_time, logfile.getvalue(), None)
-        finally:
-            # append the results to the results list
-            results.append(result)
+        # mark test as failed
+        result = (-1, "Fail [Crash]", test["Name"],
+                  end_time - start_time, logfile.getvalue(), None)
 
     # regardless of whether test has crashed, try quitting it
     try:
@@ -155,7 +119,7 @@ def run_test_group(cmdline, target, test_group):
         pass
 
     # return test results
-    return results
+    return result
 
 
 # class representing an instance of autotests run
@@ -180,6 +144,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.blacklist = blacklist
         self.whitelist = whitelist
         self.skipped = []
+        self.parallel_tests = []
+        self.non_parallel_tests = []
 
         # log file filename
         logfile = "%s.log" % target
@@ -193,80 +159,52 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.csvwriter.writerow(["test_name", "test_result", "result_str"])
 
     # set up cmdline string
-    def __get_cmdline(self, test):
+    def __get_cmdline(self):
         cmdline = self.cmdline
 
-        # append memory limitations for each test
-        # otherwise tests won't run in parallel
-        if "i686" not in self.target:
-            cmdline += " --socket-mem=%s" % test["Memory"]
-        else:
-            # affinitize startup so that tests don't fail on i686
-            cmdline = "taskset 1 " + cmdline
-            cmdline += " -m " + str(sum(map(int, test["Memory"].split(","))))
-
-        # set group prefix for autotest group
-        # otherwise they won't run in parallel
-        cmdline += " --file-prefix=%s" % test["Prefix"]
+        # affinitize startup so that tests don't fail on i686
+        cmdline = "taskset 1 " + cmdline
 
         return cmdline
 
-    def add_parallel_test_group(self, test_group):
-        self.parallel_test_groups.append(test_group)
+    def __process_result(self, result):
 
-    def add_non_parallel_test_group(self, test_group):
-        self.non_parallel_test_groups.append(test_group)
+        # unpack result tuple
+        test_result, result_str, test_name, \
+            test_time, log, report = result
 
-    def __process_results(self, results):
-        # this iterates over individual test results
-        for i, result in enumerate(results):
+        # get total run time
+        cur_time = time.time()
+        total_time = int(cur_time - self.start)
 
-            # increase total number of tests that were run
-            # do not include "start" test
-            if i > 0:
-                self.n_tests += 1
+        # print results, test run time and total time since start
+        result = ("%s:" % test_name).ljust(30)
+        result += result_str.ljust(29)
+        result += "[%02dm %02ds]" % (test_time / 60, test_time % 60)
 
-            # unpack result tuple
-            test_result, result_str, test_name, \
-                test_time, log, report = result
+        # don't print out total time every line, it's the same anyway
+        print(result + "[%02dm %02ds]" % (total_time / 60, total_time % 60))
 
-            # get total run time
-            cur_time = time.time()
-            total_time = int(cur_time - self.start)
+        # if test failed and it wasn't a "start" test
+        if test_result < 0:
+            self.fails += 1
 
-            # print results, test run time and total time since start
-            result = ("%s:" % test_name).ljust(30)
-            result += result_str.ljust(29)
-            result += "[%02dm %02ds]" % (test_time / 60, test_time % 60)
+        # collect logs
+        self.log_buffers.append(log)
 
-            # don't print out total time every line, it's the same anyway
-            if i == len(results) - 1:
-                print(result +
-                      "[%02dm %02ds]" % (total_time / 60, total_time % 60))
+        # create report if it exists
+        if report:
+            try:
+                f = open("%s_%s_report.rst" %
+                         (self.target, test_name), "w")
+            except IOError:
+                print("Report for %s could not be created!" % test_name)
             else:
-                print(result)
-
-            # if test failed and it wasn't a "start" test
-            if test_result < 0 and not i == 0:
-                self.fails += 1
-
-            # collect logs
-            self.log_buffers.append(log)
-
-            # create report if it exists
-            if report:
-                try:
-                    f = open("%s_%s_report.rst" %
-                             (self.target, test_name), "w")
-                except IOError:
-                    print("Report for %s could not be created!" % test_name)
-                else:
-                    with f:
-                        f.write(report)
-
-            # write test result to CSV file
-            if i != 0:
-                self.csvwriter.writerow([test_name, test_result, result_str])
+                with f:
+                    f.write(report)
+
+        # write test result to CSV file
+        self.csvwriter.writerow([test_name, test_result, result_str])
 
     # this function checks individual test and decides if this test should be in
     # the group by comparing it against  whitelist/blacklist. it also checks if
@@ -303,22 +241,16 @@ def __filter_test(self, test):
 
         return True
 
-    def __filter_group(self, group):
-        group["Tests"] = list(filter(self.__filter_test, group["Tests"]))
-        return len(group["Tests"]) > 0
-
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
-        # for each test group, check all tests against the filter, then remove
-        # all groups that don't have any tests
-        self.parallel_test_groups = list(
-            filter(self.__filter_group,
-                   self.parallel_test_groups)
+        self.parallel_tests = list(
+            filter(self.__filter_test,
+                   self.parallel_tests)
         )
-        self.non_parallel_test_groups = list(
-            filter(self.__filter_group,
-                   self.non_parallel_test_groups)
+        self.non_parallel_tests = list(
+            filter(self.__filter_test,
+                   self.non_parallel_tests)
         )
 
         # create a pool of worker threads
@@ -355,14 +287,16 @@ def run_all_tests(self):
             # make a note of tests start time
             self.start = time.time()
 
-            print("Parallel autotests:")
-            # assign worker threads to run test groups
-            for test_group in self.parallel_test_groups:
-                result = pool.apply_async(run_test_group,
-                                          [self.__get_cmdline(test_group),
-                                           self.target,
-                                           test_group])
-                results.append(result)
+            if len(self.parallel_tests) > 0:
+                print("Parallel autotests:")
+                # assign worker threads to run test groups
+                for test_group in self.parallel_tests:
+                    result = pool.apply_async(run_test_group,
+                                              [self.__get_cmdline(),
+                                               "",
+                                               self.target,
+                                               test_group])
+                    results.append(result)
 
             # iterate while we have group execution results to get
             while len(results) > 0:
@@ -377,18 +311,19 @@ def run_all_tests(self):
 
                     res = group_result.get()
 
-                    self.__process_results(res)
+                    self.__process_result(res)
 
                     # remove result from results list once we're done with it
                     results.remove(group_result)
 
-            print("Non-parallel autotests:")
-            # run non_parallel tests. they are run one by one, synchronously
-            for test_group in self.non_parallel_test_groups:
-                group_result = run_test_group(
-                    self.__get_cmdline(test_group), self.target, test_group)
+            if len(self.non_parallel_tests) > 0:
+                print("Non-parallel autotests:")
+                # run non_parallel tests. they are run one by one, synchronously
+                for test_group in self.non_parallel_tests:
+                    group_result = run_test_group(
+                        self.__get_cmdline(), "", self.target, test_group)
 
-                self.__process_results(group_result)
+                    self.__process_result(group_result)
 
             # get total run time
             cur_time = time.time()
-- 
2.14.4

[dpdk-dev] [PATCH v2 07/10] autotest: properly parallelize unit tests

[Reshma Pattan]

Now that everything else is in place, we can run unit tests in a
different fashion to what they were running as before. Previously,
we had all autotests as part of groups (largely obtained through
trial and error) to ensure parallel execution while still limiting
amounts of memory used by those tests.

This is no longer necessary, and as of previous commit, all tests
are now in the same group (still broken into two categories). They
still run one-by-one though. Fix this by initializing child
processes in multiprocessing Pool initialization, and putting all
tests on the queue, so that tests are executed by the first idle
worker. Tests are also affinitized to different NUMA nodes using
taskset in a round-robin fashion, to prevent over-exhausting
memory on any given NUMA node.

Non-parallel tests are executed in similar fashion, but on a
separate queue which will have only one pool worker, ensuring
non-parallel execution.

Support for FreeBSD is also added to ensure that on FreeBSD, all
tests are run sequentially even for the parallel section.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest.py        |   6 +-
 test/test/autotest_runner.py | 277 +++++++++++++++++++++++++++----------------
 2 files changed, 183 insertions(+), 100 deletions(-)

diff --git a/test/test/autotest.py b/test/test/autotest.py
index ae27daef7..12997fdf0 100644
--- a/test/test/autotest.py
+++ b/test/test/autotest.py
@@ -36,8 +36,12 @@ def usage():
 
 print(cmdline)
 
+# how many workers to run tests with. FreeBSD doesn't support multiple primary
+# processes, so make it 1, otherwise make it 4. ignored for non-parallel tests
+n_processes = 1 if "bsdapp" in target else 4
+
 runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist,
-                                        test_whitelist)
+                                        test_whitelist, n_processes)
 
 runner.parallel_tests = autotest_data.parallel_test_list[:]
 runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index d6ae57e76..36941a40a 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -6,16 +6,16 @@
 from __future__ import print_function
 import StringIO
 import csv
-import multiprocessing
+from multiprocessing import Pool, Queue
 import pexpect
 import re
 import subprocess
 import sys
 import time
+import glob
+import os
 
 # wait for prompt
-
-
 def wait_prompt(child):
     try:
         child.sendline()
@@ -28,22 +28,47 @@ def wait_prompt(child):
     else:
         return False
 
-# run a test group
-# each result tuple in results list consists of:
-#   result value (0 or -1)
-#   result string
-#   test name
-#   total test run time (double)
-#   raw test log
-#   test report (if not available, should be None)
-#
-# this function needs to be outside AutotestRunner class
-# because otherwise Pool won't work (or rather it will require
-# quite a bit of effort to make it work).
+
+# get all valid NUMA nodes
+def get_numa_nodes():
+    return [
+        int(
+            re.match(r"node(\d+)", os.path.basename(node))
+            .group(1)
+        )
+        for node in glob.glob("/sys/devices/system/node/node*")
+    ]
+
+
+# find first (or any, really) CPU on a particular node, will be used to spread
+# processes around NUMA nodes to avoid exhausting memory on particular node
+def first_cpu_on_node(node_nr):
+    cpu_path = glob.glob("/sys/devices/system/node/node%d/cpu*" % node_nr)[0]
+    cpu_name = os.path.basename(cpu_path)
+    m = re.match(r"cpu(\d+)", cpu_name)
+    return int(m.group(1))
+
+
+pool_child = None  # per-process child
 
 
-def run_test_group(cmdline, prefix, target, test):
+# we initialize each worker with a queue because we need per-pool unique
+# command-line arguments, but we cannot do different arguments in an initializer
+# because the API doesn't allow per-worker initializer arguments. so, instead,
+# we will initialize with a shared queue, and dequeue command-line arguments
+# from this queue
+def pool_init(queue, result_queue):
+    global pool_child
+
+    cmdline, prefix = queue.get()
     start_time = time.time()
+    name = ("Start %s" % prefix) if prefix != "" else "Start"
+
+    # use default prefix if no prefix was specified
+    prefix_cmdline = "--file-prefix=%s" % prefix if prefix != "" else ""
+
+    # append prefix to cmdline
+    cmdline = "%s %s" % (cmdline, prefix_cmdline)
 
     # prepare logging of init
     startuplog = StringIO.StringIO()
@@ -54,24 +79,60 @@ def run_test_group(cmdline, prefix, target, test):
         print("\n%s %s\n" % ("=" * 20, prefix), file=startuplog)
         print("\ncmdline=%s" % cmdline, file=startuplog)
 
-        child = pexpect.spawn(cmdline, logfile=startuplog)
+        pool_child = pexpect.spawn(cmdline, logfile=startuplog)
 
         # wait for target to boot
-        if not wait_prompt(child):
-            child.close()
+        if not wait_prompt(pool_child):
+            pool_child.close()
+
+            result = tuple((-1,
+                            "Fail [No prompt]",
+                            name,
+                            time.time() - start_time,
+                            startuplog.getvalue(),
+                            None))
+            pool_child = None
+        else:
+            result = tuple((0,
+                            "Success",
+                            name,
+                            time.time() - start_time,
+                            startuplog.getvalue(),
+                            None))
+    except:
+        result = tuple((-1,
+                        "Fail [Can't run]",
+                        name,
+                        time.time() - start_time,
+                        startuplog.getvalue(),
+                        None))
+        pool_child = None
 
-            return -1, "Fail [No prompt]", "Start %s" % prefix,\
-                   time.time() - start_time, startuplog.getvalue(), None
+    result_queue.put(result)
 
-    except:
-        return -1, "Fail [Can't run]", "Start %s" % prefix,\
-               time.time() - start_time, startuplog.getvalue(), None
+
+# run a test
+# each result tuple in results list consists of:
+#   result value (0 or -1)
+#   result string
+#   test name
+#   total test run time (double)
+#   raw test log
+#   test report (if not available, should be None)
+#
+# this function needs to be outside AutotestRunner class because otherwise Pool
+# won't work (or rather it will require quite a bit of effort to make it work).
+def run_test(target, test):
+    global pool_child
+
+    if pool_child is None:
+        return -1, "Fail [No test process]", test["Name"], 0, "", None
 
     # create log buffer for each test
     # in multiprocessing environment, the logging would be
     # interleaved and will create a mess, hence the buffering
     logfile = StringIO.StringIO()
-    child.logfile = logfile
+    pool_child.logfile = logfile
 
     # make a note when the test started
     start_time = time.time()
@@ -81,7 +142,7 @@ def run_test_group(cmdline, prefix, target, test):
         print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
         # run test function associated with the test
-        result = test["Func"](child, test["Command"])
+        result = test["Func"](pool_child, test["Command"])
 
         # make a note when the test was finished
         end_time = time.time()
@@ -109,15 +170,6 @@ def run_test_group(cmdline, prefix, target, test):
         result = (-1, "Fail [Crash]", test["Name"],
                   end_time - start_time, logfile.getvalue(), None)
 
-    # regardless of whether test has crashed, try quitting it
-    try:
-        child.sendline("quit")
-        child.close()
-    # if the test crashed, just do nothing instead
-    except:
-        # nop
-        pass
-
     # return test results
     return result
 
@@ -137,7 +189,7 @@ class AutotestRunner:
     blacklist = []
     whitelist = []
 
-    def __init__(self, cmdline, target, blacklist, whitelist):
+    def __init__(self, cmdline, target, blacklist, whitelist, n_processes):
         self.cmdline = cmdline
         self.target = target
         self.binary = cmdline.split()[0]
@@ -146,6 +198,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.skipped = []
         self.parallel_tests = []
         self.non_parallel_tests = []
+        self.n_processes = n_processes
+        self.active_processes = 0
 
         # log file filename
         logfile = "%s.log" % target
@@ -159,11 +213,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.csvwriter.writerow(["test_name", "test_result", "result_str"])
 
     # set up cmdline string
-    def __get_cmdline(self):
-        cmdline = self.cmdline
-
-        # affinitize startup so that tests don't fail on i686
-        cmdline = "taskset 1 " + cmdline
+    def __get_cmdline(self, cpu_nr):
+        cmdline = ("taskset -c %i " % cpu_nr) + self.cmdline
 
         return cmdline
 
@@ -241,6 +292,51 @@ def __filter_test(self, test):
 
         return True
 
+    def __run_test_group(self, test_group, worker_cmdlines):
+        group_queue = Queue()
+        init_result_queue = Queue()
+        for proc, cmdline in enumerate(worker_cmdlines):
+            prefix = "test%i" % proc if len(worker_cmdlines) > 1 else ""
+            group_queue.put(tuple((cmdline, prefix)))
+
+        # create a pool of worker threads
+        # we will initialize child in the initializer, and we don't need to
+        # close the child because when the pool worker gets destroyed, child
+        # closes the process
+        pool = Pool(processes=len(worker_cmdlines),
+                    initializer=pool_init,
+                    initargs=(group_queue, init_result_queue))
+
+        results = []
+
+        # process all initialization results
+        for _ in range(len(worker_cmdlines)):
+            self.__process_result(init_result_queue.get())
+
+        # run all tests asynchronously
+        for test in test_group:
+            result = pool.apply_async(run_test, (self.target, test))
+            results.append(result)
+
+        # tell the pool to stop all processes once done
+        pool.close()
+
+        # iterate while we have group execution results to get
+        while len(results) > 0:
+            # iterate over a copy to be able to safely delete results
+            # this iterates over a list of group results
+            for async_result in results[:]:
+                # if the thread hasn't finished yet, continue
+                if not async_result.ready():
+                    continue
+
+                res = async_result.get()
+
+                self.__process_result(res)
+
+                # remove result from results list once we're done with it
+                results.remove(async_result)
+
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
@@ -253,77 +349,60 @@ def run_all_tests(self):
                    self.non_parallel_tests)
         )
 
-        # create a pool of worker threads
-        pool = multiprocessing.Pool(processes=1)
-
-        results = []
+        parallel_cmdlines = []
+        # FreeBSD doesn't have NUMA support
+        numa_nodes = get_numa_nodes()
+        if len(numa_nodes) > 0:
+            for proc in range(self.n_processes):
+                # spread cpu affinity between NUMA nodes to have less chance of
+                # running out of memory while running multiple test apps in
+                # parallel. to do that, alternate between NUMA nodes in a round
+                # robin fashion, and pick an arbitrary CPU from that node to
+                # taskset our execution to
+                numa_node = numa_nodes[self.active_processes % len(numa_nodes)]
+                cpu_nr = first_cpu_on_node(numa_node)
+                parallel_cmdlines += [self.__get_cmdline(cpu_nr)]
+                # increase number of active processes so that the next cmdline
+                # gets a different NUMA node
+                self.active_processes += 1
+        else:
+            parallel_cmdlines = [self.cmdline] * self.n_processes
 
-        # whatever happens, try to save as much logs as possible
-        try:
+        print("Running tests with %d workers" % self.n_processes)
 
-            # create table header
-            print("")
-            print("Test name".ljust(30) + "Test result".ljust(29) +
-                  "Test".center(9) + "Total".center(9))
-            print("=" * 80)
+        # create table header
+        print("")
+        print("Test name".ljust(30) + "Test result".ljust(29) +
+              "Test".center(9) + "Total".center(9))
+        print("=" * 80)
 
-            # print out skipped autotests if there were any
-            if len(self.skipped):
-                print("Skipped autotests:")
+        if len(self.skipped):
+            print("Skipped autotests:")
 
-                # print out any skipped tests
-                for result in self.skipped:
-                    # unpack result tuple
-                    test_result, result_str, test_name, _, _, _ = result
-                    self.csvwriter.writerow([test_name, test_result,
-                                             result_str])
+            # print out any skipped tests
+            for result in self.skipped:
+                # unpack result tuple
+                test_result, result_str, test_name, _, _, _ = result
+                self.csvwriter.writerow([test_name, test_result, result_str])
 
-                    t = ("%s:" % test_name).ljust(30)
-                    t += result_str.ljust(29)
-                    t += "[00m 00s]"
+                t = ("%s:" % test_name).ljust(30)
+                t += result_str.ljust(29)
+                t += "[00m 00s]"
 
-                    print(t)
+                print(t)
 
-            # make a note of tests start time
-            self.start = time.time()
+        # make a note of tests start time
+        self.start = time.time()
 
+        # whatever happens, try to save as much logs as possible
+        try:
             if len(self.parallel_tests) > 0:
                 print("Parallel autotests:")
-                # assign worker threads to run test groups
-                for test_group in self.parallel_tests:
-                    result = pool.apply_async(run_test_group,
-                                              [self.__get_cmdline(),
-                                               "",
-                                               self.target,
-                                               test_group])
-                    results.append(result)
-
-            # iterate while we have group execution results to get
-            while len(results) > 0:
-
-                # iterate over a copy to be able to safely delete results
-                # this iterates over a list of group results
-                for group_result in results[:]:
-
-                    # if the thread hasn't finished yet, continue
-                    if not group_result.ready():
-                        continue
-
-                    res = group_result.get()
-
-                    self.__process_result(res)
-
-                    # remove result from results list once we're done with it
-                    results.remove(group_result)
+                self.__run_test_group(self.parallel_tests, parallel_cmdlines)
 
             if len(self.non_parallel_tests) > 0:
                 print("Non-parallel autotests:")
-                # run non_parallel tests. they are run one by one, synchronously
-                for test_group in self.non_parallel_tests:
-                    group_result = run_test_group(
-                        self.__get_cmdline(), "", self.target, test_group)
-
-                    self.__process_result(group_result)
+                self.__run_test_group(self.non_parallel_tests, [self.cmdline])
 
             # get total run time
             cur_time = time.time()
-- 
2.14.4

[dpdk-dev] [PATCH v2 08/10] autotest: add new test cases to autotest

[Reshma Pattan]

Autotest is enhanced with additional test cases
being added to autotest_data.py

Cc: stable@dpdk.org

Signed-off-by: Reshma Pattan <reshma.pattan@intel.com>
Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
---
 test/test/autotest_data.py | 370 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 362 insertions(+), 8 deletions(-)

diff --git a/test/test/autotest_data.py b/test/test/autotest_data.py
index c24e7bc25..6e26cc190 100644
--- a/test/test/autotest_data.py
+++ b/test/test/autotest_data.py
@@ -134,12 +134,6 @@
         "Func":    default_autotest,
         "Report":  None,
     },
-    {
-        "Name":    "PCI autotest",
-        "Command": "pci_autotest",
-        "Func":    default_autotest,
-        "Report":  None,
-    },
     {
         "Name":    "Malloc autotest",
         "Command": "malloc_autotest",
@@ -248,6 +242,311 @@
         "Func":    default_autotest,
         "Report":  None,
     },
+    {
+        "Name":    "Eventdev selftest octeontx",
+        "Command": "eventdev_selftest_octeontx",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Event ring autotest",
+        "Command": "event_ring_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Table autotest",
+        "Command": "table_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Flow classify autotest",
+        "Command": "flow_classify_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump struct sizes",
+        "Command": "dump_struct_sizes",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump mempool",
+        "Command": "dump_mempool",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump malloc stats",
+        "Command": "dump_malloc_stats",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump devargs",
+        "Command": "dump_devargs",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump log types",
+        "Command": "dump_log_types",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump_ring",
+        "Command": "dump_ring",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Quit",
+        "Command": "quit",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Set rxtx mode",
+        "Command": "set_rxtx_mode",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Set rxtx anchor",
+        "Command": "set_rxtx_anchor",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Set rxtx sc",
+        "Command": "set_rxtx_sc",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Event eth rx adapter autotest",
+        "Command": "event_eth_rx_adapter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Rawdev autotest",
+        "Command": "rawdev_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Kvargs autotest",
+        "Command": "kvargs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump physmem",
+        "Command": "dump_physmem",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump memzone",
+        "Command": "dump_memzone",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Devargs autotest",
+        "Command": "devargs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Link bonding autotest",
+        "Command": "link_bonding_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Link bonding mode4 autotest",
+        "Command": "link_bonding_mode4_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Link bonding rssconf autotest",
+        "Command": "link_bonding_rssconf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Crc autotest",
+        "Command": "crc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Distributor autotest",
+        "Command": "distributor_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Reorder autotest",
+        "Command": "reorder_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Barrier autotest",
+        "Command": "barrier_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "User delay",
+        "Command": "user_delay_us",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Bitmap test",
+        "Command": "bitmap_test",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash scaling autotest",
+        "Command": "hash_scaling_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash multiwriter autotest",
+        "Command": "hash_multiwriter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Service autotest",
+        "Command": "service_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Timer racecond autotest",
+        "Command": "timer_racecond_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Member autotest",
+        "Command": "member_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":   "Efd_autotest",
+        "Command": "efd_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Thash autotest",
+        "Command": "thash_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash function autotest",
+        "Command": "hash_functions_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+        "Name":    "Cryptodev sw mrvl autotest",
+        "Command": "cryptodev_sw_mrvl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev dpaa2 sec autotest",
+        "Command": "cryptodev_dpaa2_sec_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev dpaa sec autotest",
+        "Command": "cryptodev_dpaa_sec_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev qat autotest",
+        "Command": "cryptodev_qat_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev aesni mb autotest",
+        "Command": "cryptodev_aesni_mb_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev openssl autotest",
+        "Command": "cryptodev_openssl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev scheduler autotest",
+        "Command": "cryptodev_scheduler_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev aesni gcm autotest",
+        "Command": "cryptodev_aesni_gcm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev null autotest",
+        "Command": "cryptodev_null_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev sw snow3g autotest",
+        "Command": "cryptodev_sw_snow3g_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev sw kasumi autotest",
+        "Command": "cryptodev_sw_kasumi_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev_sw_zuc_autotest",
+        "Command": "cryptodev_sw_zuc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Reciprocal division",
+        "Command": "reciprocal_division",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red all",
+        "Command": "red_all",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
 ]
 
 # tests that should not be run when any other tests are running
@@ -259,8 +558,8 @@
         "Report":  None,
     },
     {
-        "Name":    "Eventdev sw autotest",
-        "Command": "eventdev_sw_autotest",
+        "Name":    "Eventdev selftest sw",
+        "Command": "eventdev_selftest_sw",
         "Func":    default_autotest,
         "Report":  None,
     },
@@ -312,6 +611,61 @@
         "Func":    default_autotest,
         "Report":  None,
     },
+    {
+
+        "Name":    "Pmd perf autotest",
+        "Command": "pmd_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Ring pmd perf autotest",
+        "Command": "ring_pmd_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Distributor perf autotest",
+        "Command": "distributor_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red_perf",
+        "Command": "red_perf",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Lpm6 perf autotest",
+        "Command": "lpm6_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Lpm perf autotest",
+        "Command": "lpm_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+         "Name":    "Efd perf autotest",
+         "Command": "efd_perf_autotest",
+         "Func":    default_autotest,
+         "Report":  None,
+    },
+    {
+        "Name":    "Member perf autotest",
+        "Command": "member_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Reciprocal division perf",
+        "Command": "reciprocal_division_perf",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
     #
     # Please always make sure that ring_perf is the last test!
     #
-- 
2.14.4

[dpdk-dev] [PATCH v2 09/10] autotest: update result for skipped test cases

[Reshma Pattan]

Fixed in autotest_test_funcs.py to handle test cases
which returns "Skipped" as result.
The issue was skipped test cases got timed out,
causing delay in autotests execution.

Cc: stable@dpdk.org

Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
Signed-off-by: Reshma Pattan <reshma.pattan@intel.com>
---
 test/test/autotest_test_funcs.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/test/test/autotest_test_funcs.py b/test/test/autotest_test_funcs.py
index 65fe33539..219c20828 100644
--- a/test/test/autotest_test_funcs.py
+++ b/test/test/autotest_test_funcs.py
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2010-2014 Intel Corporation
+# Copyright(c) 2010-2018 Intel Corporation
 
 # Test functions
 
@@ -12,12 +12,14 @@
 def default_autotest(child, test_name):
     child.sendline(test_name)
     result = child.expect(["Test OK", "Test Failed",
-                           "Command not found", pexpect.TIMEOUT], timeout=900)
+                           "Command not found", "Skipped", pexpect.TIMEOUT], timeout=900)
     if result == 1:
         return -1, "Fail"
     elif result == 2:
         return -1, "Fail [Not found]"
     elif result == 3:
+        return -1, "Fail [Test returns Skipped]"
+    elif result == 4:
         return -1, "Fail [Timeout]"
     return 0, "Success"
 
-- 
2.14.4

[dpdk-dev] [PATCH v2 10/10] mk: update make targets for classified testcases

[Reshma Pattan]

Makefiles are updated with new test case lists.
Test cases are classified as -
P1 - Main test cases,
P2 - Cryptodev/driver test cases,
P3 - Perf test cases which takes longer than 10s,
P4 - Logging/Dump test cases.

Makefile is updated with different targets
for the above classified groups.
Test cases for different targets are listed accordingly.

Cc: stable@dpdk.org

Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
Reviewed-by: Reshma Pattan <reshma.pattan@intel.com>
---
 mk/rte.sdkroot.mk |  4 ++--
 mk/rte.sdktest.mk | 33 +++++++++++++++++++++++++++------
 2 files changed, 29 insertions(+), 8 deletions(-)

diff --git a/mk/rte.sdkroot.mk b/mk/rte.sdkroot.mk
index f43cc7829..ea3473ebf 100644
--- a/mk/rte.sdkroot.mk
+++ b/mk/rte.sdkroot.mk
@@ -68,8 +68,8 @@ config defconfig showconfigs showversion showversionum:
 cscope gtags tags etags:
 	$(Q)$(RTE_SDK)/devtools/build-tags.sh $@ $T
 
-.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage
-test test-basic test-fast test-ring test-mempool test-perf coverage:
+.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump
+test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump:
 	$(Q)$(MAKE) -f $(RTE_SDK)/mk/rte.sdktest.mk $@
 
 test: test-build
diff --git a/mk/rte.sdktest.mk b/mk/rte.sdktest.mk
index ee1fe0c7e..13d1efb6a 100644
--- a/mk/rte.sdktest.mk
+++ b/mk/rte.sdktest.mk
@@ -18,14 +18,35 @@ DIR := $(shell basename $(RTE_OUTPUT))
 #
 # test: launch auto-tests, very simple for now.
 #
-.PHONY: test test-basic test-fast test-perf coverage
+.PHONY: test test-basic test-fast test-perf test-drivers test-dump coverage
 
-PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf
-coverage: BLACKLIST=-$(PERFLIST)
-test-fast: BLACKLIST=-$(PERFLIST)
-test-perf: WHITELIST=$(PERFLIST)
+PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf,\
+         reciprocal_division,reciprocal_division_perf,lpm_perf,red_all,\
+         barrier,hash_multiwriter,timer_racecond,efd,hash_functions,\
+         eventdev_selftest_sw,member_perf,efd_perf,lpm6_perf,red_perf,\
+         distributor_perf,ring_pmd_perf,pmd_perf,ring_perf
+DRIVERSLIST=link_bonding,link_bonding_mode4,link_bonding_rssconf,\
+            cryptodev_sw_mrvl,cryptodev_dpaa2_sec,cryptodev_dpaa_sec,\
+            cryptodev_qat,cryptodev_aesni_mb,cryptodev_openssl,\
+            cryptodev_scheduler,cryptodev_aesni_gcm,cryptodev_null,\
+            cryptodev_sw_snow3g,cryptodev_sw_kasumi,cryptodev_sw_zuc
+DUMPLIST=dump_struct_sizes,dump_mempool,dump_malloc_stats,dump_devargs,\
+         dump_log_types,dump_ring,quit,dump_physmem,dump_memzone,\
+         devargs_autotest
 
-test test-basic test-fast test-perf:
+SPACESTR:=
+SPACESTR+=
+STRIPPED_PERFLIST=$(subst $(SPACESTR),,$(PERFLIST))
+STRIPPED_DRIVERSLIST=$(subst $(SPACESTR),,$(DRIVERSLIST))
+STRIPPED_DUMPLIST=$(subst $(SPACESTR),,$(DUMPLIST))
+
+coverage: BLACKLIST=-$(STRIPPED_PERFLIST)
+test-fast: BLACKLIST=-$(STRIPPED_PERFLIST),$(STRIPPED_DRIVERSLIST),$(STRIPPED_DUMPLIST)
+test-perf: WHITELIST=$(STRIPPED_PERFLIST)
+test-drivers: WHITELIST=$(STRIPPED_DRIVERSLIST)
+test-dump: WHITELIST=$(STRIPPED_DUMPLIST)
+
+test test-basic test-fast test-perf test-drivers test-dump:
 	@mkdir -p $(AUTOTEST_DIR) ; \
 	cd $(AUTOTEST_DIR) ; \
 	if [ -f $(RTE_OUTPUT)/app/test ]; then \
-- 
2.14.4

Re: [dpdk-dev] [PATCH v2 08/10] autotest: add new test cases to autotest

[Burakov, Anatoly]

On 13-Jul-18 5:20 PM, Reshma Pattan wrote:
> Autotest is enhanced with additional test cases
> being added to autotest_data.py

You're also removing PCI autotest - commit message needs to call this 
out as well.

> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Reshma Pattan <reshma.pattan@intel.com>
> Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
> ---
>   test/test/autotest_data.py | 370 ++++++++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 362 insertions(+), 8 deletions(-)
> 
> diff --git a/test/test/autotest_data.py b/test/test/autotest_data.py

<snip>

> +    {
> +        "Name":    "Dump_ring",
> +        "Command": "dump_ring",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Quit",
> +        "Command": "quit",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },

Why is "quit" an autotest? If you want to test "quit" functionality, i 
would suggest putting it at the end of non-parallel tests, to avoid 
situation where test prematurely stops (for example, this will happen on 
FreeBSD, where there's only one thread executing these tests).

But really, having it in this list is IMO dubious.

> +    {
> +        "Name":    "Set rxtx mode",
> +        "Command": "set_rxtx_mode",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Set rxtx anchor",
> +        "Command": "set_rxtx_anchor",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Set rxtx sc",
> +        "Command": "set_rxtx_sc",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Event eth rx adapter autotest",
> +        "Command": "event_eth_rx_adapter_autotest",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Rawdev autotest",
> +        "Command": "rawdev_autotest",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Kvargs autotest",
> +        "Command": "kvargs_autotest",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Dump physmem",
> +        "Command": "dump_physmem",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Dump memzone",
> +        "Command": "dump_memzone",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },

Here and in other "dump" autotests - do these dump tests work as 
autotests? Default autotest function expects a "Test OK" at the end of 
test, and i don't think dump autotests return that, so this will 
(should?) cause a very long timeout and a test failure. These should 
have "dump_autotest" as "Func" value - that function correctly parses 
output of dump autotests (or, to be more accurate, it doesn't - as long 
as test doesn't crash, it's considered to be successful :) ).

Also, for readability, we should move these dump tests together and have 
all dump tests one after the other.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2 09/10] autotest: update result for skipped test cases

[Burakov, Anatoly]

On 13-Jul-18 5:20 PM, Reshma Pattan wrote:
> Fixed in autotest_test_funcs.py to handle test cases
> which returns "Skipped" as result.
> The issue was skipped test cases got timed out,
> causing delay in autotests execution.
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
> Signed-off-by: Reshma Pattan <reshma.pattan@intel.com>
> ---

Is this patch still applicable? Patch 5 in these series causes skipped 
tests to not get executed in the first place, so by the time we reach 
this patch it is no longer a problem, i think.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2 08/10] autotest: add new test cases to autotest

[Pattan, Reshma]

Hi,

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Friday, July 13, 2018 5:41 PM
> To: Pattan, Reshma <reshma.pattan@intel.com>; thomas@monjalon.net;
> dev@dpdk.org
> Cc: Parthasarathy, JananeeX M <jananeex.m.parthasarathy@intel.com>;
> stable@dpdk.org
> Subject: Re: [PATCH v2 08/10] autotest: add new test cases to autotest
> 
> On 13-Jul-18 5:20 PM, Reshma Pattan wrote:
> > Autotest is enhanced with additional test cases being added to
> > autotest_data.py
> 
> You're also removing PCI autotest - commit message needs to call this out as
> well.
> 

OK

> <snip>
> 
> > +    {
> > +        "Name":    "Dump_ring",
> > +        "Command": "dump_ring",
> > +        "Func":    default_autotest,
> > +        "Report":  None,
> > +    },
> > +    {
> > +        "Name":    "Quit",
> > +        "Command": "quit",
> > +        "Func":    default_autotest,
> > +        "Report":  None,
> > +    },
> 
> Why is "quit" an autotest? If you want to test "quit" functionality, i would
> suggest putting it at the end of non-parallel tests, to avoid situation where
> test prematurely stops (for example, this will happen on FreeBSD, where
> there's only one thread executing these tests).
> 
> But really, having it in this list is IMO dubious.
> 
Ok, I will remove this 

> Here and in other "dump" autotests - do these dump tests work as autotests?
> Default autotest function expects a "Test OK" at the end of test, and i don't
> think dump autotests return that, so this will
> (should?) cause a very long timeout and a test failure. These should have
> "dump_autotest" as "Func" value - that function correctly parses output of
> dump autotests (or, to be more accurate, it doesn't - as long as test doesn't
> crash, it's considered to be successful :) ).
> 
> Also, for readability, we should move these dump tests together and have all
> dump tests one after the other.
> 

Ok will add "Func" value  as dump_autotest and move all of the dump tests to end of  parallel list.

[dpdk-dev] [PATCH v3 0/9] Make unit tests great again

[Reshma Pattan]

Previously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.

In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.

This patchset merges changes done in [1], [2]

[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://patches.dpdk.org/patch/40373/

v3: Moved all dump tests together to the end of the parallel list.
    Updated the commit message about PCI auto test removal.
    Removed the unnecessary changes of the patch
    http://patches.dpdk.org/patch/40371/

Reshma Pattan (9):
  autotest: fix printing
  autotest: fix invalid code on reports
  autotest: make autotest runner python 2/3 compliant
  autotest: visually separate different test categories
  autotest: improve filtering
  autotest: remove autotest grouping
  autotest: properly parallelize unit tests
  autotest: update autotest test case list
  mk: update make targets for classified testcases

 mk/rte.sdkroot.mk            |    4 +-
 mk/rte.sdktest.mk            |   33 +-
 test/test/autotest.py        |   13 +-
 test/test/autotest_data.py   | 1099 +++++++++++++++++++++++++-----------------
 test/test/autotest_runner.py |  519 ++++++++++----------
 5 files changed, 966 insertions(+), 702 deletions(-)

-- 
2.14.4

[dpdk-dev] [PATCH v3 1/9] autotest: fix printing

[Reshma Pattan]

Previously, printing was done using tuple syntax, which caused
output to appear as a tuple as opposed to being one string. Fix
this by using addition operator instead.

Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index a692f0697..b09b57876 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -247,7 +247,7 @@ def __process_results(self, results):
 
             # don't print out total time every line, it's the same anyway
             if i == len(results) - 1:
-                print(result,
+                print(result +
                       "[%02dm %02ds]" % (total_time / 60, total_time % 60))
             else:
                 print(result)
@@ -332,8 +332,8 @@ def run_all_tests(self):
 
             # create table header
             print("")
-            print("Test name".ljust(30), "Test result".ljust(29),
-                  "Test".center(9), "Total".center(9))
+            print("Test name".ljust(30) + "Test result".ljust(29) +
+                  "Test".center(9) + "Total".center(9))
             print("=" * 80)
 
             # make a note of tests start time
-- 
2.14.4

[dpdk-dev] [PATCH v3 2/9] autotest: fix invalid code on reports

[Reshma Pattan]

There are no reports defined for any test, so this codepath was
never triggered, but it's still wrong because it's referencing
variables that aren't there. Fix it by passing target into the
test function, and reference correct log variable.

Fixes: e2cc79b75d9f ("app: rework autotest.py")
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index b09b57876..bdc32da5d 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -41,7 +41,7 @@ def wait_prompt(child):
 # quite a bit of effort to make it work).
 
 
-def run_test_group(cmdline, test_group):
+def run_test_group(cmdline, target, test_group):
     results = []
     child = None
     start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group):
             # make a note when the test was finished
             end_time = time.time()
 
+            log = logfile.getvalue()
+
             # append test data to the result tuple
-            result += (test["Name"], end_time - start_time,
-                       logfile.getvalue())
+            result += (test["Name"], end_time - start_time, log)
 
             # call report function, if any defined, and supply it with
             # target and complete log for test run
             if test["Report"]:
-                report = test["Report"](self.target, log)
+                report = test["Report"](target, log)
 
                 # append report to results tuple
                 result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self):
             for test_group in self.parallel_test_groups:
                 result = pool.apply_async(run_test_group,
                                           [self.__get_cmdline(test_group),
+                                           self.target,
                                            test_group])
                 results.append(result)
 
@@ -367,7 +369,7 @@ def run_all_tests(self):
             # run non_parallel tests. they are run one by one, synchronously
             for test_group in self.non_parallel_test_groups:
                 group_result = run_test_group(
-                    self.__get_cmdline(test_group), test_group)
+                    self.__get_cmdline(test_group), self.target, test_group)
 
                 self.__process_results(group_result)
 
-- 
2.14.4

[dpdk-dev] [PATCH v3 3/9] autotest: make autotest runner python 2/3 compliant

[Reshma Pattan]

Autotest runner was still using python 2-style print syntax. Fix
it by importing print function from the future, and fix the calls
to be python-3 style.

Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index bdc32da5d..f6b669a2e 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -3,6 +3,7 @@
 
 # The main logic behind running autotests in parallel
 
+from __future__ import print_function
 import StringIO
 import csv
 import multiprocessing
@@ -52,8 +53,8 @@ def run_test_group(cmdline, target, test_group):
         # prepare logging of init
         startuplog = StringIO.StringIO()
 
-        print >>startuplog, "\n%s %s\n" % ("=" * 20, test_group["Prefix"])
-        print >>startuplog, "\ncmdline=%s" % cmdline
+        print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog)
+        print("\ncmdline=%s" % cmdline, file=startuplog)
 
         child = pexpect.spawn(cmdline, logfile=startuplog)
 
@@ -117,7 +118,7 @@ def run_test_group(cmdline, target, test_group):
 
         try:
             # print test name to log buffer
-            print >>logfile, "\n%s %s\n" % ("-" * 20, test["Name"])
+            print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
             # run test function associated with the test
             if stripped or test["Command"] in avail_cmds:
-- 
2.14.4

[dpdk-dev] [PATCH v3 4/9] autotest: visually separate different test categories

[Reshma Pattan]

Help visually identify parallel vs. non-parallel autotests.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index f6b669a2e..d9d5f7a97 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -341,6 +341,7 @@ def run_all_tests(self):
             # make a note of tests start time
             self.start = time.time()
 
+            print("Parallel autotests:")
             # assign worker threads to run test groups
             for test_group in self.parallel_test_groups:
                 result = pool.apply_async(run_test_group,
@@ -367,6 +368,7 @@ def run_all_tests(self):
                     # remove result from results list once we're done with it
                     results.remove(group_result)
 
+            print("Non-parallel autotests:")
             # run non_parallel tests. they are run one by one, synchronously
             for test_group in self.non_parallel_test_groups:
                 group_result = run_test_group(
-- 
2.14.4

[dpdk-dev] [PATCH v3 5/9] autotest: improve filtering

[Reshma Pattan]

Improve code for filtering test groups. Also, move reading binary
symbols into filtering stage, so that tests that are meant to be
skipped are never attempted to be executed in the first place.
Before running tests, print out any tests that were skipped because
they weren't compiled.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 118 ++++++++++++++++++++++++-------------------
 1 file changed, 66 insertions(+), 52 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index d9d5f7a97..c98ec2a57 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -95,13 +95,6 @@ def run_test_group(cmdline, target, test_group):
     results.append((0, "Success", "Start %s" % test_group["Prefix"],
                     time.time() - start_time, startuplog.getvalue(), None))
 
-    # parse the binary for available test commands
-    binary = cmdline.split()[0]
-    stripped = 'not stripped' not in subprocess.check_output(['file', binary])
-    if not stripped:
-        symbols = subprocess.check_output(['nm', binary]).decode('utf-8')
-        avail_cmds = re.findall('test_register_(\w+)', symbols)
-
     # run all tests in test group
     for test in test_group["Tests"]:
 
@@ -121,10 +114,7 @@ def run_test_group(cmdline, target, test_group):
             print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
             # run test function associated with the test
-            if stripped or test["Command"] in avail_cmds:
-                result = test["Func"](child, test["Command"])
-            else:
-                result = (0, "Skipped [Not Available]")
+            result = test["Func"](child, test["Command"])
 
             # make a note when the test was finished
             end_time = time.time()
@@ -186,8 +176,10 @@ class AutotestRunner:
     def __init__(self, cmdline, target, blacklist, whitelist):
         self.cmdline = cmdline
         self.target = target
+        self.binary = cmdline.split()[0]
         self.blacklist = blacklist
         self.whitelist = whitelist
+        self.skipped = []
 
         # log file filename
         logfile = "%s.log" % target
@@ -276,53 +268,58 @@ def __process_results(self, results):
             if i != 0:
                 self.csvwriter.writerow([test_name, test_result, result_str])
 
-    # this function iterates over test groups and removes each
-    # test that is not in whitelist/blacklist
-    def __filter_groups(self, test_groups):
-        groups_to_remove = []
-
-        # filter out tests from parallel test groups
-        for i, test_group in enumerate(test_groups):
-
-            # iterate over a copy so that we could safely delete individual
-            # tests
-            for test in test_group["Tests"][:]:
-                test_id = test["Command"]
-
-                # dump tests are specified in full e.g. "Dump_mempool"
-                if "_autotest" in test_id:
-                    test_id = test_id[:-len("_autotest")]
-
-                # filter out blacklisted/whitelisted tests
-                if self.blacklist and test_id in self.blacklist:
-                    test_group["Tests"].remove(test)
-                    continue
-                if self.whitelist and test_id not in self.whitelist:
-                    test_group["Tests"].remove(test)
-                    continue
-
-            # modify or remove original group
-            if len(test_group["Tests"]) > 0:
-                test_groups[i] = test_group
-            else:
-                # remember which groups should be deleted
-                # put the numbers backwards so that we start
-                # deleting from the end, not from the beginning
-                groups_to_remove.insert(0, i)
+    # this function checks individual test and decides if this test should be in
+    # the group by comparing it against  whitelist/blacklist. it also checks if
+    # the test is compiled into the binary, and marks it as skipped if necessary
+    def __filter_test(self, test):
+        test_cmd = test["Command"]
+        test_id = test_cmd
+
+        # dump tests are specified in full e.g. "Dump_mempool"
+        if "_autotest" in test_id:
+            test_id = test_id[:-len("_autotest")]
+
+        # filter out blacklisted/whitelisted tests
+        if self.blacklist and test_id in self.blacklist:
+            return False
+        if self.whitelist and test_id not in self.whitelist:
+            return False
+
+        # if test wasn't compiled in, remove it as well
+
+        # parse the binary for available test commands
+        stripped = 'not stripped' not in \
+                   subprocess.check_output(['file', self.binary])
+        if not stripped:
+            symbols = subprocess.check_output(['nm',
+                                               self.binary]).decode('utf-8')
+            avail_cmds = re.findall('test_register_(\w+)', symbols)
+
+            if test_cmd not in avail_cmds:
+                # notify user
+                result = 0, "Skipped [Not compiled]", test_id, 0, "", None
+                self.skipped.append(tuple(result))
+                return False
 
-        # remove test groups that need to be removed
-        for i in groups_to_remove:
-            del test_groups[i]
+        return True
 
-        return test_groups
+    def __filter_group(self, group):
+        group["Tests"] = list(filter(self.__filter_test, group["Tests"]))
+        return len(group["Tests"]) > 0
 
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
-        self.parallel_test_groups = \
-            self.__filter_groups(self.parallel_test_groups)
-        self.non_parallel_test_groups = \
-            self.__filter_groups(self.non_parallel_test_groups)
+        # for each test group, check all tests against the filter, then remove
+        # all groups that don't have any tests
+        self.parallel_test_groups = list(
+            filter(self.__filter_group,
+                   self.parallel_test_groups)
+        )
+        self.non_parallel_test_groups = list(
+            filter(self.__filter_group,
+                   self.non_parallel_test_groups)
+        )
 
         # create a pool of worker threads
         pool = multiprocessing.Pool(processes=1)
@@ -338,6 +335,23 @@ def run_all_tests(self):
                   "Test".center(9) + "Total".center(9))
             print("=" * 80)
 
+            # print out skipped autotests if there were any
+            if len(self.skipped):
+                print("Skipped autotests:")
+
+                # print out any skipped tests
+                for result in self.skipped:
+                    # unpack result tuple
+                    test_result, result_str, test_name, _, _, _ = result
+                    self.csvwriter.writerow([test_name, test_result,
+                                             result_str])
+
+                    t = ("%s:" % test_name).ljust(30)
+                    t += result_str.ljust(29)
+                    t += "[00m 00s]"
+
+                    print(t)
+
             # make a note of tests start time
             self.start = time.time()
 
-- 
2.14.4

[dpdk-dev] [PATCH v3 6/9] autotest: remove autotest grouping

[Reshma Pattan]

Previously, all autotests were grouped into (seemingly arbitrary)
groups. The goal was to run all tests in parallel (so that autotest
finishes faster), but we couldn't just do it willy-nilly because
DPDK couldn't allocate and free hugepages on-demand, so we had to
find autotest groupings that could work memory-wise and still be
fast enough to not hold up shorter tests. The inflexibility of
memory subsystem has now been fixed for 18.05, so grouping
autotests is no longer necessary.

Thus, this commit moves all autotests into two groups -
parallel(izable) autotests, and non-arallel(izable) autotests
(typically performance tests). Note that this particular commit
makes running autotests dog slow because while the tests are now
in a single group, the test function itself hasn't changed much,
so all autotests are now run one-by-one, starting and stopping
the DPDK test application.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest.py        |   7 +-
 test/test/autotest_data.py   | 749 +++++++++++++++++--------------------------
 test/test/autotest_runner.py | 271 ++++++----------
 3 files changed, 408 insertions(+), 619 deletions(-)

diff --git a/test/test/autotest.py b/test/test/autotest.py
index 1cfd8cf22..ae27daef7 100644
--- a/test/test/autotest.py
+++ b/test/test/autotest.py
@@ -39,11 +39,8 @@ def usage():
 runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist,
                                         test_whitelist)
 
-for test_group in autotest_data.parallel_test_group_list:
-    runner.add_parallel_test_group(test_group)
-
-for test_group in autotest_data.non_parallel_test_group_list:
-    runner.add_non_parallel_test_group(test_group)
+runner.parallel_tests = autotest_data.parallel_test_list[:]
+runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
 
 num_fails = runner.run_all_tests()
 
diff --git a/test/test/autotest_data.py b/test/test/autotest_data.py
index aacfe0a66..c24e7bc25 100644
--- a/test/test/autotest_data.py
+++ b/test/test/autotest_data.py
@@ -3,465 +3,322 @@
 
 # Test data for autotests
 
-from glob import glob
 from autotest_test_funcs import *
 
-
-# quick and dirty function to find out number of sockets
-def num_sockets():
-    result = len(glob("/sys/devices/system/node/node*"))
-    if result == 0:
-        return 1
-    return result
-
-
-# Assign given number to each socket
-# e.g. 32 becomes 32,32 or 32,32,32,32
-def per_sockets(num):
-    return ",".join([str(num)] * num_sockets())
-
 # groups of tests that can be run in parallel
 # the grouping has been found largely empirically
-parallel_test_group_list = [
-    {
-        "Prefix":    "group_1",
-        "Memory":    per_sockets(8),
-        "Tests":
-        [
-            {
-                "Name":    "Cycles autotest",
-                "Command": "cycles_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Timer autotest",
-                "Command": "timer_autotest",
-                "Func":    timer_autotest,
-                "Report":   None,
-            },
-            {
-                "Name":    "Debug autotest",
-                "Command": "debug_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Errno autotest",
-                "Command": "errno_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Meter autotest",
-                "Command": "meter_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Common autotest",
-                "Command": "common_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Resource autotest",
-                "Command": "resource_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_2",
-        "Memory":    "16",
-        "Tests":
-        [
-            {
-                "Name":    "Memory autotest",
-                "Command": "memory_autotest",
-                "Func":    memory_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Read/write lock autotest",
-                "Command": "rwlock_autotest",
-                "Func":    rwlock_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Logs autotest",
-                "Command": "logs_autotest",
-                "Func":    logs_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "CPU flags autotest",
-                "Command": "cpuflags_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Version autotest",
-                "Command": "version_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "EAL filesystem autotest",
-                "Command": "eal_fs_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "EAL flags autotest",
-                "Command": "eal_flags_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Hash autotest",
-                "Command": "hash_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ],
-    },
-    {
-        "Prefix":    "group_3",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "LPM autotest",
-                "Command": "lpm_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "LPM6 autotest",
-                "Command": "lpm6_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Memcpy autotest",
-                "Command": "memcpy_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Memzone autotest",
-                "Command": "memzone_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "String autotest",
-                "Command": "string_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Alarm autotest",
-                "Command": "alarm_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_4",
-        "Memory":    per_sockets(128),
-        "Tests":
-        [
-            {
-                "Name":    "PCI autotest",
-                "Command": "pci_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Malloc autotest",
-                "Command": "malloc_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Multi-process autotest",
-                "Command": "multiprocess_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Mbuf autotest",
-                "Command": "mbuf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Per-lcore autotest",
-                "Command": "per_lcore_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Ring autotest",
-                "Command": "ring_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_5",
-        "Memory":    "32",
-        "Tests":
-        [
-            {
-                "Name":    "Spinlock autotest",
-                "Command": "spinlock_autotest",
-                "Func":    spinlock_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Byte order autotest",
-                "Command": "byteorder_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "TAILQ autotest",
-                "Command": "tailq_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Command-line autotest",
-                "Command": "cmdline_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Interrupts autotest",
-                "Command": "interrupt_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_6",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Function reentrancy autotest",
-                "Command": "func_reentrancy_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Mempool autotest",
-                "Command": "mempool_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Atomics autotest",
-                "Command": "atomic_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Prefetch autotest",
-                "Command": "prefetch_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Red autotest",
-                "Command": "red_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_7",
-        "Memory":    "64",
-        "Tests":
-        [
-            {
-                "Name":    "PMD ring autotest",
-                "Command": "ring_pmd_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Access list control autotest",
-                "Command": "acl_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Sched autotest",
-                "Command": "sched_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+parallel_test_list = [
+    {
+        "Name":    "Cycles autotest",
+        "Command": "cycles_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Timer autotest",
+        "Command": "timer_autotest",
+        "Func":    timer_autotest,
+        "Report":   None,
+    },
+    {
+        "Name":    "Debug autotest",
+        "Command": "debug_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Errno autotest",
+        "Command": "errno_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Meter autotest",
+        "Command": "meter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Common autotest",
+        "Command": "common_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Resource autotest",
+        "Command": "resource_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memory autotest",
+        "Command": "memory_autotest",
+        "Func":    memory_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Read/write lock autotest",
+        "Command": "rwlock_autotest",
+        "Func":    rwlock_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Logs autotest",
+        "Command": "logs_autotest",
+        "Func":    logs_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "CPU flags autotest",
+        "Command": "cpuflags_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Version autotest",
+        "Command": "version_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "EAL filesystem autotest",
+        "Command": "eal_fs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "EAL flags autotest",
+        "Command": "eal_flags_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash autotest",
+        "Command": "hash_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "LPM autotest",
+        "Command": "lpm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "LPM6 autotest",
+        "Command": "lpm6_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memcpy autotest",
+        "Command": "memcpy_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memzone autotest",
+        "Command": "memzone_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "String autotest",
+        "Command": "string_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Alarm autotest",
+        "Command": "alarm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "PCI autotest",
+        "Command": "pci_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Malloc autotest",
+        "Command": "malloc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Multi-process autotest",
+        "Command": "multiprocess_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mbuf autotest",
+        "Command": "mbuf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Per-lcore autotest",
+        "Command": "per_lcore_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Ring autotest",
+        "Command": "ring_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Spinlock autotest",
+        "Command": "spinlock_autotest",
+        "Func":    spinlock_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Byte order autotest",
+        "Command": "byteorder_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "TAILQ autotest",
+        "Command": "tailq_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Command-line autotest",
+        "Command": "cmdline_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Interrupts autotest",
+        "Command": "interrupt_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Function reentrancy autotest",
+        "Command": "func_reentrancy_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mempool autotest",
+        "Command": "mempool_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Atomics autotest",
+        "Command": "atomic_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Prefetch autotest",
+        "Command": "prefetch_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red autotest",
+        "Command": "red_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "PMD ring autotest",
+        "Command": "ring_pmd_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Access list control autotest",
+        "Command": "acl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Sched autotest",
+        "Command": "sched_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
 ]
 
 # tests that should not be run when any other tests are running
-non_parallel_test_group_list = [
-
+non_parallel_test_list = [
     {
-        "Prefix":    "eventdev",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "Eventdev common autotest",
-                "Command": "eventdev_common_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "eventdev_sw",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "Eventdev sw autotest",
-                "Command": "eventdev_sw_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "kni",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "KNI autotest",
-                "Command": "kni_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "mempool_perf",
-        "Memory":    per_sockets(256),
-        "Tests":
-        [
-            {
-                "Name":    "Mempool performance autotest",
-                "Command": "mempool_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "memcpy_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Memcpy performance autotest",
-                "Command": "memcpy_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "hash_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Hash performance autotest",
-                "Command": "hash_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power autotest",
-                "Command":    "power_autotest",
-                "Func":       default_autotest,
-                "Report":      None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power_acpi_cpufreq",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power ACPI cpufreq autotest",
-                "Command":    "power_acpi_cpufreq_autotest",
-                "Func":       default_autotest,
-                "Report":     None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power_kvm_vm",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power KVM VM  autotest",
-                "Command":    "power_kvm_vm_autotest",
-                "Func":       default_autotest,
-                "Report":     None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "timer_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Timer performance autotest",
-                "Command": "timer_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+        "Name":    "Eventdev common autotest",
+        "Command": "eventdev_common_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Eventdev sw autotest",
+        "Command": "eventdev_sw_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "KNI autotest",
+        "Command": "kni_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mempool performance autotest",
+        "Command": "mempool_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memcpy performance autotest",
+        "Command": "memcpy_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash performance autotest",
+        "Command": "hash_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":       "Power autotest",
+        "Command":    "power_autotest",
+        "Func":       default_autotest,
+        "Report":      None,
+    },
+    {
+        "Name":       "Power ACPI cpufreq autotest",
+        "Command":    "power_acpi_cpufreq_autotest",
+        "Func":       default_autotest,
+        "Report":     None,
+    },
+    {
+        "Name":       "Power KVM VM  autotest",
+        "Command":    "power_kvm_vm_autotest",
+        "Func":       default_autotest,
+        "Report":     None,
+    },
+    {
+        "Name":    "Timer performance autotest",
+        "Command": "timer_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
-
     #
     # Please always make sure that ring_perf is the last test!
     #
     {
-        "Prefix":    "ring_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Ring performance autotest",
-                "Command": "ring_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+        "Name":    "Ring performance autotest",
+        "Command": "ring_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
 ]
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index c98ec2a57..d6ae57e76 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -42,18 +42,16 @@ def wait_prompt(child):
 # quite a bit of effort to make it work).
 
 
-def run_test_group(cmdline, target, test_group):
-    results = []
-    child = None
+def run_test_group(cmdline, prefix, target, test):
     start_time = time.time()
-    startuplog = None
+
+    # prepare logging of init
+    startuplog = StringIO.StringIO()
 
     # run test app
     try:
-        # prepare logging of init
-        startuplog = StringIO.StringIO()
 
-        print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog)
+        print("\n%s %s\n" % ("=" * 20, prefix), file=startuplog)
         print("\ncmdline=%s" % cmdline, file=startuplog)
 
         child = pexpect.spawn(cmdline, logfile=startuplog)
@@ -62,88 +60,54 @@ def run_test_group(cmdline, target, test_group):
         if not wait_prompt(child):
             child.close()
 
-            results.append((-1,
-                            "Fail [No prompt]",
-                            "Start %s" % test_group["Prefix"],
-                            time.time() - start_time,
-                            startuplog.getvalue(),
-                            None))
-
-            # mark all tests as failed
-            for test in test_group["Tests"]:
-                results.append((-1, "Fail [No prompt]", test["Name"],
-                                time.time() - start_time, "", None))
-            # exit test
-            return results
+            return -1, "Fail [No prompt]", "Start %s" % prefix,\
+                   time.time() - start_time, startuplog.getvalue(), None
 
     except:
-        results.append((-1,
-                        "Fail [Can't run]",
-                        "Start %s" % test_group["Prefix"],
-                        time.time() - start_time,
-                        startuplog.getvalue(),
-                        None))
-
-        # mark all tests as failed
-        for t in test_group["Tests"]:
-            results.append((-1, "Fail [Can't run]", t["Name"],
-                            time.time() - start_time, "", None))
-        # exit test
-        return results
-
-    # startup was successful
-    results.append((0, "Success", "Start %s" % test_group["Prefix"],
-                    time.time() - start_time, startuplog.getvalue(), None))
-
-    # run all tests in test group
-    for test in test_group["Tests"]:
-
-        # create log buffer for each test
-        # in multiprocessing environment, the logging would be
-        # interleaved and will create a mess, hence the buffering
-        logfile = StringIO.StringIO()
-        child.logfile = logfile
-
-        result = ()
-
-        # make a note when the test started
-        start_time = time.time()
+        return -1, "Fail [Can't run]", "Start %s" % prefix,\
+               time.time() - start_time, startuplog.getvalue(), None
 
-        try:
-            # print test name to log buffer
-            print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
+    # create log buffer for each test
+    # in multiprocessing environment, the logging would be
+    # interleaved and will create a mess, hence the buffering
+    logfile = StringIO.StringIO()
+    child.logfile = logfile
 
-            # run test function associated with the test
-            result = test["Func"](child, test["Command"])
+    # make a note when the test started
+    start_time = time.time()
 
-            # make a note when the test was finished
-            end_time = time.time()
+    try:
+        # print test name to log buffer
+        print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
-            log = logfile.getvalue()
+        # run test function associated with the test
+        result = test["Func"](child, test["Command"])
 
-            # append test data to the result tuple
-            result += (test["Name"], end_time - start_time, log)
+        # make a note when the test was finished
+        end_time = time.time()
 
-            # call report function, if any defined, and supply it with
-            # target and complete log for test run
-            if test["Report"]:
-                report = test["Report"](target, log)
+        log = logfile.getvalue()
 
-                # append report to results tuple
-                result += (report,)
-            else:
-                # report is None
-                result += (None,)
-        except:
-            # make a note when the test crashed
-            end_time = time.time()
+        # append test data to the result tuple
+        result += (test["Name"], end_time - start_time, log)
+
+        # call report function, if any defined, and supply it with
+        # target and complete log for test run
+        if test["Report"]:
+            report = test["Report"](target, log)
+
+            # append report to results tuple
+            result += (report,)
+        else:
+            # report is None
+            result += (None,)
+    except:
+        # make a note when the test crashed
+        end_time = time.time()
 
-            # mark test as failed
-            result = (-1, "Fail [Crash]", test["Name"],
-                      end_time - start_time, logfile.getvalue(), None)
-        finally:
-            # append the results to the results list
-            results.append(result)
+        # mark test as failed
+        result = (-1, "Fail [Crash]", test["Name"],
+                  end_time - start_time, logfile.getvalue(), None)
 
     # regardless of whether test has crashed, try quitting it
     try:
@@ -155,7 +119,7 @@ def run_test_group(cmdline, target, test_group):
         pass
 
     # return test results
-    return results
+    return result
 
 
 # class representing an instance of autotests run
@@ -180,6 +144,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.blacklist = blacklist
         self.whitelist = whitelist
         self.skipped = []
+        self.parallel_tests = []
+        self.non_parallel_tests = []
 
         # log file filename
         logfile = "%s.log" % target
@@ -193,80 +159,52 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.csvwriter.writerow(["test_name", "test_result", "result_str"])
 
     # set up cmdline string
-    def __get_cmdline(self, test):
+    def __get_cmdline(self):
         cmdline = self.cmdline
 
-        # append memory limitations for each test
-        # otherwise tests won't run in parallel
-        if "i686" not in self.target:
-            cmdline += " --socket-mem=%s" % test["Memory"]
-        else:
-            # affinitize startup so that tests don't fail on i686
-            cmdline = "taskset 1 " + cmdline
-            cmdline += " -m " + str(sum(map(int, test["Memory"].split(","))))
-
-        # set group prefix for autotest group
-        # otherwise they won't run in parallel
-        cmdline += " --file-prefix=%s" % test["Prefix"]
+        # affinitize startup so that tests don't fail on i686
+        cmdline = "taskset 1 " + cmdline
 
         return cmdline
 
-    def add_parallel_test_group(self, test_group):
-        self.parallel_test_groups.append(test_group)
+    def __process_result(self, result):
 
-    def add_non_parallel_test_group(self, test_group):
-        self.non_parallel_test_groups.append(test_group)
+        # unpack result tuple
+        test_result, result_str, test_name, \
+            test_time, log, report = result
 
-    def __process_results(self, results):
-        # this iterates over individual test results
-        for i, result in enumerate(results):
+        # get total run time
+        cur_time = time.time()
+        total_time = int(cur_time - self.start)
 
-            # increase total number of tests that were run
-            # do not include "start" test
-            if i > 0:
-                self.n_tests += 1
+        # print results, test run time and total time since start
+        result = ("%s:" % test_name).ljust(30)
+        result += result_str.ljust(29)
+        result += "[%02dm %02ds]" % (test_time / 60, test_time % 60)
 
-            # unpack result tuple
-            test_result, result_str, test_name, \
-                test_time, log, report = result
+        # don't print out total time every line, it's the same anyway
+        print(result + "[%02dm %02ds]" % (total_time / 60, total_time % 60))
 
-            # get total run time
-            cur_time = time.time()
-            total_time = int(cur_time - self.start)
+        # if test failed and it wasn't a "start" test
+        if test_result < 0:
+            self.fails += 1
 
-            # print results, test run time and total time since start
-            result = ("%s:" % test_name).ljust(30)
-            result += result_str.ljust(29)
-            result += "[%02dm %02ds]" % (test_time / 60, test_time % 60)
+        # collect logs
+        self.log_buffers.append(log)
 
-            # don't print out total time every line, it's the same anyway
-            if i == len(results) - 1:
-                print(result +
-                      "[%02dm %02ds]" % (total_time / 60, total_time % 60))
+        # create report if it exists
+        if report:
+            try:
+                f = open("%s_%s_report.rst" %
+                         (self.target, test_name), "w")
+            except IOError:
+                print("Report for %s could not be created!" % test_name)
             else:
-                print(result)
-
-            # if test failed and it wasn't a "start" test
-            if test_result < 0 and not i == 0:
-                self.fails += 1
-
-            # collect logs
-            self.log_buffers.append(log)
-
-            # create report if it exists
-            if report:
-                try:
-                    f = open("%s_%s_report.rst" %
-                             (self.target, test_name), "w")
-                except IOError:
-                    print("Report for %s could not be created!" % test_name)
-                else:
-                    with f:
-                        f.write(report)
-
-            # write test result to CSV file
-            if i != 0:
-                self.csvwriter.writerow([test_name, test_result, result_str])
+                with f:
+                    f.write(report)
+
+        # write test result to CSV file
+        self.csvwriter.writerow([test_name, test_result, result_str])
 
     # this function checks individual test and decides if this test should be in
     # the group by comparing it against  whitelist/blacklist. it also checks if
@@ -303,22 +241,16 @@ def __filter_test(self, test):
 
         return True
 
-    def __filter_group(self, group):
-        group["Tests"] = list(filter(self.__filter_test, group["Tests"]))
-        return len(group["Tests"]) > 0
-
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
-        # for each test group, check all tests against the filter, then remove
-        # all groups that don't have any tests
-        self.parallel_test_groups = list(
-            filter(self.__filter_group,
-                   self.parallel_test_groups)
+        self.parallel_tests = list(
+            filter(self.__filter_test,
+                   self.parallel_tests)
         )
-        self.non_parallel_test_groups = list(
-            filter(self.__filter_group,
-                   self.non_parallel_test_groups)
+        self.non_parallel_tests = list(
+            filter(self.__filter_test,
+                   self.non_parallel_tests)
         )
 
         # create a pool of worker threads
@@ -355,14 +287,16 @@ def run_all_tests(self):
             # make a note of tests start time
             self.start = time.time()
 
-            print("Parallel autotests:")
-            # assign worker threads to run test groups
-            for test_group in self.parallel_test_groups:
-                result = pool.apply_async(run_test_group,
-                                          [self.__get_cmdline(test_group),
-                                           self.target,
-                                           test_group])
-                results.append(result)
+            if len(self.parallel_tests) > 0:
+                print("Parallel autotests:")
+                # assign worker threads to run test groups
+                for test_group in self.parallel_tests:
+                    result = pool.apply_async(run_test_group,
+                                              [self.__get_cmdline(),
+                                               "",
+                                               self.target,
+                                               test_group])
+                    results.append(result)
 
             # iterate while we have group execution results to get
             while len(results) > 0:
@@ -377,18 +311,19 @@ def run_all_tests(self):
 
                     res = group_result.get()
 
-                    self.__process_results(res)
+                    self.__process_result(res)
 
                     # remove result from results list once we're done with it
                     results.remove(group_result)
 
-            print("Non-parallel autotests:")
-            # run non_parallel tests. they are run one by one, synchronously
-            for test_group in self.non_parallel_test_groups:
-                group_result = run_test_group(
-                    self.__get_cmdline(test_group), self.target, test_group)
+            if len(self.non_parallel_tests) > 0:
+                print("Non-parallel autotests:")
+                # run non_parallel tests. they are run one by one, synchronously
+                for test_group in self.non_parallel_tests:
+                    group_result = run_test_group(
+                        self.__get_cmdline(), "", self.target, test_group)
 
-                self.__process_results(group_result)
+                    self.__process_result(group_result)
 
             # get total run time
             cur_time = time.time()
-- 
2.14.4

[dpdk-dev] [PATCH v3 7/9] autotest: properly parallelize unit tests

[Reshma Pattan]

Now that everything else is in place, we can run unit tests in a
different fashion to what they were running as before. Previously,
we had all autotests as part of groups (largely obtained through
trial and error) to ensure parallel execution while still limiting
amounts of memory used by those tests.

This is no longer necessary, and as of previous commit, all tests
are now in the same group (still broken into two categories). They
still run one-by-one though. Fix this by initializing child
processes in multiprocessing Pool initialization, and putting all
tests on the queue, so that tests are executed by the first idle
worker. Tests are also affinitized to different NUMA nodes using
taskset in a round-robin fashion, to prevent over-exhausting
memory on any given NUMA node.

Non-parallel tests are executed in similar fashion, but on a
separate queue which will have only one pool worker, ensuring
non-parallel execution.

Support for FreeBSD is also added to ensure that on FreeBSD, all
tests are run sequentially even for the parallel section.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest.py        |   6 +-
 test/test/autotest_runner.py | 277 +++++++++++++++++++++++++++----------------
 2 files changed, 183 insertions(+), 100 deletions(-)

diff --git a/test/test/autotest.py b/test/test/autotest.py
index ae27daef7..12997fdf0 100644
--- a/test/test/autotest.py
+++ b/test/test/autotest.py
@@ -36,8 +36,12 @@ def usage():
 
 print(cmdline)
 
+# how many workers to run tests with. FreeBSD doesn't support multiple primary
+# processes, so make it 1, otherwise make it 4. ignored for non-parallel tests
+n_processes = 1 if "bsdapp" in target else 4
+
 runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist,
-                                        test_whitelist)
+                                        test_whitelist, n_processes)
 
 runner.parallel_tests = autotest_data.parallel_test_list[:]
 runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index d6ae57e76..36941a40a 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -6,16 +6,16 @@
 from __future__ import print_function
 import StringIO
 import csv
-import multiprocessing
+from multiprocessing import Pool, Queue
 import pexpect
 import re
 import subprocess
 import sys
 import time
+import glob
+import os
 
 # wait for prompt
-
-
 def wait_prompt(child):
     try:
         child.sendline()
@@ -28,22 +28,47 @@ def wait_prompt(child):
     else:
         return False
 
-# run a test group
-# each result tuple in results list consists of:
-#   result value (0 or -1)
-#   result string
-#   test name
-#   total test run time (double)
-#   raw test log
-#   test report (if not available, should be None)
-#
-# this function needs to be outside AutotestRunner class
-# because otherwise Pool won't work (or rather it will require
-# quite a bit of effort to make it work).
+
+# get all valid NUMA nodes
+def get_numa_nodes():
+    return [
+        int(
+            re.match(r"node(\d+)", os.path.basename(node))
+            .group(1)
+        )
+        for node in glob.glob("/sys/devices/system/node/node*")
+    ]
+
+
+# find first (or any, really) CPU on a particular node, will be used to spread
+# processes around NUMA nodes to avoid exhausting memory on particular node
+def first_cpu_on_node(node_nr):
+    cpu_path = glob.glob("/sys/devices/system/node/node%d/cpu*" % node_nr)[0]
+    cpu_name = os.path.basename(cpu_path)
+    m = re.match(r"cpu(\d+)", cpu_name)
+    return int(m.group(1))
+
+
+pool_child = None  # per-process child
 
 
-def run_test_group(cmdline, prefix, target, test):
+# we initialize each worker with a queue because we need per-pool unique
+# command-line arguments, but we cannot do different arguments in an initializer
+# because the API doesn't allow per-worker initializer arguments. so, instead,
+# we will initialize with a shared queue, and dequeue command-line arguments
+# from this queue
+def pool_init(queue, result_queue):
+    global pool_child
+
+    cmdline, prefix = queue.get()
     start_time = time.time()
+    name = ("Start %s" % prefix) if prefix != "" else "Start"
+
+    # use default prefix if no prefix was specified
+    prefix_cmdline = "--file-prefix=%s" % prefix if prefix != "" else ""
+
+    # append prefix to cmdline
+    cmdline = "%s %s" % (cmdline, prefix_cmdline)
 
     # prepare logging of init
     startuplog = StringIO.StringIO()
@@ -54,24 +79,60 @@ def run_test_group(cmdline, prefix, target, test):
         print("\n%s %s\n" % ("=" * 20, prefix), file=startuplog)
         print("\ncmdline=%s" % cmdline, file=startuplog)
 
-        child = pexpect.spawn(cmdline, logfile=startuplog)
+        pool_child = pexpect.spawn(cmdline, logfile=startuplog)
 
         # wait for target to boot
-        if not wait_prompt(child):
-            child.close()
+        if not wait_prompt(pool_child):
+            pool_child.close()
+
+            result = tuple((-1,
+                            "Fail [No prompt]",
+                            name,
+                            time.time() - start_time,
+                            startuplog.getvalue(),
+                            None))
+            pool_child = None
+        else:
+            result = tuple((0,
+                            "Success",
+                            name,
+                            time.time() - start_time,
+                            startuplog.getvalue(),
+                            None))
+    except:
+        result = tuple((-1,
+                        "Fail [Can't run]",
+                        name,
+                        time.time() - start_time,
+                        startuplog.getvalue(),
+                        None))
+        pool_child = None
 
-            return -1, "Fail [No prompt]", "Start %s" % prefix,\
-                   time.time() - start_time, startuplog.getvalue(), None
+    result_queue.put(result)
 
-    except:
-        return -1, "Fail [Can't run]", "Start %s" % prefix,\
-               time.time() - start_time, startuplog.getvalue(), None
+
+# run a test
+# each result tuple in results list consists of:
+#   result value (0 or -1)
+#   result string
+#   test name
+#   total test run time (double)
+#   raw test log
+#   test report (if not available, should be None)
+#
+# this function needs to be outside AutotestRunner class because otherwise Pool
+# won't work (or rather it will require quite a bit of effort to make it work).
+def run_test(target, test):
+    global pool_child
+
+    if pool_child is None:
+        return -1, "Fail [No test process]", test["Name"], 0, "", None
 
     # create log buffer for each test
     # in multiprocessing environment, the logging would be
     # interleaved and will create a mess, hence the buffering
     logfile = StringIO.StringIO()
-    child.logfile = logfile
+    pool_child.logfile = logfile
 
     # make a note when the test started
     start_time = time.time()
@@ -81,7 +142,7 @@ def run_test_group(cmdline, prefix, target, test):
         print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
         # run test function associated with the test
-        result = test["Func"](child, test["Command"])
+        result = test["Func"](pool_child, test["Command"])
 
         # make a note when the test was finished
         end_time = time.time()
@@ -109,15 +170,6 @@ def run_test_group(cmdline, prefix, target, test):
         result = (-1, "Fail [Crash]", test["Name"],
                   end_time - start_time, logfile.getvalue(), None)
 
-    # regardless of whether test has crashed, try quitting it
-    try:
-        child.sendline("quit")
-        child.close()
-    # if the test crashed, just do nothing instead
-    except:
-        # nop
-        pass
-
     # return test results
     return result
 
@@ -137,7 +189,7 @@ class AutotestRunner:
     blacklist = []
     whitelist = []
 
-    def __init__(self, cmdline, target, blacklist, whitelist):
+    def __init__(self, cmdline, target, blacklist, whitelist, n_processes):
         self.cmdline = cmdline
         self.target = target
         self.binary = cmdline.split()[0]
@@ -146,6 +198,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.skipped = []
         self.parallel_tests = []
         self.non_parallel_tests = []
+        self.n_processes = n_processes
+        self.active_processes = 0
 
         # log file filename
         logfile = "%s.log" % target
@@ -159,11 +213,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.csvwriter.writerow(["test_name", "test_result", "result_str"])
 
     # set up cmdline string
-    def __get_cmdline(self):
-        cmdline = self.cmdline
-
-        # affinitize startup so that tests don't fail on i686
-        cmdline = "taskset 1 " + cmdline
+    def __get_cmdline(self, cpu_nr):
+        cmdline = ("taskset -c %i " % cpu_nr) + self.cmdline
 
         return cmdline
 
@@ -241,6 +292,51 @@ def __filter_test(self, test):
 
         return True
 
+    def __run_test_group(self, test_group, worker_cmdlines):
+        group_queue = Queue()
+        init_result_queue = Queue()
+        for proc, cmdline in enumerate(worker_cmdlines):
+            prefix = "test%i" % proc if len(worker_cmdlines) > 1 else ""
+            group_queue.put(tuple((cmdline, prefix)))
+
+        # create a pool of worker threads
+        # we will initialize child in the initializer, and we don't need to
+        # close the child because when the pool worker gets destroyed, child
+        # closes the process
+        pool = Pool(processes=len(worker_cmdlines),
+                    initializer=pool_init,
+                    initargs=(group_queue, init_result_queue))
+
+        results = []
+
+        # process all initialization results
+        for _ in range(len(worker_cmdlines)):
+            self.__process_result(init_result_queue.get())
+
+        # run all tests asynchronously
+        for test in test_group:
+            result = pool.apply_async(run_test, (self.target, test))
+            results.append(result)
+
+        # tell the pool to stop all processes once done
+        pool.close()
+
+        # iterate while we have group execution results to get
+        while len(results) > 0:
+            # iterate over a copy to be able to safely delete results
+            # this iterates over a list of group results
+            for async_result in results[:]:
+                # if the thread hasn't finished yet, continue
+                if not async_result.ready():
+                    continue
+
+                res = async_result.get()
+
+                self.__process_result(res)
+
+                # remove result from results list once we're done with it
+                results.remove(async_result)
+
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
@@ -253,77 +349,60 @@ def run_all_tests(self):
                    self.non_parallel_tests)
         )
 
-        # create a pool of worker threads
-        pool = multiprocessing.Pool(processes=1)
-
-        results = []
+        parallel_cmdlines = []
+        # FreeBSD doesn't have NUMA support
+        numa_nodes = get_numa_nodes()
+        if len(numa_nodes) > 0:
+            for proc in range(self.n_processes):
+                # spread cpu affinity between NUMA nodes to have less chance of
+                # running out of memory while running multiple test apps in
+                # parallel. to do that, alternate between NUMA nodes in a round
+                # robin fashion, and pick an arbitrary CPU from that node to
+                # taskset our execution to
+                numa_node = numa_nodes[self.active_processes % len(numa_nodes)]
+                cpu_nr = first_cpu_on_node(numa_node)
+                parallel_cmdlines += [self.__get_cmdline(cpu_nr)]
+                # increase number of active processes so that the next cmdline
+                # gets a different NUMA node
+                self.active_processes += 1
+        else:
+            parallel_cmdlines = [self.cmdline] * self.n_processes
 
-        # whatever happens, try to save as much logs as possible
-        try:
+        print("Running tests with %d workers" % self.n_processes)
 
-            # create table header
-            print("")
-            print("Test name".ljust(30) + "Test result".ljust(29) +
-                  "Test".center(9) + "Total".center(9))
-            print("=" * 80)
+        # create table header
+        print("")
+        print("Test name".ljust(30) + "Test result".ljust(29) +
+              "Test".center(9) + "Total".center(9))
+        print("=" * 80)
 
-            # print out skipped autotests if there were any
-            if len(self.skipped):
-                print("Skipped autotests:")
+        if len(self.skipped):
+            print("Skipped autotests:")
 
-                # print out any skipped tests
-                for result in self.skipped:
-                    # unpack result tuple
-                    test_result, result_str, test_name, _, _, _ = result
-                    self.csvwriter.writerow([test_name, test_result,
-                                             result_str])
+            # print out any skipped tests
+            for result in self.skipped:
+                # unpack result tuple
+                test_result, result_str, test_name, _, _, _ = result
+                self.csvwriter.writerow([test_name, test_result, result_str])
 
-                    t = ("%s:" % test_name).ljust(30)
-                    t += result_str.ljust(29)
-                    t += "[00m 00s]"
+                t = ("%s:" % test_name).ljust(30)
+                t += result_str.ljust(29)
+                t += "[00m 00s]"
 
-                    print(t)
+                print(t)
 
-            # make a note of tests start time
-            self.start = time.time()
+        # make a note of tests start time
+        self.start = time.time()
 
+        # whatever happens, try to save as much logs as possible
+        try:
             if len(self.parallel_tests) > 0:
                 print("Parallel autotests:")
-                # assign worker threads to run test groups
-                for test_group in self.parallel_tests:
-                    result = pool.apply_async(run_test_group,
-                                              [self.__get_cmdline(),
-                                               "",
-                                               self.target,
-                                               test_group])
-                    results.append(result)
-
-            # iterate while we have group execution results to get
-            while len(results) > 0:
-
-                # iterate over a copy to be able to safely delete results
-                # this iterates over a list of group results
-                for group_result in results[:]:
-
-                    # if the thread hasn't finished yet, continue
-                    if not group_result.ready():
-                        continue
-
-                    res = group_result.get()
-
-                    self.__process_result(res)
-
-                    # remove result from results list once we're done with it
-                    results.remove(group_result)
+                self.__run_test_group(self.parallel_tests, parallel_cmdlines)
 
             if len(self.non_parallel_tests) > 0:
                 print("Non-parallel autotests:")
-                # run non_parallel tests. they are run one by one, synchronously
-                for test_group in self.non_parallel_tests:
-                    group_result = run_test_group(
-                        self.__get_cmdline(), "", self.target, test_group)
-
-                    self.__process_result(group_result)
+                self.__run_test_group(self.non_parallel_tests, [self.cmdline])
 
             # get total run time
             cur_time = time.time()
-- 
2.14.4

[dpdk-dev] [PATCH v3 8/9] autotest: update autotest test case list

[Reshma Pattan]

Autotest is enhanced with additional test cases
being added to autotest_data.py

Removed non existing PCI autotest.

Cc: stable@dpdk.org

Signed-off-by: Reshma Pattan <reshma.pattan@intel.com>
Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
---
 test/test/autotest_data.py | 368 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 360 insertions(+), 8 deletions(-)

diff --git a/test/test/autotest_data.py b/test/test/autotest_data.py
index c24e7bc25..37f617146 100644
--- a/test/test/autotest_data.py
+++ b/test/test/autotest_data.py
@@ -134,12 +134,6 @@
         "Func":    default_autotest,
         "Report":  None,
     },
-    {
-        "Name":    "PCI autotest",
-        "Command": "pci_autotest",
-        "Func":    default_autotest,
-        "Report":  None,
-    },
     {
         "Name":    "Malloc autotest",
         "Command": "malloc_autotest",
@@ -248,6 +242,309 @@
         "Func":    default_autotest,
         "Report":  None,
     },
+    {
+        "Name":    "Eventdev selftest octeontx",
+        "Command": "eventdev_selftest_octeontx",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Event ring autotest",
+        "Command": "event_ring_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Table autotest",
+        "Command": "table_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Flow classify autotest",
+        "Command": "flow_classify_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+        {
+        "Name":    "Set rxtx mode",
+        "Command": "set_rxtx_mode",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Set rxtx anchor",
+        "Command": "set_rxtx_anchor",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Set rxtx sc",
+        "Command": "set_rxtx_sc",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Event eth rx adapter autotest",
+        "Command": "event_eth_rx_adapter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Rawdev autotest",
+        "Command": "rawdev_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Kvargs autotest",
+        "Command": "kvargs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Devargs autotest",
+        "Command": "devargs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Link bonding autotest",
+        "Command": "link_bonding_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Link bonding mode4 autotest",
+        "Command": "link_bonding_mode4_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Link bonding rssconf autotest",
+        "Command": "link_bonding_rssconf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Crc autotest",
+        "Command": "crc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Distributor autotest",
+        "Command": "distributor_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Reorder autotest",
+        "Command": "reorder_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Barrier autotest",
+        "Command": "barrier_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "User delay",
+        "Command": "user_delay_us",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Bitmap test",
+        "Command": "bitmap_test",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash scaling autotest",
+        "Command": "hash_scaling_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash multiwriter autotest",
+        "Command": "hash_multiwriter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Service autotest",
+        "Command": "service_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Timer racecond autotest",
+        "Command": "timer_racecond_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Member autotest",
+        "Command": "member_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":   "Efd_autotest",
+        "Command": "efd_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Thash autotest",
+        "Command": "thash_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash function autotest",
+        "Command": "hash_functions_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev sw mrvl autotest",
+        "Command": "cryptodev_sw_mrvl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev dpaa2 sec autotest",
+        "Command": "cryptodev_dpaa2_sec_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev dpaa sec autotest",
+        "Command": "cryptodev_dpaa_sec_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev qat autotest",
+        "Command": "cryptodev_qat_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev aesni mb autotest",
+        "Command": "cryptodev_aesni_mb_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev openssl autotest",
+        "Command": "cryptodev_openssl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev scheduler autotest",
+        "Command": "cryptodev_scheduler_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev aesni gcm autotest",
+        "Command": "cryptodev_aesni_gcm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev null autotest",
+        "Command": "cryptodev_null_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev sw snow3g autotest",
+        "Command": "cryptodev_sw_snow3g_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev sw kasumi autotest",
+        "Command": "cryptodev_sw_kasumi_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev_sw_zuc_autotest",
+        "Command": "cryptodev_sw_zuc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Reciprocal division",
+        "Command": "reciprocal_division",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red all",
+        "Command": "red_all",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    #
+    #Please always keep all dump tests at the end and together!
+    #
+    {
+        "Name":    "Dump physmem",
+        "Command": "dump_physmem",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump memzone",
+        "Command": "dump_memzone",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump struct sizes",
+        "Command": "dump_struct_sizes",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump mempool",
+        "Command": "dump_mempool",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump malloc stats",
+        "Command": "dump_malloc_stats",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump devargs",
+        "Command": "dump_devargs",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump log types",
+        "Command": "dump_log_types",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump_ring",
+        "Command": "dump_ring",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
 ]
 
 # tests that should not be run when any other tests are running
@@ -259,8 +556,8 @@
         "Report":  None,
     },
     {
-        "Name":    "Eventdev sw autotest",
-        "Command": "eventdev_sw_autotest",
+        "Name":    "Eventdev selftest sw",
+        "Command": "eventdev_selftest_sw",
         "Func":    default_autotest,
         "Report":  None,
     },
@@ -312,6 +609,61 @@
         "Func":    default_autotest,
         "Report":  None,
     },
+    {
+
+        "Name":    "Pmd perf autotest",
+        "Command": "pmd_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Ring pmd perf autotest",
+        "Command": "ring_pmd_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Distributor perf autotest",
+        "Command": "distributor_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red_perf",
+        "Command": "red_perf",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Lpm6 perf autotest",
+        "Command": "lpm6_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Lpm perf autotest",
+        "Command": "lpm_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+         "Name":    "Efd perf autotest",
+         "Command": "efd_perf_autotest",
+         "Func":    default_autotest,
+         "Report":  None,
+    },
+    {
+        "Name":    "Member perf autotest",
+        "Command": "member_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Reciprocal division perf",
+        "Command": "reciprocal_division_perf",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
     #
     # Please always make sure that ring_perf is the last test!
     #
-- 
2.14.4

[dpdk-dev] [PATCH v3 9/9] mk: update make targets for classified testcases

[Reshma Pattan]

Makefiles are updated with new test case lists.
Test cases are classified as -
P1 - Main test cases,
P2 - Cryptodev/driver test cases,
P3 - Perf test cases which takes longer than 10s,
P4 - Logging/Dump test cases.

Makefile is updated with different targets
for the above classified groups.
Test cases for different targets are listed accordingly.

Cc: stable@dpdk.org

Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
Reviewed-by: Reshma Pattan <reshma.pattan@intel.com>
---
 mk/rte.sdkroot.mk |  4 ++--
 mk/rte.sdktest.mk | 33 +++++++++++++++++++++++++++------
 2 files changed, 29 insertions(+), 8 deletions(-)

diff --git a/mk/rte.sdkroot.mk b/mk/rte.sdkroot.mk
index f43cc7829..ea3473ebf 100644
--- a/mk/rte.sdkroot.mk
+++ b/mk/rte.sdkroot.mk
@@ -68,8 +68,8 @@ config defconfig showconfigs showversion showversionum:
 cscope gtags tags etags:
 	$(Q)$(RTE_SDK)/devtools/build-tags.sh $@ $T
 
-.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage
-test test-basic test-fast test-ring test-mempool test-perf coverage:
+.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump
+test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump:
 	$(Q)$(MAKE) -f $(RTE_SDK)/mk/rte.sdktest.mk $@
 
 test: test-build
diff --git a/mk/rte.sdktest.mk b/mk/rte.sdktest.mk
index ee1fe0c7e..13d1efb6a 100644
--- a/mk/rte.sdktest.mk
+++ b/mk/rte.sdktest.mk
@@ -18,14 +18,35 @@ DIR := $(shell basename $(RTE_OUTPUT))
 #
 # test: launch auto-tests, very simple for now.
 #
-.PHONY: test test-basic test-fast test-perf coverage
+.PHONY: test test-basic test-fast test-perf test-drivers test-dump coverage
 
-PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf
-coverage: BLACKLIST=-$(PERFLIST)
-test-fast: BLACKLIST=-$(PERFLIST)
-test-perf: WHITELIST=$(PERFLIST)
+PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf,\
+         reciprocal_division,reciprocal_division_perf,lpm_perf,red_all,\
+         barrier,hash_multiwriter,timer_racecond,efd,hash_functions,\
+         eventdev_selftest_sw,member_perf,efd_perf,lpm6_perf,red_perf,\
+         distributor_perf,ring_pmd_perf,pmd_perf,ring_perf
+DRIVERSLIST=link_bonding,link_bonding_mode4,link_bonding_rssconf,\
+            cryptodev_sw_mrvl,cryptodev_dpaa2_sec,cryptodev_dpaa_sec,\
+            cryptodev_qat,cryptodev_aesni_mb,cryptodev_openssl,\
+            cryptodev_scheduler,cryptodev_aesni_gcm,cryptodev_null,\
+            cryptodev_sw_snow3g,cryptodev_sw_kasumi,cryptodev_sw_zuc
+DUMPLIST=dump_struct_sizes,dump_mempool,dump_malloc_stats,dump_devargs,\
+         dump_log_types,dump_ring,quit,dump_physmem,dump_memzone,\
+         devargs_autotest
 
-test test-basic test-fast test-perf:
+SPACESTR:=
+SPACESTR+=
+STRIPPED_PERFLIST=$(subst $(SPACESTR),,$(PERFLIST))
+STRIPPED_DRIVERSLIST=$(subst $(SPACESTR),,$(DRIVERSLIST))
+STRIPPED_DUMPLIST=$(subst $(SPACESTR),,$(DUMPLIST))
+
+coverage: BLACKLIST=-$(STRIPPED_PERFLIST)
+test-fast: BLACKLIST=-$(STRIPPED_PERFLIST),$(STRIPPED_DRIVERSLIST),$(STRIPPED_DUMPLIST)
+test-perf: WHITELIST=$(STRIPPED_PERFLIST)
+test-drivers: WHITELIST=$(STRIPPED_DRIVERSLIST)
+test-dump: WHITELIST=$(STRIPPED_DUMPLIST)
+
+test test-basic test-fast test-perf test-drivers test-dump:
 	@mkdir -p $(AUTOTEST_DIR) ; \
 	cd $(AUTOTEST_DIR) ; \
 	if [ -f $(RTE_OUTPUT)/app/test ]; then \
-- 
2.14.4

Re: [dpdk-dev] [PATCH v3 8/9] autotest: update autotest test case list

[Burakov, Anatoly]

On 16-Jul-18 3:12 PM, Reshma Pattan wrote:
> Autotest is enhanced with additional test cases
> being added to autotest_data.py
> 
> Removed non existing PCI autotest.
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Reshma Pattan <reshma.pattan@intel.com>
> Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
> ---

<snip>

> +    },
> +    {
> +        "Name":    "Flow classify autotest",
> +        "Command": "flow_classify_autotest",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +        {
> +        "Name":    "Set rxtx mode",
> +        "Command": "set_rxtx_mode",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Set rxtx anchor",
> +        "Command": "set_rxtx_anchor",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Set rxtx sc",
> +        "Command": "set_rxtx_sc",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },

The above three tests don't look like autotests to me. I have no idea 
what they are for, but either they need a special function, or they need 
to be taken out.

> +    {
> +        "Name":    "Event eth rx adapter autotest",
> +        "Command": "event_eth_rx_adapter_autotest",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Rawdev autotest",
> +        "Command": "rawdev_autotest",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },

<snip>

> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Barrier autotest",
> +        "Command": "barrier_autotest",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "User delay",
> +        "Command": "user_delay_us",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },

This doesn't look like autotests to me. I have no idea what it is for, 
but either it needs a special function, or it needs to be taken out.

> +    {
> +        "Name":    "Bitmap test",
> +        "Command": "bitmap_test",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Hash scaling autotest",
> +        "Command": "hash_scaling_autotest",
> +        "Func":    default_autotest,
> +        "Report":  None,
> +    },
> +    {
> +        "Name":    "Hash multiwriter autotest",
> +        "Command": "hash_multiwriter_autotest",


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 8/9] autotest: update autotest test case list

[Pattan, Reshma]

Hi,

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Monday, July 16, 2018 4:16 PM
> To: Pattan, Reshma <reshma.pattan@intel.com>; thomas@monjalon.net;
> dev@dpdk.org
> Cc: Parthasarathy, JananeeX M <jananeex.m.parthasarathy@intel.com>;
> stable@dpdk.org
> Subject: Re: [PATCH v3 8/9] autotest: update autotest test case list
> 
> 
> > +        {
> > +        "Name":    "Set rxtx mode",
> > +        "Command": "set_rxtx_mode",
> > +        "Func":    default_autotest,
> > +        "Report":  None,
> > +    },
> > +    {
> > +        "Name":    "Set rxtx anchor",
> > +        "Command": "set_rxtx_anchor",
> > +        "Func":    default_autotest,
> > +        "Report":  None,
> > +    },
> > +    {
> > +        "Name":    "Set rxtx sc",
> > +        "Command": "set_rxtx_sc",
> > +        "Func":    default_autotest,
> > +        "Report":  None,
> > +    },
> 
> The above three tests don't look like autotests to me. I have no idea what
> they are for, but either they need a special function, or they need to be taken
> out.
> 

These commands needs to be run manually from test cmd prompt to various set rxtx mode, rxtx rate and rxtx direction .
These can be used to verify pmd perf test  with vaiours set of above values.

So this can be removed from autotest.

> > +        "Name":    "User delay",
> > +        "Command": "user_delay_us",
> > +        "Func":    default_autotest,
> > +        "Report":  None,
> > +    },
> 
> This doesn't look like autotests to me. I have no idea what it is for, but either
> it needs a special function, or it needs to be taken out.
> 
This is autotest but the name does'nt have the autotest name in it. So I will retain this.

Thanks,
Reshma

Re: [dpdk-dev] [PATCH v3 8/9] autotest: update autotest test case list

[Burakov, Anatoly]

On 17-Jul-18 10:18 AM, Pattan, Reshma wrote:
> Hi,
> 
>> -----Original Message-----
>> From: Burakov, Anatoly
>> Sent: Monday, July 16, 2018 4:16 PM
>> To: Pattan, Reshma <reshma.pattan@intel.com>; thomas@monjalon.net;
>> dev@dpdk.org
>> Cc: Parthasarathy, JananeeX M <jananeex.m.parthasarathy@intel.com>;
>> stable@dpdk.org
>> Subject: Re: [PATCH v3 8/9] autotest: update autotest test case list
>>
>>
>>> +        {
>>> +        "Name":    "Set rxtx mode",
>>> +        "Command": "set_rxtx_mode",
>>> +        "Func":    default_autotest,
>>> +        "Report":  None,
>>> +    },
>>> +    {
>>> +        "Name":    "Set rxtx anchor",
>>> +        "Command": "set_rxtx_anchor",
>>> +        "Func":    default_autotest,
>>> +        "Report":  None,
>>> +    },
>>> +    {
>>> +        "Name":    "Set rxtx sc",
>>> +        "Command": "set_rxtx_sc",
>>> +        "Func":    default_autotest,
>>> +        "Report":  None,
>>> +    },
>>
>> The above three tests don't look like autotests to me. I have no idea what
>> they are for, but either they need a special function, or they need to be taken
>> out.
>>
> 
> These commands needs to be run manually from test cmd prompt to various set rxtx mode, rxtx rate and rxtx direction .
> These can be used to verify pmd perf test  with vaiours set of above values.
> 
> So this can be removed from autotest.

We do have PMD perf tests in the script - do they call these functions? 
If they are required for PMD autotests, maybe PMD autotests deserve a 
special test function calling these commands before running the tests?

(if they also work without these commands, then we can perhaps postpone 
this to 18.11)

> 
>>> +        "Name":    "User delay",
>>> +        "Command": "user_delay_us",
>>> +        "Func":    default_autotest,
>>> +        "Report":  None,
>>> +    },
>>
>> This doesn't look like autotests to me. I have no idea what it is for, but either
>> it needs a special function, or it needs to be taken out.
>>
> This is autotest but the name does'nt have the autotest name in it. So I will retain this.

OK.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 8/9] autotest: update autotest test case list

[Pattan, Reshma]

Hi,

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, July 17, 2018 10:23 AM
> To: Pattan, Reshma <reshma.pattan@intel.com>; thomas@monjalon.net;
> dev@dpdk.org
> Cc: Parthasarathy, JananeeX M <jananeex.m.parthasarathy@intel.com>;
> stable@dpdk.org
> Subject: Re: [PATCH v3 8/9] autotest: update autotest test case list
> 
> On 17-Jul-18 10:18 AM, Pattan, Reshma wrote:
> > Hi,
> >
> >> -----Original Message-----
> >> From: Burakov, Anatoly
> >> Sent: Monday, July 16, 2018 4:16 PM
> >> To: Pattan, Reshma <reshma.pattan@intel.com>; thomas@monjalon.net;
> >> dev@dpdk.org
> >> Cc: Parthasarathy, JananeeX M <jananeex.m.parthasarathy@intel.com>;
> >> stable@dpdk.org
> >> Subject: Re: [PATCH v3 8/9] autotest: update autotest test case list
> >>
> >>
> >>> +        {
> >>> +        "Name":    "Set rxtx mode",
> >>> +        "Command": "set_rxtx_mode",
> >>> +        "Func":    default_autotest,
> >>> +        "Report":  None,
> >>> +    },
> >>> +    {
> >>> +        "Name":    "Set rxtx anchor",
> >>> +        "Command": "set_rxtx_anchor",
> >>> +        "Func":    default_autotest,
> >>> +        "Report":  None,
> >>> +    },
> >>> +    {
> >>> +        "Name":    "Set rxtx sc",
> >>> +        "Command": "set_rxtx_sc",
> >>> +        "Func":    default_autotest,
> >>> +        "Report":  None,
> >>> +    },
> >>
> >> The above three tests don't look like autotests to me. I have no idea
> >> what they are for, but either they need a special function, or they
> >> need to be taken out.
> >>
> >
> > These commands needs to be run manually from test cmd prompt to
> various set rxtx mode, rxtx rate and rxtx direction .
> > These can be used to verify pmd perf test  with vaiours set of above values.
> >
> > So this can be removed from autotest.
> 
> We do have PMD perf tests in the script - do they call these functions?
> If they are required for PMD autotests, maybe PMD autotests deserve a
> special test function calling these commands before running the tests?
> 
> (if they also work without these commands, then we can perhaps postpone
> this to 18.11)
> 

I ran pmd perf test manually and it passes without having to use above set_rxtx commands. 

Thanks,
Reshma

Re: [dpdk-dev] [PATCH v3 8/9] autotest: update autotest test case list

[Burakov, Anatoly]

On 17-Jul-18 10:45 AM, Pattan, Reshma wrote:
> Hi,
> 
>> -----Original Message-----
>> From: Burakov, Anatoly
>> Sent: Tuesday, July 17, 2018 10:23 AM
>> To: Pattan, Reshma <reshma.pattan@intel.com>; thomas@monjalon.net;
>> dev@dpdk.org
>> Cc: Parthasarathy, JananeeX M <jananeex.m.parthasarathy@intel.com>;
>> stable@dpdk.org
>> Subject: Re: [PATCH v3 8/9] autotest: update autotest test case list
>>
>> On 17-Jul-18 10:18 AM, Pattan, Reshma wrote:
>>> Hi,
>>>
>>>> -----Original Message-----
>>>> From: Burakov, Anatoly
>>>> Sent: Monday, July 16, 2018 4:16 PM
>>>> To: Pattan, Reshma <reshma.pattan@intel.com>; thomas@monjalon.net;
>>>> dev@dpdk.org
>>>> Cc: Parthasarathy, JananeeX M <jananeex.m.parthasarathy@intel.com>;
>>>> stable@dpdk.org
>>>> Subject: Re: [PATCH v3 8/9] autotest: update autotest test case list
>>>>
>>>>
>>>>> +        {
>>>>> +        "Name":    "Set rxtx mode",
>>>>> +        "Command": "set_rxtx_mode",
>>>>> +        "Func":    default_autotest,
>>>>> +        "Report":  None,
>>>>> +    },
>>>>> +    {
>>>>> +        "Name":    "Set rxtx anchor",
>>>>> +        "Command": "set_rxtx_anchor",
>>>>> +        "Func":    default_autotest,
>>>>> +        "Report":  None,
>>>>> +    },
>>>>> +    {
>>>>> +        "Name":    "Set rxtx sc",
>>>>> +        "Command": "set_rxtx_sc",
>>>>> +        "Func":    default_autotest,
>>>>> +        "Report":  None,
>>>>> +    },
>>>>
>>>> The above three tests don't look like autotests to me. I have no idea
>>>> what they are for, but either they need a special function, or they
>>>> need to be taken out.
>>>>
>>>
>>> These commands needs to be run manually from test cmd prompt to
>> various set rxtx mode, rxtx rate and rxtx direction .
>>> These can be used to verify pmd perf test  with vaiours set of above values.
>>>
>>> So this can be removed from autotest.
>>
>> We do have PMD perf tests in the script - do they call these functions?
>> If they are required for PMD autotests, maybe PMD autotests deserve a
>> special test function calling these commands before running the tests?
>>
>> (if they also work without these commands, then we can perhaps postpone
>> this to 18.11)
>>
> 
> I ran pmd perf test manually and it passes without having to use above set_rxtx commands.
> 
> Thanks,
> Reshma
> 
Great.

Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

[dpdk-dev] [PATCH v4 0/9] Make unit tests great again

[Reshma Pattan]

Previously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.

In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.

This patchset merges changes done in [1], [2]

[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://patches.dpdk.org/patch/40373/

v4: Removed non auto tests set_rxtx_mode, set_rxtx_anchor and set_rxtx_sc
from autotest_data.py

Reshma Pattan (9):
  autotest: fix printing
  autotest: fix invalid code on reports
  autotest: make autotest runner python 2/3 compliant
  autotest: visually separate different test categories
  autotest: improve filtering
  autotest: remove autotest grouping
  autotest: properly parallelize unit tests
  autotest: update autotest test case list
  mk: update make targets for classified testcases

 mk/rte.sdkroot.mk            |    4 +-
 mk/rte.sdktest.mk            |   33 +-
 test/test/autotest.py        |   13 +-
 test/test/autotest_data.py   | 1081 +++++++++++++++++++++++++-----------------
 test/test/autotest_runner.py |  519 ++++++++++----------
 5 files changed, 948 insertions(+), 702 deletions(-)

-- 
2.14.4

[dpdk-dev] [PATCH v4 1/9] autotest: fix printing

[Reshma Pattan]

Previously, printing was done using tuple syntax, which caused
output to appear as a tuple as opposed to being one string. Fix
this by using addition operator instead.

Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index a692f0697..b09b57876 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -247,7 +247,7 @@ def __process_results(self, results):
 
             # don't print out total time every line, it's the same anyway
             if i == len(results) - 1:
-                print(result,
+                print(result +
                       "[%02dm %02ds]" % (total_time / 60, total_time % 60))
             else:
                 print(result)
@@ -332,8 +332,8 @@ def run_all_tests(self):
 
             # create table header
             print("")
-            print("Test name".ljust(30), "Test result".ljust(29),
-                  "Test".center(9), "Total".center(9))
+            print("Test name".ljust(30) + "Test result".ljust(29) +
+                  "Test".center(9) + "Total".center(9))
             print("=" * 80)
 
             # make a note of tests start time
-- 
2.14.4

[dpdk-dev] [PATCH v4 2/9] autotest: fix invalid code on reports

[Reshma Pattan]

There are no reports defined for any test, so this codepath was
never triggered, but it's still wrong because it's referencing
variables that aren't there. Fix it by passing target into the
test function, and reference correct log variable.

Fixes: e2cc79b75d9f ("app: rework autotest.py")
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index b09b57876..bdc32da5d 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -41,7 +41,7 @@ def wait_prompt(child):
 # quite a bit of effort to make it work).
 
 
-def run_test_group(cmdline, test_group):
+def run_test_group(cmdline, target, test_group):
     results = []
     child = None
     start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group):
             # make a note when the test was finished
             end_time = time.time()
 
+            log = logfile.getvalue()
+
             # append test data to the result tuple
-            result += (test["Name"], end_time - start_time,
-                       logfile.getvalue())
+            result += (test["Name"], end_time - start_time, log)
 
             # call report function, if any defined, and supply it with
             # target and complete log for test run
             if test["Report"]:
-                report = test["Report"](self.target, log)
+                report = test["Report"](target, log)
 
                 # append report to results tuple
                 result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self):
             for test_group in self.parallel_test_groups:
                 result = pool.apply_async(run_test_group,
                                           [self.__get_cmdline(test_group),
+                                           self.target,
                                            test_group])
                 results.append(result)
 
@@ -367,7 +369,7 @@ def run_all_tests(self):
             # run non_parallel tests. they are run one by one, synchronously
             for test_group in self.non_parallel_test_groups:
                 group_result = run_test_group(
-                    self.__get_cmdline(test_group), test_group)
+                    self.__get_cmdline(test_group), self.target, test_group)
 
                 self.__process_results(group_result)
 
-- 
2.14.4

[dpdk-dev] [PATCH v4 3/9] autotest: make autotest runner python 2/3 compliant

[Reshma Pattan]

Autotest runner was still using python 2-style print syntax. Fix
it by importing print function from the future, and fix the calls
to be python-3 style.

Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index bdc32da5d..f6b669a2e 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -3,6 +3,7 @@
 
 # The main logic behind running autotests in parallel
 
+from __future__ import print_function
 import StringIO
 import csv
 import multiprocessing
@@ -52,8 +53,8 @@ def run_test_group(cmdline, target, test_group):
         # prepare logging of init
         startuplog = StringIO.StringIO()
 
-        print >>startuplog, "\n%s %s\n" % ("=" * 20, test_group["Prefix"])
-        print >>startuplog, "\ncmdline=%s" % cmdline
+        print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog)
+        print("\ncmdline=%s" % cmdline, file=startuplog)
 
         child = pexpect.spawn(cmdline, logfile=startuplog)
 
@@ -117,7 +118,7 @@ def run_test_group(cmdline, target, test_group):
 
         try:
             # print test name to log buffer
-            print >>logfile, "\n%s %s\n" % ("-" * 20, test["Name"])
+            print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
             # run test function associated with the test
             if stripped or test["Command"] in avail_cmds:
-- 
2.14.4

[dpdk-dev] [PATCH v4 4/9] autotest: visually separate different test categories

[Reshma Pattan]

Help visually identify parallel vs. non-parallel autotests.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index f6b669a2e..d9d5f7a97 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -341,6 +341,7 @@ def run_all_tests(self):
             # make a note of tests start time
             self.start = time.time()
 
+            print("Parallel autotests:")
             # assign worker threads to run test groups
             for test_group in self.parallel_test_groups:
                 result = pool.apply_async(run_test_group,
@@ -367,6 +368,7 @@ def run_all_tests(self):
                     # remove result from results list once we're done with it
                     results.remove(group_result)
 
+            print("Non-parallel autotests:")
             # run non_parallel tests. they are run one by one, synchronously
             for test_group in self.non_parallel_test_groups:
                 group_result = run_test_group(
-- 
2.14.4

[dpdk-dev] [PATCH v4 5/9] autotest: improve filtering

[Reshma Pattan]

Improve code for filtering test groups. Also, move reading binary
symbols into filtering stage, so that tests that are meant to be
skipped are never attempted to be executed in the first place.
Before running tests, print out any tests that were skipped because
they weren't compiled.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_runner.py | 118 ++++++++++++++++++++++++-------------------
 1 file changed, 66 insertions(+), 52 deletions(-)

diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index d9d5f7a97..c98ec2a57 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -95,13 +95,6 @@ def run_test_group(cmdline, target, test_group):
     results.append((0, "Success", "Start %s" % test_group["Prefix"],
                     time.time() - start_time, startuplog.getvalue(), None))
 
-    # parse the binary for available test commands
-    binary = cmdline.split()[0]
-    stripped = 'not stripped' not in subprocess.check_output(['file', binary])
-    if not stripped:
-        symbols = subprocess.check_output(['nm', binary]).decode('utf-8')
-        avail_cmds = re.findall('test_register_(\w+)', symbols)
-
     # run all tests in test group
     for test in test_group["Tests"]:
 
@@ -121,10 +114,7 @@ def run_test_group(cmdline, target, test_group):
             print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
             # run test function associated with the test
-            if stripped or test["Command"] in avail_cmds:
-                result = test["Func"](child, test["Command"])
-            else:
-                result = (0, "Skipped [Not Available]")
+            result = test["Func"](child, test["Command"])
 
             # make a note when the test was finished
             end_time = time.time()
@@ -186,8 +176,10 @@ class AutotestRunner:
     def __init__(self, cmdline, target, blacklist, whitelist):
         self.cmdline = cmdline
         self.target = target
+        self.binary = cmdline.split()[0]
         self.blacklist = blacklist
         self.whitelist = whitelist
+        self.skipped = []
 
         # log file filename
         logfile = "%s.log" % target
@@ -276,53 +268,58 @@ def __process_results(self, results):
             if i != 0:
                 self.csvwriter.writerow([test_name, test_result, result_str])
 
-    # this function iterates over test groups and removes each
-    # test that is not in whitelist/blacklist
-    def __filter_groups(self, test_groups):
-        groups_to_remove = []
-
-        # filter out tests from parallel test groups
-        for i, test_group in enumerate(test_groups):
-
-            # iterate over a copy so that we could safely delete individual
-            # tests
-            for test in test_group["Tests"][:]:
-                test_id = test["Command"]
-
-                # dump tests are specified in full e.g. "Dump_mempool"
-                if "_autotest" in test_id:
-                    test_id = test_id[:-len("_autotest")]
-
-                # filter out blacklisted/whitelisted tests
-                if self.blacklist and test_id in self.blacklist:
-                    test_group["Tests"].remove(test)
-                    continue
-                if self.whitelist and test_id not in self.whitelist:
-                    test_group["Tests"].remove(test)
-                    continue
-
-            # modify or remove original group
-            if len(test_group["Tests"]) > 0:
-                test_groups[i] = test_group
-            else:
-                # remember which groups should be deleted
-                # put the numbers backwards so that we start
-                # deleting from the end, not from the beginning
-                groups_to_remove.insert(0, i)
+    # this function checks individual test and decides if this test should be in
+    # the group by comparing it against  whitelist/blacklist. it also checks if
+    # the test is compiled into the binary, and marks it as skipped if necessary
+    def __filter_test(self, test):
+        test_cmd = test["Command"]
+        test_id = test_cmd
+
+        # dump tests are specified in full e.g. "Dump_mempool"
+        if "_autotest" in test_id:
+            test_id = test_id[:-len("_autotest")]
+
+        # filter out blacklisted/whitelisted tests
+        if self.blacklist and test_id in self.blacklist:
+            return False
+        if self.whitelist and test_id not in self.whitelist:
+            return False
+
+        # if test wasn't compiled in, remove it as well
+
+        # parse the binary for available test commands
+        stripped = 'not stripped' not in \
+                   subprocess.check_output(['file', self.binary])
+        if not stripped:
+            symbols = subprocess.check_output(['nm',
+                                               self.binary]).decode('utf-8')
+            avail_cmds = re.findall('test_register_(\w+)', symbols)
+
+            if test_cmd not in avail_cmds:
+                # notify user
+                result = 0, "Skipped [Not compiled]", test_id, 0, "", None
+                self.skipped.append(tuple(result))
+                return False
 
-        # remove test groups that need to be removed
-        for i in groups_to_remove:
-            del test_groups[i]
+        return True
 
-        return test_groups
+    def __filter_group(self, group):
+        group["Tests"] = list(filter(self.__filter_test, group["Tests"]))
+        return len(group["Tests"]) > 0
 
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
-        self.parallel_test_groups = \
-            self.__filter_groups(self.parallel_test_groups)
-        self.non_parallel_test_groups = \
-            self.__filter_groups(self.non_parallel_test_groups)
+        # for each test group, check all tests against the filter, then remove
+        # all groups that don't have any tests
+        self.parallel_test_groups = list(
+            filter(self.__filter_group,
+                   self.parallel_test_groups)
+        )
+        self.non_parallel_test_groups = list(
+            filter(self.__filter_group,
+                   self.non_parallel_test_groups)
+        )
 
         # create a pool of worker threads
         pool = multiprocessing.Pool(processes=1)
@@ -338,6 +335,23 @@ def run_all_tests(self):
                   "Test".center(9) + "Total".center(9))
             print("=" * 80)
 
+            # print out skipped autotests if there were any
+            if len(self.skipped):
+                print("Skipped autotests:")
+
+                # print out any skipped tests
+                for result in self.skipped:
+                    # unpack result tuple
+                    test_result, result_str, test_name, _, _, _ = result
+                    self.csvwriter.writerow([test_name, test_result,
+                                             result_str])
+
+                    t = ("%s:" % test_name).ljust(30)
+                    t += result_str.ljust(29)
+                    t += "[00m 00s]"
+
+                    print(t)
+
             # make a note of tests start time
             self.start = time.time()
 
-- 
2.14.4

[dpdk-dev] [PATCH v4 6/9] autotest: remove autotest grouping

[Reshma Pattan]

Previously, all autotests were grouped into (seemingly arbitrary)
groups. The goal was to run all tests in parallel (so that autotest
finishes faster), but we couldn't just do it willy-nilly because
DPDK couldn't allocate and free hugepages on-demand, so we had to
find autotest groupings that could work memory-wise and still be
fast enough to not hold up shorter tests. The inflexibility of
memory subsystem has now been fixed for 18.05, so grouping
autotests is no longer necessary.

Thus, this commit moves all autotests into two groups -
parallel(izable) autotests, and non-arallel(izable) autotests
(typically performance tests). Note that this particular commit
makes running autotests dog slow because while the tests are now
in a single group, the test function itself hasn't changed much,
so all autotests are now run one-by-one, starting and stopping
the DPDK test application.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest.py        |   7 +-
 test/test/autotest_data.py   | 749 +++++++++++++++++--------------------------
 test/test/autotest_runner.py | 271 ++++++----------
 3 files changed, 408 insertions(+), 619 deletions(-)

diff --git a/test/test/autotest.py b/test/test/autotest.py
index 1cfd8cf22..ae27daef7 100644
--- a/test/test/autotest.py
+++ b/test/test/autotest.py
@@ -39,11 +39,8 @@ def usage():
 runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist,
                                         test_whitelist)
 
-for test_group in autotest_data.parallel_test_group_list:
-    runner.add_parallel_test_group(test_group)
-
-for test_group in autotest_data.non_parallel_test_group_list:
-    runner.add_non_parallel_test_group(test_group)
+runner.parallel_tests = autotest_data.parallel_test_list[:]
+runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
 
 num_fails = runner.run_all_tests()
 
diff --git a/test/test/autotest_data.py b/test/test/autotest_data.py
index aacfe0a66..c24e7bc25 100644
--- a/test/test/autotest_data.py
+++ b/test/test/autotest_data.py
@@ -3,465 +3,322 @@
 
 # Test data for autotests
 
-from glob import glob
 from autotest_test_funcs import *
 
-
-# quick and dirty function to find out number of sockets
-def num_sockets():
-    result = len(glob("/sys/devices/system/node/node*"))
-    if result == 0:
-        return 1
-    return result
-
-
-# Assign given number to each socket
-# e.g. 32 becomes 32,32 or 32,32,32,32
-def per_sockets(num):
-    return ",".join([str(num)] * num_sockets())
-
 # groups of tests that can be run in parallel
 # the grouping has been found largely empirically
-parallel_test_group_list = [
-    {
-        "Prefix":    "group_1",
-        "Memory":    per_sockets(8),
-        "Tests":
-        [
-            {
-                "Name":    "Cycles autotest",
-                "Command": "cycles_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Timer autotest",
-                "Command": "timer_autotest",
-                "Func":    timer_autotest,
-                "Report":   None,
-            },
-            {
-                "Name":    "Debug autotest",
-                "Command": "debug_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Errno autotest",
-                "Command": "errno_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Meter autotest",
-                "Command": "meter_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Common autotest",
-                "Command": "common_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Resource autotest",
-                "Command": "resource_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_2",
-        "Memory":    "16",
-        "Tests":
-        [
-            {
-                "Name":    "Memory autotest",
-                "Command": "memory_autotest",
-                "Func":    memory_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Read/write lock autotest",
-                "Command": "rwlock_autotest",
-                "Func":    rwlock_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Logs autotest",
-                "Command": "logs_autotest",
-                "Func":    logs_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "CPU flags autotest",
-                "Command": "cpuflags_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Version autotest",
-                "Command": "version_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "EAL filesystem autotest",
-                "Command": "eal_fs_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "EAL flags autotest",
-                "Command": "eal_flags_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Hash autotest",
-                "Command": "hash_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ],
-    },
-    {
-        "Prefix":    "group_3",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "LPM autotest",
-                "Command": "lpm_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "LPM6 autotest",
-                "Command": "lpm6_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Memcpy autotest",
-                "Command": "memcpy_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Memzone autotest",
-                "Command": "memzone_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "String autotest",
-                "Command": "string_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Alarm autotest",
-                "Command": "alarm_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_4",
-        "Memory":    per_sockets(128),
-        "Tests":
-        [
-            {
-                "Name":    "PCI autotest",
-                "Command": "pci_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Malloc autotest",
-                "Command": "malloc_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Multi-process autotest",
-                "Command": "multiprocess_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Mbuf autotest",
-                "Command": "mbuf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Per-lcore autotest",
-                "Command": "per_lcore_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Ring autotest",
-                "Command": "ring_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_5",
-        "Memory":    "32",
-        "Tests":
-        [
-            {
-                "Name":    "Spinlock autotest",
-                "Command": "spinlock_autotest",
-                "Func":    spinlock_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Byte order autotest",
-                "Command": "byteorder_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "TAILQ autotest",
-                "Command": "tailq_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Command-line autotest",
-                "Command": "cmdline_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Interrupts autotest",
-                "Command": "interrupt_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_6",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Function reentrancy autotest",
-                "Command": "func_reentrancy_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Mempool autotest",
-                "Command": "mempool_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Atomics autotest",
-                "Command": "atomic_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Prefetch autotest",
-                "Command": "prefetch_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Red autotest",
-                "Command": "red_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "group_7",
-        "Memory":    "64",
-        "Tests":
-        [
-            {
-                "Name":    "PMD ring autotest",
-                "Command": "ring_pmd_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Access list control autotest",
-                "Command": "acl_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-            {
-                "Name":    "Sched autotest",
-                "Command": "sched_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+parallel_test_list = [
+    {
+        "Name":    "Cycles autotest",
+        "Command": "cycles_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Timer autotest",
+        "Command": "timer_autotest",
+        "Func":    timer_autotest,
+        "Report":   None,
+    },
+    {
+        "Name":    "Debug autotest",
+        "Command": "debug_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Errno autotest",
+        "Command": "errno_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Meter autotest",
+        "Command": "meter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Common autotest",
+        "Command": "common_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Resource autotest",
+        "Command": "resource_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memory autotest",
+        "Command": "memory_autotest",
+        "Func":    memory_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Read/write lock autotest",
+        "Command": "rwlock_autotest",
+        "Func":    rwlock_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Logs autotest",
+        "Command": "logs_autotest",
+        "Func":    logs_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "CPU flags autotest",
+        "Command": "cpuflags_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Version autotest",
+        "Command": "version_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "EAL filesystem autotest",
+        "Command": "eal_fs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "EAL flags autotest",
+        "Command": "eal_flags_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash autotest",
+        "Command": "hash_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "LPM autotest",
+        "Command": "lpm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "LPM6 autotest",
+        "Command": "lpm6_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memcpy autotest",
+        "Command": "memcpy_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memzone autotest",
+        "Command": "memzone_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "String autotest",
+        "Command": "string_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Alarm autotest",
+        "Command": "alarm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "PCI autotest",
+        "Command": "pci_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Malloc autotest",
+        "Command": "malloc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Multi-process autotest",
+        "Command": "multiprocess_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mbuf autotest",
+        "Command": "mbuf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Per-lcore autotest",
+        "Command": "per_lcore_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Ring autotest",
+        "Command": "ring_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Spinlock autotest",
+        "Command": "spinlock_autotest",
+        "Func":    spinlock_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Byte order autotest",
+        "Command": "byteorder_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "TAILQ autotest",
+        "Command": "tailq_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Command-line autotest",
+        "Command": "cmdline_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Interrupts autotest",
+        "Command": "interrupt_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Function reentrancy autotest",
+        "Command": "func_reentrancy_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mempool autotest",
+        "Command": "mempool_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Atomics autotest",
+        "Command": "atomic_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Prefetch autotest",
+        "Command": "prefetch_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red autotest",
+        "Command": "red_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "PMD ring autotest",
+        "Command": "ring_pmd_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Access list control autotest",
+        "Command": "acl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Sched autotest",
+        "Command": "sched_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
 ]
 
 # tests that should not be run when any other tests are running
-non_parallel_test_group_list = [
-
+non_parallel_test_list = [
     {
-        "Prefix":    "eventdev",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "Eventdev common autotest",
-                "Command": "eventdev_common_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "eventdev_sw",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "Eventdev sw autotest",
-                "Command": "eventdev_sw_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "kni",
-        "Memory":    "512",
-        "Tests":
-        [
-            {
-                "Name":    "KNI autotest",
-                "Command": "kni_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "mempool_perf",
-        "Memory":    per_sockets(256),
-        "Tests":
-        [
-            {
-                "Name":    "Mempool performance autotest",
-                "Command": "mempool_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "memcpy_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Memcpy performance autotest",
-                "Command": "memcpy_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "hash_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Hash performance autotest",
-                "Command": "hash_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power autotest",
-                "Command":    "power_autotest",
-                "Func":       default_autotest,
-                "Report":      None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power_acpi_cpufreq",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power ACPI cpufreq autotest",
-                "Command":    "power_acpi_cpufreq_autotest",
-                "Func":       default_autotest,
-                "Report":     None,
-            },
-        ]
-    },
-    {
-        "Prefix":      "power_kvm_vm",
-        "Memory":      "16",
-        "Tests":
-        [
-            {
-                "Name":       "Power KVM VM  autotest",
-                "Command":    "power_kvm_vm_autotest",
-                "Func":       default_autotest,
-                "Report":     None,
-            },
-        ]
-    },
-    {
-        "Prefix":    "timer_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Timer performance autotest",
-                "Command": "timer_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+        "Name":    "Eventdev common autotest",
+        "Command": "eventdev_common_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Eventdev sw autotest",
+        "Command": "eventdev_sw_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "KNI autotest",
+        "Command": "kni_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Mempool performance autotest",
+        "Command": "mempool_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Memcpy performance autotest",
+        "Command": "memcpy_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash performance autotest",
+        "Command": "hash_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":       "Power autotest",
+        "Command":    "power_autotest",
+        "Func":       default_autotest,
+        "Report":      None,
+    },
+    {
+        "Name":       "Power ACPI cpufreq autotest",
+        "Command":    "power_acpi_cpufreq_autotest",
+        "Func":       default_autotest,
+        "Report":     None,
+    },
+    {
+        "Name":       "Power KVM VM  autotest",
+        "Command":    "power_kvm_vm_autotest",
+        "Func":       default_autotest,
+        "Report":     None,
+    },
+    {
+        "Name":    "Timer performance autotest",
+        "Command": "timer_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
-
     #
     # Please always make sure that ring_perf is the last test!
     #
     {
-        "Prefix":    "ring_perf",
-        "Memory":    per_sockets(512),
-        "Tests":
-        [
-            {
-                "Name":    "Ring performance autotest",
-                "Command": "ring_perf_autotest",
-                "Func":    default_autotest,
-                "Report":  None,
-            },
-        ]
+        "Name":    "Ring performance autotest",
+        "Command": "ring_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
     },
 ]
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index c98ec2a57..d6ae57e76 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -42,18 +42,16 @@ def wait_prompt(child):
 # quite a bit of effort to make it work).
 
 
-def run_test_group(cmdline, target, test_group):
-    results = []
-    child = None
+def run_test_group(cmdline, prefix, target, test):
     start_time = time.time()
-    startuplog = None
+
+    # prepare logging of init
+    startuplog = StringIO.StringIO()
 
     # run test app
     try:
-        # prepare logging of init
-        startuplog = StringIO.StringIO()
 
-        print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog)
+        print("\n%s %s\n" % ("=" * 20, prefix), file=startuplog)
         print("\ncmdline=%s" % cmdline, file=startuplog)
 
         child = pexpect.spawn(cmdline, logfile=startuplog)
@@ -62,88 +60,54 @@ def run_test_group(cmdline, target, test_group):
         if not wait_prompt(child):
             child.close()
 
-            results.append((-1,
-                            "Fail [No prompt]",
-                            "Start %s" % test_group["Prefix"],
-                            time.time() - start_time,
-                            startuplog.getvalue(),
-                            None))
-
-            # mark all tests as failed
-            for test in test_group["Tests"]:
-                results.append((-1, "Fail [No prompt]", test["Name"],
-                                time.time() - start_time, "", None))
-            # exit test
-            return results
+            return -1, "Fail [No prompt]", "Start %s" % prefix,\
+                   time.time() - start_time, startuplog.getvalue(), None
 
     except:
-        results.append((-1,
-                        "Fail [Can't run]",
-                        "Start %s" % test_group["Prefix"],
-                        time.time() - start_time,
-                        startuplog.getvalue(),
-                        None))
-
-        # mark all tests as failed
-        for t in test_group["Tests"]:
-            results.append((-1, "Fail [Can't run]", t["Name"],
-                            time.time() - start_time, "", None))
-        # exit test
-        return results
-
-    # startup was successful
-    results.append((0, "Success", "Start %s" % test_group["Prefix"],
-                    time.time() - start_time, startuplog.getvalue(), None))
-
-    # run all tests in test group
-    for test in test_group["Tests"]:
-
-        # create log buffer for each test
-        # in multiprocessing environment, the logging would be
-        # interleaved and will create a mess, hence the buffering
-        logfile = StringIO.StringIO()
-        child.logfile = logfile
-
-        result = ()
-
-        # make a note when the test started
-        start_time = time.time()
+        return -1, "Fail [Can't run]", "Start %s" % prefix,\
+               time.time() - start_time, startuplog.getvalue(), None
 
-        try:
-            # print test name to log buffer
-            print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
+    # create log buffer for each test
+    # in multiprocessing environment, the logging would be
+    # interleaved and will create a mess, hence the buffering
+    logfile = StringIO.StringIO()
+    child.logfile = logfile
 
-            # run test function associated with the test
-            result = test["Func"](child, test["Command"])
+    # make a note when the test started
+    start_time = time.time()
 
-            # make a note when the test was finished
-            end_time = time.time()
+    try:
+        # print test name to log buffer
+        print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
-            log = logfile.getvalue()
+        # run test function associated with the test
+        result = test["Func"](child, test["Command"])
 
-            # append test data to the result tuple
-            result += (test["Name"], end_time - start_time, log)
+        # make a note when the test was finished
+        end_time = time.time()
 
-            # call report function, if any defined, and supply it with
-            # target and complete log for test run
-            if test["Report"]:
-                report = test["Report"](target, log)
+        log = logfile.getvalue()
 
-                # append report to results tuple
-                result += (report,)
-            else:
-                # report is None
-                result += (None,)
-        except:
-            # make a note when the test crashed
-            end_time = time.time()
+        # append test data to the result tuple
+        result += (test["Name"], end_time - start_time, log)
+
+        # call report function, if any defined, and supply it with
+        # target and complete log for test run
+        if test["Report"]:
+            report = test["Report"](target, log)
+
+            # append report to results tuple
+            result += (report,)
+        else:
+            # report is None
+            result += (None,)
+    except:
+        # make a note when the test crashed
+        end_time = time.time()
 
-            # mark test as failed
-            result = (-1, "Fail [Crash]", test["Name"],
-                      end_time - start_time, logfile.getvalue(), None)
-        finally:
-            # append the results to the results list
-            results.append(result)
+        # mark test as failed
+        result = (-1, "Fail [Crash]", test["Name"],
+                  end_time - start_time, logfile.getvalue(), None)
 
     # regardless of whether test has crashed, try quitting it
     try:
@@ -155,7 +119,7 @@ def run_test_group(cmdline, target, test_group):
         pass
 
     # return test results
-    return results
+    return result
 
 
 # class representing an instance of autotests run
@@ -180,6 +144,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.blacklist = blacklist
         self.whitelist = whitelist
         self.skipped = []
+        self.parallel_tests = []
+        self.non_parallel_tests = []
 
         # log file filename
         logfile = "%s.log" % target
@@ -193,80 +159,52 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.csvwriter.writerow(["test_name", "test_result", "result_str"])
 
     # set up cmdline string
-    def __get_cmdline(self, test):
+    def __get_cmdline(self):
         cmdline = self.cmdline
 
-        # append memory limitations for each test
-        # otherwise tests won't run in parallel
-        if "i686" not in self.target:
-            cmdline += " --socket-mem=%s" % test["Memory"]
-        else:
-            # affinitize startup so that tests don't fail on i686
-            cmdline = "taskset 1 " + cmdline
-            cmdline += " -m " + str(sum(map(int, test["Memory"].split(","))))
-
-        # set group prefix for autotest group
-        # otherwise they won't run in parallel
-        cmdline += " --file-prefix=%s" % test["Prefix"]
+        # affinitize startup so that tests don't fail on i686
+        cmdline = "taskset 1 " + cmdline
 
         return cmdline
 
-    def add_parallel_test_group(self, test_group):
-        self.parallel_test_groups.append(test_group)
+    def __process_result(self, result):
 
-    def add_non_parallel_test_group(self, test_group):
-        self.non_parallel_test_groups.append(test_group)
+        # unpack result tuple
+        test_result, result_str, test_name, \
+            test_time, log, report = result
 
-    def __process_results(self, results):
-        # this iterates over individual test results
-        for i, result in enumerate(results):
+        # get total run time
+        cur_time = time.time()
+        total_time = int(cur_time - self.start)
 
-            # increase total number of tests that were run
-            # do not include "start" test
-            if i > 0:
-                self.n_tests += 1
+        # print results, test run time and total time since start
+        result = ("%s:" % test_name).ljust(30)
+        result += result_str.ljust(29)
+        result += "[%02dm %02ds]" % (test_time / 60, test_time % 60)
 
-            # unpack result tuple
-            test_result, result_str, test_name, \
-                test_time, log, report = result
+        # don't print out total time every line, it's the same anyway
+        print(result + "[%02dm %02ds]" % (total_time / 60, total_time % 60))
 
-            # get total run time
-            cur_time = time.time()
-            total_time = int(cur_time - self.start)
+        # if test failed and it wasn't a "start" test
+        if test_result < 0:
+            self.fails += 1
 
-            # print results, test run time and total time since start
-            result = ("%s:" % test_name).ljust(30)
-            result += result_str.ljust(29)
-            result += "[%02dm %02ds]" % (test_time / 60, test_time % 60)
+        # collect logs
+        self.log_buffers.append(log)
 
-            # don't print out total time every line, it's the same anyway
-            if i == len(results) - 1:
-                print(result +
-                      "[%02dm %02ds]" % (total_time / 60, total_time % 60))
+        # create report if it exists
+        if report:
+            try:
+                f = open("%s_%s_report.rst" %
+                         (self.target, test_name), "w")
+            except IOError:
+                print("Report for %s could not be created!" % test_name)
             else:
-                print(result)
-
-            # if test failed and it wasn't a "start" test
-            if test_result < 0 and not i == 0:
-                self.fails += 1
-
-            # collect logs
-            self.log_buffers.append(log)
-
-            # create report if it exists
-            if report:
-                try:
-                    f = open("%s_%s_report.rst" %
-                             (self.target, test_name), "w")
-                except IOError:
-                    print("Report for %s could not be created!" % test_name)
-                else:
-                    with f:
-                        f.write(report)
-
-            # write test result to CSV file
-            if i != 0:
-                self.csvwriter.writerow([test_name, test_result, result_str])
+                with f:
+                    f.write(report)
+
+        # write test result to CSV file
+        self.csvwriter.writerow([test_name, test_result, result_str])
 
     # this function checks individual test and decides if this test should be in
     # the group by comparing it against  whitelist/blacklist. it also checks if
@@ -303,22 +241,16 @@ def __filter_test(self, test):
 
         return True
 
-    def __filter_group(self, group):
-        group["Tests"] = list(filter(self.__filter_test, group["Tests"]))
-        return len(group["Tests"]) > 0
-
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
-        # for each test group, check all tests against the filter, then remove
-        # all groups that don't have any tests
-        self.parallel_test_groups = list(
-            filter(self.__filter_group,
-                   self.parallel_test_groups)
+        self.parallel_tests = list(
+            filter(self.__filter_test,
+                   self.parallel_tests)
         )
-        self.non_parallel_test_groups = list(
-            filter(self.__filter_group,
-                   self.non_parallel_test_groups)
+        self.non_parallel_tests = list(
+            filter(self.__filter_test,
+                   self.non_parallel_tests)
         )
 
         # create a pool of worker threads
@@ -355,14 +287,16 @@ def run_all_tests(self):
             # make a note of tests start time
             self.start = time.time()
 
-            print("Parallel autotests:")
-            # assign worker threads to run test groups
-            for test_group in self.parallel_test_groups:
-                result = pool.apply_async(run_test_group,
-                                          [self.__get_cmdline(test_group),
-                                           self.target,
-                                           test_group])
-                results.append(result)
+            if len(self.parallel_tests) > 0:
+                print("Parallel autotests:")
+                # assign worker threads to run test groups
+                for test_group in self.parallel_tests:
+                    result = pool.apply_async(run_test_group,
+                                              [self.__get_cmdline(),
+                                               "",
+                                               self.target,
+                                               test_group])
+                    results.append(result)
 
             # iterate while we have group execution results to get
             while len(results) > 0:
@@ -377,18 +311,19 @@ def run_all_tests(self):
 
                     res = group_result.get()
 
-                    self.__process_results(res)
+                    self.__process_result(res)
 
                     # remove result from results list once we're done with it
                     results.remove(group_result)
 
-            print("Non-parallel autotests:")
-            # run non_parallel tests. they are run one by one, synchronously
-            for test_group in self.non_parallel_test_groups:
-                group_result = run_test_group(
-                    self.__get_cmdline(test_group), self.target, test_group)
+            if len(self.non_parallel_tests) > 0:
+                print("Non-parallel autotests:")
+                # run non_parallel tests. they are run one by one, synchronously
+                for test_group in self.non_parallel_tests:
+                    group_result = run_test_group(
+                        self.__get_cmdline(), "", self.target, test_group)
 
-                self.__process_results(group_result)
+                    self.__process_result(group_result)
 
             # get total run time
             cur_time = time.time()
-- 
2.14.4

[dpdk-dev] [PATCH v4 7/9] autotest: properly parallelize unit tests

[Reshma Pattan]

Now that everything else is in place, we can run unit tests in a
different fashion to what they were running as before. Previously,
we had all autotests as part of groups (largely obtained through
trial and error) to ensure parallel execution while still limiting
amounts of memory used by those tests.

This is no longer necessary, and as of previous commit, all tests
are now in the same group (still broken into two categories). They
still run one-by-one though. Fix this by initializing child
processes in multiprocessing Pool initialization, and putting all
tests on the queue, so that tests are executed by the first idle
worker. Tests are also affinitized to different NUMA nodes using
taskset in a round-robin fashion, to prevent over-exhausting
memory on any given NUMA node.

Non-parallel tests are executed in similar fashion, but on a
separate queue which will have only one pool worker, ensuring
non-parallel execution.

Support for FreeBSD is also added to ensure that on FreeBSD, all
tests are run sequentially even for the parallel section.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest.py        |   6 +-
 test/test/autotest_runner.py | 277 +++++++++++++++++++++++++++----------------
 2 files changed, 183 insertions(+), 100 deletions(-)

diff --git a/test/test/autotest.py b/test/test/autotest.py
index ae27daef7..12997fdf0 100644
--- a/test/test/autotest.py
+++ b/test/test/autotest.py
@@ -36,8 +36,12 @@ def usage():
 
 print(cmdline)
 
+# how many workers to run tests with. FreeBSD doesn't support multiple primary
+# processes, so make it 1, otherwise make it 4. ignored for non-parallel tests
+n_processes = 1 if "bsdapp" in target else 4
+
 runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist,
-                                        test_whitelist)
+                                        test_whitelist, n_processes)
 
 runner.parallel_tests = autotest_data.parallel_test_list[:]
 runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index d6ae57e76..36941a40a 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -6,16 +6,16 @@
 from __future__ import print_function
 import StringIO
 import csv
-import multiprocessing
+from multiprocessing import Pool, Queue
 import pexpect
 import re
 import subprocess
 import sys
 import time
+import glob
+import os
 
 # wait for prompt
-
-
 def wait_prompt(child):
     try:
         child.sendline()
@@ -28,22 +28,47 @@ def wait_prompt(child):
     else:
         return False
 
-# run a test group
-# each result tuple in results list consists of:
-#   result value (0 or -1)
-#   result string
-#   test name
-#   total test run time (double)
-#   raw test log
-#   test report (if not available, should be None)
-#
-# this function needs to be outside AutotestRunner class
-# because otherwise Pool won't work (or rather it will require
-# quite a bit of effort to make it work).
+
+# get all valid NUMA nodes
+def get_numa_nodes():
+    return [
+        int(
+            re.match(r"node(\d+)", os.path.basename(node))
+            .group(1)
+        )
+        for node in glob.glob("/sys/devices/system/node/node*")
+    ]
+
+
+# find first (or any, really) CPU on a particular node, will be used to spread
+# processes around NUMA nodes to avoid exhausting memory on particular node
+def first_cpu_on_node(node_nr):
+    cpu_path = glob.glob("/sys/devices/system/node/node%d/cpu*" % node_nr)[0]
+    cpu_name = os.path.basename(cpu_path)
+    m = re.match(r"cpu(\d+)", cpu_name)
+    return int(m.group(1))
+
+
+pool_child = None  # per-process child
 
 
-def run_test_group(cmdline, prefix, target, test):
+# we initialize each worker with a queue because we need per-pool unique
+# command-line arguments, but we cannot do different arguments in an initializer
+# because the API doesn't allow per-worker initializer arguments. so, instead,
+# we will initialize with a shared queue, and dequeue command-line arguments
+# from this queue
+def pool_init(queue, result_queue):
+    global pool_child
+
+    cmdline, prefix = queue.get()
     start_time = time.time()
+    name = ("Start %s" % prefix) if prefix != "" else "Start"
+
+    # use default prefix if no prefix was specified
+    prefix_cmdline = "--file-prefix=%s" % prefix if prefix != "" else ""
+
+    # append prefix to cmdline
+    cmdline = "%s %s" % (cmdline, prefix_cmdline)
 
     # prepare logging of init
     startuplog = StringIO.StringIO()
@@ -54,24 +79,60 @@ def run_test_group(cmdline, prefix, target, test):
         print("\n%s %s\n" % ("=" * 20, prefix), file=startuplog)
         print("\ncmdline=%s" % cmdline, file=startuplog)
 
-        child = pexpect.spawn(cmdline, logfile=startuplog)
+        pool_child = pexpect.spawn(cmdline, logfile=startuplog)
 
         # wait for target to boot
-        if not wait_prompt(child):
-            child.close()
+        if not wait_prompt(pool_child):
+            pool_child.close()
+
+            result = tuple((-1,
+                            "Fail [No prompt]",
+                            name,
+                            time.time() - start_time,
+                            startuplog.getvalue(),
+                            None))
+            pool_child = None
+        else:
+            result = tuple((0,
+                            "Success",
+                            name,
+                            time.time() - start_time,
+                            startuplog.getvalue(),
+                            None))
+    except:
+        result = tuple((-1,
+                        "Fail [Can't run]",
+                        name,
+                        time.time() - start_time,
+                        startuplog.getvalue(),
+                        None))
+        pool_child = None
 
-            return -1, "Fail [No prompt]", "Start %s" % prefix,\
-                   time.time() - start_time, startuplog.getvalue(), None
+    result_queue.put(result)
 
-    except:
-        return -1, "Fail [Can't run]", "Start %s" % prefix,\
-               time.time() - start_time, startuplog.getvalue(), None
+
+# run a test
+# each result tuple in results list consists of:
+#   result value (0 or -1)
+#   result string
+#   test name
+#   total test run time (double)
+#   raw test log
+#   test report (if not available, should be None)
+#
+# this function needs to be outside AutotestRunner class because otherwise Pool
+# won't work (or rather it will require quite a bit of effort to make it work).
+def run_test(target, test):
+    global pool_child
+
+    if pool_child is None:
+        return -1, "Fail [No test process]", test["Name"], 0, "", None
 
     # create log buffer for each test
     # in multiprocessing environment, the logging would be
     # interleaved and will create a mess, hence the buffering
     logfile = StringIO.StringIO()
-    child.logfile = logfile
+    pool_child.logfile = logfile
 
     # make a note when the test started
     start_time = time.time()
@@ -81,7 +142,7 @@ def run_test_group(cmdline, prefix, target, test):
         print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
 
         # run test function associated with the test
-        result = test["Func"](child, test["Command"])
+        result = test["Func"](pool_child, test["Command"])
 
         # make a note when the test was finished
         end_time = time.time()
@@ -109,15 +170,6 @@ def run_test_group(cmdline, prefix, target, test):
         result = (-1, "Fail [Crash]", test["Name"],
                   end_time - start_time, logfile.getvalue(), None)
 
-    # regardless of whether test has crashed, try quitting it
-    try:
-        child.sendline("quit")
-        child.close()
-    # if the test crashed, just do nothing instead
-    except:
-        # nop
-        pass
-
     # return test results
     return result
 
@@ -137,7 +189,7 @@ class AutotestRunner:
     blacklist = []
     whitelist = []
 
-    def __init__(self, cmdline, target, blacklist, whitelist):
+    def __init__(self, cmdline, target, blacklist, whitelist, n_processes):
         self.cmdline = cmdline
         self.target = target
         self.binary = cmdline.split()[0]
@@ -146,6 +198,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.skipped = []
         self.parallel_tests = []
         self.non_parallel_tests = []
+        self.n_processes = n_processes
+        self.active_processes = 0
 
         # log file filename
         logfile = "%s.log" % target
@@ -159,11 +213,8 @@ def __init__(self, cmdline, target, blacklist, whitelist):
         self.csvwriter.writerow(["test_name", "test_result", "result_str"])
 
     # set up cmdline string
-    def __get_cmdline(self):
-        cmdline = self.cmdline
-
-        # affinitize startup so that tests don't fail on i686
-        cmdline = "taskset 1 " + cmdline
+    def __get_cmdline(self, cpu_nr):
+        cmdline = ("taskset -c %i " % cpu_nr) + self.cmdline
 
         return cmdline
 
@@ -241,6 +292,51 @@ def __filter_test(self, test):
 
         return True
 
+    def __run_test_group(self, test_group, worker_cmdlines):
+        group_queue = Queue()
+        init_result_queue = Queue()
+        for proc, cmdline in enumerate(worker_cmdlines):
+            prefix = "test%i" % proc if len(worker_cmdlines) > 1 else ""
+            group_queue.put(tuple((cmdline, prefix)))
+
+        # create a pool of worker threads
+        # we will initialize child in the initializer, and we don't need to
+        # close the child because when the pool worker gets destroyed, child
+        # closes the process
+        pool = Pool(processes=len(worker_cmdlines),
+                    initializer=pool_init,
+                    initargs=(group_queue, init_result_queue))
+
+        results = []
+
+        # process all initialization results
+        for _ in range(len(worker_cmdlines)):
+            self.__process_result(init_result_queue.get())
+
+        # run all tests asynchronously
+        for test in test_group:
+            result = pool.apply_async(run_test, (self.target, test))
+            results.append(result)
+
+        # tell the pool to stop all processes once done
+        pool.close()
+
+        # iterate while we have group execution results to get
+        while len(results) > 0:
+            # iterate over a copy to be able to safely delete results
+            # this iterates over a list of group results
+            for async_result in results[:]:
+                # if the thread hasn't finished yet, continue
+                if not async_result.ready():
+                    continue
+
+                res = async_result.get()
+
+                self.__process_result(res)
+
+                # remove result from results list once we're done with it
+                results.remove(async_result)
+
     # iterate over test groups and run tests associated with them
     def run_all_tests(self):
         # filter groups
@@ -253,77 +349,60 @@ def run_all_tests(self):
                    self.non_parallel_tests)
         )
 
-        # create a pool of worker threads
-        pool = multiprocessing.Pool(processes=1)
-
-        results = []
+        parallel_cmdlines = []
+        # FreeBSD doesn't have NUMA support
+        numa_nodes = get_numa_nodes()
+        if len(numa_nodes) > 0:
+            for proc in range(self.n_processes):
+                # spread cpu affinity between NUMA nodes to have less chance of
+                # running out of memory while running multiple test apps in
+                # parallel. to do that, alternate between NUMA nodes in a round
+                # robin fashion, and pick an arbitrary CPU from that node to
+                # taskset our execution to
+                numa_node = numa_nodes[self.active_processes % len(numa_nodes)]
+                cpu_nr = first_cpu_on_node(numa_node)
+                parallel_cmdlines += [self.__get_cmdline(cpu_nr)]
+                # increase number of active processes so that the next cmdline
+                # gets a different NUMA node
+                self.active_processes += 1
+        else:
+            parallel_cmdlines = [self.cmdline] * self.n_processes
 
-        # whatever happens, try to save as much logs as possible
-        try:
+        print("Running tests with %d workers" % self.n_processes)
 
-            # create table header
-            print("")
-            print("Test name".ljust(30) + "Test result".ljust(29) +
-                  "Test".center(9) + "Total".center(9))
-            print("=" * 80)
+        # create table header
+        print("")
+        print("Test name".ljust(30) + "Test result".ljust(29) +
+              "Test".center(9) + "Total".center(9))
+        print("=" * 80)
 
-            # print out skipped autotests if there were any
-            if len(self.skipped):
-                print("Skipped autotests:")
+        if len(self.skipped):
+            print("Skipped autotests:")
 
-                # print out any skipped tests
-                for result in self.skipped:
-                    # unpack result tuple
-                    test_result, result_str, test_name, _, _, _ = result
-                    self.csvwriter.writerow([test_name, test_result,
-                                             result_str])
+            # print out any skipped tests
+            for result in self.skipped:
+                # unpack result tuple
+                test_result, result_str, test_name, _, _, _ = result
+                self.csvwriter.writerow([test_name, test_result, result_str])
 
-                    t = ("%s:" % test_name).ljust(30)
-                    t += result_str.ljust(29)
-                    t += "[00m 00s]"
+                t = ("%s:" % test_name).ljust(30)
+                t += result_str.ljust(29)
+                t += "[00m 00s]"
 
-                    print(t)
+                print(t)
 
-            # make a note of tests start time
-            self.start = time.time()
+        # make a note of tests start time
+        self.start = time.time()
 
+        # whatever happens, try to save as much logs as possible
+        try:
             if len(self.parallel_tests) > 0:
                 print("Parallel autotests:")
-                # assign worker threads to run test groups
-                for test_group in self.parallel_tests:
-                    result = pool.apply_async(run_test_group,
-                                              [self.__get_cmdline(),
-                                               "",
-                                               self.target,
-                                               test_group])
-                    results.append(result)
-
-            # iterate while we have group execution results to get
-            while len(results) > 0:
-
-                # iterate over a copy to be able to safely delete results
-                # this iterates over a list of group results
-                for group_result in results[:]:
-
-                    # if the thread hasn't finished yet, continue
-                    if not group_result.ready():
-                        continue
-
-                    res = group_result.get()
-
-                    self.__process_result(res)
-
-                    # remove result from results list once we're done with it
-                    results.remove(group_result)
+                self.__run_test_group(self.parallel_tests, parallel_cmdlines)
 
             if len(self.non_parallel_tests) > 0:
                 print("Non-parallel autotests:")
-                # run non_parallel tests. they are run one by one, synchronously
-                for test_group in self.non_parallel_tests:
-                    group_result = run_test_group(
-                        self.__get_cmdline(), "", self.target, test_group)
-
-                    self.__process_result(group_result)
+                self.__run_test_group(self.non_parallel_tests, [self.cmdline])
 
             # get total run time
             cur_time = time.time()
-- 
2.14.4

[dpdk-dev] [PATCH v4 8/9] autotest: update autotest test case list

[Reshma Pattan]

Autotest is enhanced with additional test cases
being added to autotest_data.py

Removed non existing PCI autotest.

Cc: stable@dpdk.org

Signed-off-by: Reshma Pattan <reshma.pattan@intel.com>
Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 test/test/autotest_data.py | 350 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 342 insertions(+), 8 deletions(-)

diff --git a/test/test/autotest_data.py b/test/test/autotest_data.py
index c24e7bc25..3f856ff57 100644
--- a/test/test/autotest_data.py
+++ b/test/test/autotest_data.py
@@ -134,12 +134,6 @@
         "Func":    default_autotest,
         "Report":  None,
     },
-    {
-        "Name":    "PCI autotest",
-        "Command": "pci_autotest",
-        "Func":    default_autotest,
-        "Report":  None,
-    },
     {
         "Name":    "Malloc autotest",
         "Command": "malloc_autotest",
@@ -248,6 +242,291 @@
         "Func":    default_autotest,
         "Report":  None,
     },
+    {
+        "Name":    "Eventdev selftest octeontx",
+        "Command": "eventdev_selftest_octeontx",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Event ring autotest",
+        "Command": "event_ring_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Table autotest",
+        "Command": "table_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Flow classify autotest",
+        "Command": "flow_classify_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Event eth rx adapter autotest",
+        "Command": "event_eth_rx_adapter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "User delay",
+        "Command": "user_delay_us",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Rawdev autotest",
+        "Command": "rawdev_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Kvargs autotest",
+        "Command": "kvargs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Devargs autotest",
+        "Command": "devargs_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Link bonding autotest",
+        "Command": "link_bonding_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Link bonding mode4 autotest",
+        "Command": "link_bonding_mode4_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Link bonding rssconf autotest",
+        "Command": "link_bonding_rssconf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Crc autotest",
+        "Command": "crc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Distributor autotest",
+        "Command": "distributor_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Reorder autotest",
+        "Command": "reorder_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Barrier autotest",
+        "Command": "barrier_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Bitmap test",
+        "Command": "bitmap_test",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash scaling autotest",
+        "Command": "hash_scaling_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash multiwriter autotest",
+        "Command": "hash_multiwriter_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Service autotest",
+        "Command": "service_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Timer racecond autotest",
+        "Command": "timer_racecond_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Member autotest",
+        "Command": "member_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":   "Efd_autotest",
+        "Command": "efd_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Thash autotest",
+        "Command": "thash_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Hash function autotest",
+        "Command": "hash_functions_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev sw mrvl autotest",
+        "Command": "cryptodev_sw_mrvl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev dpaa2 sec autotest",
+        "Command": "cryptodev_dpaa2_sec_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev dpaa sec autotest",
+        "Command": "cryptodev_dpaa_sec_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev qat autotest",
+        "Command": "cryptodev_qat_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev aesni mb autotest",
+        "Command": "cryptodev_aesni_mb_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev openssl autotest",
+        "Command": "cryptodev_openssl_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev scheduler autotest",
+        "Command": "cryptodev_scheduler_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev aesni gcm autotest",
+        "Command": "cryptodev_aesni_gcm_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev null autotest",
+        "Command": "cryptodev_null_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev sw snow3g autotest",
+        "Command": "cryptodev_sw_snow3g_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev sw kasumi autotest",
+        "Command": "cryptodev_sw_kasumi_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Cryptodev_sw_zuc_autotest",
+        "Command": "cryptodev_sw_zuc_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Reciprocal division",
+        "Command": "reciprocal_division",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red all",
+        "Command": "red_all",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    #
+    #Please always keep all dump tests at the end and together!
+    #
+    {
+        "Name":    "Dump physmem",
+        "Command": "dump_physmem",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump memzone",
+        "Command": "dump_memzone",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump struct sizes",
+        "Command": "dump_struct_sizes",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump mempool",
+        "Command": "dump_mempool",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump malloc stats",
+        "Command": "dump_malloc_stats",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump devargs",
+        "Command": "dump_devargs",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump log types",
+        "Command": "dump_log_types",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Dump_ring",
+        "Command": "dump_ring",
+        "Func":    dump_autotest,
+        "Report":  None,
+    },
 ]
 
 # tests that should not be run when any other tests are running
@@ -259,8 +538,8 @@
         "Report":  None,
     },
     {
-        "Name":    "Eventdev sw autotest",
-        "Command": "eventdev_sw_autotest",
+        "Name":    "Eventdev selftest sw",
+        "Command": "eventdev_selftest_sw",
         "Func":    default_autotest,
         "Report":  None,
     },
@@ -312,6 +591,61 @@
         "Func":    default_autotest,
         "Report":  None,
     },
+    {
+
+        "Name":    "Pmd perf autotest",
+        "Command": "pmd_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Ring pmd perf autotest",
+        "Command": "ring_pmd_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Distributor perf autotest",
+        "Command": "distributor_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Red_perf",
+        "Command": "red_perf",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Lpm6 perf autotest",
+        "Command": "lpm6_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Lpm perf autotest",
+        "Command": "lpm_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+         "Name":    "Efd perf autotest",
+         "Command": "efd_perf_autotest",
+         "Func":    default_autotest,
+         "Report":  None,
+    },
+    {
+        "Name":    "Member perf autotest",
+        "Command": "member_perf_autotest",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
+    {
+        "Name":    "Reciprocal division perf",
+        "Command": "reciprocal_division_perf",
+        "Func":    default_autotest,
+        "Report":  None,
+    },
     #
     # Please always make sure that ring_perf is the last test!
     #
-- 
2.14.4

[dpdk-dev] [PATCH v4 9/9] mk: update make targets for classified testcases

[Reshma Pattan]

Makefiles are updated with new test case lists.
Test cases are classified as -
P1 - Main test cases,
P2 - Cryptodev/driver test cases,
P3 - Perf test cases which takes longer than 10s,
P4 - Logging/Dump test cases.

Makefile is updated with different targets
for the above classified groups.
Test cases for different targets are listed accordingly.

Cc: stable@dpdk.org

Signed-off-by: Jananee Parthasarathy <jananeex.m.parthasarathy@intel.com>
Reviewed-by: Reshma Pattan <reshma.pattan@intel.com>
---
 mk/rte.sdkroot.mk |  4 ++--
 mk/rte.sdktest.mk | 33 +++++++++++++++++++++++++++------
 2 files changed, 29 insertions(+), 8 deletions(-)

diff --git a/mk/rte.sdkroot.mk b/mk/rte.sdkroot.mk
index f43cc7829..ea3473ebf 100644
--- a/mk/rte.sdkroot.mk
+++ b/mk/rte.sdkroot.mk
@@ -68,8 +68,8 @@ config defconfig showconfigs showversion showversionum:
 cscope gtags tags etags:
 	$(Q)$(RTE_SDK)/devtools/build-tags.sh $@ $T
 
-.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage
-test test-basic test-fast test-ring test-mempool test-perf coverage:
+.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump
+test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump:
 	$(Q)$(MAKE) -f $(RTE_SDK)/mk/rte.sdktest.mk $@
 
 test: test-build
diff --git a/mk/rte.sdktest.mk b/mk/rte.sdktest.mk
index ee1fe0c7e..13d1efb6a 100644
--- a/mk/rte.sdktest.mk
+++ b/mk/rte.sdktest.mk
@@ -18,14 +18,35 @@ DIR := $(shell basename $(RTE_OUTPUT))
 #
 # test: launch auto-tests, very simple for now.
 #
-.PHONY: test test-basic test-fast test-perf coverage
+.PHONY: test test-basic test-fast test-perf test-drivers test-dump coverage
 
-PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf
-coverage: BLACKLIST=-$(PERFLIST)
-test-fast: BLACKLIST=-$(PERFLIST)
-test-perf: WHITELIST=$(PERFLIST)
+PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf,\
+         reciprocal_division,reciprocal_division_perf,lpm_perf,red_all,\
+         barrier,hash_multiwriter,timer_racecond,efd,hash_functions,\
+         eventdev_selftest_sw,member_perf,efd_perf,lpm6_perf,red_perf,\
+         distributor_perf,ring_pmd_perf,pmd_perf,ring_perf
+DRIVERSLIST=link_bonding,link_bonding_mode4,link_bonding_rssconf,\
+            cryptodev_sw_mrvl,cryptodev_dpaa2_sec,cryptodev_dpaa_sec,\
+            cryptodev_qat,cryptodev_aesni_mb,cryptodev_openssl,\
+            cryptodev_scheduler,cryptodev_aesni_gcm,cryptodev_null,\
+            cryptodev_sw_snow3g,cryptodev_sw_kasumi,cryptodev_sw_zuc
+DUMPLIST=dump_struct_sizes,dump_mempool,dump_malloc_stats,dump_devargs,\
+         dump_log_types,dump_ring,quit,dump_physmem,dump_memzone,\
+         devargs_autotest
 
-test test-basic test-fast test-perf:
+SPACESTR:=
+SPACESTR+=
+STRIPPED_PERFLIST=$(subst $(SPACESTR),,$(PERFLIST))
+STRIPPED_DRIVERSLIST=$(subst $(SPACESTR),,$(DRIVERSLIST))
+STRIPPED_DUMPLIST=$(subst $(SPACESTR),,$(DUMPLIST))
+
+coverage: BLACKLIST=-$(STRIPPED_PERFLIST)
+test-fast: BLACKLIST=-$(STRIPPED_PERFLIST),$(STRIPPED_DRIVERSLIST),$(STRIPPED_DUMPLIST)
+test-perf: WHITELIST=$(STRIPPED_PERFLIST)
+test-drivers: WHITELIST=$(STRIPPED_DRIVERSLIST)
+test-dump: WHITELIST=$(STRIPPED_DUMPLIST)
+
+test test-basic test-fast test-perf test-drivers test-dump:
 	@mkdir -p $(AUTOTEST_DIR) ; \
 	cd $(AUTOTEST_DIR) ; \
 	if [ -f $(RTE_OUTPUT)/app/test ]; then \
-- 
2.14.4