* [dpdk-stable] [PATCH v2 01/10] autotest: fix printing
[not found] <1531498088-20221-1-git-send-email-reshma.pattan@intel.com>
@ 2018-07-13 16:07 ` Reshma Pattan
2018-07-13 16:08 ` [dpdk-stable] [PATCH v2 02/10] autotest: fix invalid code on reports Reshma Pattan
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Reshma Pattan @ 2018-07-13 16:07 UTC (permalink / raw)
To: reshma.pattan; +Cc: Anatoly Burakov, john.mcnamara, stable
From: Anatoly Burakov <anatoly.burakov@intel.com>
Previously, printing was done using tuple syntax, which caused
output to appear as a tuple as opposed to being one string. Fix
this by using addition operator instead.
Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
test/test/autotest_runner.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index a692f0697..b09b57876 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -247,7 +247,7 @@ def __process_results(self, results):
# don't print out total time every line, it's the same anyway
if i == len(results) - 1:
- print(result,
+ print(result +
"[%02dm %02ds]" % (total_time / 60, total_time % 60))
else:
print(result)
@@ -332,8 +332,8 @@ def run_all_tests(self):
# create table header
print("")
- print("Test name".ljust(30), "Test result".ljust(29),
- "Test".center(9), "Total".center(9))
+ print("Test name".ljust(30) + "Test result".ljust(29) +
+ "Test".center(9) + "Total".center(9))
print("=" * 80)
# make a note of tests start time
--
2.14.4
^ permalink raw reply [flat|nested] 6+ messages in thread* [dpdk-stable] [PATCH v2 02/10] autotest: fix invalid code on reports
[not found] <1531498088-20221-1-git-send-email-reshma.pattan@intel.com>
2018-07-13 16:07 ` [dpdk-stable] [PATCH v2 01/10] autotest: fix printing Reshma Pattan
@ 2018-07-13 16:08 ` Reshma Pattan
2018-07-13 16:08 ` [dpdk-stable] [PATCH v2 03/10] autotest: make autotest runner python 2/3 compliant Reshma Pattan
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Reshma Pattan @ 2018-07-13 16:08 UTC (permalink / raw)
To: reshma.pattan; +Cc: Anatoly Burakov, stable
From: Anatoly Burakov <anatoly.burakov@intel.com>
There are no reports defined for any test, so this codepath was
never triggered, but it's still wrong because it's referencing
variables that aren't there. Fix it by passing target into the
test function, and reference correct log variable.
Fixes: e2cc79b75d9f ("app: rework autotest.py")
Cc: stable@dpdk.org
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
test/test/autotest_runner.py | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index b09b57876..bdc32da5d 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -41,7 +41,7 @@ def wait_prompt(child):
# quite a bit of effort to make it work).
-def run_test_group(cmdline, test_group):
+def run_test_group(cmdline, target, test_group):
results = []
child = None
start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group):
# make a note when the test was finished
end_time = time.time()
+ log = logfile.getvalue()
+
# append test data to the result tuple
- result += (test["Name"], end_time - start_time,
- logfile.getvalue())
+ result += (test["Name"], end_time - start_time, log)
# call report function, if any defined, and supply it with
# target and complete log for test run
if test["Report"]:
- report = test["Report"](self.target, log)
+ report = test["Report"](target, log)
# append report to results tuple
result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self):
for test_group in self.parallel_test_groups:
result = pool.apply_async(run_test_group,
[self.__get_cmdline(test_group),
+ self.target,
test_group])
results.append(result)
@@ -367,7 +369,7 @@ def run_all_tests(self):
# run non_parallel tests. they are run one by one, synchronously
for test_group in self.non_parallel_test_groups:
group_result = run_test_group(
- self.__get_cmdline(test_group), test_group)
+ self.__get_cmdline(test_group), self.target, test_group)
self.__process_results(group_result)
--
2.14.4
^ permalink raw reply [flat|nested] 6+ messages in thread* [dpdk-stable] [PATCH v2 03/10] autotest: make autotest runner python 2/3 compliant
[not found] <1531498088-20221-1-git-send-email-reshma.pattan@intel.com>
2018-07-13 16:07 ` [dpdk-stable] [PATCH v2 01/10] autotest: fix printing Reshma Pattan
2018-07-13 16:08 ` [dpdk-stable] [PATCH v2 02/10] autotest: fix invalid code on reports Reshma Pattan
@ 2018-07-13 16:08 ` Reshma Pattan
2018-07-13 16:08 ` [dpdk-stable] [PATCH v2 04/10] autotest: visually separate different test categories Reshma Pattan
2018-07-13 16:08 ` [dpdk-stable] [PATCH v2 05/10] autotest: improve filtering Reshma Pattan
4 siblings, 0 replies; 6+ messages in thread
From: Reshma Pattan @ 2018-07-13 16:08 UTC (permalink / raw)
To: reshma.pattan; +Cc: Anatoly Burakov, john.mcnamara, stable
From: Anatoly Burakov <anatoly.burakov@intel.com>
Autotest runner was still using python 2-style print syntax. Fix
it by importing print function from the future, and fix the calls
to be python-3 style.
Fixes: 54ca545dce4b ("make python scripts python2/3 compliant")
Cc: john.mcnamara@intel.com
Cc: stable@dpdk.org
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
test/test/autotest_runner.py | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index bdc32da5d..f6b669a2e 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -3,6 +3,7 @@
# The main logic behind running autotests in parallel
+from __future__ import print_function
import StringIO
import csv
import multiprocessing
@@ -52,8 +53,8 @@ def run_test_group(cmdline, target, test_group):
# prepare logging of init
startuplog = StringIO.StringIO()
- print >>startuplog, "\n%s %s\n" % ("=" * 20, test_group["Prefix"])
- print >>startuplog, "\ncmdline=%s" % cmdline
+ print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog)
+ print("\ncmdline=%s" % cmdline, file=startuplog)
child = pexpect.spawn(cmdline, logfile=startuplog)
@@ -117,7 +118,7 @@ def run_test_group(cmdline, target, test_group):
try:
# print test name to log buffer
- print >>logfile, "\n%s %s\n" % ("-" * 20, test["Name"])
+ print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
# run test function associated with the test
if stripped or test["Command"] in avail_cmds:
--
2.14.4
^ permalink raw reply [flat|nested] 6+ messages in thread* [dpdk-stable] [PATCH v2 04/10] autotest: visually separate different test categories
[not found] <1531498088-20221-1-git-send-email-reshma.pattan@intel.com>
` (2 preceding siblings ...)
2018-07-13 16:08 ` [dpdk-stable] [PATCH v2 03/10] autotest: make autotest runner python 2/3 compliant Reshma Pattan
@ 2018-07-13 16:08 ` Reshma Pattan
2018-07-13 16:08 ` [dpdk-stable] [PATCH v2 05/10] autotest: improve filtering Reshma Pattan
4 siblings, 0 replies; 6+ messages in thread
From: Reshma Pattan @ 2018-07-13 16:08 UTC (permalink / raw)
To: reshma.pattan; +Cc: Anatoly Burakov, stable
From: Anatoly Burakov <anatoly.burakov@intel.com>
Help visually identify parallel vs. non-parallel autotests.
Cc: stable@dpdk.org
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
test/test/autotest_runner.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index f6b669a2e..d9d5f7a97 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -341,6 +341,7 @@ def run_all_tests(self):
# make a note of tests start time
self.start = time.time()
+ print("Parallel autotests:")
# assign worker threads to run test groups
for test_group in self.parallel_test_groups:
result = pool.apply_async(run_test_group,
@@ -367,6 +368,7 @@ def run_all_tests(self):
# remove result from results list once we're done with it
results.remove(group_result)
+ print("Non-parallel autotests:")
# run non_parallel tests. they are run one by one, synchronously
for test_group in self.non_parallel_test_groups:
group_result = run_test_group(
--
2.14.4
^ permalink raw reply [flat|nested] 6+ messages in thread* [dpdk-stable] [PATCH v2 05/10] autotest: improve filtering
[not found] <1531498088-20221-1-git-send-email-reshma.pattan@intel.com>
` (3 preceding siblings ...)
2018-07-13 16:08 ` [dpdk-stable] [PATCH v2 04/10] autotest: visually separate different test categories Reshma Pattan
@ 2018-07-13 16:08 ` Reshma Pattan
4 siblings, 0 replies; 6+ messages in thread
From: Reshma Pattan @ 2018-07-13 16:08 UTC (permalink / raw)
To: reshma.pattan; +Cc: Anatoly Burakov, stable
From: Anatoly Burakov <anatoly.burakov@intel.com>
Improve code for filtering test groups. Also, move reading binary
symbols into filtering stage, so that tests that are meant to be
skipped are never attempted to be executed in the first place.
Before running tests, print out any tests that were skipped because
they weren't compiled.
Cc: stable@dpdk.org
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
test/test/autotest_runner.py | 118 ++++++++++++++++++++++++-------------------
1 file changed, 66 insertions(+), 52 deletions(-)
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index d9d5f7a97..c98ec2a57 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -95,13 +95,6 @@ def run_test_group(cmdline, target, test_group):
results.append((0, "Success", "Start %s" % test_group["Prefix"],
time.time() - start_time, startuplog.getvalue(), None))
- # parse the binary for available test commands
- binary = cmdline.split()[0]
- stripped = 'not stripped' not in subprocess.check_output(['file', binary])
- if not stripped:
- symbols = subprocess.check_output(['nm', binary]).decode('utf-8')
- avail_cmds = re.findall('test_register_(\w+)', symbols)
-
# run all tests in test group
for test in test_group["Tests"]:
@@ -121,10 +114,7 @@ def run_test_group(cmdline, target, test_group):
print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile)
# run test function associated with the test
- if stripped or test["Command"] in avail_cmds:
- result = test["Func"](child, test["Command"])
- else:
- result = (0, "Skipped [Not Available]")
+ result = test["Func"](child, test["Command"])
# make a note when the test was finished
end_time = time.time()
@@ -186,8 +176,10 @@ class AutotestRunner:
def __init__(self, cmdline, target, blacklist, whitelist):
self.cmdline = cmdline
self.target = target
+ self.binary = cmdline.split()[0]
self.blacklist = blacklist
self.whitelist = whitelist
+ self.skipped = []
# log file filename
logfile = "%s.log" % target
@@ -276,53 +268,58 @@ def __process_results(self, results):
if i != 0:
self.csvwriter.writerow([test_name, test_result, result_str])
- # this function iterates over test groups and removes each
- # test that is not in whitelist/blacklist
- def __filter_groups(self, test_groups):
- groups_to_remove = []
-
- # filter out tests from parallel test groups
- for i, test_group in enumerate(test_groups):
-
- # iterate over a copy so that we could safely delete individual
- # tests
- for test in test_group["Tests"][:]:
- test_id = test["Command"]
-
- # dump tests are specified in full e.g. "Dump_mempool"
- if "_autotest" in test_id:
- test_id = test_id[:-len("_autotest")]
-
- # filter out blacklisted/whitelisted tests
- if self.blacklist and test_id in self.blacklist:
- test_group["Tests"].remove(test)
- continue
- if self.whitelist and test_id not in self.whitelist:
- test_group["Tests"].remove(test)
- continue
-
- # modify or remove original group
- if len(test_group["Tests"]) > 0:
- test_groups[i] = test_group
- else:
- # remember which groups should be deleted
- # put the numbers backwards so that we start
- # deleting from the end, not from the beginning
- groups_to_remove.insert(0, i)
+ # this function checks individual test and decides if this test should be in
+ # the group by comparing it against whitelist/blacklist. it also checks if
+ # the test is compiled into the binary, and marks it as skipped if necessary
+ def __filter_test(self, test):
+ test_cmd = test["Command"]
+ test_id = test_cmd
+
+ # dump tests are specified in full e.g. "Dump_mempool"
+ if "_autotest" in test_id:
+ test_id = test_id[:-len("_autotest")]
+
+ # filter out blacklisted/whitelisted tests
+ if self.blacklist and test_id in self.blacklist:
+ return False
+ if self.whitelist and test_id not in self.whitelist:
+ return False
+
+ # if test wasn't compiled in, remove it as well
+
+ # parse the binary for available test commands
+ stripped = 'not stripped' not in \
+ subprocess.check_output(['file', self.binary])
+ if not stripped:
+ symbols = subprocess.check_output(['nm',
+ self.binary]).decode('utf-8')
+ avail_cmds = re.findall('test_register_(\w+)', symbols)
+
+ if test_cmd not in avail_cmds:
+ # notify user
+ result = 0, "Skipped [Not compiled]", test_id, 0, "", None
+ self.skipped.append(tuple(result))
+ return False
- # remove test groups that need to be removed
- for i in groups_to_remove:
- del test_groups[i]
+ return True
- return test_groups
+ def __filter_group(self, group):
+ group["Tests"] = list(filter(self.__filter_test, group["Tests"]))
+ return len(group["Tests"]) > 0
# iterate over test groups and run tests associated with them
def run_all_tests(self):
# filter groups
- self.parallel_test_groups = \
- self.__filter_groups(self.parallel_test_groups)
- self.non_parallel_test_groups = \
- self.__filter_groups(self.non_parallel_test_groups)
+ # for each test group, check all tests against the filter, then remove
+ # all groups that don't have any tests
+ self.parallel_test_groups = list(
+ filter(self.__filter_group,
+ self.parallel_test_groups)
+ )
+ self.non_parallel_test_groups = list(
+ filter(self.__filter_group,
+ self.non_parallel_test_groups)
+ )
# create a pool of worker threads
pool = multiprocessing.Pool(processes=1)
@@ -338,6 +335,23 @@ def run_all_tests(self):
"Test".center(9) + "Total".center(9))
print("=" * 80)
+ # print out skipped autotests if there were any
+ if len(self.skipped):
+ print("Skipped autotests:")
+
+ # print out any skipped tests
+ for result in self.skipped:
+ # unpack result tuple
+ test_result, result_str, test_name, _, _, _ = result
+ self.csvwriter.writerow([test_name, test_result,
+ result_str])
+
+ t = ("%s:" % test_name).ljust(30)
+ t += result_str.ljust(29)
+ t += "[00m 00s]"
+
+ print(t)
+
# make a note of tests start time
self.start = time.time()
--
2.14.4
^ permalink raw reply [flat|nested] 6+ messages in thread
* [dpdk-stable] [PATCH v2 02/10] autotest: fix invalid code on reports
[not found] <cover.1528404133.git.anatoly.burakov@intel.com>
@ 2018-07-13 16:20 ` Reshma Pattan
0 siblings, 0 replies; 6+ messages in thread
From: Reshma Pattan @ 2018-07-13 16:20 UTC (permalink / raw)
To: thomas, dev; +Cc: anatoly.burakov, jananeex.m.parthasarathy, stable
There are no reports defined for any test, so this codepath was
never triggered, but it's still wrong because it's referencing
variables that aren't there. Fix it by passing target into the
test function, and reference correct log variable.
Fixes: e2cc79b75d9f ("app: rework autotest.py")
Cc: stable@dpdk.org
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
test/test/autotest_runner.py | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py
index b09b57876..bdc32da5d 100644
--- a/test/test/autotest_runner.py
+++ b/test/test/autotest_runner.py
@@ -41,7 +41,7 @@ def wait_prompt(child):
# quite a bit of effort to make it work).
-def run_test_group(cmdline, test_group):
+def run_test_group(cmdline, target, test_group):
results = []
child = None
start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group):
# make a note when the test was finished
end_time = time.time()
+ log = logfile.getvalue()
+
# append test data to the result tuple
- result += (test["Name"], end_time - start_time,
- logfile.getvalue())
+ result += (test["Name"], end_time - start_time, log)
# call report function, if any defined, and supply it with
# target and complete log for test run
if test["Report"]:
- report = test["Report"](self.target, log)
+ report = test["Report"](target, log)
# append report to results tuple
result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self):
for test_group in self.parallel_test_groups:
result = pool.apply_async(run_test_group,
[self.__get_cmdline(test_group),
+ self.target,
test_group])
results.append(result)
@@ -367,7 +369,7 @@ def run_all_tests(self):
# run non_parallel tests. they are run one by one, synchronously
for test_group in self.non_parallel_test_groups:
group_result = run_test_group(
- self.__get_cmdline(test_group), test_group)
+ self.__get_cmdline(test_group), self.target, test_group)
self.__process_results(group_result)
--
2.14.4
^ permalink raw reply [flat|nested] 6+ messages in thread