From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20069.outbound.protection.outlook.com [40.107.2.69]) by dpdk.org (Postfix) with ESMTP id D49DE2661 for ; Mon, 13 Aug 2018 22:38:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=r1fk1ifLZaFiZp5CazbMc5AzhNfMpDKoO0HgqkUXj3k=; b=WpHLDsmIKU8/d/0LeyQnpdluxEx0AL3DBhgFGKspn7x3GMIQUCrNF+WgNTO4wcA1HZ57CLL2fzxWRjsvdqMQcUPYJRk9Xx4fYVOfDdHORGPX2+O29CZHSyXDj2BspNuLOXjKgutvMYk7DEAhCLPvw0ZIZnLxuIhf48NGAhZFhQM= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; Received: from mellanox.com (209.116.155.178) by VI1PR0502MB3984.eurprd05.prod.outlook.com (2603:10a6:803:24::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1017.22; Mon, 13 Aug 2018 20:38:09 +0000 From: Yongseok Koh To: Anatoly Burakov Cc: dpdk stable Date: Mon, 13 Aug 2018 13:37:45 -0700 Message-Id: <20180813203749.25171-5-yskoh@mellanox.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180813203749.25171-1-yskoh@mellanox.com> References: <20180813203749.25171-1-yskoh@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [209.116.155.178] X-ClientProxiedBy: DM5PR2001CA0017.namprd20.prod.outlook.com (2603:10b6:4:16::27) To VI1PR0502MB3984.eurprd05.prod.outlook.com (2603:10a6:803:24::26) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0d67ba46-d1c0-4024-bdd4-08d6015caec1 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989117)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:VI1PR0502MB3984; X-Microsoft-Exchange-Diagnostics: 1; VI1PR0502MB3984; 3:Uo4tn28k2BK6/998v+htpKdFTI7ZO8bC/J/DVCXtMIJtatFRP0UpZUnboiuYiUqaBMQ6i3hy+Vb2EG/PANyuOJqpPKL440UgXKOUnFpQy8REO/J4SQesCFRXu8BzAnt3I+kDpQBwN3V5ifEC/mnCyWBycWa+PyCV+j/OVQz6+LNlQoh1vxB4xGxJhN2Z7GLF28xIxjwZ5alz9sbL/P+4fQ0Uipo/Jzh6O877Dj2IKv0n+oXCe4jLRMsyXTSYEpHL; 25:A8tREisehMo7wE2exwgXqXjQXwAhrIp+Y1Ce5DgeMGf0zfFFPgY4YbzNSHJfofQ+gIzOx0fvLB1VfdVn1qpy+uxxvUx9lOiYF8l1/mGvvRi4QTeFJko9pq/KjAWb2yOHOVa6yLlZ8VObtNzkqw3L6U+5EWXf6ju+O1Px+y9Y9k99MIcD9p1dhI6sPcUDas29CGlhJmfnjxjGbEBkZu5bj9tn2iyJPHb+KtTISjVl6lCRbbPqwPKztMoUf6jh5kxAyJc2SL9fIiZC5n4ripd3MdMDszLL+MrZAYGUPohvacjjhKL88w2B/oqECrfjuWt4bIVU3Br73wnnUVXuTOo2TA==; 31:ebl/Ks1o+AKnkSOY6wXhWWD/pbnx9zwJwjK/s2OViI+aHAu67tdgrHqkri7ed0JVbMgI/Fkz8SzXPn23vy/ip1Dcs2L/WJapTjG6y06grHoQ4H9Evq88AfweGuJY9UCK26T3AeBNhQyA5cigxsHQCb4MBfvDsbAN31bqwLjEAGlzbF3qdf3CXR7YUrcXKQF2Nq0NG99LyiL/SDPr5Gf5FGtwEFeXSs+jW0giEAX4AC8= X-MS-TrafficTypeDiagnostic: VI1PR0502MB3984: X-Microsoft-Exchange-Diagnostics: 1; VI1PR0502MB3984; 20:KbbGrLcVj4231iDlKPNcDS+wAYoYP7cSRmEz7Qmlvt/ExMZCM+vq3cOLR4K2a2LVEordXTanS0gtrJDO3jL0Sfv6yHkG6Z0M+4jui6QOJJZCnbd6jJsrIs2Ll5Slf17ZGT/wSUdBksDPSl+aMeO0UpCkPmFfHon1Ryh6h4KdDVcyE1uZv3vG1TsBzX3Sr24B191WzemxQQbVjswtx3RomuaBeSelFIj7JEAX6ypmrX+8bK5WaWKMrcxPMjs/q0VeTnMqXpBv00n777++tQd8L22c4+MjqhV7X7A2vgdiqo8mNigo2wKqz5bDR331mIxGhu41mG36/IIaux/bReS+lEDUzCxSheA7jkjYWvEGXhZHhEnxhiVq+XPzXTQBoJPneLBrHEKVnf76t0F/2ILC7qMhVKdVkLaOkZ5wXtILlvCIMl74DCXP7jgGRIA5twZlb/PXCyCwLtQLgb3HYpYe8Yq1pqo5ZAIhl4Tgeie8hW4F/OIjqowt1crEyYnmu/Hj; 4:P8ihrQ/sEnitio0SbNqjjQYN+IcfWiUi2w3Q77JAl/cn1ci227raP8DTUi6hBNwx5O7GNycjbZq0hYYxh4IxeKWmr9MZ3SZS53stzCvGz+abxrrZyzatPoNtoHkNOt45KtL/mlrCyQy3mWphkRq59oiLQF6Ai7BzwmFumxppkhGqI5sCr7sjz/c7P9ylI31Evyz3fLyCW3QgScqPMxFVVAQ35ega8MAYr0mZvV/9rREs3rAYWUdzmrJbR1cCEY22uNm8X23QhZQcjii4ud3tpl5RFlUeyvDmLHtSNeB6kVOTmpo/noJBQQ/Qt5Zko6eC X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(228905959029699); X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3002001)(10201501046)(3231311)(944501410)(52105095)(93006095)(93001095)(6055026)(149027)(150027)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123562045)(20161123564045)(20161123560045)(6072148)(201708071742011)(7699016); SRVR:VI1PR0502MB3984; BCL:0; PCL:0; RULEID:; SRVR:VI1PR0502MB3984; X-Forefront-PRVS: 07630F72AD X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(366004)(346002)(376002)(136003)(39860400002)(396003)(28163001)(189003)(199004)(69234005)(3846002)(5660300001)(186003)(6116002)(305945005)(575784001)(86362001)(16526019)(6666003)(14444005)(8676002)(81156014)(97736004)(25786009)(8936002)(106356001)(66066001)(105586002)(4326008)(966005)(50226002)(81166006)(1076002)(6916009)(47776003)(48376002)(11346002)(68736007)(51416003)(7736002)(52116002)(2616005)(7696005)(478600001)(476003)(486006)(956004)(76176011)(69596002)(2906002)(1720100001)(26005)(316002)(6306002)(55016002)(53936002)(50466002)(53376002)(16586007)(21086003)(446003)(53546011)(36756003)(386003); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR0502MB3984; H:mellanox.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; VI1PR0502MB3984; 23:t2GmLglIZ7XiFngsr6ybInwTxf86giFwP+GxyIX?= =?us-ascii?Q?1/JurMPgpRxzrF3dvnyu97BYoT/hJskc1h6wV72yo64AK2EiuEndkEXEqSwE?= =?us-ascii?Q?hg29qHxMoDciZXvxKEylqfGec52NH3MZxZQVov2ChHvqZwD23anxCxRadsrk?= =?us-ascii?Q?uxoyY+ENrBxtshWJl0BpS3BGgrZwYEnbLGDc2yYxxiiqobaxQSdXKG66cXSg?= =?us-ascii?Q?1lhf7x0Eyh41hvmf+MrkcpTnCxbV/SwSwO2QGnU6y7yegrNl7gs3rcZf1uI+?= =?us-ascii?Q?DN8v3/G4IatrhFDK96T+VgX8FQYAGPppjH/3bbD4tupAHyPuTPtavLG7S1OX?= =?us-ascii?Q?zPU/9H8p6JcNO6obbC70Fbal3nLUEFBM83acrnPIfA8Dc8qdOqnpn0Km9vUY?= =?us-ascii?Q?jwh89HUoKPECuU//kdGvJMxTNMy2z943Ix3q0JQxtEWr32xn6rsztjk8XLmP?= =?us-ascii?Q?IYhHslKFByqP7KjLvTt52ajRRfajwSioetv4PH32GKszQTXNocT0H49mbz0H?= =?us-ascii?Q?kkGf73G34PybAKEVU2IplTcts8EjwLnUM8uxJ+Hmbw4Scu7K3mfKRPdPGe6y?= =?us-ascii?Q?IaKGei7i9vKD+GS4n133ltUan1lALjrYo8PeLpmiQ9G0tIPcl4LGgapDSrlN?= =?us-ascii?Q?yWpYOjaoaRCwMS8ntSkNeXc6jtg/VfY/+L84kF6A3ioWGRI2PcFb3rIAGZj3?= =?us-ascii?Q?rU16ENexwaI4QLbIjLbpIJH/DPHRV8BY0M0GEmPK1DdH9rs7WI2fXphGbyLM?= =?us-ascii?Q?90OGlFCmwD7TsmgvdsBCCnW0YAwkvK32xue9eovpPLtiiOjfh6yIbxBQJYZV?= =?us-ascii?Q?aGRK21MxupdJIgi0nQojGwwefwY/+uY6BGiAqKENHt3u6FFzp+qYOc7Tywbf?= =?us-ascii?Q?cUGreYk8O94tu7n/uAj0yqBVpe7GAH4bgBJ50JSNtCwykP0DYhYEpngpdRLS?= =?us-ascii?Q?QulOecokqKn32Ne9t8rcNOYVFWkqbGQ+XvyYWl17dBUIITuKEXtM2Wp1Nh7p?= =?us-ascii?Q?xL66+AgNUcXYAZHLXxeJeLHCvBvpYQXXcyVYJZVXUHsoFoIV1sS5CUyZHuJD?= =?us-ascii?Q?LMMc7RFRSwo0pG6FAcGpjqwstI3DJLJNlU7v2voJ4t+yef9Q5VvzUMRhui0B?= =?us-ascii?Q?rQaPnE3UfmJldRiN3etl89y12w9oxGmAsa09ruRtdbGkdq83VQj1WiaP8X2T?= =?us-ascii?Q?kgu22QNOfLWL3oazlJu2yaWUMPyni029/fYbGpE8lxtMdBvmw3FpQXTWl6Ur?= =?us-ascii?Q?Bs9T1JXQj4SC7m8nJ7lkMisY/J1VmeZ9Rkd6dqs1ibIIrZPl0YJ3D/7WEkqS?= =?us-ascii?Q?r3dI8IHchbRZpdfAP+a15j93WV6RA4lzSEKXhzxQ13tBVYAZ3PC+z35t1Zp1?= =?us-ascii?Q?3Dkxiy8n+ghjelJGrrI0NLr5APGcBQUzA0VvcNm50j5YboK/t?= X-Microsoft-Antispam-Message-Info: lNP4M5HgULxyUE9fWoNQBQ+uJy2B9tCtO8f0vPp4AU7zCkD0rkpU5ZwE3hFK5hfN/O/4azuOAnyH4D1EF1umQrEatlzGp4qnWdk21Wb34+yxF7XPu8pgbC/F8cgGP4RqHeUlKL17hqSvMygh7+pSMYVVBYAoD0kSRlwoTGjcs2bIKIYYJHmXsNAzCsc6DJlKFW13TCrAzk/WVH66743pUHeSRIspNx9RhsZXDiYpMGLWtYM4IgxCECoNkUHaibc+EnrbLPwzPDoGjllfwyFdDMTHOLyqt4niPmtOWRqHFggHFa4S09tGpo99xjtiILaDok6T84EZCCYbrZV9I386JFJ18xuK2i68dO79GcVhDr8= X-Microsoft-Exchange-Diagnostics: 1; VI1PR0502MB3984; 6:lsI3MkICSnwGfjMDbGUCZoLUtdn/kYPVDzanphpS/AbWg863eMzW3HclDMXQLWKjsK7tuxNMHxCBgoT/6AbMKmmiFlxh3P2HPsnC5FkPaMs0QAZqaxUJ605g1bxyBu10z9roU8JRP3j7NbavUDufDYZvAsVJV6eu074qR0ODeRcTUF0GVuRPNsXDptIctmbZ14c0S3NAer76e3M95O652exo2XoIed4z/4H0s6iuqmrE6G1H8fbhYZEyAmNDYSbbNd8daMtcqNQ16q/VZ4oD1Qig7OZPnWihg9sOyuefn0YcI43uH1g/cfgRcD6m5N44EquK79E/P5SWn8cxk65TbO5S8fE5KozKk+V3XitnrSL6tOYdYW8XyhWtmCsyybAxYX9Q+mpqQGzS3+QCO16WdWs8a51UxzIDmPkImsZbW2raevbgz6ntVvBqsbD1l+TXH+kZNsA+KlmNLYHZmVEmOA==; 5:h/AkcNg7iHtJ0UtLYH+EsZVR764bqtfce6Ct+w1cNE4mD3SoWcePkpioZWkpzBEMKKSPPHLLA6nOCn8C6hXc6UH0GdBNxwjFPNwEFVPZZq3OX0TYvmobORWIIfyAZa6PKn7c8XSET//njLEUTWifOG8YZonf8pOicRXT0FgVwkk=; 7:KocC/1+0mxr8TtTtAwEj/NUmrh3fPX4ys5uBhZ5MWvpRC6gCoLpjF6W1wyMeUYj5yE+RXeWAuAveORodX2XNfTkQBta0BCLVnJGsbLLy5RDZID5RtF4cZFBiGInlQjmJP5/ZCwyv8Za+SgzljWipEZ5aVibMSzc5Xaa95pOkv3n575fY7kt8cbxhPT07AqQ4/7q1iXjSbl5yJRAhBJTTCuuz25RGBBGXNqhzjVIPqa5sGCmkhRc0+R8S4I5ooY56 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Aug 2018 20:38:09.1461 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0d67ba46-d1c0-4024-bdd4-08d6015caec1 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0502MB3984 Subject: [dpdk-stable] patch 'test: improve filtering' has been queued to LTS release 17.11.4 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Aug 2018 20:38:11 -0000 Hi, FYI, your patch has been queued to LTS release 17.11.4 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 08/15/18. So please shout if anyone has objections. Thanks. Yongseok --- >>From a0a51cb787d98cd90f59c955061b5b88beb0f27d Mon Sep 17 00:00:00 2001 From: Anatoly Burakov Date: Fri, 27 Jul 2018 10:40:15 +0100 Subject: [PATCH] test: improve filtering [ upstream commit 71d97e03d812713715e409e18f2e27134cfdcd92 ] Improve code for filtering test groups. Also, move reading binary symbols into filtering stage, so that tests that are meant to be skipped are never attempted to be executed in the first place. Before running tests, print out any tests that were skipped because they weren't compiled. Signed-off-by: Anatoly Burakov --- test/test/autotest_runner.py | 118 ++++++++++++++++++++++++------------------- 1 file changed, 66 insertions(+), 52 deletions(-) diff --git a/test/test/autotest_runner.py b/test/test/autotest_runner.py index 4c02af17b..c6a9105b1 100644 --- a/test/test/autotest_runner.py +++ b/test/test/autotest_runner.py @@ -123,13 +123,6 @@ def run_test_group(cmdline, target, test_group): results.append((0, "Success", "Start %s" % test_group["Prefix"], time.time() - start_time, startuplog.getvalue(), None)) - # parse the binary for available test commands - binary = cmdline.split()[0] - stripped = 'not stripped' not in subprocess.check_output(['file', binary]) - if not stripped: - symbols = subprocess.check_output(['nm', binary]).decode('utf-8') - avail_cmds = re.findall('test_register_(\w+)', symbols) - # run all tests in test group for test in test_group["Tests"]: @@ -149,10 +142,7 @@ def run_test_group(cmdline, target, test_group): print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile) # run test function associated with the test - if stripped or test["Command"] in avail_cmds: - result = test["Func"](child, test["Command"]) - else: - result = (0, "Skipped [Not Available]") + result = test["Func"](child, test["Command"]) # make a note when the test was finished end_time = time.time() @@ -214,8 +204,10 @@ class AutotestRunner: def __init__(self, cmdline, target, blacklist, whitelist): self.cmdline = cmdline self.target = target + self.binary = cmdline.split()[0] self.blacklist = blacklist self.whitelist = whitelist + self.skipped = [] # log file filename logfile = "%s.log" % target @@ -304,53 +296,58 @@ class AutotestRunner: if i != 0: self.csvwriter.writerow([test_name, test_result, result_str]) - # this function iterates over test groups and removes each - # test that is not in whitelist/blacklist - def __filter_groups(self, test_groups): - groups_to_remove = [] - - # filter out tests from parallel test groups - for i, test_group in enumerate(test_groups): - - # iterate over a copy so that we could safely delete individual - # tests - for test in test_group["Tests"][:]: - test_id = test["Command"] - - # dump tests are specified in full e.g. "Dump_mempool" - if "_autotest" in test_id: - test_id = test_id[:-len("_autotest")] - - # filter out blacklisted/whitelisted tests - if self.blacklist and test_id in self.blacklist: - test_group["Tests"].remove(test) - continue - if self.whitelist and test_id not in self.whitelist: - test_group["Tests"].remove(test) - continue - - # modify or remove original group - if len(test_group["Tests"]) > 0: - test_groups[i] = test_group - else: - # remember which groups should be deleted - # put the numbers backwards so that we start - # deleting from the end, not from the beginning - groups_to_remove.insert(0, i) + # this function checks individual test and decides if this test should be in + # the group by comparing it against whitelist/blacklist. it also checks if + # the test is compiled into the binary, and marks it as skipped if necessary + def __filter_test(self, test): + test_cmd = test["Command"] + test_id = test_cmd + + # dump tests are specified in full e.g. "Dump_mempool" + if "_autotest" in test_id: + test_id = test_id[:-len("_autotest")] + + # filter out blacklisted/whitelisted tests + if self.blacklist and test_id in self.blacklist: + return False + if self.whitelist and test_id not in self.whitelist: + return False + + # if test wasn't compiled in, remove it as well + + # parse the binary for available test commands + stripped = 'not stripped' not in \ + subprocess.check_output(['file', self.binary]) + if not stripped: + symbols = subprocess.check_output(['nm', + self.binary]).decode('utf-8') + avail_cmds = re.findall('test_register_(\w+)', symbols) + + if test_cmd not in avail_cmds: + # notify user + result = 0, "Skipped [Not compiled]", test_id, 0, "", None + self.skipped.append(tuple(result)) + return False - # remove test groups that need to be removed - for i in groups_to_remove: - del test_groups[i] + return True - return test_groups + def __filter_group(self, group): + group["Tests"] = list(filter(self.__filter_test, group["Tests"])) + return len(group["Tests"]) > 0 # iterate over test groups and run tests associated with them def run_all_tests(self): # filter groups - self.parallel_test_groups = \ - self.__filter_groups(self.parallel_test_groups) - self.non_parallel_test_groups = \ - self.__filter_groups(self.non_parallel_test_groups) + # for each test group, check all tests against the filter, then remove + # all groups that don't have any tests + self.parallel_test_groups = list( + filter(self.__filter_group, + self.parallel_test_groups) + ) + self.non_parallel_test_groups = list( + filter(self.__filter_group, + self.non_parallel_test_groups) + ) # create a pool of worker threads pool = multiprocessing.Pool(processes=1) @@ -366,6 +363,23 @@ class AutotestRunner: "Test".center(9) + "Total".center(9)) print("=" * 80) + # print out skipped autotests if there were any + if len(self.skipped): + print("Skipped autotests:") + + # print out any skipped tests + for result in self.skipped: + # unpack result tuple + test_result, result_str, test_name, _, _, _ = result + self.csvwriter.writerow([test_name, test_result, + result_str]) + + t = ("%s:" % test_name).ljust(30) + t += result_str.ljust(29) + t += "[00m 00s]" + + print(t) + # make a note of tests start time self.start = time.time() -- 2.11.0