From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10075.outbound.protection.outlook.com [40.107.1.75]) by dpdk.org (Postfix) with ESMTP id 227761B452 for ; Sat, 16 Feb 2019 17:03:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6HzEE/8H3E2QPs3yR0DdggJJ8UZqdJhJG/EAuzi8pns=; b=o0mAc7p+LIo68Njj9qr1fM2ferPgAxSzxxMyt5IEtdQPmkGEQ3BMyzQxj6uuodYWt2E/D08OSTlcMnkTCt11b8Z5H2YKHPPPeY7HBTMxjSog1mvwxX+QvTjGjC/9A3LjZJW11VczdaHAayBVj+lzA0m8EuFCtflNflkQoHLsh6w= Received: from VI1PR05MB4269.eurprd05.prod.outlook.com (52.133.12.22) by VI1PR05MB6366.eurprd05.prod.outlook.com (20.179.26.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1622.19; Sat, 16 Feb 2019 16:03:52 +0000 Received: from VI1PR05MB4269.eurprd05.prod.outlook.com ([fe80::64cb:93f8:918:f1a1]) by VI1PR05MB4269.eurprd05.prod.outlook.com ([fe80::64cb:93f8:918:f1a1%7]) with mapi id 15.20.1622.018; Sat, 16 Feb 2019 16:03:52 +0000 From: Ali Alnubani To: "ci@dpdk.org" CC: Thomas Monjalon , "ferruh.yigit@intel.com" , "jplsek@iol.unh.edu" , Ori Kam Thread-Topic: [PATCH v3] add script to decide best tree match for patches Thread-Index: AQHUxhE1DeE9Sf4CYEOY4FrnwW/TZQ== Date: Sat, 16 Feb 2019 16:03:52 +0000 Message-ID: <20190216160206.11957-1-alialnu@mellanox.com> References: <20190212144828.18122-1-alialnu@mellanox.com> In-Reply-To: <20190212144828.18122-1-alialnu@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.11.0 x-clientproxiedby: PR2P264CA0016.FRAP264.PROD.OUTLOOK.COM (2603:10a6:101::28) To VI1PR05MB4269.eurprd05.prod.outlook.com (2603:10a6:803:40::22) authentication-results: spf=none (sender IP is ) smtp.mailfrom=alialnu@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [2001:4b98:dc0:51:216:3eff:feac:53b] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ef2b2ccd-4c1d-4794-e657-08d694285836 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600110)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:VI1PR05MB6366; x-ms-traffictypediagnostic: VI1PR05MB6366: x-ms-exchange-purlcount: 1 x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-exchange-diagnostics: =?iso-8859-1?Q?1; VI1PR05MB6366; 23:Qx1AiMZXemCQ7Pi4IEl4ElM/sCj7ZzmTmUDTmiW?= =?iso-8859-1?Q?EFBmjeLGGQpSzXgwIS07dHmJCr9VakmVEnSIV5UNJFoJRzNMU2mdVRMUuM?= =?iso-8859-1?Q?YfKhBQ0M9BFPyITvYSL3f0bglgQ4otuK+WCO4BnrPeRs7KBRxiazCQPuDR?= =?iso-8859-1?Q?TUU4DEQ+wSfT4z80u6zd+g+0MlCo8PlbW7nZDQEJ4osPj8Y5xTVRwnKdlc?= =?iso-8859-1?Q?fmAadx92ZrX5LEuJMw47mb1lZHgPYKgdqgjCKzfkrkBARbAB4iuQMURhv3?= =?iso-8859-1?Q?IWQoTMHQqM6GYHZ2P9AFBS4gc+0vGZLXL5OOFHLai7vP4/Rtu6ti/RgGOT?= =?iso-8859-1?Q?IKquvh6cHyJD+Phf1llVf/9uV7N9jF7Y0EwC4gZkrpwqTbgQd5gL90Siuq?= =?iso-8859-1?Q?0x0QY9nrbGdemJAQDsBIj0cutVzOZVax3C4ZTvw5JxRIqeWq6FRawnJ6Gj?= =?iso-8859-1?Q?rpwS/QULkjMqnXplizR3LPr71MXmtQyV+B20gQb3V9ot7uXnWfG4F61H4m?= =?iso-8859-1?Q?eaJAtTvoWlf/IbOKcD2Y0HLJBjDb7D6gLZPysHLLFClVfbOgBS0L7OFbhm?= =?iso-8859-1?Q?Bup2egLU0UXH2zKroq9bjNyUPbyJildm3D/s1FnZjElSbL6roGU5jGY3bq?= =?iso-8859-1?Q?ixQZFJZo4hBZZUdDkrErZmaLQLwJs1xQ0uebOBa0d4l6T7qfuBrFjAWNSe?= =?iso-8859-1?Q?LwIs7+tda9JB57mUdrFoZU2JJ+qERHL6yrFi7U+3bnUwE4si6I2Dpdgvil?= =?iso-8859-1?Q?5PSpLKmrKyRjpbpl9YbVb7yNelRrQlA5D5PqWiA1oNQhfhDBTzBzpBadPu?= =?iso-8859-1?Q?vyQ0eTgASMT0XTt8dqpRuJj7WIgY/QOPTujQx1tV5/OEZeLSjwMytw4T4Y?= =?iso-8859-1?Q?fbHEIe8nHHEzEwIaFo1uOEAB0jEnF8lbeu28o2Jttx2pdOWrFV7XhUbB5K?= =?iso-8859-1?Q?HYIHS8D+km2i3GNQCMK4Any+EwQ5c3Yzc8vhnHKcd4N+96hv4u0/rWw559?= =?iso-8859-1?Q?j44XnVSlhgf3r3buLR+Faz47i/AjETvrd0bUI/SPUBr+kHTWUqcAatucVz?= =?iso-8859-1?Q?jLvhApyuziR/DKnkXY/jcYn7ndgZ1zpBxkBhe3GxLmBaIjMUtXpexq0V+4?= =?iso-8859-1?Q?F2/VZexJ5/u9I0LyPs+1UXuwb3ADouLTZ2icpzAV6jqzNPjWkDAFCI71Gk?= =?iso-8859-1?Q?WL8zgC6S2xSBHxoBqyToTubdtSuZYQ7samh+OFuZ6GbvWDAKbicXJXr+Hy?= =?iso-8859-1?Q?R4yozooZtQAQpImxZcCcbtjMfrf51FGMoC2ydMWD2cHOnLewM0iVwxpHGZ?= =?iso-8859-1?Q?KcZL0ByQS3t3XQmf//EPqKUqkDqCRcfYdppKvRjXxIzMBNF8sl4Ryg0w0G?= =?iso-8859-1?Q?vOIiaMl9yihNI7PunyhOhJfGIG1Ga?= x-microsoft-antispam-prvs: x-forefront-prvs: 0950706AC1 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(376002)(346002)(136003)(366004)(39860400002)(199004)(189003)(186003)(105586002)(102836004)(107886003)(305945005)(2906002)(7736002)(4326008)(8936002)(6916009)(25786009)(2351001)(1076003)(71200400001)(71190400001)(81166006)(106356001)(81156014)(1730700003)(8676002)(99286004)(6512007)(14454004)(6306002)(50226002)(256004)(6116002)(5640700003)(14444005)(53936002)(2501003)(386003)(86362001)(6506007)(6436002)(52116002)(76176011)(46003)(97736004)(36756003)(5660300002)(478600001)(316002)(486006)(54906003)(6486002)(966005)(68736007)(446003)(2616005)(11346002)(476003); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR05MB6366; H:VI1PR05MB4269.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: 1UQ7yhK0D8lhwRJXOL9Ckz6Nbxpyz6uLYH7gh/QTVtCLEWOIkF2YXvSSil+UZe02u03QSSU1PTuQhlW61032YqvO5NGkW6X7o3/eLeK8TZ8Xa5D1GZNb6TM42HY+ScR+z1RykXcbY2/4beETpcVRXiwQFLqfKTutx802rKsACsusKtaCPwu0cofhCWUJT+EK+901epD1RC0PP4UN4oaU7UAwApIanoKzaegIY8LD6ZDoAjNacKhH0X4R8z5Bz/yv+V08s9jfc7u6BllNFROb/3RNMuISCt1KGjFPRIGPYd+VfwWyUYFu8WF4ldDto8n5mpf4UHCV8OJUKjVJQfkqwnwpMVgY9hwCjBCeETFZj5s1JPL3by9ZDG5jNGXTNxZPPwwCzkKXBiOnDMV/F2/QLMvv5Ip84a6Zh1xmhLoSu0A= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: ef2b2ccd-4c1d-4794-e657-08d694285836 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Feb 2019 16:03:51.7728 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB6366 Subject: [dpdk-ci] [PATCH v3] add script to decide best tree match for patches X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Feb 2019 16:03:54 -0000 The script can be used to get the trees that best match a patch or a series. Signed-off-by: Ali Alnubani Signed-off-by: Ori Kam --- Changes in v3: - Fixed PEP8 violations. - Fixed variable names. - Removed unused get_subject() method. tools/guess-git-tree.py | 254 ++++++++++++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 254 insertions(+) create mode 100755 tools/guess-git-tree.py diff --git a/tools/guess-git-tree.py b/tools/guess-git-tree.py new file mode 100755 index 0000000..f3f69df --- /dev/null +++ b/tools/guess-git-tree.py @@ -0,0 +1,254 @@ +#!/usr/bin/env python + +# SPDX-License-Identifier: (BSD-3-Clause AND GPL-2.0-or-later AND MIT) +# Copyright 2019 Mellanox Technologies, Ltd + +import os +import sys +import re +import argparse +import copy +import fnmatch + +from requests.exceptions import HTTPError + +from git_pw import config +from git_pw import api +from git_pw import utils + +""" +This script uses the git-pw API to retrieve Patchwork's series/patches, +and find a list of trees/repos that best match the series/patch. + +The rules on which matches are based, are taken from the MAINTAINERS file, +and currently only based on the paths of the changed files. Results can be +improved by adding more information to the MAINTAINERS file. + +TODO: + - Match using the subject of the patch/series. + - Add a configuration file to specify the priority of each tree. + +Configurations: +The script uses tokens for authentication. +If the arguments pw_{server,project,token} aren't passed, the environment +variables PW_{SERVER,PROJECT,TOKEN} should be set. If not, the script will= try +to load the git configurations pw.{server,project,token}. + +Example usage: + ./guess-git-tree.py --command list_trees_for_series 2054 + ./guess-git-tree.py --command list_trees_for_patch 2054 + +The output will be a list of trees sorted based on number of matches, +with the first line having the highest count. +""" + +CONF =3D config.CONF +CONF.debug =3D False + +MAINTAINERS_FILE_PATH =3D os.environ.get('MAINTAINERS_FILE_PATH') +if not MAINTAINERS_FILE_PATH: + print('MAINTAINERS_FILE_PATH is not set.') + sys.exit(1) +RULES =3D {} + +ignored_files_re =3D re.compile(r'^doc/|\.sh$|\.py$') + + +def configure_git_pw(args=3DNone): + """Configure git-pw.""" + conf =3D {} + conf_keys =3D ['server', 'project', 'token'] + for key in conf_keys: + value =3D getattr(args, 'pw_{}'.format(key)) + if not value: + print('--pw_{} is a required git-pw configuration'.format(key)= ) + sys.exit(1) + else: + setattr(CONF, key, value) + + +def find_filenames(diff): + """Find file changes in a given diff. + + Source: https://github.com/getpatchwork/patchwork/blob/master/patchwor= k/parser.py + Changes from source: + - Moved _filename_re into the method. + - Reduced newlines. + """ + _filename_re =3D re.compile(r'^(---|\+\+\+) (\S+)') + # normalise spaces + diff =3D diff.replace('\r', '') + diff =3D diff.strip() + '\n' + filenames =3D {} + for line in diff.split('\n'): + if len(line) <=3D 0: + continue + filename_match =3D _filename_re.match(line) + if not filename_match: + continue + filename =3D filename_match.group(2) + if filename.startswith('/dev/null'): + continue + filename =3D '/'.join(filename.split('/')[1:]) + filenames[filename] =3D True + filenames =3D sorted(filenames.keys()) + return filenames + + +def construct_rules(): + """Build a dictionary of rules from the MAINTAINERS file.""" + with open(MAINTAINERS_FILE_PATH) as fd: + maintainers =3D fd.read() + # Split into blocks of text for easier search. + maintainers =3D maintainers.split('\n\n') + + # Extract blocks that have a tree and files. + tree_file_blocks =3D [_item for _item in maintainers + if 'T: git://dpdk.org' in _item and 'F: ' in _item= ] + _dict =3D {} + for _item in tree_file_blocks: + # Get the tree url. + tree_match =3D re.search(r'T: (git://dpdk\.org[^\n]+)', _item) + if tree_match: + tree =3D tree_match.group(1) + else: + continue + if tree not in _dict: + _dict[tree] =3D {} + _dict[tree]['paths'] =3D [] + paths =3D re.findall(r'F: ([^\n]+)', _item) + _paths =3D copy.deepcopy(paths) + for path in paths: + # Remove don't-care paths + if ignored_files_re.search(path): + _paths.remove(path) + _dict[tree]['paths'] +=3D _paths + return _dict + + +def find_matches(files): + """Find trees that the changed files in a patch match, + and stop at first match for each file.""" + matches =3D [] + for _file in files: + if ignored_files_re.search(_file): + continue + match_found =3D False + for tree in RULES.keys(): + for rule in RULES[tree]['paths']: + if rule.endswith('/'): + rule =3D '{}*'.format(rule) + if fnmatch.fnmatch(_file, rule): + matches.append(tree) + match_found =3D True + break + if match_found: + break + return matches + + +def get_ordered_matches(matches): + """Order matches by occurrences.""" + match_counts =3D {item: matches.count(item) for item in matches} + return sorted(match_counts, key=3Dmatch_counts.get, reverse=3DTrue) + + +def list_trees_for_patch(patch): + """Find matching trees for a specific patch. + For a patch to match a tree, at least one changed + path has to match that tree. + """ + files =3D find_filenames(patch['diff']) + + matches =3D find_matches(files) + return matches + + +def list_trees_for_series(series): + """Find matching trees for a series.""" + patch_list =3D series['patches'] + + matches =3D [] + + for patch in patch_list: + matches =3D matches + \ + list_trees_for_patch(api_get('patches', patch['id'])) + + return matches + + +def parse_args(): + """Parse command-line arguments.""" + parser =3D argparse.ArgumentParser() + git_pw_conf_parser =3D parser.add_argument_group('git-pw configuration= s') + options_parser =3D parser.add_argument_group('optional arguments') + + options_parser.add_argument( + '--command', + choices=3D( + 'list_trees_for_patch', + 'list_trees_for_series'), + required=3DTrue, help=3D'command to perform on patch/series') + + git_pw_conf_parser.add_argument( + '--pw_server', type=3Dstr, + default=3Dos.environ.get( + 'PW_SERVER', utils.git_config('pw.server')), + help=3D'PW.SERVER') + git_pw_conf_parser.add_argument( + '--pw_project', type=3Dstr, + default=3Dos.environ.get( + 'PW_PROJECT', utils.git_config('pw.project')), + help=3D'PW.PROJECT') + git_pw_conf_parser.add_argument( + '--pw_token', type=3Dstr, + default=3Dos.environ.get('PW_TOKEN', utils.git_config('pw.toke= n')), + help=3D'PW.TOKEN') + + parser.add_argument( + 'id', type=3Dint, help=3D'patch/series id') + + args =3D parser.parse_args() + + return args + + +def main(): + """Main procedure.""" + args =3D parse_args() + configure_git_pw(args) + + command =3D args.command + _id =3D args.id + + global RULES + RULES =3D construct_rules() + + tree_list =3D [] + + if command =3D=3D 'list_trees_for_patch': + patch =3D api_get('patches', _id) + tree_list =3D list_trees_for_patch(patch) + + elif command =3D=3D 'list_trees_for_series': + series =3D api_get('series', _id) + tree_list =3D list_trees_for_series(series) + + tree_list =3D get_ordered_matches(tree_list) + + print('{}'.format('\n'.join(tree_list))) + + +def api_get(resource_type, resource_id): + """Retrieve an API resource.""" + try: + return api.detail(resource_type, resource_id) + except HTTPError as err: + if '404' in str(err): + sys.exit(1) + else: + raise + + +if __name__ =3D=3D '__main__': + main() --=20 2.11.0