From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 42F40A0C40 for ; Thu, 15 Apr 2021 23:37:27 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A37316248D; Thu, 15 Apr 2021 23:37:27 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 32B58162483 for ; Thu, 15 Apr 2021 23:37:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618522644; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UCGM6u+VZ+nCDMKcFQ+ymNof4/fVOkENpMDqqJ6XnNg=; b=axPf+jI3pKEI91eheDS7Iz2zCmqRk1jZFMzCxB9Klqhs0XmibRr7eEj5uhFA2b719Vt3nJ jkbKgkygim5HOf33aMs7QpGopxqJrqHug0VOJDPQX+7dyE5yH6DqPYd7OX9eFhjoGkFRUk eCWSZN05u2+FDV6vJ+jMhsVf8q6q+n8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-145-yrccH34rPVKaikdc6caePA-1; Thu, 15 Apr 2021 17:37:22 -0400 X-MC-Unique: yrccH34rPVKaikdc6caePA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 024EF87A83B; Thu, 15 Apr 2021 21:37:21 +0000 (UTC) Received: from dhcp-25.97.bos.redhat.com (ovpn-115-147.rdu2.redhat.com [10.10.115.147]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6E5905C5B5; Thu, 15 Apr 2021 21:37:20 +0000 (UTC) From: Aaron Conole To: ohilyard@iol.unh.edu Cc: ci@dpdk.org References: <20210415211112.77161-1-ohilyard@iol.unh.edu> Date: Thu, 15 Apr 2021 17:37:19 -0400 In-Reply-To: <20210415211112.77161-1-ohilyard@iol.unh.edu> (ohilyard@iol.unh.edu's message of "Thu, 15 Apr 2021 17:11:12 -0400") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=aconole@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-ci] [PATCH] patch parsing: Added library for parsing patch files X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" ohilyard@iol.unh.edu writes: > From: Owen Hilyard > > Due to the number of edge cases, there were difficulties doing the parsin= g using pattern matching, so a dedicated library, whatthepatch, is now used= for parsing the patches to ensure correctness. > > Also added the default behavior that if no tags are given, create_new_exe= cution_file_from_tags.py will act as though all possible tags were given. T= his is to allow patches where no tags could be found to still have test cov= erage. > --- Missing sign-off. Also, please reformat the commit message, preferring 75 column width. Unlike code, commit logs are very sensitive when viewing history, etc. > config/tests_for_tag.cfg | 30 +++++ > requirements.txt | 1 + > tools/create_new_execution_file_from_tags.py | 114 +++++++++++++++++++ > tools/patch_parser.py | 85 ++++++++++---- > 4 files changed, 205 insertions(+), 25 deletions(-) > create mode 100644 config/tests_for_tag.cfg > create mode 100644 requirements.txt > create mode 100755 tools/create_new_execution_file_from_tags.py > > diff --git a/config/tests_for_tag.cfg b/config/tests_for_tag.cfg > new file mode 100644 > index 0000000..7d95c4a > --- /dev/null > +++ b/config/tests_for_tag.cfg > @@ -0,0 +1,30 @@ > +[functional] > +core =3D dynamic_config, > + link_status_interrupt, > + mac_filter, > + mtu_update, > + scatter, > + stats_checks, > + unit_tests_mbuf > +driver =3D dynamic_config, > + link_status_interrupt, > + mac_filter, > + mtu_update, > + scatter, > + stats_checks, > + unit_tests_mbuf > +application =3D dynamic_config, > + link_status_interrupt, > + mac_filter, > + mtu_update, > + scatter, > + stats_checks, > + unit_tests_mbuf > +; Nothing in documentation > +documentation =3D > + > +[performance] > +core =3D nic_single_core_perf > +driver =3D nic_single_core_perf > +application =3D nic_single_core_perf > +documentation =3D > diff --git a/requirements.txt b/requirements.txt > new file mode 100644 > index 0000000..f20067d > --- /dev/null > +++ b/requirements.txt > @@ -0,0 +1 @@ > +whatthepatch=3D=3D1.0.2 > \ No newline at end of file > diff --git a/tools/create_new_execution_file_from_tags.py b/tools/create_= new_execution_file_from_tags.py > new file mode 100755 > index 0000000..d1d4447 > --- /dev/null > +++ b/tools/create_new_execution_file_from_tags.py > @@ -0,0 +1,114 @@ > +#!/usr/bin/env python3 > + > +# BSD LICENSE > +# > +# Copyright(c) 2020 Intel Corporation. All rights reserved. > +# Copyright =C2=A9 2018[, 2019] The University of New Hampshire. All rig= hts reserved. > +# All rights reserved. > +# > +# Redistribution and use in source and binary forms, with or without > +# modification, are permitted provided that the following conditions > +# are met: > +# > +# * Redistributions of source code must retain the above copyright > +# notice, this list of conditions and the following disclaimer. > +# * Redistributions in binary form must reproduce the above copyright > +# notice, this list of conditions and the following disclaimer in > +# the documentation and/or other materials provided with the > +# distribution. > +# * Neither the name of Intel Corporation nor the names of its > +# contributors may be used to endorse or promote products derived > +# from this software without specific prior written permission. > +# > +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR > +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT > +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, > +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, > +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY > +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE > +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. > +import sys > +from enum import Enum > + > +import itertools > +from configparser import ConfigParser > +from typing import List, Dict, Set > +import argparse > + > + > +def parse_comma_delimited_list_from_string(mod_str: str) -> List[str]: > + return list(map(str.strip, mod_str.split(','))) > + > + > +def map_tags_to_tests(tags: List[str], test_map: Dict[str, List[str]]) -= > Set[str]: > + """ > + Returns a list that is the union of all of the map lookups. > + """ > + try: > + return set( > + filter(lambda test: test !=3D '', set(itertools.chain.from_i= terable(map(lambda tag: test_map[tag], tags))))) > + except KeyError as e: > + print(f'Tag {e} is not present in tests_for_tag.cfg') > + exit(1) > + > + > +class TestingType(Enum): > + functional =3D 'functional' > + performance =3D 'performance' > + > + def __str__(self): > + return self.value > + > + > +if __name__ =3D=3D '__main__': > + parser =3D argparse.ArgumentParser( > + description=3D'Take a template execution file and add the releva= nt tests' > + ' for the given tags to it, creating a new file.') > + parser.add_argument('config_file_path', help=3D'The path to tests_fo= r_tag.cfg', default=3D'config/tests_for_tag.cfg') > + parser.add_argument('template_execution_file', help=3D'The path to t= he execution file to use as a template') > + parser.add_argument('output_path', help=3D'The path to the output ex= ecution file') > + parser.add_argument('testing_type', type=3DTestingType, choices=3Dli= st(TestingType), > + help=3D'What type of testing to create an execut= ion file for') > + parser.add_argument('tags', metavar=3D'tag', type=3Dstr, nargs=3D'*'= , help=3D'The tags to create an execution file for.') > + > + args =3D parser.parse_args() > + > + tag_to_test_map_parser =3D ConfigParser() > + tag_to_test_map_parser.read(args.config_file_path) > + > + template_execution_file_parser =3D ConfigParser() > + template_execution_file_parser.read(args.template_execution_file) > + > + test_map =3D {key: parse_comma_delimited_list_from_string(value.stri= p()) for key, value in > + tag_to_test_map_parser[str(args.testing_type)].items()} > + > + tests =3D map_tags_to_tests(args.tags, test_map) > + > + try: > + output_file =3D open(args.output_path, 'x') > + except FileExistsError: > + output_file =3D open(args.output_path, 'w') > + > + for execution_plan in template_execution_file_parser: > + # The DEFAULT section is always present and contains top-level i= tems, so it needs to be ignored > + if execution_plan !=3D 'DEFAULT': > + test_allowlist =3D parse_comma_delimited_list_from_string( > + template_execution_file_parser[execution_plan]['test_sui= tes']) > + tests_to_run =3D list(set(test_allowlist).intersection(tests= )) > + tests_to_run.sort() > + template_execution_file_parser[execution_plan]['test_suites'= ] =3D ", ".join(tests_to_run) > + > + if args.testing_type =3D=3D TestingType.functional: > + template_execution_file_parser[execution_plan]['paramete= rs'] +=3D ':func=3Dtrue:perf=3Dfalse' > + elif args.testing_type =3D=3D TestingType.performance: > + template_execution_file_parser[execution_plan]['paramete= rs'] +=3D ':func=3Dfalse:perf=3Dtrue' > + else: > + # This code should be unreachable, since this is checked= at the top of the file > + print("Fatal error: testing type is neither performance = nor functional", file=3Dsys.stderr) > + exit(1) > + > + template_execution_file_parser.write(output_file) > diff --git a/tools/patch_parser.py b/tools/patch_parser.py > index 01fc55d..cc91cd2 100755 > --- a/tools/patch_parser.py > +++ b/tools/patch_parser.py > @@ -1,27 +1,58 @@ > #!/usr/bin/env python3 > =20 > -import itertools > -import sys > +import argparse > from configparser import ConfigParser > from typing import List, Dict, Set > =20 > +import itertools > +# BSD LICENSE > +# > +# Copyright(c) 2020 Intel Corporation. All rights reserved. > +# Copyright =C2=A9 2018[, 2019] The University of New Hampshire. All rig= hts reserved. > +# All rights reserved. > +# > +# Redistribution and use in source and binary forms, with or without > +# modification, are permitted provided that the following conditions > +# are met: > +# > +# * Redistributions of source code must retain the above copyright > +# notice, this list of conditions and the following disclaimer. > +# * Redistributions in binary form must reproduce the above copyright > +# notice, this list of conditions and the following disclaimer in > +# the documentation and/or other materials provided with the > +# distribution. > +# * Neither the name of Intel Corporation nor the names of its > +# contributors may be used to endorse or promote products derived > +# from this software without specific prior written permission. > +# > +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR > +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT > +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, > +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT > +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, > +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY > +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE > +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. > +import sys > =20 > -def get_patch_files(patch_file: str) -> List[str]: > +try: > + import whatthepatch > +except ImportError: > + print("Please install whatthepatch, a patch parsing library", file= =3Dsys.stderr) > + exit(1) > + > + > +def get_changed_files_in_patch(patch_file: str) -> List[str]: > with open(patch_file, 'r') as f: > - lines =3D list(itertools.takewhile( > - lambda line: line.strip().endswith('+') or line.strip().ends= with('-'), > - itertools.dropwhile( > - lambda line: not line.strip().startswith("---"), > - f.readlines() > - ) > - )) > - filenames =3D map(lambda line: line.strip().split(' ')[0], lines= ) > - # takewhile includes the --- which starts the filenames > - return list(filenames)[1:] > + filenames =3D map(lambda diff: diff.header.new_path, whatthepatc= h.parse_patch(f.read())) > + return list(filenames) > =20 > =20 > def get_all_files_from_patches(patch_files: List[str]) -> Set[str]: > - return set(itertools.chain.from_iterable(map(get_patch_files, patch_= files))) > + return set(itertools.chain.from_iterable(map(get_changed_files_in_pa= tch, patch_files))) > =20 > =20 > def parse_comma_delimited_list_from_string(mod_str: str) -> List[str]: > @@ -47,18 +78,22 @@ def get_tags_for_patches(patch_files: Set[str], dir_a= ttrs: Dict[str, Set[str]]) > )) > =20 > =20 > -if len(sys.argv) < 3: > - print("usage: patch_parser.py ...") > - exit(1) > +if __name__ =3D=3D '__main__': > + parser =3D argparse.ArgumentParser( > + description=3D'Takes a patch file and a config file and creates = a list of tags for that patch') > + parser.add_argument('config_file_path', help=3D'The path to patch_pa= rser.cfg', default=3D'config/patch_parser.cfg') > + parser.add_argument('patch_file_paths', help=3D'A list of patch file= s', type=3Dstr, metavar=3D'patch file', nargs=3D'+') > + > + args =3D parser.parse_args() > =20 > -conf_obj =3D ConfigParser() > -conf_obj.read(sys.argv[1]) > + conf_obj =3D ConfigParser() > + conf_obj.read(args.config_file_path) > =20 > -patch_files =3D get_all_files_from_patches(sys.argv[2:]) > -dir_attrs =3D get_dictionary_attributes_from_config_file(conf_obj) > -priority_list =3D parse_comma_delimited_list_from_string(conf_obj['Prior= ity']['priority_list']) > + patch_files =3D get_all_files_from_patches(args.patch_file_paths) > + dir_attrs =3D get_dictionary_attributes_from_config_file(conf_obj) > + priority_list =3D parse_comma_delimited_list_from_string(conf_obj['P= riority']['priority_list']) > =20 > -unordered_tags: Set[str] =3D get_tags_for_patches(patch_files, dir_attrs= ) > -ordered_tags: List[str] =3D [tag for tag in priority_list if tag in unor= dered_tags] > + unordered_tags: Set[str] =3D get_tags_for_patches(patch_files, dir_a= ttrs) > + ordered_tags: List[str] =3D [tag for tag in priority_list if tag in = unordered_tags] > =20 > -print("\n".join(ordered_tags)) > + print("\n".join(ordered_tags))