From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 79346A0A02 for ; Thu, 14 Jan 2021 19:19:50 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 539D0140DEB; Thu, 14 Jan 2021 19:19:50 +0100 (CET) Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) by mails.dpdk.org (Postfix) with ESMTP id CEFE7140DE7 for ; Thu, 14 Jan 2021 19:19:49 +0100 (CET) Received: by mail-oi1-f171.google.com with SMTP id p5so6872007oif.7 for ; Thu, 14 Jan 2021 10:19:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=phzSNBK2AfwfCrJWIcBpUb1hBjV4ru8S2W0GoxYRRx4=; b=GIieDlkxZWIZi7bAOf6XUltrjoRydQSaiN+C/TLrvGj9PGyH6gas/CtO94oYyTxdtc K1KVve9hFkG+CxBjmKSFG0WQkv+H79ejFxkK5Y8QjjQJ9eCaarASiaqM9qOuAgm63QGq RllGdbLZO7/MtCRXERIcaw0ReCbRelA81vPZM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=phzSNBK2AfwfCrJWIcBpUb1hBjV4ru8S2W0GoxYRRx4=; b=ZyCrkQL/sc6CdtZog1YAHsApzQrSkhnsevDGXbABvfzc72cWflcKDu3mFLnqM87FLj 1itQ/GKAZ8bxLW2R7HfF+MSc+wh8wT+SgICNwfT5QCIOInDOfIeW9rnqOIcjgXtdnYij xz1XdHfsknvjUDwGEC/lWmLQ1YH8JKFxjV3r1xv27NKCv9QLgHxaMmYTSUEx4xb5IAa+ I1xAyQT7io6qbicAGzqEJzBKby30URTlb9iel8JxXvx5URxQwLKnjrH3/TNpeql1PrNc 3y2/e/4NqJ//fJBoBZZuir0Ad8LFIx2ggvuPcmwhBUca8/HbWdMlbLeyVoYliW2SH/x6 5uig== X-Gm-Message-State: AOAM533Ph0h61bJPIb7Qzv2Lc6+P66agUW/RM4JqJU3z6QfG3YN5Wzc4 H7hstsx4BaRsu/zZ4GGh8FUxKwgiNnUArpB/7iV7n+CbC38EUg== X-Google-Smtp-Source: ABdhPJwW1dKZ/0nryb8iChl4M0gZhF2imcgXiE9xCZ32Be3GJYrhxjiiw0ifJ5HHx/BzWc09hlj/OGSFskKCYJUUYv8= X-Received: by 2002:aca:4d8b:: with SMTP id a133mr3466118oib.79.1610648388872; Thu, 14 Jan 2021 10:19:48 -0800 (PST) MIME-Version: 1.0 References: <20210114175925.570013-1-ohilyard@iol.unh.edu> In-Reply-To: <20210114175925.570013-1-ohilyard@iol.unh.edu> From: Owen Hilyard Date: Thu, 14 Jan 2021 13:19:12 -0500 Message-ID: To: ci@dpdk.org Cc: Thomas Monjalon Content-Type: multipart/alternative; boundary="00000000000013c9c505b8e04d26" Subject: Re: [dpdk-ci] [PATCH V2] patch-tagging: Added tool to convert tags into dts execution file X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" --00000000000013c9c505b8e04d26 Content-Type: text/plain; charset="UTF-8" Please ignore this patch. I messed up directories and this patch was not created from the correct file. On Thu, Jan 14, 2021 at 12:59 PM wrote: > Fixed parsing bugs with the tag parsing script > Parsing the patch file directly proved to be too unreliabler, so > the approach to patch parsing was changed so that it relies instead on > using a directory with those patches already applied in them and then use a > git diff command to get the changed files. This avoids needing to parse the > patch files directly and should be more reliable. > > Added a tool to create an execution file from tags using another execution > file as a template > This tool is intended to allow a list of patch tags to be used in > combination with an execution file containing information on crbs to only > change which test suites are run. This tool enables using the mappings in > the tests_for_tag.cfg file to run only necessary tests. At present, the > mappings represent how tests are currently run at the UNH-IOL Community > Lab, with the same tests being run for every patch, with the requested > change from bug 511 where documentation changes do not cause any testing. > > Signed-off-by: Owen Hilyard > --- > config/tests_for_tag.cfg | 19 ++++++++++++-- > tools/patch_parser.py | 54 ++++++++++++++++++++++++---------------- > 2 files changed, 49 insertions(+), 24 deletions(-) > > diff --git a/config/tests_for_tag.cfg b/config/tests_for_tag.cfg > index 77f797f..7d95c4a 100644 > --- a/config/tests_for_tag.cfg > +++ b/config/tests_for_tag.cfg > @@ -5,11 +5,26 @@ core = dynamic_config, > mtu_update, > scatter, > stats_checks, > + unit_tests_mbuf > +driver = dynamic_config, > + link_status_interrupt, > + mac_filter, > + mtu_update, > + scatter, > + stats_checks, > + unit_tests_mbuf > +application = dynamic_config, > + link_status_interrupt, > + mac_filter, > + mtu_update, > + scatter, > stats_checks, > unit_tests_mbuf > - > ; Nothing in documentation > documentation = > > [performance] > -core = nic_single_core_perf, > +core = nic_single_core_perf > +driver = nic_single_core_perf > +application = nic_single_core_perf > +documentation = > diff --git a/tools/patch_parser.py b/tools/patch_parser.py > index 80b8194..73df91f 100755 > --- a/tools/patch_parser.py > +++ b/tools/patch_parser.py > @@ -31,30 +31,39 @@ > # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE > # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. > - > +import subprocess > > import itertools > from configparser import ConfigParser > from typing import List, Dict, Set > import argparse > +import re > > > def get_patch_files(patch_file: str) -> List[str]: > + file_match_pattern = re.compile(r"[\w\-\/.]+ +\| +\d+ [+\-]+") > with open(patch_file, 'r') as f: > - lines = list(itertools.takewhile( > - lambda line: line.strip().endswith('+') or > line.strip().endswith('-'), > - itertools.dropwhile( > - lambda line: not line.strip().startswith("---"), > - f.readlines() > + file_lines = f.readlines() > + lines = list(filter( > + lambda line: file_match_pattern.match(line.strip()), > + itertools.takewhile( > + lambda line: not line.startswith('---'), > + list( > + itertools.dropwhile( > + lambda line: not line.strip().startswith("---"), > + file_lines > + ) > + )[1:] > ) > )) > filenames = map(lambda line: line.strip().split(' ')[0], lines) > - # takewhile includes the --- which starts the filenames > - return list(filenames)[1:] > + return list(filenames) > > > def get_all_files_from_patches(patch_files: List[str]) -> Set[str]: > - return set(itertools.chain.from_iterable(map(get_patch_files, > patch_files))) > + num_patch_files: int = len(patch_files) > + files = subprocess.check_output(f"cd dpdk && git diff --name-only > HEAD~{num_patch_files}", shell=True).decode('utf-8').splitlines() > + return set(files) > > > def parse_comma_delimited_list_from_string(mod_str: str) -> List[str]: > @@ -80,21 +89,22 @@ def get_tags_for_patches(patch_files: Set[str], > dir_attrs: Dict[str, Set[str]]) > )) > > > -parser = argparse.ArgumentParser( > - description='Takes a patch file and a config file and creates a list > of tags for that patch') > -parser.add_argument('config_file_path', help='The path to > patch_parser.cfg', default='config/patch_parser.cfg') > -parser.add_argument('patch_file_paths', help='A list of patch files', > type=str, metavar='patch file', nargs='+') > +if __name__ == '__main__': > + parser = argparse.ArgumentParser( > + description='Takes a patch file and a config file and creates a > list of tags for that patch') > + parser.add_argument('config_file_path', help='The path to > patch_parser.cfg', default='config/patch_parser.cfg') > + parser.add_argument('patch_file_paths', help='A list of patch files', > type=str, metavar='patch file', nargs='+') > > -args = parser.parse_args() > + args = parser.parse_args() > > -conf_obj = ConfigParser() > -conf_obj.read(args.config_file_path) > + conf_obj = ConfigParser() > + conf_obj.read(args.config_file_path) > > -patch_files = get_all_files_from_patches(args.patch_file_paths) > -dir_attrs = get_dictionary_attributes_from_config_file(conf_obj) > -priority_list = > parse_comma_delimited_list_from_string(conf_obj['Priority']['priority_list']) > + patch_files = get_all_files_from_patches(args.patch_file_paths) > + dir_attrs = get_dictionary_attributes_from_config_file(conf_obj) > + priority_list = > parse_comma_delimited_list_from_string(conf_obj['Priority']['priority_list']) > > -unordered_tags: Set[str] = get_tags_for_patches(patch_files, dir_attrs) > -ordered_tags: List[str] = [tag for tag in priority_list if tag in > unordered_tags] > + unordered_tags: Set[str] = get_tags_for_patches(patch_files, > dir_attrs) > + ordered_tags: List[str] = [tag for tag in priority_list if tag in > unordered_tags] > > -print("\n".join(ordered_tags)) > + print("\n".join(ordered_tags)) > -- > 2.27.0 > > --00000000000013c9c505b8e04d26 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Please ignore this patch. I messed up directories and this= patch was not created from the correct file.=C2=A0

On Thu, Jan 14, 2021 at = 12:59 PM <ohilyard@iol.unh.edu> wrote:
Fi= xed parsing bugs with the tag parsing script
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Parsing the patch file directly proved to be to= o unreliabler, so the approach to patch parsing was changed so that it reli= es instead on using a directory with those patches already applied in them = and then use a git diff command to get the changed files. This avoids needi= ng to parse the patch files directly and should be more reliable.

Added a tool to create an execution file from tags using another execution = file as a template
=C2=A0 =C2=A0 =C2=A0 =C2=A0 This tool is intended to allow a list of patch = tags to be used in combination with an execution file containing informatio= n on crbs to only change which test suites are run. This tool enables using= the mappings in the tests_for_tag.cfg file to run only necessary tests. At= present, the mappings represent how tests are currently run at the UNH-IOL= Community Lab, with the same tests being run for every patch, with the req= uested change from bug 511 where documentation changes do not cause any tes= ting.

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
=C2=A0config/tests_for_tag.cfg | 19 ++++++++++++--
=C2=A0tools/patch_parser.py=C2=A0 =C2=A0 | 54 ++++++++++++++++++++++++-----= -----------
=C2=A02 files changed, 49 insertions(+), 24 deletions(-)

diff --git a/config/tests_for_tag.cfg b/config/tests_for_tag.cfg
index 77f797f..7d95c4a 100644
--- a/config/tests_for_tag.cfg
+++ b/config/tests_for_tag.cfg
@@ -5,11 +5,26 @@ core =3D dynamic_config,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 mtu_update,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 scatter,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 stats_checks,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0unit_tests_mbuf
+driver =3D dynamic_config,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0link_status_interrupt,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0mac_filter,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0mtu_update,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0scatter,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0stats_checks,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0unit_tests_mbuf
+application =3D dynamic_config,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0link_status_interrupt,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0mac_filter,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0mtu_update,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0scatter,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 stats_checks,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 unit_tests_mbuf
-
=C2=A0; Nothing in documentation
=C2=A0documentation =3D

=C2=A0[performance]
-core =3D nic_single_core_perf,
+core =3D nic_single_core_perf
+driver =3D nic_single_core_perf
+application =3D nic_single_core_perf
+documentation =3D
diff --git a/tools/patch_parser.py b/tools/patch_parser.py
index 80b8194..73df91f 100755
--- a/tools/patch_parser.py
+++ b/tools/patch_parser.py
@@ -31,30 +31,39 @@
=C2=A0# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT=
=C2=A0# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE U= SE
=C2=A0# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE= .
-
+import subprocess

=C2=A0import itertools
=C2=A0from configparser import ConfigParser
=C2=A0from typing import List, Dict, Set
=C2=A0import argparse
+import re


=C2=A0def get_patch_files(patch_file: str) -> List[str]:
+=C2=A0 =C2=A0 file_match_pattern =3D re.compile(r"[\w\-\/.]+ +\| +\d+= [+\-]+")
=C2=A0 =C2=A0 =C2=A0with open(patch_file, 'r') as f:
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 lines =3D list(itertools.takewhile(
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 lambda line: line.strip().endswi= th('+') or line.strip().endswith('-'),
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 itertools.dropwhile(
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 lambda line: not l= ine.strip().startswith("---"),
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 f.readlines()
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 file_lines =3D f.readlines()
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 lines =3D list(filter(
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 lambda line: file_match_pattern.= match(line.strip()),
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 itertools.takewhile(
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 lambda line: not l= ine.startswith('---'),
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 list(
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 iter= tools.dropwhile(
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 lambda line: not line.strip().startswith("---"),
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 file_lines
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ) +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 )[1:]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0filenames =3D map(lambda line: line.strip= ().split(' ')[0], lines)
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 # takewhile includes the --- which starts the = filenames
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 return list(filenames)[1:]
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 return list(filenames)


=C2=A0def get_all_files_from_patches(patch_files: List[str]) -> Set[str]= :
-=C2=A0 =C2=A0 return set(itertools.chain.from_iterable(map(get_patch_files= , patch_files)))
+=C2=A0 =C2=A0 num_patch_files: int =3D len(patch_files)
+=C2=A0 =C2=A0 files =3D subprocess.check_output(f"cd dpdk && = git diff --name-only HEAD~{num_patch_files}", shell=3DTrue).decode(= 9;utf-8').splitlines()
+=C2=A0 =C2=A0 return set(files)


=C2=A0def parse_comma_delimited_list_from_string(mod_str: str) -> List[s= tr]:
@@ -80,21 +89,22 @@ def get_tags_for_patches(patch_files: Set[str], dir_att= rs: Dict[str, Set[str]])
=C2=A0 =C2=A0 =C2=A0))


-parser =3D argparse.ArgumentParser(
-=C2=A0 =C2=A0 description=3D'Takes a patch file and a config file and = creates a list of tags for that patch')
-parser.add_argument('config_file_path', help=3D'The path to pa= tch_parser.cfg', default=3D'config/patch_parser.cfg')
-parser.add_argument('patch_file_paths', help=3D'A list of patc= h files', type=3Dstr, metavar=3D'patch file', nargs=3D'+= 9;)
+if __name__ =3D=3D '__main__':
+=C2=A0 =C2=A0 parser =3D argparse.ArgumentParser(
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 description=3D'Takes a patch file and a co= nfig file and creates a list of tags for that patch')
+=C2=A0 =C2=A0 parser.add_argument('config_file_path', help=3D'= The path to patch_parser.cfg', default=3D'config/patch_parser.cfg&#= 39;)
+=C2=A0 =C2=A0 parser.add_argument('patch_file_paths', help=3D'= A list of patch files', type=3Dstr, metavar=3D'patch file', nar= gs=3D'+')

-args =3D parser.parse_args()
+=C2=A0 =C2=A0 args =3D parser.parse_args()

-conf_obj =3D ConfigParser()
-conf_obj.read(args.config_file_path)
+=C2=A0 =C2=A0 conf_obj =3D ConfigParser()
+=C2=A0 =C2=A0 conf_obj.read(args.config_file_path)

-patch_files =3D get_all_files_from_patches(args.patch_file_paths)
-dir_attrs =3D get_dictionary_attributes_from_config_file(conf_obj)
-priority_list =3D parse_comma_delimited_list_from_string(conf_obj['Pri= ority']['priority_list'])
+=C2=A0 =C2=A0 patch_files =3D get_all_files_from_patches(args.patch_file_p= aths)
+=C2=A0 =C2=A0 dir_attrs =3D get_dictionary_attributes_from_config_file(con= f_obj)
+=C2=A0 =C2=A0 priority_list =3D parse_comma_delimited_list_from_string(con= f_obj['Priority']['priority_list'])

-unordered_tags: Set[str] =3D get_tags_for_patches(patch_files, dir_attrs)<= br> -ordered_tags: List[str] =3D [tag for tag in priority_list if tag in unorde= red_tags]
+=C2=A0 =C2=A0 unordered_tags: Set[str] =3D get_tags_for_patches(patch_file= s, dir_attrs)
+=C2=A0 =C2=A0 ordered_tags: List[str] =3D [tag for tag in priority_list if= tag in unordered_tags]

-print("\n".join(ordered_tags))
+=C2=A0 =C2=A0 print("\n".join(ordered_tags))
--
2.27.0

--00000000000013c9c505b8e04d26--