DPDK CI discussions
 help / color / mirror / Atom feed
From: Owen Hilyard <ohilyard@iol.unh.edu>
To: ci@dpdk.org
Cc: Thomas Monjalon <thomas@monjalon.net>
Subject: Re: [dpdk-ci] [PATCH V2] patch-tagging: Added tool to convert tags into dts execution file
Date: Thu, 14 Jan 2021 13:19:12 -0500	[thread overview]
Message-ID: <CAHx6DYCyQu1yjDSKg0kwTwKp0KG=NW4f65jYHFEKBUX0sdtm=g@mail.gmail.com> (raw)
In-Reply-To: <20210114175925.570013-1-ohilyard@iol.unh.edu>

[-- Attachment #1: Type: text/plain, Size: 6510 bytes --]

Please ignore this patch. I messed up directories and this patch was not
created from the correct file.

On Thu, Jan 14, 2021 at 12:59 PM <ohilyard@iol.unh.edu> wrote:

> Fixed parsing bugs with the tag parsing script
>         Parsing the patch file directly proved to be too unreliabler, so
> the approach to patch parsing was changed so that it relies instead on
> using a directory with those patches already applied in them and then use a
> git diff command to get the changed files. This avoids needing to parse the
> patch files directly and should be more reliable.
>
> Added a tool to create an execution file from tags using another execution
> file as a template
>         This tool is intended to allow a list of patch tags to be used in
> combination with an execution file containing information on crbs to only
> change which test suites are run. This tool enables using the mappings in
> the tests_for_tag.cfg file to run only necessary tests. At present, the
> mappings represent how tests are currently run at the UNH-IOL Community
> Lab, with the same tests being run for every patch, with the requested
> change from bug 511 where documentation changes do not cause any testing.
>
> Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
> ---
>  config/tests_for_tag.cfg | 19 ++++++++++++--
>  tools/patch_parser.py    | 54 ++++++++++++++++++++++++----------------
>  2 files changed, 49 insertions(+), 24 deletions(-)
>
> diff --git a/config/tests_for_tag.cfg b/config/tests_for_tag.cfg
> index 77f797f..7d95c4a 100644
> --- a/config/tests_for_tag.cfg
> +++ b/config/tests_for_tag.cfg
> @@ -5,11 +5,26 @@ core = dynamic_config,
>         mtu_update,
>         scatter,
>         stats_checks,
> +       unit_tests_mbuf
> +driver = dynamic_config,
> +       link_status_interrupt,
> +       mac_filter,
> +       mtu_update,
> +       scatter,
> +       stats_checks,
> +       unit_tests_mbuf
> +application = dynamic_config,
> +       link_status_interrupt,
> +       mac_filter,
> +       mtu_update,
> +       scatter,
>         stats_checks,
>         unit_tests_mbuf
> -
>  ; Nothing in documentation
>  documentation =
>
>  [performance]
> -core = nic_single_core_perf,
> +core = nic_single_core_perf
> +driver = nic_single_core_perf
> +application = nic_single_core_perf
> +documentation =
> diff --git a/tools/patch_parser.py b/tools/patch_parser.py
> index 80b8194..73df91f 100755
> --- a/tools/patch_parser.py
> +++ b/tools/patch_parser.py
> @@ -31,30 +31,39 @@
>  # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>  # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>  # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> -
> +import subprocess
>
>  import itertools
>  from configparser import ConfigParser
>  from typing import List, Dict, Set
>  import argparse
> +import re
>
>
>  def get_patch_files(patch_file: str) -> List[str]:
> +    file_match_pattern = re.compile(r"[\w\-\/.]+ +\| +\d+ [+\-]+")
>      with open(patch_file, 'r') as f:
> -        lines = list(itertools.takewhile(
> -            lambda line: line.strip().endswith('+') or
> line.strip().endswith('-'),
> -            itertools.dropwhile(
> -                lambda line: not line.strip().startswith("---"),
> -                f.readlines()
> +        file_lines = f.readlines()
> +        lines = list(filter(
> +            lambda line: file_match_pattern.match(line.strip()),
> +            itertools.takewhile(
> +                lambda line: not line.startswith('---'),
> +                list(
> +                    itertools.dropwhile(
> +                        lambda line: not line.strip().startswith("---"),
> +                        file_lines
> +                    )
> +                )[1:]
>              )
>          ))
>          filenames = map(lambda line: line.strip().split(' ')[0], lines)
> -        # takewhile includes the --- which starts the filenames
> -        return list(filenames)[1:]
> +        return list(filenames)
>
>
>  def get_all_files_from_patches(patch_files: List[str]) -> Set[str]:
> -    return set(itertools.chain.from_iterable(map(get_patch_files,
> patch_files)))
> +    num_patch_files: int = len(patch_files)
> +    files = subprocess.check_output(f"cd dpdk && git diff --name-only
> HEAD~{num_patch_files}", shell=True).decode('utf-8').splitlines()
> +    return set(files)
>
>
>  def parse_comma_delimited_list_from_string(mod_str: str) -> List[str]:
> @@ -80,21 +89,22 @@ def get_tags_for_patches(patch_files: Set[str],
> dir_attrs: Dict[str, Set[str]])
>      ))
>
>
> -parser = argparse.ArgumentParser(
> -    description='Takes a patch file and a config file and creates a list
> of tags for that patch')
> -parser.add_argument('config_file_path', help='The path to
> patch_parser.cfg', default='config/patch_parser.cfg')
> -parser.add_argument('patch_file_paths', help='A list of patch files',
> type=str, metavar='patch file', nargs='+')
> +if __name__ == '__main__':
> +    parser = argparse.ArgumentParser(
> +        description='Takes a patch file and a config file and creates a
> list of tags for that patch')
> +    parser.add_argument('config_file_path', help='The path to
> patch_parser.cfg', default='config/patch_parser.cfg')
> +    parser.add_argument('patch_file_paths', help='A list of patch files',
> type=str, metavar='patch file', nargs='+')
>
> -args = parser.parse_args()
> +    args = parser.parse_args()
>
> -conf_obj = ConfigParser()
> -conf_obj.read(args.config_file_path)
> +    conf_obj = ConfigParser()
> +    conf_obj.read(args.config_file_path)
>
> -patch_files = get_all_files_from_patches(args.patch_file_paths)
> -dir_attrs = get_dictionary_attributes_from_config_file(conf_obj)
> -priority_list =
> parse_comma_delimited_list_from_string(conf_obj['Priority']['priority_list'])
> +    patch_files = get_all_files_from_patches(args.patch_file_paths)
> +    dir_attrs = get_dictionary_attributes_from_config_file(conf_obj)
> +    priority_list =
> parse_comma_delimited_list_from_string(conf_obj['Priority']['priority_list'])
>
> -unordered_tags: Set[str] = get_tags_for_patches(patch_files, dir_attrs)
> -ordered_tags: List[str] = [tag for tag in priority_list if tag in
> unordered_tags]
> +    unordered_tags: Set[str] = get_tags_for_patches(patch_files,
> dir_attrs)
> +    ordered_tags: List[str] = [tag for tag in priority_list if tag in
> unordered_tags]
>
> -print("\n".join(ordered_tags))
> +    print("\n".join(ordered_tags))
> --
> 2.27.0
>
>

[-- Attachment #2: Type: text/html, Size: 7811 bytes --]

  reply	other threads:[~2021-01-14 18:19 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-04 19:45 [dpdk-ci] [PATCH] ci: added patch parser for patch files Owen Hilyard
2021-01-14 16:53 ` Owen Hilyard
2021-01-14 17:59   ` [dpdk-ci] [PATCH V2] patch-tagging: Added tool to convert tags into dts execution file ohilyard
2021-01-14 18:19     ` Owen Hilyard [this message]
2021-04-15 20:42       ` Aaron Conole
2021-04-15 21:14         ` Owen Hilyard
2021-04-15 20:38 ` [dpdk-ci] [PATCH] ci: added patch parser for patch files Aaron Conole
2021-04-15 21:11   ` [dpdk-ci] [PATCH] patch parsing: Added library for parsing " ohilyard
2021-04-15 21:37     ` Aaron Conole

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAHx6DYCyQu1yjDSKg0kwTwKp0KG=NW4f65jYHFEKBUX0sdtm=g@mail.gmail.com' \
    --to=ohilyard@iol.unh.edu \
    --cc=ci@dpdk.org \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).