From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5DB84A0524 for ; Fri, 5 Feb 2021 21:37:31 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 31CCB40682; Fri, 5 Feb 2021 21:37:31 +0100 (CET) Received: from mail-pf1-f225.google.com (mail-pf1-f225.google.com [209.85.210.225]) by mails.dpdk.org (Postfix) with ESMTP id E6B5A4067B for ; Fri, 5 Feb 2021 21:37:29 +0100 (CET) Received: by mail-pf1-f225.google.com with SMTP id j12so5091413pfj.12 for ; Fri, 05 Feb 2021 12:37:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=PKyVkOcXytzoJXfhQyCaz+iM5XMNLJAmrP3NiD0G0+o=; b=XFsZw+dwmVgBVdeFFYCAhb76i3eZEEYM/fSW3d9cZAbDZCoH5t6diN1s+ifEICzYjd kRI09ZSc0+ToKaH/U/iOFRQiaXC7Sa6WrDW9vaXPs1qAVjbRJC+GPOyarGscSw4A0JC8 mZPI43KR2nBhovMW3yXWjiNrpYmoK21I/0Szc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=PKyVkOcXytzoJXfhQyCaz+iM5XMNLJAmrP3NiD0G0+o=; b=XtV9/flCAb3FhzFD6pSblUcOuITluRGw1WQpBeb3Oi3s9kdzyuWOlIwrtHBY9uoRGu HBMJD77O+XPu8TpNcGxcy/ApwNYvLH5A/FknN9fDlhn79mtXPOdWU0oM7MziMToNEOd4 b0SWuNAab3hsj7pyxtQSAUMijsw0u1ZMCQ/fTQDbcurbdVzIfrw9qrCXsHZMAb+5uoJ/ aKAtfP5BMG33bbpAY2RsVcj0DN9QMoHr8A6SN8jcb516tCUI0+SRsrYwfimSJUWYBlJL GYKuPJ7Ye45QBvs9ZbHayKn5RLhSnAkn0HVpBvng54GC46fzC+yuimuFKfoDxhi6VQF5 UlnQ== X-Gm-Message-State: AOAM533PnpGaOf5wc7FysAL/6FEw3l/+TKiEaQfVA1zX8t36gVx+Kqa3 YNBQJbb3d9V0nMhyEw0hmW0Uqg14c76xJOWAbfdIAZlJGBpX41/5b5Q+R8j7UyAPaXshoWnylrh up4r4PpO2RqPw4IZv82gNTabd7Xsi5mHrDxm9sW/5NI+zEnO2UpfKF8SS4ArHOBHZ0nyXfeSX X-Google-Smtp-Source: ABdhPJz5/MRJexpEjhLL5/eAIiIGe600OOqzhw8+enpI2Ed2KttXqZqKBEZAGIeQmmnH7h+3JCVlI3Ar3G1H X-Received: by 2002:a62:76d3:0:b029:1d9:8e88:9276 with SMTP id r202-20020a6276d30000b02901d98e889276mr960814pfc.71.1612557448998; Fri, 05 Feb 2021 12:37:28 -0800 (PST) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [132.177.123.84]) by smtp-relay.gmail.com with ESMTPS id lk4sm1415885pjb.0.2021.02.05.12.37.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Feb 2021 12:37:28 -0800 (PST) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1254::1033]) by postal.iol.unh.edu (Postfix) with ESMTP id 292A76052472; Fri, 5 Feb 2021 15:37:28 -0500 (EST) From: ohilyard@iol.unh.edu To: ci@dpdk.org Cc: thomas@monjalon.net, Owen Hilyard Date: Fri, 5 Feb 2021 15:37:24 -0500 Message-Id: <20210205203725.143804-1-ohilyard@iol.unh.edu> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-ci] [PATCH 1/2] patch-tagging: Added tagging script for patches X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" From: Owen Hilyard Relevant to Bug 511 This commit contains a script, patch_parser.py, and a config file, patch_parser.cfg. These are tooling that the UNH CI team has been testing in order to reduce the number of tests that need to be run per patch. This resulted from our push to increase the number of functional tests running in the CI. While working on expanding test coverage, we found that DTS could easily take over 6 hours to run, so we decided to begin work on tagging patches and then only running the required tests. The script works by taking in an address for the config file and then a list of patch files, which it will parse and then produce a list of tags for that list of patches based on the config file. The config file is designed to work as a mapping for a base path to a set of tags. It also contains an ordered list of priorities for tags so that this may also be used by hierarchical tools rather than modular ones. The intention of the UNH team with giving this tooling to the wider DPDK community is to have people more familiar with the internal functionality of DPDK provide most of the tagging. This would allow UNH to have a better turn around time for testing by eliminating unnecessary tests, while still increasing the number of tests in the CI. The different patch tags are currently defined as such: core: Core DPDK functionality. Examples include kernel modules and librte_eal. This tag should be used sparingly as it is intended to signal to automated test suites that it is necessary to run most of the tests for DPDK and as such will consume CI resources for a long period of time. driver: For NIC drivers and other hardware interface code. This should be used as a generic tag with each driver getting it's own tag. application: Used in a similar manner to "driver". This tag is intended for code used in only in applications that DPDK provides, such as testpmd or helloworld. This tag should be accompanied by a tag which denotes which application specifically has been changed. documentation: This is intended to be used as a tag for paths which only contain documentation, such as "doc/". It's intended use is as a way to trigger the automatic re-building of the documentation website. Signed-off-by: Owen Hilyard --- config/patch_parser.cfg | 25 ++++++++++ tools/patch_parser.py | 100 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 config/patch_parser.cfg create mode 100755 tools/patch_parser.py diff --git a/config/patch_parser.cfg b/config/patch_parser.cfg new file mode 100644 index 0000000..5757f9a --- /dev/null +++ b/config/patch_parser.cfg @@ -0,0 +1,25 @@ +# Description of the categories as initially designed + +[Paths] +drivers = + driver, + core +kernel = core +doc = documentation +lib = core +meson_options.txt = core +examples = application +app = application +license = documentation +VERSION = documentation +build = core + +# This is an ordered list of the importance of each patch classification. +# It should be used to determine which classification to use on tools which +# do not support multiple patch classifications. +[Priority] +priority_list = + core, + driver, + application, + documentation diff --git a/tools/patch_parser.py b/tools/patch_parser.py new file mode 100755 index 0000000..92d50dd --- /dev/null +++ b/tools/patch_parser.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 + +# BSD LICENSE +# +# Copyright(c) 2020 Intel Corporation. All rights reserved. +# Copyright © 2018[, 2019] The University of New Hampshire. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +import subprocess +import sys + +import itertools +from configparser import ConfigParser +from typing import List, Dict, Set +import argparse + +try: + import whatthepatch +except ImportError as e: + print("Please install whatthepatch, a patch parsing library", file=sys.stderr) + exit(1) + + +def get_changed_files_in_patch(patch_file: str) -> List[str]: + with open(patch_file, 'r') as f: + filenames = map(lambda diff: diff.header.new_path, whatthepatch.parse_patch(f.read())) + return list(filenames) + + +def get_all_files_from_patches(patch_files: List[str]) -> Set[str]: + return set(itertools.chain.from_iterable(map(get_changed_files_in_patch, patch_files))) + + +def parse_comma_delimited_list_from_string(mod_str: str) -> List[str]: + return list(map(str.strip, mod_str.split(','))) + + +def get_dictionary_attributes_from_config_file(conf_obj: ConfigParser) -> Dict[str, Set[str]]: + return { + directory: parse_comma_delimited_list_from_string(module_string) for directory, module_string in + conf_obj['Paths'].items() + } + + +def get_tags_for_patch_file(patch_file: str, dir_attrs: Dict[str, Set[str]]) -> Set[str]: + return set(itertools.chain.from_iterable( + tags for directory, tags in dir_attrs.items() if patch_file.startswith(directory) + )) + + +def get_tags_for_patches(patch_files: Set[str], dir_attrs: Dict[str, Set[str]]) -> Set[str]: + return set(itertools.chain.from_iterable( + map(lambda patch_file: get_tags_for_patch_file(patch_file, dir_attrs), patch_files) + )) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + description='Takes a patch file and a config file and creates a list of tags for that patch') + parser.add_argument('config_file_path', help='The path to patch_parser.cfg', default='config/patch_parser.cfg') + parser.add_argument('patch_file_paths', help='A list of patch files', type=str, metavar='patch file', nargs='+') + + args = parser.parse_args() + + conf_obj = ConfigParser() + conf_obj.read(args.config_file_path) + + patch_files = get_all_files_from_patches(args.patch_file_paths) + dir_attrs = get_dictionary_attributes_from_config_file(conf_obj) + priority_list = parse_comma_delimited_list_from_string(conf_obj['Priority']['priority_list']) + + unordered_tags: Set[str] = get_tags_for_patches(patch_files, dir_attrs) + ordered_tags: List[str] = [tag for tag in priority_list if tag in unordered_tags] + + print("\n".join(ordered_tags)) -- 2.27.0