[dpdk-ci] [PATCH v3 00/10] Automatic patchwork delegation

[dpdk-ci] [PATCH v3 00/10] Automatic patchwork delegation

[Ali Alnubani]

This patchset adds support for automatic patch delegation.

The script 'guess_git_tree.py' was renamed to 'pw_maintainers_cli.py'
and now supports finding patch and series maintainers. It can also
delegate patches to them using git-pw.

The script 'poll-pw' was rewritten to fetch new api resources
by filtering with date and project. It now supports fetching both
patches and series.

Ali Alnubani (10):
  tools: rename guess_git_tree script
  tools: match by tree URL instead of tree name
  tools: update script usage
  tools: add functionality for detecting tree maintainers
  tools: add functionality for setting pw delegates
  add git-pw to requirements file
  tools: filter new Patchwork IDs by date
  tools: support fetching series
  tools: filter new patchwork IDs by project name
  tools: skip the IDs we already fetched

 requirements.txt                              |   3 +-
 tools/poll-pw                                 |  93 ++++++++----
 ...uess_git_tree.py => pw_maintainers_cli.py} | 132 +++++++++++++++---
 3 files changed, 181 insertions(+), 47 deletions(-)
 rename tools/{guess_git_tree.py => pw_maintainers_cli.py} (65%)

-- 
2.25.1

[dpdk-ci] [PATCH v3 01/10] tools: rename guess_git_tree script

[Ali Alnubani]

The name 'pw_maintainers_cli.py' will make more sense when
adding more operations to the script.

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
Changes in v2:
- Changed the script's name to 'pw_maintainers_cli.py' instead of
  'maintainers.py' (Suggested by Juraj Linkes).

 tools/{guess_git_tree.py => pw_maintainers_cli.py} | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
 rename tools/{guess_git_tree.py => pw_maintainers_cli.py} (98%)

diff --git a/tools/guess_git_tree.py b/tools/pw_maintainers_cli.py
similarity index 98%
rename from tools/guess_git_tree.py
rename to tools/pw_maintainers_cli.py
index c9eef39..a31f605 100755
--- a/tools/guess_git_tree.py
+++ b/tools/pw_maintainers_cli.py
@@ -31,13 +31,13 @@ variables PW_{SERVER,PROJECT,TOKEN} should be set. If not, the script will try
 to load the git configurations pw.{server,project,token}.
 
 Example usage:
-    ./guess-git-tree.py --command list_trees_for_series 2054
-    ./guess-git-tree.py --command list_trees_for_patch 2054
+    ./pw_maintainers_cli.py --command list_trees_for_series 2054
+    ./pw_maintainers_cli.py --command list_trees_for_patch 2054
 
 Or if you want to use inside other scripts:
 
     import os
-    from guess_git_tree import (Maintainers, GitPW, Diff)
+    from pw_maintainers_cli import (Maintainers, GitPW, Diff)
     _git_pw = GitPW({
         'pw_server': os.environ.get('PW_SERVER'),
         'pw_project': os.environ.get('PW_PROJECT'),
-- 
2.25.1

[dpdk-ci] [PATCH v3 02/10] tools: match by tree URL instead of tree name

[Ali Alnubani]

The tree URL has more information. It can be used to
get the maintainer of a tree, and it includes the tree name anyway.
Although we can construct the tree URL by prepending
"git://dpdk.org/next/" to the trees that have "next-" in their name,
and prepending "git://dpdk.org/" to anything else, it would be safer
to read the URL from the MAINTAINERS file just in case new trees that
don't follow this rule were added.

Also try to use named capture groups as much as possible.

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
 tools/pw_maintainers_cli.py | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)

diff --git a/tools/pw_maintainers_cli.py b/tools/pw_maintainers_cli.py
index a31f605..343e9f5 100755
--- a/tools/pw_maintainers_cli.py
+++ b/tools/pw_maintainers_cli.py
@@ -46,7 +46,8 @@ Or if you want to use inside other scripts:
     maintainers = Maintainers()
     patch_id = 52199
     files = Diff.find_filenames(_git_pw.api_get('patches', patch_id)['diff'])
-    tree = maintainers.get_tree(files)
+    tree_url = maintainers.get_tree(files)
+    tree_name = tree_url.split('/')[-1]
 """
 
 
@@ -116,9 +117,9 @@ class Diff(object):
 class Maintainers(object):
 
     file_regex = r'F:\s(.*)'
-    tree_regex = r'T: git:\/\/dpdk\.org(?:\/next)*\/(.*)'
+    tree_regex = r'T: (?P<url>git:\/\/dpdk\.org(?:\/next)*\/(?P<name>.*))'
     section_regex = r'([^\n]*)\n-+.*?(?=([^\n]*\n-+)|\Z)'
-    subsection_regex = r'[^\n](?:(?!\n{{2}}).)*?^F: {}(?:(?!\n{{2}}).)*'
+    subsection_regex = r'[^\n](?:(?!\n{{2}}).)*?^{}: {}$(?:(?!\n{{2}}).)*'
 
     def __init__(self):
         with open(MAINTAINERS_FILE_PATH) as fd:
@@ -151,8 +152,8 @@ class Maintainers(object):
             if _tree:
                 tree_list.append(_tree)
         tree = self.get_common_denominator(tree_list)
-        if tree == '':
-            tree = 'dpdk'
+        if not tree:
+            tree = 'git://dpdk.org/dpdk'
         return tree
 
     def _get_tree(self, filename):
@@ -180,7 +181,7 @@ class Maintainers(object):
 
         found_match = False
         # Find the block containing filename.
-        regex = self.subsection_regex.format(re.escape(matching_pattern))
+        regex = self.subsection_regex.format('F', re.escape(matching_pattern))
         subsection_match = re.findall(
                 regex,
                 self.maintainers_txt,
@@ -191,7 +192,7 @@ class Maintainers(object):
             tree_match = re.search(
                     self.tree_regex, subsection)
             if tree_match:
-                tree = tree_match.group(1)
+                tree = tree_match.group('url')
                 self.matched[matching_pattern] = tree
                 found_match = True
 
@@ -204,7 +205,7 @@ class Maintainers(object):
                             self.tree_regex,
                             section.group(0).split('\n\n')[0])
                     if tree_match:
-                        tree = tree_match.group(1)
+                        tree = tree_match.group('url')
 
         self.matched[matching_pattern] = tree
         return tree
@@ -228,8 +229,8 @@ class Maintainers(object):
             os.path.commonprefix(_tree_list).rstrip('-').replace(
                     'dpdk-next-net-virtio', 'dpdk-next-virtio')
         # There is no 'dpdk-next' named tree.
-        if common_prefix == 'dpdk-next':
-            common_prefix = 'dpdk'
+        if common_prefix.endswith('dpdk-next') or common_prefix.endswith('/'):
+            common_prefix = 'git://dpdk.org/dpdk'
         return common_prefix
 
 
@@ -289,4 +290,4 @@ if __name__ == '__main__':
     files = []
     for patch in patch_list:
         files += Diff.find_filenames(patch['diff'])
-    print(maintainers.get_tree(files))
+    print(maintainers.get_tree(files).split('/')[-1])
-- 
2.25.1

[dpdk-ci] [PATCH v3 03/10] tools: update script usage

[Ali Alnubani]

The resource type (whether it's a patch or a series) is now
specified by the new argument --type.
Both commands: list_trees_for_patch & list_trees_for_series
are replaced by the positional arg 'list_trees', and the type must
be always set.

This makes adding more operations require less arguments.

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
 tools/pw_maintainers_cli.py | 32 +++++++++++++++++++++-----------
 1 file changed, 21 insertions(+), 11 deletions(-)

diff --git a/tools/pw_maintainers_cli.py b/tools/pw_maintainers_cli.py
index 343e9f5..d4c0056 100755
--- a/tools/pw_maintainers_cli.py
+++ b/tools/pw_maintainers_cli.py
@@ -31,8 +31,8 @@ variables PW_{SERVER,PROJECT,TOKEN} should be set. If not, the script will try
 to load the git configurations pw.{server,project,token}.
 
 Example usage:
-    ./pw_maintainers_cli.py --command list_trees_for_series 2054
-    ./pw_maintainers_cli.py --command list_trees_for_patch 2054
+    ./pw_maintainers_cli.py --type series list_trees 2054
+    ./pw_maintainers_cli.py --type patch list_trees 2054
 
 Or if you want to use inside other scripts:
 
@@ -238,14 +238,14 @@ if __name__ == '__main__':
     """Main procedure."""
     parser = argparse.ArgumentParser()
     git_pw_conf_parser = parser.add_argument_group('git-pw configurations')
-    options_parser = parser.add_argument_group('optional arguments')
+    required_args_parser = parser.add_argument_group('required arguments')
 
-    options_parser.add_argument(
-            '--command',
+    required_args_parser.add_argument(
+            '--type',
             choices=(
-                'list_trees_for_patch',
-                'list_trees_for_series'),
-            required=True, help='Command to perform')
+                'patch',
+                'series'),
+            required=True, help='Resource type.')
 
     git_pw_conf_parser.add_argument(
             '--pw_server', type=str,
@@ -262,12 +262,18 @@ if __name__ == '__main__':
             default=os.environ.get('PW_TOKEN', utils.git_config('pw.token')),
             help='Authentication token')
 
+    parser.add_argument(
+            'command',
+            choices=[
+                'list_trees'],
+            help='Command to perform')
     parser.add_argument(
             'id', type=int, help='patch/series id')
 
     args = parser.parse_args()
 
     command = args.command
+    resource_type = args.type
     _id = args.id
 
     # Pass the needed configurations to git-pw.
@@ -279,9 +285,9 @@ if __name__ == '__main__':
     maintainers = Maintainers()
 
     patch_list = []
-    if command == 'list_trees_for_patch':
+    if resource_type == 'patch':
         patch_list.append(_git_pw.api_get('patches', _id))
-    elif command == 'list_trees_for_series':
+    else:
         series = _git_pw.api_get('series', _id)
         patch_list = [
                 _git_pw.api_get('patches', patch['id'])
@@ -290,4 +296,8 @@ if __name__ == '__main__':
     files = []
     for patch in patch_list:
         files += Diff.find_filenames(patch['diff'])
-    print(maintainers.get_tree(files).split('/')[-1])
+
+    tree = maintainers.get_tree(files)
+
+    if command == 'list_trees':
+        print(tree.split('/')[-1])
-- 
2.25.1

[dpdk-ci] [PATCH v3 04/10] tools: add functionality for detecting tree maintainers

[Ali Alnubani]

Detecting a maintainer works by searching the
'General Project Administration' section for subsections
containing the provided tree, and then returning the maintainers
specified in that subsection.

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
Changes in v3:
- Stored 'General Project Administration' in a variable (Suggested by
  Thomas Monjalon).

 tools/pw_maintainers_cli.py | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/tools/pw_maintainers_cli.py b/tools/pw_maintainers_cli.py
index d4c0056..4c95755 100755
--- a/tools/pw_maintainers_cli.py
+++ b/tools/pw_maintainers_cli.py
@@ -33,6 +33,7 @@ to load the git configurations pw.{server,project,token}.
 Example usage:
     ./pw_maintainers_cli.py --type series list_trees 2054
     ./pw_maintainers_cli.py --type patch list_trees 2054
+    ./pw_maintainers_cli.py --type patch list_maintainers 2054
 
 Or if you want to use inside other scripts:
 
@@ -48,6 +49,7 @@ Or if you want to use inside other scripts:
     files = Diff.find_filenames(_git_pw.api_get('patches', patch_id)['diff'])
     tree_url = maintainers.get_tree(files)
     tree_name = tree_url.split('/')[-1]
+    maintainers = maintainers.get_maintainers(tree_url)
 """
 
 
@@ -118,8 +120,10 @@ class Maintainers(object):
 
     file_regex = r'F:\s(.*)'
     tree_regex = r'T: (?P<url>git:\/\/dpdk\.org(?:\/next)*\/(?P<name>.*))'
+    maintainer_regex = r'M:\s(.*)'
     section_regex = r'([^\n]*)\n-+.*?(?=([^\n]*\n-+)|\Z)'
     subsection_regex = r'[^\n](?:(?!\n{{2}}).)*?^{}: {}$(?:(?!\n{{2}}).)*'
+    general_proj_admin_title = 'General Project Administration'
 
     def __init__(self):
         with open(MAINTAINERS_FILE_PATH) as fd:
@@ -141,6 +145,26 @@ class Maintainers(object):
         # Save already matched patterns.
         self.matched = {}
 
+    def get_maintainers(self, tree):
+        """
+        Return a list of a tree's maintainers."""
+        maintainers = []
+        for section in self.sections:
+            if section.group(1) == self.general_proj_admin_title:
+                # Find the block containing the tree.
+                regex = self.subsection_regex.format('T', re.escape(tree))
+                subsection_match = re.findall(
+                        regex,
+                        section.group(0),
+                        re.DOTALL | re.MULTILINE)
+                if len(subsection_match):
+                    subsection = subsection_match[-1]
+                    # Look for maintainers
+                    maintainers = re.findall(
+                            self.maintainer_regex, subsection)
+                    return maintainers
+                break
+
     def get_tree(self, files):
         """
         Return a git tree that matches a list of files."""
@@ -265,7 +289,7 @@ if __name__ == '__main__':
     parser.add_argument(
             'command',
             choices=[
-                'list_trees'],
+                'list_trees', 'list_maintainers'],
             help='Command to perform')
     parser.add_argument(
             'id', type=int, help='patch/series id')
@@ -301,3 +325,5 @@ if __name__ == '__main__':
 
     if command == 'list_trees':
         print(tree.split('/')[-1])
+    elif command == 'list_maintainers':
+        print(*maintainers.get_maintainers(tree), sep='\n')
-- 
2.25.1

[dpdk-ci] [PATCH v3 05/10] tools: add functionality for setting pw delegates

[Ali Alnubani]

A new command was added to set patch delegates in Patchwork
based on the emails found in DPDK's MAINTAINERS file.

Example usage:
  $ export MAINTAINERS_FILE_PATH=/path/to/dpdk/MAINTAINERS
  $ ./pw_maintainers_cli.py --type series set_pw_delegate SERIES_ID

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
Changes in v3:
- Added an argument to force overriding delegates (Suggested by Thomas
  Monjalon).

 tools/pw_maintainers_cli.py | 57 +++++++++++++++++++++++++++++++++++--
 1 file changed, 54 insertions(+), 3 deletions(-)

diff --git a/tools/pw_maintainers_cli.py b/tools/pw_maintainers_cli.py
index 4c95755..0e9643f 100755
--- a/tools/pw_maintainers_cli.py
+++ b/tools/pw_maintainers_cli.py
@@ -14,6 +14,7 @@ from requests.exceptions import HTTPError
 from git_pw import config
 from git_pw import api
 from git_pw import utils
+from git_pw import patch as git_pw_patch
 
 """
 Description:
@@ -84,6 +85,33 @@ class GitPW(object):
             else:
                 raise
 
+    def set_delegate(self, patch_list, delegate, force=False):
+        """Set the delegate for a patch.
+        If 'force' isn't set to True, only set a delegate for patches that
+        don't have one set already.
+
+        Reference:
+        https://github.com/getpatchwork/git-pw/blob/76b79097dc0a57c89b45dd53d9cacb7ff7b31bb2/git_pw/patch.py#L167
+        """
+        users = api.index('users', [('q', delegate)])
+        if len(users) != 1:
+            # Zero or multiple users found
+            print('Cannot choose a Patchwork user to delegate to from '
+                  'user list ({}). Skipping..'.format(users))
+            return
+        for patch in patch_list:
+            if patch['delegate'] != None and \
+                    (patch['delegate'].get('email') == users[0].get('email') or \
+                    not force):
+                print('Patch {} is already delegated to {}. '
+                      'Skipping..'.format(
+                          patch['id'], patch['delegate']['email']))
+                continue
+            print("Delegating patch {} to {}..".format(
+                patch['id'], users[0]['email']))
+            _ = api.update(
+                    'patches', patch['id'], [('delegate', users[0]['id'])])
+
 
 class Diff(object):
 
@@ -286,16 +314,22 @@ if __name__ == '__main__':
             default=os.environ.get('PW_TOKEN', utils.git_config('pw.token')),
             help='Authentication token')
 
+    parser.add_argument(
+            '--force_set_delegate',
+            action='store_true', required=False,
+            help='Force setting the PW delegate, '
+                 'even if it is already set.')
     parser.add_argument(
             'command',
             choices=[
-                'list_trees', 'list_maintainers'],
+                'list_trees', 'list_maintainers', 'set_pw_delegate'],
             help='Command to perform')
     parser.add_argument(
             'id', type=int, help='patch/series id')
 
     args = parser.parse_args()
 
+    force_set_delegate = args.force_set_delegate
     command = args.command
     resource_type = args.type
     _id = args.id
@@ -325,5 +359,22 @@ if __name__ == '__main__':
 
     if command == 'list_trees':
         print(tree.split('/')[-1])
-    elif command == 'list_maintainers':
-        print(*maintainers.get_maintainers(tree), sep='\n')
+    if command in ['list_maintainers', 'set_pw_delegate']:
+        maintainer_list = maintainers.get_maintainers(tree)
+        if command == 'list_maintainers':
+            print(*maintainer_list, sep='\n')
+        elif command == 'set_pw_delegate':
+            if len(maintainer_list) > 0:
+                # Get the email of the first maintainer in the list.
+                try:
+                    delegate = re.match(
+                            r".*\<(?P<email>.*)\>",
+                            maintainer_list[0]).group('email')
+                except AttributeError:
+                    print("Unexpected format: '{}'".format(maintainer_list[0]))
+                    sys.exit(1)
+                _git_pw.set_delegate(
+                        patch_list, delegate,
+                        force=force_set_delegate)
+            else:
+                print('No maintainers found. Not setting a delegate.')
-- 
2.25.1

[dpdk-ci] [PATCH v3 06/10] add git-pw to requirements file

[Ali Alnubani]

The module has always been required by tools/maintainers.py (previously
named tools/guess_git_tree.py).

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
 requirements.txt | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index f20067d..f2a6844 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1 +1,2 @@
-whatthepatch==1.0.2
\ No newline at end of file
+git-pw==2.1.0
+whatthepatch==1.0.2
-- 
2.25.1

[dpdk-ci] [PATCH v3 07/10] tools: filter new Patchwork IDs by date

[Ali Alnubani]

API resource IDs are guaranteed to be unique, but they aren't guaranteed
to have no gaps, for example, the following series IDs are
nonexistent: 16157, 17181, 18235.
Filtering by the date since the last check is necessary to later
add support for fetching new series IDs in addition to patch IDs.

The script now periodically fetches Patchwork's events API filtering
by the 'patch-completed' category (see [1]). It parses the responses using jq
and passes the IDs to 'callcmd'.

Instead of requiring a file that contains the next patch ID,
a file containing the timestamp of the last time the API was fetched
is now used. Each time the API is fetched for new patches, the timestamp
in the file gets updated, and the script sleeps an amount of time specified
by PAUSE_SECONDS before attempting to fetch new resources again.
There are 2 variables, 'date_now', which is recorded right before
fetching from the API, and then gets written to the file, and 'since',
which is the last date that was written to the file.

Since API responses can be in multiple pages, the script will keep
attempting to fetch the next page until jq encounters an error while
parsing a non list response with no 'payload' key.

The pause amount between each poll attempt is still 100 seconds.

The script writes the date in Universal Time (UTC) format and
expects to read the same format from the file as Patchwork stores event
objects with a naive datetime object that is not aware of the
server's timezone (see [2] and [3]).

The package jq (Command-line JSON processor) is now required by the
script.

Example usage:
$ export MAINTAINERS_FILE_PATH=/path/to/dpdk/MAINTAINERS
$ ./tools/poll-pw /path/to/last.txt \
    '/path/to/pw_maintainers_cli.py --type patch set_pw_delegate $1'

[1] https://patchwork.readthedocs.io/en/latest/usage/overview/#patch-completed
[2] https://github.com/getpatchwork/patchwork/blob/580cc8570a05c1/patchwork/models.py#L1058
[3] https://docs.python.org/3/library/datetime.html#datetime.datetime.utcnow

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
Changes in v2:
- Removed an unnecessary 'break' statement in the for loop.
Changes in v3:
- Fetch the events API endpoint for new patches as the date field in the
  /patches and /series endpoints is the mbox creation date, not
  the date they were created in Patchwork.
- Use UTC date format.
- Redirect error messages to stderr.
- Fixed typo in usage (specifed -> specified).

 tools/poll-pw | 63 ++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 42 insertions(+), 21 deletions(-)

diff --git a/tools/poll-pw b/tools/poll-pw
index bdf860a..ccc58f0 100755
--- a/tools/poll-pw
+++ b/tools/poll-pw
@@ -1,58 +1,79 @@
-#! /bin/sh -e
+#! /bin/sh
 
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2017 6WIND S.A.
 # Copyright 2018 Mellanox Technologies, Ltd
 
 URL=http://patches.dpdk.org/api
+PAUSE_SECONDS=100
 
 print_usage () {
 	cat <<- END_OF_HELP
-	usage: $(basename $0) <counter> <command>
+	usage: $(basename $0) [OPTIONS] </path/to/last.txt> <command>
 
 	Poll patchwork and call command for each patch.
-	The first patchwork id to be checked is read from the counter file.
-	The command should use '$1' to be evaluated as patchwork id.
-	When a patch is found and the command is successful,
-	then the counter is incremented.
+	The first date to filter with is read from the specified file.
+	The command should use '$1' to be evaluated as the patch id.
+	The date in the specified file is updated after each pull.
 	END_OF_HELP
 }
 
+which jq >/dev/null 2>&1
+if [ ! $? -eq 0 ] ; then
+	printf "The command 'jq' doesn't exist, please install it.\n\n" >&2
+	exit 1
+fi
+
 while getopts h arg ; do
 	case $arg in
 		h ) print_usage ; exit 0 ;;
 		? ) print_usage >&2 ; exit 1 ;;
 	esac
 done
+
 if [ $# -lt 2 ] ; then
-	printf 'missing argument\n\n' >&2
+	printf 'missing argument(s)\n\n' >&2
 	print_usage >&2
 	exit 1
 fi
 shift $(($OPTIND - 1))
-counter=$1
+since_file=$1
 shift
 cmd=$*
 
+if [ ! -f "$since_file" ] ; then
+	printf "The file '$since_file' doesn't exist.\n\n" >&2
+	exit 1
+fi
+
+date -d "$(cat $since_file | tr '\n' ' ')" >/dev/null 2>&1
+if [ ! $? -eq 0 ] ; then
+	printf "The file '$since_file' doesn't contain a valid date format.\n\n" >&2
+	exit 1
+fi
+
+URL="${URL}/events/?category=patch-completed"
+
 callcmd () # <patchwork id>
 {
 	eval $cmd
 }
 
-checkid () # <patchwork id>
-{
-	curl -sfIo /dev/null $URL/patches/$1/ ||
-	curl -sfIo /dev/null $URL/covers/$1/
-}
-
-pwid=$(cat $counter)
+set -e
 while true ; do
-	# process all recent patches
-	while checkid $pwid ; do
-		callcmd $pwid || break
-		pwid=$(($pwid + 1))
-		echo $pwid >$counter
+	date_now=$(date --utc '+%FT%T')
+	since=$(date -d $(cat $since_file | tr '\n' ' ') '+%FT%T')
+	page=1
+	while true ; do
+		ids=$(curl -s "${URL}&page=${page}&since=${since}" \
+			| jq 'try ( .[].payload.patch.id )')
+		[ -z "$(echo $ids | tr -d '\n')" ] && break
+		for id in $ids ; do
+			callcmd $id
+		done
+		page=$((page+1))
 	done
+	echo -n $date_now >$since_file
 	# pause before next check
-	sleep 100
+	sleep $PAUSE_SECONDS
 done
-- 
2.25.1

[dpdk-ci] [PATCH v3 08/10] tools: support fetching series

[Ali Alnubani]

The script can now fetch new series IDs by filtering the events
API with category 'series-completed'. See:
https://patchwork.readthedocs.io/en/latest/usage/overview/#series-completed

Example usage:
$ export MAINTAINERS_FILE_PATH=/path/to/dpdk/MAINTAINERS
$ ./tools/poll-pw series /path/to/last.txt \
    '/path/to/pw_maintainers_cli.py --type series set_pw_delegate $1'

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
Changes in v3:
- Fetch new series IDs by filtering the events API with
  category "series-completed".
- Updated usage (renamed 'patches' arg to 'patch') so that
  it feeds directly into the 'patch-completed' category.

 tools/poll-pw | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/tools/poll-pw b/tools/poll-pw
index ccc58f0..77f6b79 100755
--- a/tools/poll-pw
+++ b/tools/poll-pw
@@ -9,11 +9,11 @@ PAUSE_SECONDS=100
 
 print_usage () {
 	cat <<- END_OF_HELP
-	usage: $(basename $0) [OPTIONS] </path/to/last.txt> <command>
+	usage: $(basename $0) [OPTIONS] <patch|series> </path/to/last.txt> <command>
 
-	Poll patchwork and call command for each patch.
+	Poll patchwork and call a command for each new patch/series id.
 	The first date to filter with is read from the specified file.
-	The command should use '$1' to be evaluated as the patch id.
+	The command should use '$1' to be evaluated as the patch/series id.
 	The date in the specified file is updated after each pull.
 	END_OF_HELP
 }
@@ -31,14 +31,15 @@ while getopts h arg ; do
 	esac
 done
 
-if [ $# -lt 2 ] ; then
+if [ $# -lt 3 ] ; then
 	printf 'missing argument(s)\n\n' >&2
 	print_usage >&2
 	exit 1
 fi
 shift $(($OPTIND - 1))
-since_file=$1
-shift
+resource_type=$1
+since_file=$2
+shift 2
 cmd=$*
 
 if [ ! -f "$since_file" ] ; then
@@ -52,7 +53,13 @@ if [ ! $? -eq 0 ] ; then
 	exit 1
 fi
 
-URL="${URL}/events/?category=patch-completed"
+if [ "$resource_type" != "patch" ] & [ "$resource_type" != "series" ] ; then
+	printf "Unknown resource type '$resource_type'.\n\n" >&2
+	print_usage >&2
+	exit 1
+fi
+
+URL="${URL}/events/?category=${resource_type}-completed"
 
 callcmd () # <patchwork id>
 {
@@ -66,7 +73,7 @@ while true ; do
 	page=1
 	while true ; do
 		ids=$(curl -s "${URL}&page=${page}&since=${since}" \
-			| jq 'try ( .[].payload.patch.id )')
+			| jq "try ( .[].payload.${resource_type}.id )")
 		[ -z "$(echo $ids | tr -d '\n')" ] && break
 		for id in $ids ; do
 			callcmd $id
-- 
2.25.1

[dpdk-ci] [PATCH v3 09/10] tools: filter new patchwork IDs by project name

[Ali Alnubani]

If the script doesn't filter by project, it would be fetching patches
from all projects, which we don't want as different projects
require different checks usually. This patch modifies the script so
that it requires a project's name to fetch IDs for.

Example usage:
$ export MAINTAINERS_FILE_PATH=/path/to/dpdk/MAINTAINERS
$ ./tools/poll-pw series DPDK /path/to/last.txt \
    '/path/to/pw_maintainers_cli.py --type series set_pw_delegate $1'

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
Changes in v3:
- Filter by project using jq, since the events API doesn't have a
  'project' parameter.
- Redirect error messages to stderr.

 tools/poll-pw | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/tools/poll-pw b/tools/poll-pw
index 77f6b79..1f8db9c 100755
--- a/tools/poll-pw
+++ b/tools/poll-pw
@@ -9,7 +9,7 @@ PAUSE_SECONDS=100
 
 print_usage () {
 	cat <<- END_OF_HELP
-	usage: $(basename $0) [OPTIONS] <patch|series> </path/to/last.txt> <command>
+	usage: $(basename $0) [OPTIONS] <patch|series> <project> </path/to/last.txt> <command>
 
 	Poll patchwork and call a command for each new patch/series id.
 	The first date to filter with is read from the specified file.
@@ -31,15 +31,16 @@ while getopts h arg ; do
 	esac
 done
 
-if [ $# -lt 3 ] ; then
+if [ $# -lt 4 ] ; then
 	printf 'missing argument(s)\n\n' >&2
 	print_usage >&2
 	exit 1
 fi
 shift $(($OPTIND - 1))
 resource_type=$1
-since_file=$2
-shift 2
+project=$2
+since_file=$3
+shift 3
 cmd=$*
 
 if [ ! -f "$since_file" ] ; then
@@ -59,6 +60,12 @@ if [ "$resource_type" != "patch" ] & [ "$resource_type" != "series" ] ; then
 	exit 1
 fi
 
+curl -s $URL/projects/ | jq '.[].name' | grep -qi "^\"${project}\"$"
+if [ ! $? -eq 0 ] ; then
+	printf "The project '$project' doesn't exist.\n\n" >&2
+	exit 1
+fi
+
 URL="${URL}/events/?category=${resource_type}-completed"
 
 callcmd () # <patchwork id>
@@ -73,7 +80,8 @@ while true ; do
 	page=1
 	while true ; do
 		ids=$(curl -s "${URL}&page=${page}&since=${since}" \
-			| jq "try ( .[].payload.${resource_type}.id )")
+			| jq "try ( .[] | select( .project.name == \"$project\" ) )" \
+			| jq "try ( .payload.${resource_type}.id )")
 		[ -z "$(echo $ids | tr -d '\n')" ] && break
 		for id in $ids ; do
 			callcmd $id
-- 
2.25.1

[dpdk-ci] [PATCH v3 10/10] tools: skip the IDs we already fetched

[Ali Alnubani]

To avoid calling 'callcmd' more than once for an ID, which can happen
if a Patchwork ID was created between recording 'date_now' and fetching
the API, store the IDs we already fetched in a file and don't run 'callcmd'
again for any ID that exists in the file.

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
 tools/poll-pw | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tools/poll-pw b/tools/poll-pw
index 1f8db9c..589e356 100755
--- a/tools/poll-pw
+++ b/tools/poll-pw
@@ -66,6 +66,11 @@ if [ ! $? -eq 0 ] ; then
 	exit 1
 fi
 
+poll_pw_ids_file=/tmp/poll_pw_${resource_type}_ids
+if [ ! -f "$poll_pw_ids_file" ] ; then
+	touch $poll_pw_ids_file
+fi
+
 URL="${URL}/events/?category=${resource_type}-completed"
 
 callcmd () # <patchwork id>
@@ -84,7 +89,11 @@ while true ; do
 			| jq "try ( .payload.${resource_type}.id )")
 		[ -z "$(echo $ids | tr -d '\n')" ] && break
 		for id in $ids ; do
+			if grep -q "^${id}$" $poll_pw_ids_file ; then
+				continue
+			fi
 			callcmd $id
+			echo $id >>$poll_pw_ids_file
 		done
 		page=$((page+1))
 	done
-- 
2.25.1

Re: [dpdk-ci] [PATCH v3 00/10] Automatic patchwork delegation

[Ali Alnubani]

> -----Original Message-----
> From: ci <ci-bounces@dpdk.org> On Behalf Of Ali Alnubani
> Sent: Monday, October 18, 2021 10:45 AM
> To: ci@dpdk.org
> Cc: NBU-Contact-Thomas Monjalon <thomas@monjalon.net>;
> jerinj@marvell.com; ferruh.yigit@intel.com; david.marchand@redhat.com;
> juraj.linkes@pantheon.tech
> Subject: [dpdk-ci] [PATCH v3 00/10] Automatic patchwork delegation
> 
> This patchset adds support for automatic patch delegation.
> 
> The script 'guess_git_tree.py' was renamed to 'pw_maintainers_cli.py'
> and now supports finding patch and series maintainers. It can also
> delegate patches to them using git-pw.
> 
> The script 'poll-pw' was rewritten to fetch new api resources
> by filtering with date and project. It now supports fetching both
> patches and series.
> 

Sorry forgot to set in-reply-to.