DPDK CI discussions
 help / color / mirror / Atom feed
From: Ali Alnubani <alialnu@oss.nvidia.com>
To: <ci@dpdk.org>
Cc: <thomas@monjalon.net>, <jerinj@marvell.com>,
	<ferruh.yigit@intel.com>, <david.marchand@redhat.com>
Subject: [dpdk-ci] [PATCH 7/9] tools: filter new patchwork IDs by date
Date: Mon, 6 Sep 2021 18:45:35 +0300
Message-ID: <20210906154537.1299-8-alialnu@nvidia.com> (raw)
In-Reply-To: <20210906154537.1299-1-alialnu@nvidia.com>

API resource IDs are guaranteed to be unique, but they aren't guaranteed
to have no gaps, for example, the following series IDs are
nonexistent: 16157, 17181, 18235.
Filtering by the date since the last check is necessary to later
add support for fetching new series IDs in addition to patch IDs.

Instead of requiring a file that contains the next patch ID,
a file containing the timestamp of the last time the API was fetched
is now used.
Each time the API is fetched for new patches, the timestamp
in the file gets updated, and the script sleeps an amount of time specified
by PAUSE_SECONDS before attempting to fetch new resources again.

The pause amount between each poll attempt is still 100 seconds.

Setting the env variable 'TZ' might be necessary if your timezone
doesn't match the server's timezone.

The package jq (Command-line JSON processor) is now required by the
script.

Example usage:
$ export TZ="Europe/Paris"
$ export MAINTAINERS_FILE_PATH=/path/to/dpdk/MAINTAINERS
$ ./tools/poll-pw /path/to/last.txt \
    '/path/to/maintainers.py --command set_pw_delegate --type patch $1'

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
 tools/poll-pw | 56 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 35 insertions(+), 21 deletions(-)

diff --git a/tools/poll-pw b/tools/poll-pw
index bdf860a..e104dab 100755
--- a/tools/poll-pw
+++ b/tools/poll-pw
@@ -1,58 +1,72 @@
-#! /bin/sh -e
+#! /bin/sh
 
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2017 6WIND S.A.
 # Copyright 2018 Mellanox Technologies, Ltd
 
 URL=http://patches.dpdk.org/api
+PAUSE_SECONDS=100
 
 print_usage () {
 	cat <<- END_OF_HELP
-	usage: $(basename $0) <counter> <command>
+	usage: $(basename $0) [OPTIONS] </path/to/last.txt> <command>
 
 	Poll patchwork and call command for each patch.
-	The first patchwork id to be checked is read from the counter file.
-	The command should use '$1' to be evaluated as patchwork id.
-	When a patch is found and the command is successful,
-	then the counter is incremented.
+	The first date to filter with is read from the specified file.
+	The command should use '$1' to be evaluated as the patch id.
+	The date in the specifed file is updated after each pull.
 	END_OF_HELP
 }
 
+which jq >/dev/null 2>&1
+if [ ! $? -eq 0 ] ; then
+	echo "The command 'jq' doesn't exist, please install it."
+	exit 1
+fi
+
 while getopts h arg ; do
 	case $arg in
 		h ) print_usage ; exit 0 ;;
 		? ) print_usage >&2 ; exit 1 ;;
 	esac
 done
+
 if [ $# -lt 2 ] ; then
-	printf 'missing argument\n\n' >&2
+	printf 'missing argument(s)\n\n' >&2
 	print_usage >&2
 	exit 1
 fi
 shift $(($OPTIND - 1))
-counter=$1
+since_file=$1
 shift
 cmd=$*
 
+if [ ! -f "$since_file" ] ; then
+	echo "The file '$since_file' doesn't exist."
+	exit 1
+fi
+
+date -d "$(cat $since_file | tr '\n' ' ')" >/dev/null 2>&1
+if [ ! $? -eq 0 ] ; then
+	echo "The file '$since_file' doesn't contain a valid date format."
+	exit 1
+fi
+
+URL="${URL}/patches/?"
+
 callcmd () # <patchwork id>
 {
 	eval $cmd
 }
 
-checkid () # <patchwork id>
-{
-	curl -sfIo /dev/null $URL/patches/$1/ ||
-	curl -sfIo /dev/null $URL/covers/$1/
-}
-
-pwid=$(cat $counter)
+set -e
 while true ; do
-	# process all recent patches
-	while checkid $pwid ; do
-		callcmd $pwid || break
-		pwid=$(($pwid + 1))
-		echo $pwid >$counter
+	date_now=$(date '+%FT%T')
+	since=$(date -d $(cat $since_file | tr '\n' ' ') '+%FT%T')
+	for id in $(curl -s "${URL}since=${since}" | jq '.[].id') ; do
+		callcmd $id || break
 	done
+	echo -n $date_now >$since_file
 	# pause before next check
-	sleep 100
+	sleep $PAUSE_SECONDS
 done
-- 
2.25.1


  parent reply	other threads:[~2021-09-06 15:46 UTC|newest]

Thread overview: 56+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-06 15:45 [dpdk-ci] [PATCH 0/9] Automatic patchwork delegation Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 1/9] tools: rename guess_git_tree script Ali Alnubani
2021-09-07 11:03   ` Juraj Linkeš
2021-09-08 16:54     ` Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 2/9] tools: match by tree url instead of tree name Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 3/9] tools: update script usage Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 4/9] tools: add functionality for detecting tree maintainers Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 5/9] tools: add functionality for setting pw delegates Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 6/9] add git-pw to requirements file Ali Alnubani
2021-09-06 15:45 ` Ali Alnubani [this message]
2021-09-06 15:58   ` [dpdk-ci] [PATCH 7/9] tools: filter new patchwork IDs by date Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 8/9] tools: add support for fetching new series IDs Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 9/9] tools: filter new patchwork IDs by project name Ali Alnubani
2021-09-21 14:35 ` [dpdk-ci] [PATCH v2 00/10] Automatic patchwork delegation alialnu
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 01/10] tools: rename guess_git_tree script alialnu
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 02/10] tools: match by tree url instead of tree name alialnu
2021-09-30  8:00     ` Thomas Monjalon
2021-10-18  7:48       ` Ali Alnubani
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 03/10] tools: update script usage alialnu
2021-09-30  8:09     ` Thomas Monjalon
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 04/10] tools: add functionality for detecting tree maintainers alialnu
2021-09-30  8:29     ` Thomas Monjalon
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 05/10] tools: add functionality for setting pw delegates alialnu
2021-09-30  9:15     ` Thomas Monjalon
2021-10-18  7:48       ` Ali Alnubani
2021-10-26 14:08         ` Thomas Monjalon
2021-11-04 16:48           ` Ali Alnubani
2021-11-04 18:16             ` Thomas Monjalon
2021-11-08  7:45               ` Ali Alnubani
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 06/10] add git-pw to requirements file alialnu
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 07/10] tools: filter new patchwork IDs by date alialnu
2021-10-11 20:08     ` Ali Alnubani
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 08/10] tools: add support for fetching new series IDs alialnu
2021-09-30 10:25     ` Thomas Monjalon
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 09/10] tools: filter new patchwork IDs by project name alialnu
2021-09-30 10:28     ` Thomas Monjalon
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched alialnu
2021-09-30 10:32     ` Thomas Monjalon
2021-10-11 19:30       ` Ali Alnubani
2021-10-12  6:44         ` Thomas Monjalon
2021-10-18  8:04           ` Ali Alnubani
2021-10-26 14:07             ` Thomas Monjalon
2021-11-04 16:53               ` Ali Alnubani
2021-11-04 18:08                 ` Thomas Monjalon
2021-11-08  7:44                   ` Ali Alnubani
2021-11-08  6:28 ` [dpdk-ci] [PATCH v4 00/10] Automatic patchwork delegation Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 01/10] tools: rename guess_git_tree script Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 02/10] tools: match by tree URL instead of tree name Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 03/10] tools: update script usage Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 04/10] tools: add functionality for detecting tree maintainers Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 05/10] tools: add functionality for setting pw delegates Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 06/10] add git-pw to requirements file Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 07/10] tools: filter new Patchwork IDs by date Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 08/10] tools: support fetching series Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 09/10] tools: filter new patchwork IDs by project name Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 10/10] tools: skip the IDs we already fetched Ali Alnubani

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210906154537.1299-8-alialnu@nvidia.com \
    --to=alialnu@oss.nvidia.com \
    --cc=ci@dpdk.org \
    --cc=david.marchand@redhat.com \
    --cc=ferruh.yigit@intel.com \
    --cc=jerinj@marvell.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK CI discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/ci/0 ci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ci ci/ http://inbox.dpdk.org/ci \
		ci@dpdk.org
	public-inbox-index ci

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.ci


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git