From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <ci-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 01639A0C4B
	for <public@inbox.dpdk.org>; Mon,  8 Nov 2021 07:29:39 +0100 (CET)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id EB13240151;
	Mon,  8 Nov 2021 07:29:38 +0100 (CET)
Received: from NAM02-BN1-obe.outbound.protection.outlook.com
 (mail-bn1nam07on2089.outbound.protection.outlook.com [40.107.212.89])
 by mails.dpdk.org (Postfix) with ESMTP id 4344D40040
 for <ci@dpdk.org>; Mon,  8 Nov 2021 07:29:37 +0100 (CET)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=dAGaaNgRmxwNGwl3C2aTS8B9E6F5axfeuMMFExV5pf7lEty4lyYLvnkI84DiiXgaa8gm+eyjJU+nE2gPEp0Wt6sGrfp62LeET3JJ9wrYVCmzlFBuFy2v93QFlBbwn6Ovshc2PQM7JVOINPzUF9YrF8nZMPaNV2zvcwzupk353/h1hm49+GPAK/dzd2pncC3Sjam66cZJ7AHtitVlrPUPptLLdI6TGM5cZb7FWU1PfwaujfnSb2/36Ygy2VMmu2KsSBWHtULLRXlS2lHPButbD78DaCjvhIvyXQAkrfMsWsfRvDhyguARMC7rSyyXLzr3YERWlSCel/63macwJ57+xg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=0RS1GUaiMCllaD39RctOz7tRG2jOli1AwpKUO7ob8XE=;
 b=IAPwwPhywqN/LUor/97iUgFd+2jSCqbJtArXznQBLiNLu4S3n0RME3aZt97oZXflsKYavWYSYGmkLkHnz6VVSV1dvXjqsAm5YBBIENHjyh6+ltIB5JLIO9OrtXJ7bK9sKNewzIQ7CtDPoJnB6wDaLoAbaDE6GUBNvSAopsaH/PNEO82qPFgZl5YeHVPcR51yLVPx1DIC1Uy1q57hX50mHElkZcE75aX8NjKyZUQi26QlW8Hd2ozoOzHBEDBZgrXx6zcsPan35uMoBpp2fpAVRo5r5mWx45zzL6i6Z7wtKm9YAI/5/3CN/tJ8ai3uhIFLrGHLRRTkzqflDXfOgePWZQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
 216.228.112.34) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com;
 dmarc=pass (p=quarantine sp=quarantine pct=100) action=none
 header.from=nvidia.com; dkim=none (message not signed); arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=0RS1GUaiMCllaD39RctOz7tRG2jOli1AwpKUO7ob8XE=;
 b=nWzfl8xbHl/MTbiv8quuagH2GVXeymY0j5vaOUCTGLqad5bqddJTNBq4ryr3fcsNZrG2nZH98k/g04aYApMKN8NWG/jQGSgHkASVQIZq+E2fGA4YUje2XW/rKO96KmoBmVrT82of/olRBqmEnmmrAkq2t3MjqW2GlcMbGO/MMku4GNKyT8huBY+/zriJsBeII9nOuISkdv9nkX3SqJ8F5xKHJXL2Vwc2za2Qz7HhgYSDZ5Ldw5nzN/nS0M3I798FOwQyplG+jGAc85B2Gz3rfQmNfQg13GRAWh34ozL8ak0ShBZzqOsJ1uL6z2hvksIQAWtlP8NTRXL2rHPjSGL1dA==
Received: from DM6PR12CA0022.namprd12.prod.outlook.com (2603:10b6:5:1c0::35)
 by MW2PR12MB2348.namprd12.prod.outlook.com (2603:10b6:907:e::19) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11; Mon, 8 Nov
 2021 06:29:32 +0000
Received: from DM6NAM11FT051.eop-nam11.prod.protection.outlook.com
 (2603:10b6:5:1c0:cafe::bc) by DM6PR12CA0022.outlook.office365.com
 (2603:10b6:5:1c0::35) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10 via Frontend
 Transport; Mon, 8 Nov 2021 06:29:32 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34)
 smtp.mailfrom=nvidia.com; intel.com; dkim=none (message not signed)
 header.d=none;intel.com; dmarc=pass action=none header.from=nvidia.com;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Received: from mail.nvidia.com (216.228.112.34) by
 DM6NAM11FT051.mail.protection.outlook.com (10.13.172.243) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.4669.10 via Frontend Transport; Mon, 8 Nov 2021 06:29:32 +0000
Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com
 (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Mon, 8 Nov
 2021 06:29:29 +0000
From: Ali Alnubani <alialnu@nvidia.com>
To: <ci@dpdk.org>
CC: <thomas@monjalon.net>, <jerinj@marvell.com>, <ferruh.yigit@intel.com>,
 <david.marchand@redhat.com>, <juraj.linkes@pantheon.tech>
Date: Mon, 8 Nov 2021 08:28:12 +0200
Message-ID: <20211108062815.7614-8-alialnu@nvidia.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20211108062815.7614-1-alialnu@nvidia.com>
References: <20210906154537.1299-1-alialnu@nvidia.com>
 <20211108062815.7614-1-alialnu@nvidia.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-Originating-IP: [172.20.187.5]
X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To
 HQMAIL107.nvidia.com (172.20.187.13)
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: e764fba8-b296-43cf-88cb-08d9a2812029
X-MS-TrafficTypeDiagnostic: MW2PR12MB2348:
X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr
X-Microsoft-Antispam-PRVS: <MW2PR12MB2348AE4944C50646693B5139DA919@MW2PR12MB2348.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:2512;
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: 12zUlOjWKe4vt5/wUftJgzRrHMFp9cx0sCmcS/0HXrRrGcFkfrPHGcOc896tWlTICULJe5C1V5mkLdtIIAI8PZMberZ19c4U5nV9t8oW5XeJfcPfa60Miz7K6gb8iQ6530/SdqSXTKJUBoOVJO84gNJhVfuPC/NZQ3CM4YlpOAJ7RT5gKj/0WpXaez7wK4IlXjU+SFlp+RGyaXECNxr8+qV4LVWBOF30u9GlQA+xR2GKv/RvqPvUn7XcuuT9c13jUQDXA2n1j+5xcM8Zet6PA8bH4HendoBbEuNjA5JeNSIw66xX04+9HMzbC2MDTTp1KDX6AUOiU7/Uo/LMDJTaVbaqirFcLuADJh0Sgwb1k+Jjb1zYWOGSnlg2kbeeWth581j+BnAOjlxN7SwvxxF8jVqlscTt8QBUJlqmZBhjlKhrHG2TZ+H23sA6ORSlPFY1gJcaIyrojVNlBKAPEKv4UGNqwORDP0LkyACW18tv3dVSPrYfY7E76g+yWqdDEWBdFJmkf0hZ+FSTKuUI35BoBZlsAT4vcCjRhu8YQpaFxUtfrXCoanKH3zbgIpTtge5lt+Q3QUBVsuiet0BWT/h9KmE6pdNL0416rPedF09YVcFM9Nfw5+kPB2JzE73jFb9ysr3MinHMc6yX4MFNw14hPCEAwK4H5V1BpXCtLBxQBxaCpomTPMaB1nTP6i6bMpa8faFUQsZs5VSEmRamO60nZv+1HaBx9SbB+t8Ux8ktHcmCL4toUXeidf0IOqjZPiYknQYx+zmehMBYCvA6tqALcsF6ARXmFwNuDOgcvc+np7nxcYNRRjM1UNq2YQNs2WFH
X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE;
 SFS:(4636009)(36840700001)(46966006)(8936002)(508600001)(6286002)(82310400003)(54906003)(36860700001)(26005)(70586007)(8676002)(5660300002)(55016002)(36756003)(47076005)(6666004)(1076003)(2906002)(4326008)(966005)(186003)(16526019)(86362001)(426003)(336012)(2616005)(83380400001)(316002)(7636003)(70206006)(7696005)(6916009)(356005);
 DIR:OUT; SFP:1101; 
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Nov 2021 06:29:32.2690 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: e764fba8-b296-43cf-88cb-08d9a2812029
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34];
 Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT051.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR12MB2348
Subject: [dpdk-ci] [PATCH v4 07/10] tools: filter new Patchwork IDs by date
X-BeenThere: ci@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK CI discussions <ci.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/ci>,
 <mailto:ci-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/ci/>
List-Post: <mailto:ci@dpdk.org>
List-Help: <mailto:ci-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/ci>,
 <mailto:ci-request@dpdk.org?subject=subscribe>
Errors-To: ci-bounces@dpdk.org
Sender: "ci" <ci-bounces@dpdk.org>

API resource IDs are guaranteed to be unique, but they aren't guaranteed
to have no gaps, for example, the following series IDs are
nonexistent: 16157, 17181, 18235.
Filtering by the date since the last check is necessary to later
add support for fetching new series IDs in addition to patch IDs.

The script now periodically fetches Patchwork's events API filtering
by the 'patch-completed' category (see [1]). It parses the responses using jq
and passes the IDs to 'callcmd'.

Instead of requiring a file that contains the next patch ID,
a file containing the timestamp of the last time the API was fetched
is now used. Each time the API is fetched for new patches, the timestamp
in the file gets updated, and the script sleeps an amount of time specified
by PAUSE_SECONDS before attempting to fetch new resources again.
There are 2 variables, 'date_now', which is recorded right before
fetching from the API, and then gets written to the file, and 'since',
which is the last date that was written to the file.

Since API responses can be in multiple pages, the script will keep
attempting to fetch the next page until jq encounters an error while
parsing a non list response with no 'payload' key.

The pause amount between each poll attempt is still 100 seconds.

The script writes the date in Universal Time (UTC) format and
expects to read the same format from the file as Patchwork stores event
objects with a naive datetime object that is not aware of the
server's timezone (see [2] and [3]).

The package jq (Command-line JSON processor) is now required by the
script.

Example usage:
$ export MAINTAINERS_FILE_PATH=/path/to/dpdk/MAINTAINERS
$ ./tools/poll-pw /path/to/last.txt \
    '/path/to/pw_maintainers_cli.py --type patch set-pw-delegate $1'

[1] https://patchwork.readthedocs.io/en/latest/usage/overview/#patch-completed
[2] https://github.com/getpatchwork/patchwork/blob/580cc8570a05c1/patchwork/models.py#L1058
[3] https://docs.python.org/3/library/datetime.html#datetime.datetime.utcnow

Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
---
Changes in v2:
- Removed an unnecessary 'break' statement in the for loop.
Changes in v3:
- Fetch the events API endpoint for new patches as the date field in the
  /patches and /series endpoints is the mbox creation date, not
  the date they were created in Patchwork.
- Use UTC date format.
- Redirect error messages to stderr.
- Fixed typo in usage (specifed -> specified).

 tools/poll-pw | 63 ++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 42 insertions(+), 21 deletions(-)

diff --git a/tools/poll-pw b/tools/poll-pw
index bdf860a..ccc58f0 100755
--- a/tools/poll-pw
+++ b/tools/poll-pw
@@ -1,58 +1,79 @@
-#! /bin/sh -e
+#! /bin/sh
 
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2017 6WIND S.A.
 # Copyright 2018 Mellanox Technologies, Ltd
 
 URL=http://patches.dpdk.org/api
+PAUSE_SECONDS=100
 
 print_usage () {
 	cat <<- END_OF_HELP
-	usage: $(basename $0) <counter> <command>
+	usage: $(basename $0) [OPTIONS] </path/to/last.txt> <command>
 
 	Poll patchwork and call command for each patch.
-	The first patchwork id to be checked is read from the counter file.
-	The command should use '$1' to be evaluated as patchwork id.
-	When a patch is found and the command is successful,
-	then the counter is incremented.
+	The first date to filter with is read from the specified file.
+	The command should use '$1' to be evaluated as the patch id.
+	The date in the specified file is updated after each pull.
 	END_OF_HELP
 }
 
+which jq >/dev/null 2>&1
+if [ ! $? -eq 0 ] ; then
+	printf "The command 'jq' doesn't exist, please install it.\n\n" >&2
+	exit 1
+fi
+
 while getopts h arg ; do
 	case $arg in
 		h ) print_usage ; exit 0 ;;
 		? ) print_usage >&2 ; exit 1 ;;
 	esac
 done
+
 if [ $# -lt 2 ] ; then
-	printf 'missing argument\n\n' >&2
+	printf 'missing argument(s)\n\n' >&2
 	print_usage >&2
 	exit 1
 fi
 shift $(($OPTIND - 1))
-counter=$1
+since_file=$1
 shift
 cmd=$*
 
+if [ ! -f "$since_file" ] ; then
+	printf "The file '$since_file' doesn't exist.\n\n" >&2
+	exit 1
+fi
+
+date -d "$(cat $since_file | tr '\n' ' ')" >/dev/null 2>&1
+if [ ! $? -eq 0 ] ; then
+	printf "The file '$since_file' doesn't contain a valid date format.\n\n" >&2
+	exit 1
+fi
+
+URL="${URL}/events/?category=patch-completed"
+
 callcmd () # <patchwork id>
 {
 	eval $cmd
 }
 
-checkid () # <patchwork id>
-{
-	curl -sfIo /dev/null $URL/patches/$1/ ||
-	curl -sfIo /dev/null $URL/covers/$1/
-}
-
-pwid=$(cat $counter)
+set -e
 while true ; do
-	# process all recent patches
-	while checkid $pwid ; do
-		callcmd $pwid || break
-		pwid=$(($pwid + 1))
-		echo $pwid >$counter
+	date_now=$(date --utc '+%FT%T')
+	since=$(date -d $(cat $since_file | tr '\n' ' ') '+%FT%T')
+	page=1
+	while true ; do
+		ids=$(curl -s "${URL}&page=${page}&since=${since}" \
+			| jq 'try ( .[].payload.patch.id )')
+		[ -z "$(echo $ids | tr -d '\n')" ] && break
+		for id in $ids ; do
+			callcmd $id
+		done
+		page=$((page+1))
 	done
+	echo -n $date_now >$since_file
 	# pause before next check
-	sleep 100
+	sleep $PAUSE_SECONDS
 done
-- 
2.25.1