From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ABC67A0C54 for ; Mon, 6 Sep 2021 17:58:59 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8564E410ED; Mon, 6 Sep 2021 17:58:59 +0200 (CEST) Received: from AZHDRRW-EX02.NVIDIA.COM (azhdrrw-ex02.nvidia.com [20.64.145.131]) by mails.dpdk.org (Postfix) with ESMTP id 303D3410E9 for ; Mon, 6 Sep 2021 17:58:58 +0200 (CEST) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.49) by mxs.oss.nvidia.com (10.13.234.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.15; Mon, 6 Sep 2021 08:58:57 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=k8hAfhpHeRFU7IS22FTMp6YvIwube1w54WDfcS1PiT9UXYWi+sXE0CwUY12TF3O4hP6WINTIfk/AsVooYDWM9wPPTUO25e0p6nuS4xDkGn4JN0Tp/8sLhteCfrIq8Qoigr8U68/R1LBEfDMsaRxiNtHkCETdZz8FDKpxLBGbfCNqhHI2BHg4AezvNWhDbHLZTU0HJF41GDOG2zx6OKnUTzyCLdS1rdR3lFifdklkoYlvRWUmbQMOSMBNIraNWsoCM2c/xBvfuG0Q8GbKfs42Ka8iBRcAcuwsXiy2NrYcVcVS+z945g/aoJaRphP9Qmn1Y6RaftgbioIhJsJVGAMSMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=fqIcw898DysIUEBFCWwsoq1TJzsVqSC63l1CBvCYUSA=; b=B0+fNUx3VFYj0nziuptGviabCNAMyIsqX51YNvQjpRhtJ29bKM2gqSbOSFKUUQp2uzvF+68BNTPC6Tl5olnslaVMbxJo/4N7sL18CeP44I4Uf/WOsiF81nUMwgy5SqxajwavW/GFQVyD8geRV3VxQvL9fYiVQWjHfQRw/4vB5cpF3ICJsujo+59VMBQsU0pSlkaAFEZSqH4e/cgsoGskKoIFghQ/Ub1dxaa+vHpfJtpReObzK1rYxscH+KdT2Vw+dLon3D7psGiIKx9vxfjBISZWhk5TpnbSsduXPIOn55hrg1/a7sBvxc0iz7Z1VdWJphpco/bMe8VadikM8liy7g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fqIcw898DysIUEBFCWwsoq1TJzsVqSC63l1CBvCYUSA=; b=lqrIcOFX0TKuXnmK9P7pgPVMsuXXwBwktoxe8ddF2aL/bAqQYN6c3HKA0vEtDyWcHhEDPmJVLz0F1LZLrOYTDYpjPMLja54cBFpzpeggLkDphcDVzbmh1/xCNpuWfv1nWv/PTXflUVJnew7MzEumYv1cld0DQV3gvT1qOVrodXc5ytyz+4JUhfSmUB7ppSI5iVkKJPXDShYvJOmCwPn0hOJnabxvFvzieMguf+J/DJUy7SbxD3lISGp98EHtlG+vl+1RT5PnWI/kORAMbxrmpS+lxWjNkPZQLFe0c/I32tJhYequ1JI6xXSgzZucKSMGDJW7t7TjDdoswxvRTMPkNA== Received: from DM4PR12MB5167.namprd12.prod.outlook.com (2603:10b6:5:396::10) by DM4PR12MB5374.namprd12.prod.outlook.com (2603:10b6:5:39a::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.25; Mon, 6 Sep 2021 15:58:56 +0000 Received: from DM4PR12MB5167.namprd12.prod.outlook.com ([fe80::5ce5:bfc8:30e7:759a]) by DM4PR12MB5167.namprd12.prod.outlook.com ([fe80::5ce5:bfc8:30e7:759a%5]) with mapi id 15.20.4478.025; Mon, 6 Sep 2021 15:58:56 +0000 From: Ali Alnubani To: "ci@dpdk.org" CC: NBU-Contact-Thomas Monjalon , "jerinj@marvell.com" , "ferruh.yigit@intel.com" , "david.marchand@redhat.com" Thread-Topic: [dpdk-ci] [PATCH 7/9] tools: filter new patchwork IDs by date Thread-Index: AQHXozZr1GT2ZhsLMk2x3PFPjWPtOauXKLdA Date: Mon, 6 Sep 2021 15:58:56 +0000 Message-ID: References: <20210906154537.1299-1-alialnu@nvidia.com> <20210906154537.1299-8-alialnu@nvidia.com> In-Reply-To: <20210906154537.1299-8-alialnu@nvidia.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ccbb2c80-ea2e-42d4-0793-08d9714f3b80 x-ms-traffictypediagnostic: DM4PR12MB5374: x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-transport-forked: True x-ms-oob-tlc-oobclassifiers: OLM:2449; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: PW9NGBmwtbFZRarBXH4Kw95sfxPUjqQjU7TucUDsWPo2zCxLPd4DaFCWcLM74EjVEbcpkr8EJa7/4kMWdM2KGukqOSdZK+2BU5oScEKISF3bXHtPh+qzzvBH22Q7P/Wlzc8zomJeVxQ8WZ+xK/Du4VTehZ7FXGGUUz4VHXwVNlq9UgN7KYyPuIEn9fINan1pVDWBWohzRt4yr5CgYECppkzl+MlGYhFaSCBSThLCEC2cGS38X2IOf2l/FBGJwOvw+45eXfCFK1KaS7laf59EFJF2igl6yO0swUFBmAn2o613dSfAFYAwS0ncQu1/ZhqgZaHQ9oPwYSp0UxJkYhewzRyZbjHnKod6iJqRZ3pYty72l9tO3AZWNWbOgF7UuRLNxeEbsB4Dy/ogOOnX6cz/5mZD7yTFPPXBIMItaz2GSE90gU9Vxe3YQj+W+qAN57KOCup5j+D/lB3e9jWtmc2/KWX7j73EmCzdQKefkw6HzbfxMn2BuFiBaMbrtUrU84SML+AO67TSruh5PFIJd9VppcfpJKdqwTOuYk92Cb8fD6r8d8mHrn02QypWpBjpb6eOWsYbKg43z0V7+emiu5gOToam8/RIYOw/ZITchx+V0E+ouac/fZQ4ZCwD+SOhmsO0DRZzefDepCt+p+XHSXqEj+1QEBXOUPhtjYh9r1uAShNc0/1BnWG/LUGguVpLdSe3d1Plb8TqBbX0KbpfgxBl4o12MTuddRnwKj3zNncXPmh99eU792LMgjWurIYs+a8HXtDtcmcK453IYSn2G7gJ4eE0Yk8HH0qp9HEhgSWsJR4= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR12MB5167.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(136003)(376002)(366004)(396003)(346002)(8676002)(66476007)(53546011)(33656002)(76116006)(66446008)(64756008)(66556008)(122000001)(54906003)(38100700002)(5660300002)(478600001)(52536014)(8936002)(86362001)(4326008)(107886003)(66946007)(186003)(26005)(83380400001)(38070700005)(7696005)(316002)(9686003)(71200400001)(6506007)(6916009)(2906002)(55016002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?+GdKmG+pB50fGqCyyYZYUZXA2voBW4L6dBz6ZkYabE+gid0CqC6+EdZ7SJj5?= =?us-ascii?Q?/yEpOf7kJyuURYPB6BVm6GOZOOfwRuXTMUOEeq1MCeSdasjZEY84Dmca1+tI?= =?us-ascii?Q?/UqzIGNrF+JwSdm3nDwWgXLJr56Vn5SFROSHfBs6kLnDIZVeHHtd0HRDrUaj?= =?us-ascii?Q?fAP9F0/S+R2gUU6X+tbjbcLNULC98iYHik3xhB8HWieUFBPxNH0JWin1a7Rd?= =?us-ascii?Q?3/Mm1GCcaOLCFesstaBMaxB3KJ3+mHCpqVz92XTUCUHahZ+6+AT6GMmrsB4H?= =?us-ascii?Q?G5+uDv1bSm2q0jVulZBQI3FFD6TKylrNFwEdZYV7hiN6PhGyKuEsgTBcBwrI?= =?us-ascii?Q?bFSUtqJ6TtLX6Ksc4TYTVMz2sRoHwW8NPTE/pmfQ8q66/W8oVtISiK4G1+8A?= =?us-ascii?Q?7rgw08PTRC2hPqpeVcelsH4q9+ydoyYflrBy10DadR+ACfhQ03JoiTy0IN1+?= =?us-ascii?Q?xjgdeNJkPLr5N4eHV9cwuXPR6ykCu3Dl9fKp/oQRiDIVlL9DQ/+dB60selLT?= =?us-ascii?Q?RF9nyEcVjO18Fn1A7lz2MwiPqssm83SVCweNXN1tPdXnidmHopg1atrFT7JT?= =?us-ascii?Q?QLTYVD9cnOYkwbEL/ADihdufVtTjEJMqaoi9Cw4IAsxIv4K3X1+OWG5u2IG8?= =?us-ascii?Q?H23jAN/VpvOuG5FTDUCNTVeJp8JUQuX6nAyy+nnK2623gHlvY5bC3GQSN+y3?= =?us-ascii?Q?1amoHCe4NR3fRssBkOwEJRPRTSCi8KiUwspdCCito3yfgawoEbkbX2D2cmAk?= =?us-ascii?Q?lUsIBJjTcctQLAqyyWAsbe7y3xiP+1u+J09Cy57M4FTgxQxODLDh7Nlz7JRy?= =?us-ascii?Q?Szpe1sNGbtaDIkT+oghWorQmjMuc/lTOXrkDjU0J8ai5KdmexTiCdWmpQcUm?= =?us-ascii?Q?0ZUPbDiccDQfRPFX7h0sGx87d/KX9ZEWGMQRuR2EPqIWJ9ftXIASHBgJopme?= =?us-ascii?Q?wrwKkDGgLRokeIGUzk/2wKb54/F2xUWQ510zYCAKwGi+zqrRyAKD9I4HI1ES?= =?us-ascii?Q?pILtBEvTvn4D4TLJRrM8QZsLARfmI0BUMHC+NoLkIbS0ToUThWuRA+XYNVC9?= =?us-ascii?Q?Di5ahZRM1HyUO/nnHSGqn1U6K/1VChzNI5i5ojMYA5mNROX1IPw2QwbBJrU2?= =?us-ascii?Q?KICfN97kqWMqbQ6FZNTEgZIJPautNLAqdCl7mKaqBe/9HbNnhAs9j8vhZS0h?= =?us-ascii?Q?Lkw8SJsxWyjXJWKcOP/dQGxAh8wq9iGZtKlRqLRYbwDRfJ9NnbPvFbmM6naY?= =?us-ascii?Q?OGehGxRfKYJjyaOFos6ZjvOzicfPSOuTIWedZ0mmeiitP3zw4C5q7/uu/bUs?= =?us-ascii?Q?m0K27FBnzAQD137gOmxP4DtL?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5167.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ccbb2c80-ea2e-42d4-0793-08d9714f3b80 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2021 15:58:56.3301 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: m4BJRAvEMEphN9qlbeo13lpFSf3vVhpa91TPLq1QfOIjFMKGT6lcF2i74pimiOlGmOJSWy618X8d+3MGXGbkPw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5374 Subject: Re: [dpdk-ci] [PATCH 7/9] tools: filter new patchwork IDs by date X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" > -----Original Message----- > From: ci On Behalf Of Ali Alnubani > Sent: Monday, September 6, 2021 6:46 PM > To: ci@dpdk.org > Cc: NBU-Contact-Thomas Monjalon ; > jerinj@marvell.com; ferruh.yigit@intel.com; david.marchand@redhat.com > Subject: [dpdk-ci] [PATCH 7/9] tools: filter new patchwork IDs by date >=20 > API resource IDs are guaranteed to be unique, but they aren't guaranteed = to > have no gaps, for example, the following series IDs are > nonexistent: 16157, 17181, 18235. > Filtering by the date since the last check is necessary to later add supp= ort for > fetching new series IDs in addition to patch IDs. >=20 > Instead of requiring a file that contains the next patch ID, a file conta= ining the > timestamp of the last time the API was fetched is now used. > Each time the API is fetched for new patches, the timestamp in the file g= ets > updated, and the script sleeps an amount of time specified by > PAUSE_SECONDS before attempting to fetch new resources again. >=20 > The pause amount between each poll attempt is still 100 seconds. >=20 > Setting the env variable 'TZ' might be necessary if your timezone doesn't > match the server's timezone. >=20 > The package jq (Command-line JSON processor) is now required by the > script. >=20 > Example usage: > $ export TZ=3D"Europe/Paris" > $ export MAINTAINERS_FILE_PATH=3D/path/to/dpdk/MAINTAINERS > $ ./tools/poll-pw /path/to/last.txt \ > '/path/to/maintainers.py --command set_pw_delegate --type patch $1' >=20 > Signed-off-by: Ali Alnubani > --- > tools/poll-pw | 56 ++++++++++++++++++++++++++++++++------------------- > 1 file changed, 35 insertions(+), 21 deletions(-) >=20 > diff --git a/tools/poll-pw b/tools/poll-pw index bdf860a..e104dab 100755 > --- a/tools/poll-pw > +++ b/tools/poll-pw > @@ -1,58 +1,72 @@ > -#! /bin/sh -e > +#! /bin/sh >=20 > # SPDX-License-Identifier: BSD-3-Clause # Copyright 2017 6WIND S.A. > # Copyright 2018 Mellanox Technologies, Ltd >=20 > URL=3Dhttp://patches.dpdk.org/api > +PAUSE_SECONDS=3D100 >=20 > print_usage () { > cat <<- END_OF_HELP > - usage: $(basename $0) > + usage: $(basename $0) [OPTIONS] >=20 > Poll patchwork and call command for each patch. > - The first patchwork id to be checked is read from the counter file. > - The command should use '$1' to be evaluated as patchwork id. > - When a patch is found and the command is successful, > - then the counter is incremented. > + The first date to filter with is read from the specified file. > + The command should use '$1' to be evaluated as the patch id. > + The date in the specifed file is updated after each pull. > END_OF_HELP > } >=20 > +which jq >/dev/null 2>&1 > +if [ ! $? -eq 0 ] ; then > + echo "The command 'jq' doesn't exist, please install it." > + exit 1 > +fi > + > while getopts h arg ; do > case $arg in > h ) print_usage ; exit 0 ;; > ? ) print_usage >&2 ; exit 1 ;; > esac > done > + > if [ $# -lt 2 ] ; then > - printf 'missing argument\n\n' >&2 > + printf 'missing argument(s)\n\n' >&2 > print_usage >&2 > exit 1 > fi > shift $(($OPTIND - 1)) > -counter=3D$1 > +since_file=3D$1 > shift > cmd=3D$* >=20 > +if [ ! -f "$since_file" ] ; then > + echo "The file '$since_file' doesn't exist." > + exit 1 > +fi > + > +date -d "$(cat $since_file | tr '\n' ' ')" >/dev/null 2>&1 if [ ! $? > +-eq 0 ] ; then > + echo "The file '$since_file' doesn't contain a valid date format." > + exit 1 > +fi > + > +URL=3D"${URL}/patches/?" > + > callcmd () # > { > eval $cmd > } >=20 > -checkid () # > -{ > - curl -sfIo /dev/null $URL/patches/$1/ || > - curl -sfIo /dev/null $URL/covers/$1/ > -} > - > -pwid=3D$(cat $counter) > +set -e > while true ; do > - # process all recent patches > - while checkid $pwid ; do > - callcmd $pwid || break > - pwid=3D$(($pwid + 1)) > - echo $pwid >$counter > + date_now=3D$(date '+%FT%T') > + since=3D$(date -d $(cat $since_file | tr '\n' ' ') '+%FT%T') > + for id in $(curl -s "${URL}since=3D${since}" | jq '.[].id') ; do > + callcmd $id || break > done > + echo -n $date_now >$since_file > # pause before next check > - sleep 100 > + sleep $PAUSE_SECONDS > done Thomas pointed out that the script can fetch the same ID twice in some case= s, and that we can miss some IDs due to time synchronization. I'll look int= o that. > -- > 2.25.1