From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2371AA034F for ; Mon, 11 Oct 2021 21:30:11 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0E95B410FA; Mon, 11 Oct 2021 21:30:11 +0200 (CEST) Received: from AZHDRRW-EX02.NVIDIA.COM (azhdrrw-ex02.nvidia.com [20.64.145.131]) by mails.dpdk.org (Postfix) with ESMTP id 3883340E0F for ; Mon, 11 Oct 2021 21:30:10 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.109) by mxs.oss.nvidia.com (10.13.234.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.15; Mon, 11 Oct 2021 12:30:09 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fyfEPL5+nmddrctZPau2w7FDOrTHWC2gXmL35+ZXXEl3FYdK5/ovA+0eYkordgM5ofJaSmzctYw2BK1qhrTZ2uwnreWYnpwZjpTNHbiJz84V9Owz53TBxH6FYCtBKiV56KAAiSH/OJFExecXeFbyvKGRXf6ERqN6FPVDub1fzeyM6wpU8nHGF7kLoTYgWqIG7O6ilgMfMXWWZWnoUeVy2Fzt4XUTRqwmxDsBSTAAS03hqJJ+U6xgjshmBdMPvSK1huL1kHToR2S+/J9rDjXl6mIzJWnmKjCY7cFKNNQX0Nd8Esrw5oI2o6jpCB2vq16dNWUZ5g+inuz2JOKafWJ3lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BUBDtVMzHN56MOW2896CBvhUIOEPvAXzOkjE9ZlLVKQ=; b=G9SIzWcmZ0V29Va5hUIdCkGJTfd2CbNWKZbtCfegONvHayOQSPwS4Ty46sIap30IswmX5ClDGJCMnN72cvYO+xhnyvUFY96oIXEfsZFF2lZja63qwMjzfa0ZQqet139f/JPgXPC21Icdq6eTuoII+JFA5hY0amg0975ij2KpnZ1oBE3TfOBHXllDOYuhTItuap7AwhjbBOJtt2KtUWjj1RhGCYpyuyuNxq68auQjkutWUC9pSDYjdPlHWo9caTp4IQr7G7QpESr+h9rwFfBiP7Z1rUcqj+Ij8G33D0TR0FgF8aOfRY1PDHNKGE0SOUUM7IwE3dO8PRlj9bw/Ko3fsw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BUBDtVMzHN56MOW2896CBvhUIOEPvAXzOkjE9ZlLVKQ=; b=Of6wMnEg7IhsPTJin1ebm/a+KdahnodofFYKd8WsW4ijeuUn2nj/bYgUxJiHo8VOBwDa25prfvcYch8c/cwb1yF8stRptIU/Fq24/tx6lFXxOYwwWAhVVkF7BVe2w0kDOJ6o/fW9VczTAkvVtdWsiJ4EXynZbrTZ1cOgb+cl2oowLBJfHRHfJGWLtIhxOYtWPMZ51kfox540kWL0UjjcPSmJiSchiw7vono/WamjsMjIxccFsICCKblxLZqPxC6yg6UN3ogf/12cZ+WFumnGttRy6gxtIJwVY9ADFB6wx0pUKdn+zlEXevpa7C20JmMjsnEXHiTt+5lqIvXKLmeeUg== Received: from DM4PR12MB5167.namprd12.prod.outlook.com (2603:10b6:5:396::10) by DM4PR12MB5261.namprd12.prod.outlook.com (2603:10b6:5:398::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.19; Mon, 11 Oct 2021 19:30:08 +0000 Received: from DM4PR12MB5167.namprd12.prod.outlook.com ([fe80::3c73:8e07:c9ba:f6db]) by DM4PR12MB5167.namprd12.prod.outlook.com ([fe80::3c73:8e07:c9ba:f6db%3]) with mapi id 15.20.4587.026; Mon, 11 Oct 2021 19:30:07 +0000 From: Ali Alnubani To: NBU-Contact-Thomas Monjalon CC: "ci@dpdk.org" , "jerinj@marvell.com" , "ferruh.yigit@intel.com" , "david.marchand@redhat.com" , "juraj.linkes@pantheon.tech" Thread-Topic: [PATCH v2 10/10] tools: skip the IDs we already fetched Thread-Index: AQHXteaDE41qb2aGHky+hgvhdsrt2KvOPJfw Date: Mon, 11 Oct 2021 19:30:07 +0000 Message-ID: References: <20210906154537.1299-1-alialnu@nvidia.com> <20210921143542.4412-1-alialnu@nvidia.com> <20210921143542.4412-11-alialnu@nvidia.com> <2586128.JSg8es4y6H@thomas> In-Reply-To: <2586128.JSg8es4y6H@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 4323d519-35df-43b7-4867-08d98ced88bd x-ms-traffictypediagnostic: DM4PR12MB5261: x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-transport-forked: True x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 2OIn65IwjQtdPjzr0BKagtByOVjqDnHgeUs5p7DByC7joEEiNYg3obLZGaAb3Z/pSfHQdzU7hP8N++sk3HpjPcVh7cu4yBleHFIqlJm2xWE/ypQX3ZBqIZ2EI+MehU8ZkFI4PHDd0auNmrUybFFvklPaOuNhR9AxlyppQ0Yr1bZo9euTgwGnb/wxIetXMOOQTsv39CEalnjD6OnYQx9MPJG9ERCXNCul0SIw+nawW9kND5mmzX5o8nn+qwGPLDmyWqQBXIsnia9u5Qwfw/2S68m8xMfaG7t6OBFukj1cSaNR9yg/E9T8zEYhhlnfZF4OK04N9mmegqOSFdO/67uEy3OBJMrjpNZlfsDEqBadOjca4Okn8rn5JSSeXdb5mVVa2JcJvBMM9YPCbYSeCT68kYF4voCO3sxhO77Bnq+JE0FDIElL1MyuH3RYrYUWz6+QBMYwraMMpt9reY0UAmaZOMABKHxsnrrEmNJR1o+SxnWx6RPsEbk7l6vC5zHWxmshA9A6Z+V6MI54omNWJfw/zXg5ah9FGqXnClGnn85JbbtnFqdCUoMI83yCBHqwvdaUn9i0/0z4AsnhBxXR6MU88oVjXSYsi7YREUDbo+McjOesi18SiLb+eyRMJPhS5OEOxF7ITWOOQwRHgWWLU7P1DwB428MJQOlEVTdFmMbkUX7mPqC1qZ+k+StXb1wPGOrGPg0DiMUPgWKUFJdM8E/bkg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR12MB5167.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(8676002)(7696005)(6862004)(52536014)(53546011)(6506007)(64756008)(508600001)(26005)(9686003)(186003)(55016002)(71200400001)(4326008)(8936002)(66946007)(76116006)(66476007)(107886003)(83380400001)(66446008)(66556008)(5660300002)(33656002)(38070700005)(54906003)(316002)(86362001)(38100700002)(2906002)(122000001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?TC1D5jGfggE5HGHAgf9+8jao2O7I5nWyWHStQS2OwdPMuRdRvlnqNFwcyHiP?= =?us-ascii?Q?Tglsad4JlS9pw19ph5g9Vof2ywl6spLJhGNDhfHl+xRERxDiiunmTqJSzfFQ?= =?us-ascii?Q?2T+cE4abMncmFd59RNzibfFX4PXlX0e3rHOs4NaYGInq67IHRyfCl7se9M4O?= =?us-ascii?Q?hiO0iDH0hz5+r1bKwuYNoZ6u2TWyf7rqzYYaxRZavP+gOshYTrjNI78PWYOq?= =?us-ascii?Q?WSIY7nMaAcZs3Cn/Jr2GsptFQnNoxsX9QgHzvnMzUJTVgOr04Meb2mxOKB+e?= =?us-ascii?Q?zHF6pnVCgc9gLJXpoQBTbXAW7hjmT79f6hPvUp6ZR3zK4q2EfT/WGv9OnUcF?= =?us-ascii?Q?+AYi6+grDtzaGNJvtH3SV6xG9/4gX6vGzNZZVpqztLVzREALpoTnpV2bMa4g?= =?us-ascii?Q?tDbrmstObBX6smTk89+D8wjEbeitOfAdbglsc+5HFwvN3DdslF1NUN0rcs5c?= =?us-ascii?Q?7IZ+N3VMaUAWBzHU27Kc0W1KuJcIYGlZowdVzjOvqsgIb2QOxEJCZXvAwEHb?= =?us-ascii?Q?wwAZNjWgLq1aTsT5dPbDKqaa3g8WGjEBIhj+8mOA7O1z0JBUfvKUpizDrMDa?= =?us-ascii?Q?lwkD52TCe5yvRbWYWDnpAW7ER0dsXolSx/ZN7OuPPNv3NnZ39LwXAVa8WmdP?= =?us-ascii?Q?/8RqX5N+uIoii81qCIJds6XloucCCSoP7KLWpVHjfjS/Pm359/y7nx00s2Xv?= =?us-ascii?Q?Jha+AJDbz0Y6uzjGfQ/VZ9Ebz47CmKkLyhYYedaXV7heJKpKgjRUK2VDiLPk?= =?us-ascii?Q?oolGn2l7o3FDgDU5a2qU1vhTWwnSODNFtqBhlWdUK0OwKKvctKYJHfCVcSnm?= =?us-ascii?Q?MZSVzeiB8J5VPdbpZ2Rdx37LoP+suaHFrOOcXF1sZqMA0H7fSRveTsaiwXSY?= =?us-ascii?Q?jcMWzp6ifQ061q/+TCDNH5hayzm63vsGY4PBsD5i1MjMSH9OCI7W/2ljUuZJ?= =?us-ascii?Q?8yHVfdaNMot5HeWnDDppDCCAuud5dWMtzHT2VJs3Jr6VUSjUic2iwN6eRi0o?= =?us-ascii?Q?rJQhbvBCjHeXIe7QeF2plk4P6DbJTl/Y6hym0qaNNZIFEuDKw0FNJc+Eq+PB?= =?us-ascii?Q?pQWcjL1wcnVwrs/rMPDp1e7OzaUgNaDY+hwmEtYGsUQCyJUTQQTOzxgDcpXA?= =?us-ascii?Q?NWBA8jNFO5yEiN1NhcUkTDWMfhcHlko5zeBsfjcQCnmDn49pYb32qlNHBMUh?= =?us-ascii?Q?SklzHC3r06DnuFisp/SP3CyRbza0g5ppbWlf3HG0XQZeWbWUZxoWDalv68LW?= =?us-ascii?Q?31kblg9TzoVyWGShomb1zMcPGcrKiPqgU8B+e6qbyhc5VoDOBy+exG/gqm35?= =?us-ascii?Q?NNQ=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5167.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4323d519-35df-43b7-4867-08d98ced88bd X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Oct 2021 19:30:07.8282 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 2DJwj6gTPseXuIBSBY8sQ1fUzwWZUhx8fmyW27LcokY2sLHMmhNmaNITJCWcp9ZXOS2j0o7iSBeLCkUXmdXelw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5261 Subject: Re: [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" > -----Original Message----- > From: Thomas Monjalon > Sent: Thursday, September 30, 2021 1:33 PM > To: Ali Alnubani > Cc: ci@dpdk.org; jerinj@marvell.com; ferruh.yigit@intel.com; > david.marchand@redhat.com; juraj.linkes@pantheon.tech > Subject: Re: [PATCH v2 10/10] tools: skip the IDs we already fetched >=20 > 21/09/2021 16:35, alialnu@nvidia.com: > > From: Ali Alnubani > > > > Store the IDs we already fetched in a file and don't > > run 'callcmd' again for them. >=20 > We store all IDs. Should we manually remove olds one from time to time? >=20 Do you have a suggestion for when should we clear this file? Maybe each tim= e the script starts? > We need an explanation about the strategy, why it is needed. > I think it is because filtering by date is not enough. > In order to not miss any patch, we should request a date earlier > than the previous fetch and skip those already fetched. >=20 The reason this change was made isn't because filtering by date is not enou= gh, it's because I want to avoid feeding the same ID to 'callcmd' more than once. This can happen if a patchwork ID was created between recording date_now an= d fetching the API. I don't think we are missing any IDs, even without this change. > Where the "earlier date" is defined? There are 2 variables, "date_now", which is recorded right before fetching = from the API, and then gets written to the file, and "since", which is the last date that was written to the fi= le. >=20 > > Signed-off-by: Ali Alnubani > > --- > > tools/poll-pw | 9 +++++++++ > > 1 file changed, 9 insertions(+) > > > > diff --git a/tools/poll-pw b/tools/poll-pw > > index e197a91..dacf34b 100755 > > --- a/tools/poll-pw > > +++ b/tools/poll-pw > > @@ -65,6 +65,11 @@ if [ ! $? -eq 0 ] ; then > > exit 1 > > fi > > > > +poll_pw_ids_file=3D/tmp/poll_pw_${resource_type}_ids > > +if [ ! -f "$poll_pw_ids_file" ] ; then > > + touch $poll_pw_ids_file > > +fi > > + > > URL=3D"${URL}/${resource_type}/?project=3D${project}&" > > > > callcmd () # > > @@ -77,7 +82,11 @@ while true ; do > > date_now=3D$(date '+%FT%T') > > since=3D$(date -d $(cat $since_file | tr '\n' ' ') '+%FT%T') > > for id in $(curl -s "${URL}since=3D${since}" | jq '.[].id') ; do > > + if grep -q "^${id}$" $poll_pw_ids_file ; then > > + continue > > + fi > > callcmd $id > > + echo $id >>$poll_pw_ids_file > > done > > echo -n $date_now >$since_file > > # pause before next check > > >=20 >=20 >=20 >=20