From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6220DA0C43 for ; Mon, 18 Oct 2021 10:04:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3341140141; Mon, 18 Oct 2021 10:04:22 +0200 (CEST) Received: from AZHDRRW-EX01.nvidia.com (azhdrrw-ex01.nvidia.com [20.51.104.162]) by mails.dpdk.org (Postfix) with ESMTP id D44D24003C for ; Mon, 18 Oct 2021 10:04:20 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.174) by mxs.oss.nvidia.com (10.13.234.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.15; Mon, 18 Oct 2021 01:04:19 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TZKlwWz5HyXjIX9cG5S4AcgA/tYE9JRK8EOH9SOLeTQixmibzZTJHmIiGmRD+K9O8RbvFL7vtuomXFx4XoQ0VTTb4r/hGISIhqRVZTQjX6dKKcCIctKMCBlRNuVPWa5QE3z+g8qJpOOSf3G8jRo7X5zVkHy8hCKqKWUOKKFzwsKkViQiqtjKwy+E59OjsiTAkOOInrsy86HMBHl/kvuRbLl/vJdGqX9O7pkDRefXWFF/uf8jc+5V+xVXmIASAuefkYGQoVNRTHD4S48jRy6IydghlWjbpVyaTSsx5RDrpg/xjYRs8XNc1++HPO9PEdNm4VfH5qjfxpTu1S81/B0I3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fnXtKEKArWWf+VTUaW6blgo29S/tbypxtTlzEeQEdX4=; b=Z35q3dEGzOg7G4Ibv91jLqw3rlP4/LmZEE2+TDzhe+Aks+XlPamXzNb9u6I3ZgQsoxD4BICQrY/Tzh6H/XEi0czpBRFJqx95PajNHdlglokxuh9jmKiuae68wlfVrSZ/IdIFs76SaxecJI4HSTMqmZ8Rxgx/nIaBk/T5YFdu7qcxZ+n3xVMDWLYygtlixBbOFa3XuLXbiNPum51J8qeuhjP3wMGrur0+VsH7glM8luE9Vg1sImHHkKhI44ifhZricHJyzN99QcXu8vssPmwOuwqDyauEomnpOzJM5zv4Ggq1jM1DRktYtDp/jQZCgzFMunwTqA311HeE8MoivX04Ng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fnXtKEKArWWf+VTUaW6blgo29S/tbypxtTlzEeQEdX4=; b=HHnNHv0OCSktp5x/W6AwQq1ZK7dGnY6xEE37gUde5AFct4G6q9FfXUAVwpBH1aD7liby74OFO5I+oH+q0HkHuMonftZoJeF2pXFxEBj7fq4qvJypwstR51wvPJvTvXP13cBNIwNx5arrXqBGg3eZxbC/RwMGeEhUYew4vsUmff0s1cHo6XvTTjvlgW7Rlk/lMFGIPe0kKimc++8uDPjQSFgLyWlbJ/MkFvkOSfuq6CmfjgBvrPjPwpzLUTz0vQKjxCxmCCpsZNS9zNubQm4A2U8qfbfi8EPrkHsSJajCj3t8kLrfM1mq3wx3Y2odyUvohoo16t009s6i7ria6RDDSQ== Received: from DM4PR12MB5167.namprd12.prod.outlook.com (2603:10b6:5:396::10) by DM4PR12MB5309.namprd12.prod.outlook.com (2603:10b6:5:39d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17; Mon, 18 Oct 2021 08:04:18 +0000 Received: from DM4PR12MB5167.namprd12.prod.outlook.com ([fe80::3c73:8e07:c9ba:f6db]) by DM4PR12MB5167.namprd12.prod.outlook.com ([fe80::3c73:8e07:c9ba:f6db%3]) with mapi id 15.20.4608.018; Mon, 18 Oct 2021 08:04:18 +0000 From: Ali Alnubani To: NBU-Contact-Thomas Monjalon CC: "ci@dpdk.org" , "jerinj@marvell.com" , "ferruh.yigit@intel.com" , "david.marchand@redhat.com" , "juraj.linkes@pantheon.tech" Thread-Topic: [PATCH v2 10/10] tools: skip the IDs we already fetched Thread-Index: AQHXteaDE41qb2aGHky+hgvhdsrt2KvOPJfwgADA54CACX/sYA== Date: Mon, 18 Oct 2021 08:04:18 +0000 Message-ID: References: <20210906154537.1299-1-alialnu@nvidia.com> <2586128.JSg8es4y6H@thomas> <2024535.ONPTUEBfDP@thomas> In-Reply-To: <2024535.ONPTUEBfDP@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: acbde6bd-b536-4c3f-4e1d-08d9920de2da x-ms-traffictypediagnostic: DM4PR12MB5309: x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-microsoft-antispam-prvs: x-ms-exchange-transport-forked: True x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: YGfTLAIyEhf8Nc7kAGZNafGL5GqXXyOnl63rCyoGKyDIt4UlT+u8avaEW3lwP9OOrBY1Bm2qh/3UPHi0blzi+uQ+nb6IE5CdzxRup9eeHYh6qgHi0onAwbYUugPAsF8XwSFsWzLBJAtRdYw9hOKbLate8HlxVaa4IyTuso+BH1ar0MLQsCkhHLC5y/eNBo5sGu8wkS1o9100IxoTMC82j4eaVsKYEPMKlPwaC5xVcsrs473xro8Tk0w32QbngSf0ZVOmg2pyMxyLU/O4VzcfTXWPxMiFDEP72g2p00mZe/qSZm3LcHB94ld5Ba/2PVRHJmz9f8sZvVTY9XklbrF+C3OKQBrDFSUXWQy0fwVKoPERD+NmXE+t6NprbFOrhAjOf3OUaBN+Wzwpe2iH056GFwZoYsLp9BTyqmki4aH7iSWcHvejUTtwXurdkLBNdRgEFEHEKag687nVRUZyjuEjdNLUl+q2ppAmVXdLpBBpLXy0LF70usAEHIJgNv7SDPlii+LoztexRraeujeowC3CNgonEm9RzaOyVeoPoWv1jXt0TV8Gt8GHeGkNh2sbQox+uI/gZel7RTgCudGVvWUdW1ULjtxQ4oKWFYO5tTILS6wuT5JpWEiVdZjYc0WNT3QV7+ItgTEoBJjy1ldOdlSi/IrSjsf8xdlbFTXiF0Ej113F0FZpjllMB7rZ85fRblHARcuq2W0iUbB7rZl1KpyG/w== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR12MB5167.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(9686003)(122000001)(38070700005)(38100700002)(52536014)(71200400001)(54906003)(6862004)(83380400001)(55016002)(26005)(86362001)(508600001)(107886003)(66946007)(316002)(66476007)(64756008)(66446008)(5660300002)(66556008)(2906002)(7696005)(8936002)(33656002)(4326008)(186003)(8676002)(76116006)(53546011)(6506007); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Xk9C47wsajlWbWPDwsLjEgP7EH2d8oKACRq1me7o1fLiYo3324FNDHBOlmDO?= =?us-ascii?Q?nuVwhvrm9AnyBLPncqdA3ciB7VJGtRY2/SzXyv/+8NiLtckApdE/JD/y0RB2?= =?us-ascii?Q?jdBCsDvTT8DDT0SOlIc8nEqhweo63kmxV+YW1HnwDCllZz53o3NNFQOKZ4wH?= =?us-ascii?Q?HcdQceetSH/Hn8xhqoe2PErqU2KmPoTfPd+vAJUrRc1g0YA+Eyp5A8/Dm7BP?= =?us-ascii?Q?exVKo0aVo1aV5wxwVO1bAMyYdyd02z20dneMDxJQ+W8BbvudZpkTnLXV5bfe?= =?us-ascii?Q?a6ir1FvBlHS2Z3YC+ffu9oI+3rSD/ZJrnv6nbvjYn5/w845eHSPs5teaRBEm?= =?us-ascii?Q?QLf01UU4QR4UbTTnsaWVr2ioPaa9/dPueOJa5Jlu5+wL0SED0YAg4ZQg90dP?= =?us-ascii?Q?33666vv7kAxWtHhaEDcpO6gZgPfAMF84nXhGAqmPt6Y53dyIiWKD9fkXdeb6?= =?us-ascii?Q?r9JDC7A2l873llrp3N/FGNJR3ARLWg/YbYcWo1qixwnnCFUoyW9e/epy6oeh?= =?us-ascii?Q?AyGpRbv4jyUr/s3UOJYSLUluh9X9nHt5B4lscDtPluzQRbhCPK1Ozsl1fZwl?= =?us-ascii?Q?AvBZ6V+hWlTNmu00HjMnBQEhUIB9IzjRya1XD/Nnm0pxPfzxL+x66s5aEPmT?= =?us-ascii?Q?SCqQJnM+5Q4un9Ex+buDJtLlM1SmdBvBPsBjwvWh0N2Zx99xEMkzUN1jtaT1?= =?us-ascii?Q?s0iUzwgSQ3CdcGcaW9gm/WLczSEAMZoUPFAa3REdbrCQc22mR4ab/xWaF/PZ?= =?us-ascii?Q?igGZrYNvUGjhye0UjZVgQ93vSe4Khi0R+13GlGlMmnp5LhxRvhqjH3uapdx/?= =?us-ascii?Q?gF3dcZ78tvBNUPshWlehpPZON+Rxa19IlArw7naF9RkVMqV5WXaAUgrWSL7S?= =?us-ascii?Q?boWLliuXxxA3EnqWWD8wsW1KUsjb9Njgd8Rta/KZx/phgZlihy9AxFFQ126w?= =?us-ascii?Q?WxDc44vrdqKDpk5Mh1xcslFGIJ0C2IvU55LX+g1uS6E/DXPZZ6kZ/jIYAjT9?= =?us-ascii?Q?0qAJgooq8sJGs+o3RGmI8RIgEXAsxEfkEDSdSQwAXUfNlTtRhHbcFwiw6Zrg?= =?us-ascii?Q?J1JWvLSK2StXgvdvyeRh3TUps3DJl50NAf6vP0t0goE5KKjGrLGLCGlcRUN3?= =?us-ascii?Q?26yzDLUTWyfcvSWW7SbBsTT650EM8irXjjXMEXSTmz9lBZc/LKp/D2Xoqb6I?= =?us-ascii?Q?a+ZKTIUIovSSNWkXDeihfozwJnjlAwd7izhZu8p/b8zFD4qFt7JDclGAdKpV?= =?us-ascii?Q?rth73HAtZNKWdmZWiL8fQTnZe1hzER+MvIuofxfGVcgMn44omdMjGcQe3S65?= =?us-ascii?Q?pvt9dVQzifluV950SXI6DxFY?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5167.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: acbde6bd-b536-4c3f-4e1d-08d9920de2da X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Oct 2021 08:04:18.7047 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: SmaP28BS59ZRqYhD9UNOan50LtBMSTjddwwlsHm01OD3ifa+3jPWLd5FYe3zmpstNmDl9sF2i/UuEtbNzs5JGA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5309 Subject: Re: [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" > -----Original Message----- > From: Thomas Monjalon > Sent: Tuesday, October 12, 2021 9:45 AM > To: Ali Alnubani > Cc: ci@dpdk.org; jerinj@marvell.com; ferruh.yigit@intel.com; > david.marchand@redhat.com; juraj.linkes@pantheon.tech > Subject: Re: [PATCH v2 10/10] tools: skip the IDs we already fetched >=20 > 11/10/2021 21:30, Ali Alnubani: > > From: Thomas Monjalon > > > 21/09/2021 16:35, alialnu@nvidia.com: > > > > From: Ali Alnubani > > > > > > > > Store the IDs we already fetched in a file and don't > > > > run 'callcmd' again for them. > > > > > > We store all IDs. Should we manually remove olds one from time to tim= e? > > > > > > > Do you have a suggestion for when should we clear this file? Maybe each > time the script starts? >=20 > Yes at each start, we can remove the very old entries, like more than 10 > hours old. Wouldn't this require using and depending on a database (at least an SQLite= one) to additionally store timestamps to use for queries? Do you have a simpler way in mind to keep track of the IDs and their timest= amps to process later based on how long they have been in the file? Maybe k= eep only up to a certain number of IDs in this file (up to a 1000 maybe)? >=20 > > > We need an explanation about the strategy, why it is needed. > > > I think it is because filtering by date is not enough. > > > In order to not miss any patch, we should request a date earlier > > > than the previous fetch and skip those already fetched. > > > > The reason this change was made isn't because filtering by date is not > enough, it's because > > I want to avoid feeding the same ID to 'callcmd' more than once. > > This can happen if a patchwork ID was created between recording > date_now and fetching the API. > > I don't think we are missing any IDs, even without this change. > > > > > Where the "earlier date" is defined? > > > > There are 2 variables, "date_now", which is recorded right before fetch= ing > from the API, and then gets written > > to the file, and "since", which is the last date that was written to th= e file. >=20 > OK please update the commit log. >=20 Updated commit messages in v3.