From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3DEEFA0C4B for ; Mon, 8 Nov 2021 08:44:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F00D140DF7; Mon, 8 Nov 2021 08:44:25 +0100 (CET) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2047.outbound.protection.outlook.com [40.107.220.47]) by mails.dpdk.org (Postfix) with ESMTP id 1D86140040 for ; Mon, 8 Nov 2021 08:44:24 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LVvBJGDDpb+UxS6JwtfaZivxmJkMY/s8Z5kwcWAayszrh+MCKHDStxFl1OUr5TTiMA9LSE0oDmz1Ao/XZltkLtOlEygntkCDDyE0abgqr5I3aBetWIPCeBlrSzoeV5ihF+AcGJqtsJFuQgh0CQtnIxMEHpVqtdJtBs+XBDTo7wJ/bVsOS1Yl1EAtZGMhFWDGxhoypxGYqQeEtrICreB8JPyrPCVkPeMlOrBK8KOksUBW7m8VN3D3w15NsNrojvURltaca1EhPZw7xD2ewr+nQtvNGPOZ7XTWOc0WIAXgZxPbyLlVjKcvhz+sEJw8P3B5n+MXVWxHw7ELCZxapkrr2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=y2hUvnPaehlsMD7eESHWPH3SmHaHRN4w/9v4vTEk0jo=; b=imT1GXjDKzCgXlSbvaiAG2oumYBTlPjfWaKf9JSK02MkR7VP7K5wkbZ3ibA1rBQtb6fVYmHhjYOkTRZ5457q+Jj5fHDEUT5TmgTisXLtjSMnSuZ2WL7ompCTk/PFGtpdYJpP65bOdmDlg2QbhugGodYryL6JuX3uVD6Qh/0Vuu8Jnr9WusfoBX+MD67I3zcDfkzwlRCJAYddLBe3Ufjqhl9eYTHaI8G1VU98wIBR6mWfDs/0YWRYSD4Yq0diNONFaG+HiynqOhsP03R+v0WcurcVBNILb9BMI7Ornp1B7iW+pSn86jrCjgP0yA/78QTDmjTjmLYumLT0kAfcPYYNNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y2hUvnPaehlsMD7eESHWPH3SmHaHRN4w/9v4vTEk0jo=; b=T3JjN8ouO4zoLLoG7dX1afDzWN3dt3aR/hn44bGPsdKCB5zayWUK7v5Ige1OtUwAuUO20nCKKTpxsy0wJ5VQ9wBmvpomzApe3rwxImYpODqHeO62YyGj8aecLuQ51DBOmQZTtxkmRqrzsG79XjdIRQZSqMJAb7Re1vlB1Ao7XyRr41bfGhyX4XmtDWyohx5NlyyM/1eojT4n7Z1pzY5SG+tBDhWguu8Zo4L71UbyGWQiEdxUBwK/nGzYBeuRRrToou/5eTsaWsTPz3RgQKqzvna8j1Ha7Mzf0Xit2Stcutifhfo8aN5pIXm72nIlKPyjGikoPtmxNq59jlrKMCy3gQ== Received: from DM4PR12MB5167.namprd12.prod.outlook.com (2603:10b6:5:396::10) by DM4PR12MB5261.namprd12.prod.outlook.com (2603:10b6:5:398::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.16; Mon, 8 Nov 2021 07:44:22 +0000 Received: from DM4PR12MB5167.namprd12.prod.outlook.com ([fe80::c5e8:f3c:53d9:d6ce]) by DM4PR12MB5167.namprd12.prod.outlook.com ([fe80::c5e8:f3c:53d9:d6ce%9]) with mapi id 15.20.4669.013; Mon, 8 Nov 2021 07:44:22 +0000 From: Ali Alnubani To: NBU-Contact-Thomas Monjalon CC: "ci@dpdk.org" , "jerinj@marvell.com" , "ferruh.yigit@intel.com" , "david.marchand@redhat.com" , "juraj.linkes@pantheon.tech" Thread-Topic: [PATCH v2 10/10] tools: skip the IDs we already fetched Thread-Index: AQHXteaDE41qb2aGHky+hgvhdsrt2KvOPJfwgADA54CACX/sYIAM/EuAgA5SPqCAABZFgIAFmm2Q Date: Mon, 8 Nov 2021 07:44:21 +0000 Message-ID: References: <20210906154537.1299-1-alialnu@nvidia.com> <13296855.Cuyoo0G14J@thomas> <5098807.AlQLvJYzyh@thomas> In-Reply-To: <5098807.AlQLvJYzyh@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8208ae02-176c-49d5-7dad-08d9a28b944f x-ms-traffictypediagnostic: DM4PR12MB5261: x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: +VIkrLlzYjf6dOdCVFuaZeNKuVdAxtiaTQj+WWl8KMTuysNF5iw1wJ5DTxuEhCrALJA05NO6PlIfDA8sFp9X15ibXEhAf1hpGLKEReyb8HnCeMJkDDkHxQxCckwAZGraeOUcYkjBDgvGO/WJD0KmQ4Q7vJx3aPO21eaDSmTGL6yZ3y1dyEEWDhyA1mxtmTkbWEJaLVxSmEzI07IRK3jmpKz4CUNvJPxpVqE7zjL6eJJLOX4T/ZgWzT3XXeod6R6oAIEugnhYI67XchMjXuMrFrEY1mCvtLgRQQyKH4d+DD7VAVqEUL6/+/GlEM2qVcTaZwFkP2zbNW6Hsborsrgj0Xaibfa9meun5bSzqXPChnppShCyxRIwhP9qqNn91D3ohLIXsRJ6AQvmYt6TI/9yjRboVNXNnBvRSue3WPAIGLq1YqbU1Wsbd7BZaiHUCjWIpObvQ2pIfFW50OrLAnM6olv0UrO7Flu3MeE4VSXyVPdjudK6hmfko3yxnp9z1EmlwcLMU7f/jSuQDl95xSDrIPryu68yQ31hv/o78fWYiYUZtDrp/p6qQiUracLVn6c7npjOvkQr1qPiZ74wR3agbcd81Bob8hnk92SFq/dtgIcRvpWdGX6BZDNe/hC99DEPT1Zs4UMWUcMPtgSZu527UacAKnSFFMEvQey1MhVKO0uI+JjqwUvKFaqO4JVhBdhtIcd5l92eUYgENMn0wMnKi/TCJ701phmG/c7Im95e5RXYgwVmeoU4NeQjuuNYVMLkhRFlV0P4yErCFK6+VVRqR6iZlyudQV73vs15JkkjBvs= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR12MB5167.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(83380400001)(966005)(52536014)(55016002)(4326008)(38070700005)(71200400001)(54906003)(7696005)(33656002)(316002)(9686003)(2906002)(186003)(6916009)(8936002)(6506007)(53546011)(8676002)(86362001)(76116006)(122000001)(508600001)(66946007)(26005)(38100700002)(5660300002)(66446008)(66476007)(66556008)(64756008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?ZT3ESlHNmoTCXboYrhSfCkyaD8HNyf92I2x/zLBgDRTIf6JKzsQNtaS0LqFt?= =?us-ascii?Q?/zrobIXssUWIj6lkeUi+qVPy2QSdhDXVgbzebFhupvBKG5+VH4uLBqjRKTB4?= =?us-ascii?Q?Bv0ITNx+u/IA8JFtHUwiw1TTabzTsVM1pCb6uwOYLKP8MFSdw5Yvg//IzoZM?= =?us-ascii?Q?irvRTHfaCXi8c4niywJqfROlfoTk1NIY54a+GkibW34PgQs5dEBobW3dzq4n?= =?us-ascii?Q?R4yD8fuJ/1noFyFWS64LAy8459E9RXRnWgLOJ594/K1LdtwJAq9z3mI88D33?= =?us-ascii?Q?/i3GRKliRYXFxm5dwKkndnQZyFS++uPT+4NvFvVgb5bpnsBmxzXdA/eFusSA?= =?us-ascii?Q?i8bdNuL8SOTTpGzU4TtnnDMHIciexS3skT6eWUg1iqsY2j2fY+T43INNBNcN?= =?us-ascii?Q?We1SOPlhiPfsjTR2g9WMAMX0Y2ZOlBKoFE2KZAd7rsGbZsMoP0CrTQGFIMIC?= =?us-ascii?Q?87xaPzm6fvUvDiYnZXrwaZXLHvoiXDqcuaPrIt8FJDm+9/yXrP439VzdMoh7?= =?us-ascii?Q?t36lzB/c+nPXLFEE1yQHxwEbAf4D10dyZPBgRsCsqeOnT2zfRW+iSQpmUs99?= =?us-ascii?Q?bj69YeRODwmgYZHlOtSdmlrrnAe+wSCsZkKMByF6rJiAO3sqg0MHNslJ5SJ6?= =?us-ascii?Q?i31y8zL95rKhPZFpTiTqHi7CT6HJDRFrt1HU+u/ivQFQEHunRW8DUzFQDKyx?= =?us-ascii?Q?DtPqbgPAA0gBR1iM1X1NkqFwjKkAsbfg7Zi7xT3CFeH3/Sh3Zk7qzOmsd5M6?= =?us-ascii?Q?2HV4nlW5Aq7XciiFItRmdVDEmPrHBchiQHTLjaPY2Zl7wuzuyQWnjfwhkCqT?= =?us-ascii?Q?ylcaf7UEZ0LvkbAmsUZpVpUgxqrsalhTfp44NSUHDaWds2K9ZN41mO9BarUV?= =?us-ascii?Q?LX9mF9PczCFmPUQpPk/sJSSJWl4ooifkf9Ou7jelr3VTqdwK/WeN7gSk5m61?= =?us-ascii?Q?hjh+uOf6zgfADaxYj2q3QqglTLaEuQQ1EtsmsNrQu21x82Y13rJunR1cZ/WY?= =?us-ascii?Q?ScGH3snJzAQNzONpLfwGxDbDMWNXThfrtrwiZ+B/eA4B8zBU6z5qZIGFYzPL?= =?us-ascii?Q?43baRW1oueyvgiCM5Gr3GE19xNkdAgfrkpUqjgahzxfShDjd973JBrElmDXz?= =?us-ascii?Q?k9LTVJOTKZy4IImvkuzsB3IVVqzybIDWze60+TX4c33POaOEpYn47REqIZdn?= =?us-ascii?Q?3/ihrkE/qXhYOWpNW0GRsetqZC6zL8fEvF0+QlC4Azcowv2IM28LWBScjZse?= =?us-ascii?Q?GmOkgpG4I8N/VA1/E6gH1Zxj9yEkC7z4ck+botvGiG9/TrzOMb8O20pwYibQ?= =?us-ascii?Q?Cme3nbDqVj7gca+zl24WnPi+N+xdSninxork6NUvuNPKodWkjvzd6YxODJgk?= =?us-ascii?Q?L47+pVV1FNvs2ccb9lBe/VjilbqOgYLCt4bmByCDgSXZrvfNLWECEIoR5xNT?= =?us-ascii?Q?CGSeQjp0W10/chwmoIccyCvthUvh45l5aeimyy0ba2PP3PfyxZH4u8kS3WMD?= =?us-ascii?Q?CKCVf7sPVM23p84IMhNx5oYyPrmk4OqLJe0FofUeyyxIXFWl1XrXdweIK522?= =?us-ascii?Q?ZqosET8murzBrJJecHigwQmNVOA8liRter0fmNfIT+PpDjwyCp3AFpkKoSPv?= =?us-ascii?Q?IeUK9RdZv0jlujNB1KMzasw=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5167.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8208ae02-176c-49d5-7dad-08d9a28b944f X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Nov 2021 07:44:22.1429 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: lJSLd2HGZ5zdAbjcfMa7u23wR7zioBB0ggDQ9oCFJshIz+Z+IzBt/GJzXDgQ78LPj/h3tyqgcKvyVW8fZU86Mw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5261 Subject: Re: [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" > -----Original Message----- > From: Thomas Monjalon > Sent: Thursday, November 4, 2021 8:09 PM > To: Ali Alnubani > Cc: ci@dpdk.org; jerinj@marvell.com; ferruh.yigit@intel.com; > david.marchand@redhat.com; juraj.linkes@pantheon.tech > Subject: Re: [PATCH v2 10/10] tools: skip the IDs we already fetched >=20 > 04/11/2021 17:53, Ali Alnubani: > > From: Thomas Monjalon > > > 18/10/2021 10:04, Ali Alnubani: > > > > From: Thomas Monjalon > > > > > 11/10/2021 21:30, Ali Alnubani: > > > > > > From: Thomas Monjalon > > > > > > > 21/09/2021 16:35, alialnu@nvidia.com: > > > > > > > > From: Ali Alnubani > > > > > > > > > > > > > > > > Store the IDs we already fetched in a file and don't > > > > > > > > run 'callcmd' again for them. > > > > > > > > > > > > > > We store all IDs. Should we manually remove olds one from tim= e > to > > > time? > > > > > > > > > > > > Do you have a suggestion for when should we clear this file? Ma= ybe > > > each > > > > > time the script starts? > > > > > > > > > > Yes at each start, we can remove the very old entries, like more = than > 10 > > > > > hours old. > > > > > > > > Wouldn't this require using and depending on a database (at least a= n > SQLite > > > one) to additionally store timestamps to use for queries? > > > > Do you have a simpler way in mind to keep track of the IDs and thei= r > > > timestamps to process later based on how long they have been in the > file? > > > Maybe keep only up to a certain number of IDs in this file (up to a 1= 000 > > > maybe)? > > > > > > Yes we can store each entry as a line starting with timestamp: > > > timestamp ID > > > > If the purpose is to not allow the script to grow indefinitely, is it o= k to just > cut the file by half once it reaches a specific number of lines (like 100= 0)?. > > Do you agree with this approach? It'd be easier to implement compared t= o > storing and processing timestamps. >=20 > Yes it is simpler and should work. The file is now cut in half each time it reaches 1000 entries. See v4: https://inbox.dpdk.org/ci/20211108062815.7614-11-alialnu@nvidia.com/T/#u