From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A2706A0C52 for ; Thu, 4 Nov 2021 19:08:51 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 837BD41C27; Thu, 4 Nov 2021 19:08:51 +0100 (CET) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by mails.dpdk.org (Postfix) with ESMTP id 378F941223 for ; Thu, 4 Nov 2021 19:08:50 +0100 (CET) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id DB2B85C00DC; Thu, 4 Nov 2021 14:08:49 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Thu, 04 Nov 2021 14:08:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=fm2; bh= n0do9OAHNhKWyMIqMfQMPDkh9S19QJ24QE0Yz6N9p5M=; b=HaAC3RMnqY3EiF8R Fc/2412oR3YXXG4ODGFQdRsnhq5gcLQepLdSdDuW2Jqv3n5r/krDB6rgsb9+sLNu NGROgmPhRStQBOOeJzYplZ1+lBmhj/CQBwRWg9hHM8IF+RDyiPle+FbpE4Guk1tx oSuYIHgcfSdZzEysun6Qu+feyB4iQXY7IVphOwHIkul9M0z8LI+h46G08/+zc99x 8QIwNyytuHiU0WFEYW5B6vKVO13VysW2UYvG/N/hOAehD6lE8JJtN93NPSOSLPqE FpG6W+rMcTPG5E5V11DefH6RedZEvq2c9LzKw6WW8ESJM1tLHcZhC/LFCm47U3Uk bckkxw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=n0do9OAHNhKWyMIqMfQMPDkh9S19QJ24QE0Yz6N9p 5M=; b=JJ/UhScmVNyOPHuzHU1GIEQS+aJaMzRU4zvXv0bWhr8xqsY2sdr/NxyMp YOlY6dT5ppYILzmwrzyUtTDecEBxLgpVPxU9mdnfVXsGiz8vvrPARnXr1Fzhf1C3 +x3pLqiTOJILZ/ud3dZPJV/pMFPvy5ztm3bN8KjEhIpkllQMA3JIdVEneYAYN08y MvJTbg+ECwUJea5vcKDR8g7pvkyn16EXost/ZHdyM71xkos2gf7/y1y2/nLAn6Br k1lDuyGB4Zwb6Z06MvumNP0SU8G7UYHz4p+0rwaV3INjXWMXrqbVuC7Nx9oNWUHO 44La4BO7NKnHkURzXCnbDQ61UeCgg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrtdeggddutdegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfgjfhgggfgtsehtufertddttddvnecuhfhrohhmpefvhhhomhgr shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecugg ftrfgrthhtvghrnhepudeggfdvfeduffdtfeeglefghfeukefgfffhueejtdetuedtjeeu ieeivdffgeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrh homhepthhhohhmrghssehmohhnjhgrlhhonhdrnhgvth X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 4 Nov 2021 14:08:48 -0400 (EDT) From: Thomas Monjalon To: Ali Alnubani Cc: "ci@dpdk.org" , "jerinj@marvell.com" , "ferruh.yigit@intel.com" , "david.marchand@redhat.com" , "juraj.linkes@pantheon.tech" Date: Thu, 04 Nov 2021 19:08:47 +0100 Message-ID: <5098807.AlQLvJYzyh@thomas> In-Reply-To: References: <20210906154537.1299-1-alialnu@nvidia.com> <13296855.Cuyoo0G14J@thomas> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" 04/11/2021 17:53, Ali Alnubani: > From: Thomas Monjalon > > 18/10/2021 10:04, Ali Alnubani: > > > From: Thomas Monjalon > > > > 11/10/2021 21:30, Ali Alnubani: > > > > > From: Thomas Monjalon > > > > > > 21/09/2021 16:35, alialnu@nvidia.com: > > > > > > > From: Ali Alnubani > > > > > > > > > > > > > > Store the IDs we already fetched in a file and don't > > > > > > > run 'callcmd' again for them. > > > > > > > > > > > > We store all IDs. Should we manually remove olds one from time to > > time? > > > > > > > > > > Do you have a suggestion for when should we clear this file? Maybe > > each > > > > time the script starts? > > > > > > > > Yes at each start, we can remove the very old entries, like more than 10 > > > > hours old. > > > > > > Wouldn't this require using and depending on a database (at least an SQLite > > one) to additionally store timestamps to use for queries? > > > Do you have a simpler way in mind to keep track of the IDs and their > > timestamps to process later based on how long they have been in the file? > > Maybe keep only up to a certain number of IDs in this file (up to a 1000 > > maybe)? > > > > Yes we can store each entry as a line starting with timestamp: > > timestamp ID > > If the purpose is to not allow the script to grow indefinitely, is it ok to just cut the file by half once it reaches a specific number of lines (like 1000)?. > Do you agree with this approach? It'd be easier to implement compared to storing and processing timestamps. Yes it is simpler and should work.