Re: [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched

[Ali Alnubani <alialnu@nvidia.com>]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, November 4, 2021 8:09 PM
> To: Ali Alnubani <alialnu@nvidia.com>
> Cc: ci@dpdk.org; jerinj@marvell.com; ferruh.yigit@intel.com;
> david.marchand@redhat.com; juraj.linkes@pantheon.tech
> Subject: Re: [PATCH v2 10/10] tools: skip the IDs we already fetched
> 
> 04/11/2021 17:53, Ali Alnubani:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 18/10/2021 10:04, Ali Alnubani:
> > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > 11/10/2021 21:30, Ali Alnubani:
> > > > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > > > 21/09/2021 16:35, alialnu@nvidia.com:
> > > > > > > > From: Ali Alnubani <alialnu@nvidia.com>
> > > > > > > >
> > > > > > > > Store the IDs we already fetched in a file and don't
> > > > > > > > run 'callcmd' again for them.
> > > > > > >
> > > > > > > We store all IDs. Should we manually remove olds one from time
> to
> > > time?
> > > > > >
> > > > > > Do you have a suggestion for when should we clear this file? Maybe
> > > each
> > > > > time the script starts?
> > > > >
> > > > > Yes at each start, we can remove the very old entries, like more than
> 10
> > > > > hours old.
> > > >
> > > > Wouldn't this require using and depending on a database (at least an
> SQLite
> > > one) to additionally store timestamps to use for queries?
> > > > Do you have a simpler way in mind to keep track of the IDs and their
> > > timestamps to process later based on how long they have been in the
> file?
> > > Maybe keep only up to a certain number of IDs in this file (up to a 1000
> > > maybe)?
> > >
> > > Yes we can store each entry as a line starting with timestamp:
> > > 	timestamp ID
> >
> > If the purpose is to not allow the script to grow indefinitely, is it ok to just
> cut the file by half once it reaches a specific number of lines (like 1000)?.
> > Do you agree with this approach? It'd be easier to implement compared to
> storing and processing timestamps.
> 
> Yes it is simpler and should work.

The file is now cut in half each time it reaches 1000 entries. See v4:
https://inbox.dpdk.org/ci/20211108062815.7614-11-alialnu@nvidia.com/T/#u