Re: [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched

[Ali Alnubani <alialnu@oss.nvidia.com>]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, September 30, 2021 1:33 PM
> To: Ali Alnubani <alialnu@nvidia.com>
> Cc: ci@dpdk.org; jerinj@marvell.com; ferruh.yigit@intel.com;
> david.marchand@redhat.com; juraj.linkes@pantheon.tech
> Subject: Re: [PATCH v2 10/10] tools: skip the IDs we already fetched
> 
> 21/09/2021 16:35, alialnu@nvidia.com:
> > From: Ali Alnubani <alialnu@nvidia.com>
> >
> > Store the IDs we already fetched in a file and don't
> > run 'callcmd' again for them.
> 
> We store all IDs. Should we manually remove olds one from time to time?
> 

Do you have a suggestion for when should we clear this file? Maybe each time the script starts?

> We need an explanation about the strategy, why it is needed.
> I think it is because filtering by date is not enough.
> In order to not miss any patch, we should request a date earlier
> than the previous fetch and skip those already fetched.
> 

The reason this change was made isn't because filtering by date is not enough, it's because
I want to avoid feeding the same ID to 'callcmd' more than once.
This can happen if a patchwork ID was created between recording date_now and fetching the API.
I don't think we are missing any IDs, even without this change.

> Where the "earlier date" is defined?

There are 2 variables, "date_now", which is recorded right before fetching from the API, and then gets written
to the file, and "since", which is the last date that was written to the file.

> 
> > Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
> > ---
> >  tools/poll-pw | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/tools/poll-pw b/tools/poll-pw
> > index e197a91..dacf34b 100755
> > --- a/tools/poll-pw
> > +++ b/tools/poll-pw
> > @@ -65,6 +65,11 @@ if [ ! $? -eq 0 ] ; then
> >  	exit 1
> >  fi
> >
> > +poll_pw_ids_file=/tmp/poll_pw_${resource_type}_ids
> > +if [ ! -f "$poll_pw_ids_file" ] ; then
> > +	touch $poll_pw_ids_file
> > +fi
> > +
> >  URL="${URL}/${resource_type}/?project=${project}&"
> >
> >  callcmd () # <patchwork id>
> > @@ -77,7 +82,11 @@ while true ; do
> >  	date_now=$(date '+%FT%T')
> >  	since=$(date -d $(cat $since_file | tr '\n' ' ') '+%FT%T')
> >  	for id in $(curl -s "${URL}since=${since}" | jq '.[].id') ; do
> > +		if grep -q "^${id}$" $poll_pw_ids_file ; then
> > +			continue
> > +		fi
> >  		callcmd $id
> > +		echo $id >>$poll_pw_ids_file
> >  	done
> >  	echo -n $date_now >$since_file
> >  	# pause before next check
> >
> 
> 
> 
>