Re: [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched

[Thomas Monjalon <thomas@monjalon.net>]

21/09/2021 16:35, alialnu@nvidia.com:
> From: Ali Alnubani <alialnu@nvidia.com>
> 
> Store the IDs we already fetched in a file and don't
> run 'callcmd' again for them.

We store all IDs. Should we manually remove olds one from time to time?

We need an explanation about the strategy, why it is needed.
I think it is because filtering by date is not enough.
In order to not miss any patch, we should request a date earlier
than the previous fetch and skip those already fetched.

Where the "earlier date" is defined?

> Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
> ---
>  tools/poll-pw | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/tools/poll-pw b/tools/poll-pw
> index e197a91..dacf34b 100755
> --- a/tools/poll-pw
> +++ b/tools/poll-pw
> @@ -65,6 +65,11 @@ if [ ! $? -eq 0 ] ; then
>  	exit 1
>  fi
>  
> +poll_pw_ids_file=/tmp/poll_pw_${resource_type}_ids
> +if [ ! -f "$poll_pw_ids_file" ] ; then
> +	touch $poll_pw_ids_file
> +fi
> +
>  URL="${URL}/${resource_type}/?project=${project}&"
>  
>  callcmd () # <patchwork id>
> @@ -77,7 +82,11 @@ while true ; do
>  	date_now=$(date '+%FT%T')
>  	since=$(date -d $(cat $since_file | tr '\n' ' ') '+%FT%T')
>  	for id in $(curl -s "${URL}since=${since}" | jq '.[].id') ; do
> +		if grep -q "^${id}$" $poll_pw_ids_file ; then
> +			continue
> +		fi
>  		callcmd $id
> +		echo $id >>$poll_pw_ids_file
>  	done
>  	echo -n $date_now >$since_file
>  	# pause before next check
>