DPDK CI discussions
 help / color / mirror / Atom feed
From: Ali Alnubani <alialnu@oss.nvidia.com>
To: NBU-Contact-Thomas Monjalon <thomas@monjalon.net>
Cc: "ci@dpdk.org" <ci@dpdk.org>,
	"jerinj@marvell.com" <jerinj@marvell.com>,
	"ferruh.yigit@intel.com" <ferruh.yigit@intel.com>,
	"david.marchand@redhat.com" <david.marchand@redhat.com>,
	"juraj.linkes@pantheon.tech" <juraj.linkes@pantheon.tech>
Subject: Re: [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched
Date: Mon, 11 Oct 2021 19:30:07 +0000	[thread overview]
Message-ID: <DM4PR12MB51678826EFFFF834ECFFC85BDAB59@DM4PR12MB5167.namprd12.prod.outlook.com> (raw)
In-Reply-To: <2586128.JSg8es4y6H@thomas>

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, September 30, 2021 1:33 PM
> To: Ali Alnubani <alialnu@nvidia.com>
> Cc: ci@dpdk.org; jerinj@marvell.com; ferruh.yigit@intel.com;
> david.marchand@redhat.com; juraj.linkes@pantheon.tech
> Subject: Re: [PATCH v2 10/10] tools: skip the IDs we already fetched
> 
> 21/09/2021 16:35, alialnu@nvidia.com:
> > From: Ali Alnubani <alialnu@nvidia.com>
> >
> > Store the IDs we already fetched in a file and don't
> > run 'callcmd' again for them.
> 
> We store all IDs. Should we manually remove olds one from time to time?
> 

Do you have a suggestion for when should we clear this file? Maybe each time the script starts?

> We need an explanation about the strategy, why it is needed.
> I think it is because filtering by date is not enough.
> In order to not miss any patch, we should request a date earlier
> than the previous fetch and skip those already fetched.
> 

The reason this change was made isn't because filtering by date is not enough, it's because
I want to avoid feeding the same ID to 'callcmd' more than once.
This can happen if a patchwork ID was created between recording date_now and fetching the API.
I don't think we are missing any IDs, even without this change.

> Where the "earlier date" is defined?

There are 2 variables, "date_now", which is recorded right before fetching from the API, and then gets written
to the file, and "since", which is the last date that was written to the file.

> 
> > Signed-off-by: Ali Alnubani <alialnu@nvidia.com>
> > ---
> >  tools/poll-pw | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/tools/poll-pw b/tools/poll-pw
> > index e197a91..dacf34b 100755
> > --- a/tools/poll-pw
> > +++ b/tools/poll-pw
> > @@ -65,6 +65,11 @@ if [ ! $? -eq 0 ] ; then
> >  	exit 1
> >  fi
> >
> > +poll_pw_ids_file=/tmp/poll_pw_${resource_type}_ids
> > +if [ ! -f "$poll_pw_ids_file" ] ; then
> > +	touch $poll_pw_ids_file
> > +fi
> > +
> >  URL="${URL}/${resource_type}/?project=${project}&"
> >
> >  callcmd () # <patchwork id>
> > @@ -77,7 +82,11 @@ while true ; do
> >  	date_now=$(date '+%FT%T')
> >  	since=$(date -d $(cat $since_file | tr '\n' ' ') '+%FT%T')
> >  	for id in $(curl -s "${URL}since=${since}" | jq '.[].id') ; do
> > +		if grep -q "^${id}$" $poll_pw_ids_file ; then
> > +			continue
> > +		fi
> >  		callcmd $id
> > +		echo $id >>$poll_pw_ids_file
> >  	done
> >  	echo -n $date_now >$since_file
> >  	# pause before next check
> >
> 
> 
> 
> 


  reply	other threads:[~2021-10-11 19:30 UTC|newest]

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-06 15:45 [dpdk-ci] [PATCH 0/9] Automatic patchwork delegation Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 1/9] tools: rename guess_git_tree script Ali Alnubani
2021-09-07 11:03   ` Juraj Linkeš
2021-09-08 16:54     ` Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 2/9] tools: match by tree url instead of tree name Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 3/9] tools: update script usage Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 4/9] tools: add functionality for detecting tree maintainers Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 5/9] tools: add functionality for setting pw delegates Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 6/9] add git-pw to requirements file Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 7/9] tools: filter new patchwork IDs by date Ali Alnubani
2021-09-06 15:58   ` Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 8/9] tools: add support for fetching new series IDs Ali Alnubani
2021-09-06 15:45 ` [dpdk-ci] [PATCH 9/9] tools: filter new patchwork IDs by project name Ali Alnubani
2021-09-21 14:35 ` [dpdk-ci] [PATCH v2 00/10] Automatic patchwork delegation alialnu
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 01/10] tools: rename guess_git_tree script alialnu
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 02/10] tools: match by tree url instead of tree name alialnu
2021-09-30  8:00     ` Thomas Monjalon
2021-10-18  7:48       ` Ali Alnubani
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 03/10] tools: update script usage alialnu
2021-09-30  8:09     ` Thomas Monjalon
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 04/10] tools: add functionality for detecting tree maintainers alialnu
2021-09-30  8:29     ` Thomas Monjalon
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 05/10] tools: add functionality for setting pw delegates alialnu
2021-09-30  9:15     ` Thomas Monjalon
2021-10-18  7:48       ` Ali Alnubani
2021-10-26 14:08         ` Thomas Monjalon
2021-11-04 16:48           ` Ali Alnubani
2021-11-04 18:16             ` Thomas Monjalon
2021-11-08  7:45               ` Ali Alnubani
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 06/10] add git-pw to requirements file alialnu
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 07/10] tools: filter new patchwork IDs by date alialnu
2021-10-11 20:08     ` Ali Alnubani
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 08/10] tools: add support for fetching new series IDs alialnu
2021-09-30 10:25     ` Thomas Monjalon
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 09/10] tools: filter new patchwork IDs by project name alialnu
2021-09-30 10:28     ` Thomas Monjalon
2021-09-21 14:35   ` [dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched alialnu
2021-09-30 10:32     ` Thomas Monjalon
2021-10-11 19:30       ` Ali Alnubani [this message]
2021-10-12  6:44         ` Thomas Monjalon
2021-10-18  8:04           ` Ali Alnubani
2021-10-26 14:07             ` Thomas Monjalon
2021-11-04 16:53               ` Ali Alnubani
2021-11-04 18:08                 ` Thomas Monjalon
2021-11-08  7:44                   ` Ali Alnubani
2021-11-08  6:28 ` [dpdk-ci] [PATCH v4 00/10] Automatic patchwork delegation Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 01/10] tools: rename guess_git_tree script Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 02/10] tools: match by tree URL instead of tree name Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 03/10] tools: update script usage Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 04/10] tools: add functionality for detecting tree maintainers Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 05/10] tools: add functionality for setting pw delegates Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 06/10] add git-pw to requirements file Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 07/10] tools: filter new Patchwork IDs by date Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 08/10] tools: support fetching series Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 09/10] tools: filter new patchwork IDs by project name Ali Alnubani
2021-11-08  6:28   ` [dpdk-ci] [PATCH v4 10/10] tools: skip the IDs we already fetched Ali Alnubani
2022-01-05  1:05   ` [PATCH v4 00/10] Automatic patchwork delegation Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DM4PR12MB51678826EFFFF834ECFFC85BDAB59@DM4PR12MB5167.namprd12.prod.outlook.com \
    --to=alialnu@oss.nvidia.com \
    --cc=ci@dpdk.org \
    --cc=david.marchand@redhat.com \
    --cc=ferruh.yigit@intel.com \
    --cc=jerinj@marvell.com \
    --cc=juraj.linkes@pantheon.tech \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).