[dpdk-ci] [RFC] Proposal for allowing rerun of tests

[dpdk-ci] [RFC] Proposal for allowing rerun of tests

[Aaron Conole]

Greetings,

During the various CI pipelines, sometimes a test setup or lab will
have an internal failure unrelated to the specific patch.  Perhaps
'master' branch (or the associated -next branch) is broken and we cannot
get a successful run anyway.  Perhaps a network outage occurs during
infrastructure setup.  Perhaps some other transient error clobbers the
setup.  In all of these cases the report to the mailing flags the patch
as 'FAIL'.

It would be very helpful if maintainers had the ability to tell various
CI infrastructures to restart / rerun patch tests.  For now, this has to
be done by the individual managers of those labs.  Some labs, it isn't
possible.  Others, it's possible but is a very time-consuming process to
restart a test case.  In all cases, a maintainer needs to spend time
communicating with a lab manager.  This could be made a bit nicer.

One proposal we (Michael and I) have toyed with for our lab is having
the infrastructure monitor patchwork comments for a restart flag, and
kick off based on that information.  Patchwork tracks all of the
comments for each patch / series so we could look at the series that
are still in a state for 'merging' (new, assigned, etc) and check the
patch .comments API for new comments.  Getting the data from PW should
be pretty simple - but I think that knowing whether to kick off the
test might be more difficult.  We have concerns about which messages we
should accept (for example, can anyone ask for a series to be rerun, and
we'll need to track which rerun messages we've accepted).  The
convention needs to be something we all can work with (ie: /Re-check:
[checkname] or something as a single line in the email).

This is just a start to identify and explain the concern.  Maybe there
are other issues we've not considered, or maybe folks think this is a
terrible idea not worth spending any time developing.  I think there's
enough use for it that I am raising it here, and we can discuss it.

Thanks,
-Aaron

Re: [dpdk-ci] [RFC] Proposal for allowing rerun of tests

[Thomas Monjalon]

13/04/2021 15:50, Aaron Conole:
> During the various CI pipelines, sometimes a test setup or lab will
> have an internal failure unrelated to the specific patch.  Perhaps
> 'master' branch (or the associated -next branch) is broken and we cannot
> get a successful run anyway.  Perhaps a network outage occurs during
> infrastructure setup.  Perhaps some other transient error clobbers the
> setup.  In all of these cases the report to the mailing flags the patch
> as 'FAIL'.
> 
> It would be very helpful if maintainers had the ability to tell various
> CI infrastructures to restart / rerun patch tests.  For now, this has to
> be done by the individual managers of those labs.  Some labs, it isn't
> possible.  Others, it's possible but is a very time-consuming process to
> restart a test case.  In all cases, a maintainer needs to spend time
> communicating with a lab manager.  This could be made a bit nicer.

I agree

> One proposal we (Michael and I) have toyed with for our lab is having
> the infrastructure monitor patchwork comments for a restart flag, and
> kick off based on that information.  Patchwork tracks all of the
> comments for each patch / series so we could look at the series that
> are still in a state for 'merging' (new, assigned, etc) and check the
> patch .comments API for new comments.  Getting the data from PW should
> be pretty simple - but I think that knowing whether to kick off the
> test might be more difficult.  We have concerns about which messages we
> should accept (for example, can anyone ask for a series to be rerun, and
> we'll need to track which rerun messages we've accepted).  The
> convention needs to be something we all can work with (ie: /Re-check:
> [checkname] or something as a single line in the email).
> 
> This is just a start to identify and explain the concern.  Maybe there
> are other issues we've not considered, or maybe folks think this is a
> terrible idea not worth spending any time developing.  I think there's
> enough use for it that I am raising it here, and we can discuss it.

First question: WHO should be allowed to ask for a re-run?
	- everybody
	- patchwork delegate
	- a list of maintainers

Second question: HOW requesting a re-run?
	- comment in email with formatted message
	- patchwork button
	- postal letter

Third question: WHERE hosting this mechanism?
	- only one answer: in dpdk-ci.git consumed by labs

Re: [dpdk-ci] [RFC] Proposal for allowing rerun of tests

[David Marchand]

On Tue, Apr 13, 2021 at 4:47 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 13/04/2021 15:50, Aaron Conole:
> > During the various CI pipelines, sometimes a test setup or lab will
> > have an internal failure unrelated to the specific patch.  Perhaps
> > 'master' branch (or the associated -next branch) is broken and we cannot
> > get a successful run anyway.  Perhaps a network outage occurs during
> > infrastructure setup.  Perhaps some other transient error clobbers the
> > setup.  In all of these cases the report to the mailing flags the patch
> > as 'FAIL'.
> >
> > It would be very helpful if maintainers had the ability to tell various
> > CI infrastructures to restart / rerun patch tests.  For now, this has to
> > be done by the individual managers of those labs.  Some labs, it isn't
> > possible.  Others, it's possible but is a very time-consuming process to
> > restart a test case.  In all cases, a maintainer needs to spend time
> > communicating with a lab manager.  This could be made a bit nicer.
>

Yes, this is something that is often discussed with other maintainers.


>
> > One proposal we (Michael and I) have toyed with for our lab is having
> > the infrastructure monitor patchwork comments for a restart flag, and
> > kick off based on that information.  Patchwork tracks all of the
> > comments for each patch / series so we could look at the series that
> > are still in a state for 'merging' (new, assigned, etc) and check the
> > patch .comments API for new comments.  Getting the data from PW should
> > be pretty simple - but I think that knowing whether to kick off the
> > test might be more difficult.  We have concerns about which messages we
> > should accept (for example, can anyone ask for a series to be rerun, and
> > we'll need to track which rerun messages we've accepted).  The
> > convention needs to be something we all can work with (ie: /Re-check:
> > [checkname] or something as a single line in the email).
> >
> > This is just a start to identify and explain the concern.  Maybe there
> > are other issues we've not considered, or maybe folks think this is a
> > terrible idea not worth spending any time developing.  I think there's
> > enough use for it that I am raising it here, and we can discuss it.
>
> First question: WHO should be allowed to ask for a re-run?
>         - everybody
>         - patchwork delegate

Patchwork delegate requires to maintain a map between pw logins and an
actual mail address (if we go with email for the second point).

>         - a list of maintainers

I'd vote on any maintainer from MAINTAINERS, _but_ it must be from the
files in the repo, not in the series being tested.
So maybe the easier is to have an explicit list... ?


- author
Just listing this option for discussion, but this is dangerous, as any
user could then call reruns.


>
> Second question: HOW requesting a re-run?
>         - comment in email with formatted message
>         - patchwork button
>         - postal letter

While the postal letter has its charm, an email on the ml is better
than pw for me.
It leaves a trace on who asked and when.

And I am not sure how you could trigger a CI rerun with patchwork anyway :-).


>
> Third question: WHERE hosting this mechanism?
>         - only one answer: in dpdk-ci.git consumed by labs



-- 
David Marchand

Re: [dpdk-ci] [dpdk-dev] [RFC] Proposal for allowing rerun of tests

[Bruce Richardson]

On Tue, Apr 13, 2021 at 04:59:00PM +0200, David Marchand wrote:
> On Tue, Apr 13, 2021 at 4:47 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 13/04/2021 15:50, Aaron Conole:
> >
> > > One proposal we (Michael and I) have toyed with for our lab is having
> > > the infrastructure monitor patchwork comments for a restart flag, and
> > > kick off based on that information.  Patchwork tracks all of the
> > > comments for each patch / series so we could look at the series that
> > > are still in a state for 'merging' (new, assigned, etc) and check the
> > > patch .comments API for new comments.  Getting the data from PW should
> > > be pretty simple - but I think that knowing whether to kick off the
> > > test might be more difficult.  We have concerns about which messages we
> > > should accept (for example, can anyone ask for a series to be rerun, and
> > > we'll need to track which rerun messages we've accepted).  The
> > > convention needs to be something we all can work with (ie: /Re-check:
> > > [checkname] or something as a single line in the email).
> > >
> > > This is just a start to identify and explain the concern.  Maybe there
> > > are other issues we've not considered, or maybe folks think this is a
> > > terrible idea not worth spending any time developing.  I think there's
> > > enough use for it that I am raising it here, and we can discuss it.
> >
> > First question: WHO should be allowed to ask for a re-run?
> >         - everybody
> >         - patchwork delegate
> 
> Patchwork delegate requires to maintain a map between pw logins and an
> actual mail address (if we go with email for the second point).
> 
> >         - a list of maintainers
> 
> I'd vote on any maintainer from MAINTAINERS, _but_ it must be from the
> files in the repo, not in the series being tested.
> So maybe the easier is to have an explicit list... ?
> 
> 
> - author
> Just listing this option for discussion, but this is dangerous, as any
> user could then call reruns.
> 

I would tend towards including this, on the basis that any author can
already get a re-run just be resubmitting a new version of their patchset.
This just simplifies that for all concerned.

/Bruce

Re: [dpdk-ci] [dpdk-dev] [RFC] Proposal for allowing rerun of tests

[Thomas Monjalon]

13/04/2021 17:04, Bruce Richardson:
> On Tue, Apr 13, 2021 at 04:59:00PM +0200, David Marchand wrote:
> > On Tue, Apr 13, 2021 at 4:47 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > 13/04/2021 15:50, Aaron Conole:
> > >
> > > > One proposal we (Michael and I) have toyed with for our lab is having
> > > > the infrastructure monitor patchwork comments for a restart flag, and
> > > > kick off based on that information.  Patchwork tracks all of the
> > > > comments for each patch / series so we could look at the series that
> > > > are still in a state for 'merging' (new, assigned, etc) and check the
> > > > patch .comments API for new comments.  Getting the data from PW should
> > > > be pretty simple - but I think that knowing whether to kick off the
> > > > test might be more difficult.  We have concerns about which messages we
> > > > should accept (for example, can anyone ask for a series to be rerun, and
> > > > we'll need to track which rerun messages we've accepted).  The
> > > > convention needs to be something we all can work with (ie: /Re-check:
> > > > [checkname] or something as a single line in the email).
> > > >
> > > > This is just a start to identify and explain the concern.  Maybe there
> > > > are other issues we've not considered, or maybe folks think this is a
> > > > terrible idea not worth spending any time developing.  I think there's
> > > > enough use for it that I am raising it here, and we can discuss it.
> > >
> > > First question: WHO should be allowed to ask for a re-run?
> > >         - everybody
> > >         - patchwork delegate
> > 
> > Patchwork delegate requires to maintain a map between pw logins and an
> > actual mail address (if we go with email for the second point).
> > 
> > >         - a list of maintainers
> > 
> > I'd vote on any maintainer from MAINTAINERS, _but_ it must be from the
> > files in the repo, not in the series being tested.
> > So maybe the easier is to have an explicit list... ?
> > 
> > 
> > - author
> > Just listing this option for discussion, but this is dangerous, as any
> > user could then call reruns.
> > 
> 
> I would tend towards including this, on the basis that any author can
> already get a re-run just be resubmitting a new version of their patchset.
> This just simplifies that for all concerned.

I agree, and it would be very convenient for authors hitting
a strange failure: they can double check without bothering maintainers.

Re: [dpdk-ci] [dpdk-dev] [RFC] Proposal for allowing rerun of tests

[Aaron Conole]

Thomas Monjalon <thomas@monjalon.net> writes:

> 13/04/2021 17:04, Bruce Richardson:
>> On Tue, Apr 13, 2021 at 04:59:00PM +0200, David Marchand wrote:
>> > On Tue, Apr 13, 2021 at 4:47 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>> > >
>> > > 13/04/2021 15:50, Aaron Conole:
>> > >
>> > > > One proposal we (Michael and I) have toyed with for our lab is having
>> > > > the infrastructure monitor patchwork comments for a restart flag, and
>> > > > kick off based on that information.  Patchwork tracks all of the
>> > > > comments for each patch / series so we could look at the series that
>> > > > are still in a state for 'merging' (new, assigned, etc) and check the
>> > > > patch .comments API for new comments.  Getting the data from PW should
>> > > > be pretty simple - but I think that knowing whether to kick off the
>> > > > test might be more difficult.  We have concerns about which messages we
>> > > > should accept (for example, can anyone ask for a series to be rerun, and
>> > > > we'll need to track which rerun messages we've accepted).  The
>> > > > convention needs to be something we all can work with (ie: /Re-check:
>> > > > [checkname] or something as a single line in the email).
>> > > >
>> > > > This is just a start to identify and explain the concern.  Maybe there
>> > > > are other issues we've not considered, or maybe folks think this is a
>> > > > terrible idea not worth spending any time developing.  I think there's
>> > > > enough use for it that I am raising it here, and we can discuss it.
>> > >
>> > > First question: WHO should be allowed to ask for a re-run?
>> > >         - everybody
>> > >         - patchwork delegate
>> > 
>> > Patchwork delegate requires to maintain a map between pw logins and an
>> > actual mail address (if we go with email for the second point).
>> > 
>> > >         - a list of maintainers
>> > 
>> > I'd vote on any maintainer from MAINTAINERS, _but_ it must be from the
>> > files in the repo, not in the series being tested.
>> > So maybe the easier is to have an explicit list... ?

I agree with using the MAINTAINERS file from the repo.

>> > 
>> > - author
>> > Just listing this option for discussion, but this is dangerous, as any
>> > user could then call reruns.
>> > 
>> 
>> I would tend towards including this, on the basis that any author can
>> already get a re-run just be resubmitting a new version of their patchset.
>> This just simplifies that for all concerned.
>
> I agree, and it would be very convenient for authors hitting
> a strange failure: they can double check without bothering maintainers.

+1

Re: [dpdk-ci] [dpdk-dev] [RFC] Proposal for allowing rerun of tests

[David Marchand]

On Wed, Apr 21, 2021 at 5:03 PM Aaron Conole <aconole@redhat.com> wrote:
> >> > > First question: WHO should be allowed to ask for a re-run?
> >> > >         - everybody
> >> > >         - patchwork delegate
> >> >
> >> > Patchwork delegate requires to maintain a map between pw logins and an
> >> > actual mail address (if we go with email for the second point).
> >> >
> >> > >         - a list of maintainers
> >> >
> >> > I'd vote on any maintainer from MAINTAINERS, _but_ it must be from the
> >> > files in the repo, not in the series being tested.
> >> > So maybe the easier is to have an explicit list... ?
>
> I agree with using the MAINTAINERS file from the repo.
>
> >> >
> >> > - author
> >> > Just listing this option for discussion, but this is dangerous, as any
> >> > user could then call reruns.
> >> >
> >>
> >> I would tend towards including this, on the basis that any author can
> >> already get a re-run just be resubmitting a new version of their patchset.
> >> This just simplifies that for all concerned.
> >
> > I agree, and it would be very convenient for authors hitting
> > a strange failure: they can double check without bothering maintainers.
>
> +1

Ok for me.


-- 
David Marchand

Re: [dpdk-ci] [RFC] Proposal for allowing rerun of tests

[Kevin Traynor]

On 13/04/2021 14:50, Aaron Conole wrote:
> Greetings,
> 
> During the various CI pipelines, sometimes a test setup or lab will
> have an internal failure unrelated to the specific patch.  Perhaps
> 'master' branch (or the associated -next branch) is broken and we cannot
> get a successful run anyway.  Perhaps a network outage occurs during
> infrastructure setup.  Perhaps some other transient error clobbers the
> setup.  In all of these cases the report to the mailing flags the patch
> as 'FAIL'.
> 
> It would be very helpful if maintainers had the ability to tell various
> CI infrastructures to restart / rerun patch tests.  For now, this has to
> be done by the individual managers of those labs.  Some labs, it isn't
> possible.  Others, it's possible but is a very time-consuming process to
> restart a test case.  In all cases, a maintainer needs to spend time
> communicating with a lab manager.  This could be made a bit nicer.
> 

Just to tie two relevant threads together. I made a request in 
http://mails.dpdk.org/archives/ci/2022-January/001593.html for a 
"retest" button (or really any manual but self-sufficient way) to 
kick-off immediately what is run in periodic branch testing. Something 
might be there already, that i'm just not aware of.

This could be used by LTS maintainers, and possibly main, *-next branch 
maintainers coming up to releases.

thanks,
Kevin.

> One proposal we (Michael and I) have toyed with for our lab is having
> the infrastructure monitor patchwork comments for a restart flag, and
> kick off based on that information.  Patchwork tracks all of the
> comments for each patch / series so we could look at the series that
> are still in a state for 'merging' (new, assigned, etc) and check the
> patch .comments API for new comments.  Getting the data from PW should
> be pretty simple - but I think that knowing whether to kick off the
> test might be more difficult.  We have concerns about which messages we
> should accept (for example, can anyone ask for a series to be rerun, and
> we'll need to track which rerun messages we've accepted).  The
> convention needs to be something we all can work with (ie: /Re-check:
> [checkname] or something as a single line in the email).
> 
> This is just a start to identify and explain the concern.  Maybe there
> are other issues we've not considered, or maybe folks think this is a
> terrible idea not worth spending any time developing.  I think there's
> enough use for it that I am raising it here, and we can discuss it.
> 
> Thanks,
> -Aaron
> 

Re: [dpdklab] Re: [dpdk-ci] [RFC] Proposal for allowing rerun of tests

[Lincoln Lavoie]

Hi Kevin,

I'll add some notes around this to the 2022 planning doc.  The retests
requests for the periodic tests are tricky from the discussed patchworks /
mailing list approach, because there isn't an associated patch or patch
ID.  So, we just need to document or agree on how to "flag" what the
request is, so the system could uniquely ID what branch / test to rerun.

I'll also add this to the list for the dashboard technical debt as well, as
there may be a graphical interface stuff (i.e. a button) we could add to
the UNH-IOL hosted testing.

Cheers,
Lincoln

On Fri, Jan 21, 2022 at 9:00 AM Kevin Traynor <ktraynor@redhat.com> wrote:

> On 13/04/2021 14:50, Aaron Conole wrote:
> > Greetings,
> >
> > During the various CI pipelines, sometimes a test setup or lab will
> > have an internal failure unrelated to the specific patch.  Perhaps
> > 'master' branch (or the associated -next branch) is broken and we cannot
> > get a successful run anyway.  Perhaps a network outage occurs during
> > infrastructure setup.  Perhaps some other transient error clobbers the
> > setup.  In all of these cases the report to the mailing flags the patch
> > as 'FAIL'.
> >
> > It would be very helpful if maintainers had the ability to tell various
> > CI infrastructures to restart / rerun patch tests.  For now, this has to
> > be done by the individual managers of those labs.  Some labs, it isn't
> > possible.  Others, it's possible but is a very time-consuming process to
> > restart a test case.  In all cases, a maintainer needs to spend time
> > communicating with a lab manager.  This could be made a bit nicer.
> >
>
> Just to tie two relevant threads together. I made a request in
> http://mails.dpdk.org/archives/ci/2022-January/001593.html for a
> "retest" button (or really any manual but self-sufficient way) to
> kick-off immediately what is run in periodic branch testing. Something
> might be there already, that i'm just not aware of.
>
> This could be used by LTS maintainers, and possibly main, *-next branch
> maintainers coming up to releases.
>
> thanks,
> Kevin.
>
> > One proposal we (Michael and I) have toyed with for our lab is having
> > the infrastructure monitor patchwork comments for a restart flag, and
> > kick off based on that information.  Patchwork tracks all of the
> > comments for each patch / series so we could look at the series that
> > are still in a state for 'merging' (new, assigned, etc) and check the
> > patch .comments API for new comments.  Getting the data from PW should
> > be pretty simple - but I think that knowing whether to kick off the
> > test might be more difficult.  We have concerns about which messages we
> > should accept (for example, can anyone ask for a series to be rerun, and
> > we'll need to track which rerun messages we've accepted).  The
> > convention needs to be something we all can work with (ie: /Re-check:
> > [checkname] or something as a single line in the email).
> >
> > This is just a start to identify and explain the concern.  Maybe there
> > are other issues we've not considered, or maybe folks think this is a
> > terrible idea not worth spending any time developing.  I think there's
> > enough use for it that I am raising it here, and we can discuss it.
> >
> > Thanks,
> > -Aaron
> >
>
>

-- 
*Lincoln Lavoie*
Principal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>

Re: [dpdklab] Re: [dpdk-ci] [RFC] Proposal for allowing rerun of tests

[Kevin Traynor]

On 21/01/2022 17:57, Lincoln Lavoie wrote:
> Hi Kevin,
> 

Hi Lincoln,

> I'll add some notes around this to the 2022 planning doc.  The retests
> requests for the periodic tests are tricky from the discussed patchworks /
> mailing list approach, because there isn't an associated patch or patch
> ID.  So, we just need to document or agree on how to "flag" what the
> request is, so the system could uniquely ID what branch / test to rerun.
> 
> I'll also add this to the list for the dashboard technical debt as well, as
> there may be a graphical interface stuff (i.e. a button) we could add to
> the UNH-IOL hosted testing.
> 

 From my point of view, I don't really mind how a re-test is triggered. 
I was just suggesting a "button" as a way of saying that we mightn't 
need the complexity of parsing emails etc. as there would be just a few 
  branch maintainers and a few requests. But if it's more complex 
updating the dashboard, then that might not be best solution.

As always, please consider this just one request. If others don't see 
relative value in it, then makes sense to drop it or select something 
more important ahead of it.

thanks,
Kevin.

> Cheers,
> Lincoln
> 
> On Fri, Jan 21, 2022 at 9:00 AM Kevin Traynor <ktraynor@redhat.com> wrote:
> 
>> On 13/04/2021 14:50, Aaron Conole wrote:
>>> Greetings,
>>>
>>> During the various CI pipelines, sometimes a test setup or lab will
>>> have an internal failure unrelated to the specific patch.  Perhaps
>>> 'master' branch (or the associated -next branch) is broken and we cannot
>>> get a successful run anyway.  Perhaps a network outage occurs during
>>> infrastructure setup.  Perhaps some other transient error clobbers the
>>> setup.  In all of these cases the report to the mailing flags the patch
>>> as 'FAIL'.
>>>
>>> It would be very helpful if maintainers had the ability to tell various
>>> CI infrastructures to restart / rerun patch tests.  For now, this has to
>>> be done by the individual managers of those labs.  Some labs, it isn't
>>> possible.  Others, it's possible but is a very time-consuming process to
>>> restart a test case.  In all cases, a maintainer needs to spend time
>>> communicating with a lab manager.  This could be made a bit nicer.
>>>
>>
>> Just to tie two relevant threads together. I made a request in
>> http://mails.dpdk.org/archives/ci/2022-January/001593.html for a
>> "retest" button (or really any manual but self-sufficient way) to
>> kick-off immediately what is run in periodic branch testing. Something
>> might be there already, that i'm just not aware of.
>>
>> This could be used by LTS maintainers, and possibly main, *-next branch
>> maintainers coming up to releases.
>>
>> thanks,
>> Kevin.
>>
>>> One proposal we (Michael and I) have toyed with for our lab is having
>>> the infrastructure monitor patchwork comments for a restart flag, and
>>> kick off based on that information.  Patchwork tracks all of the
>>> comments for each patch / series so we could look at the series that
>>> are still in a state for 'merging' (new, assigned, etc) and check the
>>> patch .comments API for new comments.  Getting the data from PW should
>>> be pretty simple - but I think that knowing whether to kick off the
>>> test might be more difficult.  We have concerns about which messages we
>>> should accept (for example, can anyone ask for a series to be rerun, and
>>> we'll need to track which rerun messages we've accepted).  The
>>> convention needs to be something we all can work with (ie: /Re-check:
>>> [checkname] or something as a single line in the email).
>>>
>>> This is just a start to identify and explain the concern.  Maybe there
>>> are other issues we've not considered, or maybe folks think this is a
>>> terrible idea not worth spending any time developing.  I think there's
>>> enough use for it that I am raising it here, and we can discuss it.
>>>
>>> Thanks,
>>> -Aaron
>>>
>>
>>
>