Retesting Framework Announcement Draft

Retesting Framework Announcement Draft

[Adam Hassick]

[-- Attachment #1: Type: text/plain, Size: 3984 bytes --]

Hi All,

Below is our final draft of our announcement to the DPDK developers
informing them of our new recheck request system. We would like your
feedback and suggestions before we send it out.

Thanks,
Adam

--

Hello DPDK Developers,

Currently, various testing labs perform CI testing on new patch series sent
to dev@dpdk.org and report their results to
https://patchwork.dpdk.org/project/dpdk/list/. On each series on the patch
list, the results appear in the test category contexts for IOL (community
lab), GitHub, LoongSon, and Intel.

If a reported failure on a series seems suspect to the patch submitter or
maintainer, then there is an interest in requesting a retest on the series
for the failing label(s) in order to verify the failure is not spurious or
false positive. This retest demonstrates to the submitter or maintainer
that the failure can be reliably reproduced. Unfortunately, at present, the
best way to accomplish this is to reach out to lab maintainers via email or
slack. This is not ideal for developers in need of quick test results.

Going forward, CI testing labs will be implementing the option to request
retest for their respective test labels on patchwork via emails sent to the
dev mailing list. This feature is ready today for labels reported by the
UNH-IOL Community Lab, and will soon also be an option for the Github Robot
at least.

In order to request a retest on your patch series, send an email reply to
one of your series’s patch or cover letter emails with email content of the
format used below:

Recheck-request: <test names>

The valid delimiter is a comma optionally followed by a space or a newline
character: “,” “, “ “,\n”

Valid examples:

Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance,
iol-compile-arm64-testing,

Recheck-request: iol-compile-amd64-testing,iol-broadcom-Performance,
iol-compile-arm64-testing,

Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance,
iol-compile-arm64-testing

Invalid examples:

Recheck-request: iol-compile-amd64-testing,  iol-broadcom-Performance

Recheck-request: iol-compile-amd64-testing
iol-broadcom-Performance,iol-compile-arm64-testing,

Some important notes:

   1.

   At present, there is only support for retesting the series as it existed
   when the lab received it. As in, if the lab applied the series on DPDK
   mainline when the head was commit X, and a retest is requested, then
   retests will be run using those same sources applied on top of commit X.
   This is important to note because this means retest requests will not
   provide a solution to your patch being submitted when the tree is in a “bad
   state.”
   2.

   For any series submitted earlier than August 2023, you must submit a
   retest request in reply to a patch email, NOT in reply to a cover letter
   email.
   3.

   The initial policy is to accept no more than one retest request per
   patch per lab.
   4.

   Your patch should begin to retest within 30 minutes of your request, but
   wait time is subject to the testing queue just like any other series. As a
   result, retesting will be slower during peak submission time.


Improvements we are considering for v2 of the email retesting framework:

   1.

   Add in an option to re-apply on the latest commit on DPDK mainline. So,
   if your patch was originally applied on commit X, and you want to retest,
   but have it be applied to commit Y (latest), you could specify that. Under
   these circumstances, we would have to do a retest of all labels, since it
   would be inappropriate to mix reports for results from different commits.
   2.

   Add a policy for vetting retest requesters - so maybe only maintainers,
   or maybe only maintainers and the submitter, or another set of people.
   3.

   Add in an option to request a retest for next-* branches and/or LTS
   branches.

[-- Attachment #2: Type: text/html, Size: 15661 bytes --]

Re: Retesting Framework Announcement Draft

[Aaron Conole]

Adam Hassick <ahassick@iol.unh.edu> writes:

> Hi All,
>
> Below is our final draft of our announcement to the DPDK developers informing them of our new recheck
> request system. We would like your feedback and suggestions before we send it out.
>
> Thanks,
> Adam
>
> --
>
> Hello DPDK Developers,
>
> Currently, various testing labs perform CI testing on new patch series sent to dev@dpdk.org and report their
> results to https://patchwork.dpdk.org/project/dpdk/list/. On each series on the patch list, the results
> appear in the test category contexts for IOL (community lab), GitHub, LoongSon, and Intel.

We should probably drop Intel from this, as they haven't submitted a
result in 2 weeks.

> If a reported failure on a series seems suspect to the patch submitter or maintainer, then there is an interest in
> requesting a retest on the series for the failing label(s) in order to verify the failure is not spurious or false
> positive. This retest demonstrates to the submitter or maintainer that the failure can be reliably reproduced.
> Unfortunately, at present, the best way to accomplish this is to reach out to lab maintainers via email or slack.
> This is not ideal for developers in need of quick test results.
>
> Going forward, CI testing labs will be implementing the option to request retest for their respective test labels
> on patchwork via emails sent to the dev mailing list. This feature is ready today for labels reported by the
> UNH-IOL Community Lab, and will soon also be an option for the Github Robot at least.

ACK

> In order to request a retest on your patch series, send an email reply to one of your series’s patch or cover
> letter emails with email content of the format used below: 
>
> Recheck-request: <test names>
>
> The valid delimiter is a comma optionally followed by a space or a newline character: “,” “, “ “,\n”

Even if you support the new-line character, let's not advertise that
feature.  The original stuff we agreed on was just for a single line.
Some regex parsers won't support multi-line.

> Valid examples:
>
> Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, iol-compile-arm64-testing,
>
> Recheck-request: iol-compile-amd64-testing,iol-broadcom-Performance, iol-compile-arm64-testing,
>
> Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, iol-compile-arm64-testing
>
> Invalid examples:
>
> Recheck-request: iol-compile-amd64-testing,  iol-broadcom-Performance
>
> Recheck-request: iol-compile-amd64-testing iol-broadcom-Performance,iol-compile-arm64-testing,
>
> Some important notes:
>
> 1 At present, there is only support for retesting the series as it existed when the lab received it. As in, if the
>  lab applied the series on DPDK mainline when the head was commit X, and a retest is requested, then
>  retests will be run using those same sources applied on top of commit X. This is important to note because
>  this means retest requests will not provide a solution to your patch being submitted when the tree is in a
>  “bad state.”
>
> 2 For any series submitted earlier than August 2023, you must submit a retest request in reply to a patch
>  email, NOT in reply to a cover letter email.
>
> 3 The initial policy is to accept no more than one retest request per patch per lab.
>
> 4 Your patch should begin to retest within 30 minutes of your request, but wait time is subject to the testing
>  queue just like any other series. As a result, retesting will be slower during peak submission time.
>
> Improvements we are considering for v2 of the email retesting framework:
>
> 1 Add in an option to re-apply on the latest commit on DPDK mainline. So, if your patch was originally applied
>  on commit X, and you want to retest, but have it be applied to commit Y (latest), you could specify that.
>  Under these circumstances, we would have to do a retest of all labels, since it would be inappropriate to mix
>  reports for results from different commits.

We could say that a workaround to this is that they can always resubmit
the patch series to the mailing list in this case.

> 2 Add a policy for vetting retest requesters - so maybe only maintainers, or maybe only maintainers and the
>  submitter, or another set of people. 
>
> 3 Add in an option to request a retest for next-* branches and/or LTS branches.