Re: Retesting Framework Announcement Draft

[Aaron Conole <aconole@redhat.com>]

Adam Hassick <ahassick@iol.unh.edu> writes:

> Hi All,
>
> Below is our final draft of our announcement to the DPDK developers informing them of our new recheck
> request system. We would like your feedback and suggestions before we send it out.
>
> Thanks,
> Adam
>
> --
>
> Hello DPDK Developers,
>
> Currently, various testing labs perform CI testing on new patch series sent to dev@dpdk.org and report their
> results to https://patchwork.dpdk.org/project/dpdk/list/. On each series on the patch list, the results
> appear in the test category contexts for IOL (community lab), GitHub, LoongSon, and Intel.

We should probably drop Intel from this, as they haven't submitted a
result in 2 weeks.

> If a reported failure on a series seems suspect to the patch submitter or maintainer, then there is an interest in
> requesting a retest on the series for the failing label(s) in order to verify the failure is not spurious or false
> positive. This retest demonstrates to the submitter or maintainer that the failure can be reliably reproduced.
> Unfortunately, at present, the best way to accomplish this is to reach out to lab maintainers via email or slack.
> This is not ideal for developers in need of quick test results.
>
> Going forward, CI testing labs will be implementing the option to request retest for their respective test labels
> on patchwork via emails sent to the dev mailing list. This feature is ready today for labels reported by the
> UNH-IOL Community Lab, and will soon also be an option for the Github Robot at least.

ACK

> In order to request a retest on your patch series, send an email reply to one of your series’s patch or cover
> letter emails with email content of the format used below: 
>
> Recheck-request: <test names>
>
> The valid delimiter is a comma optionally followed by a space or a newline character: “,” “, “ “,\n”

Even if you support the new-line character, let's not advertise that
feature.  The original stuff we agreed on was just for a single line.
Some regex parsers won't support multi-line.

> Valid examples:
>
> Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, iol-compile-arm64-testing,
>
> Recheck-request: iol-compile-amd64-testing,iol-broadcom-Performance, iol-compile-arm64-testing,
>
> Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance, iol-compile-arm64-testing
>
> Invalid examples:
>
> Recheck-request: iol-compile-amd64-testing,  iol-broadcom-Performance
>
> Recheck-request: iol-compile-amd64-testing iol-broadcom-Performance,iol-compile-arm64-testing,
>
> Some important notes:
>
> 1 At present, there is only support for retesting the series as it existed when the lab received it. As in, if the
>  lab applied the series on DPDK mainline when the head was commit X, and a retest is requested, then
>  retests will be run using those same sources applied on top of commit X. This is important to note because
>  this means retest requests will not provide a solution to your patch being submitted when the tree is in a
>  “bad state.”
>
> 2 For any series submitted earlier than August 2023, you must submit a retest request in reply to a patch
>  email, NOT in reply to a cover letter email.
>
> 3 The initial policy is to accept no more than one retest request per patch per lab.
>
> 4 Your patch should begin to retest within 30 minutes of your request, but wait time is subject to the testing
>  queue just like any other series. As a result, retesting will be slower during peak submission time.
>
> Improvements we are considering for v2 of the email retesting framework:
>
> 1 Add in an option to re-apply on the latest commit on DPDK mainline. So, if your patch was originally applied
>  on commit X, and you want to retest, but have it be applied to commit Y (latest), you could specify that.
>  Under these circumstances, we would have to do a retest of all labels, since it would be inappropriate to mix
>  reports for results from different commits.

We could say that a workaround to this is that they can always resubmit
the patch series to the mailing list in this case.

> 2 Add a policy for vetting retest requesters - so maybe only maintainers, or maybe only maintainers and the
>  submitter, or another set of people. 
>
> 3 Add in an option to request a retest for next-* branches and/or LTS branches.