Re: CI Testing Automated Recheck Request Framework

[Thomas Monjalon <thomas@monjalon.net>]

Thank you, it will help a lot.

How/Where are we going to document this?
Maybe in the "Testing" web page?
https://core.dpdk.org/testing/


25/08/2023 16:51, Adam Hassick:
> Hello DPDK developers,
> 
> Currently, various testing labs perform CI testing on new patch series sent
> to dev@dpdk.org and report their results to
> https://patchwork.dpdk.org/project/dpdk/list/. On each series on the patch
> list, the results appear in the test category contexts for IOL (community
> lab), GitHub, and LoongSon.
> 
> If a reported failure on a series seems suspicious to the patch submitter
> or maintainer, then there may be an interest in requesting a retest on the
> series for the failing label(s) in order to verify the failure is not
> spurious or a false positive. This retest demonstrates to the submitter or
> maintainer that the failure can be reliably reproduced. Unfortunately, at
> present, the best way to accomplish this is to reach out to lab maintainers
> via email or Slack. This is not ideal for developers in need of quick test
> results.
> 
> Going forward, CI testing labs will be implementing the option to request
> retest for their respective test labels on patchwork via emails sent to the
> dev mailing list. This feature is ready today for labels reported by the
> UNH-IOL Community Lab, and will soon also be an option for the Github Robot
> at least.
> 
> In order to request a retest on your patch series, send an email reply to
> one of your series’s patch or cover letter emails with email content of the
> format used below:
> 
> Recheck-request: <test names>
> 
> The valid delimiter is a comma optionally followed by a space: “,” “, “
> 
> Valid examples:
> 
> Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance,
> iol-compile-arm64-testing,
> 
> Recheck-request: iol-compile-amd64-testing,iol-broadcom-Performance,
> iol-compile-arm64-testing,
> 
> Recheck-request: iol-compile-amd64-testing, iol-broadcom-Performance,
> iol-compile-arm64-testing
> 
> Invalid examples:
> 
> Recheck-request: iol-compile-amd64-testing,  iol-broadcom-Performance
> 
> Recheck-request: iol-compile-amd64-testing
> iol-broadcom-Performance,iol-compile-arm64-testing,
> 
> Some important notes:
> 
>    1.
> 
>    At present, there is only support for retesting the series as it existed
>    when the lab received it. As in, if the lab applied the series on DPDK
>    mainline when the head was commit X, and a retest is requested, then
>    retests will be run using those same sources applied on top of commit X.
>    This is important to note because this means retest requests will not
>    provide a solution to your patch being submitted when the tree is in a “bad
>    state.” Getting test results with your patch applied on the current DPDK
>    mainline could be achieved by re-submitting the patch to the mailing list
>    as a workaround.
>    2.
> 
>    For any series submitted earlier than August 2023, you must submit a
>    retest request in reply to a patch email, NOT in reply to a cover letter
>    email.
>    3.
> 
>    The initial policy is to accept no more than one retest request per
>    patch series version per lab.
>    4.
> 
>    Your patch should begin to retest within 15 minutes of your request, but
>    wait time is subject to the testing queue just like any other series. As a
>    result, retesting will be slower during peak submission time.
> 
> 
> Improvements we are considering for v2 of the email retesting framework:
> 
>    1.
> 
>    Add in an option to re-apply on the latest commit on DPDK mainline. So,
>    if your patch was originally applied on commit X, and you want to retest,
>    but have it be applied to commit Y (latest), you could specify that. Under
>    these circumstances, we would have to do a retest of all labels, since it
>    would be inappropriate to mix reports for results from different commits.
>    2.
> 
>    Add a policy for vetting retest requesters - so maybe only maintainers,
>    or maybe only maintainers and the submitter, or another set of people.
>    3.
> 
>    Add in an option to request a retest for next-* branches and/or LTS
>    branches.
>