Re: [PATCH 1/1] tools: check for pending test status when parsing emails

[Patrick Robb <probb@iol.unh.edu>]

On Mon, May 20, 2024 at 3:03 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 17/05/2024 21:22, Patrick Robb:
> > Signed-off-by: Patrick Robb <probb@iol.unh.edu>
>
> Please could you explain what it is doing?
> Having a workflow understanding would be nice.

Yes good idea.

For context, pending is already a supported check state in patchwork
server: https://patchwork.readthedocs.io/en/latest/usage/overview/#checks

1. DPDK patch is submitted. Patch is acquired by UNH Lab.
2. UNH Lab triggers some testrun pipelines in our CI system (jenkins).
The first action the pipeline takes is to create in our database a
test result record for each testrun, setting the status to PENDING. It
is important to note that one patchwork context, Like
"iol-compile-amd64-testing," may consist of many individual testruns,
each for different distros, hardware, environment etc.
3. When each testrun completes, it will send a report to Patchwork
with the new result (pass or fail). When it does this it will update
the context's results table, changing the environment's result from
pending to pass/fail. So, when the first report comes in for, say,
context "iol-compile-amd64-testing," you would see 1 pass/fail, 12
pending, or similar. Then, as subsequent testruns complete, and report
their results, the updated table comes with the new report. The
overall context result (the _Testing {PASS/FAIL/PENDING}_ at the top
of the test report email) is determined in the manner you might
expect, i.e. if there is at least one testrun fail result, overall
context is fail, else if there is at least one pending result, overall
context is pending, else if all results are passing, overall result is
passing. As an example, when testing is nearly complete, the top of
the report email may look like this:

_Testing PENDING_

Branch: tags/v22.11

a409653a123bf105970a25c594711a3cdc44d139 --> testing pass

Test environment and result as below:

+------------------------------------+-----------------------------------------------------+
|            Environment             |       dpdk_meson_compile      |
+====================================+====================+
| Ubuntu 20.04 ARM SVE                          | PASS               |
+------------------------------------+--------------------+
| Debian 12 with MUSDK                           | PENDING        |
+------------------------------------+--------------------+
| Fedora 37 (ARM)                                     | PASS               |
+------------------------------------+--------------------+
| Ubuntu 20.04 (ARM)                                | PASS               |
+------------------------------------+--------------------+
| Fedora 38 (ARM)                                     | PASS               |
+------------------------------------+--------------------+
| Fedora 39 (ARM)                                     | PENDING        |
+------------------------------------+--------------------+
| Debian 12 (arm)                                       | PASS               |
+------------------------------------+--------------------+
| CentOS Stream 9 (ARM)                         | PASS               |
+------------------------------------+--------------------+
| Debian 11 (Buster) (ARM)                        | PASS               |
+------------------------------------+--------------------+
| Ubuntu 20.04 ARM GCC Cross Compile | PASS               |
+------------------------------------+--------------------+


4. Eventually, all testruns are complete for a patchwork context, and
the table switches from pending to pass or fail.

This does not slow the delivery of results, nor does it increase the
number of test report emails sent. We still send only 1 email per
testrun.

This way it is plainly visible to the user when all testing is
complete, and it also flags for the submitter and for CI people if
some infra failure prevents a testrun from completing, or from a
result being properly emailed, etc. The idea is to provide more
complete status updates and check against infra fails better, but
without any adverse effects in user experience or load on the email
server.

>
> > --- a/tools/update-pw.sh
> > +++ b/tools/update-pw.sh
> > @@ -49,6 +49,7 @@ case $status in
> >       'SUCCESS') pwstatus='success' ;;
> >       'WARNING') pwstatus='warning' ;;
> >       'FAILURE') pwstatus='fail' ;;
> > +     'PENDING') pwstatus='pending' ;;
>
>
>