[dpdk-dev] Fwd: distributed tests of a patch

[Thomas Monjalon <thomas.monjalon@6wind.com>]

Hi,

With the help of Thomas Herbert and Stephen Finucane, we started a discussion
to have a distributed continuous integration system in patchwork:
http://thread.gmane.org/gmane.comp.version-control.patchwork/1242

The goal is to allow many contributors to run their own tests when a patch
is received and check all the results in patchwork before accepting it.
As DPDK is gaining more and more hardware support, it becomes impossible to
test every hardware combinations when making a change. Ideally, it would be
nice to have tests run by each hardware vendor. It is the same combination
issue to test every OS and every build options.
Clearly, the test coverage of the DPDK will never be 100%, but a distributed
effort can help a lot to avoid dropping the support of some hardware or OS.

-----------------------------------

From: thomas.monjalon@6wind.com
To:   patchwork@lists.ozlabs.org
CC:   therbert@redhat.com, stephen.finucane@intel.com

Hello,

Summary: it would be very nice to take into account some distributed tests
in patchwork.
Longer text below...

Patchwork is incredibly useful to list the patches received in the mailing
list of an Open Source project and classify them thanks to the associated
status. A patch is accepted after doing some review, reading the discussion,
having some specific associated tags (A/R/T column) and testing. This last
point may be automated but there is no way of gathering test results from
many contributors in a central view which is patchwork.
It would be really useful to add a new column to count the automated test
results Success, Warnings and Errors (S/W/E) as it is done for manual tags
Acked-by/Reviewed-by/Tested-by.

As it is done for managing the patches, an XML-RPC API could be used to add
a new test report associated to a patch ID. A test report could be also
received by email on a mailing list dedicated to test reports.
An advantage of sending test reports to a mailing list, is to be able
to receive them without the need of patchwork, and eventually do some
custom processing. It is also possible to offload the storage of the
test reports in the mailing list archives. Last advantage, the mailing list
administration allows to whitelist the test contributors.

With such a distributed test system, many contributors can run different test
suites on their own set of hardware / OS environment for each submitted patch,
improving the quality of the impacted project.

Below is a workflow giving more details.

Let's consider a project having 3 mailing lists:
- devel@ to receive the patches and comments.
- test@ is a mailing list without archive to send patches to test.
- reports@ is a mailing list with archives to store the test reports.
Each patch from devel@ is sent to test@ with prefix [patchwork id].
Private tests may be requested by sending patches to test@.
If an email is received in test@ without [PATCH] keyword, it is filtered out.
Each email sent to test@ is forwarded to each registered test platforms
(i.e. testers are members of the test@ mailing list).
Each test platform apply the patch (and previous patches of the series) on its
local git tree.
Each test platform must reply 1 or more reports with exactly the same subject
to the original sender or reports@ if List-Id was devel@.
A test report may be linked to a test suite doing several tests.
The first line begins with Test-Label: followed by a test name.
The second line begins with Test-Status: followed by SUCCESS, WARNING or ERROR.
The WARNING keyword may be used by tests known to have some false positives
or not so important.
The detailed report must be inline to allow easy reading from mail archives.
When the report is received in reports@, the patchwork id entry is updated to
be associated to a new test entry including the test label, status and the URL
to the report archives.
The new test result is visible in 2 patchwork views.
The patch list has a column to display the count of test success, warnings and
errors. These counters are a link to the patch view with test section unfold.
The detailed patch view has a test section (fold by default) like the headers
section. When fold, it shows only the counters. When unfold, it shows one test
result per line: the test label is followed by the status being a HTML link to
the full report.
When refreshing one of these views, it is possible to see the test progress,
i.e. how many tests are complete.
In this scheme, there is no definitive global test status.
The idea is to wait to have a common number of success and no error (removing
possible false positives) to consider a patch as good. This judgement is done
by the maintainers.
In case a test need to be re-run, it is possible to send an email to test@,
keeping the patch id and the test label. When the new report will be received,
the counters and the detailed view show only the last result of this test.

That's the end of this long story. Hope you like it and welcome the idea!