Re: [dpdk-ci] [dpdk-web] [PATCH] add general testing guidelines

["Richardson, Bruce" <bruce.richardson@intel.com>]

> -----Original Message-----
> From: web <web-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> Sent: Tuesday, March 10, 2020 10:12 PM
> To: web@dpdk.org
> Cc: ci@dpdk.org
> Subject: [dpdk-web] [PATCH] add general testing guidelines
> 
> The community lab page is replaced with a more general page about testing,
> where the community lab is part of the big picture. It includes:
> 	- testing goals
> 	- CI workflow
> 	- tools
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---

General question - is this meant to describe the ideal state or the current state of testing?
It starts off largely inspirational, but then ends with lots of details on the current
setup. Do we want to have that mix in the one page?

I've also put some stylistic comments inline below too.

/Bruce

>  content/lab/_index.md                |  21 ----
>  content/testing/_index.md            | 142 +++++++++++++++++++++++++++
>  static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes
>  3 files changed, 142 insertions(+), 21 deletions(-)  delete mode 100644
> content/lab/_index.md  create mode 100644 content/testing/_index.md
> delete mode 100644 static/img/community-lab-diagram.png
> 
> diff --git a/content/lab/_index.md b/content/lab/_index.md deleted file
> mode 100644 index 913c86f..0000000
> --- a/content/lab/_index.md
> +++ /dev/null
> @@ -1,21 +0,0 @@
> -+++
> -title = "Community Lab"
> -weight = "8"
> -+++
> -
> -{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{<
> /button >}}
> -
> -## What it's for
> -----
> -
> -- Automated performance regression testing of new patches.
> -  - When a new patch is received, automatically test
> -    and determine if it degrades performance or causes other issues
> -    on one or more architectures.
> -  - Provides input to maintainers on whether to accept patches or request
> rework.
> -- The Dashboard currently provides performance deltas for each patch
> -  compared to the expected numbers of each architecture.
> -- The scope may expand in future:
> -  test DPDK-enabled applications, use lab for demos, etc.
> -
> -![DPDK Community Lab](/img/community-lab-diagram.png)
> diff --git a/content/testing/_index.md b/content/testing/_index.md new
> file mode 100644 index 0000000..30da214
> --- /dev/null
> +++ b/content/testing/_index.md
> @@ -0,0 +1,142 @@
> ++++
> +title = "Testing"
> +weight = "8"
> ++++
> +
> +As a community project, DPDK CI is distributed and welcome any
> contribution.
> +
> +There are various ways of contributing to the CI effort:
> +
> +- suggest improvement
> +- write a new test
> +- improve infrastucture scripts

Typo: infrastRucture

> +- run a lab
> +
> +A specific effort is done
> +for the [community lab](//lab.dpdk.org/results/dashboard/about)
> +hosted at [UNH-IOL](//www.iol.unh.edu).
> +

Not sure about the phrase "specific effort". How about "One specific lab instance has been set up for the community lab..."

> +
> +## CI Testing Goals
> +---
> +
> +### General Properties
> +
> +Ideally, the CI infrastructure should have these properties:
> +
> +- reliable
> +- neutral

Neutral in what way? Presumably vendor or HW neutral?

> +- transparent visibility

What is implied by this? It is "clear methodology" or "transparent testing methods"?

> +- coverage summary
> +- trending graphs
> +- reports to author and
> +[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
> +- easy to add a test
> +- easy to reproduce a test locally
> +

The first three are what I'd consider properties, but the rest are more functions that should be included in the CI, rather than properties of it.

> +### Inputs
> +
> +There are three kinds of inputs to be tested:
> +
> +- per-patch testing integrated
> +with [patchwork](//patches.dpdk.org/project/dpdk/list/)
> +- per-commit testing after merges
> +in mainline or dpdk-next- [repositories](//git.dpdk.org/)
> +- per-release testing
> +including [release candidates](//git.dpdk.org/dpdk/refs/)
> +of all [branches](//git.dpdk.org/dpdk-stable/refs/)
> +
> +### Categories
> +
> +The tests should cover these categories:
> +
> +- compilation
> +- static analysis
> +- functional
> +- performance
> +- interoperability
> +- portability
> +- compatibility
> +

While compilation and static analysis are ok, the rest need "testing" after them since they are not nouns. Alternatively drop "static analysis" from the list and then change compilation to "compile" to use only adjectives.

> +
> +## Workflow
> +---
> +
> +### Patch Testing
> +
> +When a patch is submitted, it is received via the mailing list
> +[dpdk-dev](//inbox.dpdk.org/dev/).
> +For simple tests,
> +like
> +[checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh),
> +it can be run directly on email receiving.

s/email receiving/the received email/

> +
> +For most tests, the patch must be applied on the [right
> +tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py)
> +after its series dependencies.
> +Dealing with series and dependencies is easier via [patchwork database
> +request](//patches.dpdk.org/api/).
> +That's why the common workflow is
> +to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw).
> +The patches are saved and organized
> +in [patchwork](//patches.dpdk.org/project/dpdk/list/)
> +after being distributed by the mailing list.
> +
> +Once the patch is ready, the automatic tests can run.
> +
> +When a group of tests is finished, the author should be notified of any
> +error, and a public report must be sent to
> +[test-report@dpdk.org](//inbox.dpdk.org/test-report/).
> +
> +The reports are
> +[formatted](//git.dpdk.org/tools/dpdk-ci/tree/tools/send-patch-report.s
> +h) with a specific syntax for automatic parsing.
> +When such report email is received, the bot ci@dpdk.org will [update
> +patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh),
> +referencing the report archive and incrementing one of the counters
> +(success, warning or failure).
> +
> +A maintainer should wait the end of the tests (should be less than a
> +day), and check the results before accepting a patch.
> +
> +### Mainline Monitoring
> +
> +The repository mirror on [GitHub](//github.com/DPDK/dpdk) is preferred
> +for tests integration:
> +
> +- If polling for updates, better to load the GitHub mirror.
> +- Some services like [Travis](//travis-ci.com/DPDK/) can be hooked in
> +GitHub.
> +
> +### Release Validation
> +
> +When a new release is available, especially for release candidates,
> +some validation reports are requested to the community members.
> +
> +Such release validations are sent as replies to the release announcement.
> +
> +
> +## Tools
> +---
> +
> +### Infrastructure
> +
> +The testing infrastructure scripts are shared in the [dpdk-ci git
> +repository](//git.dpdk.org/tools/dpdk-ci/).
> +
> +The website dashboard and API of the community lab is a Django project
> +shared in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-
> site/).
> +
> +The testing infrastructure work is coordinated in three complementary
> forums:
> +
> +- [dpdk-ci mailing list](//inbox.dpdk.org/ci/)
> +- [bugzilla
> +tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20script
> +s&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_statu
> +s=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpri
> +ority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_forma
> +t=advanced&order=priority&list_id=1987)
> +- bi-weekly meeting
> +
> +### Known Test Frameworks and Tools
> +
> +- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/)
> +- [DPDK apps](//git.dpdk.org/dpdk/tree/app/)
> +- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/)
> +- [VSPERF](//git.opnfv.org/vswitchperf/about/)
> +- [OvS PVP perf](//github.com/chaudron/ovs_perf/)
> +- [downstream projects](//www.dpdk.org/ecosystem/#projects)
> diff --git a/static/img/community-lab-diagram.png b/static/img/community-
> lab-diagram.png
> deleted file mode 100644
> index a32109a..0000000
> Binary files a/static/img/community-lab-diagram.png and /dev/null differ
> --
> 2.25.1