DPDK website maintenance
 help / color / mirror / Atom feed
From: Thomas Monjalon <thomas@monjalon.net>
To: web@dpdk.org
Cc: ci@dpdk.org, bruce.richardson@intel.com
Subject: [dpdk-web] [PATCH v2] add general testing guidelines
Date: Tue, 24 Mar 2020 12:31:29 +0100
Message-ID: <20200324113129.182525-1-thomas@monjalon.net> (raw)
In-Reply-To: <20200310221130.46599-1-thomas@monjalon.net>

The community lab page is replaced with a more general page about testing,
where the community lab is part of the big picture. It includes:
	- testing goals
	- CI workflow
	- tools

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>


    - fix typo
    - reword community lab
    - explain CI properties
    - group test categories
    - reword email workflow
    - remove id from bugzilla request

 content/lab/_index.md                |  21 ----
 content/testing/_index.md            | 144 +++++++++++++++++++++++++++
 static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes
 3 files changed, 144 insertions(+), 21 deletions(-)
 delete mode 100644 content/lab/_index.md
 create mode 100644 content/testing/_index.md
 delete mode 100644 static/img/community-lab-diagram.png

diff --git a/content/lab/_index.md b/content/lab/_index.md
deleted file mode 100644
index 913c86f..0000000
--- a/content/lab/_index.md
+++ /dev/null
@@ -1,21 +0,0 @@
-title = "Community Lab"
-weight = "8"
-{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{< /button >}}
-## What it's for
-- Automated performance regression testing of new patches.
-  - When a new patch is received, automatically test
-    and determine if it degrades performance or causes other issues
-    on one or more architectures.
-  - Provides input to maintainers on whether to accept patches or request rework.
-- The Dashboard currently provides performance deltas for each patch
-  compared to the expected numbers of each architecture.
-- The scope may expand in future:
-  test DPDK-enabled applications, use lab for demos, etc.
-![DPDK Community Lab](/img/community-lab-diagram.png)
diff --git a/content/testing/_index.md b/content/testing/_index.md
new file mode 100644
index 0000000..1648f45
--- /dev/null
+++ b/content/testing/_index.md
@@ -0,0 +1,144 @@
+title = "Testing"
+weight = "8"
+As a community project, DPDK CI is distributed and welcome any contribution.
+There are various ways of contributing to the CI effort:
+- suggest improvement
+- write a new test
+- improve infrastructure scripts
+- run a lab
+One specific lab instance,
+named [community lab](//lab.dpdk.org/results/dashboard/about),
+is hosted at [UNH-IOL](//www.iol.unh.edu).
+## CI Testing Goals
+### General Properties
+Ideally, the CI infrastructure should have these properties:
+- reliable: always run, without false positive
+- vendor neutral: run same tests and configurations on all HW/SW platforms
+- transparent: show environment, configuration, source code and result
+- coverage summary
+- trending graphs
+- reports to author and
+- easy to add a test
+- easy to reproduce a test locally
+### Inputs
+There are three kinds of inputs to be tested:
+- per-patch testing integrated
+with [patchwork](//patches.dpdk.org/project/dpdk/list/)
+- per-commit testing after merges
+in mainline or dpdk-next- [repositories](//git.dpdk.org/)
+- per-release testing
+including [release candidates](//git.dpdk.org/dpdk/refs/)
+of all [branches](//git.dpdk.org/dpdk-stable/refs/)
+### Categories
+The tests should cover these categories:
+* source code checks:
+  - static analysis
+  - compilation
+* runtime tests:
+  - functional
+  - performance
+  - interoperability
+  - portability
+  - compatibility
+## Workflow
+### Patch Testing
+When a patch is submitted, it is received
+via the mailing list [dpdk-dev](//inbox.dpdk.org/dev/).
+For simple tests,
+like [checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh),
+it can be run on email reception.
+For most tests, the patch must be applied
+on the [right tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py)
+after its series dependencies.
+Dealing with series and dependencies is easier
+via [patchwork database request](//patches.dpdk.org/api/).
+That's why the common workflow is
+to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw).
+The patches are saved and organized
+in [patchwork](//patches.dpdk.org/project/dpdk/list/)
+after being distributed by the mailing list.
+Once the patch is ready, the automatic tests can run.
+When a group of tests is finished, the author should be notified of any error,
+and a public report must be sent to
+The reports are
+with a specific syntax for automatic parsing.
+When such report email is received, the bot ci@dpdk.org will
+[update patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh),
+referencing the report archive and incrementing one of the counters
+(success, warning or failure).
+A maintainer should wait the end of the tests (should be less than a day),
+and check the results before accepting a patch.
+### Mainline Monitoring
+The repository mirror on [GitHub](//github.com/DPDK/dpdk)
+is preferred for tests integration:
+- If polling for updates, better to load the GitHub mirror.
+- Some services like [Travis](//travis-ci.com/DPDK/)
+can be hooked in GitHub.
+### Release Validation
+When a new release is available, especially for release candidates,
+some validation reports are requested to the community members.
+Such release validations are sent as replies to the release announcement.
+## Tools
+### Infrastructure
+The testing infrastructure scripts are shared in
+the [dpdk-ci git repository](//git.dpdk.org/tools/dpdk-ci/).
+The website dashboard and API of the community lab is a Django project shared
+in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-site/).
+The testing infrastructure work is coordinated in three complementary forums:
+- [dpdk-ci mailing list](//inbox.dpdk.org/ci/)
+- [bugzilla tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20scripts&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpriority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_format=advanced&order=priority)
+- bi-weekly meeting
+### Known Test Frameworks and Tools
+- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/)
+- [DPDK apps](//git.dpdk.org/dpdk/tree/app/)
+- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/)
+- [VSPERF](//git.opnfv.org/vswitchperf/about/)
+- [OvS PVP perf](//github.com/chaudron/ovs_perf/)
+- [downstream projects](//www.dpdk.org/ecosystem/#projects)
diff --git a/static/img/community-lab-diagram.png b/static/img/community-lab-diagram.png
deleted file mode 100644
index a32109a..0000000
Binary files a/static/img/community-lab-diagram.png and /dev/null differ

  parent reply	other threads:[~2020-03-24 11:31 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-10 22:11 [dpdk-web] [PATCH] " Thomas Monjalon
2020-03-16  9:43 ` Richardson, Bruce
2020-03-16 11:49   ` Thomas Monjalon
2020-03-16 12:17     ` Richardson, Bruce
2020-03-16 13:21       ` Thomas Monjalon
2020-03-24 11:31 ` Thomas Monjalon [this message]
2020-03-28 21:46   ` [dpdk-web] [dpdk-ci] [PATCH v2] " Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200324113129.182525-1-thomas@monjalon.net \
    --to=thomas@monjalon.net \
    --cc=bruce.richardson@intel.com \
    --cc=ci@dpdk.org \
    --cc=web@dpdk.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK website maintenance

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/web/0 web/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 web web/ http://inbox.dpdk.org/web \
	public-inbox-index web

Example config snippet for mirrors.
Newsgroup available over NNTP:

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git