DPDK website maintenance
 help / color / Atom feed
From: Thomas Monjalon <thomas@monjalon.net>
To: web@dpdk.org
Cc: ci@dpdk.org, bruce.richardson@intel.com
Subject: [dpdk-web] [PATCH v2] add general testing guidelines
Date: Tue, 24 Mar 2020 12:31:29 +0100
Message-ID: <20200324113129.182525-1-thomas@monjalon.net> (raw)
In-Reply-To: <20200310221130.46599-1-thomas@monjalon.net>

The community lab page is replaced with a more general page about testing,
where the community lab is part of the big picture. It includes:
	- testing goals
	- CI workflow
	- tools

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

---

v2:
    - fix typo
    - reword community lab
    - explain CI properties
    - group test categories
    - reword email workflow
    - remove id from bugzilla request

---
 content/lab/_index.md                |  21 ----
 content/testing/_index.md            | 144 +++++++++++++++++++++++++++
 static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes
 3 files changed, 144 insertions(+), 21 deletions(-)
 delete mode 100644 content/lab/_index.md
 create mode 100644 content/testing/_index.md
 delete mode 100644 static/img/community-lab-diagram.png

diff --git a/content/lab/_index.md b/content/lab/_index.md
deleted file mode 100644
index 913c86f..0000000
--- a/content/lab/_index.md
+++ /dev/null
@@ -1,21 +0,0 @@
-+++
-title = "Community Lab"
-weight = "8"
-+++
-
-{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{< /button >}}
-
-## What it's for
-----
-
-- Automated performance regression testing of new patches.
-  - When a new patch is received, automatically test
-    and determine if it degrades performance or causes other issues
-    on one or more architectures.
-  - Provides input to maintainers on whether to accept patches or request rework.
-- The Dashboard currently provides performance deltas for each patch
-  compared to the expected numbers of each architecture.
-- The scope may expand in future:
-  test DPDK-enabled applications, use lab for demos, etc.
-
-![DPDK Community Lab](/img/community-lab-diagram.png)
diff --git a/content/testing/_index.md b/content/testing/_index.md
new file mode 100644
index 0000000..1648f45
--- /dev/null
+++ b/content/testing/_index.md
@@ -0,0 +1,144 @@
++++
+title = "Testing"
+weight = "8"
++++
+
+As a community project, DPDK CI is distributed and welcome any contribution.
+
+There are various ways of contributing to the CI effort:
+
+- suggest improvement
+- write a new test
+- improve infrastructure scripts
+- run a lab
+
+One specific lab instance,
+named [community lab](//lab.dpdk.org/results/dashboard/about),
+is hosted at [UNH-IOL](//www.iol.unh.edu).
+
+
+## CI Testing Goals
+---
+
+### General Properties
+
+Ideally, the CI infrastructure should have these properties:
+
+- reliable: always run, without false positive
+- vendor neutral: run same tests and configurations on all HW/SW platforms
+- transparent: show environment, configuration, source code and result
+- coverage summary
+- trending graphs
+- reports to author and
+[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
+- easy to add a test
+- easy to reproduce a test locally
+
+### Inputs
+
+There are three kinds of inputs to be tested:
+
+- per-patch testing integrated
+with [patchwork](//patches.dpdk.org/project/dpdk/list/)
+- per-commit testing after merges
+in mainline or dpdk-next- [repositories](//git.dpdk.org/)
+- per-release testing
+including [release candidates](//git.dpdk.org/dpdk/refs/)
+of all [branches](//git.dpdk.org/dpdk-stable/refs/)
+
+### Categories
+
+The tests should cover these categories:
+
+* source code checks:
+  - static analysis
+  - compilation
+* runtime tests:
+  - functional
+  - performance
+  - interoperability
+  - portability
+  - compatibility
+
+
+## Workflow
+---
+
+### Patch Testing
+
+When a patch is submitted, it is received
+via the mailing list [dpdk-dev](//inbox.dpdk.org/dev/).
+For simple tests,
+like [checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh),
+it can be run on email reception.
+
+For most tests, the patch must be applied
+on the [right tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py)
+after its series dependencies.
+Dealing with series and dependencies is easier
+via [patchwork database request](//patches.dpdk.org/api/).
+That's why the common workflow is
+to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw).
+The patches are saved and organized
+in [patchwork](//patches.dpdk.org/project/dpdk/list/)
+after being distributed by the mailing list.
+
+Once the patch is ready, the automatic tests can run.
+
+When a group of tests is finished, the author should be notified of any error,
+and a public report must be sent to
+[test-report@dpdk.org](//inbox.dpdk.org/test-report/).
+
+The reports are
+[formatted](//git.dpdk.org/tools/dpdk-ci/tree/tools/send-patch-report.sh)
+with a specific syntax for automatic parsing.
+When such report email is received, the bot ci@dpdk.org will
+[update patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh),
+referencing the report archive and incrementing one of the counters
+(success, warning or failure).
+
+A maintainer should wait the end of the tests (should be less than a day),
+and check the results before accepting a patch.
+
+### Mainline Monitoring
+
+The repository mirror on [GitHub](//github.com/DPDK/dpdk)
+is preferred for tests integration:
+
+- If polling for updates, better to load the GitHub mirror.
+- Some services like [Travis](//travis-ci.com/DPDK/)
+can be hooked in GitHub.
+
+### Release Validation
+
+When a new release is available, especially for release candidates,
+some validation reports are requested to the community members.
+
+Such release validations are sent as replies to the release announcement.
+
+
+## Tools
+---
+
+### Infrastructure
+
+The testing infrastructure scripts are shared in
+the [dpdk-ci git repository](//git.dpdk.org/tools/dpdk-ci/).
+
+The website dashboard and API of the community lab is a Django project shared
+in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-site/).
+
+The testing infrastructure work is coordinated in three complementary forums:
+
+- [dpdk-ci mailing list](//inbox.dpdk.org/ci/)
+- [bugzilla tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20scripts&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpriority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_format=advanced&order=priority)
+- bi-weekly meeting
+
+### Known Test Frameworks and Tools
+
+- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/)
+- [DPDK apps](//git.dpdk.org/dpdk/tree/app/)
+- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/)
+- [VSPERF](//git.opnfv.org/vswitchperf/about/)
+- [OvS PVP perf](//github.com/chaudron/ovs_perf/)
+- [downstream projects](//www.dpdk.org/ecosystem/#projects)
diff --git a/static/img/community-lab-diagram.png b/static/img/community-lab-diagram.png
deleted file mode 100644
index a32109a..0000000
Binary files a/static/img/community-lab-diagram.png and /dev/null differ
-- 
2.25.1


  parent reply index

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-10 22:11 [dpdk-web] [PATCH] " Thomas Monjalon
2020-03-16  9:43 ` Richardson, Bruce
2020-03-16 11:49   ` Thomas Monjalon
2020-03-16 12:17     ` Richardson, Bruce
2020-03-16 13:21       ` Thomas Monjalon
2020-03-24 11:31 ` Thomas Monjalon [this message]
2020-03-28 21:46   ` [dpdk-web] [dpdk-ci] [PATCH v2] " Thomas Monjalon

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200324113129.182525-1-thomas@monjalon.net \
    --to=thomas@monjalon.net \
    --cc=bruce.richardson@intel.com \
    --cc=ci@dpdk.org \
    --cc=web@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK website maintenance

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/web/0 web/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 web web/ http://inbox.dpdk.org/web \
		web@dpdk.org
	public-inbox-index web


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.web


AGPL code for this site: git clone https://public-inbox.org/ public-inbox