DPDK CI discussions
 help / color / mirror / Atom feed
* [dpdk-ci] [PATCH] add general testing guidelines
@ 2020-03-10 22:11 Thomas Monjalon
  2020-03-16  9:43 ` [dpdk-ci] [dpdk-web] " Richardson, Bruce
  2020-03-24 11:31 ` [dpdk-ci] [PATCH v2] " Thomas Monjalon
  0 siblings, 2 replies; 7+ messages in thread
From: Thomas Monjalon @ 2020-03-10 22:11 UTC (permalink / raw)
  To: web; +Cc: ci

The community lab page is replaced with a more general page about testing,
where the community lab is part of the big picture. It includes:
	- testing goals
	- CI workflow
	- tools

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
 content/lab/_index.md                |  21 ----
 content/testing/_index.md            | 142 +++++++++++++++++++++++++++
 static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes
 3 files changed, 142 insertions(+), 21 deletions(-)
 delete mode 100644 content/lab/_index.md
 create mode 100644 content/testing/_index.md
 delete mode 100644 static/img/community-lab-diagram.png

diff --git a/content/lab/_index.md b/content/lab/_index.md
deleted file mode 100644
index 913c86f..0000000
--- a/content/lab/_index.md
+++ /dev/null
@@ -1,21 +0,0 @@
-+++
-title = "Community Lab"
-weight = "8"
-+++
-
-{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{< /button >}}
-
-## What it's for
-----
-
-- Automated performance regression testing of new patches.
-  - When a new patch is received, automatically test
-    and determine if it degrades performance or causes other issues
-    on one or more architectures.
-  - Provides input to maintainers on whether to accept patches or request rework.
-- The Dashboard currently provides performance deltas for each patch
-  compared to the expected numbers of each architecture.
-- The scope may expand in future:
-  test DPDK-enabled applications, use lab for demos, etc.
-
-![DPDK Community Lab](/img/community-lab-diagram.png)
diff --git a/content/testing/_index.md b/content/testing/_index.md
new file mode 100644
index 0000000..30da214
--- /dev/null
+++ b/content/testing/_index.md
@@ -0,0 +1,142 @@
++++
+title = "Testing"
+weight = "8"
++++
+
+As a community project, DPDK CI is distributed and welcome any contribution.
+
+There are various ways of contributing to the CI effort:
+
+- suggest improvement
+- write a new test
+- improve infrastucture scripts
+- run a lab
+
+A specific effort is done
+for the [community lab](//lab.dpdk.org/results/dashboard/about)
+hosted at [UNH-IOL](//www.iol.unh.edu).
+
+
+## CI Testing Goals
+---
+
+### General Properties
+
+Ideally, the CI infrastructure should have these properties:
+
+- reliable
+- neutral
+- transparent visibility
+- coverage summary
+- trending graphs
+- reports to author and
+[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
+- easy to add a test
+- easy to reproduce a test locally
+
+### Inputs
+
+There are three kinds of inputs to be tested:
+
+- per-patch testing integrated
+with [patchwork](//patches.dpdk.org/project/dpdk/list/)
+- per-commit testing after merges
+in mainline or dpdk-next- [repositories](//git.dpdk.org/)
+- per-release testing
+including [release candidates](//git.dpdk.org/dpdk/refs/)
+of all [branches](//git.dpdk.org/dpdk-stable/refs/)
+
+### Categories
+
+The tests should cover these categories:
+
+- compilation
+- static analysis
+- functional
+- performance
+- interoperability
+- portability
+- compatibility
+
+
+## Workflow
+---
+
+### Patch Testing
+
+When a patch is submitted, it is received
+via the mailing list [dpdk-dev](//inbox.dpdk.org/dev/).
+For simple tests,
+like [checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh),
+it can be run directly on email receiving.
+
+For most tests, the patch must be applied
+on the [right tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py)
+after its series dependencies.
+Dealing with series and dependencies is easier
+via [patchwork database request](//patches.dpdk.org/api/).
+That's why the common workflow is
+to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw).
+The patches are saved and organized
+in [patchwork](//patches.dpdk.org/project/dpdk/list/)
+after being distributed by the mailing list.
+
+Once the patch is ready, the automatic tests can run.
+
+When a group of tests is finished, the author should be notified of any error,
+and a public report must be sent to
+[test-report@dpdk.org](//inbox.dpdk.org/test-report/).
+
+The reports are
+[formatted](//git.dpdk.org/tools/dpdk-ci/tree/tools/send-patch-report.sh)
+with a specific syntax for automatic parsing.
+When such report email is received, the bot ci@dpdk.org will
+[update patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh),
+referencing the report archive and incrementing one of the counters
+(success, warning or failure).
+
+A maintainer should wait the end of the tests (should be less than a day),
+and check the results before accepting a patch.
+
+### Mainline Monitoring
+
+The repository mirror on [GitHub](//github.com/DPDK/dpdk)
+is preferred for tests integration:
+
+- If polling for updates, better to load the GitHub mirror.
+- Some services like [Travis](//travis-ci.com/DPDK/)
+can be hooked in GitHub.
+
+### Release Validation
+
+When a new release is available, especially for release candidates,
+some validation reports are requested to the community members.
+
+Such release validations are sent as replies to the release announcement.
+
+
+## Tools
+---
+
+### Infrastructure
+
+The testing infrastructure scripts are shared in
+the [dpdk-ci git repository](//git.dpdk.org/tools/dpdk-ci/).
+
+The website dashboard and API of the community lab is a Django project shared
+in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-site/).
+
+The testing infrastructure work is coordinated in three complementary forums:
+
+- [dpdk-ci mailing list](//inbox.dpdk.org/ci/)
+- [bugzilla tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20scripts&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpriority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_format=advanced&order=priority&list_id=1987)
+- bi-weekly meeting
+
+### Known Test Frameworks and Tools
+
+- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/)
+- [DPDK apps](//git.dpdk.org/dpdk/tree/app/)
+- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/)
+- [VSPERF](//git.opnfv.org/vswitchperf/about/)
+- [OvS PVP perf](//github.com/chaudron/ovs_perf/)
+- [downstream projects](//www.dpdk.org/ecosystem/#projects)
diff --git a/static/img/community-lab-diagram.png b/static/img/community-lab-diagram.png
deleted file mode 100644
index a32109a..0000000
Binary files a/static/img/community-lab-diagram.png and /dev/null differ
-- 
2.25.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2020-03-28 21:46 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-10 22:11 [dpdk-ci] [PATCH] add general testing guidelines Thomas Monjalon
2020-03-16  9:43 ` [dpdk-ci] [dpdk-web] " Richardson, Bruce
2020-03-16 11:49   ` Thomas Monjalon
2020-03-16 12:17     ` Richardson, Bruce
2020-03-16 13:21       ` Thomas Monjalon
2020-03-24 11:31 ` [dpdk-ci] [PATCH v2] " Thomas Monjalon
2020-03-28 21:46   ` Thomas Monjalon

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).