From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 20681A0566 for ; Tue, 10 Mar 2020 23:11:59 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E34E41C02D; Tue, 10 Mar 2020 23:11:58 +0100 (CET) Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) by dpdk.org (Postfix) with ESMTP id 4340D1C02C; Tue, 10 Mar 2020 23:11:57 +0100 (CET) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id DB87656E; Tue, 10 Mar 2020 18:11:55 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Tue, 10 Mar 2020 18:11:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; s=mesmtp; bh=jjnDl1CeFSjiBEvFzurbDUl DUgJ6B0wUb8I7Oa+sFmc=; b=B296EWiRnXGpHoCkfCnsHJIijoEOxZUB6d6uRMf EP1ZKPs2/OyBL2tS7WtKyFdrdnNvebpXXw/PLSWk6Oek/90qTQPoyO60gX1UYWWz q5gbivLAd5/GFwshGtSF6uBi8S3GOcl/F3LrixhocM6cqfrt2IQo4wEMsAOKGsMT XVQQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=jjnDl1CeFSjiBEvFz urbDUlDUgJ6B0wUb8I7Oa+sFmc=; b=KUdz28CvNhFtIyqt46UmjNBZ3UFY13u+3 UOsqgHf+GIT3YkeSlBzNReSAfJALEMnwCqLiX6+moJKp+sIZUZwiKMvV91FH3A2K i23sd4PPSiDDx1IVrriSyaNlf6cbxtBChnxLxwE3K/XRwg3yB7/jAZVujagcw9VO 2dAWQipEvizSeXbBbqDLjrsHzzvU3abifJRptgsECAOx/V/c64VZjbiHRAHlYB2i tl3v8sL/zt8bOMEj80oIjjljOJXZSzpEMs8uY+E/MZK8+H3X8e3pwt1stikI7x0f eFAuKWeIf0LNwj5aFvavGLT3zieiwPaUXwSZPp6pO1WByBm3WP+hQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedruddvtddgudehtdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhephffvufffkffoggfgsedtkeertd ertddtnecuhfhrohhmpefvhhhomhgrshcuofhonhhjrghlohhnuceothhhohhmrghssehm ohhnjhgrlhhonhdrnhgvtheqnecuffhomhgrihhnpeguphgukhdrohhrghdpuhhnhhdrvg guuhdpghhithhhuhgsrdgtohhmpdhtrhgrvhhishdqtghirdgtohhmpdhophhnfhhvrdho rhhgnecukfhppeejjedrudefgedrvddtfedrudekgeenucevlhhushhtvghrufhiiigvpe dunecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhn vght X-ME-Proxy: Received: from xps.monjalon.net (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 9DE653280060; Tue, 10 Mar 2020 18:11:53 -0400 (EDT) From: Thomas Monjalon To: web@dpdk.org Cc: ci@dpdk.org Date: Tue, 10 Mar 2020 23:11:30 +0100 Message-Id: <20200310221130.46599-1-thomas@monjalon.net> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-ci] [PATCH] add general testing guidelines X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" The community lab page is replaced with a more general page about testing, where the community lab is part of the big picture. It includes: - testing goals - CI workflow - tools Signed-off-by: Thomas Monjalon --- content/lab/_index.md | 21 ---- content/testing/_index.md | 142 +++++++++++++++++++++++++++ static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes 3 files changed, 142 insertions(+), 21 deletions(-) delete mode 100644 content/lab/_index.md create mode 100644 content/testing/_index.md delete mode 100644 static/img/community-lab-diagram.png diff --git a/content/lab/_index.md b/content/lab/_index.md deleted file mode 100644 index 913c86f..0000000 --- a/content/lab/_index.md +++ /dev/null @@ -1,21 +0,0 @@ -+++ -title = "Community Lab" -weight = "8" -+++ - -{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{< /button >}} - -## What it's for ----- - -- Automated performance regression testing of new patches. - - When a new patch is received, automatically test - and determine if it degrades performance or causes other issues - on one or more architectures. - - Provides input to maintainers on whether to accept patches or request rework. -- The Dashboard currently provides performance deltas for each patch - compared to the expected numbers of each architecture. -- The scope may expand in future: - test DPDK-enabled applications, use lab for demos, etc. - -![DPDK Community Lab](/img/community-lab-diagram.png) diff --git a/content/testing/_index.md b/content/testing/_index.md new file mode 100644 index 0000000..30da214 --- /dev/null +++ b/content/testing/_index.md @@ -0,0 +1,142 @@ ++++ +title = "Testing" +weight = "8" ++++ + +As a community project, DPDK CI is distributed and welcome any contribution. + +There are various ways of contributing to the CI effort: + +- suggest improvement +- write a new test +- improve infrastucture scripts +- run a lab + +A specific effort is done +for the [community lab](//lab.dpdk.org/results/dashboard/about) +hosted at [UNH-IOL](//www.iol.unh.edu). + + +## CI Testing Goals +--- + +### General Properties + +Ideally, the CI infrastructure should have these properties: + +- reliable +- neutral +- transparent visibility +- coverage summary +- trending graphs +- reports to author and +[test-report@dpdk.org](//inbox.dpdk.org/test-report/) +- easy to add a test +- easy to reproduce a test locally + +### Inputs + +There are three kinds of inputs to be tested: + +- per-patch testing integrated +with [patchwork](//patches.dpdk.org/project/dpdk/list/) +- per-commit testing after merges +in mainline or dpdk-next- [repositories](//git.dpdk.org/) +- per-release testing +including [release candidates](//git.dpdk.org/dpdk/refs/) +of all [branches](//git.dpdk.org/dpdk-stable/refs/) + +### Categories + +The tests should cover these categories: + +- compilation +- static analysis +- functional +- performance +- interoperability +- portability +- compatibility + + +## Workflow +--- + +### Patch Testing + +When a patch is submitted, it is received +via the mailing list [dpdk-dev](//inbox.dpdk.org/dev/). +For simple tests, +like [checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh), +it can be run directly on email receiving. + +For most tests, the patch must be applied +on the [right tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py) +after its series dependencies. +Dealing with series and dependencies is easier +via [patchwork database request](//patches.dpdk.org/api/). +That's why the common workflow is +to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw). +The patches are saved and organized +in [patchwork](//patches.dpdk.org/project/dpdk/list/) +after being distributed by the mailing list. + +Once the patch is ready, the automatic tests can run. + +When a group of tests is finished, the author should be notified of any error, +and a public report must be sent to +[test-report@dpdk.org](//inbox.dpdk.org/test-report/). + +The reports are +[formatted](//git.dpdk.org/tools/dpdk-ci/tree/tools/send-patch-report.sh) +with a specific syntax for automatic parsing. +When such report email is received, the bot ci@dpdk.org will +[update patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh), +referencing the report archive and incrementing one of the counters +(success, warning or failure). + +A maintainer should wait the end of the tests (should be less than a day), +and check the results before accepting a patch. + +### Mainline Monitoring + +The repository mirror on [GitHub](//github.com/DPDK/dpdk) +is preferred for tests integration: + +- If polling for updates, better to load the GitHub mirror. +- Some services like [Travis](//travis-ci.com/DPDK/) +can be hooked in GitHub. + +### Release Validation + +When a new release is available, especially for release candidates, +some validation reports are requested to the community members. + +Such release validations are sent as replies to the release announcement. + + +## Tools +--- + +### Infrastructure + +The testing infrastructure scripts are shared in +the [dpdk-ci git repository](//git.dpdk.org/tools/dpdk-ci/). + +The website dashboard and API of the community lab is a Django project shared +in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-site/). + +The testing infrastructure work is coordinated in three complementary forums: + +- [dpdk-ci mailing list](//inbox.dpdk.org/ci/) +- [bugzilla tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20scripts&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpriority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_format=advanced&order=priority&list_id=1987) +- bi-weekly meeting + +### Known Test Frameworks and Tools + +- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/) +- [DPDK apps](//git.dpdk.org/dpdk/tree/app/) +- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/) +- [VSPERF](//git.opnfv.org/vswitchperf/about/) +- [OvS PVP perf](//github.com/chaudron/ovs_perf/) +- [downstream projects](//www.dpdk.org/ecosystem/#projects) diff --git a/static/img/community-lab-diagram.png b/static/img/community-lab-diagram.png deleted file mode 100644 index a32109a..0000000 Binary files a/static/img/community-lab-diagram.png and /dev/null differ -- 2.25.1