From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3A49FA057B for ; Tue, 24 Mar 2020 12:31:45 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D871A1C0DC; Tue, 24 Mar 2020 12:31:44 +0100 (CET) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by dpdk.org (Postfix) with ESMTP id 29F421C0D6; Tue, 24 Mar 2020 12:31:44 +0100 (CET) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id C27E25C0266; Tue, 24 Mar 2020 07:31:41 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Tue, 24 Mar 2020 07:31:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=mesmtp; bh=z4ySqom73c XQhGPw1KUZea8uWfMXSGH12+F6n/U4Jz8=; b=PK9s3adCf5jazH194vQAX/5/Ne +Nxc7UmrNO/EUGsxPv1FULMaOaArHby0kjcuJ/bOux7qB7X2QAn52MdnxVNpAQjl iiY3fDiNcwXGaXTxFw11USoR1geIQlBgO/luMspD3Apz9rfrMlCE0ta8B9pMHLNK ZL9VVNtY3daHJC0kA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=z4ySqom73cXQhGPw1KUZea8uWfMXSGH12+F6n/U4Jz8=; b=1bSpLYgr hH7qF0cfUBc12skOayvs9eHh29n2UC0gwkh9CpGoVR8hhCKg9pb93+zHlMgrY07Y Jg2y9VeXNSQduB8sH6iQ0myAWZ6ED6Fd+uXARIImDw0FQ6o2bovshvY0sVToc7Gu GgvCL9lgZu3b/1mQKuD6aD7w6ErWlOPVaCHjgz9Vg4T0PKgVD3GC4OWbyS5MYP6e 0inznFCrsgV81TTqCt7joAlzb6rDP5sAdyG66tdvTcvaDiFoHwL8ptJwv0vQJT6Z jVD8rAUrMOIISqQEMfgVk71GHYT7TIwTjuPjRFyixmUz0e1guRJG/fv8LXsVTFBk YRGn4218TnPuEg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudehudcutefuodetggdotefrodftvfcurf hrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecunecujfgurhephffvufffkffojghfggfgsedtkeertdertd dtnecuhfhrohhmpefvhhhomhgrshcuofhonhhjrghlohhnuceothhhohhmrghssehmohhn jhgrlhhonhdrnhgvtheqnecuffhomhgrihhnpeguphgukhdrohhrghdpuhhnhhdrvgguuh dpghhithhhuhgsrdgtohhmpdhtrhgrvhhishdqtghirdgtohhmpdhophhnfhhvrdhorhhg necukfhppeejjedrudefgedrvddtfedrudekgeenucevlhhushhtvghrufhiiigvpedtne curfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Received: from xps.monjalon.net (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id D5AE3328005E; Tue, 24 Mar 2020 07:31:40 -0400 (EDT) From: Thomas Monjalon To: web@dpdk.org Cc: ci@dpdk.org, bruce.richardson@intel.com Date: Tue, 24 Mar 2020 12:31:29 +0100 Message-Id: <20200324113129.182525-1-thomas@monjalon.net> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200310221130.46599-1-thomas@monjalon.net> References: <20200310221130.46599-1-thomas@monjalon.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-web] [PATCH v2] add general testing guidelines X-BeenThere: web@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK website maintenance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: web-bounces@dpdk.org Sender: "web" The community lab page is replaced with a more general page about testing, where the community lab is part of the big picture. It includes: - testing goals - CI workflow - tools Signed-off-by: Thomas Monjalon --- v2: - fix typo - reword community lab - explain CI properties - group test categories - reword email workflow - remove id from bugzilla request --- content/lab/_index.md | 21 ---- content/testing/_index.md | 144 +++++++++++++++++++++++++++ static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes 3 files changed, 144 insertions(+), 21 deletions(-) delete mode 100644 content/lab/_index.md create mode 100644 content/testing/_index.md delete mode 100644 static/img/community-lab-diagram.png diff --git a/content/lab/_index.md b/content/lab/_index.md deleted file mode 100644 index 913c86f..0000000 --- a/content/lab/_index.md +++ /dev/null @@ -1,21 +0,0 @@ -+++ -title = "Community Lab" -weight = "8" -+++ - -{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{< /button >}} - -## What it's for ----- - -- Automated performance regression testing of new patches. - - When a new patch is received, automatically test - and determine if it degrades performance or causes other issues - on one or more architectures. - - Provides input to maintainers on whether to accept patches or request rework. -- The Dashboard currently provides performance deltas for each patch - compared to the expected numbers of each architecture. -- The scope may expand in future: - test DPDK-enabled applications, use lab for demos, etc. - -![DPDK Community Lab](/img/community-lab-diagram.png) diff --git a/content/testing/_index.md b/content/testing/_index.md new file mode 100644 index 0000000..1648f45 --- /dev/null +++ b/content/testing/_index.md @@ -0,0 +1,144 @@ ++++ +title = "Testing" +weight = "8" ++++ + +As a community project, DPDK CI is distributed and welcome any contribution. + +There are various ways of contributing to the CI effort: + +- suggest improvement +- write a new test +- improve infrastructure scripts +- run a lab + +One specific lab instance, +named [community lab](//lab.dpdk.org/results/dashboard/about), +is hosted at [UNH-IOL](//www.iol.unh.edu). + + +## CI Testing Goals +--- + +### General Properties + +Ideally, the CI infrastructure should have these properties: + +- reliable: always run, without false positive +- vendor neutral: run same tests and configurations on all HW/SW platforms +- transparent: show environment, configuration, source code and result +- coverage summary +- trending graphs +- reports to author and +[test-report@dpdk.org](//inbox.dpdk.org/test-report/) +- easy to add a test +- easy to reproduce a test locally + +### Inputs + +There are three kinds of inputs to be tested: + +- per-patch testing integrated +with [patchwork](//patches.dpdk.org/project/dpdk/list/) +- per-commit testing after merges +in mainline or dpdk-next- [repositories](//git.dpdk.org/) +- per-release testing +including [release candidates](//git.dpdk.org/dpdk/refs/) +of all [branches](//git.dpdk.org/dpdk-stable/refs/) + +### Categories + +The tests should cover these categories: + +* source code checks: + - static analysis + - compilation +* runtime tests: + - functional + - performance + - interoperability + - portability + - compatibility + + +## Workflow +--- + +### Patch Testing + +When a patch is submitted, it is received +via the mailing list [dpdk-dev](//inbox.dpdk.org/dev/). +For simple tests, +like [checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh), +it can be run on email reception. + +For most tests, the patch must be applied +on the [right tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py) +after its series dependencies. +Dealing with series and dependencies is easier +via [patchwork database request](//patches.dpdk.org/api/). +That's why the common workflow is +to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw). +The patches are saved and organized +in [patchwork](//patches.dpdk.org/project/dpdk/list/) +after being distributed by the mailing list. + +Once the patch is ready, the automatic tests can run. + +When a group of tests is finished, the author should be notified of any error, +and a public report must be sent to +[test-report@dpdk.org](//inbox.dpdk.org/test-report/). + +The reports are +[formatted](//git.dpdk.org/tools/dpdk-ci/tree/tools/send-patch-report.sh) +with a specific syntax for automatic parsing. +When such report email is received, the bot ci@dpdk.org will +[update patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh), +referencing the report archive and incrementing one of the counters +(success, warning or failure). + +A maintainer should wait the end of the tests (should be less than a day), +and check the results before accepting a patch. + +### Mainline Monitoring + +The repository mirror on [GitHub](//github.com/DPDK/dpdk) +is preferred for tests integration: + +- If polling for updates, better to load the GitHub mirror. +- Some services like [Travis](//travis-ci.com/DPDK/) +can be hooked in GitHub. + +### Release Validation + +When a new release is available, especially for release candidates, +some validation reports are requested to the community members. + +Such release validations are sent as replies to the release announcement. + + +## Tools +--- + +### Infrastructure + +The testing infrastructure scripts are shared in +the [dpdk-ci git repository](//git.dpdk.org/tools/dpdk-ci/). + +The website dashboard and API of the community lab is a Django project shared +in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-site/). + +The testing infrastructure work is coordinated in three complementary forums: + +- [dpdk-ci mailing list](//inbox.dpdk.org/ci/) +- [bugzilla tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20scripts&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpriority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_format=advanced&order=priority) +- bi-weekly meeting + +### Known Test Frameworks and Tools + +- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/) +- [DPDK apps](//git.dpdk.org/dpdk/tree/app/) +- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/) +- [VSPERF](//git.opnfv.org/vswitchperf/about/) +- [OvS PVP perf](//github.com/chaudron/ovs_perf/) +- [downstream projects](//www.dpdk.org/ecosystem/#projects) diff --git a/static/img/community-lab-diagram.png b/static/img/community-lab-diagram.png deleted file mode 100644 index a32109a..0000000 Binary files a/static/img/community-lab-diagram.png and /dev/null differ -- 2.25.1