From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A13E2A0563 for ; Mon, 16 Mar 2020 10:43:12 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2FA271C07C; Mon, 16 Mar 2020 10:43:12 +0100 (CET) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by dpdk.org (Postfix) with ESMTP id E70151C02A; Mon, 16 Mar 2020 10:43:09 +0100 (CET) IronPort-SDR: /1CoVyvn5vv1h1agI2f3g4ugWzcKFHOZa6ZLNT1I9NnUQgUYqNS9dEC1455xUBegZKDac7lnk0 4jOqIeo44rBQ== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2020 02:43:07 -0700 IronPort-SDR: 2uJOAdAh/+YWSxBb0yzsmp5MWMZVqOKRfW/Ige00dF2gn+Ro1dz/bY/SQ1qO/8PewVLS18sJ2+ YZs1kRYCwCBA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,559,1574150400"; d="scan'208";a="237907067" Received: from irsmsx152.ger.corp.intel.com ([163.33.192.66]) by orsmga008.jf.intel.com with ESMTP; 16 Mar 2020 02:43:06 -0700 Received: from irsmsx103.ger.corp.intel.com ([169.254.3.138]) by IRSMSX152.ger.corp.intel.com ([169.254.6.25]) with mapi id 14.03.0439.000; Mon, 16 Mar 2020 09:43:05 +0000 From: "Richardson, Bruce" To: Thomas Monjalon , "web@dpdk.org" CC: "ci@dpdk.org" Thread-Topic: [dpdk-web] [PATCH] add general testing guidelines Thread-Index: AQHV9yjzkY7mbILj/0efL6L58oQFFqhK/YzQ Date: Mon, 16 Mar 2020 09:43:04 +0000 Message-ID: <59AF69C657FD0841A61C55336867B5B0975C93E4@IRSMSX103.ger.corp.intel.com> References: <20200310221130.46599-1-thomas@monjalon.net> In-Reply-To: <20200310221130.46599-1-thomas@monjalon.net> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-web] [PATCH] add general testing guidelines X-BeenThere: web@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK website maintenance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: web-bounces@dpdk.org Sender: "web" > -----Original Message----- > From: web On Behalf Of Thomas Monjalon > Sent: Tuesday, March 10, 2020 10:12 PM > To: web@dpdk.org > Cc: ci@dpdk.org > Subject: [dpdk-web] [PATCH] add general testing guidelines >=20 > The community lab page is replaced with a more general page about testing= , > where the community lab is part of the big picture. It includes: > - testing goals > - CI workflow > - tools >=20 > Signed-off-by: Thomas Monjalon > --- General question - is this meant to describe the ideal state or the current= state of testing? It starts off largely inspirational, but then ends with lots of details on = the current setup. Do we want to have that mix in the one page? I've also put some stylistic comments inline below too. /Bruce > content/lab/_index.md | 21 ---- > content/testing/_index.md | 142 +++++++++++++++++++++++++++ > static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes > 3 files changed, 142 insertions(+), 21 deletions(-) delete mode 100644 > content/lab/_index.md create mode 100644 content/testing/_index.md > delete mode 100644 static/img/community-lab-diagram.png >=20 > diff --git a/content/lab/_index.md b/content/lab/_index.md deleted file > mode 100644 index 913c86f..0000000 > --- a/content/lab/_index.md > +++ /dev/null > @@ -1,21 +0,0 @@ > -+++ > -title =3D "Community Lab" > -weight =3D "8" > -+++ > - > -{{< button align=3D"center" href=3D"https://lab.dpdk.org" >}} Dashboard = {{< > /button >}} > - > -## What it's for > ----- > - > -- Automated performance regression testing of new patches. > - - When a new patch is received, automatically test > - and determine if it degrades performance or causes other issues > - on one or more architectures. > - - Provides input to maintainers on whether to accept patches or reques= t > rework. > -- The Dashboard currently provides performance deltas for each patch > - compared to the expected numbers of each architecture. > -- The scope may expand in future: > - test DPDK-enabled applications, use lab for demos, etc. > - > -![DPDK Community Lab](/img/community-lab-diagram.png) > diff --git a/content/testing/_index.md b/content/testing/_index.md new > file mode 100644 index 0000000..30da214 > --- /dev/null > +++ b/content/testing/_index.md > @@ -0,0 +1,142 @@ > ++++ > +title =3D "Testing" > +weight =3D "8" > ++++ > + > +As a community project, DPDK CI is distributed and welcome any > contribution. > + > +There are various ways of contributing to the CI effort: > + > +- suggest improvement > +- write a new test > +- improve infrastucture scripts Typo: infrastRucture > +- run a lab > + > +A specific effort is done > +for the [community lab](//lab.dpdk.org/results/dashboard/about) > +hosted at [UNH-IOL](//www.iol.unh.edu). > + Not sure about the phrase "specific effort". How about "One specific lab in= stance has been set up for the community lab..." > + > +## CI Testing Goals > +--- > + > +### General Properties > + > +Ideally, the CI infrastructure should have these properties: > + > +- reliable > +- neutral Neutral in what way? Presumably vendor or HW neutral? > +- transparent visibility What is implied by this? It is "clear methodology" or "transparent testing = methods"? > +- coverage summary > +- trending graphs > +- reports to author and > +[test-report@dpdk.org](//inbox.dpdk.org/test-report/) > +- easy to add a test > +- easy to reproduce a test locally > + The first three are what I'd consider properties, but the rest are more fun= ctions that should be included in the CI, rather than properties of it. > +### Inputs > + > +There are three kinds of inputs to be tested: > + > +- per-patch testing integrated > +with [patchwork](//patches.dpdk.org/project/dpdk/list/) > +- per-commit testing after merges > +in mainline or dpdk-next- [repositories](//git.dpdk.org/) > +- per-release testing > +including [release candidates](//git.dpdk.org/dpdk/refs/) > +of all [branches](//git.dpdk.org/dpdk-stable/refs/) > + > +### Categories > + > +The tests should cover these categories: > + > +- compilation > +- static analysis > +- functional > +- performance > +- interoperability > +- portability > +- compatibility > + While compilation and static analysis are ok, the rest need "testing" after= them since they are not nouns. Alternatively drop "static analysis" from t= he list and then change compilation to "compile" to use only adjectives. > + > +## Workflow > +--- > + > +### Patch Testing > + > +When a patch is submitted, it is received via the mailing list > +[dpdk-dev](//inbox.dpdk.org/dev/). > +For simple tests, > +like > +[checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh), > +it can be run directly on email receiving. s/email receiving/the received email/ > + > +For most tests, the patch must be applied on the [right > +tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py) > +after its series dependencies. > +Dealing with series and dependencies is easier via [patchwork database > +request](//patches.dpdk.org/api/). > +That's why the common workflow is > +to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw). > +The patches are saved and organized > +in [patchwork](//patches.dpdk.org/project/dpdk/list/) > +after being distributed by the mailing list. > + > +Once the patch is ready, the automatic tests can run. > + > +When a group of tests is finished, the author should be notified of any > +error, and a public report must be sent to > +[test-report@dpdk.org](//inbox.dpdk.org/test-report/). > + > +The reports are > +[formatted](//git.dpdk.org/tools/dpdk-ci/tree/tools/send-patch-report.s > +h) with a specific syntax for automatic parsing. > +When such report email is received, the bot ci@dpdk.org will [update > +patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh), > +referencing the report archive and incrementing one of the counters > +(success, warning or failure). > + > +A maintainer should wait the end of the tests (should be less than a > +day), and check the results before accepting a patch. > + > +### Mainline Monitoring > + > +The repository mirror on [GitHub](//github.com/DPDK/dpdk) is preferred > +for tests integration: > + > +- If polling for updates, better to load the GitHub mirror. > +- Some services like [Travis](//travis-ci.com/DPDK/) can be hooked in > +GitHub. > + > +### Release Validation > + > +When a new release is available, especially for release candidates, > +some validation reports are requested to the community members. > + > +Such release validations are sent as replies to the release announcement= . > + > + > +## Tools > +--- > + > +### Infrastructure > + > +The testing infrastructure scripts are shared in the [dpdk-ci git > +repository](//git.dpdk.org/tools/dpdk-ci/). > + > +The website dashboard and API of the community lab is a Django project > +shared in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci= - > site/). > + > +The testing infrastructure work is coordinated in three complementary > forums: > + > +- [dpdk-ci mailing list](//inbox.dpdk.org/ci/) > +- [bugzilla > +tracker](//bugs.dpdk.org/buglist.cgi?product=3Dlab&component=3Djob%20scr= ipt > +s&component=3DUNH%20infra&resolution=3D---&bug_status=3DUNCONFIRMED&bug_= statu > +s=3DCONFIRMED&bug_status=3DIN_PROGRESS&columnlist=3Dproduct%2Ccomponent%= 2Cpri > +ority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_forma > +t=3Dadvanced&order=3Dpriority&list_id=3D1987) > +- bi-weekly meeting > + > +### Known Test Frameworks and Tools > + > +- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/) > +- [DPDK apps](//git.dpdk.org/dpdk/tree/app/) > +- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/) > +- [VSPERF](//git.opnfv.org/vswitchperf/about/) > +- [OvS PVP perf](//github.com/chaudron/ovs_perf/) > +- [downstream projects](//www.dpdk.org/ecosystem/#projects) > diff --git a/static/img/community-lab-diagram.png b/static/img/community- > lab-diagram.png > deleted file mode 100644 > index a32109a..0000000 > Binary files a/static/img/community-lab-diagram.png and /dev/null differ > -- > 2.25.1