From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <ci-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
	by inbox.dpdk.org (Postfix) with ESMTP id AE3BFA0559
	for <public@inbox.dpdk.org>; Mon, 16 Mar 2020 14:21:35 +0100 (CET)
Received: from [92.243.14.124] (localhost [127.0.0.1])
	by dpdk.org (Postfix) with ESMTP id 7645725D9;
	Mon, 16 Mar 2020 14:21:35 +0100 (CET)
Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com
 [66.111.4.28]) by dpdk.org (Postfix) with ESMTP id C1243FEB;
 Mon, 16 Mar 2020 14:21:33 +0100 (CET)
Received: from compute7.internal (compute7.nyi.internal [10.202.2.47])
 by mailout.nyi.internal (Postfix) with ESMTP id 1FF985C036D;
 Mon, 16 Mar 2020 09:21:33 -0400 (EDT)
Received: from mailfrontend2 ([10.202.2.163])
 by compute7.internal (MEProxy); Mon, 16 Mar 2020 09:21:33 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h=
 from:to:cc:subject:date:message-id:in-reply-to:references
 :mime-version:content-transfer-encoding:content-type; s=mesmtp;
 bh=ON8NUcvKTIT8FSsS0QzDTtdf7iCyKYhM7swQq4W6Veg=; b=OWRHN79nDr98
 3oFr9lJUUrf9oShC/L/Jyh5kKm+GxOOvKZHAPoZHrhsOTOi29xftuJIKgnlTrzGw
 TgsFfpusWiTlHSNCR1OC8PY/yiBicZFVlJIYbePBbP62ZcSbxy2vQueO2nD/xgcH
 /+TVGdZHPEcH8KM1p12wH4ruPvDvSwA=
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
 messagingengine.com; h=cc:content-transfer-encoding:content-type
 :date:from:in-reply-to:message-id:mime-version:references
 :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
 :x-sasl-enc; s=fm2; bh=ON8NUcvKTIT8FSsS0QzDTtdf7iCyKYhM7swQq4W6V
 eg=; b=daIEuHpxRNoTCa0sA4XdFV8RtQplZBmDaJzb0/+CqcmnJ90I20pd+hTo2
 9ita859tDIxVrMj3hd/5DUHgkVW73B+mCby1Ey/ABpOJrkZAyK7zNFRHTg7cDSXw
 gFh338pq4WSNxgDH+hZ/F0vOQXW73/i7sYQMFv5pQvmGfX/m7Y83moPvQnSpQvRe
 fxHZOhY+fRF4JJ1SYOzriRqlHUqwxiW1/SnTD+/UkTTqpUwt2qZtuh8Zjuwr/uJN
 rlLPj7Om1I9TmzK5arKndZin9i7p/fhLKFgDWxJAF4LGBJR2lRllGlrePTfPO4pA
 HWdA9N/BQqfCpr5YpIVqtJ3t1OMrw==
X-ME-Sender: <xms:XH1vXkP6SBX8TUb4VbS3K_juHlBrwgVy1DCpQlk_Y4CksWIoXhGTJg>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudeffedgheduucetufdoteggodetrfdotf
 fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
 uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne
 cujfgurhephffvufffkfgjfhgggfgtsehtqhertddttdejnecuhfhrohhmpefvhhhomhgr
 shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecuff
 homhgrihhnpeguphgukhdrohhrghdpuhhnhhdrvgguuhenucfkphepjeejrddufeegrddv
 tdefrddukeegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrh
 homhepthhhohhmrghssehmohhnjhgrlhhonhdrnhgvth
X-ME-Proxy: <xmx:XH1vXia4gCLmAXZgk-GPy0sj681FQAwsv-2UAiLXQRWOnkGC0QjMYw>
 <xmx:XH1vXoqUxkFsHIBA8aiC2d3jZ2DpJk4QTHGMjqUHFUDpM5L4YEL-Cg>
 <xmx:XH1vXma-A--6URi7uqopQsASkDvMelLoD-TUxoYPoNNKWfjBEUlPQw>
 <xmx:XX1vXmZIOv3YxrHCHP1BQSEcy6catdhDoyfyq54Y8Equ_nMchsg3Fg>
Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184])
 by mail.messagingengine.com (Postfix) with ESMTPA id 47C49306215A;
 Mon, 16 Mar 2020 09:21:32 -0400 (EDT)
From: Thomas Monjalon <thomas@monjalon.net>
To: "Richardson, Bruce" <bruce.richardson@intel.com>
Cc: "web@dpdk.org" <web@dpdk.org>, "ci@dpdk.org" <ci@dpdk.org>
Date: Mon, 16 Mar 2020 14:21:31 +0100
Message-ID: <3720266.e99z0qppnp@xps>
In-Reply-To: <59AF69C657FD0841A61C55336867B5B0975C978B@IRSMSX103.ger.corp.intel.com>
References: <20200310221130.46599-1-thomas@monjalon.net>
 <11237691.zAa99ISigo@xps>
 <59AF69C657FD0841A61C55336867B5B0975C978B@IRSMSX103.ger.corp.intel.com>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"
Subject: Re: [dpdk-ci] [dpdk-web] [PATCH] add general testing guidelines
X-BeenThere: ci@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK CI discussions <ci.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/ci>,
 <mailto:ci-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/ci/>
List-Post: <mailto:ci@dpdk.org>
List-Help: <mailto:ci-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/ci>,
 <mailto:ci-request@dpdk.org?subject=subscribe>
Errors-To: ci-bounces@dpdk.org
Sender: "ci" <ci-bounces@dpdk.org>

16/03/2020 13:17, Richardson, Bruce:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 16/03/2020 10:43, Richardson, Bruce:
> > > From: web <web-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> > > >
> > > > The community lab page is replaced with a more general page about
> > > > testing, where the community lab is part of the big picture. It
> > includes:
> > > > 	- testing goals
> > > > 	- CI workflow
> > > > 	- tools
> > >
> > > General question - is this meant to describe the ideal state or the
> > current state of testing?
> >=20
> > Yes it is about goals and workflow details, plus reference the tools wh=
ich
> > are used.
> > I think these three items give a big picture.
> >=20
> > > It starts off largely inspirational, but then ends with lots of
> > > details on the current setup. Do we want to have that mix in the one
> > page?
> >=20
> > I plan to add one more page to list what is currently done in which lab.
> > So the split is:
> > 	- one guideline page to setup a lab
> > 	- one status page
> >=20
> Do we anticipate others setting up public labs? I suppose it doesn't hurt=
 to
> document this even if no other labs are set up.

I cannot speak for others, but we can already document the community lab,
the Intel lab and the smaller free "labs", i.e. services, from Coverity,
Travis and OBS.
I will work on it as a second step.


> > I understand your question about splitting the guidelines in several pa=
ges
> > but I don't whether it would help reading, or the opposite, makes harder
> > to get all informations about what is a lab?
> >=20
> Ok, happy to keep it all in one place.
>=20
> > > I've also put some stylistic comments inline below too.
> > [...]
> > > > +There are various ways of contributing to the CI effort:
> > > > +
> > > > +- suggest improvement
> > > > +- write a new test
> > > > +- improve infrastucture scripts
> > >
> > > Typo: infrastRucture
> >=20
> > OK thanks
> >=20
> > > > +- run a lab
> > > > +
> > > > +A specific effort is done
> > > > +for the [community lab](//lab.dpdk.org/results/dashboard/about)
> > > > +hosted at [UNH-IOL](//www.iol.unh.edu).
> > > > +
> > >
> > > Not sure about the phrase "specific effort". How about "One specific =
lab
> > instance has been set up for the community lab..."
> >=20
> > What about this?
> > One specific lab instance, named community lab, is hosted at UNH-IOL.
> >=20
> > [...]
> > > > +Ideally, the CI infrastructure should have these properties:
> > > > +
> > > > +- reliable
> > > > +- neutral
> > >
> > > Neutral in what way? Presumably vendor or HW neutral?
> >=20
> > Neutral is a property which can apply to vendors, HW or SW (OS).
> > Do you think "vendor neutral" is better?
> >=20
> We just need to clarify what type(s) of neutrality is in mind. Vendor neu=
tral is probably what people most look for, so I suggest changing it to tha=
t - it implies HW neutral I think.

OK

> > > > +- transparent visibility
> > >
> > > What is implied by this? It is "clear methodology" or "transparent
> > testing methods"?
> >=20
> > I mean we should be able to see which tests are run, what is the
> > environment, how tests are configured and what is the result.
> > The property to measure is really the visibility.
> >=20
> I think just "transparency" is probably the best title then.

OK for transparency.

> However, to clarify things, I think it might be good to have a one-line e=
xplanation of each bullet point to allow additional clarity. For example:
>=20
> - reliable: users need to know that any failures are genuine, and not the=
 result of intermittent issues in the test infrastructure
> - neutral: the lab should not focus on one HW or SW platform to the exclu=
sion or detriment of others
> - transparent: the details of what tests have been run and the result of =
those tests should be visible to all users.
>=20
> > > > +- coverage summary
> > > > +- trending graphs
> > > > +- reports to author and
> > > > +[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
> > > > +- easy to add a test
> > > > +- easy to reproduce a test locally
> > > > +
> > >
> > > The first three are what I'd consider properties, but the rest are
> > > more functions that should be included in the CI, rather than propert=
ies
> > of it.
> >=20
> > I don't agree:
> > 	- coverage apply to all tests: what is the coverage of each test?
> > 	- trending graphs can be done for any test or measure
> > 	- reporting must be done for all tests
> > 	- ability to add test is an infrastructure property
> > 	- ability to reproduce is a test property
> >=20
> Yes, I agree they are needed, they just don't fit in the same list as rel=
iability, transparency and neutrality, which are abstract concepts vs cover=
age reports, trend graphs etc. which are concrete outputs. Therefore I sugg=
est they be put in a separate list on the page, not dropped altogether.

Transparency is also achieved through some outputs.
And the last two ones (add and reproduce) are not outputs.

I understand your concern but I don't know how to separate lists.


> > [...]
> > > > +The tests should cover these categories:
> > > > +
> > > > +- compilation
> > > > +- static analysis
> > > > +- functional
> > > > +- performance
> > > > +- interoperability
> > > > +- portability
> > > > +- compatibility
> > > > +
> > >
> > > While compilation and static analysis are ok, the rest need "testing"
> > after them since they are not nouns. Alternatively drop "static analysi=
s"
> > from the list and then change compilation to "compile" to use only
> > adjectives.
> >=20
> > I don't want to drop static analysis, I think it is a test category.
> > We could add "testing" after each item, but I feel it is just an overhe=
ad.
> > I don't know what to do here.
>=20
> Maybe split it into two levels of bullets:
>=20
> - compilation
> - static analysis
> - test case runs, including
>   - functional tests
>   - performance tests
>   - interoperability tests
>   - ....
>=20
> If that doesn't work for you, it's ok to leave as-is. Everyone will under=
stand it, it just reads a little weird to me. =F0=9F=98=8A

OK for the nested bullet for runtime tests.

> > [...]
> > > > +### Patch Testing
> > > > +
> > > > +When a patch is submitted, it is received via the mailing list
> > > > +[dpdk-dev](//inbox.dpdk.org/dev/).
> > > > +For simple tests,
> > > > +like
> > > > +[checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh)
> > > > +, it can be run directly on email receiving.
> > >
> > > s/email receiving/the received email/
> >=20
> > I wanted to emphasize on the trigger which "receiving" instead of
> > "polling".
> > But "on the received email" is also OK. Is it a better English?
> >=20
> Ah, ok, I understand a little better. To emphasise the triggering rather =
than what the scan runs on, I suggest "on email reception", or change "run"=
 to "triggered" as in - "triggered on receipt of the patch email, and run o=
n the email itself."

OK thanks.
Note: I tried to understand the difference between receiving, reception and=
 receipt before sending this patch.
Looks like I failed to pick the best English (or Irish) form ;)

Thanks for the good review.