From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <ci-bounces@dpdk.org> Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id AE3BFA0559 for <public@inbox.dpdk.org>; Mon, 16 Mar 2020 14:21:35 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7645725D9; Mon, 16 Mar 2020 14:21:35 +0100 (CET) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by dpdk.org (Postfix) with ESMTP id C1243FEB; Mon, 16 Mar 2020 14:21:33 +0100 (CET) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 1FF985C036D; Mon, 16 Mar 2020 09:21:33 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Mon, 16 Mar 2020 09:21:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=mesmtp; bh=ON8NUcvKTIT8FSsS0QzDTtdf7iCyKYhM7swQq4W6Veg=; b=OWRHN79nDr98 3oFr9lJUUrf9oShC/L/Jyh5kKm+GxOOvKZHAPoZHrhsOTOi29xftuJIKgnlTrzGw TgsFfpusWiTlHSNCR1OC8PY/yiBicZFVlJIYbePBbP62ZcSbxy2vQueO2nD/xgcH /+TVGdZHPEcH8KM1p12wH4ruPvDvSwA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=ON8NUcvKTIT8FSsS0QzDTtdf7iCyKYhM7swQq4W6V eg=; b=daIEuHpxRNoTCa0sA4XdFV8RtQplZBmDaJzb0/+CqcmnJ90I20pd+hTo2 9ita859tDIxVrMj3hd/5DUHgkVW73B+mCby1Ey/ABpOJrkZAyK7zNFRHTg7cDSXw gFh338pq4WSNxgDH+hZ/F0vOQXW73/i7sYQMFv5pQvmGfX/m7Y83moPvQnSpQvRe fxHZOhY+fRF4JJ1SYOzriRqlHUqwxiW1/SnTD+/UkTTqpUwt2qZtuh8Zjuwr/uJN rlLPj7Om1I9TmzK5arKndZin9i7p/fhLKFgDWxJAF4LGBJR2lRllGlrePTfPO4pA HWdA9N/BQqfCpr5YpIVqtJ3t1OMrw== X-ME-Sender: <xms:XH1vXkP6SBX8TUb4VbS3K_juHlBrwgVy1DCpQlk_Y4CksWIoXhGTJg> X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudeffedgheduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfgjfhgggfgtsehtqhertddttdejnecuhfhrohhmpefvhhhomhgr shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecuff homhgrihhnpeguphgukhdrohhrghdpuhhnhhdrvgguuhenucfkphepjeejrddufeegrddv tdefrddukeegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrh homhepthhhohhmrghssehmohhnjhgrlhhonhdrnhgvth X-ME-Proxy: <xmx:XH1vXia4gCLmAXZgk-GPy0sj681FQAwsv-2UAiLXQRWOnkGC0QjMYw> <xmx:XH1vXoqUxkFsHIBA8aiC2d3jZ2DpJk4QTHGMjqUHFUDpM5L4YEL-Cg> <xmx:XH1vXma-A--6URi7uqopQsASkDvMelLoD-TUxoYPoNNKWfjBEUlPQw> <xmx:XX1vXmZIOv3YxrHCHP1BQSEcy6catdhDoyfyq54Y8Equ_nMchsg3Fg> Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 47C49306215A; Mon, 16 Mar 2020 09:21:32 -0400 (EDT) From: Thomas Monjalon <thomas@monjalon.net> To: "Richardson, Bruce" <bruce.richardson@intel.com> Cc: "web@dpdk.org" <web@dpdk.org>, "ci@dpdk.org" <ci@dpdk.org> Date: Mon, 16 Mar 2020 14:21:31 +0100 Message-ID: <3720266.e99z0qppnp@xps> In-Reply-To: <59AF69C657FD0841A61C55336867B5B0975C978B@IRSMSX103.ger.corp.intel.com> References: <20200310221130.46599-1-thomas@monjalon.net> <11237691.zAa99ISigo@xps> <59AF69C657FD0841A61C55336867B5B0975C978B@IRSMSX103.ger.corp.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-ci] [dpdk-web] [PATCH] add general testing guidelines X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions <ci.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/ci>, <mailto:ci-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/ci/> List-Post: <mailto:ci@dpdk.org> List-Help: <mailto:ci-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/ci>, <mailto:ci-request@dpdk.org?subject=subscribe> Errors-To: ci-bounces@dpdk.org Sender: "ci" <ci-bounces@dpdk.org> 16/03/2020 13:17, Richardson, Bruce: > From: Thomas Monjalon <thomas@monjalon.net> > > 16/03/2020 10:43, Richardson, Bruce: > > > From: web <web-bounces@dpdk.org> On Behalf Of Thomas Monjalon > > > > > > > > The community lab page is replaced with a more general page about > > > > testing, where the community lab is part of the big picture. It > > includes: > > > > - testing goals > > > > - CI workflow > > > > - tools > > > > > > General question - is this meant to describe the ideal state or the > > current state of testing? > >=20 > > Yes it is about goals and workflow details, plus reference the tools wh= ich > > are used. > > I think these three items give a big picture. > >=20 > > > It starts off largely inspirational, but then ends with lots of > > > details on the current setup. Do we want to have that mix in the one > > page? > >=20 > > I plan to add one more page to list what is currently done in which lab. > > So the split is: > > - one guideline page to setup a lab > > - one status page > >=20 > Do we anticipate others setting up public labs? I suppose it doesn't hurt= to > document this even if no other labs are set up. I cannot speak for others, but we can already document the community lab, the Intel lab and the smaller free "labs", i.e. services, from Coverity, Travis and OBS. I will work on it as a second step. > > I understand your question about splitting the guidelines in several pa= ges > > but I don't whether it would help reading, or the opposite, makes harder > > to get all informations about what is a lab? > >=20 > Ok, happy to keep it all in one place. >=20 > > > I've also put some stylistic comments inline below too. > > [...] > > > > +There are various ways of contributing to the CI effort: > > > > + > > > > +- suggest improvement > > > > +- write a new test > > > > +- improve infrastucture scripts > > > > > > Typo: infrastRucture > >=20 > > OK thanks > >=20 > > > > +- run a lab > > > > + > > > > +A specific effort is done > > > > +for the [community lab](//lab.dpdk.org/results/dashboard/about) > > > > +hosted at [UNH-IOL](//www.iol.unh.edu). > > > > + > > > > > > Not sure about the phrase "specific effort". How about "One specific = lab > > instance has been set up for the community lab..." > >=20 > > What about this? > > One specific lab instance, named community lab, is hosted at UNH-IOL. > >=20 > > [...] > > > > +Ideally, the CI infrastructure should have these properties: > > > > + > > > > +- reliable > > > > +- neutral > > > > > > Neutral in what way? Presumably vendor or HW neutral? > >=20 > > Neutral is a property which can apply to vendors, HW or SW (OS). > > Do you think "vendor neutral" is better? > >=20 > We just need to clarify what type(s) of neutrality is in mind. Vendor neu= tral is probably what people most look for, so I suggest changing it to tha= t - it implies HW neutral I think. OK > > > > +- transparent visibility > > > > > > What is implied by this? It is "clear methodology" or "transparent > > testing methods"? > >=20 > > I mean we should be able to see which tests are run, what is the > > environment, how tests are configured and what is the result. > > The property to measure is really the visibility. > >=20 > I think just "transparency" is probably the best title then. OK for transparency. > However, to clarify things, I think it might be good to have a one-line e= xplanation of each bullet point to allow additional clarity. For example: >=20 > - reliable: users need to know that any failures are genuine, and not the= result of intermittent issues in the test infrastructure > - neutral: the lab should not focus on one HW or SW platform to the exclu= sion or detriment of others > - transparent: the details of what tests have been run and the result of = those tests should be visible to all users. >=20 > > > > +- coverage summary > > > > +- trending graphs > > > > +- reports to author and > > > > +[test-report@dpdk.org](//inbox.dpdk.org/test-report/) > > > > +- easy to add a test > > > > +- easy to reproduce a test locally > > > > + > > > > > > The first three are what I'd consider properties, but the rest are > > > more functions that should be included in the CI, rather than propert= ies > > of it. > >=20 > > I don't agree: > > - coverage apply to all tests: what is the coverage of each test? > > - trending graphs can be done for any test or measure > > - reporting must be done for all tests > > - ability to add test is an infrastructure property > > - ability to reproduce is a test property > >=20 > Yes, I agree they are needed, they just don't fit in the same list as rel= iability, transparency and neutrality, which are abstract concepts vs cover= age reports, trend graphs etc. which are concrete outputs. Therefore I sugg= est they be put in a separate list on the page, not dropped altogether. Transparency is also achieved through some outputs. And the last two ones (add and reproduce) are not outputs. I understand your concern but I don't know how to separate lists. > > [...] > > > > +The tests should cover these categories: > > > > + > > > > +- compilation > > > > +- static analysis > > > > +- functional > > > > +- performance > > > > +- interoperability > > > > +- portability > > > > +- compatibility > > > > + > > > > > > While compilation and static analysis are ok, the rest need "testing" > > after them since they are not nouns. Alternatively drop "static analysi= s" > > from the list and then change compilation to "compile" to use only > > adjectives. > >=20 > > I don't want to drop static analysis, I think it is a test category. > > We could add "testing" after each item, but I feel it is just an overhe= ad. > > I don't know what to do here. >=20 > Maybe split it into two levels of bullets: >=20 > - compilation > - static analysis > - test case runs, including > - functional tests > - performance tests > - interoperability tests > - .... >=20 > If that doesn't work for you, it's ok to leave as-is. Everyone will under= stand it, it just reads a little weird to me. =F0=9F=98=8A OK for the nested bullet for runtime tests. > > [...] > > > > +### Patch Testing > > > > + > > > > +When a patch is submitted, it is received via the mailing list > > > > +[dpdk-dev](//inbox.dpdk.org/dev/). > > > > +For simple tests, > > > > +like > > > > +[checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh) > > > > +, it can be run directly on email receiving. > > > > > > s/email receiving/the received email/ > >=20 > > I wanted to emphasize on the trigger which "receiving" instead of > > "polling". > > But "on the received email" is also OK. Is it a better English? > >=20 > Ah, ok, I understand a little better. To emphasise the triggering rather = than what the scan runs on, I suggest "on email reception", or change "run"= to "triggered" as in - "triggered on receipt of the patch email, and run o= n the email itself." OK thanks. Note: I tried to understand the difference between receiving, reception and= receipt before sending this patch. Looks like I failed to pick the best English (or Irish) form ;) Thanks for the good review.