[dpdk-ci] [PATCH] add general testing guidelines

[dpdk-ci] [PATCH] add general testing guidelines

[Thomas Monjalon]

The community lab page is replaced with a more general page about testing,
where the community lab is part of the big picture. It includes:
	- testing goals
	- CI workflow
	- tools

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
 content/lab/_index.md                |  21 ----
 content/testing/_index.md            | 142 +++++++++++++++++++++++++++
 static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes
 3 files changed, 142 insertions(+), 21 deletions(-)
 delete mode 100644 content/lab/_index.md
 create mode 100644 content/testing/_index.md
 delete mode 100644 static/img/community-lab-diagram.png

diff --git a/content/lab/_index.md b/content/lab/_index.md
deleted file mode 100644
index 913c86f..0000000
--- a/content/lab/_index.md
+++ /dev/null
@@ -1,21 +0,0 @@
-+++
-title = "Community Lab"
-weight = "8"
-+++
-
-{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{< /button >}}
-
-## What it's for
-----
-
-- Automated performance regression testing of new patches.
-  - When a new patch is received, automatically test
-    and determine if it degrades performance or causes other issues
-    on one or more architectures.
-  - Provides input to maintainers on whether to accept patches or request rework.
-- The Dashboard currently provides performance deltas for each patch
-  compared to the expected numbers of each architecture.
-- The scope may expand in future:
-  test DPDK-enabled applications, use lab for demos, etc.
-
-![DPDK Community Lab](/img/community-lab-diagram.png)
diff --git a/content/testing/_index.md b/content/testing/_index.md
new file mode 100644
index 0000000..30da214
--- /dev/null
+++ b/content/testing/_index.md
@@ -0,0 +1,142 @@
++++
+title = "Testing"
+weight = "8"
++++
+
+As a community project, DPDK CI is distributed and welcome any contribution.
+
+There are various ways of contributing to the CI effort:
+
+- suggest improvement
+- write a new test
+- improve infrastucture scripts
+- run a lab
+
+A specific effort is done
+for the [community lab](//lab.dpdk.org/results/dashboard/about)
+hosted at [UNH-IOL](//www.iol.unh.edu).
+
+
+## CI Testing Goals
+---
+
+### General Properties
+
+Ideally, the CI infrastructure should have these properties:
+
+- reliable
+- neutral
+- transparent visibility
+- coverage summary
+- trending graphs
+- reports to author and
+[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
+- easy to add a test
+- easy to reproduce a test locally
+
+### Inputs
+
+There are three kinds of inputs to be tested:
+
+- per-patch testing integrated
+with [patchwork](//patches.dpdk.org/project/dpdk/list/)
+- per-commit testing after merges
+in mainline or dpdk-next- [repositories](//git.dpdk.org/)
+- per-release testing
+including [release candidates](//git.dpdk.org/dpdk/refs/)
+of all [branches](//git.dpdk.org/dpdk-stable/refs/)
+
+### Categories
+
+The tests should cover these categories:
+
+- compilation
+- static analysis
+- functional
+- performance
+- interoperability
+- portability
+- compatibility
+
+
+## Workflow
+---
+
+### Patch Testing
+
+When a patch is submitted, it is received
+via the mailing list [dpdk-dev](//inbox.dpdk.org/dev/).
+For simple tests,
+like [checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh),
+it can be run directly on email receiving.
+
+For most tests, the patch must be applied
+on the [right tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py)
+after its series dependencies.
+Dealing with series and dependencies is easier
+via [patchwork database request](//patches.dpdk.org/api/).
+That's why the common workflow is
+to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw).
+The patches are saved and organized
+in [patchwork](//patches.dpdk.org/project/dpdk/list/)
+after being distributed by the mailing list.
+
+Once the patch is ready, the automatic tests can run.
+
+When a group of tests is finished, the author should be notified of any error,
+and a public report must be sent to
+[test-report@dpdk.org](//inbox.dpdk.org/test-report/).
+
+The reports are
+[formatted](//git.dpdk.org/tools/dpdk-ci/tree/tools/send-patch-report.sh)
+with a specific syntax for automatic parsing.
+When such report email is received, the bot ci@dpdk.org will
+[update patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh),
+referencing the report archive and incrementing one of the counters
+(success, warning or failure).
+
+A maintainer should wait the end of the tests (should be less than a day),
+and check the results before accepting a patch.
+
+### Mainline Monitoring
+
+The repository mirror on [GitHub](//github.com/DPDK/dpdk)
+is preferred for tests integration:
+
+- If polling for updates, better to load the GitHub mirror.
+- Some services like [Travis](//travis-ci.com/DPDK/)
+can be hooked in GitHub.
+
+### Release Validation
+
+When a new release is available, especially for release candidates,
+some validation reports are requested to the community members.
+
+Such release validations are sent as replies to the release announcement.
+
+
+## Tools
+---
+
+### Infrastructure
+
+The testing infrastructure scripts are shared in
+the [dpdk-ci git repository](//git.dpdk.org/tools/dpdk-ci/).
+
+The website dashboard and API of the community lab is a Django project shared
+in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-site/).
+
+The testing infrastructure work is coordinated in three complementary forums:
+
+- [dpdk-ci mailing list](//inbox.dpdk.org/ci/)
+- [bugzilla tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20scripts&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpriority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_format=advanced&order=priority&list_id=1987)
+- bi-weekly meeting
+
+### Known Test Frameworks and Tools
+
+- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/)
+- [DPDK apps](//git.dpdk.org/dpdk/tree/app/)
+- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/)
+- [VSPERF](//git.opnfv.org/vswitchperf/about/)
+- [OvS PVP perf](//github.com/chaudron/ovs_perf/)
+- [downstream projects](//www.dpdk.org/ecosystem/#projects)
diff --git a/static/img/community-lab-diagram.png b/static/img/community-lab-diagram.png
deleted file mode 100644
index a32109a..0000000
Binary files a/static/img/community-lab-diagram.png and /dev/null differ
-- 
2.25.1

Re: [dpdk-ci] [dpdk-web] [PATCH] add general testing guidelines

[Richardson, Bruce]

> -----Original Message-----
> From: web <web-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> Sent: Tuesday, March 10, 2020 10:12 PM
> To: web@dpdk.org
> Cc: ci@dpdk.org
> Subject: [dpdk-web] [PATCH] add general testing guidelines
> 
> The community lab page is replaced with a more general page about testing,
> where the community lab is part of the big picture. It includes:
> 	- testing goals
> 	- CI workflow
> 	- tools
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---

General question - is this meant to describe the ideal state or the current state of testing?
It starts off largely inspirational, but then ends with lots of details on the current
setup. Do we want to have that mix in the one page?

I've also put some stylistic comments inline below too.

/Bruce

>  content/lab/_index.md                |  21 ----
>  content/testing/_index.md            | 142 +++++++++++++++++++++++++++
>  static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes
>  3 files changed, 142 insertions(+), 21 deletions(-)  delete mode 100644
> content/lab/_index.md  create mode 100644 content/testing/_index.md
> delete mode 100644 static/img/community-lab-diagram.png
> 
> diff --git a/content/lab/_index.md b/content/lab/_index.md deleted file
> mode 100644 index 913c86f..0000000
> --- a/content/lab/_index.md
> +++ /dev/null
> @@ -1,21 +0,0 @@
> -+++
> -title = "Community Lab"
> -weight = "8"
> -+++
> -
> -{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{<
> /button >}}
> -
> -## What it's for
> -----
> -
> -- Automated performance regression testing of new patches.
> -  - When a new patch is received, automatically test
> -    and determine if it degrades performance or causes other issues
> -    on one or more architectures.
> -  - Provides input to maintainers on whether to accept patches or request
> rework.
> -- The Dashboard currently provides performance deltas for each patch
> -  compared to the expected numbers of each architecture.
> -- The scope may expand in future:
> -  test DPDK-enabled applications, use lab for demos, etc.
> -
> -![DPDK Community Lab](/img/community-lab-diagram.png)
> diff --git a/content/testing/_index.md b/content/testing/_index.md new
> file mode 100644 index 0000000..30da214
> --- /dev/null
> +++ b/content/testing/_index.md
> @@ -0,0 +1,142 @@
> ++++
> +title = "Testing"
> +weight = "8"
> ++++
> +
> +As a community project, DPDK CI is distributed and welcome any
> contribution.
> +
> +There are various ways of contributing to the CI effort:
> +
> +- suggest improvement
> +- write a new test
> +- improve infrastucture scripts

Typo: infrastRucture

> +- run a lab
> +
> +A specific effort is done
> +for the [community lab](//lab.dpdk.org/results/dashboard/about)
> +hosted at [UNH-IOL](//www.iol.unh.edu).
> +

Not sure about the phrase "specific effort". How about "One specific lab instance has been set up for the community lab..."

> +
> +## CI Testing Goals
> +---
> +
> +### General Properties
> +
> +Ideally, the CI infrastructure should have these properties:
> +
> +- reliable
> +- neutral

Neutral in what way? Presumably vendor or HW neutral?

> +- transparent visibility

What is implied by this? It is "clear methodology" or "transparent testing methods"?

> +- coverage summary
> +- trending graphs
> +- reports to author and
> +[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
> +- easy to add a test
> +- easy to reproduce a test locally
> +

The first three are what I'd consider properties, but the rest are more functions that should be included in the CI, rather than properties of it.

> +### Inputs
> +
> +There are three kinds of inputs to be tested:
> +
> +- per-patch testing integrated
> +with [patchwork](//patches.dpdk.org/project/dpdk/list/)
> +- per-commit testing after merges
> +in mainline or dpdk-next- [repositories](//git.dpdk.org/)
> +- per-release testing
> +including [release candidates](//git.dpdk.org/dpdk/refs/)
> +of all [branches](//git.dpdk.org/dpdk-stable/refs/)
> +
> +### Categories
> +
> +The tests should cover these categories:
> +
> +- compilation
> +- static analysis
> +- functional
> +- performance
> +- interoperability
> +- portability
> +- compatibility
> +

While compilation and static analysis are ok, the rest need "testing" after them since they are not nouns. Alternatively drop "static analysis" from the list and then change compilation to "compile" to use only adjectives.

> +
> +## Workflow
> +---
> +
> +### Patch Testing
> +
> +When a patch is submitted, it is received via the mailing list
> +[dpdk-dev](//inbox.dpdk.org/dev/).
> +For simple tests,
> +like
> +[checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh),
> +it can be run directly on email receiving.

s/email receiving/the received email/

> +
> +For most tests, the patch must be applied on the [right
> +tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py)
> +after its series dependencies.
> +Dealing with series and dependencies is easier via [patchwork database
> +request](//patches.dpdk.org/api/).
> +That's why the common workflow is
> +to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw).
> +The patches are saved and organized
> +in [patchwork](//patches.dpdk.org/project/dpdk/list/)
> +after being distributed by the mailing list.
> +
> +Once the patch is ready, the automatic tests can run.
> +
> +When a group of tests is finished, the author should be notified of any
> +error, and a public report must be sent to
> +[test-report@dpdk.org](//inbox.dpdk.org/test-report/).
> +
> +The reports are
> +[formatted](//git.dpdk.org/tools/dpdk-ci/tree/tools/send-patch-report.s
> +h) with a specific syntax for automatic parsing.
> +When such report email is received, the bot ci@dpdk.org will [update
> +patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh),
> +referencing the report archive and incrementing one of the counters
> +(success, warning or failure).
> +
> +A maintainer should wait the end of the tests (should be less than a
> +day), and check the results before accepting a patch.
> +
> +### Mainline Monitoring
> +
> +The repository mirror on [GitHub](//github.com/DPDK/dpdk) is preferred
> +for tests integration:
> +
> +- If polling for updates, better to load the GitHub mirror.
> +- Some services like [Travis](//travis-ci.com/DPDK/) can be hooked in
> +GitHub.
> +
> +### Release Validation
> +
> +When a new release is available, especially for release candidates,
> +some validation reports are requested to the community members.
> +
> +Such release validations are sent as replies to the release announcement.
> +
> +
> +## Tools
> +---
> +
> +### Infrastructure
> +
> +The testing infrastructure scripts are shared in the [dpdk-ci git
> +repository](//git.dpdk.org/tools/dpdk-ci/).
> +
> +The website dashboard and API of the community lab is a Django project
> +shared in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-
> site/).
> +
> +The testing infrastructure work is coordinated in three complementary
> forums:
> +
> +- [dpdk-ci mailing list](//inbox.dpdk.org/ci/)
> +- [bugzilla
> +tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20script
> +s&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_statu
> +s=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpri
> +ority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_forma
> +t=advanced&order=priority&list_id=1987)
> +- bi-weekly meeting
> +
> +### Known Test Frameworks and Tools
> +
> +- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/)
> +- [DPDK apps](//git.dpdk.org/dpdk/tree/app/)
> +- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/)
> +- [VSPERF](//git.opnfv.org/vswitchperf/about/)
> +- [OvS PVP perf](//github.com/chaudron/ovs_perf/)
> +- [downstream projects](//www.dpdk.org/ecosystem/#projects)
> diff --git a/static/img/community-lab-diagram.png b/static/img/community-
> lab-diagram.png
> deleted file mode 100644
> index a32109a..0000000
> Binary files a/static/img/community-lab-diagram.png and /dev/null differ
> --
> 2.25.1

Re: [dpdk-ci] [dpdk-web] [PATCH] add general testing guidelines

[Thomas Monjalon]

16/03/2020 10:43, Richardson, Bruce:
> From: web <web-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> > 
> > The community lab page is replaced with a more general page about testing,
> > where the community lab is part of the big picture. It includes:
> > 	- testing goals
> > 	- CI workflow
> > 	- tools
> 
> General question - is this meant to describe the ideal state or the current state of testing?

Yes it is about goals and workflow details, plus reference the tools which are used.
I think these three items give a big picture.

> It starts off largely inspirational, but then ends with lots of details on the current
> setup. Do we want to have that mix in the one page?

I plan to add one more page to list what is currently done in which lab.
So the split is:
	- one guideline page to setup a lab
	- one status page

I understand your question about splitting the guidelines in several pages
but I don't whether it would help reading, or the opposite, makes harder
to get all informations about what is a lab?

> I've also put some stylistic comments inline below too.
[...]
> > +There are various ways of contributing to the CI effort:
> > +
> > +- suggest improvement
> > +- write a new test
> > +- improve infrastucture scripts
> 
> Typo: infrastRucture

OK thanks

> > +- run a lab
> > +
> > +A specific effort is done
> > +for the [community lab](//lab.dpdk.org/results/dashboard/about)
> > +hosted at [UNH-IOL](//www.iol.unh.edu).
> > +
> 
> Not sure about the phrase "specific effort". How about "One specific lab instance has been set up for the community lab..."

What about this?
One specific lab instance, named community lab, is hosted at UNH-IOL.

[...]
> > +Ideally, the CI infrastructure should have these properties:
> > +
> > +- reliable
> > +- neutral
> 
> Neutral in what way? Presumably vendor or HW neutral?

Neutral is a property which can apply to vendors, HW or SW (OS).
Do you think "vendor neutral" is better?

> > +- transparent visibility
> 
> What is implied by this? It is "clear methodology" or "transparent testing methods"?

I mean we should be able to see which tests are run,
what is the environment, how tests are configured and what is the result.
The property to measure is really the visibility.

> > +- coverage summary
> > +- trending graphs
> > +- reports to author and
> > +[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
> > +- easy to add a test
> > +- easy to reproduce a test locally
> > +
> 
> The first three are what I'd consider properties,
> but the rest are more functions that should be included in the CI, rather than properties of it.

I don't agree:
	- coverage apply to all tests: what is the coverage of each test?
	- trending graphs can be done for any test or measure
	- reporting must be done for all tests
	- ability to add test is an infrastructure property
	- ability to reproduce is a test property

[...]
> > +The tests should cover these categories:
> > +
> > +- compilation
> > +- static analysis
> > +- functional
> > +- performance
> > +- interoperability
> > +- portability
> > +- compatibility
> > +
> 
> While compilation and static analysis are ok, the rest need "testing" after them since they are not nouns. Alternatively drop "static analysis" from the list and then change compilation to "compile" to use only adjectives.

I don't want to drop static analysis, I think it is a test category.
We could add "testing" after each item, but I feel it is just an overhead.
I don't know what to do here.

[...]
> > +### Patch Testing
> > +
> > +When a patch is submitted, it is received via the mailing list
> > +[dpdk-dev](//inbox.dpdk.org/dev/).
> > +For simple tests,
> > +like
> > +[checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh),
> > +it can be run directly on email receiving.
> 
> s/email receiving/the received email/

I wanted to emphasize on the trigger which "receiving" instead of "polling".
But "on the received email" is also OK. Is it a better English?

Thanks for the review.

Re: [dpdk-ci] [dpdk-web] [PATCH] add general testing guidelines

[Richardson, Bruce]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, March 16, 2020 11:49 AM
> To: Richardson, Bruce <bruce.richardson@intel.com>
> Cc: web@dpdk.org; ci@dpdk.org
> Subject: Re: [dpdk-web] [PATCH] add general testing guidelines
> 
> 16/03/2020 10:43, Richardson, Bruce:
> > From: web <web-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> > >
> > > The community lab page is replaced with a more general page about
> > > testing, where the community lab is part of the big picture. It
> includes:
> > > 	- testing goals
> > > 	- CI workflow
> > > 	- tools
> >
> > General question - is this meant to describe the ideal state or the
> current state of testing?
> 
> Yes it is about goals and workflow details, plus reference the tools which
> are used.
> I think these three items give a big picture.
> 
> > It starts off largely inspirational, but then ends with lots of
> > details on the current setup. Do we want to have that mix in the one
> page?
> 
> I plan to add one more page to list what is currently done in which lab.
> So the split is:
> 	- one guideline page to setup a lab
> 	- one status page
> 
Do we anticipate others setting up public labs? I suppose it doesn't hurt to
document this even if no other labs are set up.

> I understand your question about splitting the guidelines in several pages
> but I don't whether it would help reading, or the opposite, makes harder
> to get all informations about what is a lab?
> 
Ok, happy to keep it all in one place.

> > I've also put some stylistic comments inline below too.
> [...]
> > > +There are various ways of contributing to the CI effort:
> > > +
> > > +- suggest improvement
> > > +- write a new test
> > > +- improve infrastucture scripts
> >
> > Typo: infrastRucture
> 
> OK thanks
> 
> > > +- run a lab
> > > +
> > > +A specific effort is done
> > > +for the [community lab](//lab.dpdk.org/results/dashboard/about)
> > > +hosted at [UNH-IOL](//www.iol.unh.edu).
> > > +
> >
> > Not sure about the phrase "specific effort". How about "One specific lab
> instance has been set up for the community lab..."
> 
> What about this?
> One specific lab instance, named community lab, is hosted at UNH-IOL.
> 
> [...]
> > > +Ideally, the CI infrastructure should have these properties:
> > > +
> > > +- reliable
> > > +- neutral
> >
> > Neutral in what way? Presumably vendor or HW neutral?
> 
> Neutral is a property which can apply to vendors, HW or SW (OS).
> Do you think "vendor neutral" is better?
> 
We just need to clarify what type(s) of neutrality is in mind. Vendor neutral is probably what people most look for, so I suggest changing it to that - it implies HW neutral I think.

> > > +- transparent visibility
> >
> > What is implied by this? It is "clear methodology" or "transparent
> testing methods"?
> 
> I mean we should be able to see which tests are run, what is the
> environment, how tests are configured and what is the result.
> The property to measure is really the visibility.
> 
I think just "transparency" is probably the best title then. However, to clarify things, I think it might be good to have a one-line explanation of each bullet point to allow additional clarity. For example:

- reliable: users need to know that any failures are genuine, and not the result of intermittent issues in the test infrastructure
- neutral: the lab should not focus on one HW or SW platform to the exclusion or detriment of others
- transparent: the details of what tests have been run and the result of those tests should be visible to all users.

> > > +- coverage summary
> > > +- trending graphs
> > > +- reports to author and
> > > +[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
> > > +- easy to add a test
> > > +- easy to reproduce a test locally
> > > +
> >
> > The first three are what I'd consider properties, but the rest are
> > more functions that should be included in the CI, rather than properties
> of it.
> 
> I don't agree:
> 	- coverage apply to all tests: what is the coverage of each test?
> 	- trending graphs can be done for any test or measure
> 	- reporting must be done for all tests
> 	- ability to add test is an infrastructure property
> 	- ability to reproduce is a test property
> 
Yes, I agree they are needed, they just don't fit in the same list as reliability, transparency and neutrality, which are abstract concepts vs coverage reports, trend graphs etc. which are concrete outputs. Therefore I suggest they be put in a separate list on the page, not dropped altogether.

> [...]
> > > +The tests should cover these categories:
> > > +
> > > +- compilation
> > > +- static analysis
> > > +- functional
> > > +- performance
> > > +- interoperability
> > > +- portability
> > > +- compatibility
> > > +
> >
> > While compilation and static analysis are ok, the rest need "testing"
> after them since they are not nouns. Alternatively drop "static analysis"
> from the list and then change compilation to "compile" to use only
> adjectives.
> 
> I don't want to drop static analysis, I think it is a test category.
> We could add "testing" after each item, but I feel it is just an overhead.
> I don't know what to do here.

Maybe split it into two levels of bullets:

- compilation
- static analysis
- test case runs, including
  - functional tests
  - performance tests
  - interoperability tests
  - ....

If that doesn't work for you, it's ok to leave as-is. Everyone will understand it, it just reads a little weird to me. 😊

> 
> [...]
> > > +### Patch Testing
> > > +
> > > +When a patch is submitted, it is received via the mailing list
> > > +[dpdk-dev](//inbox.dpdk.org/dev/).
> > > +For simple tests,
> > > +like
> > > +[checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh)
> > > +, it can be run directly on email receiving.
> >
> > s/email receiving/the received email/
> 
> I wanted to emphasize on the trigger which "receiving" instead of
> "polling".
> But "on the received email" is also OK. Is it a better English?
> 
Ah, ok, I understand a little better. To emphasise the triggering rather than what the scan runs on, I suggest "on email reception", or change "run" to "triggered" as in - "triggered on receipt of the patch email, and run on the email itself."

/Bruce

Re: [dpdk-ci] [dpdk-web] [PATCH] add general testing guidelines

[Thomas Monjalon]

16/03/2020 13:17, Richardson, Bruce:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 16/03/2020 10:43, Richardson, Bruce:
> > > From: web <web-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> > > >
> > > > The community lab page is replaced with a more general page about
> > > > testing, where the community lab is part of the big picture. It
> > includes:
> > > > 	- testing goals
> > > > 	- CI workflow
> > > > 	- tools
> > >
> > > General question - is this meant to describe the ideal state or the
> > current state of testing?
> > 
> > Yes it is about goals and workflow details, plus reference the tools which
> > are used.
> > I think these three items give a big picture.
> > 
> > > It starts off largely inspirational, but then ends with lots of
> > > details on the current setup. Do we want to have that mix in the one
> > page?
> > 
> > I plan to add one more page to list what is currently done in which lab.
> > So the split is:
> > 	- one guideline page to setup a lab
> > 	- one status page
> > 
> Do we anticipate others setting up public labs? I suppose it doesn't hurt to
> document this even if no other labs are set up.

I cannot speak for others, but we can already document the community lab,
the Intel lab and the smaller free "labs", i.e. services, from Coverity,
Travis and OBS.
I will work on it as a second step.


> > I understand your question about splitting the guidelines in several pages
> > but I don't whether it would help reading, or the opposite, makes harder
> > to get all informations about what is a lab?
> > 
> Ok, happy to keep it all in one place.
> 
> > > I've also put some stylistic comments inline below too.
> > [...]
> > > > +There are various ways of contributing to the CI effort:
> > > > +
> > > > +- suggest improvement
> > > > +- write a new test
> > > > +- improve infrastucture scripts
> > >
> > > Typo: infrastRucture
> > 
> > OK thanks
> > 
> > > > +- run a lab
> > > > +
> > > > +A specific effort is done
> > > > +for the [community lab](//lab.dpdk.org/results/dashboard/about)
> > > > +hosted at [UNH-IOL](//www.iol.unh.edu).
> > > > +
> > >
> > > Not sure about the phrase "specific effort". How about "One specific lab
> > instance has been set up for the community lab..."
> > 
> > What about this?
> > One specific lab instance, named community lab, is hosted at UNH-IOL.
> > 
> > [...]
> > > > +Ideally, the CI infrastructure should have these properties:
> > > > +
> > > > +- reliable
> > > > +- neutral
> > >
> > > Neutral in what way? Presumably vendor or HW neutral?
> > 
> > Neutral is a property which can apply to vendors, HW or SW (OS).
> > Do you think "vendor neutral" is better?
> > 
> We just need to clarify what type(s) of neutrality is in mind. Vendor neutral is probably what people most look for, so I suggest changing it to that - it implies HW neutral I think.

OK

> > > > +- transparent visibility
> > >
> > > What is implied by this? It is "clear methodology" or "transparent
> > testing methods"?
> > 
> > I mean we should be able to see which tests are run, what is the
> > environment, how tests are configured and what is the result.
> > The property to measure is really the visibility.
> > 
> I think just "transparency" is probably the best title then.

OK for transparency.

> However, to clarify things, I think it might be good to have a one-line explanation of each bullet point to allow additional clarity. For example:
> 
> - reliable: users need to know that any failures are genuine, and not the result of intermittent issues in the test infrastructure
> - neutral: the lab should not focus on one HW or SW platform to the exclusion or detriment of others
> - transparent: the details of what tests have been run and the result of those tests should be visible to all users.
> 
> > > > +- coverage summary
> > > > +- trending graphs
> > > > +- reports to author and
> > > > +[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
> > > > +- easy to add a test
> > > > +- easy to reproduce a test locally
> > > > +
> > >
> > > The first three are what I'd consider properties, but the rest are
> > > more functions that should be included in the CI, rather than properties
> > of it.
> > 
> > I don't agree:
> > 	- coverage apply to all tests: what is the coverage of each test?
> > 	- trending graphs can be done for any test or measure
> > 	- reporting must be done for all tests
> > 	- ability to add test is an infrastructure property
> > 	- ability to reproduce is a test property
> > 
> Yes, I agree they are needed, they just don't fit in the same list as reliability, transparency and neutrality, which are abstract concepts vs coverage reports, trend graphs etc. which are concrete outputs. Therefore I suggest they be put in a separate list on the page, not dropped altogether.

Transparency is also achieved through some outputs.
And the last two ones (add and reproduce) are not outputs.

I understand your concern but I don't know how to separate lists.


> > [...]
> > > > +The tests should cover these categories:
> > > > +
> > > > +- compilation
> > > > +- static analysis
> > > > +- functional
> > > > +- performance
> > > > +- interoperability
> > > > +- portability
> > > > +- compatibility
> > > > +
> > >
> > > While compilation and static analysis are ok, the rest need "testing"
> > after them since they are not nouns. Alternatively drop "static analysis"
> > from the list and then change compilation to "compile" to use only
> > adjectives.
> > 
> > I don't want to drop static analysis, I think it is a test category.
> > We could add "testing" after each item, but I feel it is just an overhead.
> > I don't know what to do here.
> 
> Maybe split it into two levels of bullets:
> 
> - compilation
> - static analysis
> - test case runs, including
>   - functional tests
>   - performance tests
>   - interoperability tests
>   - ....
> 
> If that doesn't work for you, it's ok to leave as-is. Everyone will understand it, it just reads a little weird to me. 😊

OK for the nested bullet for runtime tests.

> > [...]
> > > > +### Patch Testing
> > > > +
> > > > +When a patch is submitted, it is received via the mailing list
> > > > +[dpdk-dev](//inbox.dpdk.org/dev/).
> > > > +For simple tests,
> > > > +like
> > > > +[checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh)
> > > > +, it can be run directly on email receiving.
> > >
> > > s/email receiving/the received email/
> > 
> > I wanted to emphasize on the trigger which "receiving" instead of
> > "polling".
> > But "on the received email" is also OK. Is it a better English?
> > 
> Ah, ok, I understand a little better. To emphasise the triggering rather than what the scan runs on, I suggest "on email reception", or change "run" to "triggered" as in - "triggered on receipt of the patch email, and run on the email itself."

OK thanks.
Note: I tried to understand the difference between receiving, reception and receipt before sending this patch.
Looks like I failed to pick the best English (or Irish) form ;)

Thanks for the good review.

[dpdk-ci] [PATCH v2] add general testing guidelines

[Thomas Monjalon]

The community lab page is replaced with a more general page about testing,
where the community lab is part of the big picture. It includes:
	- testing goals
	- CI workflow
	- tools

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

---

v2:
    - fix typo
    - reword community lab
    - explain CI properties
    - group test categories
    - reword email workflow
    - remove id from bugzilla request

---
 content/lab/_index.md                |  21 ----
 content/testing/_index.md            | 144 +++++++++++++++++++++++++++
 static/img/community-lab-diagram.png | Bin 135778 -> 0 bytes
 3 files changed, 144 insertions(+), 21 deletions(-)
 delete mode 100644 content/lab/_index.md
 create mode 100644 content/testing/_index.md
 delete mode 100644 static/img/community-lab-diagram.png

diff --git a/content/lab/_index.md b/content/lab/_index.md
deleted file mode 100644
index 913c86f..0000000
--- a/content/lab/_index.md
+++ /dev/null
@@ -1,21 +0,0 @@
-+++
-title = "Community Lab"
-weight = "8"
-+++
-
-{{< button align="center" href="https://lab.dpdk.org" >}} Dashboard {{< /button >}}
-
-## What it's for
-----
-
-- Automated performance regression testing of new patches.
-  - When a new patch is received, automatically test
-    and determine if it degrades performance or causes other issues
-    on one or more architectures.
-  - Provides input to maintainers on whether to accept patches or request rework.
-- The Dashboard currently provides performance deltas for each patch
-  compared to the expected numbers of each architecture.
-- The scope may expand in future:
-  test DPDK-enabled applications, use lab for demos, etc.
-
-![DPDK Community Lab](/img/community-lab-diagram.png)
diff --git a/content/testing/_index.md b/content/testing/_index.md
new file mode 100644
index 0000000..1648f45
--- /dev/null
+++ b/content/testing/_index.md
@@ -0,0 +1,144 @@
++++
+title = "Testing"
+weight = "8"
++++
+
+As a community project, DPDK CI is distributed and welcome any contribution.
+
+There are various ways of contributing to the CI effort:
+
+- suggest improvement
+- write a new test
+- improve infrastructure scripts
+- run a lab
+
+One specific lab instance,
+named [community lab](//lab.dpdk.org/results/dashboard/about),
+is hosted at [UNH-IOL](//www.iol.unh.edu).
+
+
+## CI Testing Goals
+---
+
+### General Properties
+
+Ideally, the CI infrastructure should have these properties:
+
+- reliable: always run, without false positive
+- vendor neutral: run same tests and configurations on all HW/SW platforms
+- transparent: show environment, configuration, source code and result
+- coverage summary
+- trending graphs
+- reports to author and
+[test-report@dpdk.org](//inbox.dpdk.org/test-report/)
+- easy to add a test
+- easy to reproduce a test locally
+
+### Inputs
+
+There are three kinds of inputs to be tested:
+
+- per-patch testing integrated
+with [patchwork](//patches.dpdk.org/project/dpdk/list/)
+- per-commit testing after merges
+in mainline or dpdk-next- [repositories](//git.dpdk.org/)
+- per-release testing
+including [release candidates](//git.dpdk.org/dpdk/refs/)
+of all [branches](//git.dpdk.org/dpdk-stable/refs/)
+
+### Categories
+
+The tests should cover these categories:
+
+* source code checks:
+  - static analysis
+  - compilation
+* runtime tests:
+  - functional
+  - performance
+  - interoperability
+  - portability
+  - compatibility
+
+
+## Workflow
+---
+
+### Patch Testing
+
+When a patch is submitted, it is received
+via the mailing list [dpdk-dev](//inbox.dpdk.org/dev/).
+For simple tests,
+like [checkpatch](//git.dpdk.org/tools/dpdk-ci/tree/tests/checkpatch.sh),
+it can be run on email reception.
+
+For most tests, the patch must be applied
+on the [right tree](//git.dpdk.org/tools/dpdk-ci/tree/tools/guess_git_tree.py)
+after its series dependencies.
+Dealing with series and dependencies is easier
+via [patchwork database request](//patches.dpdk.org/api/).
+That's why the common workflow is
+to [poll patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/poll-pw).
+The patches are saved and organized
+in [patchwork](//patches.dpdk.org/project/dpdk/list/)
+after being distributed by the mailing list.
+
+Once the patch is ready, the automatic tests can run.
+
+When a group of tests is finished, the author should be notified of any error,
+and a public report must be sent to
+[test-report@dpdk.org](//inbox.dpdk.org/test-report/).
+
+The reports are
+[formatted](//git.dpdk.org/tools/dpdk-ci/tree/tools/send-patch-report.sh)
+with a specific syntax for automatic parsing.
+When such report email is received, the bot ci@dpdk.org will
+[update patchwork](//git.dpdk.org/tools/dpdk-ci/tree/tools/update-pw.sh),
+referencing the report archive and incrementing one of the counters
+(success, warning or failure).
+
+A maintainer should wait the end of the tests (should be less than a day),
+and check the results before accepting a patch.
+
+### Mainline Monitoring
+
+The repository mirror on [GitHub](//github.com/DPDK/dpdk)
+is preferred for tests integration:
+
+- If polling for updates, better to load the GitHub mirror.
+- Some services like [Travis](//travis-ci.com/DPDK/)
+can be hooked in GitHub.
+
+### Release Validation
+
+When a new release is available, especially for release candidates,
+some validation reports are requested to the community members.
+
+Such release validations are sent as replies to the release announcement.
+
+
+## Tools
+---
+
+### Infrastructure
+
+The testing infrastructure scripts are shared in
+the [dpdk-ci git repository](//git.dpdk.org/tools/dpdk-ci/).
+
+The website dashboard and API of the community lab is a Django project shared
+in the [dpdk-ci-site git repository](//git.dpdk.org/tools/dpdk-ci-site/).
+
+The testing infrastructure work is coordinated in three complementary forums:
+
+- [dpdk-ci mailing list](//inbox.dpdk.org/ci/)
+- [bugzilla tracker](//bugs.dpdk.org/buglist.cgi?product=lab&component=job%20scripts&component=UNH%20infra&resolution=---&bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cpriority%2Cbug_status%2Cassigned_to%2Cshort_desc%2Cchangeddate&query_format=advanced&order=priority)
+- bi-weekly meeting
+
+### Known Test Frameworks and Tools
+
+- [DPDK devtools](//git.dpdk.org/dpdk/tree/devtools/)
+- [DPDK apps](//git.dpdk.org/dpdk/tree/app/)
+- [DPDK Test Suite (DTS)](//git.dpdk.org/tools/dts/)
+- [VSPERF](//git.opnfv.org/vswitchperf/about/)
+- [OvS PVP perf](//github.com/chaudron/ovs_perf/)
+- [downstream projects](//www.dpdk.org/ecosystem/#projects)
diff --git a/static/img/community-lab-diagram.png b/static/img/community-lab-diagram.png
deleted file mode 100644
index a32109a..0000000
Binary files a/static/img/community-lab-diagram.png and /dev/null differ
-- 
2.25.1

Re: [dpdk-ci] [PATCH v2] add general testing guidelines

[Thomas Monjalon]

24/03/2020 12:31, Thomas Monjalon:
> The community lab page is replaced with a more general page about testing,
> where the community lab is part of the big picture. It includes:
> 	- testing goals
> 	- CI workflow
> 	- tools
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

Applied: http://core.dpdk.org/testing/