[dpdk-ci] Fwd: Re: [dpdk-moving] proposal for DPDK CI improvement

[dpdk-ci] Fwd: Re: [dpdk-moving] proposal for DPDK CI improvement

[Thomas Monjalon]

From: Thomas Monjalon <thomas.monjalon@6wind.com>
To: moving@dpdk.org, Liu, Yong <yong.liu@intel.com>

2016-11-05 04:47, Liu, Yong:
> Currently, DPDK CI has done in a distributed way. Several companies running
> their own CI tests internally. Some companies running their own CI tests
> internally. Some of them (Intel and IBM) provided their test reports to
> mailing list, but others keep their test results for internal use only.

I'm confident we'll have more contributors to the distributed CI when it
will be well advertised (see below).

> There are two possible approaches that we can consider for improving DPDK CI:
> 
> 1. Create a centralized DPDK CI lab and buildup required infrastructure.
> 2. Continue with a distributed model but improve reporting and visibility.

I think these two approaches are good:
1. The centralized open lab can help as a reference
2. The distributed CI instances will bring more diversity

> We think the main advantages of a centralized approach are:
> Transparency: Everybody can see and access the test infrastructure, see what
>               exactly how the servers are configured and what tests have been
>               run and their result. The community can review and agree
>               collectively when new tests are required.
> Flexibility:  Testing can be performed on demand. Instead of a developer
>               submitting a patch, having it tested by a distributed CI
>               infrastructure and then getting test results. The developer can
>               access the CI infrastructure and trigger the tests manually
>               before submitting the patch, thus speeding up the development
>               process, make short test cycle.

It is possible to offer such flexibility in private CI labs by offering
an email address where we can send some patches to be tested.
However there can be an issue of hardware bandwith/availability to solve.
This is the same issue for open/centralized labs or private labs.
A test lab accepting any private request can be abused. That's why I think
forcing to send the patches publically to the mailing list is a good policy.

> Independence: Instead of each vendor providing their own performance results,
>               having these generated in a centralized lab run by an 
>               independent body will increase confidence that DPDK users have
>               in the test results.
> 
> There is one example of how this was done for another project.
> (https://wiki.fd.io/view/CSIT).
> 
> In their wiki page, you can get the idea about how to configure the servers
> and run test cases on these servers. The test report for all releases can be
> found on the page. You can also browse the detail test report for each release
> if click the link. If click their Jekin's link, you can see the trend of
> project status.

I do not see an explanation of how to use the CSIT lab on demand (what you
described as "Flexibility"). How does it work?

> The main disadvantages of a centralized approach are relocating equipment
> from separate vendor labs will require a project budget. We can depend on
> budget to decide which infrastructure should be deployed in the public test
> lab.
> 
> For distributed model, we essentially continue as what we are at present.
> Vendors can independently choose the CI tests that they run, and the reports
> that they choose to make public. We can add enhancements to Patchwork to
> display test results to make tracking easier, and can also look at other ways
> to make test reports more visible.

Yes, I'm working on it.
One year ago, we discussed a CI integration in patchwork:
	https://lists.ozlabs.org/pipermail/patchwork/2015-July/001363.html
It is now implemented in patchwork and available on dpdk.org:
	http://dpdk.org/ml/archives/dev/2016-September/046282.html
A first basic test (checkpatch) is integrated. See this example:
	http://dpdk.org/patch/16953
The detailed report in test-report mailing list archives is referenced
with an hyperlink (in the "Description" column).
The next step (work in progress) is to publish some scripts in a new git
repository dpdk-ci to help integrating more test labs in patchwork.
It basically requires only to receive and send some emails from the lab.

Note that any open or centralized lab can be also integrated in patchwork.

Note also that this patchwork integration covers only the tests run when a
patch is submitted. The tests run regularly on a git tree won't appear
in this interface.

> The main advantages of a distributed approach are:
> There's no requirement for a project budget.

The other major advantage is to have a better test coverage.
Many companies need to have some internal DPDK tests for their needs.
If they use them to provide some public reports, they can avoid having
some regressions with their specific use cases or hardware.
That's why the distributed CI approach is a win-win.

> The disadvantages of a distributed approach are:
> We lost the benefits of transparency, independence and the ability to run
> tests on demand that are described under the centralized approach above. CI
> testing and the publication of the results remains under the control of
> vendors (or others who choose to run CI tests).
> 
> Based on the above, we would like to propose that a centralized CI lab.
> Details of the required budget and funding for this will obviously need to be
> determined, but for now our proposal will focus purely on the technical scope
> and benefits.

Thanks for the detailed description of the tests that you expect.
I think the CI discussion must be thought with two major questions:
	When? and Where?
When:
	- regularly on a git tree
	- after each patch submission -> report available via patchwork
Where:
	- in a private lab
	- in a foundation lab
Both private and foundation labs can be more or less open.

My conclusion: every kind of tests have some benefits and are welcome!