From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id DA6B55587 for ; Mon, 7 Nov 2016 06:15:52 +0100 (CET) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP; 06 Nov 2016 21:15:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.31,604,1473145200"; d="scan'208";a="898488914" Received: from stv-crb-56.sh.intel.com (HELO [10.239.128.116]) ([10.239.128.116]) by orsmga003.jf.intel.com with ESMTP; 06 Nov 2016 21:15:50 -0800 Message-ID: <58200E0A.4010804@intel.com> Date: Mon, 07 Nov 2016 13:15:54 +0800 From: "Liu, Yong" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Thomas Monjalon , moving@dpdk.org References: <86228AFD5BCD8E4EBFD2B90117B5E81E60310FA1@SHSMSX103.ccr.corp.intel.com> <3804736.OkjAMiHs6v@xps13> In-Reply-To: <3804736.OkjAMiHs6v@xps13> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-moving] proposal for DPDK CI improvement X-BeenThere: moving@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK community structure changes List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Nov 2016 05:15:53 -0000 Thanks, Thomas. On 11/06/2016 03:15 AM, Thomas Monjalon wrote: > 2016-11-05 04:47, Liu, Yong: >> Currently, DPDK CI has done in a distributed way. Several companies running >> their own CI tests internally. Some companies running their own CI tests >> internally. Some of them (Intel and IBM) provided their test reports to >> mailing list, but others keep their test results for internal use only. > I'm confident we'll have more contributors to the distributed CI when it > will be well advertised (see below). Agree. > >> There are two possible approaches that we can consider for improving DPDK CI: >> >> 1. Create a centralized DPDK CI lab and buildup required infrastructure. >> 2. Continue with a distributed model but improve reporting and visibility. > I think these two approaches are good: > 1. The centralized open lab can help as a reference > 2. The distributed CI instances will bring more diversity > >> We think the main advantages of a centralized approach are: >> Transparency: Everybody can see and access the test infrastructure, see what >> exactly how the servers are configured and what tests have been >> run and their result. The community can review and agree >> collectively when new tests are required. >> Flexibility: Testing can be performed on demand. Instead of a developer >> submitting a patch, having it tested by a distributed CI >> infrastructure and then getting test results. The developer can >> access the CI infrastructure and trigger the tests manually >> before submitting the patch, thus speeding up the development >> process, make short test cycle. > It is possible to offer such flexibility in private CI labs by offering > an email address where we can send some patches to be tested. > However there can be an issue of hardware bandwith/availability to solve. > This is the same issue for open/centralized labs or private labs. > A test lab accepting any private request can be abused. That's why I think > forcing to send the patches publically to the mailing list is a good policy. > >> Independence: Instead of each vendor providing their own performance results, >> having these generated in a centralized lab run by an >> independent body will increase confidence that DPDK users have >> in the test results. >> >> There is one example of how this was done for another project. >> (https://wiki.fd.io/view/CSIT). >> >> In their wiki page, you can get the idea about how to configure the servers >> and run test cases on these servers. The test report for all releases can be >> found on the page. You can also browse the detail test report for each release >> if click the link. If click their Jekin's link, you can see the trend of >> project status. > I do not see an explanation of how to use the CSIT lab on demand (what you > described as "Flexibility"). How does it work? As I known, CSIT lab have weekly meeting and everyone can join the meeting. They will discuss about the requirements in the meeting and decide on what to do in the next step. We are supporting CSIT team to create dpdk performance regression on x86 platform. Maybe they can share their report to us. >> The main disadvantages of a centralized approach are relocating equipment >> from separate vendor labs will require a project budget. We can depend on >> budget to decide which infrastructure should be deployed in the public test >> lab. >> >> For distributed model, we essentially continue as what we are at present. >> Vendors can independently choose the CI tests that they run, and the reports >> that they choose to make public. We can add enhancements to Patchwork to >> display test results to make tracking easier, and can also look at other ways >> to make test reports more visible. > Yes, I'm working on it. > One year ago, we discussed a CI integration in patchwork: > https://lists.ozlabs.org/pipermail/patchwork/2015-July/001363.html > It is now implemented in patchwork and available on dpdk.org: > http://dpdk.org/ml/archives/dev/2016-September/046282.html > A first basic test (checkpatch) is integrated. See this example: > http://dpdk.org/patch/16953 > The detailed report in test-report mailing list archives is referenced > with an hyperlink (in the "Description" column). > The next step (work in progress) is to publish some scripts in a new git > repository dpdk-ci to help integrating more test labs in patchwork. > It basically requires only to receive and send some emails from the lab. > > Note that any open or centralized lab can be also integrated in patchwork. > > Note also that this patchwork integration covers only the tests run when a > patch is submitted. The tests run regularly on a git tree won't appear > in this interface. My main concern about patchwork is that patchwork's page can't show enough information. Developers may need detail build log or function case log to narrow down issue. As you mentioned, daily regression status for main/LTS/other develop branches need also demonstrate to the community. >> The main advantages of a distributed approach are: >> There's no requirement for a project budget. > The other major advantage is to have a better test coverage. > Many companies need to have some internal DPDK tests for their needs. > If they use them to provide some public reports, they can avoid having > some regressions with their specific use cases or hardware. > That's why the distributed CI approach is a win-win. Agree, we need align on the format of the public reports. > >> The disadvantages of a distributed approach are: >> We lost the benefits of transparency, independence and the ability to run >> tests on demand that are described under the centralized approach above. CI >> testing and the publication of the results remains under the control of >> vendors (or others who choose to run CI tests). >> >> Based on the above, we would like to propose that a centralized CI lab. >> Details of the required budget and funding for this will obviously need to be >> determined, but for now our proposal will focus purely on the technical scope >> and benefits. > Thanks for the detailed description of the tests that you expect. > I think the CI discussion must be thought with two major questions: > When? and Where? > When: > - regularly on a git tree > - after each patch submission -> report available via patchwork > Where: > - in a private lab > - in a foundation lab > Both private and foundation labs can be more or less open. > > My conclusion: every kind of tests have some benefits and are welcome! > Daily Performance Test: When: - regularly on master Where: - propose in foundation lab and enable the ability for guest access Automated Patch Compilation Test (Pre check-in Patch Testing): When: - after one patch-set submission Where: - Either in foundation or private lab Automated Daily Compilation Test: When: - regularly on master/LTS/developing branch Where: - Either in foundation or private lab Regression Unit Test: When: - regularly on master/LTS/developing branch Where: - Either in foundation or private lab Regression Function Test: When: - regularly on master/LTS/developing branch Where: - Either in foundation or private lab Again, deployment of foundation lab is based on the budget.