From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id D4058106A for ; Sat, 5 Nov 2016 05:47:31 +0100 (CET) Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP; 04 Nov 2016 21:47:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.31,595,1473145200"; d="scan'208";a="27661650" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by fmsmga006.fm.intel.com with ESMTP; 04 Nov 2016 21:47:30 -0700 Received: from fmsmsx114.amr.corp.intel.com (10.18.116.8) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.248.2; Fri, 4 Nov 2016 21:47:30 -0700 Received: from shsmsx101.ccr.corp.intel.com (10.239.4.153) by FMSMSX114.amr.corp.intel.com (10.18.116.8) with Microsoft SMTP Server (TLS) id 14.3.248.2; Fri, 4 Nov 2016 21:47:29 -0700 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.139]) by SHSMSX101.ccr.corp.intel.com ([169.254.1.104]) with mapi id 14.03.0248.002; Sat, 5 Nov 2016 12:47:27 +0800 From: "Liu, Yong" To: "moving@dpdk.org" Thread-Topic: proposal for DPDK CI improvement Thread-Index: AdI3H6aOR4pSQwpSRVuot1dMSoDmOw== Date: Sat, 5 Nov 2016 04:47:26 +0000 Message-ID: <86228AFD5BCD8E4EBFD2B90117B5E81E60310FA1@SHSMSX103.ccr.corp.intel.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiOWU1NGNiNWUtNWVmMy00NGU4LThjZjYtMjJjZDc4ZWM2NTZjIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE1LjkuNi42IiwiVHJ1c3RlZExhYmVsSGFzaCI6IlFRQkl0Q054SjhOT2ZKbmpDdFZkQUtxS0c1MDR0VzdCSGVQaUd1aWFSclE9In0= x-ctpclassification: CTP_IC x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: [dpdk-moving] proposal for DPDK CI improvement X-BeenThere: moving@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK community structure changes List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Nov 2016 04:47:32 -0000 I'm marvin and on behalf of Intel STV team. As we are moving to LF, there's one chance for us to discuss on how to enhance DPDK CI infrastructure. Currently, DPDK CI has done in a distributed way. Several companies running their own CI tests internally. Some companies running their own CI tests internally. Some of them (Intel and IBM) provided their test reports to mailing list, but others keep their test results for internal use only. There are two possible approaches that we can consider for improving DPDK C= I: 1. Create a centralized DPDK CI lab and buildup required infrastructure. 2. Continue with a distributed model but improve reporting and visibility. We think the main advantages of a centralized approach are: Transparency: Everybody can see and access the test infrastructure, see wha= t exactly how the servers are configured and what tests have be= en run and their result. The community can review and agree collectively when new tests are required. Flexibility: Testing can be performed on demand. Instead of a developer submitting a patch, having it tested by a distributed CI infrastructure and then getting test results. The developer c= an access the CI infrastructure and trigger the tests manually before submitting the patch, thus speeding up the development process, make short test cycle. Independence: Instead of each vendor providing their own performance result= s, having these generated in a centralized lab run by an=20 independent body will increase confidence that DPDK users hav= e in the test results. There is one example of how this was done for another project. (https://wiki.fd.io/view/CSIT). In their wiki page, you can get the idea about how to configure the servers and run test cases on these servers. The test report for all releases can b= e found on the page. You can also browse the detail test report for each rele= ase if click the link. If click their Jekin's link, you can see the trend of project status. =20 The main disadvantages of a centralized approach are relocating equipment from separate vendor labs will require a project budget. We can depend on budget to decide which infrastructure should be deployed in the public test lab. For distributed model, we essentially continue as what we are at present. Vendors can independently choose the CI tests that they run, and the report= s that they choose to make public. We can add enhancements to Patchwork to display test results to make tracking easier, and can also look at other wa= ys to make test reports more visible. The main advantages of a distributed approach are: There's no requirement for a project budget. The disadvantages of a distributed approach are: We lost the benefits of transparency, independence and the ability to run tests on demand that are described under the centralized approach above. CI testing and the publication of the results remains under the control of vendors (or others who choose to run CI tests). Based on the above, we would like to propose that a centralized CI lab. Details of the required budget and funding for this will obviously need to = be determined, but for now our proposal will focus purely on the technical sco= pe and benefits. ---------------------------------------------------------------------------= --- At the moment, the hardware requirements below are identified only for Inte= l platforms. We'd like to encourage input from other vendors on additional hardware requirements to support their platforms. The items below are prioritized so that we can determine the cut-off point based on the final budget that is available for this activity. Daily Performance Test Scope: An l3fwd performance test will be run once per day on the master branch to identify any performance degradation. The test will use a software packet generator (could be pktgen or TRex). =20 Test execution time: Almost 60 mins for RFC2544 Priority: 1 =20 Hardware Requirements:=20 For x86 this will require two 2U servers, one to run the packet generator and one to act as the device under test (DUT).=20 In order to make sure the test results are consistent from one run to the next, we recommend to allocate dedicated servers. But if budget doesn't allow us to place too much servers, we can optimize our performance testing by make performance test beds to share with other testing like regression unit and building test, then we can maximize = the utilization of infrastructure. =20 Hardware requirements for other CPU architectures need to be determin= ed by the relevant vendors. Automated Patch Compilation Test (Pre check-in Patch Testing): Scope: When developers submit patches to the dev@dpdk.org mailing list = a back-end automation script will be triggered automatically to run a compilation test.=20 The results will be sent to the test-report@dpdk.org mailing list and t= o the patch author. To deliver timely report for patch, the automated pat= ch compilation test only verifies patch within few OSVs. So Patch compilat= ion test can't achieve same coverage with daily compilation test on master branch.=20 Testing should ideally be performed on all supported CPU platforms (x86= , ARM, Power 8, TILE-Gx etc.), but this will depend on which vendors are willing to contribute hardware platforms to the DPDK CI lab. Testing will be performed on multiple operating systems (Fedora, Ubuntu= , RHEL etc.), kernel versions (latest stable Linux Kernel + default Linu= x Kernel in OSV) and compiler versions (GCC/ICC/Clang). Test execution time:=20 5 mins per patch, average 30 min per patch set=20 Priority: 2 Hardware Requirements:=20 For x86 this will require one dedicated 2U server. Because the tests will be run frequently (every time a new patch or patch set is submitted), it's not realistic to run this testing on shared servers.= =20 Combinations of operating systems, kernels and compiler versions will= be configured as separate VMs on the server. Hardware requirements for other CPU architectures need to be determined by the relevant vendors. =20 Automated Daily Compilation Test: Scope: This is similar to the previous item but run once per day on the master branch and on the latest stable/LTS release branches. Since code merging will cause some issues, which breaks compilation for master branch. Automated Daily Compilation Test is used to monitor and avoid this kind of issues. In general, Automated Daily Compilation Test will verify latest branch with 3 compilers (ICC/GCC/Clang) and 4 buildi= ng options on each mainstream OSV. Currently, intel daily build test was performed on almost 14 OVS with different Linux kernel. It includes all the mainstream operating systems, including Ubuntu, Redhat, Fedora, SUS= E, CentOS, Windriver, FreeBSD and MontaVista Linux. =20 Test execution time:=20 30 mins per one platform. We can perform build testing on different OSV in parallel. Priority: 3 =20 Hardware Requirements: For x86 this will require one 2U server. Because the tests will be ru= n at the same time every day, they could be scheduled to run on a share= d server (approximate current test duration is ~2 hours on 2.6GHz CPU w= ith 64GB RAM). Combinations of operating systems, kernels and compiler versions will be configured as separate VMs on the server. =20 Hardware requirements for other CPU architectures need to be determined= by the relevant vendors. =20 Regression Unit Test: Scope: Unit tests will be run once per day on the master branch and on = the latest stable/LTS release branches with one mainstream operating system= . =20 Test execution time: 2 hours to complete all automated unit test. Priority: 4 Hardware Requirements: For x86 this will require one 2U server. Because the tests will be run at the same time every day, they could = be scheduled to run on a shared server (approximate current test duratio= n is ~1 hour on 2.6GHz CPU with 64GB RAM). =20 Regression Function Test: Since this function test depends on NIC features. It's difficult to mak= e standard test plan & cases for different platforms. This test can be executed in the distributed lab. After platform owners complete testing= , they can provide reports and test plan to maintainer for reviewing.