From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id 5A5F21E33 for ; Mon, 7 Nov 2016 15:22:29 +0100 (CET) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP; 07 Nov 2016 06:22:28 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.31,606,1473145200"; d="scan'208";a="633871" Received: from fmsmsx108.amr.corp.intel.com ([10.18.124.206]) by orsmga002.jf.intel.com with ESMTP; 07 Nov 2016 06:22:28 -0800 Received: from fmsmsx155.amr.corp.intel.com (10.18.116.71) by FMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP Server (TLS) id 14.3.248.2; Mon, 7 Nov 2016 06:22:27 -0800 Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by FMSMSX155.amr.corp.intel.com (10.18.116.71) with Microsoft SMTP Server (TLS) id 14.3.248.2; Mon, 7 Nov 2016 06:22:27 -0800 Received: from shsmsx102.ccr.corp.intel.com ([169.254.2.206]) by SHSMSX152.ccr.corp.intel.com ([169.254.6.2]) with mapi id 14.03.0248.002; Mon, 7 Nov 2016 22:22:25 +0800 From: "Xu, Qian Q" To: Thomas Monjalon CC: "Liu, Yong" , "ci@dpdk.org" Thread-Topic: [dpdk-moving] proposal for DPDK CI improvement Thread-Index: AdI3H6aOR4pSQwpSRVuot1dMSoDmOwANjkGAAFuk5jD//7FNAP//XedggADNB4D//2bQ8A== Date: Mon, 7 Nov 2016 14:22:24 +0000 Message-ID: <82F45D86ADE5454A95A89742C8D1410E3923C008@shsmsx102.ccr.corp.intel.com> References: <86228AFD5BCD8E4EBFD2B90117B5E81E60310FA1@SHSMSX103.ccr.corp.intel.com> <1689822.FXvyOjK9nz@xps13> <82F45D86ADE5454A95A89742C8D1410E3923BEAD@shsmsx102.ccr.corp.intel.com> <2991873.gcDsJIJtl2@xps13> In-Reply-To: <2991873.gcDsJIJtl2@xps13> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-ci] [dpdk-moving] proposal for DPDK CI improvement X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Nov 2016 14:22:30 -0000 See below.=20 -----Original Message----- From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]=20 Sent: Monday, November 7, 2016 8:51 PM To: Xu, Qian Q Cc: Liu, Yong ; ci@dpdk.org Subject: Re: [dpdk-moving] proposal for DPDK CI improvement I'm removing the list moving@dpdk.org as we are discussing implementation d= etails. 2016-11-07 12:20, Xu, Qian Q: > > a). Currently, there is only " S/W/F for Success/Warning/Fail counters" > > in tests, so does it refer to build test or functional test or=20 > > performance test? >=20 > It can be any test, including performance ones. > A major performance regression must be seen as a failed test.=20 >=20 > [Qian] If it refer to any test, so how do we know which test has been=20 > done. For example, some patch may only do build, some may do perf and=20 > some may do functional. > How to differentiate these tests execution? =20 Why do you want to differentiate them in patchwork? There are 3 views of test reports: 1/ On a patchwork page listing patches, we see the counters S/W/F so we can= give attention to patches having some warnings or failures. 2/ On a patchwork page showing only one patch, we see the list of tests, th= e results and the links to the reports. 3/ In a detailed report from the test-report ml archives, we can see exactl= y what's going wrong or what was tested successfully (and what are the numb= ers in case of a performance test). ---To differentiate the test(build, function and performance) that can give= a clear view of what tests has been done, and what tests failed. And we do= n't need click=20 the link to know which test failed.=20 Now, all tests are sharing one column for the Pass or fail or warnings. For= example, we see one patch Test failed, we don't know which test failed or = even don't know if=20 Which tests are executed. For example, if build failed, but no functional t= ests executed, then it showed as Failed; while if the build passed but one = function test is failed, then=20 It also showed as Failed. So from Failed, we don't know failures at which l= evel tests.=20 For one patch, I think we may have different requirements for build test, f= unction test and performance test failures. =20 To pass build is the 1st priority task, no errors are allowed. While as to = functional tests, some failures on function may be acceptable if the pass r= ate is over 90% . As to=20 the performance, the criteria for failure is not clear now, since performan= ce will have some fluctuation and some features will bring some performance= drop as the feature=20 implementation cost. If we can't differentiate them, we can't easily judge = the failure is really critical failures, and we even don't know which tests= are executed. Similarly, for a patch, we have Acked-by, Tested-by and Reviewed-by, which = differentiate the different aspect of the patch review status. If we just p= ut it as Pass or failed, how=20 Can we know if it's reviewed failure or tested failure or acked failure? We= even don't know which activity is done here. Even if there is a link to cl= ick see more details about=20 Acted-by, Tested-by and Reviewed-by, it's not so convenient. So it's the si= milar thoughts to differentiate the tests.=20 Btw, do you want to each patch have function and performance test? As we di= scussed before, currently we have 4 repos, then we may not sure if the patc= h should go to which repo,=20 It's fine for the build, but for the function test, if it applied to wrong = repo, then it may fail the tests. And we may need think how to solve the is= sue.=20 In short term, I suggest that we only keep Build test for each patch, and r= un function and performance test on daily basis for the git tree.=20