From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <qian.q.xu@intel.com> Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id E0EDF293B for <ci@dpdk.org>; Mon, 26 Jun 2017 05:47:13 +0200 (CEST) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 Jun 2017 20:47:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.39,393,1493708400"; d="scan'208";a="1186905052" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by fmsmga002.fm.intel.com with ESMTP; 25 Jun 2017 20:47:12 -0700 Received: from shsmsx103.ccr.corp.intel.com (10.239.4.69) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.319.2; Sun, 25 Jun 2017 20:47:12 -0700 Received: from shsmsx102.ccr.corp.intel.com ([169.254.2.146]) by SHSMSX103.ccr.corp.intel.com ([169.254.4.116]) with mapi id 14.03.0319.002; Mon, 26 Jun 2017 11:47:10 +0800 From: "Xu, Qian Q" <qian.q.xu@intel.com> To: "Richardson, Bruce" <bruce.richardson@intel.com>, Thomas Monjalon <thomas@monjalon.net>, "Wei, FangfangX" <fangfangx.wei@intel.com> CC: "ci@dpdk.org" <ci@dpdk.org>, "O'Driscoll, Tim" <tim.odriscoll@intel.com>, Eugene Voronov <eugene@mellanox.com> Thread-Topic: [dpdk-ci] script to determine target repo (was DPDK Lab) Thread-Index: AQHStCGuVaJiVBER0kWSfXBHnGYchKHVrwQw//+DKwCAWaRGAIAACZ2AgAADiQCAA6y6IP//h/4AgATc4tA= Date: Mon, 26 Jun 2017 03:47:09 +0000 Message-ID: <82F45D86ADE5454A95A89742C8D1410E3B69BC88@shsmsx102.ccr.corp.intel.com> References: <26FA93C7ED1EAA44AB77D62FBE1D27BA722C837C@IRSMSX108.ger.corp.intel.com> <067B569323FEB248B5CB480E1954F4346F4174FD@SHSMSX101.ccr.corp.intel.com> <59AF69C657FD0841A61C55336867B5B0721805ED@IRSMSX104.ger.corp.intel.com> <1814490.xy7qWLUraa@xps> <82F45D86ADE5454A95A89742C8D1410E3B698B4E@shsmsx102.ccr.corp.intel.com> <59AF69C657FD0841A61C55336867B5B072181A27@IRSMSX104.ger.corp.intel.com> In-Reply-To: <59AF69C657FD0841A61C55336867B5B072181A27@IRSMSX104.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 10.0.102.7 dlp-reaction: no-action x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-ci] script to determine target repo (was DPDK Lab) X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions <ci.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/ci>, <mailto:ci-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/ci/> List-Post: <mailto:ci@dpdk.org> List-Help: <mailto:ci-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/ci>, <mailto:ci-request@dpdk.org?subject=subscribe> X-List-Received-Date: Mon, 26 Jun 2017 03:47:14 -0000 > -----Original Message----- > From: Richardson, Bruce > Sent: Friday, June 23, 2017 5:30 PM > To: Xu, Qian Q <qian.q.xu@intel.com>; Thomas Monjalon > <thomas@monjalon.net>; Wei, FangfangX <fangfangx.wei@intel.com> > Cc: ci@dpdk.org; O'Driscoll, Tim <tim.odriscoll@intel.com>; Eugene Vorono= v > <eugene@mellanox.com> > Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab) >=20 >=20 >=20 > > -----Original Message----- > > From: Xu, Qian Q > > Sent: Friday, June 23, 2017 9:44 AM > > To: Thomas Monjalon <thomas@monjalon.net>; Wei, FangfangX > > <fangfangx.wei@intel.com> > > Cc: ci@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; > > O'Driscoll, Tim <tim.odriscoll@intel.com>; Eugene Voronov > > <eugene@mellanox.com> > > Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab) > > > > Thomas/Bruce > > 1. For determining the repo tree to target, I don't believe that we > > can ever > > > come up with a 100% accurate rule, as the tree to which a set is to > > > be applied can be difficult to determine, so it may be done on the > > > basis of > > on-list discussion. > > > A 90% accurate rule it what we may have to accept. > > > > -- Then if we find the performance issue, then maybe it's a false > > alarm due to apply to the wrong repo. So, we may face many false > > alarms according with the time. > > Then people may not treat the performance issue as a problem, so I > > still think we need to try 100% accurate to have a more trustable > > result when we send out the alarm. >=20 > I find that rather improbable, and not worth considering. For that to per= a > problem multiple unlikely events have to occur: > 1) we mis-identify the tree on which the set is to be applied (we should = be able > to get to 90% accuracy here) > 2) the patchset must apply cleanly to the "wrong" tree (this is reasonabl= y likely, > but it's still another condition that has to be met for us to have a prob= lem) > 3) the patchset has to cause a performance regression in the "wrong" tree > 4) but NOT cause a regression when in the right tree. >=20 > If we assume 90% accuracy of tree identification, optimistically that 90%= of > patches will apply to the wrong tree, that 5% of patches cause a performa= nce > regression (an overestimate IMHO), and that even 1/3 of those won't cause= a > performance regression in the right tree (a very overestimate IMHO, I wou= ld > expect just about none of them to even have this), it still means that on= ly about > 1 patch in 1000 will show as a false positive performance regression. >=20 > 0.1 (mis-identify) * 0.9 (applies ok) * 0.05 (regression) * 0.33 (no regr= ession) =3D > 0.0015, or 0.15% >=20 > So worst case, I still don't think we have a problem for the scenario you= describe. OK, Bruce, so the question is that how can we ensure 90% accuracy? How to c= heck if it's 90% or 80%?=20