From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <bruce.richardson@intel.com>
Received: from mga07.intel.com (mga07.intel.com [134.134.136.100])
 by dpdk.org (Postfix) with ESMTP id 6262E3252
 for <ci@dpdk.org>; Fri, 23 Jun 2017 11:30:32 +0200 (CEST)
Received: from fmsmga006.fm.intel.com ([10.253.24.20])
 by orsmga105.jf.intel.com with ESMTP; 23 Jun 2017 02:30:30 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.39,377,1493708400"; d="scan'208";a="118410710"
Received: from irsmsx103.ger.corp.intel.com ([163.33.3.157])
 by fmsmga006.fm.intel.com with ESMTP; 23 Jun 2017 02:30:29 -0700
Received: from irsmsx104.ger.corp.intel.com ([169.254.5.26]) by
 IRSMSX103.ger.corp.intel.com ([169.254.3.9]) with mapi id 14.03.0319.002;
 Fri, 23 Jun 2017 10:30:28 +0100
From: "Richardson, Bruce" <bruce.richardson@intel.com>
To: "Xu, Qian Q" <qian.q.xu@intel.com>, Thomas Monjalon <thomas@monjalon.net>, 
 "Wei, FangfangX" <fangfangx.wei@intel.com>
CC: "ci@dpdk.org" <ci@dpdk.org>, "O'Driscoll, Tim" <tim.odriscoll@intel.com>, 
 Eugene Voronov <eugene@mellanox.com>
Thread-Topic: [dpdk-ci] script to determine target repo (was DPDK Lab)
Thread-Index: AQHStCGpMpJWjq4ArUimVAHX1/Hj4aHVn+UAgAAHowCAWaRFAIAAGZEQ///zlgCAAyepAIAAGwoQ
Date: Fri, 23 Jun 2017 09:30:28 +0000
Message-ID: <59AF69C657FD0841A61C55336867B5B072181A27@IRSMSX104.ger.corp.intel.com>
References: <26FA93C7ED1EAA44AB77D62FBE1D27BA722C837C@IRSMSX108.ger.corp.intel.com>
 <067B569323FEB248B5CB480E1954F4346F4174FD@SHSMSX101.ccr.corp.intel.com>
 <59AF69C657FD0841A61C55336867B5B0721805ED@IRSMSX104.ger.corp.intel.com>
 <1814490.xy7qWLUraa@xps>
 <82F45D86ADE5454A95A89742C8D1410E3B698B4E@shsmsx102.ccr.corp.intel.com>
In-Reply-To: <82F45D86ADE5454A95A89742C8D1410E3B698B4E@shsmsx102.ccr.corp.intel.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiOWM2ZjkyMTAtZjFlYy00ZjVhLThkYTktYjZlNzAyOWI5MTNjIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IjB2dHJKOHFhcU9wRGtYaUVkcnl4TjBhc3ZZZWpuc1RaMTRsYzRPRjc0cUE9In0=
x-ctpclassification: CTP_IC
dlp-product: dlpe-windows
dlp-version: 10.0.102.7
dlp-reaction: no-action
x-originating-ip: [163.33.239.182]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: Re: [dpdk-ci] script to determine target repo (was DPDK Lab)
X-BeenThere: ci@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK CI discussions <ci.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/ci>,
 <mailto:ci-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/ci/>
List-Post: <mailto:ci@dpdk.org>
List-Help: <mailto:ci-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/ci>,
 <mailto:ci-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 23 Jun 2017 09:30:33 -0000



> -----Original Message-----
> From: Xu, Qian Q
> Sent: Friday, June 23, 2017 9:44 AM
> To: Thomas Monjalon <thomas@monjalon.net>; Wei, FangfangX
> <fangfangx.wei@intel.com>
> Cc: ci@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> O'Driscoll, Tim <tim.odriscoll@intel.com>; Eugene Voronov
> <eugene@mellanox.com>
> Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab)
>=20
> Thomas/Bruce
> 1. For determining the repo tree to target, I don't believe that we can
> ever
> > come up with a 100% accurate rule, as the tree to which a set is to be
> > applied can be difficult to determine, so it may be done on the basis o=
f
> on-list discussion.
> > A 90% accurate rule it what we may have to accept.
>=20
> -- Then if we find the performance issue, then maybe it's a false alarm
> due to apply to the wrong repo. So, we may face many false alarms
> according with the time.
> Then people may not treat the performance issue as a problem, so I still
> think we need to try 100% accurate to have a more trustable result when w=
e
> send out the alarm.

I find that rather improbable, and not worth considering. For that to per a=
 problem multiple unlikely events have to occur:
1) we mis-identify the tree on which the set is to be applied (we should be=
 able to get to 90% accuracy here)
2) the patchset must apply cleanly to the "wrong" tree (this is reasonably =
likely, but it's still another condition that has to be met for us to have =
a problem)
3) the patchset has to cause a performance regression in the "wrong" tree
4) but NOT cause a regression when in the right tree.

If we assume 90% accuracy of tree identification, optimistically that 90% o=
f patches will apply to the wrong tree, that 5% of patches cause a performa=
nce regression (an overestimate IMHO), and that even 1/3 of those won't cau=
se a performance regression in the right tree (a very overestimate IMHO, I =
would expect just about none of them to even have this), it still means tha=
t only about 1 patch in 1000 will show as a false positive performance regr=
ession.

0.1 (mis-identify) * 0.9 (applies ok) * 0.05 (regression) * 0.33 (no regres=
sion) =3D 0.0015, or 0.15%

So worst case, I still don't think we have a problem for the scenario you d=
escribe.

/Bruce

>=20
>=20
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > Sent: Wednesday, June 21, 2017 4:33 PM
> > To: Wei, FangfangX <fangfangx.wei@intel.com>
> > Cc: ci@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Xu,
> > Qian Q <qian.q.xu@intel.com>; O'Driscoll, Tim
> > <tim.odriscoll@intel.com>; Eugene Voronov <eugene@mellanox.com>
> > Subject: Re: [dpdk-ci] script to determine target repo (was DPDK Lab)
> >
> > I agree with Bruce.
> >
> > Sorry for not having written the scripts yet.
> > Someone else in Mellanox should do it in July.
> > In the meantime, do not hesitate to share your code if it speed up
> things.
> >
> > Thanks a lot
> >
> >
> > 21/06/2017 10:20, Richardson, Bruce:
> > > Hi Fangfang,
> > >
> > > My comments on the questions you asked:
> > >
> > > 1. For determining the repo tree to target, I don't believe that we
> > > can ever
> > come up with a 100% accurate rule, as the tree to which a set is to be
> > applied can be difficult to determine, so it may be done on the basis o=
f
> on-list discussion.
> > A 90% accurate rule it what we may have to accept. However, since
> > applying a patchset to a tree should not be a time-consuming
> > operation, I suggest any script produce a list of possible trees in
> > priority order to try. If not net, then try main, etc. etc.
> > >
> > > 2. Using the order from patch titles is correct.
> > >
> > > /Bruce