Re: [dpdk-ci] script to determine target repo (was DPDK Lab)

[Jerin Jacob <jerin.jacob@caviumnetworks.com>]

-----Original Message-----
> Date: Fri, 23 Jun 2017 09:30:28 +0000
> From: "Richardson, Bruce" <bruce.richardson@intel.com>
> To: "Xu, Qian Q" <qian.q.xu@intel.com>, Thomas Monjalon
>  <thomas@monjalon.net>, "Wei, FangfangX" <fangfangx.wei@intel.com>
> CC: "ci@dpdk.org" <ci@dpdk.org>, "O'Driscoll, Tim"
>  <tim.odriscoll@intel.com>, Eugene Voronov <eugene@mellanox.com>
> Subject: Re: [dpdk-ci] script to determine target repo (was DPDK Lab)
> 
> 
> 
> > -----Original Message-----
> > From: Xu, Qian Q
> > Sent: Friday, June 23, 2017 9:44 AM
> > To: Thomas Monjalon <thomas@monjalon.net>; Wei, FangfangX
> > <fangfangx.wei@intel.com>
> > Cc: ci@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> > O'Driscoll, Tim <tim.odriscoll@intel.com>; Eugene Voronov
> > <eugene@mellanox.com>
> > Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab)
> > 
> > Thomas/Bruce
> > 1. For determining the repo tree to target, I don't believe that we can
> > ever
> > > come up with a 100% accurate rule, as the tree to which a set is to be
> > > applied can be difficult to determine, so it may be done on the basis of
> > on-list discussion.
> > > A 90% accurate rule it what we may have to accept.
> > 
> > -- Then if we find the performance issue, then maybe it's a false alarm
> > due to apply to the wrong repo. So, we may face many false alarms
> > according with the time.
> > Then people may not treat the performance issue as a problem, so I still
> > think we need to try 100% accurate to have a more trustable result when we
> > send out the alarm.
> 
> I find that rather improbable, and not worth considering. For that to per a problem multiple unlikely events have to occur:
> 1) we mis-identify the tree on which the set is to be applied (we should be able to get to 90% accuracy here)
> 2) the patchset must apply cleanly to the "wrong" tree (this is reasonably likely, but it's still another condition that has to be met for us to have a problem)
> 3) the patchset has to cause a performance regression in the "wrong" tree
> 4) but NOT cause a regression when in the right tree.
> 
> If we assume 90% accuracy of tree identification, optimistically that 90% of patches will apply to the wrong tree, that 5% of patches cause a performance regression (an overestimate IMHO), and that even 1/3 of those won't cause a performance regression in the right tree (a very overestimate IMHO, I would expect just about none of them to even have this), it still means that only about 1 patch in 1000 will show as a false positive performance regression.
> 
> 0.1 (mis-identify) * 0.9 (applies ok) * 0.05 (regression) * 0.33 (no regression) = 0.0015, or 0.15%
> 
> So worst case, I still don't think we have a problem for the scenario you describe.

Another option could be to fix the problem at source. Where the targeted
script can run at the source, identify the target repo and embedded that
information in 1st patch in series. I guess, that will provide a means
(user can change the target repo at source if required)
to fixup the remaining 10% accuracy.



> > 
> > 
> > > -----Original Message-----
> > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > Sent: Wednesday, June 21, 2017 4:33 PM
> > > To: Wei, FangfangX <fangfangx.wei@intel.com>
> > > Cc: ci@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Xu,
> > > Qian Q <qian.q.xu@intel.com>; O'Driscoll, Tim
> > > <tim.odriscoll@intel.com>; Eugene Voronov <eugene@mellanox.com>
> > > Subject: Re: [dpdk-ci] script to determine target repo (was DPDK Lab)
> > >
> > > I agree with Bruce.
> > >
> > > Sorry for not having written the scripts yet.
> > > Someone else in Mellanox should do it in July.
> > > In the meantime, do not hesitate to share your code if it speed up
> > things.
> > >
> > > Thanks a lot
> > >
> > >
> > > 21/06/2017 10:20, Richardson, Bruce:
> > > > Hi Fangfang,
> > > >
> > > > My comments on the questions you asked:
> > > >
> > > > 1. For determining the repo tree to target, I don't believe that we
> > > > can ever
> > > come up with a 100% accurate rule, as the tree to which a set is to be
> > > applied can be difficult to determine, so it may be done on the basis of
> > on-list discussion.
> > > A 90% accurate rule it what we may have to accept. However, since
> > > applying a patchset to a tree should not be a time-consuming
> > > operation, I suggest any script produce a list of possible trees in
> > > priority order to try. If not net, then try main, etc. etc.
> > > >
> > > > 2. Using the order from patch titles is correct.
> > > >
> > > > /Bruce