From: "Xu, Qian Q" <qian.q.xu@intel.com>
To: "Richardson, Bruce" <bruce.richardson@intel.com>,
Thomas Monjalon <thomas@monjalon.net>,
"Wei, FangfangX" <fangfangx.wei@intel.com>
Cc: "ci@dpdk.org" <ci@dpdk.org>,
"O'Driscoll, Tim" <tim.odriscoll@intel.com>,
Eugene Voronov <eugene@mellanox.com>
Subject: Re: [dpdk-ci] script to determine target repo (was DPDK Lab)
Date: Mon, 26 Jun 2017 03:47:09 +0000 [thread overview]
Message-ID: <82F45D86ADE5454A95A89742C8D1410E3B69BC88@shsmsx102.ccr.corp.intel.com> (raw)
In-Reply-To: <59AF69C657FD0841A61C55336867B5B072181A27@IRSMSX104.ger.corp.intel.com>
> -----Original Message-----
> From: Richardson, Bruce
> Sent: Friday, June 23, 2017 5:30 PM
> To: Xu, Qian Q <qian.q.xu@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Wei, FangfangX <fangfangx.wei@intel.com>
> Cc: ci@dpdk.org; O'Driscoll, Tim <tim.odriscoll@intel.com>; Eugene Voronov
> <eugene@mellanox.com>
> Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab)
>
>
>
> > -----Original Message-----
> > From: Xu, Qian Q
> > Sent: Friday, June 23, 2017 9:44 AM
> > To: Thomas Monjalon <thomas@monjalon.net>; Wei, FangfangX
> > <fangfangx.wei@intel.com>
> > Cc: ci@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> > O'Driscoll, Tim <tim.odriscoll@intel.com>; Eugene Voronov
> > <eugene@mellanox.com>
> > Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab)
> >
> > Thomas/Bruce
> > 1. For determining the repo tree to target, I don't believe that we
> > can ever
> > > come up with a 100% accurate rule, as the tree to which a set is to
> > > be applied can be difficult to determine, so it may be done on the
> > > basis of
> > on-list discussion.
> > > A 90% accurate rule it what we may have to accept.
> >
> > -- Then if we find the performance issue, then maybe it's a false
> > alarm due to apply to the wrong repo. So, we may face many false
> > alarms according with the time.
> > Then people may not treat the performance issue as a problem, so I
> > still think we need to try 100% accurate to have a more trustable
> > result when we send out the alarm.
>
> I find that rather improbable, and not worth considering. For that to per a
> problem multiple unlikely events have to occur:
> 1) we mis-identify the tree on which the set is to be applied (we should be able
> to get to 90% accuracy here)
> 2) the patchset must apply cleanly to the "wrong" tree (this is reasonably likely,
> but it's still another condition that has to be met for us to have a problem)
> 3) the patchset has to cause a performance regression in the "wrong" tree
> 4) but NOT cause a regression when in the right tree.
>
> If we assume 90% accuracy of tree identification, optimistically that 90% of
> patches will apply to the wrong tree, that 5% of patches cause a performance
> regression (an overestimate IMHO), and that even 1/3 of those won't cause a
> performance regression in the right tree (a very overestimate IMHO, I would
> expect just about none of them to even have this), it still means that only about
> 1 patch in 1000 will show as a false positive performance regression.
>
> 0.1 (mis-identify) * 0.9 (applies ok) * 0.05 (regression) * 0.33 (no regression) =
> 0.0015, or 0.15%
>
> So worst case, I still don't think we have a problem for the scenario you describe.
OK, Bruce, so the question is that how can we ensure 90% accuracy? How to check if it's 90% or 80%?
next prev parent reply other threads:[~2017-06-26 3:47 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <26FA93C7ED1EAA44AB77D62FBE1D27BA722C837C@IRSMSX108.ger.corp.intel.com>
[not found] ` <HE1PR0501MB21370D6ACD0931D36925D085BD0D0@HE1PR0501MB2137.eurprd05.prod.outlook.com>
[not found] ` <82F45D86ADE5454A95A89742C8D1410E3B5BA2BB@shsmsx102.ccr.corp.intel.com>
[not found] ` <1564461.MHdNEdeViC@xps13>
2017-04-13 5:25 ` [dpdk-ci] DPDK Lab Xu, Qian Q
2017-04-13 6:46 ` [dpdk-ci] script to determine target repo (was DPDK Lab) Thomas Monjalon
2017-04-25 6:23 ` Xu, Qian Q
2017-04-25 6:24 ` Xu, Qian Q
2017-04-25 6:37 ` Wei, FangfangX
2017-04-25 6:41 ` Wei, FangfangX
2017-04-25 6:48 ` Thomas Monjalon
2017-04-25 6:50 ` Thomas Monjalon
2017-06-21 7:45 ` Wei, FangfangX
2017-06-21 8:20 ` Richardson, Bruce
2017-06-21 8:33 ` Thomas Monjalon
2017-06-23 8:43 ` Xu, Qian Q
2017-06-23 9:30 ` Richardson, Bruce
2017-06-26 3:47 ` Xu, Qian Q [this message]
2017-06-26 6:23 ` Jerin Jacob
2017-06-26 13:26 ` Xu, Qian Q
2017-06-30 8:41 ` Wei, FangfangX
2017-06-30 9:22 ` Thomas Monjalon
2017-04-13 8:24 ` [dpdk-ci] DPDK Lab Richardson, Bruce
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=82F45D86ADE5454A95A89742C8D1410E3B69BC88@shsmsx102.ccr.corp.intel.com \
--to=qian.q.xu@intel.com \
--cc=bruce.richardson@intel.com \
--cc=ci@dpdk.org \
--cc=eugene@mellanox.com \
--cc=fangfangx.wei@intel.com \
--cc=thomas@monjalon.net \
--cc=tim.odriscoll@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).