DPDK CI discussions
 help / color / mirror / Atom feed
From: "Richardson, Bruce" <bruce.richardson@intel.com>
To: "Xu, Qian Q" <qian.q.xu@intel.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	 "Wei, FangfangX" <fangfangx.wei@intel.com>
Cc: "ci@dpdk.org" <ci@dpdk.org>,
	"O'Driscoll, Tim" <tim.odriscoll@intel.com>,
	 Eugene Voronov <eugene@mellanox.com>
Subject: Re: [dpdk-ci] script to determine target repo (was DPDK Lab)
Date: Fri, 23 Jun 2017 09:30:28 +0000	[thread overview]
Message-ID: <59AF69C657FD0841A61C55336867B5B072181A27@IRSMSX104.ger.corp.intel.com> (raw)
In-Reply-To: <82F45D86ADE5454A95A89742C8D1410E3B698B4E@shsmsx102.ccr.corp.intel.com>



> -----Original Message-----
> From: Xu, Qian Q
> Sent: Friday, June 23, 2017 9:44 AM
> To: Thomas Monjalon <thomas@monjalon.net>; Wei, FangfangX
> <fangfangx.wei@intel.com>
> Cc: ci@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> O'Driscoll, Tim <tim.odriscoll@intel.com>; Eugene Voronov
> <eugene@mellanox.com>
> Subject: RE: [dpdk-ci] script to determine target repo (was DPDK Lab)
> 
> Thomas/Bruce
> 1. For determining the repo tree to target, I don't believe that we can
> ever
> > come up with a 100% accurate rule, as the tree to which a set is to be
> > applied can be difficult to determine, so it may be done on the basis of
> on-list discussion.
> > A 90% accurate rule it what we may have to accept.
> 
> -- Then if we find the performance issue, then maybe it's a false alarm
> due to apply to the wrong repo. So, we may face many false alarms
> according with the time.
> Then people may not treat the performance issue as a problem, so I still
> think we need to try 100% accurate to have a more trustable result when we
> send out the alarm.

I find that rather improbable, and not worth considering. For that to per a problem multiple unlikely events have to occur:
1) we mis-identify the tree on which the set is to be applied (we should be able to get to 90% accuracy here)
2) the patchset must apply cleanly to the "wrong" tree (this is reasonably likely, but it's still another condition that has to be met for us to have a problem)
3) the patchset has to cause a performance regression in the "wrong" tree
4) but NOT cause a regression when in the right tree.

If we assume 90% accuracy of tree identification, optimistically that 90% of patches will apply to the wrong tree, that 5% of patches cause a performance regression (an overestimate IMHO), and that even 1/3 of those won't cause a performance regression in the right tree (a very overestimate IMHO, I would expect just about none of them to even have this), it still means that only about 1 patch in 1000 will show as a false positive performance regression.

0.1 (mis-identify) * 0.9 (applies ok) * 0.05 (regression) * 0.33 (no regression) = 0.0015, or 0.15%

So worst case, I still don't think we have a problem for the scenario you describe.

/Bruce

> 
> 
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > Sent: Wednesday, June 21, 2017 4:33 PM
> > To: Wei, FangfangX <fangfangx.wei@intel.com>
> > Cc: ci@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Xu,
> > Qian Q <qian.q.xu@intel.com>; O'Driscoll, Tim
> > <tim.odriscoll@intel.com>; Eugene Voronov <eugene@mellanox.com>
> > Subject: Re: [dpdk-ci] script to determine target repo (was DPDK Lab)
> >
> > I agree with Bruce.
> >
> > Sorry for not having written the scripts yet.
> > Someone else in Mellanox should do it in July.
> > In the meantime, do not hesitate to share your code if it speed up
> things.
> >
> > Thanks a lot
> >
> >
> > 21/06/2017 10:20, Richardson, Bruce:
> > > Hi Fangfang,
> > >
> > > My comments on the questions you asked:
> > >
> > > 1. For determining the repo tree to target, I don't believe that we
> > > can ever
> > come up with a 100% accurate rule, as the tree to which a set is to be
> > applied can be difficult to determine, so it may be done on the basis of
> on-list discussion.
> > A 90% accurate rule it what we may have to accept. However, since
> > applying a patchset to a tree should not be a time-consuming
> > operation, I suggest any script produce a list of possible trees in
> > priority order to try. If not net, then try main, etc. etc.
> > >
> > > 2. Using the order from patch titles is correct.
> > >
> > > /Bruce

  reply	other threads:[~2017-06-23  9:30 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <26FA93C7ED1EAA44AB77D62FBE1D27BA722C837C@IRSMSX108.ger.corp.intel.com>
     [not found] ` <HE1PR0501MB21370D6ACD0931D36925D085BD0D0@HE1PR0501MB2137.eurprd05.prod.outlook.com>
     [not found]   ` <82F45D86ADE5454A95A89742C8D1410E3B5BA2BB@shsmsx102.ccr.corp.intel.com>
     [not found]     ` <1564461.MHdNEdeViC@xps13>
2017-04-13  5:25       ` [dpdk-ci] DPDK Lab Xu, Qian Q
2017-04-13  6:46         ` [dpdk-ci] script to determine target repo (was DPDK Lab) Thomas Monjalon
2017-04-25  6:23           ` Xu, Qian Q
2017-04-25  6:24             ` Xu, Qian Q
2017-04-25  6:37             ` Wei, FangfangX
2017-04-25  6:41               ` Wei, FangfangX
2017-04-25  6:48               ` Thomas Monjalon
2017-04-25  6:50             ` Thomas Monjalon
2017-06-21  7:45               ` Wei, FangfangX
2017-06-21  8:20                 ` Richardson, Bruce
2017-06-21  8:33                   ` Thomas Monjalon
2017-06-23  8:43                     ` Xu, Qian Q
2017-06-23  9:30                       ` Richardson, Bruce [this message]
2017-06-26  3:47                         ` Xu, Qian Q
2017-06-26  6:23                         ` Jerin Jacob
2017-06-26 13:26                           ` Xu, Qian Q
2017-06-30  8:41                     ` Wei, FangfangX
2017-06-30  9:22                       ` Thomas Monjalon
2017-04-13  8:24         ` [dpdk-ci] DPDK Lab Richardson, Bruce

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=59AF69C657FD0841A61C55336867B5B072181A27@IRSMSX104.ger.corp.intel.com \
    --to=bruce.richardson@intel.com \
    --cc=ci@dpdk.org \
    --cc=eugene@mellanox.com \
    --cc=fangfangx.wei@intel.com \
    --cc=qian.q.xu@intel.com \
    --cc=thomas@monjalon.net \
    --cc=tim.odriscoll@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).