Re: [dpdk-ci] Intel PerPatch Build

["Xu, Qian Q" <qian.q.xu@intel.com>]

Then the conclusion is that we just kept current model, 1 report for 1 patch, right? 
Also keep in mind some things about the error comparison, we can discuss it again if we see many these issues.  
As to the next step, I think Fangfang can check how to update the per patch build result to the patchwork to make the result more visible/readable in website. 

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Wednesday, November 30, 2016 4:33 PM
> To: Xu, Qian Q <qian.q.xu@intel.com>
> Cc: ci@dpdk.org; Wei, FangfangX <fangfangx.wei@intel.com>; Liu, Yong
> <yong.liu@intel.com>
> Subject: Re: Intel PerPatch Build
> 
> 2016-11-30 06:49, Xu, Qian Q:
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > 2016-11-29 03:56, Xu, Qian Q:
> > > > I feel we need to split the report.
> > > > What do you think of having a report per OS? or a report per build?
> > > > It would show easily how big is the failure by looking at the
> > > > counters and
> > > descriptions in patchwork.
> > > > The email volume would be bigger but is it a problem?
> > > > ----current report is for per patch build report, and for each
> > > > patch, we now
> > > have 18 builds. We will send out 1 build report for 1 patch.  If we
> > > send 18 reports for 1 patch, then it may be too many reports.
> > >
> > > Why is it too many reports?
> > >
> > For each patch, we have 18 builds tests and what you want is to have
> > 18 reports for each patch. If yes to it, normally we will have average ~30
> patches(maybe not accurate) every day, then We will have 30x18=540 mails
> report every day, if we have 100 patches one day, then we will have 1800
> reports. So I mean it would be too many reports. It would be a disaster for the
> mailing list.
> > I'm not sure if you like thousands of mails from one mailing list. Or do I
> misunderstand your points?
> 
> I think this mailing-list does not aim to be human readable.
> The per-patch reports feed patchwork and the daily reports should feed another
> monitoring tool.
> That's why I wonder wether it is important to minimize the number of emails.
> 
> > > In one build test, there can be several errors (even if we stop at the first
> error).
> > > And one error can be seen in several build tests.
> > > So we cannot really count the number of real errors, but we can
> > > count the number of tests which are failing.
> > > Yes we could add the number of failed tests in a test report.
> >
> > Yes, we can show the total numbers of failed tests in one test report, and it's
> also very simple.
> >
> > > I just thought that it would be simpler to understand if sending 1 report per
> test.
> >
> > Not sure if it's simpler but I can see mail spam if 1 report per test.
> 
> It is not a spam if you are not registered to the mailing list :)
> 
> > > An example of the benefit of splitting:
> > > An error with recent GCC is a failure.
> > > An error with ICC or an old GCC may be just a warning.
> >
> > Currently ICC and old GCC is not included in the per patch build system.
> >
> > > Finer is the grain of the reports, better is the patchwork overview.
> >
> > Agreed, but maybe there is other way. Fetch the failure numbers from the
> report. I also wonder if in future, there would be IBM or ARM build or even more
> regression functional test report.
> > If we put Intel IA build as 18 Lines for tracking how big the failure,
> > then IBM or ARM may have more lines, then how about functional test report?
> If we run 100 functional tests per patch, then We need 100 test reports for each
> patch, then if 100 patches for one day, then we have 10k mails one day......
> >
> > From our monitor on the build error, not many errors currently, most
> > failures are due to the apply patch failures, and some
> > configurations(gcc-debug, gcc-combined) build errors. Build is relatively Stable
> now. So not sure how much value we can get from the split the patch report.
> Before we provide the patchset report, and now we have already split it to per
> patch report, which is also an Improved grain of the report.
> 
> Yes good point. OK to continue with only one report for the Intel build tests as
> errors are rare.
> 
> > > Today, we have few tests only. But later we could have more and more
> > > test instances sending the same kind of reports.
> > >
> > > Another example:
> > > If a test is failing for some reasons, it will fail for every patches.
> > > We must have a way to ignore it when reading the results.
> >
> > How to ignore it? Don't send the failure report??
> 
> I meant visually ignore it in the list of tests for a patch.
> But there is probably a better approach, like comparing with the results of the
> previous tests when making the report.
> I suggest to keep it in mind for later if we see this kind of issue.
> 
> > We need analyze the failure reason then decide if the failure is same
> > for every patch, so the analysis is needed here.
> 
> > > If it is a separate report, it is easier to ignore.
> >
> >
> >
> > > PS: please Qian, could you configure your email client to use reply quoting?
> > Yes, is it better now?
> 
> Yes a lot better, thanks.