Re: [dpdk-ci] Intel PerPatch Build

[Thomas Monjalon <thomas.monjalon@6wind.com>]

2016-11-30 09:25, Xu, Qian Q:
> Then the conclusion is that we just kept current model, 1 report for 1 patch, right?

Yes
Thanks for the discussion
(I keep this discussion in mind in case it needs some improvement)

> Also keep in mind some things about the error comparison, we can discuss it again if we see many these issues. 

Yes

> As to the next step, I think Fangfang can check how to update the per patch build result to the patchwork to make the result more visible/readable in website.

Yes please.
I'm also waiting to apply the patches. Should I apply them right now?


> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > Sent: Wednesday, November 30, 2016 4:33 PM
> > To: Xu, Qian Q <qian.q.xu@intel.com>
> > Cc: ci@dpdk.org; Wei, FangfangX <fangfangx.wei@intel.com>; Liu, Yong
> > <yong.liu@intel.com>
> > Subject: Re: Intel PerPatch Build
> > 
> > 2016-11-30 06:49, Xu, Qian Q:
> > > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > > > 2016-11-29 03:56, Xu, Qian Q:
> > > > > I feel we need to split the report.
> > > > > What do you think of having a report per OS? or a report per build?
> > > > > It would show easily how big is the failure by looking at the
> > > > > counters and
> > > > descriptions in patchwork.
> > > > > The email volume would be bigger but is it a problem?
> > > > > ----current report is for per patch build report, and for each
> > > > > patch, we now
> > > > have 18 builds. We will send out 1 build report for 1 patch.  If we
> > > > send 18 reports for 1 patch, then it may be too many reports.
> > > >
> > > > Why is it too many reports?
> > > >
> > > For each patch, we have 18 builds tests and what you want is to have
> > > 18 reports for each patch. If yes to it, normally we will have average ~30
> > patches(maybe not accurate) every day, then We will have 30x18=540 mails
> > report every day, if we have 100 patches one day, then we will have 1800
> > reports. So I mean it would be too many reports. It would be a disaster for the
> > mailing list.
> > > I'm not sure if you like thousands of mails from one mailing list. Or do I
> > misunderstand your points?
> > 
> > I think this mailing-list does not aim to be human readable.
> > The per-patch reports feed patchwork and the daily reports should feed another
> > monitoring tool.
> > That's why I wonder wether it is important to minimize the number of emails.
> > 
> > > > In one build test, there can be several errors (even if we stop at the first
> > error).
> > > > And one error can be seen in several build tests.
> > > > So we cannot really count the number of real errors, but we can
> > > > count the number of tests which are failing.
> > > > Yes we could add the number of failed tests in a test report.
> > >
> > > Yes, we can show the total numbers of failed tests in one test report, and it's
> > also very simple.
> > >
> > > > I just thought that it would be simpler to understand if sending 1 report per
> > test.
> > >
> > > Not sure if it's simpler but I can see mail spam if 1 report per test.
> > 
> > It is not a spam if you are not registered to the mailing list :)
> > 
> > > > An example of the benefit of splitting:
> > > > An error with recent GCC is a failure.
> > > > An error with ICC or an old GCC may be just a warning.
> > >
> > > Currently ICC and old GCC is not included in the per patch build system.
> > >
> > > > Finer is the grain of the reports, better is the patchwork overview.
> > >
> > > Agreed, but maybe there is other way. Fetch the failure numbers from the
> > report. I also wonder if in future, there would be IBM or ARM build or even more
> > regression functional test report.
> > > If we put Intel IA build as 18 Lines for tracking how big the failure,
> > > then IBM or ARM may have more lines, then how about functional test report?
> > If we run 100 functional tests per patch, then We need 100 test reports for each
> > patch, then if 100 patches for one day, then we have 10k mails one day......
> > >
> > > From our monitor on the build error, not many errors currently, most
> > > failures are due to the apply patch failures, and some
> > > configurations(gcc-debug, gcc-combined) build errors. Build is relatively Stable
> > now. So not sure how much value we can get from the split the patch report.
> > Before we provide the patchset report, and now we have already split it to per
> > patch report, which is also an Improved grain of the report.
> > 
> > Yes good point. OK to continue with only one report for the Intel build tests as
> > errors are rare.
> > 
> > > > Today, we have few tests only. But later we could have more and more
> > > > test instances sending the same kind of reports.
> > > >
> > > > Another example:
> > > > If a test is failing for some reasons, it will fail for every patches.
> > > > We must have a way to ignore it when reading the results.
> > >
> > > How to ignore it? Don't send the failure report??
> > 
> > I meant visually ignore it in the list of tests for a patch.
> > But there is probably a better approach, like comparing with the results of the
> > previous tests when making the report.
> > I suggest to keep it in mind for later if we see this kind of issue.
> > 
> > > We need analyze the failure reason then decide if the failure is same
> > > for every patch, so the analysis is needed here.
> > 
> > > > If it is a separate report, it is easier to ignore.
> > >
> > >
> > >
> > > > PS: please Qian, could you configure your email client to use reply quoting?
> > > Yes, is it better now?
> > 
> > Yes a lot better, thanks.