Re: [dpdk-ci] Intel PerPatch Build

[Thomas Monjalon <thomas.monjalon@6wind.com>]

2016-11-30 06:49, Xu, Qian Q:
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > 2016-11-29 03:56, Xu, Qian Q:
> > > I feel we need to split the report.
> > > What do you think of having a report per OS? or a report per build?
> > > It would show easily how big is the failure by looking at the counters and
> > descriptions in patchwork.
> > > The email volume would be bigger but is it a problem?
> > > ----current report is for per patch build report, and for each patch, we now
> > have 18 builds. We will send out 1 build report for 1 patch.  If we send 18 reports
> > for 1 patch, then it may be too many reports.
> > 
> > Why is it too many reports?
> > 
> For each patch, we have 18 builds tests and what you want is to have 18 reports for each patch. If yes to it, normally we will have average ~30 patches(maybe not accurate) every day, then 
> We will have 30x18=540 mails report every day, if we have 100 patches one day, then we will have 1800 reports. So I mean it would be too many reports. It would be a disaster for the mailing list. 
> I'm not sure if you like thousands of mails from one mailing list. Or do I misunderstand your points?

I think this mailing-list does not aim to be human readable.
The per-patch reports feed patchwork and the daily reports should feed
another monitoring tool.
That's why I wonder wether it is important to minimize the number of emails.

> > In one build test, there can be several errors (even if we stop at the first error).
> > And one error can be seen in several build tests.
> > So we cannot really count the number of real errors, but we can count the
> > number of tests which are failing.
> > Yes we could add the number of failed tests in a test report.
> 
> Yes, we can show the total numbers of failed tests in one test report, and it's also very simple. 
> 
> > I just thought that it would be simpler to understand if sending 1 report per test.
> 
> Not sure if it's simpler but I can see mail spam if 1 report per test.

It is not a spam if you are not registered to the mailing list :)

> > An example of the benefit of splitting:
> > An error with recent GCC is a failure.
> > An error with ICC or an old GCC may be just a warning.
> 
> Currently ICC and old GCC is not included in the per patch build system. 
> 
> > Finer is the grain of the reports, better is the patchwork overview.
> 
> Agreed, but maybe there is other way. Fetch the failure numbers from the report. I also wonder if in future, there would be IBM or ARM build or even more regression functional test report. 
> If we put Intel IA build as 18 Lines for tracking how big the failure, then IBM or ARM may have more lines, then how about functional test report? If we run 100 functional tests per patch, then 
> We need 100 test reports for each patch, then if 100 patches for one day, then we have 10k mails one day......
> 
> From our monitor on the build error, not many errors currently, most failures are due to the apply patch failures, and some configurations(gcc-debug, gcc-combined) build errors. Build is relatively 
> Stable now. So not sure how much value we can get from the split the patch report. Before we provide the patchset report, and now we have already split it to per patch report, which is also an 
> Improved grain of the report.

Yes good point. OK to continue with only one report for the Intel build tests
as errors are rare.

> > Today, we have few tests only. But later we could have more and more test
> > instances sending the same kind of reports.
> > 
> > Another example:
> > If a test is failing for some reasons, it will fail for every patches.
> > We must have a way to ignore it when reading the results.
> 
> How to ignore it? Don't send the failure report??

I meant visually ignore it in the list of tests for a patch.
But there is probably a better approach, like comparing with the results
of the previous tests when making the report.
I suggest to keep it in mind for later if we see this kind of issue.

> We need analyze the failure reason then decide if the failure is same
> for every patch, so the analysis is needed here.

> > If it is a separate report, it is easier to ignore.
> 
> 
> 
> > PS: please Qian, could you configure your email client to use reply quoting?
> Yes, is it better now? 

Yes a lot better, thanks.