Re: [dpdk-ci] [dpdk-dev] [PATCH 00/14] Unit tests fixes for CI

Re: [dpdk-ci] [dpdk-dev] [PATCH 00/14] Unit tests fixes for CI

[Thomas Monjalon]

04/06/2019 17:49, Michael Santana Francisco:
> On 6/4/19 4:59 AM, David Marchand wrote:
> > - the "perf" tests are taking way too long for my taste,
> 
> We should still fix them. However I don't know if we should be running 
> the perf test for every job and every patch on travis. It takes too 
> long. The travis queue will be delayed too far behind for it to be of 
> any use.
> 
> OTOH we could have one job as part of the travis build dedicated to 
> running tests (or just perf test). It's still time consuming but better 
> than running the test on every travis job. For this to work we would 
> need to decreased the timeout for the perf tests as the timeout for it 
> and the travis are both 10 minutes

+Cc ci@dpdk.org

I don't think we should run the perf tests in basic CI like Travis.
We can run perf tests if the purpose is to compare the performance
with previous releases, as some other tests in the community lab.

Re: [dpdk-ci] [dpdk-dev] [PATCH 00/14] Unit tests fixes for CI

[Aaron Conole]

Thomas Monjalon <thomas@monjalon.net> writes:

> 04/06/2019 17:49, Michael Santana Francisco:
>> On 6/4/19 4:59 AM, David Marchand wrote:
>> > - the "perf" tests are taking way too long for my taste,

+1 here.

>> 
>> We should still fix them. However I don't know if we should be running 
>> the perf test for every job and every patch on travis. It takes too 
>> long. The travis queue will be delayed too far behind for it to be of 
>> any use.
>> 
>> OTOH we could have one job as part of the travis build dedicated to 
>> running tests (or just perf test). It's still time consuming but better 
>> than running the test on every travis job. For this to work we would 
>> need to decreased the timeout for the perf tests as the timeout for it 
>> and the travis are both 10 minutes
>
> +Cc ci@dpdk.org
>
> I don't think we should run the perf tests in basic CI like Travis.
> We can run perf tests if the purpose is to compare the performance
> with previous releases, as some other tests in the community lab.

+1 - some of the perf tests aren't going to complete in any sort of
reasonable time.  While we could claim it's a separate problem, we
should also not enable something that will make the travis runs so much
longer.

I do like the idea of running tests in the travis build, and I think it
would make sense to have just a single job for it (or maybe one for
clang and one for gcc?  maybe even that is overkill).

I would rather not do performance tests during the travis run, though.
It doesn't really make sense.  Travis isn't any kind of an 'optimized'
environment, so I don't know what 'performance' should mean.