Re: [dpdk-dev] [PATCH v2 2/2] ci: add test suite run without hugepage

[Ruifeng Wang <Ruifeng.Wang@arm.com>]

> -----Original Message-----
> From: Aaron Conole <aconole@redhat.com>
> Sent: Thursday, March 5, 2020 01:31
> To: Ruifeng Wang <Ruifeng.Wang@arm.com>
> Cc: maicolgabriel@hotmail.com; bruce.richardson@intel.com; dev@dpdk.org;
> david.marchand@redhat.com; Gavin Hu <Gavin.Hu@arm.com>; Honnappa
> Nagarahalli <Honnappa.Nagarahalli@arm.com>; juraj.linkes@pantheon.tech;
> nd <nd@arm.com>
> Subject: Re: [PATCH v2 2/2] ci: add test suite run without hugepage
> 
> Ruifeng Wang <ruifeng.wang@arm.com> writes:
> 
> > This test suite is derived from fast-tests suite. Cases in this suite
> > are run with '--no-huge' flag.
> >
> > The suite aims to cover as many as possible test cases out of the
> > fast-tests suites in the environments without huge pages support, like
> > containers.
> >
> > Signed-off-by: Ruifeng Wang <ruifeng.wang@arm.com>
> > Reviewed-by: Gavin Hu <gavin.hu@arm.com>
> > ---
> 
> I like this much more.  Few comments.
> 
> >  .travis.yml          | 10 +++++--
> >  app/test/meson.build | 71
> >  ++++++++++++++++++++++++++++++++++++++++++++
> 
> You should update doc/guides/prog_guide/meson_ut.rst to include some
> detail about the new tests suite.
> 
Thanks. Will update document in next version.

> >  2 files changed, 79 insertions(+), 2 deletions(-)
> >
> > diff --git a/.travis.yml b/.travis.yml index b64a81bd0..eed1d96db
> > 100644
> > --- a/.travis.yml
> > +++ b/.travis.yml
> > @@ -40,7 +40,7 @@ jobs:
> >    - env: DEF_LIB="static"
> >      arch: amd64
> >      compiler: gcc
> > -  - env: DEF_LIB="shared" RUN_TESTS=1
> > +  - env: DEF_LIB="shared" RUN_TESTS=1 TEST_SUITES="fast-tests
> nohuge-tests"
> >      arch: amd64
> >      compiler: gcc
> >    - env: DEF_LIB="shared" BUILD_DOCS=1 @@ -63,7 +63,7 @@ jobs:
> >    - env: DEF_LIB="static"
> >      arch: amd64
> >      compiler: clang
> > -  - env: DEF_LIB="shared" RUN_TESTS=1
> > +  - env: DEF_LIB="shared" RUN_TESTS=1 TEST_SUITES="fast-tests
> nohuge-tests"
> >      arch: amd64
> >      compiler: clang
> >    - env: DEF_LIB="shared" BUILD_DOCS=1 @@ -101,6 +101,9 @@ jobs:
> >    - env: DEF_LIB="static"
> >      arch: arm64
> >      compiler: gcc
> > +  - env: DEF_LIB="shared" RUN_TESTS=1 TEST_SUITES="nohuge-tests"
> > +    arch: arm64
> > +    compiler: gcc
> >    - env: DEF_LIB="shared" BUILD_DOCS=1
> >      arch: arm64
> >      compiler: gcc
> > @@ -124,3 +127,6 @@ jobs:
> >    - env: DEF_LIB="shared"
> >      arch: arm64
> >      compiler: clang
> > +  - env: DEF_LIB="shared" RUN_TESTS=1 TEST_SUITES="nohuge-tests"
> > +    arch: arm64
> > +    compiler: clang
> > diff --git a/app/test/meson.build b/app/test/meson.build index
> > 0a2ce710f..162a1a76f 100644
> > --- a/app/test/meson.build
> > +++ b/app/test/meson.build
> > @@ -237,6 +237,60 @@ fast_test_names = [
> >          'thash_autotest',
> >  ]
> 
> Shouldn't we also trim the list of fast-tests?  Otherwise, these tests will run
> twice.
> 
I think you mean to have exclusive lists for fast-tests and nohuge-tests.
Overlapped cases will run twice if both test suites are opted in.
But the two runs are not the same, one runs with hugepage and 
the other runs in no-huge mode.

If fast-tests list is splited,  we will need to always run multiple suites
to cover all fast tests.
We can keep x86 to run only fast-tests suite to avoid extra test runs if
they are not necessary. Thoughts?

> ex: https://travis-ci.com/ovsrobot/dpdk/jobs/292037684
> 
> > +nohuge_test_names = [
> > +        'byteorder_autotest',
> > +        'cmdline_autotest',
> > +        'common_autotest',
> > +        'cpuflags_autotest',
> > +        'cycles_autotest',
> > +        'debug_autotest',
> > +        'eal_flags_n_opt_autotest',
> > +        'eal_flags_no_huge_autotest',
> > +        'eal_flags_vdev_opt_autotest',
> > +        'eal_fs_autotest',
> > +        'errno_autotest',
> > +        'event_ring_autotest',
> > +        'fib_autotest',
> > +        'fib6_autotest',
> > +        'interrupt_autotest',
> > +        'logs_autotest',
> > +        'lpm_autotest',
> > +        'lpm6_autotest',
> > +        'memcpy_autotest',
> > +        'meter_autotest',
> > +        'per_lcore_autotest',
> > +        'prefetch_autotest',
> > +        'rcu_qsbr_autotest',
> > +        'red_autotest',
> > +        'rib_autotest',
> > +        'rib6_autotest',
> > +        'ring_autotest',
> > +        'rwlock_rda_autotest',
> > +        'rwlock_rds_wrm_autotest',
> > +        'rwlock_rde_wro_autotest',
> > +        'sched_autotest',
> > +        'spinlock_autotest',
> > +        'string_autotest',
> > +        'tailq_autotest',
> > +        'user_delay_us',
> > +        'version_autotest',
> > +        'crc_autotest',
> > +        'delay_us_sleep_autotest',
> > +        'eventdev_common_autotest',
> > +        'fbarray_autotest',
> > +        'ipsec_autotest',
> > +        'kni_autotest',
> > +        'kvargs_autotest',
> > +        'member_autotest',
> > +        'metrics_autotest',
> > +        'power_cpufreq_autotest',
> > +        'power_autotest',
> > +        'power_kvm_vm_autotest',
> > +        'reorder_autotest',
> > +        'service_autotest',
> > +        'thash_autotest',
> > +]
> > +
> >  perf_test_names = [
> >          'ring_perf_autotest',
> >          'mempool_perf_autotest',
> > @@ -341,6 +395,10 @@ if dpdk_conf.has('RTE_LIBRTE_RING_PMD')
> >  	fast_test_names += 'latencystats_autotest'
> >  	driver_test_names += 'link_bonding_mode4_autotest'
> >  	fast_test_names += 'pdump_autotest'
> > +	nohuge_test_names += 'ring_pmd_autotest'
> > +	nohuge_test_names += 'bitratestats_autotest'
> > +	nohuge_test_names += 'latencystats_autotest'
> > +	nohuge_test_names += 'pdump_autotest'
> >  endif
> >
> >  if dpdk_conf.has('RTE_LIBRTE_POWER')
> > @@ -430,6 +488,19 @@ foreach arg : fast_test_names
> >  	endif
> >  endforeach
> >
> > +foreach arg : nohuge_test_names
> > +	if host_machine.system() == 'linux'
> > +		test(arg, dpdk_test,
> > +			  env : ['DPDK_TEST=' + arg],
> > +			  args : test_args +
> > +				 ['--no-huge'] + ['-m 1024'] +
> > +				 ['--file-prefix=@0@'.format(arg)],
> > +		timeout : timeout_seconds_fast,
> > +		is_parallel : false,
> > +		suite : 'nohuge-tests')
> > +	endif
> > +endforeach
> > +
> >  foreach arg : perf_test_names
> >  	test(arg, dpdk_test,
> >  	env : ['DPDK_TEST=' + arg],