Unit tests not defined in any test suite

Unit tests not defined in any test suite

[Bruce Richardson]

Hi all,

As part of the recent rework of the build of our unit test binary
(dpdk-test), the individual autotests were updated in the source code to
specify the test suite they belong to. The replacement of the
registration macro was done automatically using the existing suite lists
in the meson.build file, but following the replacement, it appears that
there are quite a number of test cases which are not in any test suite -
and as such as not being run automatically for patches in the IOL test
lab.

Below is a list of these test cases, and the files each is defined in.
We should endeavour to ensure each test case is in a suite. Currently we
have 3 suites - fast-test, perf-test, and driver-test, but more suites
can be added as necessary.

I've attempted to automatically identify the maintainers of the various
tests using "get-maintainers" script - hence the list of those on CC. If
possible, can each of you identify what test cases you may be maintainer
for, and add them to a test suite, to help improve our test coverage.
[Apologies for any mis-identifications]

Thanks,
/Bruce

alarm_autotest (test_alarm.c)
bitratestats_autotest (test_bitratestats.c)
cfgfile_autotest (test_cfgfile.c)
cksum_perf_autotest (test_cksum_perf.c)
compressdev_autotest (test_compressdev.c)
cryptodev_scheduler_autotest (test_cryptodev.c)
cryptodev_dpaa2_sec_raw_api_autotest (test_cryptodev.c)
cryptodev_dpaa_sec_raw_api_autotest (test_cryptodev.c)
cryptodev_mlx5_autotest (test_cryptodev.c)
cryptodev_ccp_autotest (test_cryptodev.c)
cryptodev_virtio_autotest (test_cryptodev.c)
cryptodev_octeontx_autotest (test_cryptodev.c)
cryptodev_caam_jr_autotest (test_cryptodev.c)
cryptodev_nitrox_autotest (test_cryptodev.c)
cryptodev_bcmfs_autotest (test_cryptodev.c)
cryptodev_openssl_asym_autotest (test_cryptodev_asym.c)
cryptodev_octeontx_asym_autotest (test_cryptodev_asym.c)
cryptodev_cn9k_asym_autotest (test_cryptodev_asym.c)
cryptodev_cn10k_asym_autotest (test_cryptodev_asym.c)
cryptodev_crosscheck (test_cryptodev_crosscheck.c)
event_crypto_adapter_autotest (test_event_crypto_adapter.c)
event_eth_rx_adapter_autotest (test_event_eth_rx_adapter.c)
event_eth_rx_intr_adapter_autotest (test_event_eth_rx_adapter.c)
event_eth_tx_adapter_autotest (test_event_eth_tx_adapter.c)
event_timer_adapter_test (test_event_timer_adapter.c)
eventdev_selftest_sw (test_eventdev.c)
eventdev_selftest_octeontx (test_eventdev.c)
eventdev_selftest_dpaa2 (test_eventdev.c)
eventdev_selftest_dlb2 (test_eventdev.c)
eventdev_selftest_cn9k (test_eventdev.c)
eventdev_selftest_cn10k (test_eventdev.c)
external_mem_autotest (test_external_mem.c)
graph_autotest (test_graph.c)
node_list_dump (test_graph.c)
graph_perf_autotest (test_graph_perf.c)
ipsec_sad_autotest (test_ipsec_sad.c)
latencystats_autotest (test_latencystats.c)
link_bonding_autotest (test_link_bonding.c)
link_bonding_mode4_autotest (test_link_bonding_mode4.c)
link_bonding_rssconf_autotest (test_link_bonding_rssconf.c)
metrics_autotest (test_metrics.c)
pcapng_autotest (test_pcapng.c)
pdcp_autotest (test_pdcp.c)
pdump_autotest (test_pdump.c)
ring_pmd_autotest (test_pmd_ring.c)
ring_pmd_perf_autotest (test_pmd_ring_perf.c)
power_caps_autotest (test_power_cpufreq.c)
rawdev_autotest (test_rawdev.c)
red_autotest (test_red.c)
resource_autotest (test_resource.c)
ring_stress_autotest (test_ring_stress.c)
inline_macsec_autotest (test_security_inline_macsec.c)
inline_ipsec_autotest (test_security_inline_proto.c)
inline_ipsec_sg_autotest (test_security_inline_proto.c)
event_inline_ipsec_autotest (test_security_inline_proto.c)
table_autotest (test_table.c)
timer_secondary_autotest (test_timer_secondary.c)
vdev_autotest (test_vdev.c)

Re: Unit tests not defined in any test suite

[David Marchand]

Hello Bruce,

On Thu, Sep 14, 2023 at 6:16 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> Hi all,
>
> As part of the recent rework of the build of our unit test binary
> (dpdk-test), the individual autotests were updated in the source code to
> specify the test suite they belong to. The replacement of the
> registration macro was done automatically using the existing suite lists
> in the meson.build file, but following the replacement, it appears that
> there are quite a number of test cases which are not in any test suite -
> and as such as not being run automatically for patches in the IOL test
> lab.
>
> Below is a list of these test cases, and the files each is defined in.
> We should endeavour to ensure each test case is in a suite. Currently we
> have 3 suites - fast-test, perf-test, and driver-test, but more suites
> can be added as necessary.
>
> I've attempted to automatically identify the maintainers of the various
> tests using "get-maintainers" script - hence the list of those on CC. If
> possible, can each of you identify what test cases you may be maintainer
> for, and add them to a test suite, to help improve our test coverage.
> [Apologies for any mis-identifications]
>
> Thanks,
> /Bruce
>
> alarm_autotest (test_alarm.c)
> bitratestats_autotest (test_bitratestats.c)

Those two tests above (and the one on the graph library) made me raise
an eyebrow as I remembered them being part of the fast-tests list.
I thought I had checked the tests lists before/after rework..., but
looking again I did not do this properly (the blame is on the
reviewers and commiters).

Before the rework, GHA would run 106 fast-tests ut (look for
fast-tests in follow log):
https://pipelines.actions.githubusercontent.com/serviceHosts/aad9342c-d08c-4ebc-a424-c37e5c0e7e92/_apis/pipelines/1/runs/1767/signedlogcontent/8?urlExpires=2023-09-15T07%3A37%3A45.5644150Z&urlSigningMethod=HMACV1&urlSignature=0g%2BN%2B0Q64RQolmslxH5N%2B91J7kvDAcVAanOmYnvO5cE%3D
After the rework, only 92 remain:
https://pipelines.actions.githubusercontent.com/serviceHosts/aad9342c-d08c-4ebc-a424-c37e5c0e7e92/_apis/pipelines/1/runs/1771/signedlogcontent/8?urlExpires=2023-09-15T07%3A38%3A01.6216684Z&urlSigningMethod=HMACV1&urlSignature=uVfBMml1XXXr1CLVggaPDMLN8US6n9TcudVcyqFgsd0%3D

I have a fix ready that I will send shortly.
And the list of non tagged tests must be reevaluated.



> cfgfile_autotest (test_cfgfile.c)
> cksum_perf_autotest (test_cksum_perf.c)
> compressdev_autotest (test_compressdev.c)
> cryptodev_scheduler_autotest (test_cryptodev.c)
> cryptodev_dpaa2_sec_raw_api_autotest (test_cryptodev.c)
> cryptodev_dpaa_sec_raw_api_autotest (test_cryptodev.c)
> cryptodev_mlx5_autotest (test_cryptodev.c)
> cryptodev_ccp_autotest (test_cryptodev.c)
> cryptodev_virtio_autotest (test_cryptodev.c)
> cryptodev_octeontx_autotest (test_cryptodev.c)
> cryptodev_caam_jr_autotest (test_cryptodev.c)
> cryptodev_nitrox_autotest (test_cryptodev.c)
> cryptodev_bcmfs_autotest (test_cryptodev.c)
> cryptodev_openssl_asym_autotest (test_cryptodev_asym.c)
> cryptodev_octeontx_asym_autotest (test_cryptodev_asym.c)
> cryptodev_cn9k_asym_autotest (test_cryptodev_asym.c)
> cryptodev_cn10k_asym_autotest (test_cryptodev_asym.c)
> cryptodev_crosscheck (test_cryptodev_crosscheck.c)
> event_crypto_adapter_autotest (test_event_crypto_adapter.c)
> event_eth_rx_adapter_autotest (test_event_eth_rx_adapter.c)
> event_eth_rx_intr_adapter_autotest (test_event_eth_rx_adapter.c)
> event_eth_tx_adapter_autotest (test_event_eth_tx_adapter.c)
> event_timer_adapter_test (test_event_timer_adapter.c)
> eventdev_selftest_sw (test_eventdev.c)
> eventdev_selftest_octeontx (test_eventdev.c)
> eventdev_selftest_dpaa2 (test_eventdev.c)
> eventdev_selftest_dlb2 (test_eventdev.c)
> eventdev_selftest_cn9k (test_eventdev.c)
> eventdev_selftest_cn10k (test_eventdev.c)
> external_mem_autotest (test_external_mem.c)
> graph_autotest (test_graph.c)
> node_list_dump (test_graph.c)
> graph_perf_autotest (test_graph_perf.c)
> ipsec_sad_autotest (test_ipsec_sad.c)
> latencystats_autotest (test_latencystats.c)
> link_bonding_autotest (test_link_bonding.c)
> link_bonding_mode4_autotest (test_link_bonding_mode4.c)
> link_bonding_rssconf_autotest (test_link_bonding_rssconf.c)
> metrics_autotest (test_metrics.c)
> pcapng_autotest (test_pcapng.c)
> pdcp_autotest (test_pdcp.c)
> pdump_autotest (test_pdump.c)
> ring_pmd_autotest (test_pmd_ring.c)
> ring_pmd_perf_autotest (test_pmd_ring_perf.c)
> power_caps_autotest (test_power_cpufreq.c)
> rawdev_autotest (test_rawdev.c)
> red_autotest (test_red.c)
> resource_autotest (test_resource.c)
> ring_stress_autotest (test_ring_stress.c)
> inline_macsec_autotest (test_security_inline_macsec.c)
> inline_ipsec_autotest (test_security_inline_proto.c)
> inline_ipsec_sg_autotest (test_security_inline_proto.c)
> event_inline_ipsec_autotest (test_security_inline_proto.c)
> table_autotest (test_table.c)
> timer_secondary_autotest (test_timer_secondary.c)
> vdev_autotest (test_vdev.c)
>


-- 
David Marchand

Re: Unit tests not defined in any test suite

[Bruce Richardson]

On Fri, Sep 15, 2023 at 09:55:49AM +0200, David Marchand wrote:
> Hello Bruce,
> 
> On Thu, Sep 14, 2023 at 6:16 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > Hi all,
> >
> > As part of the recent rework of the build of our unit test binary
> > (dpdk-test), the individual autotests were updated in the source code to
> > specify the test suite they belong to. The replacement of the
> > registration macro was done automatically using the existing suite lists
> > in the meson.build file, but following the replacement, it appears that
> > there are quite a number of test cases which are not in any test suite -
> > and as such as not being run automatically for patches in the IOL test
> > lab.
> >
> > Below is a list of these test cases, and the files each is defined in.
> > We should endeavour to ensure each test case is in a suite. Currently we
> > have 3 suites - fast-test, perf-test, and driver-test, but more suites
> > can be added as necessary.
> >
> > I've attempted to automatically identify the maintainers of the various
> > tests using "get-maintainers" script - hence the list of those on CC. If
> > possible, can each of you identify what test cases you may be maintainer
> > for, and add them to a test suite, to help improve our test coverage.
> > [Apologies for any mis-identifications]
> >
> > Thanks,
> > /Bruce
> >
> > alarm_autotest (test_alarm.c)
> > bitratestats_autotest (test_bitratestats.c)
> 
> Those two tests above (and the one on the graph library) made me raise
> an eyebrow as I remembered them being part of the fast-tests list.
> I thought I had checked the tests lists before/after rework..., but
> looking again I did not do this properly (the blame is on the
> reviewers and commiters).
> 
Blame also falls on my as patch author, my apologies. I missed the fact
that the test lists given at the top of the old meson.build file were
incomplete, and were added to by additional tests later down the file in
the if/else blocks. [This is, after all, the main reason I did the rework,
to make this more comprehensible, so I suppose my mistake helped prove the
need :-) ]

Apologies to those on the CC list for whom I have accidentally removed your
test from the list. I'll work with David to get things fixed back the way
they were, and then we can do a new audit list of tests.

/Bruce