[dts] Correct way of skipping tests

[dts] Correct way of skipping tests

[Angela Czubak]

Hi,

I am working on adding support for Cavium NICs in DTS and I have 
stumbled upon several issues. I can see there are two *xls file in conf 
directory. What is the difference between dpdk_support_test_case.xls 
and dpdk_test_case_checklist.xls? I understand that adding a NIC and a 
test case to either make this test case be skipped for the NIC but I do 
not really know which one should be used and when.
Are these spreadsheets meant to be modified manually? I just want to 
know if a binary patch on either of these files is OK or maybe there is 
a better way to add information about which tests should be skipped for 
a particular NIC.
What is more, is there a generic way of skipping some parts of test 
cases? For instance I can see in TestSuite_checksum_offload.py that 
sending out IP/SCTP packets is skipped if the driver is fm10k. It seems 
very specific and adding more checks in that manner to every test case 
seems not to be very flexible. Is there already a structure somewhere 
holding similar type of information, for example that SCTP tx offload is 
not supported for a list of drivers (including fm10k and possibly 
others)? I have not managed to detect this kind of variable and I am 
wondering if there is a reason this kind of setting lacking capabilities 
is missing (and just hardcoded comparisons seem to be used).

Regards,
Angela Czubak

Re: [dts] Correct way of skipping tests

[Liu, Yong]

Hi Angela,
Support list and check list are the two dimensions for check whether cases can run on current validation environment. 

If you've known that only one or two kinds of NIC supported by some cases, you can add it into support list.
When case name matched in support list, NIC/OS/Target will be checked in sequence. If any of them not matched, case will be skipped.
If you've known that one case can't run in certain environment like i686 or NIC, you can add it into check list.
When case name matched in check list, NIC/OS/Target will be checked in sequence. If any of them matched, case will be skipped.

These information is tracked by excel for it need to be readable. Binary patch is hard to read, so commit log should mention what has been changed.
Other format like Json/XML should be better choice. Currently, we have on plan on that.

For long term, we will add suite based configuration file and in that file we can track some special things like fm10k not support SCTP checksum.

Thanks,
Marvin

> -----Original Message-----
> From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Angela Czubak
> Sent: Saturday, April 01, 2017 1:08 AM
> To: dts@dpdk.org
> Subject: [dts] Correct way of skipping tests
> 
> Hi,
> 
> I am working on adding support for Cavium NICs in DTS and I have
> stumbled upon several issues. I can see there are two *xls file in conf
> directory. What is the difference between dpdk_support_test_case.xls
> and dpdk_test_case_checklist.xls? I understand that adding a NIC and a
> test case to either make this test case be skipped for the NIC but I do
> not really know which one should be used and when.
> Are these spreadsheets meant to be modified manually? I just want to
> know if a binary patch on either of these files is OK or maybe there is
> a better way to add information about which tests should be skipped for
> a particular NIC.
> What is more, is there a generic way of skipping some parts of test
> cases? For instance I can see in TestSuite_checksum_offload.py that
> sending out IP/SCTP packets is skipped if the driver is fm10k. It seems
> very specific and adding more checks in that manner to every test case
> seems not to be very flexible. Is there already a structure somewhere
> holding similar type of information, for example that SCTP tx offload is
> not supported for a list of drivers (including fm10k and possibly
> others)? I have not managed to detect this kind of variable and I am
> wondering if there is a reason this kind of setting lacking capabilities
> is missing (and just hardcoded comparisons seem to be used).
> 
> Regards,
> Angela Czubak