Opensource ethdev tests

[Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>]

Hi Thomas and community,

May I ask to add https://ts-factory.io/ to the DPDK ecosystem.
I'm not 100% that it is suitable for the ecosystem since it is not 
consuming DPDK, but rather testing DPDK ethdev.
Anyway it could be useful for ethdev PMD developers and maintainers.

I'll not repeat what is written on the site [1] and documentation 
(including the framework [2] and test scenarios documentation [3]) to 
keep the mail small enough.

[1] https://ts-factory.io/
[2] https://ts-factory.io/doc/test-environment/
[3] https://ts-factory.io/doc/dpdk-ethdev-ts/

First of all I'd like to thank Xilinx/AMD for making these tests opensource.
Testing framework (Test Environment) used by these tests is an 
opensource as well.

The database has examples of testing log for QEMU virtio [4], Solarflare 
SFN8522 [5], Intel X710 [6] and Mellanox ConnectX-5 [7] NICs.

[4] 
https://ts-factory.io/bublik/v2/runs?runData=pci-1af4%3BTS_NAME%3Ddpdk-ethdev-ts
[5] 
https://ts-factory.io/bublik/v2/runs?runData=pci-1924%3BTS_NAME%3Ddpdk-ethdev-ts
[6] 
https://ts-factory.io/bublik/v2/runs?runData=pci-8086-1572%3BTS_NAME%3Ddpdk-ethdev-ts
[7] 
https://ts-factory.io/bublik/v2/runs?runData=pci-15b3-1017%3BTS_NAME%3Ddpdk-ethdev-ts

Full list of sample DPDK ethdev logs [8].

[8] https://ts-factory.io/bublik/v2/runs?runData=TS_NAME%3Ddpdk-ethdev-ts

Testing results are classified into 6 categories. There are 3 results: 
passed, failed and skipped (when test fails to do its job because tested 
functionality itself or some required per-conditions are not supported). 
Each result could be either expected in accordance with filled in 
expectations or unexpected if obtained result does not match 
expectations. These expectations could differ for different NICs, tested 
DPDK version etc.

High rate of expected results for SFN8522 and virtio is explained by 
origin of the tests. Expectations for these NICs are mostly filled in.
High number of unexpected results for i40e and mlx5 drivers does not 
mean these drivers or NICs are bad. First of all it is tests which could 
be wrong, too strict or just have bugs. Second, expectations (because of 
missing functionality or known aspects of the behaviour) for these NICs 
are not filled in in many-many cases.

Let's get down to few examples of unexpected results.

1. QEMU virtio. VLAN tagged packet is not delivered. it is a virtio 
testing when two VMs talk to each other via Linux bridge.
    Sent packet is observed on Peer (line 38), but DPDK fails to receive 
it (line 65). Most likely it is some kind of misconfiguration.
https://ts-factory.io/bublik/v2/log/93205?focusId=93288&mode=treeAndinfoAndlog

2. QEMU virtio. Inconsistent number of xstats on get number (38) and 
actual get (just 13 returned).
     The trick here is that xstats API is called just after 
rte_eth_dev_configiure(). I.e. queues are not configured and device is 
not started yet.
https://ts-factory.io/bublik/v2/log/93205?focusId=93983&mode=treeAndinfoAndlog

3. Intel X710. CWR TCP flag loss in dummy TSO case (i.e. when TSO 
payload is less than TCP MSS).
     Sent packet has CWR bit set in TCP flags, but the packet received 
on Peer does not have it.
     Since it is a dummy TSO case it is hardly critical, but still 
interesting aspect of the behaviour.
https://ts-factory.io/bublik/v2/log/70553?focusId=72767&mode=treeAndinfoAndlog

4. ConnectX-5. Prepared but stuck on Tx burst single segment TSO packet.
     Tx prepare on line 51 accepts the packet, but attempt to transmit 
fails on line 62.
     Of course the packet layout is specific since typically TSO header 
goes in its own segment,
     but the behavbiour is still unfriendly since application never 
knows if Tx burst returns 0 since Tx ring is full or something else is 
wrong.
https://ts-factory.io/bublik/v2/log/85618?focusId=87546&mode=treeAndinfoAndlog

5. ConnectX-5: Sometimes huge number of tests bring fruits like this 
when all further tests fail because of NIC probe failure.
     Most likely it was a bug in a particular driver version since the 
problem is not always repeatable.
https://ts-factory.io/bublik/v2/log/48348?focusId=51740&mode=treeAndinfoAndlog

Performance testing results using testpmd are not representative since 
used hosts are too weak and not really tuned for performance testing. If 
testing hosts are good, these tests can do its job as well.

For example a number of bugs in net/virtion were found using these tests 
and corresponding patches sent upstream in the past.

These tests are fully automatic and suitable for release testing as well 
as everyday regressions tracking as soon as expectations are filled in.

Also having expectations for different NICs filled in allows to generate 
comparison reports to understand the difference in behaviour and 
supported features.

The testing framework supports collection of gcov-based coverage and 
generates reports if requested, for example [9] for i40e.

[9] https://ts-factory.io/logs/2022/09/30/fror-x710-p0-7/tce_log_dpdk.html

Besides DPDK ethdev tests there are testing logs for Linux net drivers 
for same NICs [10].

[10] https://ts-factory.io/bublik/v2/runs?runData=TS_NAME%3Dnet-drv-ts

Andrew.