From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5E969A00C3; Mon, 3 Oct 2022 12:00:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A4B3B40DFB; Mon, 3 Oct 2022 12:00:04 +0200 (CEST) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id 98C8340695 for ; Mon, 3 Oct 2022 12:00:03 +0200 (CEST) Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id 168785E; Mon, 3 Oct 2022 13:00:03 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru 168785E DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1664791203; bh=kh76vTUuGWZf4RyeLuaYLZhTNCE3nJtORwo0XJWQcsA=; h=Date:From:Subject:To:Cc:From; b=T0Mjkw2VbejHHO9Wgf20jCMse6mtWIGQ3qBHbzQlk0oIqZrTdxanuUU8RK12fC75j /+HrLX8g6oLbsR14mZMrfjBhnhs6CXZo1AE5nLuwFxneA/Sgqc4SgAAfChSfOKiRa8 800YWKBGgyThsZw4fSeKLIW6JOwORctL27o74Aqo= Message-ID: <792fd30f-acd9-b07d-e4eb-7d88de654538@oktetlabs.ru> Date: Mon, 3 Oct 2022 13:00:02 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.13.0 From: Andrew Rybchenko Subject: Opensource ethdev tests To: Thomas Monjalon Content-Language: en-US Cc: "dev@dpdk.org" , Ferruh Yigit , Ajit Khaparde , Jerin Jacob Kollanukkaran , Dongdong Liu , Qiming Yang , Yuying Zhang , Beilei Xing , Qi Zhang , Hemant Agrawal , Maxime Coquelin , Viacheslav Ovsiienko , Stephen Hemminger Organization: OKTET Labs Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Thomas and community, May I ask to add https://ts-factory.io/ to the DPDK ecosystem. I'm not 100% that it is suitable for the ecosystem since it is not consuming DPDK, but rather testing DPDK ethdev. Anyway it could be useful for ethdev PMD developers and maintainers. I'll not repeat what is written on the site [1] and documentation (including the framework [2] and test scenarios documentation [3]) to keep the mail small enough. [1] https://ts-factory.io/ [2] https://ts-factory.io/doc/test-environment/ [3] https://ts-factory.io/doc/dpdk-ethdev-ts/ First of all I'd like to thank Xilinx/AMD for making these tests opensource. Testing framework (Test Environment) used by these tests is an opensource as well. The database has examples of testing log for QEMU virtio [4], Solarflare SFN8522 [5], Intel X710 [6] and Mellanox ConnectX-5 [7] NICs. [4] https://ts-factory.io/bublik/v2/runs?runData=pci-1af4%3BTS_NAME%3Ddpdk-ethdev-ts [5] https://ts-factory.io/bublik/v2/runs?runData=pci-1924%3BTS_NAME%3Ddpdk-ethdev-ts [6] https://ts-factory.io/bublik/v2/runs?runData=pci-8086-1572%3BTS_NAME%3Ddpdk-ethdev-ts [7] https://ts-factory.io/bublik/v2/runs?runData=pci-15b3-1017%3BTS_NAME%3Ddpdk-ethdev-ts Full list of sample DPDK ethdev logs [8]. [8] https://ts-factory.io/bublik/v2/runs?runData=TS_NAME%3Ddpdk-ethdev-ts Testing results are classified into 6 categories. There are 3 results: passed, failed and skipped (when test fails to do its job because tested functionality itself or some required per-conditions are not supported). Each result could be either expected in accordance with filled in expectations or unexpected if obtained result does not match expectations. These expectations could differ for different NICs, tested DPDK version etc. High rate of expected results for SFN8522 and virtio is explained by origin of the tests. Expectations for these NICs are mostly filled in. High number of unexpected results for i40e and mlx5 drivers does not mean these drivers or NICs are bad. First of all it is tests which could be wrong, too strict or just have bugs. Second, expectations (because of missing functionality or known aspects of the behaviour) for these NICs are not filled in in many-many cases. Let's get down to few examples of unexpected results. 1. QEMU virtio. VLAN tagged packet is not delivered. it is a virtio testing when two VMs talk to each other via Linux bridge.    Sent packet is observed on Peer (line 38), but DPDK fails to receive it (line 65). Most likely it is some kind of misconfiguration. https://ts-factory.io/bublik/v2/log/93205?focusId=93288&mode=treeAndinfoAndlog 2. QEMU virtio. Inconsistent number of xstats on get number (38) and actual get (just 13 returned).     The trick here is that xstats API is called just after rte_eth_dev_configiure(). I.e. queues are not configured and device is not started yet. https://ts-factory.io/bublik/v2/log/93205?focusId=93983&mode=treeAndinfoAndlog 3. Intel X710. CWR TCP flag loss in dummy TSO case (i.e. when TSO payload is less than TCP MSS).     Sent packet has CWR bit set in TCP flags, but the packet received on Peer does not have it.     Since it is a dummy TSO case it is hardly critical, but still interesting aspect of the behaviour. https://ts-factory.io/bublik/v2/log/70553?focusId=72767&mode=treeAndinfoAndlog 4. ConnectX-5. Prepared but stuck on Tx burst single segment TSO packet.     Tx prepare on line 51 accepts the packet, but attempt to transmit fails on line 62.     Of course the packet layout is specific since typically TSO header goes in its own segment,     but the behavbiour is still unfriendly since application never knows if Tx burst returns 0 since Tx ring is full or something else is wrong. https://ts-factory.io/bublik/v2/log/85618?focusId=87546&mode=treeAndinfoAndlog 5. ConnectX-5: Sometimes huge number of tests bring fruits like this when all further tests fail because of NIC probe failure.     Most likely it was a bug in a particular driver version since the problem is not always repeatable. https://ts-factory.io/bublik/v2/log/48348?focusId=51740&mode=treeAndinfoAndlog Performance testing results using testpmd are not representative since used hosts are too weak and not really tuned for performance testing. If testing hosts are good, these tests can do its job as well. For example a number of bugs in net/virtion were found using these tests and corresponding patches sent upstream in the past. These tests are fully automatic and suitable for release testing as well as everyday regressions tracking as soon as expectations are filled in. Also having expectations for different NICs filled in allows to generate comparison reports to understand the difference in behaviour and supported features. The testing framework supports collection of gcov-based coverage and generates reports if requested, for example [9] for i40e. [9] https://ts-factory.io/logs/2022/09/30/fror-x710-p0-7/tce_log_dpdk.html Besides DPDK ethdev tests there are testing logs for Linux net drivers for same NICs [10]. [10] https://ts-factory.io/bublik/v2/runs?runData=TS_NAME%3Dnet-drv-ts Andrew.