From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 073F3432D0; Wed, 8 Nov 2023 08:20:06 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F358C4067C; Wed, 8 Nov 2023 08:20:05 +0100 (CET) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id 96B2C402D4 for ; Wed, 8 Nov 2023 08:20:04 +0100 (CET) Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id 7D52E69; Wed, 8 Nov 2023 10:20:03 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru 7D52E69 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1699428003; bh=txYuV+pM9uJ4jc7HE+7tu1ZvSwBU6banQeo3TiDW1V8=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=gGivQRiEh5/3ApV3lVj5i/8J5/UbBsdUwiFiFJy4gqyboAULM7sS1Qrr3kliSgCSJ wnWXO7IaEhSw8ZrosgTQ/dQ/sxD2gNs9zCrnn3TzZOsSHJIGC9oja/ogxB4alhc1m7 0ZNYqoTxdleYvWsHMnszovRMwlpoovEC5oPoWKMA= Content-Type: multipart/alternative; boundary="------------5NaLdGc7raUHUYWW5cZ9oBP2" Message-ID: <0cd6a9fb-bc39-4b69-be5d-3470e2374016@oktetlabs.ru> Date: Wed, 8 Nov 2023 10:20:03 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Setting up DPDK PMD Test Suite Content-Language: en-US To: Adam Hassick Cc: Konstantin Ushakov , Patrick Robb , ci@dpdk.org References: <7788077c-4e0b-f68f-2e09-3994a49ae715@oktetlabs.ru> <1f53aade-73a7-baaf-aecb-2b9a33ab6682@oktetlabs.ru> <4979713b-8e5f-417b-b1af-21f54130eab7@oktetlabs.ru> <8e26b8e6-8d8b-4925-9b30-3fbc5e103e18@oktetlabs.ru> From: Andrew Rybchenko Organization: OKTET Labs In-Reply-To: X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org This is a multi-part message in MIME format. --------------5NaLdGc7raUHUYWW5cZ9oBP2 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi Adam, On 11/7/23 23:30, Adam Hassick wrote: > Hi Andrew, > > The runner machine was missing a dependency for one of the scripts, > "pixz". After installing that, it appears to have worked. I can see > the results listed on the ts-factory Bublik instance. If you use copy of dpdk-ethdev-ts has 398e272495143884274f5a53c6fe0cc16df41052, you don't need to pass --trc-tag=pci-8086-1572 any more since corresponding changeset updates expectations to have the same for pci-8086-1583. > In the latest revision of ts-rigs, there appears to be a syntax error > at line 42 within the script located at > "ts-rigs/scripts/publish_logs/prj/ts-factory/publish", within the if > condition. I fixed it locally to get it to run. Sorry, but I've failed to find what's wrong there. > Taking a quick look at a comparison against your most recent X710 run, > it looks like we're NOK on around ~400 more test cases. By percentage > of tests, we're 1% off, however, it looks like whole subsets of the > test suite that contain low numbers of tests are failing. I wonder if > this is due to differences between the Intel X710 and XL710 or issues > in our dev testbed. As far as I can see LLDP packets spoil testing results: https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63 As far as I can see main prologue disables FW LLDP on Tester https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80 but I guess it could be still enabled on DUT side and DPDK do not provide means to disable it as far as I know. I vaguely remember that Intel provides FW configuration tools which can do it. It is interesting since DPDK gets unexpected LLDP packets but may be packets sent by FW go via loopback and visible to PF as well. Other possible source of LLDP packet is a switch if NICs are connected via switch. If so, LLDP should be disabled on corresponding switch ports. As far as I can see fixing the problem should make results much closer. However, I already see some differences in behaviour which should be simply fixed in TRC. For example, X710 gets 9 packets less than configuration number of Rx descriptors, but XL710 gets 10 packets less. Also I see that performance tests are not run because of failed prologue: https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true I'll investigate it, but I guess the source of difference is that we always run tests on single interface. Just add -p0 (--cfg=iol-dts-xl710-p0) to your configuration name. You don't need to change ts-rigs for it since the suffix is handled by generic code. It simply comments the second instance and forces take the first interface only into account. Initially it was introduced to run independent tests on different ports to be able to share configuration, but I guess right now it has limitations for some packages like representors which require entire NIC. Regards, Andrew. > Thanks, > Adam (dropped history, to keep mail size small) --------------5NaLdGc7raUHUYWW5cZ9oBP2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit
Hi Adam,

On 11/7/23 23:30, Adam Hassick wrote:
Hi Andrew,

The runner machine was missing a dependency for one of the scripts, "pixz". After installing that, it appears to have worked. I can see the results listed on the ts-factory Bublik instance.

If you use copy of dpdk-ethdev-ts has 398e272495143884274f5a53c6fe0cc16df41052, you don't need to pass --trc-tag=pci-8086-1572 any more since corresponding changeset updates expectations to have the same for pci-8086-1583.

In the latest revision of ts-rigs, there appears to be a syntax error at line 42 within the script located at "ts-rigs/scripts/publish_logs/prj/ts-factory/publish", within the if condition. I fixed it locally to get it to run.

Sorry, but I've failed to find what's wrong there.

Taking a quick look at a comparison against your most recent X710 run, it looks like we're NOK on around ~400 more test cases. By percentage of tests, we're 1% off, however, it looks like whole subsets of the test suite that contain low numbers of tests are failing. I wonder if this is due to differences between the Intel X710 and XL710 or issues in our dev testbed.

As far as I can see LLDP packets spoil testing results:
https://ts-factory.io/bublik/v2/log/362398?focusId=362760&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_63

As far as I can see main prologue disables FW LLDP on Tester
https://ts-factory.io/bublik/v2/log/362398?focusId=362400&mode=treeAndinfoAndlog&experimental=true&lineNumber=1_80
but I guess it could be still enabled on DUT side and DPDK do not provide means to disable it as far as I know. I vaguely remember that Intel provides FW configuration tools which can do it.
It is interesting since DPDK gets unexpected LLDP packets but may be packets sent by FW go via loopback and visible to PF as well.
Other possible source of LLDP packet is a switch if NICs are connected via switch. If so, LLDP should be disabled on corresponding switch ports.

As far as I can see fixing the problem should make results much closer. However, I already see some differences in behaviour which should be simply fixed in TRC. For example, X710 gets 9 packets less than configuration number of Rx descriptors, but XL710 gets 10 packets less.

Also I see that performance tests are not run because of failed prologue:
https://ts-factory.io/bublik/v2/log/362398?focusId=369564&mode=treeAndinfoAndlog&experimental=true
I'll investigate it, but I guess the source of difference is that we always run tests on single interface. Just add -p0 (--cfg=iol-dts-xl710-p0) to your configuration name. You don't need to change ts-rigs for it since the suffix is handled by generic code. It simply comments the second instance and forces take the first interface only into account. Initially it was introduced to run independent tests on different ports to be able to share configuration, but I guess right now it has limitations for some packages like representors which require entire NIC.

Regards,
Andrew.

Thanks,
Adam

(dropped history, to keep mail size small) --------------5NaLdGc7raUHUYWW5cZ9oBP2--