From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5B957456F4; Tue, 30 Jul 2024 13:16:33 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C31D8402EA; Tue, 30 Jul 2024 13:16:29 +0200 (CEST) Received: from inbox.dpdk.org (inbox.dpdk.org [95.142.172.178]) by mails.dpdk.org (Postfix) with ESMTP id 7FCF3400D7 for ; Tue, 30 Jul 2024 12:14:00 +0200 (CEST) Received: by inbox.dpdk.org (Postfix, from userid 33) id 6A6F0456F3; Tue, 30 Jul 2024 12:14:00 +0200 (CEST) From: bugzilla@dpdk.org To: dts@dpdk.org Subject: [DTS/tests Bug 1507] Fixed test traffic patterns make test results unreliable Date: Tue, 30 Jul 2024 10:14:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: DTS X-Bugzilla-Component: tests X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: major X-Bugzilla-Who: mb@smartsharesystems.com X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: dts@dpdk.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter target_milestone Message-ID: Content-Type: multipart/alternative; boundary=17223344400.D2EB1A.235588 Content-Transfer-Encoding: 7bit X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All MIME-Version: 1.0 X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org --17223344400.D2EB1A.235588 Date: Tue, 30 Jul 2024 12:14:00 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All https://bugs.dpdk.org/show_bug.cgi?id=3D1507 Bug ID: 1507 Summary: Fixed test traffic patterns make test results unreliable Product: DTS Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: major Priority: Normal Component: tests Assignee: dts@dpdk.org Reporter: mb@smartsharesystems.com Target Milestone: --- It has recently come to my attention that a lot of testing is performed with fixed packet size and type. This has two bad consequences: 1. The same code path in the SUT is taken every time. This is bad, because it reduces the code coverage of functional testing. Worst case, newly added code (intended to be tested) may be in a non-taken = code path, and thus not tested at all. 2. When the same code path is taken every time, the CPU's branch predictor achieves near-perfect hit rate, and thus the performance test results do not reflect real-life performance. This is very bad, because performance results are often used to quantify the benefit of suggested performance optimizations, and they might in fact have= the opposite effect in real life (with real-life traffic mix). Testing should be performed with a realistic mix of packets, as seen on the internet. For some cases, packet size matters, and for some cases, packet type matter= s. --=20 You are receiving this mail because: You are the assignee for the bug.= --17223344400.D2EB1A.235588 Date: Tue, 30 Jul 2024 12:14:00 +0200 MIME-Version: 1.0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All
Bug ID 1507
Summary Fixed test traffic patterns make test results unreliable
Product DTS
Version unspecified
Hardware All
OS All
Status UNCONFIRMED
Severity major
Priority Normal
Component tests
Assignee dts@dpdk.org
Reporter mb@smartsharesystems.com
Target Milestone --- 

It has recently come to my attenti=
on that a lot of testing is performed with
fixed packet size and type.

This has two bad consequences:

1. The same code path in the SUT is taken every time.
This is bad, because it reduces the code coverage of functional testing.
Worst case, newly added code (intended to be tested) may be in a non-taken =
code
path, and thus not tested at all.

2. When the same code path is taken every time, the CPU's branch predictor
achieves near-perfect hit rate, and thus the performance test results do not
reflect real-life performance.
This is very bad, because performance results are often used to quantify the
benefit of suggested performance optimizations, and they might in fact have=
 the
opposite effect in real life (with real-life traffic mix).

Testing should be performed with a realistic mix of packets, as seen on the
internet.
For some cases, packet size matters, and for some cases, packet type matter=
s.

You are receiving this mail because:
  • You are the assignee for the bug.
=20=20=20=20=20=20=20=20=20=20
= --17223344400.D2EB1A.235588--