test suite reviews and discussions
 help / color / mirror / Atom feed
From: Lijuan Tu <lijuan.tu@intel.com>
To: dliu@iol.unh.edu, ohilyard@iol.unh.edu, alialnu@nvidia.com
Cc: dts@dpdk.org, Lijuan Tu <lijuan.tu@intel.com>
Subject: [dts] [v2 4/6] tests/nic_single_core_perf: revise delta to ratio
Date: Wed, 31 Mar 2021 09:34:08 +0000	[thread overview]
Message-ID: <20210331093410.505773-5-lijuan.tu@intel.com> (raw)
In-Reply-To: <20210331093410.505773-1-lijuan.tu@intel.com>

the delta used to difines the difference between real pps and expeced
pps, the case fails if it exceed the absolute number specified by
'accepted_tolerance' in conf/nic_single_core_perf.cfg, the deault is
1Mpps, which is not friendly when the expeced is only a few of million,
so updated the accepted_tolerance to a percentage, considering the CI
intergation, so the type is float.

Here are main changes:
* the meaning of accepted_tolerance
    before: threshold for (real-mpps - expected-mpps)
    after: threshold for (real-mpps - expected-mpps)/expected-mpps
* Pass criteria
    before:
      real-mpps - expected-mpps > -accepted_tolerance
    after:
      (real-mpps - expected-mpps)/expected-mpps > -accepted_tolerance
* rst table
    before:
      ... +---------------+-----------------+--------+
      ... | Expected-Mpps | Real - Expected | Status |
      ... +===============+=================+========+
      ... | 9.900         | -0.040          | PASS   |
      ... +---------------+-----------------+--------+

    after:
      ... +---------------+------------+--------+
      ... | Expected-Mpps | Fluc Ratio | Status |
      ... +===============+============+========+
      ... | 9.900         | -0.004     | PASS   |
      ... +---------------+------------+--------+
* delta in JSON:
    before: real-mpps - expected-mpps
    after: (real-mpps - expected-mpps)/expected-mpps

Signed-off-by: Lijuan Tu <lijuan.tu@intel.com>
---
 conf/nic_single_core_perf.cfg           | 10 +++++---
 tests/TestSuite_nic_single_core_perf.py | 32 ++++++++++++-------------
 2 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/conf/nic_single_core_perf.cfg b/conf/nic_single_core_perf.cfg
index e6b0a5af..72b8c533 100644
--- a/conf/nic_single_core_perf.cfg
+++ b/conf/nic_single_core_perf.cfg
@@ -11,8 +11,12 @@
 #  - test_duration is how many seconds each combination performance will
 #  be recorded.
 #
-#  - accepted_tolerance defines the accepted tolerance between test
-# results and expected numbers.
+#  - accepted_tolerance defines the accepted tolerance for the
+#  fluctuation ratio of performance, it is a float(e.g. 0.05).
+#  Assume:
+#    F = (Real-Mpps - Expected-Mpps)/Expected-Mpps
+#    T = accepted_tolerance
+#  If F < -T, the case is PASS, else it is FAIL
 #
 #  - expected_throughput is a dictionary defining expected throughput
 # numbers based on NIC, and the pattern is
@@ -38,7 +42,7 @@ test_parameters = {'1C/1T': {64: [512, 2048]},
 rx_desc_16byte = 'y'
 test_duration = 60
 throughput_stat_sample_interval = 2
-accepted_tolerance = 1
+accepted_tolerance = 0.05
 expected_throughput = {
     'fortville_spirit': {
         '1C/1T': {64: {512: 0.00, 2048: 0.00}},
diff --git a/tests/TestSuite_nic_single_core_perf.py b/tests/TestSuite_nic_single_core_perf.py
index 095b508a..dd35caac 100644
--- a/tests/TestSuite_nic_single_core_perf.py
+++ b/tests/TestSuite_nic_single_core_perf.py
@@ -116,12 +116,12 @@ class TestNicSingleCorePerf(TestCase):
         # {'$framesize':{"$nb_desc": 'throughput'}
         self.throughput = {}
 
-        # Accepted tolerance in Mpps
-        self.gap = self.get_suite_cfg()['accepted_tolerance']
+        # Accepted tolerance is ratio
+        self.gap = self.get_suite_cfg().get('accepted_tolerance', 0.1)
 
         # header to print test result table
         self.table_header = ['Fwd_core', 'Frame Size', 'TXD/RXD', 'Real-Mpps', 'Rate',
-                             'Expected-Mpps', 'Real - Expected', 'Status']
+                             'Expected-Mpps', 'Fluc Ratio', 'Status']
         self.test_result = {}
 
     def flows(self):
@@ -332,12 +332,16 @@ class TestNicSingleCorePerf(TestCase):
                     ret_data[header[3]] = "{:.3f}".format(_real)
                     ret_data[header[4]] = "{:.3f}%".format(_real * 100 / wirespeed)
                     ret_data[header[5]] = "{:.3f}".format(_exp)
-                    delta = _real - _exp
-                    ret_data[header[6]] = "{:.3f}".format(delta)
-                    if delta > -self.gap:
-                        ret_data[header[7]] = 'PASS'
+                    delta = (_real - _exp)/_exp
+                    if _exp != 0:
+                        ret_data[header[6]] = "{:.3f}".format(delta)
+                        if delta > -self.gap:
+                            ret_data[header[7]] = 'PASS'
+                        else:
+                            ret_data[header[7]] = 'FAIL'
                     else:
-                        ret_data[header[7]] = 'FAIL'
+                        ret_data[header[6]] = "N/A"
+                        ret_data[header[7]] = 'PASS'
 
                     ret_datas[frame_size][nb_desc] = deepcopy(ret_data)
                 self.test_result[fwd_config] = deepcopy(ret_datas)
@@ -376,15 +380,9 @@ class TestNicSingleCorePerf(TestCase):
                     row_dict0 = dict()
                     row_dict0['performance'] = list()
                     row_dict0['parameters'] = list()
-                    result_throughput = float(row_in['Real-Mpps'])
-                    expected_throughput = float(row_in['Expected-Mpps'])
-                    # delta value and accepted tolerance in percentage
-                    delta = result_throughput - expected_throughput
-                    if delta > -self.gap:
-                        row_dict0['status'] = 'PASS'
-                    else:
-                        row_dict0['status'] = 'FAIL'
-                    row_dict1 = dict(name="Throughput", value=result_throughput, unit="Mpps", delta=delta)
+                    row_dict0['status'] = row_in['Status']
+                    row_dict1 = dict(name="Throughput", value=row_in['Real-Mpps'], unit="Mpps",
+                                     delta=row_in['Fluc Ratio'])
                     row_dict2 = dict(name="Txd/Rxd", value=row_in["TXD/RXD"], unit="descriptor")
                     row_dict3 = dict(name="frame_size", value=row_in["Frame Size"], unit="bytes")
                     row_dict4 = dict(name="Fwd_core", value=row_in["Fwd_core"])
-- 
2.25.1


  parent reply	other threads:[~2021-03-31  1:35 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-31  9:34 [dts] [v2 0/6] revise tests/nic_signle_core_perf Lijuan Tu
2021-03-31  9:34 ` [dts] [v2 1/6] tests/nic_single_core_perf: beauty JSON string Lijuan Tu
2021-03-31  9:34 ` [dts] [v2 2/6] tests/nic_single_core_perf: get better throughput stats Lijuan Tu
2021-03-31  9:34 ` [dts] [v2 3/6] tests/nic_single_core_perf: revise rst table Lijuan Tu
2021-03-31  9:34 ` Lijuan Tu [this message]
2021-03-31  9:34 ` [dts] [v2 5/6] tests/nic_single_core_perf: add expected mpps into JSON Lijuan Tu
2021-03-31  9:34 ` [dts] [v2 6/6] tests/nic_single_core_perf: revise rx desc size Lijuan Tu
2021-06-25  5:21 ` [dts] [v2 0/6] revise tests/nic_signle_core_perf Tu, Lijuan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210331093410.505773-5-lijuan.tu@intel.com \
    --to=lijuan.tu@intel.com \
    --cc=alialnu@nvidia.com \
    --cc=dliu@iol.unh.edu \
    --cc=dts@dpdk.org \
    --cc=ohilyard@iol.unh.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).