test suite reviews and discussions
 help / color / mirror / Atom feed
From: Lijuan Tu <lijuan.tu@intel.com>
To: dliu@iol.unh.edu, ohilyard@iol.unh.edu, alialnu@nvidia.com
Cc: dts@dpdk.org, Lijuan Tu <lijuan.tu@intel.com>
Subject: [dts] [v2 2/6] tests/nic_single_core_perf: get better throughput stats
Date: Wed, 31 Mar 2021 09:34:06 +0000	[thread overview]
Message-ID: <20210331093410.505773-3-lijuan.tu@intel.com> (raw)
In-Reply-To: <20210331093410.505773-1-lijuan.tu@intel.com>

Snice throughput is easily fluctuated, we want to imporve reliable
for results by following:

* get multiple throughput statistics during the run time.
* remove the maximum and minimum and get average.
* sample interval could be configed by users

Signed-off-by: Lijuan Tu <lijuan.tu@intel.com>
---
 conf/nic_single_core_perf.cfg           |  4 +++
 tests/TestSuite_nic_single_core_perf.py | 42 +++++++++++++++++++------
 2 files changed, 37 insertions(+), 9 deletions(-)

diff --git a/conf/nic_single_core_perf.cfg b/conf/nic_single_core_perf.cfg
index 4d3279ba..e6b0a5af 100644
--- a/conf/nic_single_core_perf.cfg
+++ b/conf/nic_single_core_perf.cfg
@@ -5,6 +5,9 @@
 # numbers, and the pattern is
 #    {'frame size': ['descriptor number #1', 'descriptor number #2']}
 #
+#  - throughput_stat_sample_interval defines interval of get throughput
+#  statistics (second). If not set, it is 5 seconds by default.
+#
 #  - test_duration is how many seconds each combination performance will
 #  be recorded.
 #
@@ -34,6 +37,7 @@ test_parameters = {'1C/1T': {64: [512, 2048]},
                     '1C/2T': {64: [512, 2048]}}
 rx_desc_16byte = 'y'
 test_duration = 60
+throughput_stat_sample_interval = 2
 accepted_tolerance = 1
 expected_throughput = {
     'fortville_spirit': {
diff --git a/tests/TestSuite_nic_single_core_perf.py b/tests/TestSuite_nic_single_core_perf.py
index edb4465d..a0a257dd 100644
--- a/tests/TestSuite_nic_single_core_perf.py
+++ b/tests/TestSuite_nic_single_core_perf.py
@@ -41,6 +41,7 @@ from exception import VerifyFailure
 from settings import HEADER_SIZE, UPDATE_EXPECTED, load_global_setting
 from pmd_output import PmdOutput
 from copy import deepcopy
+from numpy import mean
 import rst
 from pktgen import PacketGeneratorHelper
 
@@ -96,6 +97,7 @@ class TestNicSingleCorePerf(TestCase):
 
         # traffic duraion in second
         self.test_duration = self.get_suite_cfg()['test_duration']
+        self.throughput_stat_sample_interval = self.get_suite_cfg().get('throughput_stat_sample_interval', 5)
 
         # load the expected throughput for required nic
         if self.nic in ["ConnectX4_LX_MT4117"]:
@@ -265,21 +267,43 @@ class TestNicSingleCorePerf(TestCase):
                     # run packet generator
                     streams = self.pktgen_helper.prepare_stream_from_tginput(tgenInput, 100, vm_config, self.tester.pktgen)
                     # set traffic option
-                    traffic_opt = {'duration': self.test_duration}
-                    # _, pps = self.tester.traffic_generator_throughput(tgenInput, rate_percent=100, delay=30)
-                    _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
-                    self.verify(packets_received > 0, "No traffic detected")
-                    throughput = packets_received / 1000000.0
-                    self.throughput[fwd_config][frame_size][nb_desc] = throughput
+                    traffic_opt = {
+                        'method': 'throughput',
+                        'rate': 100,
+                        'duration': self.test_duration,
+                        'interval': self.throughput_stat_sample_interval,
+                        }
+                    stats = self.tester.pktgen.measure(stream_ids=streams, traffic_opt=traffic_opt)
+
+                    #####################################################
+                    # Remove max and min if count >=5, then get average
+                    #####################################################
+                    if isinstance(stats, list):
+                        total_pps_rxs = []
+                        c = len(stats)
+                        for i in range(c):
+                            stats_pps = stats[i][1]
+                            if isinstance(stats_pps, tuple):
+                                total_pps_rxs.append(stats_pps[1])
+                            else:
+                                total_pps_rxs.append(stats_pps)
+                        if c >= 5:
+                            total_pps_rxs.remove(max(total_pps_rxs))
+                            total_pps_rxs.remove(min(total_pps_rxs))
+                        total_pps_rx = mean(total_pps_rxs)
+                    else:
+                        total_pps_rx = stats
+
+                    self.verify(total_pps_rx > 0, "No traffic detected, please check your configuration")
+                    total_mpps_rx = total_pps_rx / 1000000.0
+                    self.throughput[fwd_config][frame_size][nb_desc] = total_mpps_rx
 
                     self.dut.send_expect("stop", "testpmd> ")
                     self.dut.send_expect("quit", "# ", 30)
 
-                    self.verify(throughput,
-                        "No traffic detected, please check your configuration")
                     self.logger.info("Trouthput of " +
                         "framesize: {}, rxd/txd: {} is :{} Mpps".format(
-                            frame_size, nb_desc, throughput))
+                            frame_size, nb_desc, total_mpps_rx))
 
         return self.throughput
 
-- 
2.25.1


  parent reply	other threads:[~2021-03-31  1:35 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-31  9:34 [dts] [v2 0/6] revise tests/nic_signle_core_perf Lijuan Tu
2021-03-31  9:34 ` [dts] [v2 1/6] tests/nic_single_core_perf: beauty JSON string Lijuan Tu
2021-03-31  9:34 ` Lijuan Tu [this message]
2021-03-31  9:34 ` [dts] [v2 3/6] tests/nic_single_core_perf: revise rst table Lijuan Tu
2021-03-31  9:34 ` [dts] [v2 4/6] tests/nic_single_core_perf: revise delta to ratio Lijuan Tu
2021-03-31  9:34 ` [dts] [v2 5/6] tests/nic_single_core_perf: add expected mpps into JSON Lijuan Tu
2021-03-31  9:34 ` [dts] [v2 6/6] tests/nic_single_core_perf: revise rx desc size Lijuan Tu
2021-06-25  5:21 ` [dts] [v2 0/6] revise tests/nic_signle_core_perf Tu, Lijuan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210331093410.505773-3-lijuan.tu@intel.com \
    --to=lijuan.tu@intel.com \
    --cc=alialnu@nvidia.com \
    --cc=dliu@iol.unh.edu \
    --cc=dts@dpdk.org \
    --cc=ohilyard@iol.unh.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).