[dts] [DTS][PATCH V4 2/3] test_plans/nic_single_core_perf_test_plan.rst: add new test plan.

[dts] [DTS][PATCH V4 2/3] test_plans/nic_single_core_perf_test_plan.rst: add new test plan.

[wang fei]

For next branch: add new test plan for nic_single_core_perf test suite.

Signed-off-by: wang fei <feix.y.wang@intel.com>
---
 test_plans/nic_single_core_perf_test_plan.rst | 116 ++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)
 create mode 100644 test_plans/nic_single_core_perf_test_plan.rst

diff --git a/test_plans/nic_single_core_perf_test_plan.rst b/test_plans/nic_single_core_perf_test_plan.rst
new file mode 100644
index 0000000..ec67ad4
--- /dev/null
+++ b/test_plans/nic_single_core_perf_test_plan.rst
@@ -0,0 +1,116 @@
+.. Copyright (c) <2016>, Intel Corporation
+   All rights reserved.
+   
+   Redistribution and use in source and binary forms, with or without
+   modification, are permitted provided that the following conditions
+   are met:
+   
+   - Redistributions of source code must retain the above copyright
+     notice, this list of conditions and the following disclaimer.
+   
+   - Redistributions in binary form must reproduce the above copyright
+     notice, this list of conditions and the following disclaimer in
+     the documentation and/or other materials provided with the
+     distribution.
+   
+   - Neither the name of Intel Corporation nor the names of its
+     contributors may be used to endorse or promote products derived
+     from this software without specific prior written permission.
+   
+   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+   FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+   COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+   INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+   (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+   SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+   HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+   STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+   ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+   OF THE POSSIBILITY OF SUCH DAMAGE.
+
+======================================================================
+Benchmark the performance of single core forwarding with FVL25G/NNT10G
+======================================================================
+
+Prerequisites
+=============
+
+1. Hardware:
+    1) nic_single_core_perf test for FVL25G: two dual port FVL25G nics,
+        all installed on the same socket, pick one port per nic
+    3) nic_single_core_perf test for NNT10G : four 82599 nics,
+        all installed on the same socket, pick one port per nic
+  
+2. Software: 
+    dpdk: git clone http://dpdk.org/git/dpdk
+    scapy: http://www.secdev.org/projects/scapy/
+    dts (next branch): git clone http://dpdk.org/git/tools/dts, 
+                       then "git checkout next" 
+    Trex code: http://trex-tgn.cisco.com/trex/release/v2.26.tar.gz 
+    python-prettytable: 
+        apt install python-prettytable (for ubuntu os) 
+        or dnf install python-prettytable (for fedora os). 
+
+3. Connect all the selected nic ports to traffic generator(IXIA,TREX,
+   PKTGEN) ports(TG ports).
+    2 TG 25g ports for FVL25G ports
+    4 TG 10g ports for 4 NNT10G ports
+    
+Test Case : Single Core Performance Measurement
+===============================================
+1) Bind tested ports to igb_uio
+
+2) Start testpmd::
+
+     ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x6 -n 4 -- -i \
+         --portmask=0xf  --txd=128 --rxd=128
+        testpmd> start
+        
+3) Configure traffic generator to send traffic
+    configure one below stream for each TG port
+        dst mac: peer nic port mac
+        src ip : random
+        dst ip : random
+        packet length : 64 byte
+        
+4)  check throughput and compare it with the expected value.
+
+5)  for NNT10G, repeat above step 1-4 for txd=rxd=512,2048 separately.
+    for FVL25G nic, just test txd=rxd=512,2048 following above steps 
+    1-4.
+
+6) Result tables for different NICs:
+
+        FVL25G:
++------------+---------+-------------+---------+---------------------+
+| Frame Size | TXD/RXD |  Throughput |   Rate  | Expected Throughput |
++------------+---------+-------------+---------+---------------------+
+|     64     |   512   | xxxxxx Mpps |   xxx % |     xxx    Mpps     |
+|     64     |   2048  | xxxxxx Mpps |   xxx % |     xxx    Mpps     |
++------------+---------+-------------+---------+---------------------+
+
+        for NNT10G:
++------------+---------+-------------+---------+---------------------+
+| Frame Size | TXD/RXD |  Throughput |   Rate  | Expected Throughput |
++------------+---------+-------------+---------+---------------------+
+|     64     |   128   | xxxxxx Mpps |   xxx % |       xxx  Mpps     |
+|     64     |   512   | xxxxxx Mpps |   xxx % |       xxx  Mpps     |
+|     64     |   2048  | xxxxxx Mpps |   xxx % |       xxx  Mpps     |
++------------+---------+-------------+---------+---------------------+
+
+Note : The values for the expected throughput may vary due to different
+       platform and OS, and traffic generator, please correct threshold
+       values accordingly. (the above expected values for FVL 25G and
+       NNT10G  were got from the combination of Purly,Ubuntu 16.04, and
+       traffic generator IXIA) 
+
+Case will raise failure if actual throughputs have more than 1Mpps gap 
+from expected ones. 
+
-- 
2.7.4

[dts] [DTS][PATCH V4 3/3] conf/nic_single_core_perf.cfg: add suite configuration file

[wang fei]

For next branch: add the expected throughput values for NICs: 
"niantic"(4 ports), "fortville_25g"(2 ports), and the values 
will be parsed by the test suite.

Signed-off-by: wang fei <feix.y.wang@intel.com>
---
 conf/nic_single_core_perf.cfg | 10 ++++++++++
 1 file changed, 10 insertions(+)
 create mode 100644 conf/nic_single_core_perf.cfg

diff --git a/conf/nic_single_core_perf.cfg b/conf/nic_single_core_perf.cfg
new file mode 100644
index 0000000..8746326
--- /dev/null
+++ b/conf/nic_single_core_perf.cfg
@@ -0,0 +1,10 @@
+[suite]
+
+# list expected throughput values for 64byte packet size and different
+# txd/rxd, these values may vary dut to different test enviroment,
+# please adjust accordingly, the below nnt and fvl25g expected
+# throughputs were tested under Purly,Ubuntu 16.04, IXIA
+# the formate for the expected throughput is a dictinary which means 
+#    {"packet size": {"descriptor ring size": "throughput"}}

+throughput_nnt = {64: {128: 53.435, 512: 53.699, 2048: 42.798}}
+throughput_fvl25g = {64: {512: 43.777, 2048: 43.651}}
-- 
2.7.4

[dts] [DTS][PATCH v4 1/3] tests/TestSuite_nic_single_core_perf.py: add a new test suite for performance

[wang fei]

For next branch: conducts single core performace test for NICs: 
"niantic"(4 ports required),"fortville_25g"(2 ports required), 
the traffic generator is TREX, and there is a flag to determine 
if save the test result as a separate file or not.


Signed-off-by: wang fei <feix.y.wang@intel.com>
---
 tests/TestSuite_nic_single_core_perf.py | 284 ++++++++++++++++++++++++++++++++
 1 file changed, 284 insertions(+)
 create mode 100644 tests/TestSuite_nic_single_core_perf.py

diff --git a/tests/TestSuite_nic_single_core_perf.py b/tests/TestSuite_nic_single_core_perf.py
new file mode 100644
index 0000000..2d6a047
--- /dev/null
+++ b/tests/TestSuite_nic_single_core_perf.py
@@ -0,0 +1,284 @@
+#
+# Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
+DPDK Test suite.
+"""
+
+import utils
+import re
+import time
+from test_case import TestCase
+from time import sleep
+from settings import HEADER_SIZE
+from pmd_output import PmdOutput
+from copy import deepcopy
+from prettytable import PrettyTable
+
+class TestNicSingleCorePerf(TestCase):
+
+    def set_up_all(self):
+        """
+        Run at the start of each test suite.
+        PMD prerequisites.
+        """
+
+        self.frame_sizes = [64]
+        self.headers_size = HEADER_SIZE['eth'] + HEADER_SIZE['ip'] + HEADER_SIZE['tcp']
+        self.ixgbe_descriptors = [128, 512, 2048]
+        self.i40e_descriptors = [512, 2048]
+
+        # traffic duraion in second
+        self.trafficDuration = 60
+
+        #load the expected throughput for required nic
+        self.expected_throughput_nnt = self.parse_string(self.get_suite_cfg()["throughput_nnt"])
+        self.expected_throughput_fvl25g = self.parse_string(self.get_suite_cfg()["throughput_fvl25g"])
+
+        # The acdepted gap between expected throughput and actual throughput, 1 Mpps
+        self.gap = 1
+
+        # header to print test result table
+        self.table_header = ['Frame Size', 'TXD/RXD', 'Throughput', 'Rate', 'Expected Throughput']
+
+        # Update config file and rebuild to get best perf on FVL
+        if self.nic in ["fortville_25g"]:
+            self.dut.send_expect("sed -i -e 's/CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC=n/CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC=y/' ./config/common_base", "#", 20)
+            self.dut.build_install_dpdk(self.target)
+
+        # Based on h/w type, choose how many ports to use
+        self.dut_ports = self.dut.get_ports()
+
+        self.socket = self.dut.get_numa_id(self.dut_ports[0])
+
+        self.pmdout = PmdOutput(self.dut)
+
+        self.test_result = {}
+
+        # determine if to save test result as a separated file
+        self.save_result_flag =  True
+
+    def set_up(self):
+        """
+        Run before each test case.
+        """
+        if self.nic == "niantic":
+            self.descriptors = self.ixgbe_descriptors
+        elif self.nic in ["fortville_25g"]:
+            self.descriptors = self.i40e_descriptors
+        else:
+            raise Exception("Not required NIC")
+
+    def test_nic_single_core_perf(self):
+        """
+        Run nic single core performance 
+        """
+        self.verify(len(self.dut_ports) == 2 or len(self.dut_ports) == 4, "Require 2 or 4 ports to test")
+        self.verify(self.nic in ['niantic','fortville_25g'], "Not required NIC ")
+        if len(self.dut_ports) == 2:
+            self.perf_test(2)   
+        elif len(self.dut_ports) == 4:
+            self.perf_test(4)
+
+    def perf_test(self, port_num):
+        """
+        Single core Performance Benchmarking test
+        """
+        # traffic option
+        options = {
+             'rate' : '100%',
+             #'ip': {'action': 'inc', 'mask' : '255.255.255.0', 'step': '0.0.0.1'}
+            }
+
+        header = self.table_header
+        if port_num == 2:
+            pci0 = self.dut.ports_info[0]['pci']
+            pci1 = self.dut.ports_info[1]['pci']
+            eal = "-w %s -w %s" % (pci0, pci1)
+        elif port_num == 4:
+            pci0 = self.dut.ports_info[0]['pci']
+            pci1 = self.dut.ports_info[1]['pci']
+            pci2 = self.dut.ports_info[2]['pci']
+            pci3 = self.dut.ports_info[3]['pci']
+            eal = "-w %s -w %s -w %s -w %s" % (pci0, pci1, pci2, pci3)
+
+        # run testpmd with 2 cores
+        core_config = "1S/2C/1T"
+        core_list = self.dut.get_core_list(core_config, socket=self.socket)
+        port_mask = utils.create_mask(self.dut_ports)
+
+        for frame_size in self.frame_sizes:
+            ret_datas = {}
+            for descriptor in self.descriptors:
+                self.logger.info("Executing Test Using cores: %s" % core_list)
+                self.pmdout.start_testpmd(core_config, "--portmask=%s --txd=%d --rxd=%d" % (port_mask, descriptor, descriptor),eal, socket=self.socket)
+                self.dut.send_expect("start", "testpmd> ", 15)
+
+                self.logger.info("Running with frame size %d " % frame_size)
+
+                # create pcap file
+                payload_size = frame_size - self.headers_size
+                self.tester.scapy_append(
+                        'wrpcap("test.pcap", [Ether(src="52:00:00:00:00:00")/IP(src="1.2.3.4",dst="1.1.1.1")/TCP()/("X"*%d)])' % payload_size)
+                self.tester.scapy_execute()
+
+                # send the traffic
+                streams = self.prepare_stream(port_num, options)      
+                _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, delay=self.trafficDuration)
+
+                throughput = packets_received / 1000000.0
+
+                self.dut.send_expect("stop", "testpmd> ")
+                self.dut.send_expect("quit", "# ", 30)
+
+                self.logger.info("Throughput result for Descriptor :%s is :%s Mpps" % (descriptor, throughput))
+
+                wirespeed = self.wirespeed(self.nic, frame_size, port_num)
+
+                # one entry for test result record
+                ret_data = {}
+                ret_data[header[0]] = frame_size
+                ret_data[header[1]] = descriptor
+                ret_data[header[2]] = str(float("%.3f" % throughput)) + " Mpps"
+                ret_data[header[3]] = str(float("%.3f" % (throughput * 100 / wirespeed))) + "%"
+                if self.nic == "niantic":
+                    ret_data[header[4]] = self.expected_throughput_nnt[str(frame_size)][str(descriptor)] + " Mpps"
+                elif self.nic == "fortville_25g":
+                    ret_data[header[4]] = self.expected_throughput_fvl25g[str(frame_size)][str(descriptor)] + " Mpps"
+                ret_datas[descriptor] = deepcopy(ret_data)
+                self.test_result[frame_size] = deepcopy(ret_datas)
+        
+        for frame_size in self.frame_sizes:
+            for descriptor in self.descriptors:
+                self.verify(self.test_result[frame_size][descriptor][header[2]] > 0, "No traffic detected")
+
+        # Print results
+        self.result_table_create(header)
+        for frame_size in self.frame_sizes:
+            for descriptor in self.descriptors:
+                table_row = [self.test_result[frame_size][descriptor][header[0]]]
+                table_row.append(self.test_result[frame_size][descriptor][header[1]])
+                table_row.append(self.test_result[frame_size][descriptor][header[2]])
+                table_row.append(self.test_result[frame_size][descriptor][header[3]])
+                table_row.append(self.test_result[frame_size][descriptor][header[4]])
+                self.result_table_add(table_row)
+
+        self.result_table_print()
+
+        # save test results as a file
+        if self.save_result_flag:
+            self.save_result(self.test_result)
+
+        # check if the gap between expected throughput and actual throughput exceed accepted gap 
+        for frame_size in self.frame_sizes:
+            for descriptor in self.descriptors:
+                self.verify(float(self.test_result[frame_size][descriptor][header[4]].split()[0]) -
+                    float(self.test_result[frame_size][descriptor][header[2]].split()[0]) < self.gap, "Exceeded Gap")
+
+    def prepare_stream(self, port_num, options):
+        '''
+        create streams for ports, one port one stream
+        '''
+        if port_num == 2:
+            txport0 = self.tester.get_local_port(self.dut.get_ports()[0])
+            txport1 = self.tester.get_local_port(self.dut.get_ports()[1])
+            stream_id0 = self.tester.pktgen.add_stream(txport0, txport1, r'/root/test.pcap')
+            stream_id1 = self.tester.pktgen.add_stream(txport1, txport0, r'/root/test.pcap')
+            self.tester.pktgen.config_stream(stream_id0, options)
+            self.tester.pktgen.config_stream(stream_id1, options)
+            return [stream_id0, stream_id1]
+        elif port_num == 4:
+            txport0 = self.tester.get_local_port(self.dut.get_ports()[0])
+            txport1 = self.tester.get_local_port(self.dut.get_ports()[1])
+            txport2 = self.tester.get_local_port(self.dut.get_ports()[2])
+            txport3 = self.tester.get_local_port(self.dut.get_ports()[3])
+            stream_id0 = self.tester.pktgen.add_stream(txport0, txport1, r'/root/test.pcap')
+            stream_id1 = self.tester.pktgen.add_stream(txport1, txport0, r'/root/test.pcap')
+            stream_id2 = self.tester.pktgen.add_stream(txport2, txport3, r'/root/test.pcap')
+            stream_id3 = self.tester.pktgen.add_stream(txport3, txport2, r'/root/test.pcap')
+            self.tester.pktgen.config_stream(stream_id0, options)
+            self.tester.pktgen.config_stream(stream_id1, options)
+            self.tester.pktgen.config_stream(stream_id2, options)
+            self.tester.pktgen.config_stream(stream_id3, options)
+            return [stream_id0, stream_id1, stream_id2, stream_id3]
+
+    def save_result(self, data):
+        '''
+        Saves the test results as a separated file named with self.nic+_single_core_perf.txt
+        in output folder if self.save_result_flag is True
+        '''
+        header = self.table_header
+        table = PrettyTable(header)
+        for frame_size in self.frame_sizes:
+            for descriptor in self.descriptors:
+                table_row = [self.test_result[frame_size][descriptor][header[0]]]
+                table_row.append(self.test_result[frame_size][descriptor][header[1]])
+                table_row.append(self.test_result[frame_size][descriptor][header[2]])
+                table_row.append(self.test_result[frame_size][descriptor][header[3]])
+                table_row.append(self.test_result[frame_size][descriptor][header[4]])
+                table.add_row(table_row)
+        file_to_save = open("output/%s_single_core_perf.txt" % self.nic, 'w')
+        file_to_save.write(str(table))
+        file_to_save.close()
+
+    def parse_string(self, string):
+        '''
+        Parse a string in the formate of a dictionary and convert it into a real dictionary. 
+        '''
+        element_pattern = re.compile(".\d+:.*?}")
+        string_elements = element_pattern.findall(string)
+        ret = {}
+        for element in string_elements:
+            ex_pattern = re.compile("(\d+): *{(.*)}")
+            ex_ret = ex_pattern.search(element)
+            ret[ex_ret.groups()[0]] = ex_ret.groups()[1]
+            inner_datas = ex_ret.groups()[1].split(",")
+            ret_inner = {}
+            for data in inner_datas:
+                match_inner = data.split(":")
+                ret_inner[match_inner[0].strip()] = match_inner[1].strip()
+            ret[ex_ret.groups()[0]] = deepcopy(ret_inner)
+        return ret
+
+    def tear_down(self):
+        """
+        Run after each test case.
+        """
+        if self.nic in ["fortville_25g"]:
+            self.dut.send_expect("sed -i -e 's/CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC=y/CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC=n/' ./config/common_base", "#", 20)    
+
+    def tear_down_all(self):
+        """
+        Run after each test suite.
+        """
+        self.dut.kill_all()
-- 
2.7.4

Re: [dts] [DTS][PATCH v4 1/3] tests/TestSuite_nic_single_core_perf.py: add a new test suite for performance

[Liu, Yong]

Thanks Fei, applied into next branch.

On 02/05/2018 03:58 PM, wang fei wrote:
> For next branch: conducts single core performace test for NICs:
> "niantic"(4 ports required),"fortville_25g"(2 ports required),
> the traffic generator is TREX, and there is a flag to determine
> if save the test result as a separate file or not.
>
>
> Signed-off-by: wang fei<feix.y.wang@intel.com>
> ---