[dts][PATCH V3 1/8] tests/efd: split performance plan and suite

[Hongbo Li <hongbox.li@intel.com>]

split performance plan and suite

Signed-off-by: Hongbo Li <hongbox.li@intel.com>
---

v2:
add patch 0
submit index.rst file separately
v3:
solve format problems

 test_plans/efd_test_plan.rst      |  77 +----------
 test_plans/perf_efd_test_plan.rst | 105 +++++++++++++++
 tests/TestSuite_efd.py            | 138 --------------------
 tests/TestSuite_perf_efd.py       | 210 ++++++++++++++++++++++++++++++
 4 files changed, 316 insertions(+), 214 deletions(-)
 create mode 100644 test_plans/perf_efd_test_plan.rst
 create mode 100644 tests/TestSuite_perf_efd.py

diff --git a/test_plans/efd_test_plan.rst b/test_plans/efd_test_plan.rst
index ee440c02..70dd3b53 100644
--- a/test_plans/efd_test_plan.rst
+++ b/test_plans/efd_test_plan.rst
@@ -49,79 +49,4 @@ Start test application and run EFD performance unit test::
 Verify lookup and lookup bulk cpu cycles are reasonable.
 Verify when key size increased, no significant increment in cpu cycles.
 Verify when value bits increased, no significant increment in cpu cycles.
-Compare with cuckoo hash performance result, lookup cycles should be less.
-
-Performance test cases
-==============================
-In EFD sample, EFD work as a flow-level load balancer. Flows are received at
-a front end server before being forwarded to the target back end server for
-processing. This case will measure the performance of flow distribution with
-different parameters.
-
-Nodes: number of back end nodes
-Entries: number of flows to be added in EFD table
-Value bits: number of bits of value that be stored in EFD table
-
-Test Case: Load balancer performance based on node numbers
-----------------------------------------------------------------------
-This case will measure the performance based on node numbers.
-
-+--------------+-------+-----------+------------+
-| Value Bits   | Nodes | Entries   | Throughput |
-+--------------+-------+-----------+------------+
-|  8           |   1   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   2   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   3   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   4   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   5   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   6   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   7   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   8   |    2M     |            |
-+--------------+-------+-----------+------------+
-
-Test Case: Load balancer performance based on flow numbers
------------------------------------------------------------------------
-This case will measure the performance based on flow numbers.
-
-+--------------+-------+-----------+------------+
-| Value Bits   | Nodes | Entries   | Throughput |
-+--------------+-------+-----------+------------+
-|  8           |   2   |    1M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   2   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   2   |    4M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   2   |    8M     |            |
-+--------------+-------+-----------+------------+
-|  8           |   2   |    16M    |            |
-+--------------+-------+-----------+------------+
-|  8           |   2   |    32M    |            |
-+--------------+-------+-----------+------------+
-
-Test Case: Load balancer performance based on value bits
------------------------------------------------------------------------
-Modify different value size which must be between 1 and 32, and it need
-to be configured with the ``-Dc_args=-DRTE_EFD_VALUE_NUM_BITS`` option
-at compile time.
-
-This case will measure the performance based on value bits.
-
-+--------------+-------+-----------+------------+
-| Value Bits   | Nodes | Entries   | Throughput |
-+--------------+-------+-----------+------------+
-|  8           |   2   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  16          |   2   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  24          |   2   |    2M     |            |
-+--------------+-------+-----------+------------+
-|  32          |   2   |    2M     |            |
-+--------------+-------+-----------+------------+
+Compare with cuckoo hash performance result, lookup cycles should be less.
\ No newline at end of file
diff --git a/test_plans/perf_efd_test_plan.rst b/test_plans/perf_efd_test_plan.rst
new file mode 100644
index 00000000..fe3fedd8
--- /dev/null
+++ b/test_plans/perf_efd_test_plan.rst
@@ -0,0 +1,105 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+   Copyright(c) 2010-2017 Intel Corporation
+
+==================================================
+Sample Application Tests: Elastic Flow Distributor
+==================================================
+
+Description
+-----------
+EFD is a distributor library that uses perfect hashing to determine a
+target/value for a given incoming flow key.
+It has the following advantages: 
+1. It uses perfect hashing it does not store the key itself and hence
+lookup performance is not dependent on the key size. 
+2. Target/value can be any arbitrary value hence the system designer
+and/or operator can better optimize service rates and inter-cluster
+network traffic locating. 
+3. Since the storage requirement is much smaller than a hash-based flow
+table (i.e. better fit for CPU cache), EFD can scale to millions of flow
+keys.
+4. With the current optimized library implementation, performance is fully
+scalable with any number of CPU cores.
+
+For more details, please reference to dpdk online programming guide.
+
+Prerequisites
+=============
+Two ports connect to packet generator.
+
+DUT board must be two sockets system and each cpu have more than 16 lcores.
+
+Performance test cases
+==============================
+In EFD sample, EFD work as a flow-level load balancer. Flows are received at
+a front end server before being forwarded to the target back end server for
+processing. This case will measure the performance of flow distribution with
+different parameters.
+
+Nodes: number of back end nodes
+Entries: number of flows to be added in EFD table
+Value bits: number of bits of value that be stored in EFD table
+
+Test Case: Load balancer performance based on node numbers
+----------------------------------------------------------------------
+This case will measure the performance based on node numbers.
+
++--------------+-------+-----------+------------+
+| Value Bits   | Nodes | Entries   | Throughput |
++--------------+-------+-----------+------------+
+|  8           |   1   |    2M     |            |
++--------------+-------+-----------+------------+
+|  8           |   2   |    2M     |            |
++--------------+-------+-----------+------------+
+|  8           |   3   |    2M     |            |
++--------------+-------+-----------+------------+
+|  8           |   4   |    2M     |            |
++--------------+-------+-----------+------------+
+|  8           |   5   |    2M     |            |
++--------------+-------+-----------+------------+
+|  8           |   6   |    2M     |            |
++--------------+-------+-----------+------------+
+|  8           |   7   |    2M     |            |
++--------------+-------+-----------+------------+
+|  8           |   8   |    2M     |            |
++--------------+-------+-----------+------------+
+
+Test Case: Load balancer performance based on flow numbers
+-----------------------------------------------------------------------
+This case will measure the performance based on flow numbers.
+
++--------------+-------+-----------+------------+
+| Value Bits   | Nodes | Entries   | Throughput |
++--------------+-------+-----------+------------+
+|  8           |   2   |    1M     |            |
++--------------+-------+-----------+------------+
+|  8           |   2   |    2M     |            |
++--------------+-------+-----------+------------+
+|  8           |   2   |    4M     |            |
++--------------+-------+-----------+------------+
+|  8           |   2   |    8M     |            |
++--------------+-------+-----------+------------+
+|  8           |   2   |    16M    |            |
++--------------+-------+-----------+------------+
+|  8           |   2   |    32M    |            |
++--------------+-------+-----------+------------+
+
+Test Case: Load balancer performance based on value bits
+-----------------------------------------------------------------------
+Modify different value size which must be between 1 and 32, and it need
+to be configured with the ``-Dc_args=-DRTE_EFD_VALUE_NUM_BITS`` option
+at compile time.
+
+This case will measure the performance based on value bits.
+
++--------------+-------+-----------+------------+
+| Value Bits   | Nodes | Entries   | Throughput |
++--------------+-------+-----------+------------+
+|  8           |   2   |    2M     |            |
++--------------+-------+-----------+------------+
+|  16          |   2   |    2M     |            |
++--------------+-------+-----------+------------+
+|  24          |   2   |    2M     |            |
++--------------+-------+-----------+------------+
+|  32          |   2   |    2M     |            |
++--------------+-------+-----------+------------+
diff --git a/tests/TestSuite_efd.py b/tests/TestSuite_efd.py
index 9d94a3d4..071a3564 100644
--- a/tests/TestSuite_efd.py
+++ b/tests/TestSuite_efd.py
@@ -68,144 +68,6 @@ class TestEFD(TestCase):
         self.dut.send_expect("quit", "# ")
         self.verify("Test OK" in out, "Test failed")
 
-    def test_perf_efd_nodenum(self):
-        """
-        Run EFD perf evaluation for number of nodes
-        """
-        self.verify(len(self.dut_ports) >= 2, "Not enough ports")
-        node_nums = [1, 2, 3, 4, 5, 6, 7, 8]
-
-        flow_num = 1024 * 1024 * 2
-
-        table_header = ["Value Bits", "Nodes", "Flow Entries", "Throughput(mpps)"]
-
-        self.result_table_create(table_header)
-        # perf of different nodes
-        for node_num in node_nums:
-            pps = self._efd_perf_evaluate(node_num, flow_num)
-
-            self.result_table_add([8, node_num, "2M", pps])
-
-        self.result_table_print()
-
-    def test_perf_efd_flownums(self):
-        """
-        Run EFD perf evaluation for millions of flows
-        """
-        self.logger.warning(
-            "Millions of flow required huge memory, please allocate 16G hugepage"
-        )
-        self.dut.setup_memory_linux(hugepages=8192)
-        self.verify(len(self.dut_ports) >= 2, "Not enough ports")
-        flow_nums = [
-            1024 * 1024,
-            1024 * 1024 * 2,
-            1024 * 1024 * 4,
-            1024 * 1024 * 8,
-            1024 * 1024 * 16,
-            1024 * 1024 * 32,
-        ]
-
-        table_header = ["Value Bits", "Nodes", "Million Flows", "Throughput(mpps)"]
-
-        self.result_table_create(table_header)
-
-        # perf of different flow numbers
-        for flow_num in flow_nums:
-            pps = self._efd_perf_evaluate(2, flow_num)
-
-            self.result_table_add([8, 2, flow_num / (1024 * 1024), pps])
-
-        self.result_table_print()
-
-    def test_perf_efd_valuesize(self):
-        """
-        Run EFD perf evaluation for different value size
-        """
-        self.verify(len(self.dut_ports) >= 2, "Not enough ports")
-        val_bitnums = [8, 16, 24, 32]
-        flow_num = 1024 * 1024 * 2
-
-        table_header = ["Value Bits", "Nodes", "Flow Entries", "Throughput(mpps)"]
-
-        self.result_table_create(table_header)
-        # perf of different value bit lengths
-        for val_bitnum in val_bitnums:
-            # change value length and rebuild dpdk
-            extra_options = "-Dc_args=-DRTE_EFD_VALUE_NUM_BITS=%d" % val_bitnum
-            self.dut.build_install_dpdk(self.target, extra_options=extra_options)
-            self.build_server_node_efd()
-
-            pps = self._efd_perf_evaluate(2, flow_num)
-            self.result_table_add([val_bitnum, 2, "2M", pps])
-
-        self.result_table_print()
-        extra_options = "-Dc_args=-DRTE_EFD_VALUE_NUM_BITS=8"
-        self.dut.build_install_dpdk(self.target, extra_options=extra_options)
-        self.build_server_node_efd()
-
-    def _efd_perf_evaluate(self, node_num, flow_num):
-        # extended flow number into etgen module
-
-        # output port is calculated from overall ports number
-        server_cmd_fmt = "%s %s -- -p 0x3 -n %d -f %s"
-        node_cmd_fmt = "%s %s --proc-type=secondary -- -n %d"
-        socket = self.dut.get_numa_id(self.dut_ports[0])
-
-        pcap = os.sep.join([self.output_path, "efd.pcap"])
-        self.tester.scapy_append(
-            'wrpcap("%s", [Ether()/IP(src="0.0.0.0", dst="0.0.0.0")/("X"*26)])' % pcap
-        )
-        self.tester.scapy_execute()
-
-        tgen_input = []
-        rx_port = self.tester.get_local_port(self.dut_ports[0])
-        tx_port = self.tester.get_local_port(self.dut_ports[1])
-
-        pcap = os.sep.join([self.output_path, "efd.pcap"])
-        tgen_input.append((tx_port, rx_port, pcap))
-        tgen_input.append((rx_port, tx_port, pcap))
-
-        cores = self.dut.get_core_list("1S/%dC/1T" % (node_num + 2), socket)
-
-        self.verify(len(cores), "Can't find enough cores")
-
-        eal_para = self.dut.create_eal_parameters(cores=cores[0:2], ports=[0, 1])
-        server_cmd = server_cmd_fmt % (
-            self.server_app,
-            eal_para,
-            node_num,
-            hex(flow_num),
-        )
-        # create table may need few minutes
-        self.dut.send_expect(server_cmd, "Finished Process Init", timeout=240)
-
-        node_sessions = []
-        for node in range(node_num):
-
-            eal_para = self.dut.create_eal_parameters(cores=[cores[2 + node]])
-            node_cmd = node_cmd_fmt % (self.node_app, eal_para, node)
-            node_session = self.dut.new_session(suite="node%d" % node)
-            node_sessions.append(node_session)
-            node_session.send_expect(node_cmd, "Finished Process Init", timeout=30)
-
-        # clear streams before add new streams
-        self.tester.pktgen.clear_streams()
-        # run packet generator
-        streams = self.pktgen_helper.prepare_stream_from_tginput(
-            tgen_input, 100, None, self.tester.pktgen
-        )
-        _, pps = self.tester.pktgen.measure_throughput(stream_ids=streams)
-
-        for node_session in node_sessions:
-            node_session.send_expect("^C", "#")
-            self.dut.close_session(node_session)
-
-        self.dut.send_expect("^C", "#")
-
-        pps /= 1000000.0
-        return pps
-
     def set_fields(self):
         """set ip protocol field behavior"""
         fields_config = {
diff --git a/tests/TestSuite_perf_efd.py b/tests/TestSuite_perf_efd.py
new file mode 100644
index 00000000..5d638d2d
--- /dev/null
+++ b/tests/TestSuite_perf_efd.py
@@ -0,0 +1,210 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2017 Intel Corporation
+#
+
+"""
+DPDK Test suite.
+"""
+import os
+import re
+
+import framework.utils as utils
+from framework.pktgen import PacketGeneratorHelper
+from framework.test_case import TestCase
+
+
+class TestEFD(TestCase):
+    def set_up_all(self):
+        """
+        Run at the start of each test suite.
+        """
+
+        self.build_server_node_efd()
+
+        self.dut_ports = self.dut.get_ports()
+        self.node_app = self.dut.apps_name["node"]
+        self.server_app = self.dut.apps_name["server"]
+        self.app_test_path = self.dut.apps_name["test"]
+        # get dts output path
+        if self.logger.log_path.startswith(os.sep):
+            self.output_path = self.logger.log_path
+        else:
+            cur_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+            self.output_path = os.sep.join([cur_path, self.logger.log_path])
+        # create an instance to set stream field setting
+        self.pktgen_helper = PacketGeneratorHelper()
+
+    def build_server_node_efd(self):
+        apps = ["node", "server"]
+        for app in apps:
+            out = self.dut.build_dpdk_apps("./examples/server_node_efd/%s" % app)
+            self.verify("Error" not in out, "Compilation %s error" % app)
+            self.verify("No such" not in out, "Compilation %s error" % app)
+
+    def set_up(self):
+        """
+        Run before each test case.
+        """
+        pass
+
+    def test_perf_efd_nodenum(self):
+        """
+        Run EFD perf evaluation for number of nodes
+        """
+        self.verify(len(self.dut_ports) >= 2, "Not enough ports")
+        node_nums = [1, 2, 3, 4, 5, 6, 7, 8]
+
+        flow_num = 1024 * 1024 * 2
+
+        table_header = ["Value Bits", "Nodes", "Flow Entries", "Throughput(mpps)"]
+
+        self.result_table_create(table_header)
+        # perf of different nodes
+        for node_num in node_nums:
+            pps = self._efd_perf_evaluate(node_num, flow_num)
+
+            self.result_table_add([8, node_num, "2M", pps])
+
+        self.result_table_print()
+
+    def test_perf_efd_flownums(self):
+        """
+        Run EFD perf evaluation for millions of flows
+        """
+        self.logger.warning(
+            "Millions of flow required huge memory, please allocate 16G hugepage"
+        )
+        self.dut.setup_memory_linux(hugepages=8192)
+        self.verify(len(self.dut_ports) >= 2, "Not enough ports")
+        flow_nums = [
+            1024 * 1024,
+            1024 * 1024 * 2,
+            1024 * 1024 * 4,
+            1024 * 1024 * 8,
+            1024 * 1024 * 16,
+            1024 * 1024 * 32,
+        ]
+
+        table_header = ["Value Bits", "Nodes", "Million Flows", "Throughput(mpps)"]
+
+        self.result_table_create(table_header)
+
+        # perf of different flow numbers
+        for flow_num in flow_nums:
+            pps = self._efd_perf_evaluate(2, flow_num)
+
+            self.result_table_add([8, 2, flow_num / (1024 * 1024), pps])
+
+        self.result_table_print()
+
+    def test_perf_efd_valuesize(self):
+        """
+        Run EFD perf evaluation for different value size
+        """
+        self.verify(len(self.dut_ports) >= 2, "Not enough ports")
+        val_bitnums = [8, 16, 24, 32]
+        flow_num = 1024 * 1024 * 2
+
+        table_header = ["Value Bits", "Nodes", "Flow Entries", "Throughput(mpps)"]
+
+        self.result_table_create(table_header)
+        # perf of different value bit lengths
+        for val_bitnum in val_bitnums:
+            # change value length and rebuild dpdk
+            extra_options = "-Dc_args=-DRTE_EFD_VALUE_NUM_BITS=%d" % val_bitnum
+            self.dut.build_install_dpdk(self.target, extra_options=extra_options)
+            self.build_server_node_efd()
+
+            pps = self._efd_perf_evaluate(2, flow_num)
+            self.result_table_add([val_bitnum, 2, "2M", pps])
+
+        self.result_table_print()
+        extra_options = "-Dc_args=-DRTE_EFD_VALUE_NUM_BITS=8"
+        self.dut.build_install_dpdk(self.target, extra_options=extra_options)
+        self.build_server_node_efd()
+
+    def _efd_perf_evaluate(self, node_num, flow_num):
+        # extended flow number into etgen module
+
+        # output port is calculated from overall ports number
+        server_cmd_fmt = "%s %s -- -p 0x3 -n %d -f %s"
+        node_cmd_fmt = "%s %s --proc-type=secondary -- -n %d"
+        socket = self.dut.get_numa_id(self.dut_ports[0])
+
+        pcap = os.sep.join([self.output_path, "efd.pcap"])
+        self.tester.scapy_append(
+            'wrpcap("%s", [Ether()/IP(src="0.0.0.0", dst="0.0.0.0")/("X"*26)])' % pcap
+        )
+        self.tester.scapy_execute()
+
+        tgen_input = []
+        rx_port = self.tester.get_local_port(self.dut_ports[0])
+        tx_port = self.tester.get_local_port(self.dut_ports[1])
+
+        pcap = os.sep.join([self.output_path, "efd.pcap"])
+        tgen_input.append((tx_port, rx_port, pcap))
+        tgen_input.append((rx_port, tx_port, pcap))
+
+        cores = self.dut.get_core_list("1S/%dC/1T" % (node_num + 2), socket)
+
+        self.verify(len(cores), "Can't find enough cores")
+
+        eal_para = self.dut.create_eal_parameters(cores=cores[0:2], ports=[0, 1])
+        server_cmd = server_cmd_fmt % (
+            self.server_app,
+            eal_para,
+            node_num,
+            hex(flow_num),
+        )
+        # create table may need few minutes
+        self.dut.send_expect(server_cmd, "Finished Process Init", timeout=240)
+
+        node_sessions = []
+        for node in range(node_num):
+
+            eal_para = self.dut.create_eal_parameters(cores=[cores[2 + node]])
+            node_cmd = node_cmd_fmt % (self.node_app, eal_para, node)
+            node_session = self.dut.new_session(suite="node%d" % node)
+            node_sessions.append(node_session)
+            node_session.send_expect(node_cmd, "Finished Process Init", timeout=30)
+
+        # clear streams before add new streams
+        self.tester.pktgen.clear_streams()
+        # run packet generator
+        streams = self.pktgen_helper.prepare_stream_from_tginput(
+            tgen_input, 100, None, self.tester.pktgen
+        )
+        _, pps = self.tester.pktgen.measure_throughput(stream_ids=streams)
+
+        for node_session in node_sessions:
+            node_session.send_expect("^C", "#")
+            self.dut.close_session(node_session)
+
+        self.dut.send_expect("^C", "#")
+
+        pps /= 1000000.0
+        return pps
+
+    def set_fields(self):
+        """set ip protocol field behavior"""
+        fields_config = {
+            "ip": {
+                # self.flow_num not used by this suite
+                # 'dst': {'range': self.flow_num, 'action': 'inc'}
+                "dst": {"range": 64, "action": "inc"}
+            },
+        }
+        return fields_config
+
+    def tear_down(self):
+        """
+        Run after each test case.
+        """
+        self.dut.kill_all()
+        pass
+
+    def tear_down_all(self):
+        """
+        Run after each test suite.
+        """
+        pass
-- 
2.17.1