[dts][PATCH V1 2/7] tests/l2fwd: Separated performance cases

[Hongbo Li <hongbox.li@intel.com>]

Separated performance test cases

Signed-off-by: Hongbo Li <hongbox.li@intel.com>
---
 test_plans/l2fwd_test_plan.rst      |  36 -----
 test_plans/perf_l2fwd_test_plan.rst |  90 +++++++++++
 tests/TestSuite_l2fwd.py            | 106 -------------
 tests/TestSuite_perf_l2fwd.py       | 233 ++++++++++++++++++++++++++++
 4 files changed, 323 insertions(+), 142 deletions(-)
 create mode 100644 test_plans/perf_l2fwd_test_plan.rst
 create mode 100644 tests/TestSuite_perf_l2fwd.py

diff --git a/test_plans/l2fwd_test_plan.rst b/test_plans/l2fwd_test_plan.rst
index d8803cb5..1bf9ad45 100644
--- a/test_plans/l2fwd_test_plan.rst
+++ b/test_plans/l2fwd_test_plan.rst
@@ -69,39 +69,3 @@ Trigger the packet generator of bursting packets from ``port A``, then check if
 ``port 0`` could receive them and ``port 1`` could forward them back. Stop it
 and then trigger the packet generator of bursting packets from ``port B``, then
 check if ``port 1`` could receive them and ``port 0`` could forward them back.
-
-Test Case: ``64/128/256/512/1024/1500`` bytes packet forwarding test
-====================================================================
-
-Set the packet stream to be sent out from packet generator before testing as below.
-
-+-------+---------+---------+---------+-----------+
-| Frame |    1q   |    2q   |   4q    |    8 q    |
-| Size  |         |         |         |           |
-+-------+---------+---------+---------+-----------+
-|  64   |         |         |         |           |
-+-------+---------+---------+---------+-----------+
-|  65   |         |         |         |           |
-+-------+---------+---------+---------+-----------+
-|  128  |         |         |         |           |
-+-------+---------+---------+---------+-----------+
-|  256  |         |         |         |           |
-+-------+---------+---------+---------+-----------+
-|  512  |         |         |         |           |
-+-------+---------+---------+---------+-----------+
-|  1024 |         |         |         |           |
-+-------+---------+---------+---------+-----------+
-|  1280 |         |         |         |           |
-+-------+---------+---------+---------+-----------+
-|  1518 |         |         |         |           |
-+-------+---------+---------+---------+-----------+
-
-Then run the test application as below::
-
-    $ ./x86_64-native-linuxapp-gcc/examples/dpdk-l2fwd -n 2 -c f -- -q 1 -p 0x3
-
-The -n command is used to select the number of memory channels. It should match the number of memory channels on that setup.
-
-Trigger the packet generator of bursting packets to the port 0 and 1 on the onboard
-NIC to be tested. Then measure the forwarding throughput for different packet sizes
-and different number of queues.
diff --git a/test_plans/perf_l2fwd_test_plan.rst b/test_plans/perf_l2fwd_test_plan.rst
new file mode 100644
index 00000000..e5fbaf30
--- /dev/null
+++ b/test_plans/perf_l2fwd_test_plan.rst
@@ -0,0 +1,90 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+   Copyright(c) 2010-2017 Intel Corporation
+
+===================
+L2 Forwarding Tests
+===================
+
+This test application is a basic packet processing application using Intel®
+DPDK. It is a layer-2 (L2) forwarding application which takes traffic from
+a single RX port and transmits it with few modification on a single TX port.
+
+For a packet received on a RX port (RX_PORT), it would be transmitted from a
+TX port (TX_PORT=RX_PORT+1) if RX_PORT is even; otherwise from a TX port
+(TX_PORT=RX_PORT-1) if RX_PORT is odd. Before being transmitted, the source
+mac address of the packet would be replaced by the mac address of the TX port,
+while the destination mac address would be replaced by 00:09:c0:00:00:TX_PORT_ID.
+
+The test application should be run with the wanted paired ports configured using
+the coremask parameter via the command line. i.e. port 0 and 1 is a valid pair,
+while port 1 and 2 isn't. The test is performed by running the test application
+and using a traffic generator. Tests are run with receiving a variety of size of
+packets generated by the traffic generator and forwarding back to the traffic
+generator. The packet loss and the throughput are the right ones need to be
+measured.
+
+The ``l2fwd`` application is run with EAL parameters and parameters for
+the application itself. For details about the EAL parameters, see the relevant
+DPDK **Getting Started Guide**. This application supports two parameters for
+itself.
+
+- ``-p PORTMASK``: hexadecimal bitmask of ports to configure
+- ``-q NQ``: number of queue per lcore (default is 1)
+
+Prerequisites
+=============
+
+If using vfio the kernel must be >= 3.6+ and VT-d must be enabled in bios.When
+using vfio, use the following commands to load the vfio driver and bind it
+to the device under test::
+
+   modprobe vfio
+   modprobe vfio-pci
+   usertools/dpdk-devbind.py --bind=vfio-pci device_bus_id
+
+Assume port 0 and 1 are connected to the traffic generator, to run the test
+application in linuxapp environment with 4 lcores, 2 ports and 8 RX queues
+per lcore::
+
+    $ ./x86_64-native-linuxapp-gcc/examples/dpdk-l2fwd -n 1 -c f -- -q 8 -p 0x3
+
+Also, if the ports to be tested are different, the port mask should be changed.
+The lcore used to run the test application and the number of queue used for a
+lcore could be changed. For benchmarking, the EAL parameters and the parameters
+for the application itself for different test cases should be the same.
+
+Test Case: ``64/128/256/512/1024/1500`` bytes packet forwarding test
+====================================================================
+
+Set the packet stream to be sent out from packet generator before testing as below.
+
++-------+---------+---------+---------+-----------+
+| Frame |    1q   |    2q   |   4q    |    8 q    |
+| Size  |         |         |         |           |
++-------+---------+---------+---------+-----------+
+|  64   |         |         |         |           |
++-------+---------+---------+---------+-----------+
+|  65   |         |         |         |           |
++-------+---------+---------+---------+-----------+
+|  128  |         |         |         |           |
++-------+---------+---------+---------+-----------+
+|  256  |         |         |         |           |
++-------+---------+---------+---------+-----------+
+|  512  |         |         |         |           |
++-------+---------+---------+---------+-----------+
+|  1024 |         |         |         |           |
++-------+---------+---------+---------+-----------+
+|  1280 |         |         |         |           |
++-------+---------+---------+---------+-----------+
+|  1518 |         |         |         |           |
++-------+---------+---------+---------+-----------+
+
+Then run the test application as below::
+
+    $ ./x86_64-native-linuxapp-gcc/examples/dpdk-l2fwd -n 2 -c f -- -q 1 -p 0x3
+
+The -n command is used to select the number of memory channels. It should match the number of memory channels on that setup.
+
+Trigger the packet generator of bursting packets to the port 0 and 1 on the onboard
+NIC to be tested. Then measure the forwarding throughput for different packet sizes
+and different number of queues.
diff --git a/tests/TestSuite_l2fwd.py b/tests/TestSuite_l2fwd.py
index 9e8fd2d5..7594958b 100644
--- a/tests/TestSuite_l2fwd.py
+++ b/tests/TestSuite_l2fwd.py
@@ -150,112 +150,6 @@ class TestL2fwd(TestCase):
 
             self.quit_l2fwd()
 
-    def test_perf_l2fwd_performance(self):
-        """
-        Benchmark performance for frame_sizes.
-        """
-        ports = []
-        for port in range(self.number_of_ports):
-            ports.append(self.dut_ports[port])
-
-        port_mask = utils.create_mask(ports)
-        cores = self.dut.get_core_list(self.core_config, socket=self.ports_socket)
-
-        eal_params = self.dut.create_eal_parameters(cores=cores)
-        eal_param = ""
-        for i in ports:
-            eal_param += " -a %s" % self.dut.ports_info[i]["pci"]
-
-        for frame_size in self.frame_sizes:
-
-            payload_size = frame_size - self.headers_size
-
-            tgen_input = []
-            cnt = 1
-            for port in range(self.number_of_ports):
-                rx_port = self.tester.get_local_port(
-                    self.dut_ports[port % self.number_of_ports]
-                )
-                tx_port = self.tester.get_local_port(
-                    self.dut_ports[(port + 1) % self.number_of_ports]
-                )
-                destination_mac = self.dut.get_mac_address(
-                    self.dut_ports[(port + 1) % self.number_of_ports]
-                )
-                pcap = os.sep.join(
-                    [self.output_path, "l2fwd_{0}_{1}.pcap".format(port, cnt)]
-                )
-                self.tester.scapy_append(
-                    'wrpcap("%s", [Ether(dst="%s")/IP()/UDP()/("X"*%d)])'
-                    % (pcap, destination_mac, payload_size)
-                )
-                tgen_input.append((tx_port, rx_port, pcap))
-                time.sleep(3)
-                self.tester.scapy_execute()
-                cnt += 1
-
-            for queues in self.test_queues:
-
-                command_line = "./%s %s %s -- -q %s -p %s &" % (
-                    self.app_l2fwd_path,
-                    eal_params,
-                    eal_param,
-                    str(queues["queues"]),
-                    port_mask,
-                )
-
-                #                self.dut.send_expect(command_line, "memory mapped", 60)
-                self.dut.send_expect(command_line, "L2FWD: entering main loop", 60)
-                # wait 5 second after l2fwd boot up.
-                # It is aimed to make sure trex detect link up status.
-                if self.tester.is_pktgen:
-                    time.sleep(5)
-                info = (
-                    "Executing l2fwd using %s queues, frame size %d and %s setup.\n"
-                    % (queues["queues"], frame_size, self.core_config)
-                )
-
-                self.logger.info(info)
-                self.rst_report(info, annex=True)
-                self.rst_report(command_line + "\n\n", frame=True, annex=True)
-
-                # clear streams before add new streams
-                self.tester.pktgen.clear_streams()
-                # run packet generator
-                streams = self.pktgen_helper.prepare_stream_from_tginput(
-                    tgen_input, 100, None, self.tester.pktgen
-                )
-                _, pps = self.tester.pktgen.measure_throughput(stream_ids=streams)
-
-                Mpps = pps / 1000000.0
-                queues["Mpps"][frame_size] = Mpps
-                queues["pct"][frame_size] = (
-                    Mpps
-                    * 100
-                    / float(self.wirespeed(self.nic, frame_size, self.number_of_ports))
-                )
-
-                self.quit_l2fwd()
-
-        # Look for transmission error in the results
-        for frame_size in self.frame_sizes:
-            for n in range(len(self.test_queues)):
-                self.verify(
-                    self.test_queues[n]["Mpps"][frame_size] > 0, "No traffic detected"
-                )
-
-        # Prepare the results for table
-        for frame_size in self.frame_sizes:
-            results_row = []
-            results_row.append(frame_size)
-            for queue in self.test_queues:
-                results_row.append(queue["Mpps"][frame_size])
-                results_row.append(queue["pct"][frame_size])
-
-            self.result_table_add(results_row)
-
-        self.result_table_print()
-
     def tear_down(self):
         """
         Run after each test case.
diff --git a/tests/TestSuite_perf_l2fwd.py b/tests/TestSuite_perf_l2fwd.py
new file mode 100644
index 00000000..2cbf8899
--- /dev/null
+++ b/tests/TestSuite_perf_l2fwd.py
@@ -0,0 +1,233 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2019 Intel Corporation
+#
+
+"""
+DPDK Test suite.
+Test Layer-2 Forwarding support
+"""
+import os
+import time
+
+import framework.utils as utils
+from framework.pktgen import PacketGeneratorHelper
+from framework.settings import HEADER_SIZE
+from framework.test_case import TestCase
+
+
+class TestL2fwd(TestCase):
+    def set_up_all(self):
+        """
+        Run at the start of each test suite.
+
+        L2fwd prerequisites.
+        """
+        self.frame_sizes = [64, 65, 128, 256, 512, 1024, 1280, 1518]
+
+        self.test_queues = [
+            {"queues": 1, "Mpps": {}, "pct": {}},
+            {"queues": 2, "Mpps": {}, "pct": {}},
+            {"queues": 4, "Mpps": {}, "pct": {}},
+            {"queues": 8, "Mpps": {}, "pct": {}},
+        ]
+
+        self.core_config = "1S/4C/1T"
+        self.number_of_ports = 2
+        self.headers_size = HEADER_SIZE["eth"] + HEADER_SIZE["ip"] + HEADER_SIZE["udp"]
+
+        self.dut_ports = self.dut.get_ports_performance(force_different_nic=False)
+
+        self.verify(
+            len(self.dut_ports) >= self.number_of_ports,
+            "Not enough ports for " + self.nic,
+        )
+
+        self.ports_socket = self.dut.get_numa_id(self.dut_ports[0])
+
+        # compile
+        out = self.dut.build_dpdk_apps("./examples/l2fwd")
+        self.app_l2fwd_path = self.dut.apps_name["l2fwd"]
+        self.verify("Error" not in out, "Compilation error")
+        self.verify("No such" not in out, "Compilation error")
+
+        self.table_header = ["Frame"]
+        for queue in self.test_queues:
+            self.table_header.append("%d queues Mpps" % queue["queues"])
+            self.table_header.append("% linerate")
+
+        self.result_table_create(self.table_header)
+
+        # get dts output path
+        if self.logger.log_path.startswith(os.sep):
+            self.output_path = self.logger.log_path
+        else:
+            cur_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+            self.output_path = os.sep.join([cur_path, self.logger.log_path])
+        # create an instance to set stream field setting
+        self.pktgen_helper = PacketGeneratorHelper()
+
+    def set_up(self):
+        """
+        Run before each test case.
+        """
+        pass
+
+    def quit_l2fwd(self):
+        self.dut.send_expect("fg", "l2fwd ", 5)
+        self.dut.send_expect("^C", "# ", 5)
+
+    def notest_port_testing(self):
+        """
+        Check port forwarding.
+        """
+        # the cases use the first two ports
+        port_mask = utils.create_mask([self.dut_ports[0], self.dut_ports[1]])
+        eal_params = self.dut.create_eal_parameters()
+
+        self.dut.send_expect(
+            "./%s %s -- -q 8 -p %s  &" % (self.app_l2fwd_path, eal_params, port_mask),
+            "L2FWD: entering main loop",
+            60,
+        )
+
+        for i in [0, 1]:
+            tx_port = self.tester.get_local_port(self.dut_ports[i])
+            rx_port = self.tester.get_local_port(self.dut_ports[1 - i])
+
+            tx_interface = self.tester.get_interface(tx_port)
+            rx_interface = self.tester.get_interface(rx_port)
+
+            self.tester.scapy_background()
+            self.tester.scapy_append('p = sniff(iface="%s", count=1)' % rx_interface)
+            self.tester.scapy_append("number_packets=len(p)")
+            self.tester.scapy_append("RESULT = str(number_packets)")
+
+            self.tester.scapy_foreground()
+            self.tester.scapy_append(
+                'sendp([Ether()/IP()/UDP()/("X"*46)], iface="%s")' % tx_interface
+            )
+
+            self.tester.scapy_execute()
+            number_packets = self.tester.scapy_get_result()
+            self.verify(number_packets == "1", "Failed to switch L2 frame")
+
+        self.quit_l2fwd()
+
+    def test_perf_l2fwd_performance(self):
+        """
+        Benchmark performance for frame_sizes.
+        """
+        ports = []
+        for port in range(self.number_of_ports):
+            ports.append(self.dut_ports[port])
+
+        port_mask = utils.create_mask(ports)
+        cores = self.dut.get_core_list(self.core_config, socket=self.ports_socket)
+
+        eal_params = self.dut.create_eal_parameters(cores=cores)
+        eal_param = ""
+        for i in ports:
+            eal_param += " -a %s" % self.dut.ports_info[i]["pci"]
+
+        for frame_size in self.frame_sizes:
+
+            payload_size = frame_size - self.headers_size
+
+            tgen_input = []
+            cnt = 1
+            for port in range(self.number_of_ports):
+                rx_port = self.tester.get_local_port(
+                    self.dut_ports[port % self.number_of_ports]
+                )
+                tx_port = self.tester.get_local_port(
+                    self.dut_ports[(port + 1) % self.number_of_ports]
+                )
+                destination_mac = self.dut.get_mac_address(
+                    self.dut_ports[(port + 1) % self.number_of_ports]
+                )
+                pcap = os.sep.join(
+                    [self.output_path, "l2fwd_{0}_{1}.pcap".format(port, cnt)]
+                )
+                self.tester.scapy_append(
+                    'wrpcap("%s", [Ether(dst="%s")/IP()/UDP()/("X"*%d)])'
+                    % (pcap, destination_mac, payload_size)
+                )
+                tgen_input.append((tx_port, rx_port, pcap))
+                time.sleep(3)
+                self.tester.scapy_execute()
+                cnt += 1
+
+            for queues in self.test_queues:
+
+                command_line = "./%s %s %s -- -q %s -p %s &" % (
+                    self.app_l2fwd_path,
+                    eal_params,
+                    eal_param,
+                    str(queues["queues"]),
+                    port_mask,
+                )
+
+                #                self.dut.send_expect(command_line, "memory mapped", 60)
+                self.dut.send_expect(command_line, "L2FWD: entering main loop", 60)
+                # wait 5 second after l2fwd boot up.
+                # It is aimed to make sure trex detect link up status.
+                if self.tester.is_pktgen:
+                    time.sleep(5)
+                info = (
+                    "Executing l2fwd using %s queues, frame size %d and %s setup.\n"
+                    % (queues["queues"], frame_size, self.core_config)
+                )
+
+                self.logger.info(info)
+                self.rst_report(info, annex=True)
+                self.rst_report(command_line + "\n\n", frame=True, annex=True)
+
+                # clear streams before add new streams
+                self.tester.pktgen.clear_streams()
+                # run packet generator
+                streams = self.pktgen_helper.prepare_stream_from_tginput(
+                    tgen_input, 100, None, self.tester.pktgen
+                )
+                _, pps = self.tester.pktgen.measure_throughput(stream_ids=streams)
+
+                Mpps = pps / 1000000.0
+                queues["Mpps"][frame_size] = Mpps
+                queues["pct"][frame_size] = (
+                    Mpps
+                    * 100
+                    / float(self.wirespeed(self.nic, frame_size, self.number_of_ports))
+                )
+
+                self.quit_l2fwd()
+
+        # Look for transmission error in the results
+        for frame_size in self.frame_sizes:
+            for n in range(len(self.test_queues)):
+                self.verify(
+                    self.test_queues[n]["Mpps"][frame_size] > 0, "No traffic detected"
+                )
+
+        # Prepare the results for table
+        for frame_size in self.frame_sizes:
+            results_row = []
+            results_row.append(frame_size)
+            for queue in self.test_queues:
+                results_row.append(queue["Mpps"][frame_size])
+                results_row.append(queue["pct"][frame_size])
+
+            self.result_table_add(results_row)
+
+        self.result_table_print()
+
+    def tear_down(self):
+        """
+        Run after each test case.
+        """
+        self.dut.send_expect("fg", "l2fwd|# ", 5)
+        self.dut.send_expect("^C", "# ", 5)
+
+    def tear_down_all(self):
+        """
+        Run after each test suite.
+        """
+        pass
-- 
2.17.1