Re: [dts] [PATCH] added status checks test plan and test suite

["Tu, Lijuan" <lijuan.tu@intel.com>]

Applied, thanks

-----Original Message-----
From: dts <dts-bounces@dpdk.org> On Behalf Of Owen Hilyard
Sent: 2020年6月17日 5:26
To: dts@dpdk.org
Cc: dpdklab@iol.unh.edu; ohilyard@iol.unh.edu
Subject: [dts] [PATCH] added status checks test plan and test suite

add status checks test plan
add status checks test suite

Signed-off-by: Owen Hilyard <ohilyard@iol.unh.edu>
---
 test_plans/stats_checks_test_plan.rst | 119 ++++++++++++++++
 tests/TestSuite_stats_checks.py       | 196 ++++++++++++++++++++++++++
 2 files changed, 315 insertions(+)
 create mode 100644 test_plans/stats_checks_test_plan.rst
 create mode 100644 tests/TestSuite_stats_checks.py

diff --git a/test_plans/stats_checks_test_plan.rst b/test_plans/stats_checks_test_plan.rst
new file mode 100644
index 0000000..ec58804
--- /dev/null
+++ b/test_plans/stats_checks_test_plan.rst
@@ -0,0 +1,119 @@
+.. # BSD LICENSE
+    #
+    # Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
+    # Copyright © 2018[, 2019] The University of New Hampshire. All rights reserved.
+    # All rights reserved.
+    #
+    # Redistribution and use in source and binary forms, with or without
+    # modification, are permitted provided that the following conditions
+    # are met:
+    #
+    #   * Redistributions of source code must retain the above copyright
+    #     notice, this list of conditions and the following disclaimer.
+    #   * Redistributions in binary form must reproduce the above copyright
+    #     notice, this list of conditions and the following disclaimer in
+    #     the documentation and/or other materials provided with the
+    #     distribution.
+    #   * Neither the name of Intel Corporation nor the names of its
+    #     contributors may be used to endorse or promote products derived
+    #     from this software without specific prior written permission.
+    #
+    # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+    # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+    # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+    # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+    # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+    # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+    # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+    # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+    # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+    # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+    # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+=================
+Stats Check tests
+=================
+
+The support of stats checks by Poll Mode Drivers consists of the 
+ability of the driver to properly report statistics upon request. Such 
+statistics should include number of packets and bytes sent and 
+recieved, as well as the number of dropped packets and transmission errors.
+
+.. note::
+
+   Maximum Packet Length = MTU(Maximum Transmission Unit) + 14(src mac + dst mac) + 4(CRC)
+   e.g., 1518 = 1500 + 14 + 4
+
+Prerequisites
+=============
+
+If using vfio the kernel must be >= 3.6+ and VT-d must be enabled in 
+bios.When using vfio, use the following commands to load the vfio 
+driver and bind it to the device under test::
+
+   modprobe vfio
+   modprobe vfio-pci
+   usertools/dpdk-devbind.py --bind=vfio-pci device_bus_id
+
+Assuming that ports ``0`` and ``1`` of the test target are directly 
+connected to the traffic generator, launch the ``testpmd`` application 
+with the following
+arguments::
+
+  ./build/app/testpmd -c ffffff -n 6 -- -i --portmask=0x3 
+ --max-pkt-len=9600 \
+  --tx-offloads=0x00008000
+
+The -n command is used to select the number of memory channels. It should match the number of memory channels on that setup.
+
+Setting tx-offload to 0x8000 and the maximum packet length to 9600 (CRC 
+included) makes input Jumbo Frames to be stored in multiple buffers by 
+the hardware RX engine.
+
+Start packet forwarding in the ``testpmd`` application with the 
+``start`` command. Then, for each port on the target make the Traffic 
+Generator transmit a packet to the port of arbitrary size less than the 
+MTU of the target port, checking that the same amount of frames and 
+bytes are received back by the Traffic Generator from the port.
+
+Functional Tests of Status Checks
+================================
+
+Testing the support of Status Checks in Poll Mode Drivers consists of 
+configuring the gathering the initial status of a port, sending a 
+packet to that port, and checking the status of the port. The initial 
+status and the new status are then compared for expected differences.
+The fields checked are RX-packets, RX-bytes, RX-errors, TX-packets, 
+TX-errors, and TX-bytes.
+
+Test Case: Status Checks
+====================================================
+
+Check the initial state of the ports (Single example port shown)::
+
+  testpmd> show port stats all
+    ######################## NIC statistics for port 0  ########################
+    RX-packets: 0        RX-missed: 0         RX-bytes:  0
+    RX-errors: 0
+    RX-nombuf:  0
+    TX-packets: 0          TX-errors: 0          TX-bytes:  0
+
+    Throughput (since last show)
+    Rx-pps:            0
+    Tx-pps:            0
+    
+ ######################################################################
+ ######
+
+Send a packet with size 50 bytes (Single example port show) ::
+
+  testpmd> show port stats all
+    ######################## NIC statistics for port 0  ########################
+    RX-packets: 1        RX-missed: 0         RX-bytes:  50
+    RX-errors: 0
+    RX-nombuf:  0
+    TX-packets: 0          TX-errors: 0          TX-bytes:  0
+
+    Throughput (since last show)
+    Rx-pps:            0
+    Tx-pps:            0
+    
+ ######################################################################
+ ######
+
+
+Verify that the increase in RX-bytes and RX-packets is as-expected, and no other information changed.
diff --git a/tests/TestSuite_stats_checks.py b/tests/TestSuite_stats_checks.py new file mode 100644 index 0000000..e1c443b
--- /dev/null
+++ b/tests/TestSuite_stats_checks.py
@@ -0,0 +1,196 @@
+# BSD LICENSE
+#
+# Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
+# Copyright © 2018[, 2019] The University of New Hampshire. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without # 
+modification, are permitted provided that the following conditions # 
+are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # 
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # 
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # 
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT # 
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # 
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # 
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # 
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # 
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # 
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # 
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
+DPDK Test suite.
+Stats Checks example.
+"""
+from time import sleep
+from typing import List, Iterator, Tuple
+
+import utils
+from pmd_output import PmdOutput
+
+from port import Port
+
+from test_case import TestCase
+
+ETHER_HEADER_LEN = 18
+IP_HEADER_LEN = 20
+ETHER_STANDARD_MTU = 1518
+
+
+class TestStatsChecks(TestCase):
+    #
+    #
+    # Helper methods and setup methods.
+    #
+    # Some of these methods may not be used because they were inlined from a child
+    # of TestCase. This was done because the current test system doesn't support
+    # inheritance.
+    #
+    def tear_down(self):
+        """
+        Run after each test case.
+        """
+        self.dut.kill_all()
+
+    def tear_down_all(self):
+        """
+        When the case of this test suite finished, the environment should
+        clear up.
+        """
+        self.tester.send_expect(f"ifconfig {self.tester.get_interface(self.tester.get_local_port(self.rx_port))} " +
+                                f"mtu {ETHER_STANDARD_MTU}", "# ")
+        super().tear_down_all()
+
+    def exec(self, command: str) -> str:
+        """
+        An abstraction to remove repeated code throughout the subclasses of this class
+        """
+        return self.dut.send_expect(command, "testpmd>")
+
+    def get_mac_address_for_port(self, port_id: int) -> str:
+        return self.dut.get_mac_address(port_id)
+
+    def send_scapy_packet(self, port_id: int, packet: str):
+        itf = self.tester.get_interface(port_id)
+
+        self.tester.scapy_foreground()
+        mac = self.dut.get_mac_address(port_id)
+        self.tester.scapy_append(f'dutmac="{mac}"')
+        self.tester.scapy_append(f'sendp({packet}, iface="{itf}")')
+        return self.tester.scapy_execute()
+
+    def send_packet_of_size_to_port(self, port_id: int, pktsize: int):
+
+        # The packet total size include ethernet header, ip header, and payload.
+        # ethernet header length is 18 bytes, ip standard header length is 20 bytes.
+        # pktlen = pktsize - ETHER_HEADER_LEN
+        padding = pktsize - IP_HEADER_LEN
+        out = self.send_scapy_packet(port_id,
+                                     f'Ether(dst=dutmac, src="52:00:00:00:00:00")/IP()/Raw(load="\x50"*{padding})')
+        return out
+
+    def send_packet_of_size_to_tx_port(self, pktsize, received=True):
+        """
+        Send 1 packet to portid
+        """
+        tx_pkts_ori, tx_err_ori, tx_bytes_ori = [int(_) for _ in self.get_port_status_rx(self.tx_port)]
+        rx_pkts_ori, rx_err_ori, rx_bytes_ori = [int(_) for _ in 
+ self.get_port_status_tx(self.rx_port)]
+
+        out = self.send_packet_of_size_to_port(self.tx_port, pktsize)
+
+        sleep(5)
+
+        tx_pkts, tx_err, tx_bytes = [int(_) for _ in self.get_port_status_rx(self.tx_port)]
+        rx_pkts, rx_err, rx_bytes = [int(_) for _ in 
+ self.get_port_status_tx(self.rx_port)]
+
+        tx_pkts_difference = tx_pkts - tx_pkts_ori
+        tx_err_difference = tx_err - tx_err_ori
+        tx_bytes_difference = tx_bytes - tx_bytes_ori
+        rx_pkts_difference = rx_pkts - rx_pkts_ori
+        rx_err_difference = rx_err - rx_err_ori
+        rx_bytes_difference = rx_bytes - rx_bytes_ori
+
+        if received:
+            self.verify(tx_pkts_difference >= 1, "No packet was sent")
+            self.verify(tx_bytes_difference == pktsize + ETHER_HEADER_LEN)
+            self.verify(tx_pkts_difference == rx_pkts_difference, "different numbers of packets sent and received")
+            self.verify(tx_bytes_difference == rx_bytes_difference, "different number of bytes sent and received")
+            self.verify(tx_err_difference == 0, "unexpected tx error")
+            self.verify(rx_err_difference == 0, "unexpected rx error")
+        else:
+            self.verify(rx_err_difference == 1 or tx_pkts_difference == 0 or tx_err_difference == 1,
+                        "packet that either should have either caused an error " +
+                        "or been rejected for transmission was not")
+        return out
+
+    def get_port_status_rx(self, portid) -> Tuple[str, str, str]:
+        stats = self.pmdout.get_pmd_stats(portid)
+        return stats['RX-packets'], stats['RX-errors'], 
+ stats['RX-bytes']
+
+    def get_port_status_tx(self, portid) -> Tuple[str, str, str]:
+        stats = self.pmdout.get_pmd_stats(portid)
+        return stats['TX-packets'], stats['TX-errors'], 
+ stats['TX-bytes']
+
+    def set_up_all(self):
+        """
+        Prerequisite steps for each test suit.
+        """
+        self.dut_ports = self.dut.get_ports()
+        self.verify(len(self.dut_ports) >= 2, "Insufficient ports")
+        self.rx_port = self.dut_ports[0]
+        self.tx_port = self.dut_ports[1]
+
+        cores = self.dut.get_core_list("1S/2C/1T")
+        self.coremask = utils.create_mask(cores)
+
+        self.port_mask = utils.create_mask([self.rx_port, 
+ self.tx_port])
+
+        self.pmdout = PmdOutput(self.dut)
+
+    def set_up(self):
+        """
+        This is to clear up environment before the case run.
+        """
+        self.dut.kill_all()
+
+    def tear_down(self):
+        """
+        Run after each test case.
+        """
+        self.dut.kill_all()
+
+    def tear_down_all(self):
+        """
+        When the case of this test suite finished, the environment should
+        clear up.
+        """
+        self.dut.kill_all()
+
+    #
+    #
+    #
+    # Test cases.
+    #
+
+    def test_stats_checks(self):
+        self.pmdout.start_testpmd("Default")
+        self.exec("port start all")
+        self.exec("set fwd mac")
+        self.exec("start")
+
+        self.send_packet_of_size_to_tx_port(50, received=True)
+
+        self.exec("stop")
+        self.pmdout.quit()
--
2.25.1