[dts] [PATCH V1]userspace_ethtool: update test script and test plan

[Yufen Mo <yufengx.mo@intel.com>]

From: yufengmx <yufengx.mo@intel.com>

change vlan packet check number when doing vlan testing. Add process to resize linux ethtool eeprom dump
data size as the size dump by dpdk ethtool. Ignore auto test link connect status on FVL. update related
test cases' test plan, add testing configuration and requirements.

Signed-off-by: yufengmx <yufengx.mo@intel.com>
---
 test_plans/userspace_ethtool_test_plan.rst | 167 ++++++++++++++++++-----------
 tests/TestSuite_userspace_ethtool.py       |  85 +++++++++------
 2 files changed, 156 insertions(+), 96 deletions(-)

diff --git a/test_plans/userspace_ethtool_test_plan.rst b/test_plans/userspace_ethtool_test_plan.rst
index 5ee79a1..83e7fac 100644
--- a/test_plans/userspace_ethtool_test_plan.rst
+++ b/test_plans/userspace_ethtool_test_plan.rst
@@ -34,31 +34,42 @@
  Userspace Ethtool
 ==================
 
-This feature is designed to provide one rte_ethtool shim layer based on 
-rte_ethdev API. The implementation also along with a command prompt driven
-demonstration application. It only contained of 18 popular used Ethtool and
-Netdevice ops as described in rte_ethtool.h.
+This feature is designed to provide one rte_ethtool shim layer based on
+rte_ethdev API. The Ethtool sample application shows an implementation 
+of an ethtool-like API and provides a console environment that allows 
+its use to query and change Ethernet card parameters. The Ethtool sample 
+is based upon a simple L2 frame reflector.
 
 Prerequisites
 =============
-
+notice: On FVL, test case "test_dump_driver_info" need a physical link disconnect,
+this case must do manually at this condition.
+ 
 Assume port 0 and 1 are connected to the traffic generator, to run the test
-application in linuxapp environment with 4 lcores, 2 ports.
+application in linux app environment with 4 lcores, 2 ports.
 
 	ethtool -c f -n 4
 
-The sample should be validated on Forville, Niantic and i350 Nics. 
+The sample should be validated on Forville, Niantic and i350 Nics.
+
+other requirements:
+*. igxbe driver (version >= 4.3.13).
+*. ethtool of linux is the default reference tool.
+*. md5sum is a tool to do dumped bin format file comparison.
+*. insert two nic cards on No.0 socket
 
-Test Case: Dump driver infor test
+Test Case: Dump driver information test
 =================================
-User "drvinfo" command to dump driver information and then check that dumped
-information was exactly the same as fact.
+User "drvinfo" command to dump driver information and then check that
+dumped information, which are dumped separately by dpdk's ethtool and 
+linux's ethtool, were exactly the same.
 
 	EthApp> drvinfo
 	Port 0 driver: rte_ixgbe_pmd (ver: RTE 2.1.0)
 	Port 1 driver: rte_ixgbe_pmd (ver: RTE 2.1.0)
 
 Use "link" command to dump all ports link status.
+Notice:: On FVL, link detect need a physical link disconnect.
 	EthApp> link
 	Port 0: Up
 	Port 1: Up
@@ -68,7 +79,7 @@ Change tester port link status to down and re-check link status.
 	Port 0: Down
 	Port 1: Down
 
-Send few packets to l2fwd and check that command "portstats" dumped correct
+Send a few packets to l2fwd and check that command "portstats" dumps correct
 port statistics.
     EthApp> portstats 0
     Port 0 stats
@@ -78,34 +89,38 @@ port statistics.
 Test Case: Retrieve eeprom test
 ===============================
 Unbind ports from igb_uio and bind them to default driver.
-Dump eeprom binary by ethtool.
+Dump eeprom binary by linux's ethtool and dpdk's ethtool separately.
 
 ethtool --eeprom-dump INTF_0 raw on > ethtool_eeprom_0.bin
 ethtool --eeprom-dump INTF_1 raw on > ethtool_eeprom_1.bin
 
-Retrieve eeprom on specified port and compare csum with the file dumped by ethtool.
+Retrieve eeprom on specified port using dpdk's ethtool and 
+compare csum with the file dumped by ethtool.
 
 	EthApp> eeprom 0 eeprom_0.bin
 	EthApp> eeprom 1 eeprom_1.bin
 
-md5sum ethtool_eeprom_0.bin
-md5sum eeprom_0.bin > eeprom_0.bin
-	
-diff ethtool_eeprom_0.hex eeprom_0.hex
+	md5sum ethtool_eeprom_0.bin
+	md5sum eeprom_0.bin
+
+compare md5sum value of the two bin files.
 
 Test Case: Retrieve register test
 ===============================
-Retrieve register on specified port, do not known how to check the binary?
+Retrieve register on specified port.
 
 	EthApp> regs 0 reg_0.bin
 	EthApp> regs 1 reg_1.bin	
 
 Unbind ports from igb_uio and bind them to default driver.
+
+    dpdk/tools/dpdk_nic_bind.py --bind=ixgbe x:xx.x
+
 Check that dumped register information is correct.
 
-ethtool -d INTF_0 raw on file reg_0.bin
-ethtool -d INTF_1 raw on file reg_0.bin
-	
+ethtool -d INTF_0 raw off file reg_0.bin
+ethtool -d INTF_1 raw off file reg_0.bin
+
 Test Case: Ring param test
 ==========================
 Dump port 0 ring size by ringparam command and check numbers are correct.
@@ -126,52 +141,30 @@ Port 0 ring paramaeters
   Rx Pending: 256 (256 max)
   Tx Pending: 2048 (4096 max)
 	
-Test Case: Pause test
-=====================
-Enable port 0 Rx pause frame and then create two packets flows in IXIA port.
-One flow is 100000 normally packet and the second flow is pause frame.
-Check that port 0 Rx speed dropped. For example, niantic will drop from
-14.8Mpps to 7.49Mpps.
-
-	EthApp> pause 0 rx
-
-Use "parse" command to print port pause status, check that port 0 rx has been
-paused.
-	EthApp> pause 0
-	Port 0: Rx Paused
-
-Unpause port 0 rx and then restart port0, check that packets Rx speed is normal.
-	EthApp> pause 0 none
-    EthApp> 
+send packet by scapy on Tester
 
-Pause port 0 TX pause frame.
-	EthApp> pause 0 tx
-
-Use "parse" command to print port pause status, check that port 1 tx has been
-paused.
-    EthApp> pause 0
-    Port 0: Tx Paused
-
-Enable flow control in IXIA port and send packets from IXIA with line rate.
-Check that IXIA receive flow control packets and IXIA transmit speed dropped.
-IXIA Rx packets more then Tx packets to check that received pause frame.
-
-Unpause port 0 tx and restart port 0. Then send packets to port0, check that
-packets forwarded normally from port 0.
-	EthApp> pause 0 none
-    EthApp> stop 0
-    EthApp> open 0
+check tx/rx packets
+EthApp>  portstats 0
 
 Test Case: Vlan test
 ====================
+enable vlan filter flag in main.c of dpdk's ethtool 
+	
+	sed -i -e '/cfg_port.txmode.mq_mode = ETH_MQ_TX_NONE;$/a\\cfg_port.rxmode.hw_vlan_filter=1;' examples/ethtool/ethtool-app/main.c
+
+re-compile examples/ethtool
+	
+	make -C examples/ethtool
+
 Add vlan 0 to port 0 and vlan 1 to port1, send packet without vlan to port0,1
 Verify port0 and port1 recevied vlan packets
+
 	EthApp> vlan 0 add 0
 	VLAN vid 0 added
 
 	EthApp> vlan 1 add 1
 	VLAN vid 1 added
-	
+
 Send packet with vlan0,1 to port0&1. Verify port0 and port1 received vlan
 packets
 
@@ -189,7 +182,11 @@ port0 and port1 can not receive vlan packet.
 Test Case: Mac address test
 ===========================
 Use "macaddr" command to dump port mac address and then check that dumped
-information is exactly the same as fact.
+information is exactly the same as ifconfig do.
+
+set a new mac address by dpdk's ethtool, send and sniff packet and check packet
+forwared status 
+
 	EthApp> macaddr 0
 	Port 0 MAC Address: XX:XX:XX:XX:XX:XX
 	EthApp> macaddr 1
@@ -211,8 +208,8 @@ Use "macaddr" command to change port mac address and then check mac changed.
 	MAC address changed
 	EthApp> macaddr 0
 	Port 0 MAC Address: 00:10:00:00:00:00
-	
-Verified  mac adress in forwarded packets has been changed.
+
+Verified mac adress in forwarded packets has been changed.
 
 Test Case: Port config test
 ===========================
@@ -224,18 +221,58 @@ Use "open" command to re-enable port0. Send packets to port0 and verify
 packets received and forwarded.
 	EthApp> open 0
 
-
 Test case: Mtu config test
 ==========================
-Use "mtu" command to change port 0 mtu from default 1518 to 1000.
+Use "mtu" command to change port 0 mtu from default 1519 to 9000 on Tester's port.
 
-Send packet size over 1000 and check that packet will be detected as error.
+Send packet size over 1519 and check that packet will be detected as error.
 
-    EthApp> mtu 0 1000
+    EthApp> mtu 0 1519
     Port 0 stats
        In: 0 (0 bytes)
       Out: 0 (0 bytes)
       Err: 1
 
-Change mtu to default value and send packet size over 1000 and check that
-packet will normally received.
+Change mtu to default value and send packet size over 1519 and check that
+packet will normally be received.
+
+Test Case: Pause tx/rx test(performance test)
+=====================
+Enable port 0 Rx pause frame and then create two packets flows on IXIA port.
+One flow is 100000 normally packet and the second flow is pause frame.
+Check that dut's port 0 Rx speed dropped status. For example, niantic will drop 
+from 14.8Mpps to 7.49Mpps.
+
+	EthApp> pause 0 rx
+
+Use "pause" command to print dut's port pause status, check that dut's port 0 rx 
+has been paused.
+
+	EthApp> pause 0
+	Port 0: Rx Paused
+
+Release pause status of port 0 rx and then restart port 0, check that packets Rx 
+speed is normal.
+	EthApp> pause 0 none
+    EthApp> 
+
+Pause port 0 TX pause frame.
+	EthApp> pause 0 tx
+
+Use "pause" command to print port pause status, check that port 0 tx has been
+paused.
+    EthApp> pause 0
+    Port 0: Tx Paused
+
+Enable flow control in IXIA port and send packets from IXIA with line rate.
+Record line rate before send packet.
+Check that IXIA receive flow control packets and IXIA transmit speed dropped.
+IXIA Rx packets more then Tx packets to check that received pause frame.Compare 
+the line rates in the time before and after the Pause packets are injected
+
+Unpause port 0 tx and restart port 0. Then send packets to port0, check that
+packets forwarded normally from port 0.
+
+	EthApp> pause 0 none
+    EthApp> stop 0
+    EthApp> open 0
diff --git a/tests/TestSuite_userspace_ethtool.py b/tests/TestSuite_userspace_ethtool.py
index c479093..c23f841 100644
--- a/tests/TestSuite_userspace_ethtool.py
+++ b/tests/TestSuite_userspace_ethtool.py
@@ -38,12 +38,12 @@ import dts
 import time
 import re
 from test_case import TestCase
-from pmd_output import PmdOutput
 from packet import Packet, sniff_packets, load_sniff_packets
 import random
 from etgen import IxiaPacketGenerator
 from settings import HEADER_SIZE
 from settings import SCAPY2IXIA
+from utils import RED
 
 
 class TestUserspaceEthtool(TestCase, IxiaPacketGenerator):
@@ -127,6 +127,22 @@ class TestUserspaceEthtool(TestCase, IxiaPacketGenerator):
         else:
             return 1518
 
+    def resize_linux_eeprom_file(self, dpdk_eeprom_file, linux_eeprom_file):
+        basePath = self.dut.base_dir
+        with open( basePath + dpdk_eeprom_file, 'rb') as fpDpdk:
+            dpdk_bytes = fpDpdk.read()
+            dpdk_length = len(dpdk_bytes)
+
+        with open( basePath + linux_eeprom_file, 'rb') as fplinux:
+            linux_bytes = fplinux.read()
+            linux_length = len(linux_bytes)
+        
+        self.verify(dpdk_length <= linux_length, 
+                    "linux ethtool haven't dump out enough data as dpdk ethtool")
+
+        with open( basePath + linux_eeprom_file, 'wb') as fplinux:
+            fplinux.write(linux_bytes[:dpdk_length])
+
     def strip_md5(self, filename):
         md5_info = self.dut.send_expect("md5sum %s" % filename, "# ")
         md5_pattern = r"(\w+)  (\w+)"
@@ -155,30 +171,33 @@ class TestUserspaceEthtool(TestCase, IxiaPacketGenerator):
                 version = m.group(3)
                 print dts.GREEN("Detect port %s with %s driver\n" % (port, driver))
 
-        # check link status dump function
-        for port in self.ports:
-            tester_port = self.tester.get_local_port(port)
-            intf = self.tester.get_interface(tester_port)
-            self.tester.send_expect("ip link set dev %s down" % intf, "# ")
-        # wait for link stable
-        time.sleep(5)
-
-        out = self.dut.send_expect("link", "EthApp>", 60)
-        link_pattern = r"Port (\d+): (.*)"
-        link_infos = out.split("\r\n")
-        for link_info in link_infos:
-            m = re.match(link_pattern, link_info)
-            if m:
-                port = m.group(1)
-                status = m.group(2)
-                self.verify(status == "Down", "Userspace tool failed to detect link down")
-
-        for port in self.ports:
-            tester_port = self.tester.get_local_port(port)
-            intf = self.tester.get_interface(tester_port)
-            self.tester.send_expect("ip link set dev %s up" % intf, "# ")
-        # wait for link stable
-        time.sleep(5)
+        # ethtool doesn't support port disconnect by tools of linux 
+        # only detect physical link disconnect status
+        if self.nic.startswith("fortville") == False:  
+            # check link status dump function
+            for port in self.ports:
+                tester_port = self.tester.get_local_port(port)
+                intf = self.tester.get_interface(tester_port)
+                self.tester.send_expect("ip link set dev %s down" % intf, "# ")
+            # wait for link stable
+            time.sleep(5)
+    
+            out = self.dut.send_expect("link", "EthApp>", 60)
+            link_pattern = r"Port (\d+): (.*)"
+            link_infos = out.split("\r\n")
+            for link_info in link_infos:
+                m = re.match(link_pattern, link_info)
+                if m:
+                    port = m.group(1)
+                    status = m.group(2)
+                    self.verify(status == "Down", "Userspace tool failed to detect link down")
+    
+            for port in self.ports:
+                tester_port = self.tester.get_local_port(port)
+                intf = self.tester.get_interface(tester_port)
+                self.tester.send_expect("ip link set dev %s up" % intf, "# ")
+            # wait for link stable
+            time.sleep(5)
 
         # check port stats function
         pkt = Packet(pkt_type='UDP')
@@ -271,8 +290,10 @@ class TestUserspaceEthtool(TestCase, IxiaPacketGenerator):
 
         for index in range(len(self.ports)):
             md5 = self.strip_md5(portsinfo[index]['eeprom_file'])
+            self.resize_linux_eeprom_file( portsinfo[index]['eeprom_file'], portsinfo[index]['ethtool_eeprom'])
             md5_ref = self.strip_md5(portsinfo[index]['ethtool_eeprom'])
-            print dts.GREEN("Reference eeprom md5 %s" % md5_ref)
+            print dts.GREEN("Reference eeprom md5 %s" % md5)
+            print dts.GREEN("Reference eeprom md5_ref %s" % md5_ref)
             self.verify(md5 == md5_ref, "Dumped eeprom not same as linux dumped")
 
     def test_ring_parameter(self):
@@ -323,7 +344,6 @@ class TestUserspaceEthtool(TestCase, IxiaPacketGenerator):
             tester_port = self.tester.get_local_port(port)
             intf = self.tester.get_interface(tester_port)
             pkt.send_pkt(tx_port=intf)
-            time.sleep(2)
             rx_pkts, tx_pkts = self.strip_portstats(port)
             self.verify(rx_pkts == ori_rx_pkts + 1, "Failed to Rx vlan packet")
             self.verify(tx_pkts == ori_tx_pkts + 1, "Failed to Tx vlan packet")
@@ -372,8 +392,8 @@ class TestUserspaceEthtool(TestCase, IxiaPacketGenerator):
             inst = sniff_packets(intf, timeout=5)
             pkt.send_pkt(tx_port=intf)
             pkts = load_sniff_packets(inst)
-            self.verify(len(pkts) == 2, "Packet not forwarded as expected")
-            src_mac = pkts[1].strip_layer_element("layer2", "src")
+            self.verify(len(pkts) == 1, "Packet not forwarded as expected")
+            src_mac = pkts[0].strip_layer_element("layer2", "src")
             self.verify(src_mac == valid_mac, "Forwarded packet not match default mac")
 
         # check multicase will not be valid mac
@@ -405,9 +425,11 @@ class TestUserspaceEthtool(TestCase, IxiaPacketGenerator):
             rx_pkts, tx_pkts = self.strip_portstats(index)
             self.verify(rx_pkts == ori_rx_pkts, "Failed to stop port")
             # restart port and check packet can normally forwarded
+            time.sleep(2)
             self.dut.send_expect("open %d" % index, "EthApp>")
             # wait few time for port ready
-            time.sleep(0.5)
+            rx_pkts, tx_pkts = self.strip_portstats(index)
+            time.sleep(2)
             pkt.send_pkt(tx_port=intf)
             rx_pkts_open, tx_pkts_open = self.strip_portstats(index)
             self.verify(rx_pkts_open == rx_pkts + 1, "Failed to reopen port rx")
@@ -430,11 +452,12 @@ class TestUserspaceEthtool(TestCase, IxiaPacketGenerator):
             self.tester.send_expect("ifconfig %s mtu 9000" % (intf), "# ")
             for mtu in mtus:
                 self.dut.send_expect("mtu %d %d" % (index, mtu), "EthApp>")
+                ori_rx_pkts, _ = self.strip_portstats(index)
                 pkt_size = mtu + HEADER_SIZE['eth']
                 pkt = Packet(pkt_len=pkt_size)
                 pkt.send_pkt(tx_port=intf)
                 rx_pkts, _ = self.strip_portstats(index)
-                self.verify(rx_pkts == 1, "Packet match mtu not forwarded as expected")
+                self.verify(rx_pkts == ori_rx_pkts, "Packet match mtu not forwarded as expected")
                 pkt = Packet(pkt_len=mtu + 1)
                 pkt.send_pkt(tx_port=intf)
                 rx_pkts_over, _ = self.strip_portstats(index)
-- 
1.9.3