Re: [dts] [PATCH V1]userspace_ethtool: update test script and test plan

["Liu, Yong" <yong.liu@intel.com>]

Applied, thanks.

> -----Original Message-----
> From: dts [mailto:dts-bounces@dpdk.org] On Behalf Of Yufen Mo
> Sent: Wednesday, May 04, 2016 7:59 PM
> To: dts@dpdk.org
> Cc: Mo, YufengX
> Subject: [dts] [PATCH V1]userspace_ethtool: update test script and test
> plan
> 
> From: yufengmx <yufengx.mo@intel.com>
> 
> change vlan packet check number when doing vlan testing. Add process to
> resize linux ethtool eeprom dump
> data size as the size dump by dpdk ethtool. Ignore auto test link connect
> status on FVL. update related
> test cases' test plan, add testing configuration and requirements.
> 
> Signed-off-by: yufengmx <yufengx.mo@intel.com>
> ---
>  test_plans/userspace_ethtool_test_plan.rst | 167 ++++++++++++++++++------
> -----
>  tests/TestSuite_userspace_ethtool.py       |  85 +++++++++------
>  2 files changed, 156 insertions(+), 96 deletions(-)
> 
> diff --git a/test_plans/userspace_ethtool_test_plan.rst
> b/test_plans/userspace_ethtool_test_plan.rst
> index 5ee79a1..83e7fac 100644
> --- a/test_plans/userspace_ethtool_test_plan.rst
> +++ b/test_plans/userspace_ethtool_test_plan.rst
> @@ -34,31 +34,42 @@
>   Userspace Ethtool
>  ==================
> 
> -This feature is designed to provide one rte_ethtool shim layer based on
> -rte_ethdev API. The implementation also along with a command prompt
> driven
> -demonstration application. It only contained of 18 popular used Ethtool
> and
> -Netdevice ops as described in rte_ethtool.h.
> +This feature is designed to provide one rte_ethtool shim layer based on
> +rte_ethdev API. The Ethtool sample application shows an implementation
> +of an ethtool-like API and provides a console environment that allows
> +its use to query and change Ethernet card parameters. The Ethtool sample
> +is based upon a simple L2 frame reflector.
> 
>  Prerequisites
>  =============
> -
> +notice: On FVL, test case "test_dump_driver_info" need a physical link
> disconnect,
> +this case must do manually at this condition.
> +
>  Assume port 0 and 1 are connected to the traffic generator, to run the
> test
> -application in linuxapp environment with 4 lcores, 2 ports.
> +application in linux app environment with 4 lcores, 2 ports.
> 
>  	ethtool -c f -n 4
> 
> -The sample should be validated on Forville, Niantic and i350 Nics.
> +The sample should be validated on Forville, Niantic and i350 Nics.
> +
> +other requirements:
> +*. igxbe driver (version >= 4.3.13).
> +*. ethtool of linux is the default reference tool.
> +*. md5sum is a tool to do dumped bin format file comparison.
> +*. insert two nic cards on No.0 socket
> 
> -Test Case: Dump driver infor test
> +Test Case: Dump driver information test
>  =================================
> -User "drvinfo" command to dump driver information and then check that
> dumped
> -information was exactly the same as fact.
> +User "drvinfo" command to dump driver information and then check that
> +dumped information, which are dumped separately by dpdk's ethtool and
> +linux's ethtool, were exactly the same.
> 
>  	EthApp> drvinfo
>  	Port 0 driver: rte_ixgbe_pmd (ver: RTE 2.1.0)
>  	Port 1 driver: rte_ixgbe_pmd (ver: RTE 2.1.0)
> 
>  Use "link" command to dump all ports link status.
> +Notice:: On FVL, link detect need a physical link disconnect.
>  	EthApp> link
>  	Port 0: Up
>  	Port 1: Up
> @@ -68,7 +79,7 @@ Change tester port link status to down and re-check link
> status.
>  	Port 0: Down
>  	Port 1: Down
> 
> -Send few packets to l2fwd and check that command "portstats" dumped
> correct
> +Send a few packets to l2fwd and check that command "portstats" dumps
> correct
>  port statistics.
>      EthApp> portstats 0
>      Port 0 stats
> @@ -78,34 +89,38 @@ port statistics.
>  Test Case: Retrieve eeprom test
>  ===============================
>  Unbind ports from igb_uio and bind them to default driver.
> -Dump eeprom binary by ethtool.
> +Dump eeprom binary by linux's ethtool and dpdk's ethtool separately.
> 
>  ethtool --eeprom-dump INTF_0 raw on > ethtool_eeprom_0.bin
>  ethtool --eeprom-dump INTF_1 raw on > ethtool_eeprom_1.bin
> 
> -Retrieve eeprom on specified port and compare csum with the file dumped
> by ethtool.
> +Retrieve eeprom on specified port using dpdk's ethtool and
> +compare csum with the file dumped by ethtool.
> 
>  	EthApp> eeprom 0 eeprom_0.bin
>  	EthApp> eeprom 1 eeprom_1.bin
> 
> -md5sum ethtool_eeprom_0.bin
> -md5sum eeprom_0.bin > eeprom_0.bin
> -
> -diff ethtool_eeprom_0.hex eeprom_0.hex
> +	md5sum ethtool_eeprom_0.bin
> +	md5sum eeprom_0.bin
> +
> +compare md5sum value of the two bin files.
> 
>  Test Case: Retrieve register test
>  ===============================
> -Retrieve register on specified port, do not known how to check the binary?
> +Retrieve register on specified port.
> 
>  	EthApp> regs 0 reg_0.bin
>  	EthApp> regs 1 reg_1.bin
> 
>  Unbind ports from igb_uio and bind them to default driver.
> +
> +    dpdk/tools/dpdk_nic_bind.py --bind=ixgbe x:xx.x
> +
>  Check that dumped register information is correct.
> 
> -ethtool -d INTF_0 raw on file reg_0.bin
> -ethtool -d INTF_1 raw on file reg_0.bin
> -
> +ethtool -d INTF_0 raw off file reg_0.bin
> +ethtool -d INTF_1 raw off file reg_0.bin
> +
>  Test Case: Ring param test
>  ==========================
>  Dump port 0 ring size by ringparam command and check numbers are correct.
> @@ -126,52 +141,30 @@ Port 0 ring paramaeters
>    Rx Pending: 256 (256 max)
>    Tx Pending: 2048 (4096 max)
> 
> -Test Case: Pause test
> -=====================
> -Enable port 0 Rx pause frame and then create two packets flows in IXIA
> port.
> -One flow is 100000 normally packet and the second flow is pause frame.
> -Check that port 0 Rx speed dropped. For example, niantic will drop from
> -14.8Mpps to 7.49Mpps.
> -
> -	EthApp> pause 0 rx
> -
> -Use "parse" command to print port pause status, check that port 0 rx has
> been
> -paused.
> -	EthApp> pause 0
> -	Port 0: Rx Paused
> -
> -Unpause port 0 rx and then restart port0, check that packets Rx speed is
> normal.
> -	EthApp> pause 0 none
> -    EthApp>
> +send packet by scapy on Tester
> 
> -Pause port 0 TX pause frame.
> -	EthApp> pause 0 tx
> -
> -Use "parse" command to print port pause status, check that port 1 tx has
> been
> -paused.
> -    EthApp> pause 0
> -    Port 0: Tx Paused
> -
> -Enable flow control in IXIA port and send packets from IXIA with line
> rate.
> -Check that IXIA receive flow control packets and IXIA transmit speed
> dropped.
> -IXIA Rx packets more then Tx packets to check that received pause frame.
> -
> -Unpause port 0 tx and restart port 0. Then send packets to port0, check
> that
> -packets forwarded normally from port 0.
> -	EthApp> pause 0 none
> -    EthApp> stop 0
> -    EthApp> open 0
> +check tx/rx packets
> +EthApp>  portstats 0
> 
>  Test Case: Vlan test
>  ====================
> +enable vlan filter flag in main.c of dpdk's ethtool
> +
> +	sed -i -e '/cfg_port.txmode.mq_mode =
> ETH_MQ_TX_NONE;$/a\\cfg_port.rxmode.hw_vlan_filter=1;'
> examples/ethtool/ethtool-app/main.c
> +
> +re-compile examples/ethtool
> +
> +	make -C examples/ethtool
> +
>  Add vlan 0 to port 0 and vlan 1 to port1, send packet without vlan to
> port0,1
>  Verify port0 and port1 recevied vlan packets
> +
>  	EthApp> vlan 0 add 0
>  	VLAN vid 0 added
> 
>  	EthApp> vlan 1 add 1
>  	VLAN vid 1 added
> -
> +
>  Send packet with vlan0,1 to port0&1. Verify port0 and port1 received vlan
>  packets
> 
> @@ -189,7 +182,11 @@ port0 and port1 can not receive vlan packet.
>  Test Case: Mac address test
>  ===========================
>  Use "macaddr" command to dump port mac address and then check that dumped
> -information is exactly the same as fact.
> +information is exactly the same as ifconfig do.
> +
> +set a new mac address by dpdk's ethtool, send and sniff packet and check
> packet
> +forwared status
> +
>  	EthApp> macaddr 0
>  	Port 0 MAC Address: XX:XX:XX:XX:XX:XX
>  	EthApp> macaddr 1
> @@ -211,8 +208,8 @@ Use "macaddr" command to change port mac address and
> then check mac changed.
>  	MAC address changed
>  	EthApp> macaddr 0
>  	Port 0 MAC Address: 00:10:00:00:00:00
> -
> -Verified  mac adress in forwarded packets has been changed.
> +
> +Verified mac adress in forwarded packets has been changed.
> 
>  Test Case: Port config test
>  ===========================
> @@ -224,18 +221,58 @@ Use "open" command to re-enable port0. Send packets
> to port0 and verify
>  packets received and forwarded.
>  	EthApp> open 0
> 
> -
>  Test case: Mtu config test
>  ==========================
> -Use "mtu" command to change port 0 mtu from default 1518 to 1000.
> +Use "mtu" command to change port 0 mtu from default 1519 to 9000 on
> Tester's port.
> 
> -Send packet size over 1000 and check that packet will be detected as
> error.
> +Send packet size over 1519 and check that packet will be detected as
> error.
> 
> -    EthApp> mtu 0 1000
> +    EthApp> mtu 0 1519
>      Port 0 stats
>         In: 0 (0 bytes)
>        Out: 0 (0 bytes)
>        Err: 1
> 
> -Change mtu to default value and send packet size over 1000 and check that
> -packet will normally received.
> +Change mtu to default value and send packet size over 1519 and check that
> +packet will normally be received.
> +
> +Test Case: Pause tx/rx test(performance test)
> +=====================
> +Enable port 0 Rx pause frame and then create two packets flows on IXIA
> port.
> +One flow is 100000 normally packet and the second flow is pause frame.
> +Check that dut's port 0 Rx speed dropped status. For example, niantic
> will drop
> +from 14.8Mpps to 7.49Mpps.
> +
> +	EthApp> pause 0 rx
> +
> +Use "pause" command to print dut's port pause status, check that dut's
> port 0 rx
> +has been paused.
> +
> +	EthApp> pause 0
> +	Port 0: Rx Paused
> +
> +Release pause status of port 0 rx and then restart port 0, check that
> packets Rx
> +speed is normal.
> +	EthApp> pause 0 none
> +    EthApp>
> +
> +Pause port 0 TX pause frame.
> +	EthApp> pause 0 tx
> +
> +Use "pause" command to print port pause status, check that port 0 tx has
> been
> +paused.
> +    EthApp> pause 0
> +    Port 0: Tx Paused
> +
> +Enable flow control in IXIA port and send packets from IXIA with line
> rate.
> +Record line rate before send packet.
> +Check that IXIA receive flow control packets and IXIA transmit speed
> dropped.
> +IXIA Rx packets more then Tx packets to check that received pause
> frame.Compare
> +the line rates in the time before and after the Pause packets are
> injected
> +
> +Unpause port 0 tx and restart port 0. Then send packets to port0, check
> that
> +packets forwarded normally from port 0.
> +
> +	EthApp> pause 0 none
> +    EthApp> stop 0
> +    EthApp> open 0
> diff --git a/tests/TestSuite_userspace_ethtool.py
> b/tests/TestSuite_userspace_ethtool.py
> index c479093..c23f841 100644
> --- a/tests/TestSuite_userspace_ethtool.py
> +++ b/tests/TestSuite_userspace_ethtool.py
> @@ -38,12 +38,12 @@ import dts
>  import time
>  import re
>  from test_case import TestCase
> -from pmd_output import PmdOutput
>  from packet import Packet, sniff_packets, load_sniff_packets
>  import random
>  from etgen import IxiaPacketGenerator
>  from settings import HEADER_SIZE
>  from settings import SCAPY2IXIA
> +from utils import RED
> 
> 
>  class TestUserspaceEthtool(TestCase, IxiaPacketGenerator):
> @@ -127,6 +127,22 @@ class TestUserspaceEthtool(TestCase,
> IxiaPacketGenerator):
>          else:
>              return 1518
> 
> +    def resize_linux_eeprom_file(self, dpdk_eeprom_file,
> linux_eeprom_file):
> +        basePath = self.dut.base_dir
> +        with open( basePath + dpdk_eeprom_file, 'rb') as fpDpdk:
> +            dpdk_bytes = fpDpdk.read()
> +            dpdk_length = len(dpdk_bytes)
> +
> +        with open( basePath + linux_eeprom_file, 'rb') as fplinux:
> +            linux_bytes = fplinux.read()
> +            linux_length = len(linux_bytes)
> +
> +        self.verify(dpdk_length <= linux_length,
> +                    "linux ethtool haven't dump out enough data as dpdk
> ethtool")
> +
> +        with open( basePath + linux_eeprom_file, 'wb') as fplinux:
> +            fplinux.write(linux_bytes[:dpdk_length])
> +
>      def strip_md5(self, filename):
>          md5_info = self.dut.send_expect("md5sum %s" % filename, "# ")
>          md5_pattern = r"(\w+)  (\w+)"
> @@ -155,30 +171,33 @@ class TestUserspaceEthtool(TestCase,
> IxiaPacketGenerator):
>                  version = m.group(3)
>                  print dts.GREEN("Detect port %s with %s driver\n" % (port,
> driver))
> 
> -        # check link status dump function
> -        for port in self.ports:
> -            tester_port = self.tester.get_local_port(port)
> -            intf = self.tester.get_interface(tester_port)
> -            self.tester.send_expect("ip link set dev %s down" % intf, "#
> ")
> -        # wait for link stable
> -        time.sleep(5)
> -
> -        out = self.dut.send_expect("link", "EthApp>", 60)
> -        link_pattern = r"Port (\d+): (.*)"
> -        link_infos = out.split("\r\n")
> -        for link_info in link_infos:
> -            m = re.match(link_pattern, link_info)
> -            if m:
> -                port = m.group(1)
> -                status = m.group(2)
> -                self.verify(status == "Down", "Userspace tool failed to
> detect link down")
> -
> -        for port in self.ports:
> -            tester_port = self.tester.get_local_port(port)
> -            intf = self.tester.get_interface(tester_port)
> -            self.tester.send_expect("ip link set dev %s up" % intf, "# ")
> -        # wait for link stable
> -        time.sleep(5)
> +        # ethtool doesn't support port disconnect by tools of linux
> +        # only detect physical link disconnect status
> +        if self.nic.startswith("fortville") == False:
> +            # check link status dump function
> +            for port in self.ports:
> +                tester_port = self.tester.get_local_port(port)
> +                intf = self.tester.get_interface(tester_port)
> +                self.tester.send_expect("ip link set dev %s down" % intf,
> "# ")
> +            # wait for link stable
> +            time.sleep(5)
> +
> +            out = self.dut.send_expect("link", "EthApp>", 60)
> +            link_pattern = r"Port (\d+): (.*)"
> +            link_infos = out.split("\r\n")
> +            for link_info in link_infos:
> +                m = re.match(link_pattern, link_info)
> +                if m:
> +                    port = m.group(1)
> +                    status = m.group(2)
> +                    self.verify(status == "Down", "Userspace tool failed
> to detect link down")
> +
> +            for port in self.ports:
> +                tester_port = self.tester.get_local_port(port)
> +                intf = self.tester.get_interface(tester_port)
> +                self.tester.send_expect("ip link set dev %s up" % intf,
> "# ")
> +            # wait for link stable
> +            time.sleep(5)
> 
>          # check port stats function
>          pkt = Packet(pkt_type='UDP')
> @@ -271,8 +290,10 @@ class TestUserspaceEthtool(TestCase,
> IxiaPacketGenerator):
> 
>          for index in range(len(self.ports)):
>              md5 = self.strip_md5(portsinfo[index]['eeprom_file'])
> +
> self.resize_linux_eeprom_file( portsinfo[index]['eeprom_file'],
> portsinfo[index]['ethtool_eeprom'])
>              md5_ref = self.strip_md5(portsinfo[index]['ethtool_eeprom'])
> -            print dts.GREEN("Reference eeprom md5 %s" % md5_ref)
> +            print dts.GREEN("Reference eeprom md5 %s" % md5)
> +            print dts.GREEN("Reference eeprom md5_ref %s" % md5_ref)
>              self.verify(md5 == md5_ref, "Dumped eeprom not same as linux
> dumped")
> 
>      def test_ring_parameter(self):
> @@ -323,7 +344,6 @@ class TestUserspaceEthtool(TestCase,
> IxiaPacketGenerator):
>              tester_port = self.tester.get_local_port(port)
>              intf = self.tester.get_interface(tester_port)
>              pkt.send_pkt(tx_port=intf)
> -            time.sleep(2)
>              rx_pkts, tx_pkts = self.strip_portstats(port)
>              self.verify(rx_pkts == ori_rx_pkts + 1, "Failed to Rx vlan
> packet")
>              self.verify(tx_pkts == ori_tx_pkts + 1, "Failed to Tx vlan
> packet")
> @@ -372,8 +392,8 @@ class TestUserspaceEthtool(TestCase,
> IxiaPacketGenerator):
>              inst = sniff_packets(intf, timeout=5)
>              pkt.send_pkt(tx_port=intf)
>              pkts = load_sniff_packets(inst)
> -            self.verify(len(pkts) == 2, "Packet not forwarded as
> expected")
> -            src_mac = pkts[1].strip_layer_element("layer2", "src")
> +            self.verify(len(pkts) == 1, "Packet not forwarded as
> expected")
> +            src_mac = pkts[0].strip_layer_element("layer2", "src")
>              self.verify(src_mac == valid_mac, "Forwarded packet not match
> default mac")
> 
>          # check multicase will not be valid mac
> @@ -405,9 +425,11 @@ class TestUserspaceEthtool(TestCase,
> IxiaPacketGenerator):
>              rx_pkts, tx_pkts = self.strip_portstats(index)
>              self.verify(rx_pkts == ori_rx_pkts, "Failed to stop port")
>              # restart port and check packet can normally forwarded
> +            time.sleep(2)
>              self.dut.send_expect("open %d" % index, "EthApp>")
>              # wait few time for port ready
> -            time.sleep(0.5)
> +            rx_pkts, tx_pkts = self.strip_portstats(index)
> +            time.sleep(2)
>              pkt.send_pkt(tx_port=intf)
>              rx_pkts_open, tx_pkts_open = self.strip_portstats(index)
>              self.verify(rx_pkts_open == rx_pkts + 1, "Failed to reopen
> port rx")
> @@ -430,11 +452,12 @@ class TestUserspaceEthtool(TestCase,
> IxiaPacketGenerator):
>              self.tester.send_expect("ifconfig %s mtu 9000" % (intf), "# ")
>              for mtu in mtus:
>                  self.dut.send_expect("mtu %d %d" % (index, mtu),
> "EthApp>")
> +                ori_rx_pkts, _ = self.strip_portstats(index)
>                  pkt_size = mtu + HEADER_SIZE['eth']
>                  pkt = Packet(pkt_len=pkt_size)
>                  pkt.send_pkt(tx_port=intf)
>                  rx_pkts, _ = self.strip_portstats(index)
> -                self.verify(rx_pkts == 1, "Packet match mtu not forwarded
> as expected")
> +                self.verify(rx_pkts == ori_rx_pkts, "Packet match mtu not
> forwarded as expected")
>                  pkt = Packet(pkt_len=mtu + 1)
>                  pkt.send_pkt(tx_port=intf)
>                  rx_pkts_over, _ = self.strip_portstats(index)
> --
> 1.9.3