[PATCH V2 0/3] add case3 and optimize verify expected value

[PATCH V2 0/3] add case3 and optimize verify expected value

[Wei Ling]

1.test_plans/pvp_diff_qemu_version: add case3 to test virtio 1.1 path.
2.tests/pvp_diff_qemu_version: add case3 to test virtio1.1 and optimize
  verify expected value.
3.conf/pvp_diff_qemu_version: add expected value to verify.

Wei Ling (3):
  test_plans/pvp_diff_qemu_version: add case3 to test virtio 1.1 path
  tests/pvp_diff_qemu_version: add case3 to test virtio1.1 and optimize
    verify expected
  conf/pvp_diff_qemu_version: add expected value to verify

 conf/pvp_diff_qemu_version.cfg                |  41 ++-
 .../pvp_diff_qemu_version_test_plan.rst       |  56 ++++-
 tests/TestSuite_pvp_diff_qemu_version.py      | 236 ++++++++++--------
 3 files changed, 216 insertions(+), 117 deletions(-)

-- 
2.25.1

[dts][PATCH V2 1/3] test_plans/pvp_diff_qemu_version: add case3 to test virtio 1.1 path

[Wei Ling]

1.Add case3 to test virtio 1.1 path.
2.Modify the QEMU version as the version upgrade.

Signed-off-by: Wei Ling <weix.ling@intel.com>
---
 .../pvp_diff_qemu_version_test_plan.rst       | 56 ++++++++++++++++---
 1 file changed, 48 insertions(+), 8 deletions(-)

diff --git a/test_plans/pvp_diff_qemu_version_test_plan.rst b/test_plans/pvp_diff_qemu_version_test_plan.rst
index 83a7b269..f29dbf4b 100644
--- a/test_plans/pvp_diff_qemu_version_test_plan.rst
+++ b/test_plans/pvp_diff_qemu_version_test_plan.rst
@@ -8,7 +8,7 @@ vhost/virtio different qemu version test plan
 Description
 ===========
 
-This test plan will test pvp different qemu version test cases, also cover virtio 1.0 and virtio 0.95.
+This test plan will test pvp different qemu version test cases, also cover virtio 0.95, virtio 1.0 and virtio 1.1.
 
 Test flow
 =========
@@ -27,9 +27,9 @@ Test Case 1: PVP multi qemu version test with virtio 0.95 mergeable path
     testpmd>set fwd mac
     testpmd>start
 
-2. Check dut machine already has installed different version qemu, includes [qemu_2.5, qemu_2.6, qemu_2.7, qemu_2.8, qemu_2.9, qemu_2.10, qemu_2.11, qemu_2.12, qemu_3.0].
+2. Check dut machine already has installed different version qemu, includes [qemu-5.2.0, qemu-6.2.0, qemu-7.0.0, qemu-7.1.0, qemu-7.2.0].
 
-3. Go to the absolute_path of different version qemu,then launch VM with different version qemu::
+3. Go to the absolute_path of different version qemu,then launch VM with different version qemu, note: we need add "disable-modern=true" to enable virtio 0.95::
 
     qemu-system-x86_64 -name vm2 -enable-kvm -cpu host -smp 2 -m 4096 \
     -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
@@ -40,10 +40,11 @@ Test Case 1: PVP multi qemu version test with virtio 0.95 mergeable path
     -netdev user,id=netdev0,hostfwd=tcp:127.0.0.1:6002-:22 \
     -chardev socket,id=char0,path=./vhost-net \
     -netdev type=vhost-user,id=netdev1,chardev=char0,vhostforce \
-    -device virtio-net-pci,netdev=netdev1,mac=52:54:00:00:00:01,mrg_rxbuf=on \
+    -device virtio-net-pci,netdev=netdev1,mac=52:54:00:00:00:01,disable-modern=true,mrg_rxbuf=on \
     -vnc :10
 
 4. On VM, bind virtio net to vfio-pci and run testpmd ::
+
     ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0x3 -n 3 -- -i \
     --nb-cores=1 --txd=1024 --rxd=1024
     testpmd>set fwd mac
@@ -65,7 +66,7 @@ Test Case 2: PVP test with virtio 1.0 mergeable path
     testpmd>set fwd mac
     testpmd>start
 
-2. Check dut machine already has installed different version qemu, includes [qemu_2.5, qemu_2.6, qemu_2.7, qemu_2.8, qemu_2.9, qemu_2.10, qemu_2.11, qemu_2.12, qemu_3.0].
+2. Check dut machine already has installed different version qemu, includes [qemu-5.2.0, qemu-6.2.0, qemu-7.0.0, qemu-7.1.0, qemu-7.2.0].
 
 3. Go to the absolute_path of different version qemu,then launch VM with different version qemu, note: we need add "disable-modern=false" to enable virtio 1.0::
 
@@ -81,13 +82,52 @@ Test Case 2: PVP test with virtio 1.0 mergeable path
     -device virtio-net-pci,netdev=netdev1,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on \
     -vnc :10
 
-3. On VM, bind virtio net to vfio-pci and run testpmd::
+4. On VM, bind virtio net to vfio-pci and run testpmd::
 
     ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0x3 -n 3 -- -i \
     --nb-cores=1 --txd=1024 --rxd=1024
     testpmd>set fwd mac
     testpmd>start
 
-4. Send packet by packet generator with different packet sizes(68,128,256,512,1024,1280,1518),repeat below command to get throughput 10 times,then calculate the average throughput::
+5. Send packet by packet generator with different packet sizes(68,128,256,512,1024,1280,1518),repeat below command to get throughput 10 times,then calculate the average throughput::
+
+    testpmd>show port stats all
+
+Test Case 3: PVP test with virtio 1.1 mergeable path
+====================================================
+
+1. Bind one port to vfio-pci, then launch testpmd by below command::
 
-    testpmd>show port stats all
\ No newline at end of file
+    rm -rf vhost-net*
+    ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0xe -n 4 \
+    --vdev 'eth_vhost0,iface=vhost-net,queues=1' -- \
+    -i --nb-cores=1 --txd=1024 --rxd=1024
+    testpmd>set fwd mac
+    testpmd>start
+
+2. Check dut machine already has installed different version qemu, includes [qemu-5.2.0, qemu-6.2.0, qemu-7.0.0, qemu-7.1.0, qemu-7.2.0].
+
+3. Go to the absolute_path of different version qemu,then launch VM with different version qemu, note: we need add "disable-modern=false,packed=on" to enable virtio 1.1::
+
+    qemu-system-x86_64 -name vm2 -enable-kvm -cpu host -smp 2 -m 4096 \
+    -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
+    -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu16.img  \
+    -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
+    -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
+    -monitor unix:/tmp/vm2_monitor.sock,server,nowait -net nic,macaddr=00:00:00:08:e8:aa,addr=1f \
+    -netdev user,id=netdev0,hostfwd=tcp:127.0.0.1:6002-:22 \
+    -chardev socket,id=char0,path=./vhost-net \
+    -netdev type=vhost-user,id=netdev1,chardev=char0,vhostforce \
+    -device virtio-net-pci,netdev=netdev1,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,packed=on \
+    -vnc :10
+
+4. On VM, bind virtio net to vfio-pci and run testpmd::
+
+    ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0x3 -n 3 -- -i \
+    --nb-cores=1 --txd=1024 --rxd=1024
+    testpmd>set fwd mac
+    testpmd>start
+
+5. Send packet by packet generator with different packet sizes(68,128,256,512,1024,1280,1518),repeat below command to get throughput 10 times,then calculate the average throughput::
+
+    testpmd>show port stats all
-- 
2.25.1

[dts][PATCH V2 2/3] tests/pvp_diff_qemu_version: add case3 to test virtio1.1 and optimize verify expected

[Wei Ling]

1.Add case3 to test virtio 1.1 path.
2.Modify the casename with virtio protocol version.
3.Optimize the verify the expected value from the hard code to read from
  conf/pvp_diff_qemu_version.cfg to adapt different platform.

Signed-off-by: Wei Ling <weix.ling@intel.com>
---
 tests/TestSuite_pvp_diff_qemu_version.py | 236 +++++++++++++----------
 1 file changed, 136 insertions(+), 100 deletions(-)

diff --git a/tests/TestSuite_pvp_diff_qemu_version.py b/tests/TestSuite_pvp_diff_qemu_version.py
index 02682815..6a7830f4 100644
--- a/tests/TestSuite_pvp_diff_qemu_version.py
+++ b/tests/TestSuite_pvp_diff_qemu_version.py
@@ -2,23 +2,11 @@
 # Copyright(c) 2019 Intel Corporation
 #
 
-"""
-DPDK Test suite.
-
-Vhost PVP performance using differnet Qemu test suite.
-Can config the qemu version in config file like:
-qemu =
-    path=qemu-2.5/bin/qemu-system-x86_64;
-    path=qemu-2.6/bin/qemu-system-x86_64;
-"""
 import re
 import time
 
-from scapy.utils import wrpcap
-
-import framework.utils as utils
 from framework.pktgen import PacketGeneratorHelper
-from framework.settings import HEADER_SIZE
+from framework.settings import HEADER_SIZE, UPDATE_EXPECTED, load_global_setting
 from framework.test_case import TestCase
 from framework.virt_common import VM
 
@@ -34,7 +22,6 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         self.cores_num = len(
             [n for n in self.dut.cores if int(n["socket"]) == self.socket]
         )
-
         self.verify(len(self.dut_ports) >= 1, "Insufficient ports for testing")
         self.verify(
             self.cores_num >= 3, "There has not enought cores to test this suite"
@@ -42,7 +29,6 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         self.cores = self.dut.get_core_list("1S/3C/1T", socket=self.socket)
         self.vm_dut = None
         self.packet_params_set()
-
         self.logger.info(
             "You can config all the path of qemu version you want to"
             + " tested in the conf file %s.cfg" % self.suite_name
@@ -53,7 +39,6 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         )
         res = self.verify_qemu_version_config()
         self.verify(res is True, "The path of qemu version in config file not right")
-
         self.out_path = "/tmp"
         out = self.tester.send_expect("ls -d %s" % self.out_path, "# ")
         if "No such file or directory" in out:
@@ -62,36 +47,36 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         self.pktgen_helper = PacketGeneratorHelper()
         self.base_dir = self.dut.base_dir.replace("~", "/root")
         self.pci_info = self.dut.ports_info[0]["pci"]
-        self.number_of_ports = 1
+        self.nb_ports = 1
         self.path = self.dut.apps_name["test-pmd"]
         self.testpmd_name = self.path.split("/")[-1]
+        self.gap = self.get_suite_cfg()["accepted_tolerance"]
+        self.vhost_user = self.dut.new_session(suite="vhost-user")
 
     def set_up(self):
         """
         Run before each test case.
         """
-        self.vhost = self.dut.new_session(suite="vhost-user")
         self.dut.send_expect("rm -rf %s/vhost-net*" % self.base_dir, "#")
         self.dut.send_expect("killall -s INT %s" % self.testpmd_name, "#")
         self.dut.send_expect("killall -I qemu-system-x86_64", "#", 20)
+        self.throughput = dict()
+        self.test_result = []
+        self.table_header = [
+            "QemuVersion",
+            "FrameSize(B)",
+            "Throughput(Mpps)",
+            "LineRate(%)",
+        ]
 
     def packet_params_set(self):
-        self.frame_sizes = [64, 128, 256, 512, 1024, 1500]
+        self.frame_sizes = [64, 128, 256, 512, 1024, 1280, 1518]
         # get the frame_sizes from cfg file
         if "packet_sizes" in self.get_suite_cfg():
             self.frame_sizes = self.get_suite_cfg()["packet_sizes"]
-
         self.virtio1_mac = "52:54:00:00:00:01"
         self.src1 = "192.168.4.1"
         self.dst1 = "192.168.3.1"
-        self.header_row = [
-            "QemuVersion",
-            "FrameSize(B)",
-            "Throughput(Mpps)",
-            "LineRate(%)",
-            "Cycle",
-            "Expected Throughput(Mpps)",
-        ]
 
     def get_qemu_list_from_config(self):
         """
@@ -122,7 +107,6 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         qemu_num = len(self.qemu_list)
         for i in range(qemu_num):
             qemu_path = self.qemu_list[i]["path"]
-
             out = self.dut.send_expect("ls %s" % qemu_path, "#")
             if "No such file or directory" in out:
                 self.logger.error(
@@ -137,13 +121,14 @@ class TestVhostPVPDiffQemuVersion(TestCase):
                     % (qemu_path, self.dut.get_ip_address())
                 )
                 return False
-
             out = self.dut.send_expect("%s --version" % qemu_path, "#")
             result = re.search("QEMU\s*emulator\s*version\s*(\d*.\d*)", out)
             version = result.group(1)
             # update the version info to self.qemu_list
             self.qemu_list[i].update({"version": "qemu-%s" % version})
-
+        self.qemu_versions = list()
+        for i in self.qemu_list:
+            self.qemu_versions.append(i["version"])
         # print all the qemu version you config
         config_qemu_version = ""
         for i in range(len(self.qemu_list)):
@@ -151,7 +136,6 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         self.logger.info(
             "The suite will test the qemu version of: %s" % config_qemu_version
         )
-
         return True
 
     def rm_vm_qemu_path_config(self):
@@ -167,7 +151,7 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         self.verify(qemu_index < params_num, "Please config qemu path in conf gile")
         self.vm.params.pop(qemu_index)
 
-    def start_vm(self, path, modem):
+    def start_one_vm(self, qemu_path, qemu_mode):
         """
         start vm
         """
@@ -176,15 +160,17 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         vm_params["driver"] = "vhost-user"
         vm_params["opt_path"] = "%s/vhost-net" % self.base_dir
         vm_params["opt_mac"] = self.virtio1_mac
-        if modem == 1:
-            vm_params["opt_settings"] = "disable-modern=false,mrg_rxbuf=on"
-        elif modem == 0:
+        if qemu_mode == 0:
             vm_params["opt_settings"] = "disable-modern=true,mrg_rxbuf=on"
+        elif qemu_mode == 1:
+            vm_params["opt_settings"] = "disable-modern=false,mrg_rxbuf=on"
+        elif qemu_mode == 2:
+            vm_params["opt_settings"] = "disable-modern=false,mrg_rxbuf=on,packed=on"
         self.vm.set_vm_device(**vm_params)
         self.vm.load_config()
         self.rm_vm_qemu_path_config()
         # set qemu version info
-        self.vm.set_qemu_emulator(path)
+        self.vm.set_qemu_emulator(qemu_path)
         # Due to we have change the params info before,
         # so need to start vm with load_config=False
         try:
@@ -204,9 +190,9 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         )
         para = " -- -i --nb-cores=1 --txd=1024 --rxd=1024"
         command_line_client = self.path + eal_params + para
-        self.vhost.send_expect(command_line_client, "testpmd> ", 30)
-        self.vhost.send_expect("set fwd mac", "testpmd> ", 30)
-        self.vhost.send_expect("start", "testpmd> ", 30)
+        self.vhost_user.send_expect(command_line_client, "testpmd> ", 30)
+        self.vhost_user.send_expect("set fwd mac", "testpmd> ", 30)
+        self.vhost_user.send_expect("start", "testpmd> ", 30)
 
     def vm_testpmd_start(self):
         """
@@ -220,26 +206,9 @@ class TestVhostPVPDiffQemuVersion(TestCase):
             self.vm_dut.send_expect("set fwd mac", "testpmd> ", 20)
             self.vm_dut.send_expect("start", "testpmd> ")
 
-    @property
-    def check_value(self):
-        check_dict = dict.fromkeys(self.frame_sizes)
-        linerate = {
-            64: 0.07,
-            128: 0.10,
-            256: 0.17,
-            512: 0.18,
-            1024: 0.35,
-            1280: 0.40,
-            1500: 0.45,
-        }
-        for size in self.frame_sizes:
-            speed = self.wirespeed(self.nic, size, self.number_of_ports)
-            check_dict[size] = round(speed * linerate[size], 2)
-        return check_dict
-
-    def send_verify(self, qemu_version, vlan_id1=0, tag="Performance"):
-        self.result_table_create(self.header_row)
-        perf_result = {}
+    def perf_test(self, qemu_version, vlan_id1=0):
+        self.result_table_create(self.table_header)
+        self.throughput[qemu_version] = {}
         for frame_size in self.frame_sizes:
             info = "Running test %s, and %d frame size." % (
                 self.running_case,
@@ -264,42 +233,30 @@ class TestVhostPVPDiffQemuVersion(TestCase):
             tgenInput.append(
                 (port, port, "%s/pvp_diff_qemu_version.pcap" % self.out_path)
             )
-
             self.tester.pktgen.clear_streams()
             streams = self.pktgen_helper.prepare_stream_from_tginput(
                 tgenInput, 100, None, self.tester.pktgen
             )
             # set traffic option
-            traffic_opt = {"delay": 5, "duration": 20}
+            traffic_opt = {
+                "delay": 5,
+                "duration": self.get_suite_cfg()["test_duration"],
+            }
             _, pps = self.tester.pktgen.measure_throughput(
                 stream_ids=streams, options=traffic_opt
             )
             Mpps = pps / 1000000.0
             line_rate = Mpps * 100 / float(self.wirespeed(self.nic, frame_size, 1))
+            self.throughput[qemu_version][frame_size] = Mpps
             # update print table info
             data_row = [
                 qemu_version,
                 frame_size,
                 str(Mpps),
                 str(line_rate),
-                tag,
-                self.check_value[frame_size],
             ]
             self.result_table_add(data_row)
-            perf_result[frame_size] = Mpps
-
         self.result_table_print()
-        for frame_size in perf_result.keys():
-            self.verify(
-                perf_result[frame_size] > self.check_value[frame_size],
-                "%s of frame size %d speed verify failed, expect %s, result %s"
-                % (
-                    self.running_case,
-                    frame_size,
-                    self.check_value[frame_size],
-                    perf_result[frame_size],
-                ),
-            )
 
     def close_testpmd_and_qemu(self):
         """
@@ -307,57 +264,136 @@ class TestVhostPVPDiffQemuVersion(TestCase):
         close the qemu
         """
         self.vm_dut.send_expect("quit", "#", 20)
-        self.vhost.send_expect("quit", "#", 20)
+        self.vhost_user.send_expect("quit", "#", 20)
         self.vm.stop()
         self.dut.send_expect("killall -I %s" % self.testpmd_name, "#", 20)
         self.dut.send_expect("rm -rf %s/vhost-net*" % self.base_dir, "#")
 
-    def test_perf_vhost_pvp_diffrent_qemu_version_mergeable_mac(self):
+    def test_perf_vhost_pvp_diff_qemu_version_virtio95_mergeable_path(self):
+        """
+        Test Case 1: PVP multi qemu version test with virtio 0.95 mergeable path
+        """
+        self.test_target = self.running_case
+        self.expected_throughput = self.get_suite_cfg()["expected_throughput"][
+            self.test_target
+        ]
+        for i in range(len(self.qemu_list)):
+            qemu_path = self.qemu_list[i]["path"]
+            qemu_version = self.qemu_list[i]["version"]
+            self.start_vhost_testpmd()
+            self.logger.info("now testing the qemu path of %s" % qemu_path)
+            self.start_one_vm(qemu_path=qemu_path, qemu_mode=0)
+            self.vm_testpmd_start()
+            vlan_id1 = 1000
+            self.perf_test(qemu_version, vlan_id1)
+            self.close_testpmd_and_qemu()
+        self.handle_expected()
+        self.handle_results()
+
+    def test_perf_vhost_pvp_diff_qemu_version_virtio10_mergeable_path(self):
         """
-        Test the performance of one vm with virtio 0.95 on mergeable path
+        Test Case 2: PVP test with virtio 1.0 mergeable path
         """
+        self.test_target = self.running_case
+        self.expected_throughput = self.get_suite_cfg()["expected_throughput"][
+            self.test_target
+        ]
         for i in range(len(self.qemu_list)):
-            path = self.qemu_list[i]["path"]
-            version = self.qemu_list[i]["version"]
+            qemu_path = self.qemu_list[i]["path"]
+            qemu_version = self.qemu_list[i]["version"]
             self.start_vhost_testpmd()
-            self.start_vm(path, 0)
-            # Start testpmd in vm
+            self.logger.info("now testing the qemu path of %s" % qemu_path)
+            self.start_one_vm(qemu_path=qemu_path, qemu_mode=1)
             self.vm_testpmd_start()
-            self.logger.info("now testing the qemu path of %s" % path)
-            time.sleep(5)
             vlan_id1 = 1000
-            self.send_verify(version, vlan_id1, "virtio-0.95, Mergeable")
+            self.perf_test(qemu_version, vlan_id1)
             self.close_testpmd_and_qemu()
+        self.handle_expected()
+        self.handle_results()
 
-    def test_perf_vhost_pvp_diffrent_qemu_version_modern_mergeable_mac(self):
+    def test_perf_vhost_pvp_diff_qemu_version_virtio11_mergeable_path(self):
         """
-        Test the performance of one vm with virtio 1.0 on mergeable path
+        Test Case 3: PVP test with virtio 1.1 mergeable path
         """
+        self.test_target = self.running_case
+        self.expected_throughput = self.get_suite_cfg()["expected_throughput"][
+            self.test_target
+        ]
         for i in range(len(self.qemu_list)):
-            path = self.qemu_list[i]["path"]
-            version = self.qemu_list[i]["version"]
+            qemu_path = self.qemu_list[i]["path"]
+            qemu_version = self.qemu_list[i]["version"]
             self.start_vhost_testpmd()
-            self.start_vm(path, 1)
-            # Start testpmd in vm
+            self.logger.info("now testing the qemu path of %s" % qemu_path)
+            self.start_one_vm(qemu_path=qemu_path, qemu_mode=2)
             self.vm_testpmd_start()
-            self.logger.info("now testing the qemu path of %s" % path)
-            time.sleep(5)
             vlan_id1 = 1000
-            self.send_verify(version, vlan_id1, "virtio-1.0, Mergeable")
+            self.perf_test(qemu_version, vlan_id1)
             self.close_testpmd_and_qemu()
+        self.handle_expected()
+        self.handle_results()
+
+    def handle_expected(self):
+        """
+        Update expected numbers to configurate file: $DTS_CFG_FOLDER/$suite_name.cfg
+        """
+        if load_global_setting(UPDATE_EXPECTED) == "yes":
+            for qemu_version in self.qemu_versions:
+                for frame_size in self.frame_sizes:
+                    self.expected_throughput[qemu_version][frame_size] = round(
+                        self.throughput[qemu_version][frame_size], 3
+                    )
+
+    def handle_results(self):
+        """
+        results handled process:
+        1, save to self.test_results
+        2, create test results table
+        """
+        # save test results to self.test_result
+        header = self.table_header
+        header.append("Expected Throughput(Mpps)")
+        header.append("Status")
+        self.result_table_create(self.table_header)
+        for qv in self.qemu_list:
+            qemu_version = qv["version"]
+            for frame_size in self.frame_sizes:
+                wirespeed = self.wirespeed(self.nic, frame_size, self.nb_ports)
+                ret_data = {}
+                ret_data[header[0]] = qemu_version
+                ret_data[header[1]] = str(frame_size)
+                _real = float(self.throughput[qemu_version][frame_size])
+                _exp = float(self.expected_throughput[qemu_version][frame_size])
+                ret_data[header[2]] = "{:.3f}".format(_real)
+                ret_data[header[3]] = "{:.3f}%".format(_real * 100 / wirespeed)
+                ret_data[header[4]] = "{:.3f}".format(_exp)
+                gap = _exp * -self.gap * 0.01
+                if _real > _exp + gap:
+                    ret_data[header[5]] = "PASS"
+                else:
+                    ret_data[header[5]] = "FAIL"
+                self.test_result.append(ret_data)
+
+        for test_result in self.test_result:
+            table_row = list()
+            for i in range(len(header)):
+                table_row.append(test_result[header[i]])
+            self.result_table_add(table_row)
+        # present test results to screen
+        self.result_table_print()
+        self.verify(
+            "FAIL" not in self.test_result,
+            "Excessive gap between test results and expectations",
+        )
 
     def tear_down(self):
         """
         Run after each test case.
-        Clear qemu and testpmd to avoid blocking the following TCs
         """
-        self.dut.close_session(self.vhost)
         self.dut.send_expect("killall -s INT qemu-system-x86_64", "#")
         self.dut.send_expect("killall -s INT %s" % self.testpmd_name, "#")
-        time.sleep(2)
 
     def tear_down_all(self):
         """
         Run after each test suite.
         """
-        pass
+        self.dut.close_session(self.vhost_user)
-- 
2.25.1

[dts][PATCH V2 3/3] conf/pvp_diff_qemu_version: add expected value to verify

[Wei Ling]

Add expected value to verify the running result. 

Signed-off-by: Wei Ling <weix.ling@intel.com>
---
 conf/pvp_diff_qemu_version.cfg | 41 ++++++++++++++++++++++++++--------
 1 file changed, 32 insertions(+), 9 deletions(-)

diff --git a/conf/pvp_diff_qemu_version.cfg b/conf/pvp_diff_qemu_version.cfg
index 0353c246..e89f7b8e 100644
--- a/conf/pvp_diff_qemu_version.cfg
+++ b/conf/pvp_diff_qemu_version.cfg
@@ -1,20 +1,43 @@
-# Configuration sample: vhost_sample.cfg
 [vm0]
 cpu =
-    model=host,number=4,cpupin=4 5;
+    model=host,number=8,cpupin=20 21 22 23 24 25 26 27;
 mem =
-    size=4096,hugepage=yes;
+    size=8192,hugepage=yes;
 disk =
-    file=/home/img/vm.img;
+    file=/home/image/ubuntu2004.img;
 login =
     user=root,password=tester;
 vnc =
     displayNum=4;
 daemon =
     enable=yes;
-# config all the path of qemu version you want to tested
 qemu =
-    path=path/qemu-2.5/bin/qemu-system-x86_64;
-    path=path/qemu-2.6/bin/qemu-system-x86_64;
-    path=path/qemu-2.7/bin/qemu-system-x86_64;
-    path=path/qemu-2.8/bin/qemu-system-x86_64;
+    path=/home/QEMU/qemu-5.2.0/bin/qemu-system-x86_64;
+    path=/home/QEMU/qemu-6.2.0/bin/qemu-system-x86_64;
+    path=/home/QEMU/qemu-7.0.0/bin/qemu-system-x86_64;
+    path=/home/QEMU/qemu-7.1.0/bin/qemu-system-x86_64;
+    path=/home/QEMU/qemu-7.2.0/bin/qemu-system-x86_64;
+[suite]
+update_expected = True
+packet_sizes = [64, 128, 256, 512, 1024, 1280, 1518]
+test_duration = 30
+accepted_tolerance = 5
+expected_throughput = {
+    'test_perf_vhost_pvp_diff_qemu_version_virtio95_mergeable_path': {
+        'qemu-5.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-6.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-7.0': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-7.1': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-7.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}},
+    'test_perf_vhost_pvp_diff_qemu_version_virtio10_mergeable_path': {
+        'qemu-5.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-6.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-7.0': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-7.1': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-7.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}},
+    'test_perf_vhost_pvp_diff_qemu_version_virtio11_mergeable_path': {
+        'qemu-5.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-6.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-7.0': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-7.1': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0},
+        'qemu-7.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}}}
-- 
2.25.1

RE: [dts][PATCH V2 3/3] conf/pvp_diff_qemu_version: add expected value to verify

[He, Xingguang]

> -----Original Message-----
> From: Wei Ling <weix.ling@intel.com>
> Sent: Monday, April 24, 2023 3:34 PM
> To: dts@dpdk.org
> Cc: Ling, WeiX <weix.ling@intel.com>
> Subject: [dts][PATCH V2 3/3] conf/pvp_diff_qemu_version: add expected
> value to verify
> 
> Add expected value to verify the running result.
> 
> Signed-off-by: Wei Ling <weix.ling@intel.com>
> ---
Acked-by: Xingguang He<xingguang.he@intel.com>

[dts][PATCH V2 3/3] conf/pvp_diff_qemu_version: add expected value to verify

[lijuan.tu]

On Mon, 24 Apr 2023 15:33:40 +0800, Wei Ling <weix.ling@intel.com> wrote:
> Add expected value to verify the running result. 
> 
> Signed-off-by: Wei Ling <weix.ling@intel.com>

Acked-by: Lijuan Tu <lijuan.tu@intel.com>
Series applied, thanks