test suite reviews and discussions
 help / color / mirror / Atom feed
* [dts][PATCH V3 1/2] tests/meson_tests: add new testsuite for meson tests
@ 2022-05-26  9:19 Li, WeiyuanX
  2022-05-26  9:19 ` [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan " Li, WeiyuanX
  0 siblings, 1 reply; 5+ messages in thread
From: Li, WeiyuanX @ 2022-05-26  9:19 UTC (permalink / raw)
  To: dts, lijuan.tu; +Cc: Weiyuan Li

From: Weiyuan Li <weiyuanx.li@intel.com>

Add new testsuite for meson tests.

Signed-off-by: Weiyuan Li <weiyuanx.li@intel.com>
---

v2:
-fix format issue.
v3:
-add verification that the test result is empty

 tests/TestSuite_meson_tests.py | 233 +++++++++++++++++++++++++++++++++
 1 file changed, 233 insertions(+)
 create mode 100644 tests/TestSuite_meson_tests.py

diff --git a/tests/TestSuite_meson_tests.py b/tests/TestSuite_meson_tests.py
new file mode 100644
index 00000000..4b1f7df2
--- /dev/null
+++ b/tests/TestSuite_meson_tests.py
@@ -0,0 +1,233 @@
+# BSD LICENSE
+#
+# Copyright(c) 2022 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
+DPDK Test Suite.
+
+meson autotest
+"""
+import os
+import re
+from mimetypes import init
+from sre_constants import SUCCESS
+
+import framework.utils as utils
+from framework.test_case import TestCase
+
+
+class TestMesonTests(TestCase):
+    def set_up_all(self):
+        """
+        Run at the start of each test suite.
+        """
+        # icc compilation cost long long time.
+        self.cores = self.dut.get_core_list("all")
+        self.dut_ip = self.dut.get_ip_address()
+        self.timeout = 600
+        # -t 2
+        self.ratio = 6
+        self.dut_pathlog = "fast-test.log"
+        self.execute_wait_time = self.ratio * self.timeout * 10
+        # skip scope
+        self.SKIP_SCOPE = ""
+        # Test log storage directory
+        self.base_output = os.path.join(
+            os.path.dirname(os.path.dirname(os.path.realpath(__file__))), "output"
+        )
+
+    def set_up(self):
+        """
+        Run before each test case.
+        """
+        self.meson_auto_test = {}
+
+    def delete_exists_files(self):
+        out = self.tester.send_command(
+            f"ls -l {self.base_output}/{self.dut_pathlog}", "# "
+        )
+        if self.dut_pathlog in out:
+            self.tester.send_command(f"rm -rf {self.base_output}/{self.dut_pathlog}")
+
+    def find_text(self, res):
+        pattern = r"(DPDK:fast-tests|DPDK:perf-tests|DPDK:debug-tests|DPDK:extra-tests|DPDK:driver-tests) / (\w+)\s+(\w+)\s+"
+        regex = re.compile(pattern)
+        mo = regex.search(res)
+
+        if mo is None:
+            return 0
+        case_name = mo.group(2)
+        test_result = mo.group(3)
+
+        self.meson_auto_test[case_name] = test_result
+
+    def check_sub_case(self):
+        success = []
+        skip = []
+        timeout = []
+        failed = []
+
+        self.verify(
+            bool(self.meson_auto_test),
+            f"Test failed the meson no test results were obtained.",
+        )
+
+        for k, v in self.meson_auto_test.items():
+            if v == "FAIL" and k not in self.SKIP_SCOPE:
+                failed.append(k)
+            elif v == "OK":
+                success.append(k)
+            elif v == "SKIP":
+                skip.append(k)
+            elif v == "TIMEOUT":
+                timeout.append(k)
+            else:
+                failed.append(k)
+
+        for item in skip:
+            self.logger.debug(utils.RED(f"Skip sub case:{item}"))
+
+        for item in timeout:
+            self.logger.debug(utils.RED(f"TimeOut sub case:{item}"))
+
+        if (len(failed) + len(timeout)) > 0:
+            sub_fail = ""
+            sub_timeout = ""
+            if len(failed) > 0:
+                sub_fail = "[" + ",".join(failed) + "] sub case failed. "
+            if len(timeout) > 0:
+                sub_timeout = "[" + ",".join(timeout) + "] sub case time out. "
+            self.verify(False, f"Test failed. {sub_fail}{sub_timeout}")
+
+    def check_scp_file_valid_between_dut(self):
+        out = self.tester.send_command(
+            f"scp root@{self.dut_ip}:/root/{self.dut_pathlog} {self.base_output}",
+            timeout=15,
+        )
+        if "Are you sure you want to continue connecting" in out:
+            out = self.tester.send_command("yes", timeout=20)
+        for item in range(30):
+            if "password" in out:
+                self.tester.send_command(self.dut.get_password(), timeout=20)
+                break
+
+        out = self.tester.send_command(
+            f"ls -l {self.base_output}/{self.dut_pathlog}", "# "
+        )
+        self.verify(
+            "No such file or directory" not in out, "No test result log was obtained!"
+        )
+
+    def insmod_kni(self):
+        out = self.dut.send_expect("lsmod | grep rte_kni", "# ")
+
+        if "rte_kni" in out:
+            self.dut.send_expect("rmmod rte_kni.ko", "# ")
+
+        out = self.dut.send_expect(
+            "insmod ./%s/kmod/rte_kni.ko lo_mode=lo_mode_fifo" % (self.target), "# "
+        )
+        self.verify("Error" not in out, "Error loading KNI module: " + out)
+
+    def check_meson_test_result(self, res=None):
+        if not os.path.exists(f"{self.base_output}/{self.dut_pathlog}"):
+            self.verify(False, "meson tests log file is not exists!!")
+        if res is None:
+            with open(f"{self.base_output}/{self.dut_pathlog}", "r") as file_obj:
+                for files in file_obj:
+                    self.find_text(files)
+        else:
+            self.find_text(res)
+        self.check_sub_case()
+
+    def test_fasts(self):
+        # init file name
+        self.dut_pathlog = "fast-test.log"
+        self.delete_exists_files()
+        self.insmod_kni()
+        cmds = f'meson test -C x86_64-native-linuxapp-gcc/ --suite DPDK:fast-tests -t {self.ratio} --test-args="-c 0xff" |tee /root/{self.dut_pathlog}'
+        out = self.dut.send_expect(cmds, "# ", self.execute_wait_time)
+        self.logger.info(out)
+        self.check_scp_file_valid_between_dut()
+        self.check_meson_test_result()
+
+    def test_driver(self):
+        # init file name
+        self.dut_pathlog = "driver-test.log"
+        self.delete_exists_files()
+        cmds = f'meson test -C x86_64-native-linuxapp-gcc/ --suite DPDK:driver-tests -t {self.ratio} --test-args="-c 0xff" |tee /root/{self.dut_pathlog}'
+        out = self.dut.send_expect(cmds, "# ", self.execute_wait_time)
+        self.logger.info(out)
+        self.check_scp_file_valid_between_dut()
+        self.check_meson_test_result()
+
+    def test_debug(self):
+        self.logger.warning(
+            "Debug-tests all case failed commands are not registered, there is a bug https://bugs.dpdk.org/show_bug.cgi?id=1002"
+        )
+
+    def test_extra(self):
+        self.dut_pathlog = "extra-test.log"
+        # delete exists files
+        self.delete_exists_files()
+        cmds = f'meson test -C x86_64-native-linuxapp-gcc/ --suite DPDK:extra-tests -t {self.ratio} --test-args="-c 0xff" |tee /root/{self.dut_pathlog}'
+        out = self.dut.send_expect(cmds, "# ", self.execute_wait_time)
+        self.logger.info(out)
+        self.check_scp_file_valid_between_dut()
+        self.check_meson_test_result()
+        self.logger.warning(
+            "Extra-tests are know issues which are recorded in DPDK commit and meson.build (detail see test plan)"
+        )
+
+    def test_perf(self):
+        # init file name
+        self.dut_pathlog = "perf-test.log"
+        # delete exists files
+        self.delete_exists_files()
+        cmds = f'meson test -C x86_64-native-linuxapp-gcc/ --suite DPDK:perf-tests -t {self.ratio} --test-args="-c 0xff" |tee /root/{self.dut_pathlog}'
+        out = self.dut.send_expect(cmds, "# ", self.execute_wait_time)
+        self.logger.info(out)
+        self.check_scp_file_valid_between_dut()
+        self.check_meson_test_result()
+
+    def tear_down(self):
+        """
+        Run after each test case.
+        """
+        out = self.dut.send_expect("lsmod | grep rte_kni", "# ")
+
+        if "rte_kni" in out:
+            self.dut.send_expect("rmmod rte_kni.ko", "# ")
+
+    def tear_down_all(self):
+        """
+        Run after each test suite.
+        """
+        pass
-- 
2.17.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan for meson tests
  2022-05-26  9:19 [dts][PATCH V3 1/2] tests/meson_tests: add new testsuite for meson tests Li, WeiyuanX
@ 2022-05-26  9:19 ` Li, WeiyuanX
  2022-05-26 10:53   ` Chen, LingliX
                     ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Li, WeiyuanX @ 2022-05-26  9:19 UTC (permalink / raw)
  To: dts, lijuan.tu; +Cc: Weiyuan Li

From: Weiyuan Li <weiyuanx.li@intel.com>

Add new test plan for meson tests.

Signed-off-by: Weiyuan Li <weiyuanx.li@intel.com>
---

v2:
-fix format issue.
v3:
-add verification that the test result is empty

 test_plans/index.rst                 |   1 +
 test_plans/meson_tests_test_plan.rst | 185 +++++++++++++++++++++++++++
 2 files changed, 186 insertions(+)
 create mode 100644 test_plans/meson_tests_test_plan.rst

diff --git a/test_plans/index.rst b/test_plans/index.rst
index 6978a2ef..45a23e7e 100644
--- a/test_plans/index.rst
+++ b/test_plans/index.rst
@@ -281,6 +281,7 @@ The following are the test plans for the DPDK DTS automated test system.
     unit_tests_ringpmd_test_plan
     unit_tests_ring_test_plan
     unit_tests_timer_test_plan
+    meson_tests_test_plan
 
     cmdline_test_plan
     hello_world_test_plan
diff --git a/test_plans/meson_tests_test_plan.rst b/test_plans/meson_tests_test_plan.rst
new file mode 100644
index 00000000..e199ec81
--- /dev/null
+++ b/test_plans/meson_tests_test_plan.rst
@@ -0,0 +1,185 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+   Copyright (c) <2022>, Intel Corporation
+
+===========
+Meson tests
+===========
+
+Description
+===========
+This test plan describes how to run unit tests with meson.
+Test tests have been classified into five different groups.
+
+- Fast tests.
+- Performance tests.
+- Driver tests.
+- Extra tests.
+- Debug tests.
+
+For more details, please refer to `Running DPDK Unit Tests with Meson <http://doc.dpdk.org/guides/prog_guide/meson_ut.html>`_.
+
+Prerequisites
+=============
+
+Required Libraries
+------------------
+* crypto relative cases require dependencies `libIPSec_MB` and `libcrypto`::
+
+     cryptodev_aesni_mb_autotest
+     cryptodev_aesni_gcm_autotest
+     cryptodev_null_autotest
+     cryptodev_openssl_autotest
+     cryptodev_openssl_asym_autotest
+     cryptodev_qat_autotest
+     cryptodev_sw_kasumi_autotest
+     cryptodev_sw_snow3g_autotest
+     cryptodev_sw_zuc_autotest
+     cryptodev_scheduler_autotest
+
+  .. note::
+
+     For more details, please refer to `Crypto Device Drivers <http://doc.dpdk.org/guides/cryptodevs/index.html>`_
+
+* The following cases require dependencies `libpcap`::
+
+   bpf_convert_autotest
+   eal_flags_mem_autotest
+   flow_classify_autotest
+   efd_autotest
+   efd_perf_autotest
+
+General set up
+--------------
+
+* Compile DPDK::
+
+    # CC=gcc meson --werror -Denable_kmods=True -Dlibdir=lib --default-library=static <dpdk build dir>
+    # ninja -C <dpdk build dir> -j 110
+
+* Load rte_kni driver::
+
+    # lsmod | grep rte_kni
+    # rmmod rte_kni.ko
+    # insmod ./<dpdk build dir>/kmod/rte_kni.ko lo_mode=lo_mode_fifo
+
+   .. note::
+
+      Test case fast-tests/kni_autotest requires to load kni.ko.
+
+* Get the pci device id and interface of DUT and tester.
+   For example, 0000:18:00.0 and 0000:18:00.1 is pci device id,
+   ens785f0 and ens785f1 is interface::
+
+    <dpdk dir># ./usertools/dpdk-devbind.py -s
+	...
+    0000:18:00.0 'Device 159b' if=ens785f0 drv=ice unused=vfio-pci
+    0000:18:00.1 'Device 159b' if=ens785f1 drv=ice unused=vfio-pci
+
+* Bind the DUT port to dpdk::
+
+    <dpdk dir># ./usertools/dpdk-devbind.py -b vfio-pci 0000:18:00.0 0000:18:00.1
+
+.. note::
+
+   kni_autotest requires at least one ethernet port, but other can be none.
+
+Test case
+=========
+
+Test case 1: test_fasts
+-------------------------------
+This section explains how to run the meson tests for fast-tests.
+
+test steps
+~~~~~~~~~~
+1. Run test on DUT::
+
+    meson test -C x86_64-native-linuxapp-gcc --suite DPDK:fast-tests -t 1 --test-args="-c 0xff"
+
+   * `-t` is used to specify the timeout of test case, unit is second.
+   * `--test-args option` is used to specify test specific parameters,
+     distributor_autotest and distributor_perf_autotest require core number less than 64.
+
+2. Open and view the results on DUT::
+
+    <dpdk dir># cat ./x86_64-native-linuxapp-gcc/meson-logs/testlog.txt
+
+3. Check the result output log. All case results are "OK", for example::
+
+    1/99 DPDK:fast-tests / acl_autotest                   OK              3.43s
+    2/99 DPDK:fast-tests / atomic_autotest                OK              6.21s
+    3/99 DPDK:fast-tests / bitmap_autotest                OK              1.18s
+    4/99 DPDK:fast-tests / bpf_autotest                   OK              1.17s
+    5/99 DPDK:fast-tests / bpf_convert_autotest           OK              1.19s
+
+Test case 2: test_driver
+-------------------------------
+This section explains how to run the meson tests for driver-tests.
+
+test steps
+~~~~~~~~~~
+1. Run test on DUT::
+
+    meson test -C x86_64-native-linuxapp-gcc --suite DPDK:driver-tests -t 1 --test-args="-c 0xff"
+
+2. Following test case 1 step 2 and step 3.
+
+Test case 3: test_debug
+-------------------------------
+This section explains how to run the meson tests for debug-tests.
+
+test steps
+~~~~~~~~~~
+1. Run test on DUT::
+
+    meson test -C x86_64-native-linuxapp-gcc --suite DPDK:debug-tests -t 1 --test-args="-c 0xff"
+
+2.Following test case 1 step 2 and step 3.
+
+.. warning::
+
+   `The bug <https://bugs.dpdk.org/show_bug.cgi?id=1002>_` impacts the following cases::
+
+      DPDK:debug-tests/dump_struct_sizes
+      DPDK:debug-tests/dump_mempool
+      DPDK:debug-tests/dump_malloc_stats
+      DPDK:debug-tests/dump_devargs
+      DPDK:debug-tests/dump_log_types
+      DPDK:debug-tests/dump_ring
+      DPDK:debug-tests/dump_physmem
+      DPDK:debug-tests/dump_memzone
+
+Test case 4: test_extra
+-------------------------------
+This section explains how to run the meson tests for extra-tests.
+
+test steps
+~~~~~~~~~~
+1. Run test on DUT::
+
+    meson test -C x86_64-native-linuxapp-gcc --suite DPDK:extra-tests -t 1 --test-args="-c 0xff"
+
+2. Following test case 1 step 2 and step 3.
+
+.. warning::
+
+   Extra-tests are know issues which are recorded in app/test/meson.build::
+
+    #Tests known to have issues or which don't belong in other tests lists.
+    extra_test_names = [
+            'alarm_autotest', # ee00af60170b ("test: remove strict timing requirements some tests")
+            'cycles_autotest', # ee00af60170b ("test: remove strict timing requirements some tests")
+            'delay_us_sleep_autotest', # ee00af60170b ("test: remove strict timing requirements some tests")
+            'red_autotest', # https://bugs.dpdk.org/show_bug.cgi?id=826]
+
+Test case 5: test_perf
+-------------------------------
+This section explains how to run the meson tests for perf-tests.
+
+test steps
+~~~~~~~~~~
+1. Run test on DUT::
+
+    meson test -C x86_64-native-linuxapp-gcc --suite DPDK:perf-tests -t 1 --test-args="-c 0xff"
+
+2. Following test case 1 step 2 and step 3.
\ No newline at end of file
-- 
2.17.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

* RE: [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan for meson tests
  2022-05-26  9:19 ` [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan " Li, WeiyuanX
@ 2022-05-26 10:53   ` Chen, LingliX
  2022-05-28  8:57   ` lijuan.tu
  2022-05-28  8:59   ` lijuan.tu
  2 siblings, 0 replies; 5+ messages in thread
From: Chen, LingliX @ 2022-05-26 10:53 UTC (permalink / raw)
  To: dts; +Cc: Li, WeiyuanX

> -----Original Message-----
> From: Li, WeiyuanX <weiyuanx.li@intel.com>
> Sent: Thursday, May 26, 2022 5:20 PM
> To: dts@dpdk.org; Tu, Lijuan <lijuan.tu@intel.com>
> Cc: Li, WeiyuanX <weiyuanx.li@intel.com>
> Subject: [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan for meson
> tests
> 
> From: Weiyuan Li <weiyuanx.li@intel.com>
> 
> Add new test plan for meson tests.
> 
> Signed-off-by: Weiyuan Li <weiyuanx.li@intel.com>
> ---
> 
> v2:
> -fix format issue.
> v3:
> -add verification that the test result is empty
> 

Tested-by: Lingli Chen <linglix.chen@intel.com>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan for meson tests
  2022-05-26  9:19 ` [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan " Li, WeiyuanX
  2022-05-26 10:53   ` Chen, LingliX
@ 2022-05-28  8:57   ` lijuan.tu
  2022-05-28  8:59   ` lijuan.tu
  2 siblings, 0 replies; 5+ messages in thread
From: lijuan.tu @ 2022-05-28  8:57 UTC (permalink / raw)
  To: dts, lijuan.tu, Li, WeiyuanX; +Cc: Weiyuan Li

On Thu, 26 May 2022 17:19:36 +0800, "Li, WeiyuanX" <weiyuanx.li@intel.com> wrote:
> From: Weiyuan Li <weiyuanx.li@intel.com>
> 
> Add new test plan for meson tests.
> 
> Signed-off-by: Weiyuan Li <weiyuanx.li@intel.com>


Series applied, thanks

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan for meson tests
  2022-05-26  9:19 ` [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan " Li, WeiyuanX
  2022-05-26 10:53   ` Chen, LingliX
  2022-05-28  8:57   ` lijuan.tu
@ 2022-05-28  8:59   ` lijuan.tu
  2 siblings, 0 replies; 5+ messages in thread
From: lijuan.tu @ 2022-05-28  8:59 UTC (permalink / raw)
  To: dts, lijuan.tu, Li, WeiyuanX; +Cc: Weiyuan Li

On Thu, 26 May 2022 17:19:36 +0800, "Li, WeiyuanX" <weiyuanx.li@intel.com> wrote:
> From: Weiyuan Li <weiyuanx.li@intel.com>
> 
> Add new test plan for meson tests.
> 
> Signed-off-by: Weiyuan Li <weiyuanx.li@intel.com>

Reviewed-by: Lijuan Tu <lijuan.tu@intel.com>
Series applied, thanks

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2022-05-28  8:59 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-26  9:19 [dts][PATCH V3 1/2] tests/meson_tests: add new testsuite for meson tests Li, WeiyuanX
2022-05-26  9:19 ` [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan " Li, WeiyuanX
2022-05-26 10:53   ` Chen, LingliX
2022-05-28  8:57   ` lijuan.tu
2022-05-28  8:59   ` lijuan.tu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).