[RFC PATCH v1 0/3] dts: pf_smoke port

[RFC PATCH v1 0/3] dts: pf_smoke port

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

This series ports the functionality of the pf_msoke test sutie from old
DTS to the new framework. It is listed as an RFC mainly due to the fact
that is uses different verification steps than most other test suites
by utilizing checksums to differentiate packets sent by the framework
and ones that are just noise on the wire. It should be noted however
that this will not work as expected on mlx5 due to the following
bugzilla ticket:

https://bugs.dpdk.org/show_bug.cgi?id=1514

Jeremy Spewock (3):
  dts: add ability to modify number of queues on a port to testpmd
  dts: add pf smoke testing suite
  dts: added pf_smoke_tests to yaml schema

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py |  36 +++++
 dts/tests/TestSuite_pf_smoke_tests.py         | 129 ++++++++++++++++++
 3 files changed, 167 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_pf_smoke_tests.py

-- 
2.45.2

[RFC PATCH v1 1/3] dts: add ability to modify number of queues on a port to testpmd

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

The ability to change the configuration of a port at runtime is a
crucial aspect of DPDK. This patch adds both the steps required to
modify the number of queues on a port at runtime and also the
verification steps to ensure that the command behaved as expected.

Depends-on: patch-142762 ("dts: add text parser for testpmd verbose
 output")
Depends-on: patch-142696 ("dts: add VLAN methods to testpmd shell")

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 36 +++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index 6bde7f536f..6eb6360bf7 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -1191,6 +1191,42 @@ def set_verbose(self, level: int, verify: bool = True) -> None:
                     f"Testpmd failed to set verbose level to {level}."
                 )
 
+    def set_num_queues_all(self, num_queues: int, is_rx: bool, verify: bool = True) -> None:
+        """Modify the number of Rx/Tx queues configured on all ports.
+
+        Args:
+            num_queues: Number of queues to set on all ports.
+            is_rx: If :data:`True` then the number of Rx queues will be modified, otherwise the
+                number of Tx queues will be modified.
+            verify: If :data:`True` then an additional command will be sent to check the info of
+                `port_id` and verify that the number of queues is equal to `num_queues`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and testpmd failed to
+                update the number of queues on the ports.
+        """
+        queue_type = "rxq" if is_rx else "txq"
+        self.port_stop_all(verify=verify)
+        port_config_output = self.send_command(f"port config all {queue_type} {num_queues}")
+        # ports have to be started before the output can be verified.
+        self.port_start_all(verify=verify)
+        if verify:
+            all_ports_modified = all(
+                queues == num_queues
+                for queues in map(
+                    lambda info: info.rx_queues_num if is_rx else info.tx_queues_num,
+                    self.show_port_info_all(),
+                )
+            )
+            if not all_ports_modified:
+                self._logger.debug(
+                    f"Failed to set number of queues on all ports to "
+                    f"{num_queues}:\n{port_config_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    "Testpmd failed to update the number of queues on all ports."
+                )
+
     def _close(self) -> None:
         """Overrides :meth:`~.interactive_shell.close`."""
         self.stop()
-- 
2.45.2

[RFC PATCH v1 2/3] dts: add pf smoke testing suite

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

This patch adds a smoke testing suite for Physical Function features.
The goal of this suite is to test some of the most basic features of
DPDK on a physical function and bail out early if any of these features
aren't supported as expected. Unlike DTS smoke tests, these ones are not
included as a switch in the config file and thus are an additional test
suite that developers can include alongside others at their own
discretion.

Depends-on: patch-142691 ("dts: add send_packets to test suites and
 rework packet addressing")

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
 dts/tests/TestSuite_pf_smoke_tests.py | 129 ++++++++++++++++++++++++++
 1 file changed, 129 insertions(+)
 create mode 100644 dts/tests/TestSuite_pf_smoke_tests.py

diff --git a/dts/tests/TestSuite_pf_smoke_tests.py b/dts/tests/TestSuite_pf_smoke_tests.py
new file mode 100644
index 0000000000..82c84c7c8d
--- /dev/null
+++ b/dts/tests/TestSuite_pf_smoke_tests.py
@@ -0,0 +1,129 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 University of New Hampshire
+"""Physical Function (PF) smoke testing suite.
+
+This test suite tests some of the more common DPDK functionality on a PF. Things such as
+jumbroframes, Receive Side Scaling (RSS) functions, and being able to modify the number of queues
+at runtime should all be supported by PMDs that are capable of running DPDK. Since this is a smoke
+testing suite, it is considered a blocking suite that will stop following ones from running.
+"""
+
+from typing import ClassVar
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.exception import InteractiveCommandExecutionError, TestCaseVerifyError
+from framework.params.testpmd import SimpleForwardingModes
+from framework.remote_session.testpmd_shell import TestPmdShell, VerboseOLFlag
+from framework.test_suite import TestSuite
+
+
+class TestPfSmokeTests(TestSuite):
+    """DPDK Physical Function Testing Suite.
+
+    This test suite is designed to verify the basic functions of DPDK on a PF. The MTU of the ports
+    on the traffic generator are increased to 9000 to support jumboframes for one of the test
+    cases, and then reverted back to 1500 once the test suite is complete. Some functionality in
+    this test suite also relies on the ability of testpmd to recognize and flag invalid checksum
+    values in its verbose output.
+
+    Attributes:
+        is_blocking: This test suite will block the execution of all other test suites
+            in the build target after it.
+    """
+
+    is_blocking: ClassVar[bool] = True
+    jumbo_frame_len: ClassVar[int] = 9000
+    num_queues: int = 4
+    rx_port: int = 0
+
+    def set_up_suite(self) -> None:
+        """Increase the MTU of the traffic generator to support jumboframes."""
+        for port_link in self._port_links:
+            self.tg_node.main_session.configure_port_mtu(self.jumbo_frame_len, port_link.tg_port)
+
+    def test_jumbo_frame_support(self) -> None:
+        """Verify that the PF is able to send and receive jumboframes."""
+        with TestPmdShell(
+            self.sut_node,
+            max_pkt_len=self.jumbo_frame_len,
+            mbuf_size=[self.jumbo_frame_len + 128],
+            forward_mode=SimpleForwardingModes.mac,
+        ) as testpmd:
+            testpmd.start()
+            # Take 26 bytes off the MTU size to account for Ethernet headers
+            payload_len = self.jumbo_frame_len - 26
+            packet = Ether() / Raw("X" * payload_len)
+            recv = self.send_packet_and_capture(packet)
+            self.verify(
+                any(hasattr(p, "load") and "X" * 20 in str(p.load) for p in recv),
+                f"Jumboframe was not received even when MTU was set to {self.jumbo_frame_len}.",
+            )
+
+    def test_rss_functionality(self) -> None:
+        """Test that Receive Side Scaling functions are working as intended.
+
+        The primary things to test in this case are that packets that are sent with different
+        destination IP addresses are handled by different queues and that the RSS hash of every
+        packet is unique. Verification of these functionalities is done by sending packets with
+        invalid checksums so that the packets sent by this test suite can be differentiated from
+        other packets sent to the same port. This makes the assumption that other packets sent to
+        the port will all have valid checksums.
+        """
+        with TestPmdShell(
+            self.sut_node,
+            forward_mode=SimpleForwardingModes.rxonly,
+            rx_queues=self.num_queues,
+            tx_queues=self.num_queues,
+        ) as testpmd:
+            testpmd.set_verbose(1)
+            send_pkts = [
+                Ether() / IP(dst=f"192.168.0.{i+1}", chksum=0x0) for i in range(self.num_queues * 4)
+            ]
+            testpmd.start()
+            self.send_packets(send_pkts)
+            verbose_stats = TestPmdShell.extract_verbose_output(testpmd.stop())
+            # Filter down the packets to only the ones with back checksums
+            for block in verbose_stats:
+                block.packets = list(
+                    filter(
+                        lambda x: VerboseOLFlag.RTE_MBUF_F_RX_IP_CKSUM_BAD in x.ol_flags,
+                        block.packets,
+                    )
+                )
+            # Remove any output blocks that don't have any packets left after filtering
+            verbose_stats = list(filter(lambda x: len(x.packets) > 0, verbose_stats))
+            rss_hashes = [p.rss_hash for block in verbose_stats for p in block.packets]
+            self.verify(
+                all(rss_h is not None for rss_h in rss_hashes),
+                "At least one packet did not have an RSS hash.",
+            )
+            self.verify(
+                len(set(rss_hashes)) == len(rss_hashes),
+                "RSS hashes were not unique.",
+            )
+            self.verify(
+                all(any(q == b.queue_id for b in verbose_stats) for q in range(self.num_queues)),
+                "Not all ports were used when packets were sent with different source addresses.",
+            )
+
+    def test_runtime_modify_num_queues(self) -> None:
+        """Ensure that the number of queues on a port can be changed at runtime."""
+        with TestPmdShell(
+            self.sut_node, rx_queues=self.num_queues, tx_queues=self.num_queues
+        ) as testpmd:
+            testpmd.port_stop_all()
+            try:
+                testpmd.set_num_queues_all(2, True, verify=True)
+                testpmd.set_num_queues_all(2, False, verify=True)
+            except InteractiveCommandExecutionError as e:
+                raise TestCaseVerifyError(
+                    "Failed to change the number of queues on a port at runtime."
+                ) from e
+
+    def tear_down_suite(self) -> None:
+        """Revert MTU back to a standard value of 1500."""
+        for port_link in self._port_links:
+            self.tg_node.main_session.configure_port_mtu(1500, port_link.tg_port)
-- 
2.45.2

[RFC PATCH v1 3/3] dts: added pf_smoke_tests to yaml schema

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

Add the PF smoke testing suite to the yaml schema so that it can be
specified in conf.yaml.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..910134f9e4 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "pf_smoke_tests"
       ]
     },
     "test_target": {
-- 
2.45.2

[PATCH v2 0/2] dts: pf_smoke port

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

v2:
 * update dependencies and how the test suite uses them
 * use a unique src MAC on packets to find them in verbose output
   rather than checksums since using checksums isn't supported on all
   NICs
 * formatting fixes and doc-string updates

Jeremy Spewock (2):
  dts: add ability to modify number of queues on a port to testpmd
  dts: add pf smoke testing suite

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py |  36 ++++++
 dts/tests/TestSuite_pf_smoke_tests.py         | 121 ++++++++++++++++++
 3 files changed, 159 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_pf_smoke_tests.py

-- 
2.46.0

[PATCH v2 1/2] dts: add ability to modify number of queues on a port to testpmd

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

The ability to change the configuration of a port at runtime is a
crucial aspect of DPDK. This patch adds both the steps required to
modify the number of queues on a port at runtime and also the
verification steps to ensure that the command behaved as expected.

Depends-on: patch-143033 ("dts: add text parser for testpmd verbose
 output")
Depends-on: patch-143385 ("dts: add VLAN methods to testpmd shell")

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 36 +++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index bac5791a00..586a48fe87 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -1450,6 +1450,42 @@ def set_verbose(self, level: int, verify: bool = True) -> None:
                     f"Testpmd failed to set verbose level to {level}."
                 )
 
+    def set_num_queues_all(self, num_queues: int, is_rx: bool, verify: bool = True) -> None:
+        """Modify the number of Rx/Tx queues configured on all ports.
+
+        Args:
+            num_queues: Number of queues to set on all ports.
+            is_rx: If :data:`True` then the number of Rx queues will be modified, otherwise the
+                number of Tx queues will be modified.
+            verify: If :data:`True` then an additional command will be sent to check the info of
+                `port_id` and verify that the number of queues is equal to `num_queues`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and testpmd failed to
+                update the number of queues on the ports.
+        """
+        queue_type = "rxq" if is_rx else "txq"
+        self.port_stop_all(verify=verify)
+        port_config_output = self.send_command(f"port config all {queue_type} {num_queues}")
+        # ports have to be started before the output can be verified.
+        self.port_start_all(verify=verify)
+        if verify:
+            all_ports_modified = all(
+                queues == num_queues
+                for queues in map(
+                    lambda info: info.rx_queues_num if is_rx else info.tx_queues_num,
+                    self.show_port_info_all(),
+                )
+            )
+            if not all_ports_modified:
+                self._logger.debug(
+                    f"Failed to set number of queues on all ports to "
+                    f"{num_queues}:\n{port_config_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    "Testpmd failed to update the number of queues on all ports."
+                )
+
     def _close(self) -> None:
         """Overrides :meth:`~.interactive_shell.close`."""
         self.stop()
-- 
2.46.0

[PATCH v2 2/2] dts: add pf smoke testing suite

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

This patch adds a smoke testing suite for Physical Function features.
The goal of this suite is to test some of the most basic features of
DPDK on a physical function and bail out early if any of these features
aren't supported as expected. Unlike DTS smoke tests, these ones are not
included as a switch in the config file and thus are an additional test
suite that developers can include alongside others at their own
discretion.

Depends-on: patch-143594 ("dts: add send_packets to test suites and
 rework packet addressing")

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_pf_smoke_tests.py      | 121 +++++++++++++++++++++
 2 files changed, 123 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_pf_smoke_tests.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..910134f9e4 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "pf_smoke_tests"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_pf_smoke_tests.py b/dts/tests/TestSuite_pf_smoke_tests.py
new file mode 100644
index 0000000000..287132e9dd
--- /dev/null
+++ b/dts/tests/TestSuite_pf_smoke_tests.py
@@ -0,0 +1,121 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 University of New Hampshire
+"""Physical Function (PF) smoke testing suite.
+
+This test suite tests some of the more common DPDK functionality on a PF. Things such as
+jumbroframes, Receive Side Scaling (RSS) functions, and being able to modify the number of queues
+at runtime should all be supported by PMDs that are capable of running DPDK. Since this is a smoke
+testing suite, it is considered a blocking suite that will stop following ones from running.
+"""
+
+from typing import ClassVar
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.exception import InteractiveCommandExecutionError, TestCaseVerifyError
+from framework.params.testpmd import SimpleForwardingModes
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+
+class TestPfSmokeTests(TestSuite):
+    """DPDK Physical Function Testing Suite.
+
+    This test suite is designed to verify the basic functions of DPDK on a PF. The MTU of the ports
+    on the traffic generator are increased to 9000 to support jumboframes for one of the test
+    cases, and then reverted back to 1500 once the test suite is complete.
+
+    Attributes:
+        is_blocking: This test suite will block the execution of all other test suites
+            in the build target after it.
+    """
+
+    is_blocking: ClassVar[bool] = True
+    jumbo_frame_len: ClassVar[int] = 9000
+    num_queues: int = 4
+
+    def set_up_suite(self) -> None:
+        """Increase the MTU of the traffic generator to support jumboframes."""
+        for port_link in self._port_links:
+            self.tg_node.main_session.configure_port_mtu(self.jumbo_frame_len, port_link.tg_port)
+
+    def test_jumbo_frame_support(self) -> None:
+        """Verify that the PF is able to send and receive jumboframes."""
+        with TestPmdShell(
+            self.sut_node,
+            max_pkt_len=self.jumbo_frame_len,
+            mbuf_size=[self.jumbo_frame_len + 128],
+            forward_mode=SimpleForwardingModes.mac,
+        ) as testpmd:
+            testpmd.start()
+            # Take 26 bytes off the MTU size to account for Ethernet headers
+            payload_len = self.jumbo_frame_len - 26
+            packet = Ether() / Raw("X" * payload_len)
+            recv = self.send_packet_and_capture(packet)
+            self.verify(
+                any(hasattr(p, "load") and "X" * 20 in str(p.load) for p in recv),
+                f"Jumboframe was not received even when MTU was set to {self.jumbo_frame_len}.",
+            )
+
+    def test_rss_functionality(self) -> None:
+        """Test that Receive Side Scaling functions are working as intended.
+
+        The primary things to test in this case are that packets that are sent with different
+        destination IP addresses are handled by different queues and that the RSS hash of every
+        packet is unique. Verification of these functionalities is done by sending packets with
+        a unique source MAC addresses so that the packets sent by this test suite can be
+        differentiated from other packets sent to the same port.
+        """
+        with TestPmdShell(
+            self.sut_node,
+            forward_mode=SimpleForwardingModes.rxonly,
+            rx_queues=self.num_queues,
+            tx_queues=self.num_queues,
+        ) as testpmd:
+            testpmd.set_verbose(1)
+            src_max = "00:00:00:00:00:01"
+            send_pkts = [
+                Ether(src=src_max) / IP(dst=f"192.168.0.{i+1}") for i in range(self.num_queues * 4)
+            ]
+            testpmd.start()
+            self.send_packets(send_pkts)
+            verbose_stats = TestPmdShell.extract_verbose_output(testpmd.stop())
+            # Filter down the packets to only the ones with the correct source MAC
+            verbose_stats = list(filter(lambda x: x.src_mac == src_max, verbose_stats))
+            self.verify(
+                len(verbose_stats) > 0, "Testpmd did not receive any packets from the test case."
+            )
+            rss_hashes = [p.rss_hash for p in verbose_stats]
+            self.verify(
+                all(rss_h is not None for rss_h in rss_hashes),
+                "At least one packet did not have an RSS hash.",
+            )
+            self.verify(
+                len(set(rss_hashes)) == len(rss_hashes),
+                "RSS hashes were not unique.",
+            )
+            self.verify(
+                all(any(q == p.queue_id for p in verbose_stats) for q in range(self.num_queues)),
+                "Not all ports were used when packets were sent with different destination "
+                "addresses.",
+            )
+
+    def test_runtime_modify_num_queues(self) -> None:
+        """Ensure that the number of queues on a port can be changed at runtime."""
+        with TestPmdShell(
+            self.sut_node, rx_queues=self.num_queues, tx_queues=self.num_queues
+        ) as testpmd:
+            try:
+                testpmd.set_num_queues_all(2, True, verify=True)
+                testpmd.set_num_queues_all(2, False, verify=True)
+            except InteractiveCommandExecutionError as e:
+                raise TestCaseVerifyError(
+                    "Failed to change the number of queues on a port at runtime."
+                ) from e
+
+    def tear_down_suite(self) -> None:
+        """Revert MTU back to a standard value of 1500."""
+        for port_link in self._port_links:
+            self.tg_node.main_session.configure_port_mtu(1500, port_link.tg_port)
-- 
2.46.0