[PATCH v1 0/2] dts: port over port_control testing suite

[PATCH v1 0/2] dts: port over port_control testing suite

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

This series ports over most of the test coverage provided from the
port_control testing suite in the Old DTS framework. The only
functionality that is missing is testing port functions in a VM through
QEMU and testing the support of resetting ports. Since we have no
method of handling virtual machines in the new framework and have yet
to express interest in adding it, these test cases were omitted from
this suite. Additionally the ability to reset a port is not something
that is supported by all devices and devices do not seem to expose
information regarding whether they support resetting ports through
testpmd.

Jeremy Spewock (2):
  dts: add methods for closing and restarting ports to testpmd
  dts: add port control testing suite

 dts/framework/config/conf_yaml_schema.json    |  3 +-
 dts/framework/remote_session/testpmd_shell.py | 21 +++++
 dts/tests/TestSuite_port_control.py           | 80 +++++++++++++++++++
 3 files changed, 103 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_port_control.py

-- 
2.45.2

[PATCH v1 1/2] dts: add methods for closing ports to testpmd

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

Closing ports is a standard configuration feature that is available in
testpmd but the framework lacks the ability to access this command
through the Testpmd API. This patch adds a method that performs this
action and verifies the results of sending the command to allow
developers to have more control over the state of the ports that
testpmd is aware of.

Depends-on: patch-142952 ("dts: add ability to start/stop testpmd
ports")

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 21 +++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index ca24b28070..51593c61f5 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -805,6 +805,27 @@ def start_all_ports(self, verify: bool = True) -> None:
 
         self.ports_started = True
 
+    @requires_stopped_ports
+    def close_all_ports(self, verify: bool = True) -> None:
+        """Close all ports.
+
+        Args:
+            verify: If :data:`True` the output of the close command will be scanned in an attempt
+                to verify that all ports were stopped successfully. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and at lease one port
+                failed to close.
+        """
+        port_close_output = self.send_command("port close all")
+        if verify:
+            if type(self._app_params.ports) is list:
+                num_ports = len(self._app_params.ports)
+                if not all(
+                    f"Port {p_id} is closed" in port_close_output for p_id in range(num_ports)
+                ):
+                    raise InteractiveCommandExecutionError("Ports were not closed successfully.")
+
     def show_port_info_all(self) -> list[TestPmdPort]:
         """Returns the information of all the ports.
 
-- 
2.45.2

[PATCH v1 2/2] dts: add port control testing suite

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

This patch ports over the port_control test suite from the Old DTS
framework and adapts the functionality to fit with the current testing
framework. The test suite provides validation of basic port control
functions such as starting, stopping, and closing ports. It should
be noted that this test suite is not completely 1-to-1 with the one
from Old DTS as it does exclude test cases that use QEMU for testing as
this is not something we are looking to add to the framework in the
near future. It also excludes test cases for resetting ports as this
feature is something that is not supported by all devices and does not
expose a capability regarding if it is through testpmd.

Depends-on: patch-142949 ("dts: add ability to send/receive multiple
packets")

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json |  3 +-
 dts/tests/TestSuite_port_control.py        | 80 ++++++++++++++++++++++
 2 files changed, 82 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_port_control.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..78cbd17dad 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "port_control"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_port_control.py b/dts/tests/TestSuite_port_control.py
new file mode 100644
index 0000000000..9e843512ab
--- /dev/null
+++ b/dts/tests/TestSuite_port_control.py
@@ -0,0 +1,80 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 University of New Hampshire
+"""Port Control Testing Suite.
+
+This test suite serves to show that ports within testpmd support basic configuration functions.
+Things such as starting a port, stopping a port, and closing a port should all be supported by the
+device. Additionally, after each of these configuration steps (outside of closing the port) it
+should still be possible to start the port again and verify that the port is able to forward a
+large amount of packets (1000 are sent in the test cases).
+"""
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Packet, Raw  # type: ignore[import-untyped]
+
+from framework.params.testpmd import SimpleForwardingModes
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+
+class TestPortControl(TestSuite):
+    """DPDK Port Control Testing Suite."""
+
+    def send_packets_and_verify(self) -> None:
+        """Send 1000 packets and verify that all packets were forwarded back.
+
+        Packets sent are identical and are all ethernet frames with a payload of 30 "X" characters.
+        This payload is used to differentiate noise on the wire from packets sent by this
+        framework.
+        """
+        payload = "X" * 30
+        num_pakts = 1000
+        send_p = Ether() / Raw(payload)
+        recv_pakts: list[Packet] = []
+        # The scapy sniffer can only handle a little under 200 packets per 1000 at a time, so this
+        # is limited to 100 per burst.
+        for _ in range(int(num_pakts / 100)):
+            recv_pakts += self.send_packets_and_capture([send_p] * 100)
+        recv_pakts += self.send_packets_and_capture([send_p] * (num_pakts % 100))
+        recv_pakts = [
+            p
+            for p in recv_pakts
+            if (
+                # Remove padding from the bytes.
+                hasattr(p, "load")
+                and p.load.decode("utf-8").replace("\x00", "") == payload
+            )
+        ]
+        self.verify(
+            len(recv_pakts) == num_pakts,
+            f"Received {len(recv_pakts)} packets when {num_pakts} were expected.",
+        )
+
+    def test_start_ports(self) -> None:
+        """Ensure that the port can receive traffic after explicitly being started."""
+        with TestPmdShell(self.sut_node, forward_mode=SimpleForwardingModes.mac) as testpmd:
+            testpmd.start_all_ports()
+            testpmd.start()
+            self.send_packets_and_verify()
+
+    def test_stop_ports(self) -> None:
+        """Verify that the link goes down after stopping ports.
+
+        This case also verifies that the port can be started again and properly forward traffic
+        after being stopped.
+        """
+        with TestPmdShell(self.sut_node, forward_mode=SimpleForwardingModes.mac) as testpmd:
+            testpmd.stop_all_ports()
+            self.verify(
+                all(not p.is_link_up for p in testpmd.show_port_info_all()),
+                "Failed to stop all ports.",
+            )
+            testpmd.start()
+            self.send_packets_and_verify()
+
+    def test_close_ports(self) -> None:
+        """Verify that ports can be closed and no longer appear in testpmd when they are."""
+        with TestPmdShell(self.sut_node) as testpmd:
+            testpmd.close_all_ports()
+            self.verify(
+                len(testpmd.show_port_info_all()) == 0, "Failed to close all ports in testpmd."
+            )
-- 
2.45.2

[PATCH v2 0/2] dts: port over port_control testing suite

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

v2:
 * apply to next-dts

Jeremy Spewock (2):
  dts: add method for closing ports to testpmd
  dts: add port control testing suite

 dts/framework/config/conf_yaml_schema.json    |  3 +-
 dts/framework/remote_session/testpmd_shell.py | 18 +++++
 dts/tests/TestSuite_port_control.py           | 80 +++++++++++++++++++
 3 files changed, 100 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_port_control.py

-- 
2.46.0

[PATCH v2 1/2] dts: add method for closing ports to testpmd

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

Closing ports is a standard configuration feature that is available in
testpmd but the framework lacks the ability to access this command
through the Testpmd API. This patch adds a method that performs this
action and verifies the results of sending the command to allow
developers to have more control over the state of the ports that
testpmd is aware of.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index 77902a468d..c7161df374 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -844,6 +844,24 @@ def set_ports_queues(self, number_of: int) -> None:
         self.send_command(f"port config all rxq {number_of}")
         self.send_command(f"port config all txq {number_of}")
 
+    @requires_stopped_ports
+    def close_all_ports(self, verify: bool = True) -> None:
+        """Close all ports.
+
+        Args:
+            verify: If :data:`True` the output of the close command will be scanned in an attempt
+                to verify that all ports were stopped successfully. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and at lease one port
+                failed to close.
+        """
+        port_close_output = self.send_command("port close all")
+        if verify:
+            num_ports = len(self.ports)
+            if not all(f"Port {p_id} is closed" in port_close_output for p_id in range(num_ports)):
+                raise InteractiveCommandExecutionError("Ports were not closed successfully.")
+
     def show_port_info_all(self) -> list[TestPmdPort]:
         """Returns the information of all the ports.
 
-- 
2.46.0

[PATCH v2 2/2] dts: add port control testing suite

[jspewock]

From: Jeremy Spewock <jspewock@iol.unh.edu>

This patch ports over the port_control test suite from the Old DTS
framework and adapts the functionality to fit with the current testing
framework. The test suite provides validation of basic port control
functions such as starting, stopping, and closing ports. It should
be noted that this test suite is not completely 1-to-1 with the one
from Old DTS as it does exclude test cases that use QEMU for testing as
this is not something we are looking to add to the framework in the
near future. It also excludes test cases for resetting ports as this
feature is something that is not supported by all devices and does not
expose a capability regarding if it is through testpmd.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
---

It should also be noted that this suite has issues running in some mlx5
NICs. This is due to the fact that ports are validated as "stopped"
using their link status which does not update upon stopping ports of an
mlx5 NIC. I am not sure at this time if this is intentional or just a
bug, but it doesn't seem there is a consistent way to check if a port is
stopped other than link status.

 dts/framework/config/conf_yaml_schema.json |  3 +-
 dts/tests/TestSuite_port_control.py        | 80 ++++++++++++++++++++++
 2 files changed, 82 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_port_control.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index df390e8ae2..b0d8e1f6a6 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "port_control"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_port_control.py b/dts/tests/TestSuite_port_control.py
new file mode 100644
index 0000000000..9e843512ab
--- /dev/null
+++ b/dts/tests/TestSuite_port_control.py
@@ -0,0 +1,80 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 University of New Hampshire
+"""Port Control Testing Suite.
+
+This test suite serves to show that ports within testpmd support basic configuration functions.
+Things such as starting a port, stopping a port, and closing a port should all be supported by the
+device. Additionally, after each of these configuration steps (outside of closing the port) it
+should still be possible to start the port again and verify that the port is able to forward a
+large amount of packets (1000 are sent in the test cases).
+"""
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Packet, Raw  # type: ignore[import-untyped]
+
+from framework.params.testpmd import SimpleForwardingModes
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+
+class TestPortControl(TestSuite):
+    """DPDK Port Control Testing Suite."""
+
+    def send_packets_and_verify(self) -> None:
+        """Send 1000 packets and verify that all packets were forwarded back.
+
+        Packets sent are identical and are all ethernet frames with a payload of 30 "X" characters.
+        This payload is used to differentiate noise on the wire from packets sent by this
+        framework.
+        """
+        payload = "X" * 30
+        num_pakts = 1000
+        send_p = Ether() / Raw(payload)
+        recv_pakts: list[Packet] = []
+        # The scapy sniffer can only handle a little under 200 packets per 1000 at a time, so this
+        # is limited to 100 per burst.
+        for _ in range(int(num_pakts / 100)):
+            recv_pakts += self.send_packets_and_capture([send_p] * 100)
+        recv_pakts += self.send_packets_and_capture([send_p] * (num_pakts % 100))
+        recv_pakts = [
+            p
+            for p in recv_pakts
+            if (
+                # Remove padding from the bytes.
+                hasattr(p, "load")
+                and p.load.decode("utf-8").replace("\x00", "") == payload
+            )
+        ]
+        self.verify(
+            len(recv_pakts) == num_pakts,
+            f"Received {len(recv_pakts)} packets when {num_pakts} were expected.",
+        )
+
+    def test_start_ports(self) -> None:
+        """Ensure that the port can receive traffic after explicitly being started."""
+        with TestPmdShell(self.sut_node, forward_mode=SimpleForwardingModes.mac) as testpmd:
+            testpmd.start_all_ports()
+            testpmd.start()
+            self.send_packets_and_verify()
+
+    def test_stop_ports(self) -> None:
+        """Verify that the link goes down after stopping ports.
+
+        This case also verifies that the port can be started again and properly forward traffic
+        after being stopped.
+        """
+        with TestPmdShell(self.sut_node, forward_mode=SimpleForwardingModes.mac) as testpmd:
+            testpmd.stop_all_ports()
+            self.verify(
+                all(not p.is_link_up for p in testpmd.show_port_info_all()),
+                "Failed to stop all ports.",
+            )
+            testpmd.start()
+            self.send_packets_and_verify()
+
+    def test_close_ports(self) -> None:
+        """Verify that ports can be closed and no longer appear in testpmd when they are."""
+        with TestPmdShell(self.sut_node) as testpmd:
+            testpmd.close_all_ports()
+            self.verify(
+                len(testpmd.show_port_info_all()) == 0, "Failed to close all ports in testpmd."
+            )
-- 
2.46.0