[PATCH v1 0/2] dts: mtu update and jumbo frames test suite

[PATCH v1 0/2] dts: mtu update and jumbo frames test suite

[Nicholas Pratte]

v(1):
  * refactored suite consolidating these two test suites:
      inbox.dpdk.org/dev/20241023153906.1522920-2-luca.vizzarro@arm.com/
      inbox.dpdk.org/dev/20240726141307.14410-3-npratte@iol.unh.edu/
  * Suite is 'redesigned' to assess both runtime and pre-runtime mtu
      configuration in testpmd, as these use different components of
      ethdev.
  * add some additional packet forward testing to the original mtu
      update suite to synchronize with test cases present in
      jumbo frames
  * removed or consolidated redundant test cases from original jumbo
    frames suite to mirror 'mtu_update's functional tests.

For anyone who may need better context on the current situation of mtu
and how it is understood within dpdk as an application, you may find the
following inbox archives insightful.

inbox.dpdk.org/dev/0f4866fc-b76d-cf83-75e8-86326c02814b@intel.com/
inbox.dpdk.org/dev/20211018134854.1258938-5-ferruh.yigit@intel.com/
inbox.dpdk.org/dev/e2554b78-cdda-aa33-ac6d-59a543a10640@intel.com/

Nicholas Pratte (2):
  dts: add fwd restart decorator to rx capabilities
  dts: add mtu update and jumbo frames test suite

 dts/framework/remote_session/testpmd_shell.py |  26 +-
 dts/tests/TestSuite_mtu_update_fwding.py      | 278 ++++++++++++++++++
 2 files changed, 303 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_mtu_update_fwding.py

-- 
2.47.1

[PATCH v1 1/2] dts: add fwd restart decorator to rx capabilities

[Nicholas Pratte]

Some testpmd runtime functions require that forwarding be stopped before
attempting to execute them, depending on the NIC and vendor. Adding a
decorator to these testpmdshell methods to stop, execute, and then start
forwarding again abstracts this concern away for test suite developers,
and makes for cleaner, easier to read code.

Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 26 ++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index c01ee74b21..c008cb3792 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -1390,6 +1390,28 @@ def _wrapper(self: "TestPmdShell", *args: P.args, **kwargs: P.kwargs):
     return _wrapper
 
 
+def requires_forwarding_restart(func: TestPmdShellMethod) -> TestPmdShellMethod:
+    """Decorator for :class:`TestPmdShell` commands methods that requires forwarding restart.
+
+    If the decorated method is called while a :class:`TestPmdShell` is actively forwarding, then
+    forwarding is ceased, the wrapped function is executed, and forwarding is started again.
+
+    Args:
+        func: The :class:`TestPmdShell` method to decorate.
+    """
+
+    @functools.wraps(func)
+    def _wrapper(self: "TestPmdShell", *args: P.args, **kwargs: P.kwargs):
+        if self.currently_forwarding:
+            self._logger.debug("Forwarding needs to be restarted to continue.")
+            self.stop()
+            retval = func(self, *args, **kwargs)
+            self.start()
+            return retval
+
+    return _wrapper
+
+
 def add_remove_mtu(mtu: int = 1500) -> Callable[[TestPmdShellMethod], TestPmdShellMethod]:
     """Configure MTU to `mtu` on all ports, run the decorated function, then revert.
 
@@ -1438,6 +1460,7 @@ class TestPmdShell(DPDKShell):
     _command_extra_chars: ClassVar[str] = "\n"
 
     ports_started: bool
+    currently_forwarding: bool
 
     def __init__(
         self,
@@ -1462,6 +1485,7 @@ def __init__(
             name,
         )
         self.ports_started = not self._app_params.disable_device_start
+        self.currently_forwarding = not self._app_params.auto_start
         self._ports = None
 
     @property
@@ -1836,7 +1860,7 @@ def csum_set_hw(
                                                            {port_id}:\n{csum_output}"""
                     )
 
-    @requires_started_ports
+    @requires_forwarding_restart
     @requires_stopped_ports
     def set_port_mtu(self, port_id: int, mtu: int, verify: bool = True) -> None:
         """Change the MTU of a port using testpmd.
-- 
2.47.1

[PATCH v1 2/2] dts: add mtu update and jumbo frames test suite

[Nicholas Pratte]

A functional test suite that assesses MTU updating and forwarding within
a DPDK application.

This suite consolidates the previous 'mtu_update' and 'jumbo_frames' test
suites from the old dts framework into a single, comprehensive test suite,
and it covers all of mtu the  adjustment options within the ethdev api.

Bugzilla ID: 1421

Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
Signed-off-by: Alex Chapman <alex.chapman@arm.com>
---
 dts/tests/TestSuite_mtu_update_fwding.py | 278 +++++++++++++++++++++++
 1 file changed, 278 insertions(+)
 create mode 100644 dts/tests/TestSuite_mtu_update_fwding.py

diff --git a/dts/tests/TestSuite_mtu_update_fwding.py b/dts/tests/TestSuite_mtu_update_fwding.py
new file mode 100644
index 0000000000..caafc41576
--- /dev/null
+++ b/dts/tests/TestSuite_mtu_update_fwding.py
@@ -0,0 +1,278 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+# Copyright(c) 2023-2024 University of New Hampshire
+"""MTU update and jumbo frame forwarding test suite.
+
+A suite of tests to ensures the consistency of jumbo and standard frame transmission within a DPDK
+application. If a NIC receives a packet that is greater than its assigned MTU length, then that
+packet should be dropped. Likewise, if a NIC receives a packet that is less than or equal to a
+designated MTU length, the packet should be accepted.
+
+The definition of MTU between individual vendors varies with a +/- difference of 9 bytes, at most.
+To universally test MTU functionality, and not concern over individual vendor behavior, this test
+suite compensates using a 9 byte upper and lower bound when testing a set MTU boundary.
+"""
+
+from scapy.layers.inet import IP
+from scapy.layers.l2 import Ether
+from scapy.packet import Raw
+
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite, func_test
+
+STANDARD_FRAME = 1518  # --max-pkt-len will subtract l2 information at a minimum of 18 bytes.
+JUMBO_FRAME = 9018
+
+STANDARD_MTU = 1500
+JUMBO_MTU = 9000
+
+IP_HEADER_LEN = 20
+VENDOR_AGNOSTIC_PADDING = 9  # Used as a work around for varying MTU definitions between vendors.
+
+
+class TestMtuUpdateFwding(TestSuite):
+    """DPDK PMD jumbo frames and MTU update test suite.
+
+    Assess the expected behavior of frames greater than, less then, or equal to a designated MTU
+    size in a DPDK application.
+
+    Verify the behavior of both runtime MTU and pre-runtime MTU adjustments within DPDK
+    applications. (TestPMD's CLI and runtime MTU adjustment options leverage different logical
+    in components within ethdev to set a value).
+
+    Test cases will verify that any frame greater than an assigned MTU are dropped and packets
+    less than or equal to a designated MTU are forwarded and fully intact.
+    """
+
+    def set_up_suite(self) -> None:
+        """Set up the test suite.
+
+        Setup:
+            Set traffic generator MTU lengths to a size greater than scope of all
+            test cases.
+        """
+        self.tg_node.main_session.configure_port_mtu(JUMBO_MTU + 200, self._tg_port_egress)
+        self.tg_node.main_session.configure_port_mtu(JUMBO_MTU + 200, self._tg_port_ingress)
+
+    def send_packet_and_verify(self, pkt_size: int, should_receive: bool) -> None:
+        """Generate, send a packet, and assess its behavior based on a given packet size.
+
+        Generates a packet based on a specified size and sends it to the SUT. The desired packet's
+        payload size is calculated, and a string of arbrity size, containing a single character,
+        is placed in the packet as payload. This method assesses whether or not it was forwarded,
+        depending on the test case, and does so via a check of the previously-inserted packet
+        payload.
+
+        Args:
+            pkt_size: Size of packet to be generated and sent.
+            should_receive: Indicate whether the test case expects to receive the packet or not.
+        """
+        padding = pkt_size - IP_HEADER_LEN
+        # Insert '    ' as placeholder 'CRC' error correction.
+        packet = Ether() / Raw(load="    ") / IP(len=pkt_size) / Raw(load="X" * padding)
+        received_packets = self.send_packet_and_capture(packet)
+        found = any(
+            ("X" * padding) in str(packets.load)
+            for packets in received_packets
+            if hasattr(packets, "load")
+        )
+
+        if should_receive:
+            self.verify(found, "Did not receive packet.")
+        else:
+            self.verify(not found, "Received packet.")
+
+    def assess_mtu_boundary(self, testpmd_shell: TestPmdShell, mtu: int) -> None:
+        """Sets the new MTU and verifies packets at the set boundary.
+
+        Ensure that packets smaller than or equal to a set MTU will be received and packets larger
+        will not.
+
+        First, start testpmd and update the MTU. Then ensure the new value appears
+        on port info for all ports.
+        Next, start packet capturing and send 3 different lengths of packet and verify
+        they are handled correctly.
+            # 1. VENDOR_AGNOSTIC_PADDING units smaller than the MTU specified.
+            # 2. Equal to the MTU specified.
+            # 3. VENDOR_AGNOSTIC_PADDING units larger than the MTU specified (should be fragmented).
+        Finally, stop packet capturing.
+
+        Args:
+            testpmd_shell: Active testpmd shell of a given test case.
+            mtu: New Maximum Transmission Unit to be tested.
+        """
+        # Send 3 packets of different sizes (accounting for vendor inconsistencies).
+        # 1. VENDOR_AGNOSTIC_PADDING units smaller than the MTU specified.
+        # 2. Equal to the MTU specified.
+        # 3. VENDOR_AGNOSTIC_PADDING units larger than the MTU specified.
+        smaller_frame_size: int = mtu - VENDOR_AGNOSTIC_PADDING
+        equal_frame_size: int = mtu
+        larger_frame_size: int = mtu + VENDOR_AGNOSTIC_PADDING
+
+        self.send_packet_and_verify(pkt_size=smaller_frame_size, should_receive=True)
+        self.send_packet_and_verify(pkt_size=equal_frame_size, should_receive=True)
+
+        current_mtu = testpmd_shell.show_port_info(0).mtu
+        self.verify(current_mtu is not None, "Error grabbing testpmd MTU value.")
+        if current_mtu and (
+            current_mtu >= STANDARD_MTU + VENDOR_AGNOSTIC_PADDING and mtu == STANDARD_MTU
+        ):
+            self.send_packet_and_verify(pkt_size=larger_frame_size, should_receive=True)
+        else:
+            self.send_packet_and_verify(pkt_size=larger_frame_size, should_receive=False)
+
+    @func_test
+    def test_runtime_mtu_updating_and_forwarding(self) -> None:
+        """Verify runtime MTU adjustments and assess packet forwarding behavior.
+
+        Test:
+            Start TestPMD in a paired topology.
+            Set port MTU to 1500.
+            Send packets of 1493, 1500 and 1509 bytes.
+                Verify the first two packets are forwarded and the last is dropped.
+
+            Set port MTU to 2400.
+            Send packets of 1493, 1500 and 1509 bytes.
+                Verify all three packets are forwarded.
+            Send packets of 2393, 2400 and 2409 bytes.
+                Verify the first two packets are forwarded and the last is dropped.
+
+            Set port MTU to 4800.
+            Send packets of 1493, 1500 and 1509 bytes.
+                Verify all three packets are forwarded.
+            Send packets of 4793, 4800 and 4809 bytes.
+                Verify the first two packets are forwarded and the last is dropped.
+
+            Set port MTU to 9000.
+            Send packets of 1493, 1500 and 1509 bytes.
+                Verify all three packets are forwarded.
+            Send packets of 8993, 9000 and 9009 bytes.
+                Verify the first two packets are forwarded and the last is dropped.
+        Verify:
+            Verifies the successful forwarding of packets via a search for an inserted payload.
+            If the payload is found, the packet was transmitted successfully. Otherwise, the packet
+            is considered dropped.
+
+            Verify that standard MTU packets forward, in addition to packets within the limits of
+            an MTU size set during runtime.
+        """
+        with TestPmdShell(
+            self.sut_node,
+            tx_offloads=0x8000,
+            mbuf_size=[JUMBO_MTU + 200],
+        ) as testpmd:
+            # Configure the new MTU.
+
+            # Start packet capturing.
+            testpmd.start()
+
+            testpmd.set_port_mtu_all(1500, verify=True)
+            self.assess_mtu_boundary(testpmd, 1500)
+
+            testpmd.set_port_mtu_all(2400, verify=True)
+            self.assess_mtu_boundary(testpmd, 1500)
+            self.assess_mtu_boundary(testpmd, 2400)
+
+            testpmd.set_port_mtu_all(4800, verify=True)
+            self.assess_mtu_boundary(testpmd, 1500)
+            self.assess_mtu_boundary(testpmd, 4800)
+
+            testpmd.set_port_mtu_all(9000, verify=True)
+            self.assess_mtu_boundary(testpmd, 1500)
+            self.assess_mtu_boundary(testpmd, 9000)
+
+    @func_test
+    def test_cli_mtu_forwarding_for_std_packets(self) -> None:
+        """Assesses packet forwarding of standard MTU packets after pre-runtime MTU adjustments.
+
+        Test:
+            Start TestPMD with MTU size of 1518 bytes, set pre-runtime.
+            Send packets of size 1493, 1500 and 1509 bytes.
+            Verify the first two packets are forwarded and the last is dropped.
+        Verify:
+            Verifies the successful forwarding of packets via a search for an inserted payload.
+            If the payload is found, the packet was transmitted successfully. Otherwise, the packet
+            is considered dropped.
+
+            Verify the first two packets are forwarded and the last is dropped after pre-runtime
+            MTU modification.
+        """
+        with TestPmdShell(
+            self.sut_node,
+            tx_offloads=0x8000,
+            mbuf_size=[JUMBO_MTU + 200],
+            mbcache=200,
+            max_pkt_len=STANDARD_FRAME,
+        ) as testpmd:
+            testpmd.start()
+
+            self.send_packet_and_verify(STANDARD_MTU - VENDOR_AGNOSTIC_PADDING, should_receive=True)
+            self.send_packet_and_verify(STANDARD_MTU, should_receive=True)
+            self.send_packet_and_verify(
+                STANDARD_MTU + VENDOR_AGNOSTIC_PADDING, should_receive=False
+            )
+
+    @func_test
+    def test_cli_jumbo_forwarding_for_jumbo_mtu(self) -> None:
+        """Assess packet forwarding of packets within the bounds of a pre-runtime MTU adjustment.
+
+        Test:
+            Start TestPMD with MTU size of 9018 bytes, set pre-runtime.
+            Send packets of size 8993, 9000 and 1509 bytes.
+        Verify:
+            Verifies the successful forwarding of packets via a search for an inserted payload.
+            If the payload is found, the packet was transmitted successfully. Otherwise, the packet
+            is considered dropped.
+
+            Verify that all packets are forwarded after pre-runtime MTU modification.
+        """
+        with TestPmdShell(
+            self.sut_node,
+            tx_offloads=0x8000,
+            mbuf_size=[JUMBO_MTU + 200],
+            mbcache=200,
+            max_pkt_len=JUMBO_FRAME,
+        ) as testpmd:
+            testpmd.start()
+
+            self.send_packet_and_verify(JUMBO_MTU - VENDOR_AGNOSTIC_PADDING, should_receive=True)
+            self.send_packet_and_verify(JUMBO_MTU, should_receive=True)
+            self.send_packet_and_verify(STANDARD_MTU + VENDOR_AGNOSTIC_PADDING, should_receive=True)
+
+    @func_test
+    def test_cli_mtu_std_packets_for_jumbo_mtu(self) -> None:
+        """Assess boundary of jumbo MTU value set pre-runtime.
+
+        Test:
+            Start TestPMD with MTU size of 9018 bytes, set pre-runtime.
+            Send a packets of size 8993, 9000 and 9009 bytes.
+            Verify the first two packets are forwarded and the last is dropped.
+        Verify:
+            Verifies the successful forwarding of packets via a search for an inserted payload.
+            If the payload is found, the packet was transmitted successfully. Otherwise, the packet
+            is considered dropped.
+
+            Verify the first two packets are forwarded and the last is dropped after pre-runtime
+            MTU modification.
+        """
+        with TestPmdShell(
+            self.sut_node,
+            tx_offloads=0x8000,
+            mbuf_size=[JUMBO_MTU + 200],
+            mbcache=200,
+            max_pkt_len=JUMBO_FRAME,
+        ) as testpmd:
+            testpmd.start()
+
+            self.send_packet_and_verify(JUMBO_MTU - VENDOR_AGNOSTIC_PADDING, should_receive=True)
+            self.send_packet_and_verify(JUMBO_MTU, should_receive=True)
+            self.send_packet_and_verify(JUMBO_MTU + VENDOR_AGNOSTIC_PADDING, should_receive=False)
+
+    def tear_down_suite(self) -> None:
+        """Tear down the test suite.
+
+        Teardown:
+            Set the MTU size of the traffic generator back to the standard 1518 byte size.
+        """
+        self.tg_node.main_session.configure_port_mtu(STANDARD_MTU, self._tg_port_egress)
+        self.tg_node.main_session.configure_port_mtu(STANDARD_MTU, self._tg_port_ingress)
-- 
2.47.1