[RFC PATCH 0/1] Initial Implementation For Jumbo Frames

[RFC PATCH 0/1] Initial Implementation For Jumbo Frames

[Nicholas Pratte]

The following is a rough design for the implementation of the jumbo
frames test suite in new DTS. The test suite uses the same
testing methodology from the test suite implementation in old
DTS. In doing so, much of the logic in this implementation has been
stripped away for the sake of maintaining an OS-agnostic design
philosopgy. Thus, there are some concerns here that need further
discussion.

All test cases behave accordingly on lab hardware. However, issues with
the testpmd shell, all of which will be fixed with the upcoming context
manager, prevent the test suite from functioning properly if all test
cases are run sequentially. Thus, for testing, each test case needs to
be run individually. So, expect issues if attempting to test.

Old DTS implements some basic logic that detects 1GB NICs, specifically
when testing jumbo frame packets greater than the specified MTU length.
This is because, according to a commit message from 2016, 1GB NICs will
automatically adjust their size to +4 bytes, meaning that when setting
an MTU of 9000, testpmd adjust the MTU size to 9004. To compensate, some
logic was inserted in the old suite to adjust packet sizes accordingly.
Given that Juraj's capabilities patch is in development, it would be
possible to set a restriction on this test suite to remove the support
of 1GB NICs, which could be determined from testpmd outputs; it could
also be possible that this problem is entirely null and void today,
essentially allowing us to forget about it.

The current implementation is not developed with both the capabilities
patch and the params patch in mind; however, the current design can and
will be refactored to do so.

Nicholas Pratte (1):
  Initial Implementation For Jumbo Frames Test Suite

 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_jumboframes.py         | 210 +++++++++++++++++++++
 2 files changed, 212 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_jumboframes.py

-- 
2.44.0

[RFC PATCH] Initial Implementation For Jumbo Frames Test Suite

[Nicholas Pratte]

The following test suite reflects the fundamental outline for how the
jumbo frames test suite may be designed. The test suite consists of five
individual test cases, each of which assesses the behavior of packet
transmissions for both 1518 byte and 9000 byte frames.

The edge cases are ripped directly from the old DTS framework, and the
general methodology is the same as well. The process, at this point, has
been refactored to operate within the new DTS framework.

Bugzilla ID: 1421

Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_jumboframes.py         | 210 +++++++++++++++++++++
 2 files changed, 212 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_jumboframes.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index 4731f4511d..6b9d749022 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "jumboframes"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_jumboframes.py b/dts/tests/TestSuite_jumboframes.py
new file mode 100644
index 0000000000..dd253101aa
--- /dev/null
+++ b/dts/tests/TestSuite_jumboframes.py
@@ -0,0 +1,210 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023-2024 University of New Hampshire
+"""Jumbo frame consistency and compatibility test suite.
+
+The test suite ensures the consistency of jumbo frames transmission within
+Poll Mode Drivers using a series of individual test cases. If a Poll Mode
+Driver receives a packet that is greater than its assigned MTU length, then
+that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver
+receives a packet that is less than or equal to a its designated MTU length, then the
+packet should be transmitted by the Poll Mode Driver, completeing a cycle within the
+testbed and getting received by the traffic generator. Thus, the following test suite
+evaluates the behavior within all possible edge cases, ensuring that a test Poll
+Mode Driver strictly abides by the above implications.
+"""
+
+from framework.test_suite import TestSuite
+from framework.remote_session.testpmd_shell import TestPmdShell
+
+from scapy.layers.inet import IP  # type: ignore[import]
+from scapy.layers.l2 import Ether  # type: ignore[import]
+from scapy.packet import Raw  # type: ignore[import]
+
+ETHER_HEADER_LEN = 18
+IP_HEADER_LEN = 20
+ETHER_STANDARD_MTU = 1518
+ETHER_JUMBO_FRAME_MTU = 9000
+
+
+class TestJumboframes(TestSuite):
+    """DPDK PMD jumbo frames test suite.
+
+    Asserts the expected behavior of frames greater than, less then, or equal to
+    a designated MTU size in the testpmd application. If a packet size greater
+    than the designated testpmd MTU length is retrieved, the test fails. If a
+    packet size less than or equal to the designated testpmd MTU length is retrieved,
+    the test passes.
+    """
+
+    def set_up_suite(self) -> None:
+        """Set up the test suite.
+
+        Setup:
+            Set traffic generator MTU lengths to a size greater than scope of all
+            test cases.
+        """
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_JUMBO_FRAME_MTU + 200, self._tg_port_egress
+        )
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_JUMBO_FRAME_MTU + 200, self._tg_port_ingress
+        )
+
+    def send_packet_and_verify(self, pktsize: int, should_receive: bool = True) -> None:
+        """Generate, send, and capture packets to verify that the sent packet was received or not.
+
+        Generates a packet based on a specified size and sends it to the SUT. The desired packet's
+        payload size is calculated, and arbitrary, byte-sized characters are inserted into the
+        packet before sending. Packets are captured, and depending on the test case, packet
+        payloads are checked to determine if the sent payload was received.
+
+        Args:
+            pktsize: Size of packet to be generated and sent.
+            should_receive: Indicate whether the test case expects to receive the packet or not.
+        """
+        pktlength = pktsize - ETHER_HEADER_LEN
+        padding = pktlength - IP_HEADER_LEN
+
+        packet = Ether() / IP(len=pktlength) / Raw(load="\x50" * padding)
+        received_packets = self.send_packet_and_capture(packet)
+
+        found = any(
+            ("\x50" * padding) in str(packets.load)
+            for packets in received_packets
+            if hasattr(packets, "load")
+        )
+
+        print(found)
+        if should_receive:
+            self.verify(found, "Packet pass assert error")
+        else:
+            self.verify(not found, "Packet drop assert error")
+
+    def test_jumboframes_normal_nojumbo(self) -> None:
+        """Assess the boundaries of packets sent less than or equal to the standard MTU length.
+
+        PMDs are set to the standard MTU length of 1518 to assess behavior of sent packets less than
+        or equal to this size. Sends two packets: one that is less than 1518 bytes, and another that
+        is equal to 1518 bytes. The test case expects to receive both packets.
+
+        Test:
+            Start testpmd and send packets of sizes 1517 and 1518.
+        """
+        testpmd = self.sut_node.create_interactive_shell(
+            TestPmdShell,
+            app_parameters=(
+                "--max-pkt-len=%s " % (ETHER_STANDARD_MTU) + "--port-topology=paired "
+                "--tx-offloads=0x8000 "
+            ),
+            privileged=True,
+        )
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_STANDARD_MTU - 1)
+        self.send_packet_and_verify(ETHER_STANDARD_MTU)
+        testpmd.close()
+
+    def test_jumboframes_jumbo_nojumbo(self) -> None:
+        """Assess the boundaries of packets sent greater than standard MTU length.
+
+        PMDs are set to the standard MTU length of 1518 bytes to assess behavior of sent packets
+        greater than this size. Sends one packet with a frame size of 1519. The test cases does
+        not expect to receive this packet.
+
+        Test:
+            Start testpmd with standard MTU size of 1518. Send a packet of 1519 and verify it was
+            not received.
+        """
+        testpmd = self.sut_node.create_interactive_shell(
+            TestPmdShell,
+            app_parameters=(
+                "--max-pkt-len=%s " % (ETHER_STANDARD_MTU) + "--port-topology=paired "
+                "--tx-offloads=0x8000 "
+            ),
+            privileged=True,
+        )
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_STANDARD_MTU + 1, False)
+        testpmd.close()
+
+    def test_jumboframes_normal_jumbo(self) -> None:
+        """Assess the consistency of standard 1518 byte packets using a 9000 byte jumbo MTU length.
+
+        PMDs are set to a jumbo frame size of 9000 bytes. Packets of sizes 1517 and 1518 are sent
+        to assess the boundaries of packets less than or equal to the standard MTU length of 1518.
+        The test case expects to receive both packets.
+
+        Test:
+            Start testpmd with a jumbo frame size of 9000 bytes. Send a packet of 1517 and 1518
+            and verify they were received.
+        """
+        testpmd = self.sut_node.create_interactive_shell(
+            TestPmdShell,
+            app_parameters=(
+                "--max-pkt-len=%s " % (ETHER_JUMBO_FRAME_MTU) + "--port-topology=paired "
+                "--tx-offloads=0x8000 "
+            ),
+            privileged=True,
+        )
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_STANDARD_MTU - 1)
+        self.send_packet_and_verify(ETHER_STANDARD_MTU)
+        testpmd.close()
+
+    def test_jumboframes_jumbo_jumbo(self) -> None:
+        """Assess the boundaries packets sent at an MTU size of 9000 bytes.
+
+        PMDs are set to a jumbo frames size of 9000 bytes. Packets of size 1519, 8999, and 9000
+        are sent. The test expects to receive all packets.
+
+        Test:
+            Start testpmd with an MTU length of 9000 bytes. Send packets of size 1519, 8999,
+            and 9000 and verify that all packets were received.
+        """
+        testpmd = self.sut_node.create_interactive_shell(
+            TestPmdShell,
+            app_parameters=(
+                "--max-pkt-len=%s " % (ETHER_JUMBO_FRAME_MTU) + "--port-topology=paired "
+                "--tx-offloads=0x8000 "
+            ),
+            privileged=True,
+        )
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_STANDARD_MTU + 1)
+        self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU - 1)
+        self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU)
+        testpmd.close()
+
+    def test_jumboframes_bigger_jumbo(self) -> None:
+        """Assess the behavior of packets send greater than a specified MTU length of 9000 bytes.
+
+        PMDs are set to a jumbo frames size of 9000 bytes. A packet of size 9001 is sent to the SUT.
+        The test case does not expect to receive the packet.
+
+        Test:
+            Start testpmd with an MTU length of 9000 bytes. Send a packet of 9001 bytes and verify
+            it was not received.
+        """
+        testpmd = self.sut_node.create_interactive_shell(
+            TestPmdShell,
+            app_parameters=(
+                "--max-pkt-len=%s " % (ETHER_JUMBO_FRAME_MTU) + "--port-topology=paired "
+                "--tx-offloads=0x8000 "
+            ),
+            privileged=True,
+        )
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU + 1, False)
+        testpmd.close()
+
+    def tear_down_suite(self) -> None:
+        """Tear down the test suite.
+
+        Teardown:
+            Set the MTU size of the traffic generator back to the standard 1518 byte size.
+        """
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_STANDARD_MTU - ETHER_HEADER_LEN, self._tg_port_egress
+        )
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_STANDARD_MTU - ETHER_HEADER_LEN, self._tg_port_ingress
+        )
-- 
2.44.0

[RFC PATCH v3 0/2] Initial Implementation For Jumbo Frames

[Nicholas Pratte]

v3:
  * Refactored to use TestPMDShell context manager.

NOTE: Assessing the boundaries and discern the correct assumption
for ethernet overhead is still to be discussed. Thus, while each
individual test case may pass, the test cases may not yet be precise.

Nicholas Pratte (2):
  dts: add port config mtu options to testpmd shell
  dts: Initial Implementation For Jumbo Frames Test Suite

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py |  20 +-
 dts/tests/TestSuite_jumboframes.py            | 182 ++++++++++++++++++
 3 files changed, 203 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_jumboframes.py

-- 
2.44.0

[RFC PATCH v2 0/1] Initial Implementation For Jumbo Frames

[Nicholas Pratte]

v2:
  - Suite has been refactored to include latest DPDK merge.
  - Include a 6 second timer to ensure each test case passes (This is
    just temporary placement in lieu of the context manager)


Nicholas Pratte (1):
  Initial Implementation For Jumbo Frames Test Suite

 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_jumboframes.py         | 181 +++++++++++++++++++++
 2 files changed, 183 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_jumboframes.py

-- 
2.44.0

[RFC PATCH v2] Initial Implementation For Jumbo Frames Test Suite

[Nicholas Pratte]

The following test suite reflects the fundamental outline for how the
jumbo frames test suite may be designed. The test suite consists of five
individual test cases, each of which assesses the behavior of packet
transmissions for both 1518 byte and 9000 byte frames.

The edge cases are ripped directly from the old DTS framework, and the
general methodology is the same as well. The process, at this point, has
been refactored to operate within the new DTS framework.

Bugzilla ID: 1421
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_jumboframes.py         | 182 +++++++++++++++++++++
 2 files changed, 184 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_jumboframes.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..a1028f128b 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "jumboframes"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_jumboframes.py b/dts/tests/TestSuite_jumboframes.py
new file mode 100644
index 0000000000..f1edcb789a
--- /dev/null
+++ b/dts/tests/TestSuite_jumboframes.py
@@ -0,0 +1,182 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023-2024 University of New Hampshire
+"""Jumbo frame consistency and compatibility test suite.
+
+The test suite ensures the consistency of jumbo frames transmission within
+Poll Mode Drivers using a series of individual test cases. If a Poll Mode
+Driver receives a packet that is greater than its assigned MTU length, then
+that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver
+receives a packet that is less than or equal to a its designated MTU length, then the
+packet should be transmitted by the Poll Mode Driver, completing a cycle within the
+testbed and getting received by the traffic generator. Thus, the following test suite
+evaluates the behavior within all possible edge cases, ensuring that a test Poll
+Mode Driver strictly abides by the above implications.
+"""
+
+from time import sleep
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+ETHER_HEADER_LEN = 18
+IP_HEADER_LEN = 20
+ETHER_STANDARD_MTU = 1518
+ETHER_JUMBO_FRAME_MTU = 9000
+
+
+class TestJumboframes(TestSuite):
+    """DPDK PMD jumbo frames test suite.
+
+    Asserts the expected behavior of frames greater than, less then, or equal to
+    a designated MTU size in the testpmd application. If a packet size greater
+    than the designated testpmd MTU length is retrieved, the test fails. If a
+    packet size less than or equal to the designated testpmd MTU length is retrieved,
+    the test passes.
+    """
+
+    def set_up_suite(self) -> None:
+        """Set up the test suite.
+
+        Setup:
+            Set traffic generator MTU lengths to a size greater than scope of all
+            test cases.
+        """
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_JUMBO_FRAME_MTU + 200, self._tg_port_egress
+        )
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_JUMBO_FRAME_MTU + 200, self._tg_port_ingress
+        )
+
+    def send_packet_and_verify(self, pktsize: int, should_receive: bool = True) -> None:
+        """Generate, send, and capture packets to verify that the sent packet was received or not.
+
+        Generates a packet based on a specified size and sends it to the SUT. The desired packet's
+        payload size is calculated, and arbitrary, byte-sized characters are inserted into the
+        packet before sending. Packets are captured, and depending on the test case, packet
+        payloads are checked to determine if the sent payload was received.
+
+        Args:
+            pktsize: Size of packet to be generated and sent.
+            should_receive: Indicate whether the test case expects to receive the packet or not.
+        """
+        pktlength = pktsize - ETHER_HEADER_LEN
+        padding = pktlength - IP_HEADER_LEN
+
+        packet = Ether() / IP(len=pktlength) / Raw(load="\x50" * padding)
+        received_packets = self.send_packet_and_capture(packet)
+
+        found = any(
+            ("\x50" * padding) in str(packets.load)
+            for packets in received_packets
+            if hasattr(packets, "load")
+        )
+
+        print(found)
+        if should_receive:
+            self.verify(found, "Packet pass assert error")
+        else:
+            self.verify(not found, "Packet drop assert error")
+
+    def test_jumboframes_normal_nojumbo(self) -> None:
+        """Assess the boundaries of packets sent less than or equal to the standard MTU length.
+
+        PMDs are set to the standard MTU length of 1518 to assess behavior of sent packets less than
+        or equal to this size. Sends two packets: one that is less than 1518 bytes, and another that
+        is equal to 1518 bytes. The test case expects to receive both packets.
+
+        Test:
+            Start testpmd and send packets of sizes 1517 and 1518.
+        """
+        testpmd = TestPmdShell(self.sut_node, max_pkt_len=ETHER_STANDARD_MTU, tx_offloads=0x8000)
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_STANDARD_MTU - 1)
+        self.send_packet_and_verify(ETHER_STANDARD_MTU)
+        testpmd.close()
+        sleep(6)
+
+    def test_jumboframes_jumbo_nojumbo(self) -> None:
+        """Assess the boundaries of packets sent greater than standard MTU length.
+
+        PMDs are set to the standard MTU length of 1518 bytes to assess behavior of sent packets
+        greater than this size. Sends one packet with a frame size of 1519. The test cases does
+        not expect to receive this packet.
+
+        Test:
+            Start testpmd with standard MTU size of 1518. Send a packet of 1519 and verify it was
+            not received.
+        """
+        testpmd = TestPmdShell(self.sut_node, max_pkt_len=ETHER_STANDARD_MTU, tx_offloads=0x8000)
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_STANDARD_MTU + 1, False)
+        testpmd.close()
+        sleep(6)
+
+    def test_jumboframes_normal_jumbo(self) -> None:
+        """Assess the consistency of standard 1518 byte packets using a 9000 byte jumbo MTU length.
+
+        PMDs are set to a jumbo frame size of 9000 bytes. Packets of sizes 1517 and 1518 are sent
+        to assess the boundaries of packets less than or equal to the standard MTU length of 1518.
+        The test case expects to receive both packets.
+
+        Test:
+            Start testpmd with a jumbo frame size of 9000 bytes. Send a packet of 1517 and 1518
+            and verify they were received.
+        """
+        testpmd = TestPmdShell(self.sut_node, max_pkt_len=ETHER_JUMBO_FRAME_MTU, tx_offloads=0x8000)
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_STANDARD_MTU - 1)
+        self.send_packet_and_verify(ETHER_STANDARD_MTU)
+        testpmd.close()
+        sleep(6)
+
+    def test_jumboframes_jumbo_jumbo(self) -> None:
+        """Assess the boundaries packets sent at an MTU size of 9000 bytes.
+
+        PMDs are set to a jumbo frames size of 9000 bytes. Packets of size 1519, 8999, and 9000
+        are sent. The test expects to receive all packets.
+
+        Test:
+            Start testpmd with an MTU length of 9000 bytes. Send packets of size 1519, 8999,
+            and 9000 and verify that all packets were received.
+        """
+        testpmd = TestPmdShell(self.sut_node, max_pkt_len=ETHER_JUMBO_FRAME_MTU, tx_offloads=0x8000)
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_STANDARD_MTU + 1)
+        self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU - 1)
+        self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU)
+        testpmd.close()
+        sleep(6)
+
+    def test_jumboframes_bigger_jumbo(self) -> None:
+        """Assess the behavior of packets send greater than a specified MTU length of 9000 bytes.
+
+        PMDs are set to a jumbo frames size of 9000 bytes. A packet of size 9001 is sent to the SUT.
+        The test case does not expect to receive the packet.
+
+        Test:
+            Start testpmd with an MTU length of 9000 bytes. Send a packet of 9001 bytes and verify
+            it was not received.
+        """
+        testpmd = TestPmdShell(self.sut_node, max_pkt_len=ETHER_JUMBO_FRAME_MTU, tx_offloads=0x8000)
+        testpmd.start()
+        self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU + 1, False)
+        testpmd.close()
+        sleep(6)
+
+    def tear_down_suite(self) -> None:
+        """Tear down the test suite.
+
+        Teardown:
+            Set the MTU size of the traffic generator back to the standard 1518 byte size.
+        """
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_STANDARD_MTU - ETHER_HEADER_LEN, self._tg_port_egress
+        )
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_STANDARD_MTU - ETHER_HEADER_LEN, self._tg_port_ingress
+        )
-- 
2.44.0

Re: [RFC PATCH v2] Initial Implementation For Jumbo Frames Test Suite

[Stephen Hemminger]

On Fri, 21 Jun 2024 17:19:21 -0400
Nicholas Pratte <npratte@iol.unh.edu> wrote:

> +The test suite ensures the consistency of jumbo frames transmission within
> +Poll Mode Drivers using a series of individual test cases. If a Poll Mode
> +Driver receives a packet that is greater than its assigned MTU length, then
> +that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver
> +receives a packet that is less than or equal to a its designated MTU length, then the
> +packet should be transmitted by the Poll Mode Driver, completing a cycle within the
> +testbed and getting received by the traffic generator. Thus, the following test suite
> +evaluates the behavior within all possible edge cases, ensuring that a test Poll
> +Mode Driver strictly abides by the above implications.

There are some weird drivers where MRU and MTU are not the same thing.
I believe the e1000 HW only allowed setting buffer size to a power of 2.
At least on Linux, that meant that with 1500 byte MTU it would receive an up to 2K packet.
This never caused any problem for upper layer protocols, just some picky conformance tests.

Re: [RFC PATCH v2] Initial Implementation For Jumbo Frames Test Suite

[Honnappa Nagarahalli]

> On Jun 21, 2024, at 5:18 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> On Fri, 21 Jun 2024 17:19:21 -0400
> Nicholas Pratte <npratte@iol.unh.edu> wrote:
> 
>> +The test suite ensures the consistency of jumbo frames transmission within
>> +Poll Mode Drivers using a series of individual test cases. If a Poll Mode
>> +Driver receives a packet that is greater than its assigned MTU length, then
>> +that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver
>> +receives a packet that is less than or equal to a its designated MTU length, then the
>> +packet should be transmitted by the Poll Mode Driver, completing a cycle within the
>> +testbed and getting received by the traffic generator. Thus, the following test suite
>> +evaluates the behavior within all possible edge cases, ensuring that a test Poll
>> +Mode Driver strictly abides by the above implications.
> 
> There are some weird drivers where MRU and MTU are not the same thing.
> I believe the e1000 HW only allowed setting buffer size to a power of 2.
> At least on Linux, that meant that with 1500 byte MTU it would receive an up to 2K packet.
> This never caused any problem for upper layer protocols, just some picky conformance tests.
The test cases should not concern themselves with individual PMD behaviors. They should be based on the API definition.

Re: [RFC PATCH v2] Initial Implementation For Jumbo Frames Test Suite

[Nicholas Pratte]

The previous comments led me to go on an investigation about how MTUs
are allocated in testpmd. --max-pkt-len, which the documentation
states is defaulted to 1518, will shave off 18 bytes to account for
the 14 byte Ethernet frame and the Dot1Q headers. This is the case
when using Mellanox, but for both Intel and Broadcom, when explicitly
setting --max-pkt-len to 1518, the MTU listed on 'show port info' is
1492. So there are 26 bytes being shaved off, leaving 8 unknown bytes.
Does anyone know what these extra 8 bytes are? I wondered if these
might be VXLAN, FCS or something else, but it might just be easier to
ask and see if anyone knows; I can't find anything about it in
documentation.

As far as how this relates to the test suite at hand, the
send_packet_and_capture() method will need to be reworked to
compensate for the extra 4 Dot1Q header bytes, but I'm still curious
about the extra 8 bytes on the Intel and Broadcom devices I've tested
on; again, these bytes are not present on Mellanox, which is just
bound to their specific kernel drivers. When I manually set the
--max-pkt-len to 1518, with the MTU then set to 1492 bytes, packets
sent to the SUT can only be, including frame size, 1514 bytes in size.
So I'm looking at the DPDK MTU output plus 22, even when using 'port
config mtu 0 1500,' for instance, so I'm not sure why 26 bytes is
subtracted here.

On Fri, Jun 21, 2024 at 7:55 PM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
>
>
>
> > On Jun 21, 2024, at 5:18 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> >
> > On Fri, 21 Jun 2024 17:19:21 -0400
> > Nicholas Pratte <npratte@iol.unh.edu> wrote:
> >
> >> +The test suite ensures the consistency of jumbo frames transmission within
> >> +Poll Mode Drivers using a series of individual test cases. If a Poll Mode
> >> +Driver receives a packet that is greater than its assigned MTU length, then
> >> +that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver
> >> +receives a packet that is less than or equal to a its designated MTU length, then the
> >> +packet should be transmitted by the Poll Mode Driver, completing a cycle within the
> >> +testbed and getting received by the traffic generator. Thus, the following test suite
> >> +evaluates the behavior within all possible edge cases, ensuring that a test Poll
> >> +Mode Driver strictly abides by the above implications.
> >
> > There are some weird drivers where MRU and MTU are not the same thing.
> > I believe the e1000 HW only allowed setting buffer size to a power of 2.
> > At least on Linux, that meant that with 1500 byte MTU it would receive an up to 2K packet.
> > This never caused any problem for upper layer protocols, just some picky conformance tests.
> The test cases should not concern themselves with individual PMD behaviors. They should be based on the API definition.
>

Re: [RFC PATCH v2] Initial Implementation For Jumbo Frames Test Suite

[Thomas Monjalon]

Nicholas, you are writing a test for the API.
You should not adapt to the driver behaviour.
If the driver does not report what we can expect from the API definition,
it is a bug.

Ferruh, please can you explain what is the problem with MTU sizes?


25/06/2024 21:57, Nicholas Pratte:
> The previous comments led me to go on an investigation about how MTUs
> are allocated in testpmd. --max-pkt-len, which the documentation
> states is defaulted to 1518, will shave off 18 bytes to account for
> the 14 byte Ethernet frame and the Dot1Q headers. This is the case
> when using Mellanox, but for both Intel and Broadcom, when explicitly
> setting --max-pkt-len to 1518, the MTU listed on 'show port info' is
> 1492. So there are 26 bytes being shaved off, leaving 8 unknown bytes.
> Does anyone know what these extra 8 bytes are? I wondered if these
> might be VXLAN, FCS or something else, but it might just be easier to
> ask and see if anyone knows; I can't find anything about it in
> documentation.
> 
> As far as how this relates to the test suite at hand, the
> send_packet_and_capture() method will need to be reworked to
> compensate for the extra 4 Dot1Q header bytes, but I'm still curious
> about the extra 8 bytes on the Intel and Broadcom devices I've tested
> on; again, these bytes are not present on Mellanox, which is just
> bound to their specific kernel drivers. When I manually set the
> --max-pkt-len to 1518, with the MTU then set to 1492 bytes, packets
> sent to the SUT can only be, including frame size, 1514 bytes in size.
> So I'm looking at the DPDK MTU output plus 22, even when using 'port
> config mtu 0 1500,' for instance, so I'm not sure why 26 bytes is
> subtracted here.
> 
> On Fri, Jun 21, 2024 at 7:55 PM Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com> wrote:
> >
> >
> >
> > > On Jun 21, 2024, at 5:18 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> > >
> > > On Fri, 21 Jun 2024 17:19:21 -0400
> > > Nicholas Pratte <npratte@iol.unh.edu> wrote:
> > >
> > >> +The test suite ensures the consistency of jumbo frames transmission within
> > >> +Poll Mode Drivers using a series of individual test cases. If a Poll Mode
> > >> +Driver receives a packet that is greater than its assigned MTU length, then
> > >> +that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver
> > >> +receives a packet that is less than or equal to a its designated MTU length, then the
> > >> +packet should be transmitted by the Poll Mode Driver, completing a cycle within the
> > >> +testbed and getting received by the traffic generator. Thus, the following test suite
> > >> +evaluates the behavior within all possible edge cases, ensuring that a test Poll
> > >> +Mode Driver strictly abides by the above implications.
> > >
> > > There are some weird drivers where MRU and MTU are not the same thing.
> > > I believe the e1000 HW only allowed setting buffer size to a power of 2.
> > > At least on Linux, that meant that with 1500 byte MTU it would receive an up to 2K packet.
> > > This never caused any problem for upper layer protocols, just some picky conformance tests.
> > The test cases should not concern themselves with individual PMD behaviors. They should be based on the API definition.
> >
> 

Re: [RFC PATCH v2] Initial Implementation For Jumbo Frames Test Suite

[Nicholas Pratte]

If I would like to create a test suite that is driver agnostic, then I
would need to increase the boundaries at which I conduct each
individual test case. Right now, I test jumbo frame behavior -1, +1,
or at the MTU boundary. But, packets in TestPMD are dropped or
forwarded based on the MTU size plus additional ethernet overhead. We
would need to agree on a universal expected value for ethernet
overhead across all devices, or come up with another solution.

I created a Bugzilla ticket to discuss this further, it might make
sense to discuss this outside of this thread.

https://bugs.dpdk.org/show_bug.cgi?id=1476




On Tue, Jun 25, 2024 at 5:57 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> Nicholas, you are writing a test for the API.
> You should not adapt to the driver behaviour.
> If the driver does not report what we can expect from the API definition,
> it is a bug.
>
> Ferruh, please can you explain what is the problem with MTU sizes?
>
>
> 25/06/2024 21:57, Nicholas Pratte:
> > The previous comments led me to go on an investigation about how MTUs
> > are allocated in testpmd. --max-pkt-len, which the documentation
> > states is defaulted to 1518, will shave off 18 bytes to account for
> > the 14 byte Ethernet frame and the Dot1Q headers. This is the case
> > when using Mellanox, but for both Intel and Broadcom, when explicitly
> > setting --max-pkt-len to 1518, the MTU listed on 'show port info' is
> > 1492. So there are 26 bytes being shaved off, leaving 8 unknown bytes.
> > Does anyone know what these extra 8 bytes are? I wondered if these
> > might be VXLAN, FCS or something else, but it might just be easier to
> > ask and see if anyone knows; I can't find anything about it in
> > documentation.
> >
> > As far as how this relates to the test suite at hand, the
> > send_packet_and_capture() method will need to be reworked to
> > compensate for the extra 4 Dot1Q header bytes, but I'm still curious
> > about the extra 8 bytes on the Intel and Broadcom devices I've tested
> > on; again, these bytes are not present on Mellanox, which is just
> > bound to their specific kernel drivers. When I manually set the
> > --max-pkt-len to 1518, with the MTU then set to 1492 bytes, packets
> > sent to the SUT can only be, including frame size, 1514 bytes in size.
> > So I'm looking at the DPDK MTU output plus 22, even when using 'port
> > config mtu 0 1500,' for instance, so I'm not sure why 26 bytes is
> > subtracted here.
> >
> > On Fri, Jun 21, 2024 at 7:55 PM Honnappa Nagarahalli
> > <Honnappa.Nagarahalli@arm.com> wrote:
> > >
> > >
> > >
> > > > On Jun 21, 2024, at 5:18 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> > > >
> > > > On Fri, 21 Jun 2024 17:19:21 -0400
> > > > Nicholas Pratte <npratte@iol.unh.edu> wrote:
> > > >
> > > >> +The test suite ensures the consistency of jumbo frames transmission within
> > > >> +Poll Mode Drivers using a series of individual test cases. If a Poll Mode
> > > >> +Driver receives a packet that is greater than its assigned MTU length, then
> > > >> +that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver
> > > >> +receives a packet that is less than or equal to a its designated MTU length, then the
> > > >> +packet should be transmitted by the Poll Mode Driver, completing a cycle within the
> > > >> +testbed and getting received by the traffic generator. Thus, the following test suite
> > > >> +evaluates the behavior within all possible edge cases, ensuring that a test Poll
> > > >> +Mode Driver strictly abides by the above implications.
> > > >
> > > > There are some weird drivers where MRU and MTU are not the same thing.
> > > > I believe the e1000 HW only allowed setting buffer size to a power of 2.
> > > > At least on Linux, that meant that with 1500 byte MTU it would receive an up to 2K packet.
> > > > This never caused any problem for upper layer protocols, just some picky conformance tests.
> > > The test cases should not concern themselves with individual PMD behaviors. They should be based on the API definition.
> > >
> >
>
>
>
>
>

Re: [RFC PATCH v2] Initial Implementation For Jumbo Frames Test Suite

[Stephen Hemminger]

On Tue, 25 Jun 2024 15:57:02 -0400
Nicholas Pratte <npratte@iol.unh.edu> wrote:

> The previous comments led me to go on an investigation about how MTUs
> are allocated in testpmd. --max-pkt-len, which the documentation
> states is defaulted to 1518, will shave off 18 bytes to account for
> the 14 byte Ethernet frame and the Dot1Q headers. This is the case
> when using Mellanox, but for both Intel and Broadcom, when explicitly
> setting --max-pkt-len to 1518, the MTU listed on 'show port info' is
> 1492. So there are 26 bytes being shaved off, leaving 8 unknown bytes.
> Does anyone know what these extra 8 bytes are? I wondered if these
> might be VXLAN, FCS or something else, but it might just be easier to
> ask and see if anyone knows; I can't find anything about it in
> documentation.
> 
> As far as how this relates to the test suite at hand, the
> send_packet_and_capture() method will need to be reworked to
> compensate for the extra 4 Dot1Q header bytes, but I'm still curious
> about the extra 8 bytes on the Intel and Broadcom devices I've tested
> on; again, these bytes are not present on Mellanox, which is just
> bound to their specific kernel drivers. When I manually set the
> --max-pkt-len to 1518, with the MTU then set to 1492 bytes, packets
> sent to the SUT can only be, including frame size, 1514 bytes in size.
> So I'm looking at the DPDK MTU output plus 22, even when using 'port
> config mtu 0 1500,' for instance, so I'm not sure why 26 bytes is
> subtracted here.
> 
> On Fri, Jun 21, 2024 at 7:55 PM Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com> wrote:
> >
> >
> >  
> > > On Jun 21, 2024, at 5:18 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> > >
> > > On Fri, 21 Jun 2024 17:19:21 -0400
> > > Nicholas Pratte <npratte@iol.unh.edu> wrote:
> > >  
> > >> +The test suite ensures the consistency of jumbo frames transmission within
> > >> +Poll Mode Drivers using a series of individual test cases. If a Poll Mode
> > >> +Driver receives a packet that is greater than its assigned MTU length, then
> > >> +that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver
> > >> +receives a packet that is less than or equal to a its designated MTU length, then the
> > >> +packet should be transmitted by the Poll Mode Driver, completing a cycle within the
> > >> +testbed and getting received by the traffic generator. Thus, the following test suite
> > >> +evaluates the behavior within all possible edge cases, ensuring that a test Poll
> > >> +Mode Driver strictly abides by the above implications.  
> > >
> > > There are some weird drivers where MRU and MTU are not the same thing.
> > > I believe the e1000 HW only allowed setting buffer size to a power of 2.
> > > At least on Linux, that meant that with 1500 byte MTU it would receive an up to 2K packet.
> > > This never caused any problem for upper layer protocols, just some picky conformance tests.  
> > The test cases should not concern themselves with individual PMD behaviors. They should be based on the API definition.
> >  

There is confusion in DPDK about Maximum Transmission Unit (MTU) vs Maximum Receive Unit (MRU).
For almost all things, they are the same thing, but technically there is a difference.

https://swamy2064.wordpress.com/2018/12/01/mtu-and-mru/

[RFC PATCH v3 0/2] Initial Implementation For Jumbo Frames

[Nicholas Pratte]

v3:
  * Refactored to use TestPMDShell context manager.

NOTE: Assessing the boundaries and discern the correct assumption
for ethernet overhead is still to be discussed. Thus, while each
individual test case may pass, the test cases may not yet be precise.

Nicholas Pratte (2):
  dts: add port config mtu options to testpmd shell
  dts: Initial Implementation For Jumbo Frames Test Suite

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py |  20 +-
 dts/tests/TestSuite_jumboframes.py            | 182 ++++++++++++++++++
 3 files changed, 203 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_jumboframes.py

-- 
2.44.0

[RFC PATCH v3 1/2] dts: add port config mtu options to testpmd shell

[Nicholas Pratte]

Testpmd offers mtu configuration options that omit ethernet overhead
calculations when set. This patch adds easy-of-use methods to leverage
these runtime options.

Bugzilla ID: 1421

Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 20 ++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index eda6eb320f..83f7961359 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -804,7 +804,25 @@ def show_port_stats(self, port_id: int) -> TestPmdPortStats:
 
         return TestPmdPortStats.parse(output)
 
-    def _close(self) -> None:
+    def configure_port_mtu(self, port_id: int, mtu_length: int) -> None:
+        """Set the MTU length on a designated port.
+
+        Args:
+            port_id: The ID of the port being configured.
+            mtu_length: The length, in bytes, of the MTU being set.
+        """
+        self.send_command(f"port config mtu {port_id} {mtu_length}")
+
+    def configure_port_mtu_all(self, mtu_length: int) -> None:
+        """Set the MTU length on all designated ports.
+
+        Args:
+            mtu_length: The MTU length to be set on all ports.
+        """
+        for port in self.show_port_info_all():
+            self.send_command(f"port config mtu {port.id} {mtu_length}")
+
+    def close(self) -> None:
         """Overrides :meth:`~.interactive_shell.close`."""
         self.stop()
         self.send_command("quit", "Bye...")
-- 
2.44.0

Re: [RFC PATCH v3 1/2] dts: add port config mtu options to testpmd shell

[Jeremy Spewock]

Hey Nick,

Looks good to me, I just had one comment about what looks like a
mistake in a merge, and then another more general question.

On Fri, Jul 26, 2024 at 10:13 AM Nicholas Pratte <npratte@iol.unh.edu> wrote:
>
> Testpmd offers mtu configuration options that omit ethernet overhead
> calculations when set. This patch adds easy-of-use methods to leverage
> these runtime options.
>
> Bugzilla ID: 1421
>
> Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
> ---
>  dts/framework/remote_session/testpmd_shell.py | 20 ++++++++++++++++++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index eda6eb320f..83f7961359 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -804,7 +804,25 @@ def show_port_stats(self, port_id: int) -> TestPmdPortStats:
>
>          return TestPmdPortStats.parse(output)
>
> -    def _close(self) -> None:
> +    def configure_port_mtu(self, port_id: int, mtu_length: int) -> None:

Was there a reason you decided to omit the verify parameter? I think
it would still be valuable to have just so the developer knows that if
they get past this step they are guaranteed to have a modified MTU. I
think you can do it with port info and I actually wrote the same
method in my (albeit, old and outdated now) scatter expansion patch if
you wanted to see an example of what I mean [1]. Additionally, I think
on some NICs you also have to stop the port before you can adjust the
MTU, so this could fail in some cases.

> +        """Set the MTU length on a designated port.
> +
> +        Args:
> +            port_id: The ID of the port being configured.
> +            mtu_length: The length, in bytes, of the MTU being set.
> +        """
> +        self.send_command(f"port config mtu {port_id} {mtu_length}")
> +
> +    def configure_port_mtu_all(self, mtu_length: int) -> None:
> +        """Set the MTU length on all designated ports.
> +
> +        Args:
> +            mtu_length: The MTU length to be set on all ports.
> +        """
> +        for port in self.show_port_info_all():
> +            self.send_command(f"port config mtu {port.id} {mtu_length}")
> +
> +    def close(self) -> None:

This looks like something that went wrong in the merge, this method is
called _close on main and that is the one that
SingleActiveIntactiveShells use to properly close.

>          """Overrides :meth:`~.interactive_shell.close`."""
>          self.stop()
>          self.send_command("quit", "Bye...")
> --
> 2.44.0
>
[1] https://patchwork.dpdk.org/project/dpdk/patch/20240709175341.183888-2-jspewock@iol.unh.edu/

Re: [RFC PATCH v3 1/2] dts: add port config mtu options to testpmd shell

[Juraj Linkeš]

>> +    def configure_port_mtu_all(self, mtu_length: int) -> None:
>> +        """Set the MTU length on all designated ports.
>> +
>> +        Args:
>> +            mtu_length: The MTU length to be set on all ports.
>> +        """
>> +        for port in self.show_port_info_all():
>> +            self.send_command(f"port config mtu {port.id} {mtu_length}")

There's a patch in my capabilities series [0] that could be use here. 
I'll create a standalone patch from it.

[0] 
https://patches.dpdk.org/project/dpdk/patch/20240821145315.97974-8-juraj.linkes@pantheon.tech/

>> +
>> +    def close(self) -> None:
> 
> This looks like something that went wrong in the merge, this method is
> called _close on main and that is the one that
> SingleActiveIntactiveShells use to properly close.
> 
>>           """Overrides :meth:`~.interactive_shell.close`."""
>>           self.stop()
>>           self.send_command("quit", "Bye...")
>> --
>> 2.44.0
>>
> [1] https://patchwork.dpdk.org/project/dpdk/patch/20240709175341.183888-2-jspewock@iol.unh.edu/

These two are basically the same patch. Let's coordinate this and create 
one standalone patch that will be used for all three patches (I'm 
counting my capabilities patch as well here).

[RFC PATCH v3 2/2] dts: Initial Implementation For Jumbo Frames Test Suite

[Nicholas Pratte]

The following test suite reflects the fundamental outline for how the
jumbo frames test suite may be designed. The test suite consists of five
individual test cases, each of which assesses the behavior of packet
transmissions for both 1518 byte and 9000 byte frames.

The edge cases are ripped directly from the old DTS framework, and the
general methodology is the same as well. The process, at this point, has
been refactored to operate within the new DTS framework.

Bugzilla ID: 1421

Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_jumboframes.py         | 182 +++++++++++++++++++++
 2 files changed, 184 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_jumboframes.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..a1028f128b 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "jumboframes"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_jumboframes.py b/dts/tests/TestSuite_jumboframes.py
new file mode 100644
index 0000000000..dd8092f2a4
--- /dev/null
+++ b/dts/tests/TestSuite_jumboframes.py
@@ -0,0 +1,182 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023-2024 University of New Hampshire
+"""Jumbo frame consistency and compatibility test suite.
+
+The test suite ensures the consistency of jumbo frames transmission within
+Poll Mode Drivers using a series of individual test cases. If a Poll Mode
+Driver receives a packet that is greater than its assigned MTU length, then
+that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver
+receives a packet that is less than or equal to a its designated MTU length, then the
+packet should be transmitted by the Poll Mode Driver, completing a cycle within the
+testbed and getting received by the traffic generator. Thus, the following test suite
+evaluates the behavior within all possible edge cases, ensuring that a test Poll
+Mode Driver strictly abides by the above implications.
+"""
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+IP_HEADER_LEN = 20
+ETHER_STANDARD_FRAME = 1500
+ETHER_JUMBO_FRAME_MTU = 9000
+
+
+class TestJumboframes(TestSuite):
+    """DPDK PMD jumbo frames test suite.
+
+    Asserts the expected behavior of frames greater than, less then, or equal to
+    a designated MTU size in the testpmd application. If a packet size greater
+    than the designated testpmd MTU length is retrieved, the test fails. If a
+    packet size less than or equal to the designated testpmd MTU length is retrieved,
+    the test passes.
+    """
+
+    def set_up_suite(self) -> None:
+        """Set up the test suite.
+
+        Setup:
+            Set traffic generator MTU lengths to a size greater than scope of all
+            test cases.
+        """
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_JUMBO_FRAME_MTU + 200, self._tg_port_egress
+        )
+        self.tg_node.main_session.configure_port_mtu(
+            ETHER_JUMBO_FRAME_MTU + 200, self._tg_port_ingress
+        )
+
+    def send_packet_and_verify(self, pktsize: int, should_receive: bool = True) -> None:
+        """Generate, send, and capture packets to verify that the sent packet was received or not.
+
+        Generates a packet based on a specified size and sends it to the SUT. The desired packet's
+        payload size is calculated, and arbitrary, byte-sized characters are inserted into the
+        packet before sending. Packets are captured, and depending on the test case, packet
+        payloads are checked to determine if the sent payload was received.
+
+        Args:
+            pktsize: Size of packet to be generated and sent.
+            should_receive: Indicate whether the test case expects to receive the packet or not.
+        """
+        padding = pktsize - IP_HEADER_LEN
+        # Insert extra space for placeholder 'CRC' Error correction.
+        packet = Ether() / Raw("    ") / IP(len=pktsize) / Raw(load="X" * padding)
+        received_packets = self.send_packet_and_capture(packet)
+        found = any(
+            ("X" * padding) in str(packets.load)
+            for packets in received_packets
+            if hasattr(packets, "load")
+        )
+
+        if should_receive:
+            self.verify(found, "Did not receive packet")
+        else:
+            self.verify(not found, "Received packet")
+
+    def test_jumboframes_normal_nojumbo(self) -> None:
+        """Assess the boundaries of packets sent less than or equal to the standard MTU length.
+
+        PMDs are set to the standard MTU length of 1518 to assess behavior of sent packets less than
+        or equal to this size. Sends two packets: one that is less than 1518 bytes, and another that
+        is equal to 1518 bytes. The test case expects to receive both packets.
+
+        Test:
+            Start testpmd and send packets of sizes 1517 and 1518.
+        """
+        with TestPmdShell(
+            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
+        ) as testpmd:
+            testpmd.configure_port_mtu_all(ETHER_STANDARD_FRAME)
+            testpmd.start()
+
+            self.send_packet_and_verify(ETHER_STANDARD_FRAME - 5)
+            self.send_packet_and_verify(ETHER_STANDARD_FRAME)
+
+    def test_jumboframes_jumbo_nojumbo(self) -> None:
+        """Assess the boundaries of packets sent greater than standard MTU length.
+
+        PMDs are set to the standard MTU length of 1518 bytes to assess behavior of sent packets
+        greater than this size. Sends one packet with a frame size of 1519. The test cases does
+        not expect to receive this packet.
+
+        Test:
+            Start testpmd with standard MTU size of 1518. Send a packet of 1519 and verify it was
+            not received.
+        """
+        with TestPmdShell(
+            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
+        ) as testpmd:
+            testpmd.configure_port_mtu_all(ETHER_STANDARD_FRAME)
+            testpmd.start()
+
+            self.send_packet_and_verify(ETHER_STANDARD_FRAME + 5, False)
+
+    def test_jumboframes_normal_jumbo(self) -> None:
+        """Assess the consistency of standard 1518 byte packets using a 9000 byte jumbo MTU length.
+
+        PMDs are set to a jumbo frame size of 9000 bytes. Packets of sizes 1517 and 1518 are sent
+        to assess the boundaries of packets less than or equal to the standard MTU length of 1518.
+        The test case expects to receive both packets.
+
+        Test:
+            Start testpmd with a jumbo frame size of 9000 bytes. Send a packet of 1517 and 1518
+            and verify they were received.
+        """
+        with TestPmdShell(
+            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
+        ) as testpmd:
+            testpmd.configure_port_mtu_all(ETHER_JUMBO_FRAME_MTU)
+            testpmd.start()
+
+            self.send_packet_and_verify(ETHER_STANDARD_FRAME - 5)
+            self.send_packet_and_verify(ETHER_STANDARD_FRAME)
+
+    def test_jumboframes_jumbo_jumbo(self) -> None:
+        """Assess the boundaries packets sent at an MTU size of 9000 bytes.
+
+        PMDs are set to a jumbo frames size of 9000 bytes. Packets of size 1519, 8999, and 9000
+        are sent. The test expects to receive all packets.
+
+        Test:
+            Start testpmd with an MTU length of 9000 bytes. Send packets of size 1519, 8999,
+            and 9000 and verify that all packets were received.
+        """
+        with TestPmdShell(
+            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
+        ) as testpmd:
+            testpmd.configure_port_mtu_all(ETHER_JUMBO_FRAME_MTU)
+            testpmd.start()
+
+            self.send_packet_and_verify(ETHER_STANDARD_FRAME + 5)
+            self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU - 5)
+            self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU)
+
+    def test_jumboframes_bigger_jumbo(self) -> None:
+        """Assess the behavior of packets send greater than a specified MTU length of 9000 bytes.
+
+        PMDs are set to a jumbo frames size of 9000 bytes. A packet of size 9001 is sent to the SUT.
+        The test case does not expect to receive the packet.
+
+        Test:
+            Start testpmd with an MTU length of 9000 bytes. Send a packet of 9001 bytes and verify
+            it was not received.
+        """
+        with TestPmdShell(
+            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
+        ) as testpmd:
+            testpmd.configure_port_mtu_all(ETHER_JUMBO_FRAME_MTU)
+            testpmd.start()
+
+            self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU + 5, False)
+
+    def tear_down_suite(self) -> None:
+        """Tear down the test suite.
+
+        Teardown:
+            Set the MTU size of the traffic generator back to the standard 1518 byte size.
+        """
+        self.tg_node.main_session.configure_port_mtu(ETHER_STANDARD_FRAME, self._tg_port_egress)
+        self.tg_node.main_session.configure_port_mtu(ETHER_STANDARD_FRAME, self._tg_port_ingress)
-- 
2.44.0

Re: [RFC PATCH v3 2/2] dts: Initial Implementation For Jumbo Frames Test Suite

[Jeremy Spewock]

Just some very light comments here as well, mostly about doc-string
but also a few questions/suggestions.

On Fri, Jul 26, 2024 at 10:14 AM Nicholas Pratte <npratte@iol.unh.edu> wrote:
<snip>
> diff --git a/dts/tests/TestSuite_jumboframes.py b/dts/tests/TestSuite_jumboframes.py
> new file mode 100644
> index 0000000000..dd8092f2a4
> --- /dev/null
> +++ b/dts/tests/TestSuite_jumboframes.py
> @@ -0,0 +1,182 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2023-2024 University of New Hampshire

It's probably fine to just make the date on this copyright 2024.

> +"""Jumbo frame consistency and compatibility test suite.
> +
> +The test suite ensures the consistency of jumbo frames transmission within
> +Poll Mode Drivers using a series of individual test cases. If a Poll Mode
> +Driver receives a packet that is greater than its assigned MTU length, then
> +that packet will be dropped, and thus not received. Likewise, if a Poll Mode Driver

This sentence is a little confusing because you're saying "if a packet
is received with X condition then it isn't received." Maybe it would
be more clear to say it isn't forwarded?

> +receives a packet that is less than or equal to a its designated MTU length, then the

I think this was a typo, this should probably be either "equal to its"
or "equal to a" rather than both.

> +packet should be transmitted by the Poll Mode Driver, completing a cycle within the
> +testbed and getting received by the traffic generator. Thus, the following test suite
> +evaluates the behavior within all possible edge cases, ensuring that a test Poll
> +Mode Driver strictly abides by the above implications.
> +"""
<snip>
> +
> +class TestJumboframes(TestSuite):
> +    """DPDK PMD jumbo frames test suite.
> +
> +    Asserts the expected behavior of frames greater than, less then, or equal to

I think this should be "less than" rather than "less then".

> +    a designated MTU size in the testpmd application. If a packet size greater
> +    than the designated testpmd MTU length is retrieved, the test fails. If a
> +    packet size less than or equal to the designated testpmd MTU length is retrieved,
> +    the test passes.
> +    """
> +
> +    def set_up_suite(self) -> None:
> +        """Set up the test suite.
> +
> +        Setup:
> +            Set traffic generator MTU lengths to a size greater than scope of all
> +            test cases.
> +        """
> +        self.tg_node.main_session.configure_port_mtu(
> +            ETHER_JUMBO_FRAME_MTU + 200, self._tg_port_egress
> +        )
> +        self.tg_node.main_session.configure_port_mtu(
> +            ETHER_JUMBO_FRAME_MTU + 200, self._tg_port_ingress
> +        )

I know 9000 is a common jumboframe MTU to support, but do we have to
worry about a case where some NICs wouldn't support an MTU this high?
That could also be a case of the NICs where that would be a problem
are maybe older and not expected to be as common? I'm not completely
sure what the standards and expectations are.

> +
> +    def send_packet_and_verify(self, pktsize: int, should_receive: bool = True) -> None:
> +        """Generate, send, and capture packets to verify that the sent packet was received or not.

I feel like the "verify that..." asserts that you are verifying
something positive when it could be positive or negative. Maybe
"verify whether" would fit better here.

> +
> +        Generates a packet based on a specified size and sends it to the SUT. The desired packet's
> +        payload size is calculated, and arbitrary, byte-sized characters are inserted into the
> +        packet before sending. Packets are captured, and depending on the test case, packet
> +        payloads are checked to determine if the sent payload was received.
> +
> +        Args:
> +            pktsize: Size of packet to be generated and sent.
> +            should_receive: Indicate whether the test case expects to receive the packet or not.
> +        """
> +        padding = pktsize - IP_HEADER_LEN
> +        # Insert extra space for placeholder 'CRC' Error correction.
> +        packet = Ether() / Raw("    ") / IP(len=pktsize) / Raw(load="X" * padding)

This CRC error correction is interesting, I thought generally that the
Ether header length included the CRC, but it makes sense to try and
account for it if it isn't .

> +        received_packets = self.send_packet_and_capture(packet)
> +        found = any(
> +            ("X" * padding) in str(packets.load)
> +            for packets in received_packets
> +            if hasattr(packets, "load")
> +        )
> +
> +        if should_receive:
> +            self.verify(found, "Did not receive packet")
> +        else:
> +            self.verify(not found, "Received packet")
> +
<snip>
> +
> +    def test_jumboframes_jumbo_nojumbo(self) -> None:
> +        """Assess the boundaries of packets sent greater than standard MTU length.
> +
> +        PMDs are set to the standard MTU length of 1518 bytes to assess behavior of sent packets
> +        greater than this size. Sends one packet with a frame size of 1519. The test cases does

Since the bounds were increased to account for the difference in PMDs,
we should probably update this number in the doc-string accordingly.

> +        not expect to receive this packet.
> +
> +        Test:
> +            Start testpmd with standard MTU size of 1518. Send a packet of 1519 and verify it was
> +            not received.
> +        """
> +        with TestPmdShell(
> +            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
> +        ) as testpmd:
> +            testpmd.configure_port_mtu_all(ETHER_STANDARD_FRAME)
> +            testpmd.start()
> +
> +            self.send_packet_and_verify(ETHER_STANDARD_FRAME + 5, False)
> +
> +    def test_jumboframes_normal_jumbo(self) -> None:
> +        """Assess the consistency of standard 1518 byte packets using a 9000 byte jumbo MTU length.
> +
> +        PMDs are set to a jumbo frame size of 9000 bytes. Packets of sizes 1517 and 1518 are sent
> +        to assess the boundaries of packets less than or equal to the standard MTU length of 1518.
> +        The test case expects to receive both packets.
> +
> +        Test:
> +            Start testpmd with a jumbo frame size of 9000 bytes. Send a packet of 1517 and 1518
> +            and verify they were received.
> +        """
> +        with TestPmdShell(
> +            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
> +        ) as testpmd:
> +            testpmd.configure_port_mtu_all(ETHER_JUMBO_FRAME_MTU)
> +            testpmd.start()
> +
> +            self.send_packet_and_verify(ETHER_STANDARD_FRAME - 5)

Does it still make sense to take off extra bytes here since we are so
far away from the MTU boundary? I think this would be consistent still
even if it were 1517.

> +            self.send_packet_and_verify(ETHER_STANDARD_FRAME)
> +
> +    def test_jumboframes_jumbo_jumbo(self) -> None:
> +        """Assess the boundaries packets sent at an MTU size of 9000 bytes.

Should this first line have something like "Asses the boundaries with
packets..." to tie the sentence together a little more?

> +
> +        PMDs are set to a jumbo frames size of 9000 bytes. Packets of size 1519, 8999, and 9000

It would probably good to also reflect here that it is 1523, 8995 and 9000.

> +        are sent. The test expects to receive all packets.
> +
> +        Test:
> +            Start testpmd with an MTU length of 9000 bytes. Send packets of size 1519, 8999,
> +            and 9000 and verify that all packets were received.

and here as well.

> +        """
> +        with TestPmdShell(
> +            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
> +        ) as testpmd:
> +            testpmd.configure_port_mtu_all(ETHER_JUMBO_FRAME_MTU)
> +            testpmd.start()
> +
> +            self.send_packet_and_verify(ETHER_STANDARD_FRAME + 5)
> +            self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU - 5)
> +            self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU)
> +
> +    def test_jumboframes_bigger_jumbo(self) -> None:
> +        """Assess the behavior of packets send greater than a specified MTU length of 9000 bytes.

Typo: send should probably be sent.

> +
> +        PMDs are set to a jumbo frames size of 9000 bytes. A packet of size 9001 is sent to the SUT.

We should probably also reflect here that the packet size is really 9005.

> +        The test case does not expect to receive the packet.
> +
> +        Test:
> +            Start testpmd with an MTU length of 9000 bytes. Send a packet of 9001 bytes and verify

Here as well.


> +            it was not received.
> +        """
> +        with TestPmdShell(
> +            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
> +        ) as testpmd:
> +            testpmd.configure_port_mtu_all(ETHER_JUMBO_FRAME_MTU)
> +            testpmd.start()
> +
> +            self.send_packet_and_verify(ETHER_JUMBO_FRAME_MTU + 5, False)
> +
> +    def tear_down_suite(self) -> None:
> +        """Tear down the test suite.
> +
> +        Teardown:
> +            Set the MTU size of the traffic generator back to the standard 1518 byte size.
> +        """
> +        self.tg_node.main_session.configure_port_mtu(ETHER_STANDARD_FRAME, self._tg_port_egress)
> +        self.tg_node.main_session.configure_port_mtu(ETHER_STANDARD_FRAME, self._tg_port_ingress)
> --
> 2.44.0
>

Re: [RFC PATCH v3 2/2] dts: Initial Implementation For Jumbo Frames Test Suite

[Alex Chapman]

Hi,
Ive been looking into the MTU terminology and would just like to clarify 
some naming conventions and doc strings.

On 7/26/24 15:13, Nicholas Pratte wrote:
<snip>
> +IP_HEADER_LEN = 20
> +ETHER_STANDARD_FRAME = 1500
> +ETHER_JUMBO_FRAME_MTU = 9000

For these constants, I am confused why one is "FRAME" and the other is 
"MTU". The value of 'ETHER_STANDARD_FRAME' is 1500 (the standard MTU 
size), it would make sense to rename it to 'ETHER_STANDARD_MTU', to keep 
naming consistent.

If the value was 1518 instead of 1500, then `ETHER_STANDARD_FRAME` would 
be appropriate.

<snip>
> +    def test_jumboframes_normal_nojumbo(self) -> None:
> +        """Assess the boundaries of packets sent less than or equal to the standard MTU length.
> +
> +        PMDs are set to the standard MTU length of 1518 to assess behavior of sent packets less than
> +        or equal to this size. Sends two packets: one that is less than 1518 bytes, and another that
> +        is equal to 1518 bytes. The test case expects to receive both packets.
> +
> +        Test:
> +            Start testpmd and send packets of sizes 1517 and 1518.
> +        """
> +        with TestPmdShell(
> +            self.sut_node, tx_offloads=0x8000, mbuf_size=[9200], mbcache=200
> +        ) as testpmd:
> +            testpmd.configure_port_mtu_all(ETHER_STANDARD_FRAME)
> +            testpmd.start()

Renaming 'ETHER_STANDARD_FRAME' to 'ETHER_STANDARD_MTU' would reduce 
confusion here too.
e.g.
`testpmd.configure_port_mtu_all(ETHER_STANDARD_MTU)`

Additionally, you state you are sending packets of sizes 1517 and 1518. 
but you then call:
`self.send_packet_and_verify(ETHER_STANDARD_FRAME - 5)`
`self.send_packet_and_verify(ETHER_STANDARD_FRAME)`

Calculating to:
`self.send_packet_and_verify(1495)`
`self.send_packet_and_verify(1500)`

Which is confusing.
I believe this is because you are accounting for the 4 bytes of VLAN's 
in your calculations, but you might want to explain this.


Overall very solid and clean test suite, just wanted to get 
clarification on a few areas 🙂.
Alex

Re: [RFC PATCH v3 2/2] dts: Initial Implementation For Jumbo Frames Test Suite

[Nicholas Pratte]

Hi Alex, thanks for the review!

See my comments below.

<snip>
> > +IP_HEADER_LEN = 20
> > +ETHER_STANDARD_FRAME = 1500
> > +ETHER_JUMBO_FRAME_MTU = 9000
>
> For these constants, I am confused why one is "FRAME" and the other is
> "MTU". The value of 'ETHER_STANDARD_FRAME' is 1500 (the standard MTU
> size), it would make sense to rename it to 'ETHER_STANDARD_MTU', to keep
> naming consistent.
>
> If the value was 1518 instead of 1500, then `ETHER_STANDARD_FRAME` would
> be appropriate.

You are correct! I will make the changes.
>
<snip>
> Renaming 'ETHER_STANDARD_FRAME' to 'ETHER_STANDARD_MTU' would reduce
> confusion here too.
> e.g.
> `testpmd.configure_port_mtu_all(ETHER_STANDARD_MTU)`
>
> Additionally, you state you are sending packets of sizes 1517 and 1518.
> but you then call:
> `self.send_packet_and_verify(ETHER_STANDARD_FRAME - 5)`
> `self.send_packet_and_verify(ETHER_STANDARD_FRAME)`

Ack. I must have missed the docstring when I adjusted the boundaries
at which we test jumbo frame sizes ethernet overhead issues we've been
having. Adding the +5, -5 bytes is sort of a temporary
measure/placeholder while we wait to gather more information on how to
properly assess and test the ethernet overhead issue. See my next
comment for more clarification.
>
> Calculating to:
> `self.send_packet_and_verify(1495)`
> `self.send_packet_and_verify(1500)`
>
> Which is confusing.
> I believe this is because you are accounting for the 4 bytes of VLAN's
> in your calculations, but you might want to explain this.

What the +5 bytes situation really is for, currently, is to assess the
boundaries of a set MTU size, sorry for the confusion there. For
example, if we have a MTU of 1500, some vendors assume an ethernet
overhead of +22 bytes on top of the 1500 byte MTU (for a total frame
size of 1522 bytes), and other vendors, such as Mellanox, add +18
bytes of ethernet overhead +1500 byte MTU (for a total frame size of
1518). The +5 tries to compensate for this by adding +5 or -5 bytes to
properly test greater than or less than a 1500 byte MTU for all
vendors, but this gets tricky when you are trying to run tests at the
MTU size itself. If you look back to the oldest version I have of this
suite, you'll see that each test case was originally sending 1499 byte
packets, 1501 packets, and 1500 packets in some cases, but we can't
run tests like this because of the differing assumptions from vendor
to vendor (you can find the old suite in a different email thread, I
messed up the sending of the series so I apologize for that).

Here's the original version of this patch:
https://inbox.dpdk.org/dev/20240524183604.6925-2-npratte@iol.unh.edu/

The calculation of ethernet overhead basically comes down to a single
'if' statement in testpmd's code. You can find this method in
testpmd.c and do some digging if you're interested (look under
'app/test-pmd/testpmd.c' and search for 'get_eth_overhead')

>
>
> Overall very solid and clean test suite, just wanted to get
> clarification on a few areas 🙂.
> Alex

I wrote this test suite a while back when I was just starting out, and
a lot of this information was new to me at the time, so it's not
surprising to see some of the misuse of definitions you've laid out
here; I appreciate the feedback!

-Nicholas

Re: [RFC PATCH v3 0/2] Initial Implementation For Jumbo Frames

[Alex Chapman]

Hey Nick,

After going through this test suite (jumbo frames) and mtu update, it 
seems as though mtu update contains all of the test cases of jumbo 
frames and more.

To reduce the amount of maintenance and code duplication, it would make 
sense to remove this test suite and supercede it with mtu update.

If you have any questions, let me know :)

Thanks,
Alex