[RFC v1 0/2] dts: Ethertype ethdev api test suite

[RFC v1 0/2] dts: Ethertype ethdev api test suite

[Nicholas Pratte]

Some general input would be helpful in understanding the validity of
certain test cases within this suite. Each test case is based on the old
DTS test plan. As it currently exists, none of the NICs available for
testing (Intel, Broadcom, Mellanox) properly interface with ethdev api
functions related to tpid changes; some may not support it these
features.

There is a comment on the old DTS test plan related to extended vlans
that needs extra clarification:

"Due to the kernel enables Qinq and cannot be closed, the DPDK only add
`extend on` to make the VLAN filter work normally. Therefore, if the
i40e firmware version >= 8.4 the DPDK can only add `extend on` to make
the VLAN filter work normally"

Capabilities needed related to certain VLAN offloading functionalities,
and these are implemented within the test suite currently.

Some interesting points to highlight in writing this test suite is the
need to modify the tpid at the 'Ether' layer within Scapy, as doing so
modifies the correct tpid needed for testing; this might seem
counter-intuitive at first glance.

Nicholas Pratte (2):
  dts: add additional vlan configuration to testpmd shell class
  dts: port ethertype ethdev api test suite to new dts framework

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py |  55 +++
 dts/tests/TestSuite_ethertype_config.py       | 381 ++++++++++++++++++
 3 files changed, 438 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_ethertype_config.py

-- 
2.44.0

[RFC v1 1/2] dts: add additional vlan configuration to testpmd shell class

[Nicholas Pratte]

The ethertype test suite requires many different internal runtime
vlan offload options to test ethertype configuration. The following patch
 adds a new RxVlanOffloadOptions class that contains all the different
vlan configuration options available within testpmd. Additionally, an
extra method has beena added to adjust both the inner and outer tpids of
ingressing and egressing packets.

Bugzilla ID: 1505

Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 55 +++++++++++++++++++
 1 file changed, 55 insertions(+)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index 288be9a085..268e34f0eb 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -70,6 +70,10 @@ class VLANOffloadFlag(Flag):
     #:
     QINQ_STRIP = auto()
 
+    def __str__(self):
+        """Return flag name when cast to a string."""
+        return self.name
+
     @classmethod
     def from_str_dict(cls, d):
         """Makes an instance from a dict containing the flag member names with an "on" value.
@@ -915,6 +919,26 @@ def show_port_stats_all(self) -> list[TestPmdPortStats]:
         iter = re.finditer(r"(^  #*.+#*$[^#]+)^  #*\r$", output, re.MULTILINE)
         return [TestPmdPortStats.parse(block.group(1)) for block in iter]
 
+    def set_vlan_tpid(self, port_id: int, tpid: int, inner_id: bool) -> None:
+        """Set ethertype tpid values using the ethdev api.
+
+        Args:
+            port_id: the ID of the port the tpid is being changed on.
+            tpid: The tpid value being changed on the port.
+            inner_id: If :data:`True`, set the inner tpid to the specified value. If
+                :data:`False`, change the outer tpid to the specified value.
+
+        Raises:
+            InteractiveCommandExecutionError: If either `port_id` or `tpid` value is invalid.
+        """
+        if tpid < 0 or tpid > 65535:
+            raise InteractiveCommandExecutionError("Invalid TPID value given.")
+        output = self.send_command(
+            f'vlan set {"inner" if inner_id else "outer"} tpid {tpid} {port_id}'
+        )
+        if output.startswith("Invalid port"):
+            raise InteractiveCommandExecutionError(f"Invalid port ID {port_id} given.")
+
     def show_port_stats(self, port_id: int) -> TestPmdPortStats:
         """Returns the given port statistics.
 
@@ -978,6 +1002,37 @@ def rx_vlan(self, vlan: int, port: int, add: bool, verify: bool = True) -> None:
                     f"Testpmd failed to {'add' if add else 'remove'} tag {vlan} on port {port}."
                 )
 
+    def set_vlan_offload_option(
+        self, port: int, offload_option: VLANOffloadFlag, on: bool, verify: bool = True
+    ) -> None:
+        """Enable extended vlans on the specified port.
+
+        Args:
+            port: The port number to use, should be within 0-32.
+            on: If :data:`True`, extended vlan turn on, otherwise it turns off.
+            offload_options: The rx vlan offload option to be set.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan stripping was enabled on the specified port.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and stripping
+                fails to update.
+        """
+        offload_output = self.send_command(
+            f"vlan set {str(offload_option).lower()} {'on' if on else 'off'} {port}"
+        )
+        if verify:
+            if on ^ (str(offload_option) in str(self.show_port_info(port_id=port).vlan_offload)):
+                self._logger.debug(
+                    f"""Failed to set {offload_option} {'on' if on else 'off'} port {port}:
+                        \n{offload_output}
+                    """
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set {offload_option} {'on' if on else 'off'} port {port}."
+                )
+
     def port_stop_all(self, verify: bool = True) -> None:
         """Stop all ports.
 
-- 
2.44.0

Re: [RFC v1 1/2] dts: add additional vlan configuration to testpmd shell class

[Jeremy Spewock]

Hey Nick,

Patch looks pretty good to me, just a few minor comments below.

On Mon, Aug 5, 2024 at 1:13 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
>
> The ethertype test suite requires many different internal runtime
> vlan offload options to test ethertype configuration. The following patch
>  adds a new RxVlanOffloadOptions class that contains all the different
> vlan configuration options available within testpmd. Additionally, an
> extra method has beena added to adjust both the inner and outer tpids of

Extra "a" here at the end of "been."

> ingressing and egressing packets.
>
> Bugzilla ID: 1505
>
> Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
> ---
<snip>
> +    def set_vlan_tpid(self, port_id: int, tpid: int, inner_id: bool) -> None:

Is this one of those things that you can't really verify from testpmd
output? That makes sense if it is, but we should probably note why
this method is different in the doc-string.

> +        """Set ethertype tpid values using the ethdev api.
> +
> +        Args:
> +            port_id: the ID of the port the tpid is being changed on.

"the" should be capitalized here.

> +            tpid: The tpid value being changed on the port.
> +            inner_id: If :data:`True`, set the inner tpid to the specified value. If
> +                :data:`False`, change the outer tpid to the specified value.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If either `port_id` or `tpid` value is invalid.
> +        """
> +        if tpid < 0 or tpid > 65535:
> +            raise InteractiveCommandExecutionError("Invalid TPID value given.")
> +        output = self.send_command(
> +            f'vlan set {"inner" if inner_id else "outer"} tpid {tpid} {port_id}'
> +        )
> +        if output.startswith("Invalid port"):
> +            raise InteractiveCommandExecutionError(f"Invalid port ID {port_id} given.")
> +
<snip>
>
> +    def set_vlan_offload_option(
> +        self, port: int, offload_option: VLANOffloadFlag, on: bool, verify: bool = True
> +    ) -> None:
> +        """Enable extended vlans on the specified port.
> +
> +        Args:
> +            port: The port number to use, should be within 0-32.
> +            on: If :data:`True`, extended vlan turn on, otherwise it turns off.
> +            offload_options: The rx vlan offload option to be set.
> +            verify: If :data:`True`, the output of the command and show port info
> +                is scanned to verify that vlan stripping was enabled on the specified port.
> +                If not, it is considered an error.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If `verify` is :data:`True` and stripping
> +                fails to update.
> +        """
> +        offload_output = self.send_command(
> +            f"vlan set {str(offload_option).lower()} {'on' if on else 'off'} {port}"

Because flags can be combinations of multiple values, this might not
work if you passed in a flag that had STRIP and FILTER turned on at
the same time, for example. I think the string version would come out
to be something like STRIP|FILTER which definitely wouldn't be what
you want. Unfortunately, I think the only way around that though is to
have conditional values for each option and then checking to see if
the flag contains that option. Something like

if VLANOffloadFlag.STRIP in options:
    self.send_command(vlan set strip on 0)

Obviously with proper string formatting. Although, another thing you
might be able to do is if you can iterate through the members of the
VLANOffloadFlag class, you could do something more like:

for offload in VLANOffloadFlag.__members__:
    if offload in options:
        self.send_command(f"vlan set {offload.name.lower()} on 0")

And that would be a shorter way that covers you for any combination of
offloads. Could be fancy and make it a one-liner too if you wanted to,
but that's up to you.

> +        )
> +        if verify:
> +            if on ^ (str(offload_option) in str(self.show_port_info(port_id=port).vlan_offload)):

Is there a reason you can't directly compare the flags? The different
possible combinations of options will all have unique values, so I
think you should just be able to do `offload_option ==
self.show_port_info(port_id=port).vlan_offload)`



> +                self._logger.debug(
> +                    f"""Failed to set {offload_option} {'on' if on else 'off'} port {port}:
> +                        \n{offload_output}
> +                    """
> +                )
> +                raise InteractiveCommandExecutionError(
> +                    f"Testpmd failed to set {offload_option} {'on' if on else 'off'} port {port}."
> +                )
> +
>      def port_stop_all(self, verify: bool = True) -> None:
>          """Stop all ports.
>
> --
> 2.44.0
>

[RFC v1 2/2] dts: port ethertype ethdev api test suite to new dts framework

[Nicholas Pratte]

Based off the test cases from the old DTS test suite, the following test
suite assess the behavior of ethertype configuration options found
within the ethdev api.

Bugzilla-ID: 1505

depends-on: 142696 ("dts: add VLAN methods to testpmd shell")
depends-on: 142762 ("dts: add text parser for testpmd verbose output")
depends-on: 139227 ("dts: skip test cases based on capabilities")

Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_ethertype_config.py    | 381 +++++++++++++++++++++
 2 files changed, 383 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_ethertype_config.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..b5a9bcdf2a 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "ethertype_config"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_ethertype_config.py b/dts/tests/TestSuite_ethertype_config.py
new file mode 100644
index 0000000000..4b3e99a081
--- /dev/null
+++ b/dts/tests/TestSuite_ethertype_config.py
@@ -0,0 +1,381 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023-2024 University of New Hampshire
+
+"""VLAN tpid ethdev api assessment test suite.
+
+Assesses the behavior of Poll Mode Drivers as it relates to their ability to adequately interface
+with VLAN tpid functionalities within the ethdev api. Associated test cases within this suite may
+check and analyse both verbose output within testpmd and/or forwarding behaviors within a paired
+topology. An ethernet device should detect and forward packets as expected when using different
+tpids for both standard VLAN and extended QinQ networking. Moreover, if a Poll Mode Driver is
+configured with a different tpid, then packets with a corresponding tpid should be read properly
+within testpmd verbose output, and packets should be forwarded and not dropped.
+"""
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Dot1Q, Ether  # type: ignore[import-untyped]
+from scapy.packet import Packet, Raw  # type: ignore[import-untyped]
+
+from framework.params.testpmd import SimpleForwardingModes
+from framework.remote_session.testpmd_shell import (
+    NicCapability,
+    TestPmdShell,
+    VLANOffloadFlag,
+)
+from framework.test_suite import TestSuite, requires
+
+
+class TestEthertypeConfig(TestSuite):
+    """The vlan tpid configuration ethdev test suite for DPDK PMDs."""
+
+    def send_packet_and_verify(self, packet: Packet, should_receive: bool) -> None:
+        """Sends a specified packet and verify both reception and tpid values.
+
+        Sends a specified packet across a paired topology, and if a packet is to be received, check
+        that said packet is within the list of received packets and verify that the tpid is
+        unchanged. Otherwise, check that the specified packet was not received.
+
+        Args:
+            packet: The packet to be sent across the topology.
+            should_receive: If :data:`True', ensure that the packet was received and the Dot1Q tpid
+                is unchanged.
+        """
+        received_packets = [
+            packets
+            for packets in self.send_packet_and_capture(packet)
+            if hasattr(packets, "load") and str(packets.load) == "X" * 80
+        ]
+        if should_receive:
+            self.verify(len(received_packets) == 1, "Packet should be received.")
+            self.verify(
+                received_packets[0][Ether].type == packet[Ether].type,
+                "tpid value changed during transmission",
+            )
+        else:
+            self.verify(not received_packets, "Packet should not be received.")
+
+    def verify_verbose_output(
+        self, testpmd_shell: TestPmdShell, packet: Packet, expected_output: str
+    ) -> None:
+        """Send a specified packet and analyse verbose testpmd output for vlan information.
+
+        Sends a specified packet across a paired topology. Forwarding is not checked, but verbose
+        output is analysed to ensure that the packet's ptype matches the expected output parameter
+        which, with respect to this test suite, will either check for standard VLAN or QinQ
+        tagging information.
+
+        Args:
+            testpmd_shell: A test case's associated testpmd shell. This is needed to stop the
+                testpmd shell and gather verbose output for assessment.
+            packet: The packet to be sent across the topology.
+            expected_output: Output to be checked for within testpmd verbose output after a packet
+                has been forwarded.
+        """
+        self.send_packet_and_capture(packet)
+        verbose_output = testpmd_shell.extract_verbose_output(testpmd_shell.stop())
+        try:
+            self.verify(
+                (
+                    any(
+                        self._sut_port_egress.mac_address in packets.src_mac
+                        for queues in verbose_output
+                        for packets in queues.packets
+                    )
+                ),
+                "Packet not discovered in verbose output.",
+            )
+            self.verify(
+                (
+                    any(
+                        self._sut_port_egress.mac_address in packets.src_mac
+                        and expected_output in packets.sw_ptype
+                        for queues in verbose_output
+                        for packets in queues.packets
+                    )
+                ),
+                f"{expected_output} not found in verbose output.",
+            )
+        finally:
+            testpmd_shell.start()
+
+    def test_change_vlan_tpid(self) -> None:
+        """Set PMD's default tpid and assess testpmd verbose output for VLAN information.
+
+        Check that a PMD can properly detect that a Dot1Q layer is present on a received packet
+        when its tpid is changed to a different value.
+
+        Test:
+            Start testpmd with rxonly forwarding mode enabled.
+            Set testpmd verbose level to 3.
+            Send a VLAN enabled packet with a tpid value of 0xA100 across the testbed.
+            Assess that the packet's verbose output within testpmd contains 'L2_ETHER_VLAN'.
+        """
+        with TestPmdShell(self.sut_node, forward_mode=SimpleForwardingModes.rxonly) as testpmd:
+            testpmd.start()
+            testpmd.set_verbose(3)
+            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
+
+            # Send a basic packet and verify the TPID changes.
+            packet = Ether(type=0xA100) / Dot1Q() / IP() / Raw(load="X" * 80)
+            self.verify_verbose_output(testpmd, packet, "L2_ETHER_VLAN")
+
+    @requires(NicCapability.vlan_filter)
+    @requires(NicCapability.vlan_extended)
+    def test_vlan_filtering_on_off(self) -> None:
+        """Test VLAN filter offload with variable tpid values.
+
+        Assesses VLAN filtering behavior when a packet is sent with both an identical and different
+        tpid from what is set on a PMD. Behavior is validated via packet forwarding. Packets with a
+        different tpid from the PMD should be dropped, and packet with an identical tpid should be
+        forwarded across the testbed.
+
+        Test:
+            Start testpmd with mac forwarding mode enabled.
+            Enable both filter and extend VLAN offloading options.
+            Send a packet with a nondefault tpid (0xA100) and verify that it was not received.
+            Use the ethdev api to change the default tpid on the device to 0xA100.
+            Send a packet with tpid 0xA100, and verify that it was forwarded across the testbed.
+        """
+        with TestPmdShell(
+            self.sut_node,
+            forward_mode=SimpleForwardingModes.mac,
+        ) as testpmd:
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=True)
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.EXTEND, on=True)
+            testpmd.start()
+
+            packet = Ether(type=0xA100) / Dot1Q(vlan=16) / IP() / Raw("X" * 80)
+            self.send_packet_and_verify(packet, should_receive=False)
+
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=False)
+            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
+            self.send_packet_and_verify(packet, should_receive=True)
+
+    @requires(NicCapability.vlan_extended)
+    @requires(NicCapability.vlan_strip)
+    @requires(NicCapability.vlan_filter)
+    def test_adding_vlan_tag_identifier_with_changing_vlan_tpid(self) -> None:
+        """Test VLAN filter offload with vlan tagging and variable tpid values.
+
+        Assess VLAN filtering on tagged vlan packets with both differing and identical tpid values.
+        Behavior is validated via packet forwarding within the testbed. A packet with a differing
+        tpid value the device should be dropped, and packets with identical tpid values should be
+        forwarded, regardless of any VLAN tag present. Moreover, packets with VLAN tags should
+        perform as expected. Packets with differing VLAN tags from a device filter should be
+        dropped, and packets with accepted VLAN tags should be forwarded.
+
+        Test:
+            Start testpmd with mac forwarding mode enabled.
+            Enable filter, strip and extend offloading features within testpmd.
+            Add VLAN tag value 16 to the rx VLAN filter on testpmd.
+            Send a packet with default tpid value (0x8100) and vlan tag 16, verify that the packet
+                has been received.
+            Use the ethdev api to change the device's tpid value to 0xA100.
+            Send a packet with VLAN tag of 16 and a tpid of 0xA100 to the SUT, and verify that
+                the packet was received.
+            Remove VLAN tag value 16 from the rx VLAN filter on testpmd.
+            Send a packet with VLAN tag 16 and tpid value 0xA100 and verify that it was not
+                received.
+        """
+        with TestPmdShell(
+            self.sut_node,
+            forward_mode=SimpleForwardingModes.mac,
+        ) as testpmd:
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=True)
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=False)
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.EXTEND, on=True)
+            testpmd.start()
+
+            testpmd.rx_vlan(16, 0, add=True)
+            packet = Ether() / Dot1Q(vlan=16) / IP() / Raw(load="X" * 80)
+            self.send_packet_and_verify(packet, should_receive=True)
+
+            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
+            packet[Ether].type = 0xA100
+            self.send_packet_and_verify(packet, should_receive=True)
+
+            testpmd.rx_vlan(16, 0, add=False)
+            self.send_packet_and_verify(packet, should_receive=False)
+
+    @requires(NicCapability.vlan_filter)
+    @requires(NicCapability.vlan_strip)
+    def test_vlan_header_stripping_with_changing_vlan_tpid(self) -> None:
+        """Test VLAN stripping offload with changing VLAN tpid.
+
+        Changing tpid values should not affect the expected behavior of the VLAN stripping offload
+        functionality. Thus, the following test case assesses that vlan tags are properly stripped
+        despite the change in VLAN tpid values.
+
+        Test:
+            Start testpmd with mac forwarding mode enabled.
+            Enable filter and strip offloading options within testpmd.
+            Send a VLAN tagged packet and verify it was received with Dot1Q layer removed across
+                the testbed when VLAN filtering offload is turned off and stipping is turned on.
+            Use the ethdev api to change the device default tpid to 0xA100.
+            Send a packet with vlan tag value 16 and tpid value 0xA100 across the testbed and
+                verify that it was received with Dot1Q layer removed when filter offload is
+                disable and stripping enabled.
+            Turn off VLAN strip offload option in testpmd.
+            Send a packet with vlan tag value 16 and tpid value 0xA100 and verify that is
+                received across the testbed with Dot1Q layer present.
+        """
+        with TestPmdShell(
+            self.sut_node,
+            forward_mode=SimpleForwardingModes.mac,
+        ) as testpmd:
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=False)
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=True)
+            testpmd.start()
+
+            # Test that packets are received without VLAN filter
+            packet = Ether() / Dot1Q(vlan=16) / IP() / Raw(load="X" * 80)
+            received_packets = [
+                packets
+                for packets in self.send_packet_and_capture(packet)
+                if hasattr(packets, "load") and str(packets.load) == "X" * 80
+            ]
+            self.verify(len(received_packets) > 0, "Expected packet not received during strip")
+            self.verify(Dot1Q not in received_packets[0], "Dot1Q tag not stripped during transfer.")
+
+            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
+            packet[Ether].type = 0xA100
+            received_packets = [
+                packets
+                for packets in self.send_packet_and_capture(packet)
+                if hasattr(packets, "load") and str(packets.load) == "X" * 80
+            ]
+            self.verify(len(received_packets) > 0, "Expected packet not received during strip")
+            self.verify(Dot1Q not in received_packets[0], "Dot1Q tag not stripped during transfer.")
+
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=False)
+            received_packets = [
+                packets
+                for packets in self.send_packet_and_capture(packet)
+                if hasattr(packets, "load") and str(packets.load) == "X" * 80
+            ]
+            self.verify(len(received_packets) > 0, "Expected packet not received during strip")
+            self.verify(Dot1Q in received_packets[0], "Dot1Q tag was stripped during transfer.")
+
+    @requires(NicCapability.vlan_filter)
+    @requires(NicCapability.vlan_strip)
+    def test_vlan_header_inserting_with_changing_vlan_tpid(self) -> None:
+        """Test tx VLAN behavior with varying tpid values.
+
+        Test tx Dot1Q layer insertion behavior when the default tpid value is changed using the
+        ethdev api. An ethernet device should insert Dot1Q layers on egressing packet regardless of
+        the tpid value set, and egressing packets should contain the tpid value that was explicitly
+        set.
+
+        Test:
+            Start testpmd with mac forwarding mode enabled.
+            Add tx VLAN value of 16 using testpmd.
+            Send a packet without a Dot1Q layer across the testbed, and verify that the packet was
+                received with a Dot1Q layer present, a VLAN tag value of 16, and a tpid value of
+                0x8100.
+            Use the ethdev api to change the default device tpid value to 0xA100
+            Send a packet without a Dot1Q layer across the testbed, and verify that the packet was
+                received with a Dot1Q layer present, a VLAN tag value of 16, and a tpid value of
+                0xA100.
+            Reset tx VLAN options to its default state using testpmd.
+            Send a packet without a Dot1Q layer across the testbed and verify that it is received
+                without a Dot1Q layer present.
+        """
+        with TestPmdShell(self.sut_node, forward_mode=SimpleForwardingModes.mac) as testpmd:
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=False)
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=False)
+            testpmd.start()
+
+            testpmd.tx_vlan_set(1, vlan=16)
+            packet = Ether() / IP() / Raw(load="X" * 80)
+            received_packets = list(
+                packets
+                for packets in self.send_packet_and_capture(packet)
+                if hasattr(packets, "load") and str(packets.load) == "X" * 80
+            )
+            self.verify(len(received_packets) > 0, "Expected packets not received.")
+            # Separate verifications for easier debugging.
+            self.verify(Dot1Q in received_packets[0], "Expected Dot1Q layer not found.")
+            self.verify(
+                received_packets[0][Ether].type == 0x8100, "Ethertype changed during transmission."
+            )
+            self.verify(
+                received_packets[0][Dot1Q].vlan == 16,
+                "Vlan tag number changed during transmission.",
+            )
+
+            testpmd.set_vlan_tpid(1, 0xA100, inner_id=False)
+            received_packets = [
+                packets
+                for packets in self.send_packet_and_capture(packet)
+                if hasattr(packets, "load") and str(packets.load) == "X" * 80
+            ]
+            self.verify(len(received_packets) > 0, "Expected packets not received.")
+            # Separate verifications for easier debugging.
+            self.verify(Dot1Q in received_packets[0], "Expected Dot1Q layer not found.")
+            self.verify(
+                received_packets[0][Ether].type == 0xA100, "Ethertype changed during transmission."
+            )
+            self.verify(
+                received_packets[0][Dot1Q].vlan == 16,
+                "Vlan tag number changed during transmission.",
+            )
+
+            testpmd.tx_vlan_reset(1)
+            received_packets = [
+                packets
+                for packets in self.send_packet_and_capture(packet)
+                if hasattr(packets, "load") and str(packets.load) == "X" * 80
+            ]
+            self.verify(len(received_packets) > 0, "Expected packets not received.")
+            self.verify(
+                Dot1Q not in received_packets[0].layers(), "Dot1q detected in received packet."
+            )
+
+    @requires(NicCapability.qinq_strip)
+    def change_stag_and_ctag_within_qinq(self) -> None:
+        """Assess QinQ packet recognition with varying c_tag and s_tag values.
+
+        Assesses testpmd verbose output to validate the proper functionality of QinQ packets when
+        inner and outer tpid values are changed. If a sent QinQ packet's tpid values are changed to
+        correspond to the values that are set within testpmd, then that sent packet should be
+        acknowledged and the expected standard VLAN and QinQ verbose output should be present
+        within testpmd.
+
+        Test:
+            Start testpmd with rxonly forwarding mode enabled.
+            Enable qinq_strip VLAN offload option within testpmd.
+            Set testpmd verbose level to 3.
+            Use the ethdev api to set the default outer tpid value to 0x88A8.
+            Use the ethdev api to set the default inner tpid value to 0x8100.
+            Send a QinQ packet with an outer tpid value of 0x88A8 and an inner tpid value of 0x8100
+                and verify that 'L2_ETHER_VLAN INNER_L2_ETHER_QINQ' is present in verbose output.
+            Use the ethdev api to set the default outer tpid value to 0x9100.
+            Use the ethdev api to set the default inner tpid value to 0xA100.
+            Send a QinQ packet with an outer tpid value of 0x9100 and an inner tpid value of 0xA100
+                and verify that 'L2_ETHER_VLAN INNER_L2_ETHER_QINQ' is present in verbose output.
+            Use the ethdev api to set the default outer tpid value to 0x8100.
+            Use the ethdev api to set the default inner tpid value to 0x8100.
+            Send a QinQ packet with an outer tpid value of 0x8100 and an inner tpid value of 0x8100
+                and verify that 'L2_ETHER_VLAN INNER_L2_ETHER_QINQ' is present in verbose output.
+        """
+        with TestPmdShell(self.sut_node, forward_mode=SimpleForwardingModes.rxonly) as testpmd:
+            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.QINQ_STRIP, on=True)
+            testpmd.set_verbose(3)
+            testpmd.start()
+
+            testpmd.set_vlan_tpid(0, 0x88A8, inner_id=False)
+            testpmd.set_vlan_tpid(0, 0x8100, inner_id=True)
+            packet = Ether(type=0x88A8) / Dot1Q(type=0x8100) / Dot1Q() / IP() / Raw(load="X" * 80)
+            self.verify_verbose_output(testpmd, packet, "L2_ETHER_VLAN INNER_L2_ETHER_QINQ")
+
+            testpmd.set_vlan_tpid(0, 0x9100, inner_id=False)
+            testpmd.set_vlan_tpid(0, 0xA100, inner_id=True)
+            packet[Ether].type = 0x9100
+            packet[Dot1Q][0].type = 0xA100
+            self.verify_verbose_output(testpmd, packet, "L2_ETHER_VLAN INNER_L2_ETHER_QINQ")
+
+            testpmd.set_vlan_tpid(0, 0x8100, inner_id=False)
+            testpmd.set_vlan_tpid(0, 0x8100, inner_id=True)
+            packet[Ether].type = 0x8100
+            packet[Dot1Q][0].type = 0x8100
+            self.verify_verbose_output(testpmd, packet, "L2_ETHER_VLAN INNER_L2_ETHER_QINQ")
-- 
2.44.0

Re: [RFC v1 2/2] dts: port ethertype ethdev api test suite to new dts framework

[Jeremy Spewock]

On Mon, Aug 5, 2024 at 1:13 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
>
> Based off the test cases from the old DTS test suite, the following test
> suite assess the behavior of ethertype configuration options found
> within the ethdev api.
>
> Bugzilla-ID: 1505
>
> depends-on: 142696 ("dts: add VLAN methods to testpmd shell")
> depends-on: 142762 ("dts: add text parser for testpmd verbose output")
> depends-on: 139227 ("dts: skip test cases based on capabilities")
>
> Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
<snip>
> +    def verify_verbose_output(
> +        self, testpmd_shell: TestPmdShell, packet: Packet, expected_output: str
> +    ) -> None:
> +        """Send a specified packet and analyse verbose testpmd output for vlan information.
> +
> +        Sends a specified packet across a paired topology. Forwarding is not checked, but verbose
> +        output is analysed to ensure that the packet's ptype matches the expected output parameter
> +        which, with respect to this test suite, will either check for standard VLAN or QinQ
> +        tagging information.
> +
> +        Args:
> +            testpmd_shell: A test case's associated testpmd shell. This is needed to stop the
> +                testpmd shell and gather verbose output for assessment.
> +            packet: The packet to be sent across the topology.
> +            expected_output: Output to be checked for within testpmd verbose output after a packet
> +                has been forwarded.
> +        """
> +        self.send_packet_and_capture(packet)
> +        verbose_output = testpmd_shell.extract_verbose_output(testpmd_shell.stop())
> +        try:
> +            self.verify(
> +                (
> +                    any(
> +                        self._sut_port_egress.mac_address in packets.src_mac
> +                        for queues in verbose_output
> +                        for packets in queues.packets
> +                    )
> +                ),
> +                "Packet not discovered in verbose output.",
> +            )
> +            self.verify(
> +                (
> +                    any(
> +                        self._sut_port_egress.mac_address in packets.src_mac
> +                        and expected_output in packets.sw_ptype
> +                        for queues in verbose_output
> +                        for packets in queues.packets
> +                    )
> +                ),
> +                f"{expected_output} not found in verbose output.",
> +            )
> +        finally:
> +            testpmd_shell.start()

Why is it that you need to start forwarding again if the test case
fails? If either of the self.verify() calls fail it should bail out of
the whole test case, so you would only need this if you were re-using
the shell between test cases which it doesn't look like is the case
from quickly scrolling through, or if you were ignoring the failures.

> +
> +    def test_change_vlan_tpid(self) -> None:
> +        """Set PMD's default tpid and assess testpmd verbose output for VLAN information.
> +
> +        Check that a PMD can properly detect that a Dot1Q layer is present on a received packet
> +        when its tpid is changed to a different value.
> +
> +        Test:
> +            Start testpmd with rxonly forwarding mode enabled.
> +            Set testpmd verbose level to 3.
> +            Send a VLAN enabled packet with a tpid value of 0xA100 across the testbed.
> +            Assess that the packet's verbose output within testpmd contains 'L2_ETHER_VLAN'.
> +        """
> +        with TestPmdShell(self.sut_node, forward_mode=SimpleForwardingModes.rxonly) as testpmd:
> +            testpmd.start()
> +            testpmd.set_verbose(3)
> +            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
> +
> +            # Send a basic packet and verify the TPID changes.
> +            packet = Ether(type=0xA100) / Dot1Q() / IP() / Raw(load="X" * 80)
> +            self.verify_verbose_output(testpmd, packet, "L2_ETHER_VLAN")

I just updated the verbose parser patch, so you should be able to use
an actual flag now instead of a string.

> +
> +    @requires(NicCapability.vlan_filter)
> +    @requires(NicCapability.vlan_extended)
> +    def test_vlan_filtering_on_off(self) -> None:
> +        """Test VLAN filter offload with variable tpid values.
> +
> +        Assesses VLAN filtering behavior when a packet is sent with both an identical and different
> +        tpid from what is set on a PMD. Behavior is validated via packet forwarding. Packets with a
> +        different tpid from the PMD should be dropped, and packet with an identical tpid should be
> +        forwarded across the testbed.
> +
> +        Test:
> +            Start testpmd with mac forwarding mode enabled.
> +            Enable both filter and extend VLAN offloading options.
> +            Send a packet with a nondefault tpid (0xA100) and verify that it was not received.
> +            Use the ethdev api to change the default tpid on the device to 0xA100.
> +            Send a packet with tpid 0xA100, and verify that it was forwarded across the testbed.
> +        """
> +        with TestPmdShell(
> +            self.sut_node,
> +            forward_mode=SimpleForwardingModes.mac,
> +        ) as testpmd:
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=True)
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.EXTEND, on=True)

If you end up making those changes in the testpmd method you can
combine these two lines into: `testpmd.set_vlan_offload_option(0,
VLANOffloadFlag.FILTER | VLANOffloadFlag.EXTEND, on=True)`

> +            testpmd.start()
> +
> +            packet = Ether(type=0xA100) / Dot1Q(vlan=16) / IP() / Raw("X" * 80)
> +            self.send_packet_and_verify(packet, should_receive=False)
> +
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=False)
> +            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
> +            self.send_packet_and_verify(packet, should_receive=True)
> +
> +    @requires(NicCapability.vlan_extended)
> +    @requires(NicCapability.vlan_strip)
> +    @requires(NicCapability.vlan_filter)
> +    def test_adding_vlan_tag_identifier_with_changing_vlan_tpid(self) -> None:
> +        """Test VLAN filter offload with vlan tagging and variable tpid values.
> +
> +        Assess VLAN filtering on tagged vlan packets with both differing and identical tpid values.
> +        Behavior is validated via packet forwarding within the testbed. A packet with a differing
> +        tpid value the device should be dropped, and packets with identical tpid values should be
> +        forwarded, regardless of any VLAN tag present. Moreover, packets with VLAN tags should
> +        perform as expected. Packets with differing VLAN tags from a device filter should be
> +        dropped, and packets with accepted VLAN tags should be forwarded.
> +
> +        Test:
> +            Start testpmd with mac forwarding mode enabled.
> +            Enable filter, strip and extend offloading features within testpmd.

It might be worth noting here that you actually disable strip rather
than enabling it.

> +            Add VLAN tag value 16 to the rx VLAN filter on testpmd.
> +            Send a packet with default tpid value (0x8100) and vlan tag 16, verify that the packet
> +                has been received.
> +            Use the ethdev api to change the device's tpid value to 0xA100.
> +            Send a packet with VLAN tag of 16 and a tpid of 0xA100 to the SUT, and verify that
> +                the packet was received.
> +            Remove VLAN tag value 16 from the rx VLAN filter on testpmd.
> +            Send a packet with VLAN tag 16 and tpid value 0xA100 and verify that it was not
> +                received.
> +        """
> +        with TestPmdShell(
> +            self.sut_node,
> +            forward_mode=SimpleForwardingModes.mac,
> +        ) as testpmd:
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=True)
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=False)
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.EXTEND, on=True)
> +            testpmd.start()
> +
> +            testpmd.rx_vlan(16, 0, add=True)
> +            packet = Ether() / Dot1Q(vlan=16) / IP() / Raw(load="X" * 80)
> +            self.send_packet_and_verify(packet, should_receive=True)
> +
> +            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
> +            packet[Ether].type = 0xA100
> +            self.send_packet_and_verify(packet, should_receive=True)
> +
> +            testpmd.rx_vlan(16, 0, add=False)
> +            self.send_packet_and_verify(packet, should_receive=False)
> +
> +    @requires(NicCapability.vlan_filter)
> +    @requires(NicCapability.vlan_strip)
> +    def test_vlan_header_stripping_with_changing_vlan_tpid(self) -> None:
> +        """Test VLAN stripping offload with changing VLAN tpid.
> +
> +        Changing tpid values should not affect the expected behavior of the VLAN stripping offload
> +        functionality. Thus, the following test case assesses that vlan tags are properly stripped
> +        despite the change in VLAN tpid values.
> +
> +        Test:
> +            Start testpmd with mac forwarding mode enabled.
> +            Enable filter and strip offloading options within testpmd.

It looks like you are actually disabling filtering in the test case.

> +            Send a VLAN tagged packet and verify it was received with Dot1Q layer removed across
> +                the testbed when VLAN filtering offload is turned off and stipping is turned on.
> +            Use the ethdev api to change the device default tpid to 0xA100.
> +            Send a packet with vlan tag value 16 and tpid value 0xA100 across the testbed and
> +                verify that it was received with Dot1Q layer removed when filter offload is
> +                disable and stripping enabled.
> +            Turn off VLAN strip offload option in testpmd.
> +            Send a packet with vlan tag value 16 and tpid value 0xA100 and verify that is
> +                received across the testbed with Dot1Q layer present.
> +        """
> +        with TestPmdShell(
> +            self.sut_node,
> +            forward_mode=SimpleForwardingModes.mac,
> +        ) as testpmd:
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=False)
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=True)
> +            testpmd.start()
> +
> +            # Test that packets are received without VLAN filter
> +            packet = Ether() / Dot1Q(vlan=16) / IP() / Raw(load="X" * 80)
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packet not received during strip")
> +            self.verify(Dot1Q not in received_packets[0], "Dot1Q tag not stripped during transfer.")
> +
> +            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
> +            packet[Ether].type = 0xA100
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packet not received during strip")
> +            self.verify(Dot1Q not in received_packets[0], "Dot1Q tag not stripped during transfer.")
> +
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=False)
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packet not received during strip")
> +            self.verify(Dot1Q in received_packets[0], "Dot1Q tag was stripped during transfer.")
> +
<snip>
> +            testpmd.set_vlan_tpid(1, 0xA100, inner_id=False)
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packets not received.")
> +            # Separate verifications for easier debugging.
> +            self.verify(Dot1Q in received_packets[0], "Expected Dot1Q layer not found.")
> +            self.verify(
> +                received_packets[0][Ether].type == 0xA100, "Ethertype changed during transmission."
> +            )
> +            self.verify(
> +                received_packets[0][Dot1Q].vlan == 16,
> +                "Vlan tag number changed during transmission.",
> +            )

I wonder if this check for a VLAN ID is something you can bake into
send_packet_and_verify so that you can just call that a few times
instead of having to do the filtering yourself repeatedly.

> +
> +            testpmd.tx_vlan_reset(1)
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packets not received.")
> +            self.verify(
> +                Dot1Q not in received_packets[0].layers(), "Dot1q detected in received packet."
> +            )

If you do put the logic into send_packet_and_verify maybe this case
could be if the expected VLAN ID is set to None then there shouldn't
be one?

> +
<snip>
> 2.44.0
>