Re: [RFC v1 2/2] dts: port ethertype ethdev api test suite to new dts framework

[Jeremy Spewock <jspewock@iol.unh.edu>]

On Mon, Aug 5, 2024 at 1:13 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
>
> Based off the test cases from the old DTS test suite, the following test
> suite assess the behavior of ethertype configuration options found
> within the ethdev api.
>
> Bugzilla-ID: 1505
>
> depends-on: 142696 ("dts: add VLAN methods to testpmd shell")
> depends-on: 142762 ("dts: add text parser for testpmd verbose output")
> depends-on: 139227 ("dts: skip test cases based on capabilities")
>
> Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
<snip>
> +    def verify_verbose_output(
> +        self, testpmd_shell: TestPmdShell, packet: Packet, expected_output: str
> +    ) -> None:
> +        """Send a specified packet and analyse verbose testpmd output for vlan information.
> +
> +        Sends a specified packet across a paired topology. Forwarding is not checked, but verbose
> +        output is analysed to ensure that the packet's ptype matches the expected output parameter
> +        which, with respect to this test suite, will either check for standard VLAN or QinQ
> +        tagging information.
> +
> +        Args:
> +            testpmd_shell: A test case's associated testpmd shell. This is needed to stop the
> +                testpmd shell and gather verbose output for assessment.
> +            packet: The packet to be sent across the topology.
> +            expected_output: Output to be checked for within testpmd verbose output after a packet
> +                has been forwarded.
> +        """
> +        self.send_packet_and_capture(packet)
> +        verbose_output = testpmd_shell.extract_verbose_output(testpmd_shell.stop())
> +        try:
> +            self.verify(
> +                (
> +                    any(
> +                        self._sut_port_egress.mac_address in packets.src_mac
> +                        for queues in verbose_output
> +                        for packets in queues.packets
> +                    )
> +                ),
> +                "Packet not discovered in verbose output.",
> +            )
> +            self.verify(
> +                (
> +                    any(
> +                        self._sut_port_egress.mac_address in packets.src_mac
> +                        and expected_output in packets.sw_ptype
> +                        for queues in verbose_output
> +                        for packets in queues.packets
> +                    )
> +                ),
> +                f"{expected_output} not found in verbose output.",
> +            )
> +        finally:
> +            testpmd_shell.start()

Why is it that you need to start forwarding again if the test case
fails? If either of the self.verify() calls fail it should bail out of
the whole test case, so you would only need this if you were re-using
the shell between test cases which it doesn't look like is the case
from quickly scrolling through, or if you were ignoring the failures.

> +
> +    def test_change_vlan_tpid(self) -> None:
> +        """Set PMD's default tpid and assess testpmd verbose output for VLAN information.
> +
> +        Check that a PMD can properly detect that a Dot1Q layer is present on a received packet
> +        when its tpid is changed to a different value.
> +
> +        Test:
> +            Start testpmd with rxonly forwarding mode enabled.
> +            Set testpmd verbose level to 3.
> +            Send a VLAN enabled packet with a tpid value of 0xA100 across the testbed.
> +            Assess that the packet's verbose output within testpmd contains 'L2_ETHER_VLAN'.
> +        """
> +        with TestPmdShell(self.sut_node, forward_mode=SimpleForwardingModes.rxonly) as testpmd:
> +            testpmd.start()
> +            testpmd.set_verbose(3)
> +            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
> +
> +            # Send a basic packet and verify the TPID changes.
> +            packet = Ether(type=0xA100) / Dot1Q() / IP() / Raw(load="X" * 80)
> +            self.verify_verbose_output(testpmd, packet, "L2_ETHER_VLAN")

I just updated the verbose parser patch, so you should be able to use
an actual flag now instead of a string.

> +
> +    @requires(NicCapability.vlan_filter)
> +    @requires(NicCapability.vlan_extended)
> +    def test_vlan_filtering_on_off(self) -> None:
> +        """Test VLAN filter offload with variable tpid values.
> +
> +        Assesses VLAN filtering behavior when a packet is sent with both an identical and different
> +        tpid from what is set on a PMD. Behavior is validated via packet forwarding. Packets with a
> +        different tpid from the PMD should be dropped, and packet with an identical tpid should be
> +        forwarded across the testbed.
> +
> +        Test:
> +            Start testpmd with mac forwarding mode enabled.
> +            Enable both filter and extend VLAN offloading options.
> +            Send a packet with a nondefault tpid (0xA100) and verify that it was not received.
> +            Use the ethdev api to change the default tpid on the device to 0xA100.
> +            Send a packet with tpid 0xA100, and verify that it was forwarded across the testbed.
> +        """
> +        with TestPmdShell(
> +            self.sut_node,
> +            forward_mode=SimpleForwardingModes.mac,
> +        ) as testpmd:
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=True)
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.EXTEND, on=True)

If you end up making those changes in the testpmd method you can
combine these two lines into: `testpmd.set_vlan_offload_option(0,
VLANOffloadFlag.FILTER | VLANOffloadFlag.EXTEND, on=True)`

> +            testpmd.start()
> +
> +            packet = Ether(type=0xA100) / Dot1Q(vlan=16) / IP() / Raw("X" * 80)
> +            self.send_packet_and_verify(packet, should_receive=False)
> +
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=False)
> +            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
> +            self.send_packet_and_verify(packet, should_receive=True)
> +
> +    @requires(NicCapability.vlan_extended)
> +    @requires(NicCapability.vlan_strip)
> +    @requires(NicCapability.vlan_filter)
> +    def test_adding_vlan_tag_identifier_with_changing_vlan_tpid(self) -> None:
> +        """Test VLAN filter offload with vlan tagging and variable tpid values.
> +
> +        Assess VLAN filtering on tagged vlan packets with both differing and identical tpid values.
> +        Behavior is validated via packet forwarding within the testbed. A packet with a differing
> +        tpid value the device should be dropped, and packets with identical tpid values should be
> +        forwarded, regardless of any VLAN tag present. Moreover, packets with VLAN tags should
> +        perform as expected. Packets with differing VLAN tags from a device filter should be
> +        dropped, and packets with accepted VLAN tags should be forwarded.
> +
> +        Test:
> +            Start testpmd with mac forwarding mode enabled.
> +            Enable filter, strip and extend offloading features within testpmd.

It might be worth noting here that you actually disable strip rather
than enabling it.

> +            Add VLAN tag value 16 to the rx VLAN filter on testpmd.
> +            Send a packet with default tpid value (0x8100) and vlan tag 16, verify that the packet
> +                has been received.
> +            Use the ethdev api to change the device's tpid value to 0xA100.
> +            Send a packet with VLAN tag of 16 and a tpid of 0xA100 to the SUT, and verify that
> +                the packet was received.
> +            Remove VLAN tag value 16 from the rx VLAN filter on testpmd.
> +            Send a packet with VLAN tag 16 and tpid value 0xA100 and verify that it was not
> +                received.
> +        """
> +        with TestPmdShell(
> +            self.sut_node,
> +            forward_mode=SimpleForwardingModes.mac,
> +        ) as testpmd:
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=True)
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=False)
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.EXTEND, on=True)
> +            testpmd.start()
> +
> +            testpmd.rx_vlan(16, 0, add=True)
> +            packet = Ether() / Dot1Q(vlan=16) / IP() / Raw(load="X" * 80)
> +            self.send_packet_and_verify(packet, should_receive=True)
> +
> +            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
> +            packet[Ether].type = 0xA100
> +            self.send_packet_and_verify(packet, should_receive=True)
> +
> +            testpmd.rx_vlan(16, 0, add=False)
> +            self.send_packet_and_verify(packet, should_receive=False)
> +
> +    @requires(NicCapability.vlan_filter)
> +    @requires(NicCapability.vlan_strip)
> +    def test_vlan_header_stripping_with_changing_vlan_tpid(self) -> None:
> +        """Test VLAN stripping offload with changing VLAN tpid.
> +
> +        Changing tpid values should not affect the expected behavior of the VLAN stripping offload
> +        functionality. Thus, the following test case assesses that vlan tags are properly stripped
> +        despite the change in VLAN tpid values.
> +
> +        Test:
> +            Start testpmd with mac forwarding mode enabled.
> +            Enable filter and strip offloading options within testpmd.

It looks like you are actually disabling filtering in the test case.

> +            Send a VLAN tagged packet and verify it was received with Dot1Q layer removed across
> +                the testbed when VLAN filtering offload is turned off and stipping is turned on.
> +            Use the ethdev api to change the device default tpid to 0xA100.
> +            Send a packet with vlan tag value 16 and tpid value 0xA100 across the testbed and
> +                verify that it was received with Dot1Q layer removed when filter offload is
> +                disable and stripping enabled.
> +            Turn off VLAN strip offload option in testpmd.
> +            Send a packet with vlan tag value 16 and tpid value 0xA100 and verify that is
> +                received across the testbed with Dot1Q layer present.
> +        """
> +        with TestPmdShell(
> +            self.sut_node,
> +            forward_mode=SimpleForwardingModes.mac,
> +        ) as testpmd:
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.FILTER, on=False)
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=True)
> +            testpmd.start()
> +
> +            # Test that packets are received without VLAN filter
> +            packet = Ether() / Dot1Q(vlan=16) / IP() / Raw(load="X" * 80)
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packet not received during strip")
> +            self.verify(Dot1Q not in received_packets[0], "Dot1Q tag not stripped during transfer.")
> +
> +            testpmd.set_vlan_tpid(0, 0xA100, inner_id=False)
> +            packet[Ether].type = 0xA100
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packet not received during strip")
> +            self.verify(Dot1Q not in received_packets[0], "Dot1Q tag not stripped during transfer.")
> +
> +            testpmd.set_vlan_offload_option(0, VLANOffloadFlag.STRIP, on=False)
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packet not received during strip")
> +            self.verify(Dot1Q in received_packets[0], "Dot1Q tag was stripped during transfer.")
> +
<snip>
> +            testpmd.set_vlan_tpid(1, 0xA100, inner_id=False)
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packets not received.")
> +            # Separate verifications for easier debugging.
> +            self.verify(Dot1Q in received_packets[0], "Expected Dot1Q layer not found.")
> +            self.verify(
> +                received_packets[0][Ether].type == 0xA100, "Ethertype changed during transmission."
> +            )
> +            self.verify(
> +                received_packets[0][Dot1Q].vlan == 16,
> +                "Vlan tag number changed during transmission.",
> +            )

I wonder if this check for a VLAN ID is something you can bake into
send_packet_and_verify so that you can just call that a few times
instead of having to do the filtering yourself repeatedly.

> +
> +            testpmd.tx_vlan_reset(1)
> +            received_packets = [
> +                packets
> +                for packets in self.send_packet_and_capture(packet)
> +                if hasattr(packets, "load") and str(packets.load) == "X" * 80
> +            ]
> +            self.verify(len(received_packets) > 0, "Expected packets not received.")
> +            self.verify(
> +                Dot1Q not in received_packets[0].layers(), "Dot1q detected in received packet."
> +            )

If you do put the logic into send_packet_and_verify maybe this case
could be if the expected VLAN ID is set to None then there shouldn't
be one?

> +
<snip>
> 2.44.0
>