[RFC v1 0/3] VXLAN-GPE test suite

[RFC v1 0/3] VXLAN-GPE test suite

[Dean Marx]

This test suite and the supporting testpmd shell changes originate from
an old dts test plan called "vxlan_gpe_support_in_i40e", and the
necessary testpmd commands to run the suite are only fully supported on
the i40e NICs I have tested on (bnxt and mlx5 NICs don't support.)
However, VXLAN-GPE tunnelling is technically supported in the ethdev
API, so I decided to write it anyways.

The suite consists originally consists of two test cases, but I
compressed them into one where the udp_tunnel_port command is used to
add VXLAN-GPE port 4790 to the filter list on testpmd. To my
understanding, A UDP / VXLAN packet with dport 4790 is sent and captured
to verify the VXLAN layer is still in the received packet. 
Then, the port 4790 tunnel is removed from the session and the same
packet is sent, but when received should not have the VXLAN layer. I am
unsure based on the test plan description whether the VXLAN layer is
fully removed, or if the verbose output on testpmd just prints the VXLAN
layer in stdout in the first case and not the second, but I decided to
implement the former for now.


Dean Marx (3):
  dts: add UDP tunnel command to testpmd shell
  dts: VXLAN gpe support test suite
  dts: conf schema VXLAN gpe support

 dts/framework/config/conf_yaml_schema.json    |  3 +-
 dts/framework/remote_session/testpmd_shell.py | 51 +++++++++++-
 dts/tests/TestSuite_vxlan_gpe_support.py      | 77 +++++++++++++++++++
 3 files changed, 129 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_vxlan_gpe_support.py

-- 
2.44.0

[RFC v1 1/3] dts: add UDP tunnel command to testpmd shell

[Dean Marx]

add udp_tunnel_port command to testpmd shell class,
also ports over set verbose method from vlan suite

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 51 ++++++++++++++++++-
 1 file changed, 50 insertions(+), 1 deletion(-)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index eda6eb320f..26114091d6 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -804,7 +804,56 @@ def show_port_stats(self, port_id: int) -> TestPmdPortStats:
 
         return TestPmdPortStats.parse(output)
 
-    def _close(self) -> None:
+    def set_verbose(self, level: int, verify: bool = True):
+        """Set debug verbosity level.
+
+        Args:
+            level: 0 - silent except for error
+                1 - fully verbose except for Tx packets
+                2 - fully verbose except for Rx packets
+                >2 - fully verbose
+            verify: If :data:`True` the command output will be scanned to verify that verbose level
+                is properly set. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and verbose level
+            is not correctly set.
+        """
+        verbose_output = self.send_command(f"set verbose {level}")
+        if verify:
+            if "Change verbose level" not in verbose_output:
+                self._logger.debug(f"Failed to set verbose level to {level}: \n{verbose_output}")
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set verbose level to {level}."
+                )
+
+    def udp_tunnel_port(
+        self, port_id: int, add: bool, udp_port: int, protocol: str, verify: bool = True
+    ):
+        """Configures a UDP tunnel on the specified port, for the specified protocol.
+
+        Args:
+            port_id: ID of the port to configure tunnel on.
+            add: If :data:`True`, adds tunnel, otherwise removes tunnel.
+            udp_port: ID of the UDP port to configure tunnel on.
+            protocol: Name of tunnelling protocol to use; options are vxlan, geneve, ecpri
+            verify: If :data:`True`, checks the output of the command to verify that
+                no errors were thrown.
+
+        Raises:
+            InteractiveCommandExecutionError: If verify is :data:`True` and command
+                output shows an error.
+        """
+        action = "add" if add else "rm"
+        cmd_output = self.send_command(
+            f"port config {port_id} udp_tunnel_port {action} {protocol} {udp_port}"
+        )
+        if verify:
+            if "Operation not supported" in cmd_output or "Bad arguments" in cmd_output:
+                self._logger.debug(f"Failed to set UDP tunnel: \n{cmd_output}")
+                raise InteractiveCommandExecutionError(f"Failed to set UDP tunnel: \n{cmd_output}")
+
+    def close(self) -> None:
         """Overrides :meth:`~.interactive_shell.close`."""
         self.stop()
         self.send_command("quit", "Bye...")
-- 
2.44.0

Re: [RFC v1 1/3] dts: add UDP tunnel command to testpmd shell

[Jeremy Spewock]

Hey Dean, these changes look good to me, I just had a few minor
comments/suggestions.

One thing I did notice was that the methods added here don't have
type-hints for their return-types, obviously functionally it makes no
difference since they don't return anything, but just adding the note
that says the return None is helpful for type checkers and
understanding the method at a glance.

On Thu, Jul 25, 2024 at 12:23 PM Dean Marx <dmarx@iol.unh.edu> wrote:
>
> add udp_tunnel_port command to testpmd shell class,
> also ports over set verbose method from vlan suite
>
> Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
> ---
>  dts/framework/remote_session/testpmd_shell.py | 51 ++++++++++++++++++-
>  1 file changed, 50 insertions(+), 1 deletion(-)
>
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index eda6eb320f..26114091d6 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -804,7 +804,56 @@ def show_port_stats(self, port_id: int) -> TestPmdPortStats:
>
>          return TestPmdPortStats.parse(output)
>
> -    def _close(self) -> None:

It looks like this method might have been renamed by mistake in a
rebase, the name on main right now is _close. This could cause some
weird behavior in your testing since this is what the context manager
uses to close the session, but I don't think it would have any drastic
effect since the channel is still closed.

> +    def set_verbose(self, level: int, verify: bool = True):
> +        """Set debug verbosity level.
> +
> +        Args:
> +            level: 0 - silent except for error
> +                1 - fully verbose except for Tx packets
> +                2 - fully verbose except for Rx packets
> +                >2 - fully verbose
> +            verify: If :data:`True` the command output will be scanned to verify that verbose level
> +                is properly set. Defaults to :data:`True`.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If `verify` is :data:`True` and verbose level
> +            is not correctly set.
> +        """
> +        verbose_output = self.send_command(f"set verbose {level}")
> +        if verify:
> +            if "Change verbose level" not in verbose_output:
> +                self._logger.debug(f"Failed to set verbose level to {level}: \n{verbose_output}")
> +                raise InteractiveCommandExecutionError(
> +                    f"Testpmd failed to set verbose level to {level}."
> +                )
> +
> +    def udp_tunnel_port(
> +        self, port_id: int, add: bool, udp_port: int, protocol: str, verify: bool = True
> +    ):
> +        """Configures a UDP tunnel on the specified port, for the specified protocol.
> +
> +        Args:
> +            port_id: ID of the port to configure tunnel on.
> +            add: If :data:`True`, adds tunnel, otherwise removes tunnel.
> +            udp_port: ID of the UDP port to configure tunnel on.
> +            protocol: Name of tunnelling protocol to use; options are vxlan, geneve, ecpri

If there are explicit choices that this has to be like this it might
be better to put these options into an enum and then pass that in as
the parameter here. That way it is very clear from just calling the
methods what your options are.



> +            verify: If :data:`True`, checks the output of the command to verify that
> +                no errors were thrown.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If verify is :data:`True` and command
> +                output shows an error.
> +        """
> +        action = "add" if add else "rm"
> +        cmd_output = self.send_command(
> +            f"port config {port_id} udp_tunnel_port {action} {protocol} {udp_port}"
> +        )
> +        if verify:
> +            if "Operation not supported" in cmd_output or "Bad arguments" in cmd_output:
> +                self._logger.debug(f"Failed to set UDP tunnel: \n{cmd_output}")
> +                raise InteractiveCommandExecutionError(f"Failed to set UDP tunnel: \n{cmd_output}")
> +
> +    def close(self) -> None:
>          """Overrides :meth:`~.interactive_shell.close`."""
>          self.stop()
>          self.send_command("quit", "Bye...")
> --
> 2.44.0
>

[RFC v1 2/3] dts: VXLAN gpe support test suite

[Dean Marx]

Test suite for verifying vxlan gpe support on NIC, as well as expected
behavior while sending vxlan packets through tunnel

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/tests/TestSuite_vxlan_gpe_support.py | 77 ++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
 create mode 100644 dts/tests/TestSuite_vxlan_gpe_support.py

diff --git a/dts/tests/TestSuite_vxlan_gpe_support.py b/dts/tests/TestSuite_vxlan_gpe_support.py
new file mode 100644
index 0000000000..981f878a4c
--- /dev/null
+++ b/dts/tests/TestSuite_vxlan_gpe_support.py
@@ -0,0 +1,77 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 University of New Hampshire
+
+"""VXLAN-GPE support test suite.
+
+This suite verifies virtual extensible local area network packets
+are only received in the same state when a UDP tunnel port for VXLAN tunneling
+protocols is enabled. GPE is the Generic Protocol Extension for VXLAN,
+which is used for configuring fields in the VXLAN header through GPE tunnels.
+
+If a GPE tunnel is configured for the corresponding UDP port within a sent packet,
+that packet should be received with its VXLAN layer. If there is no GPE tunnel,
+the packet should be received without its VXLAN layer.
+
+"""
+
+from scapy.layers.inet import IP, UDP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.layers.vxlan import VXLAN  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.params.testpmd import SimpleForwardingModes
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+
+class TestVxlanGpeSupport(TestSuite):
+    """DPDK VXLAN-GPE test suite.
+
+    This suite consists of one test case (Port 4790 is designated for VXLAN-GPE streams):
+    1. VXLAN-GPE ipv4 packet detect - configures a GPE tunnel on port 4790
+        and sends packets with a matching UDP destination port. This packet
+        should be received by the traffic generator with its VXLAN layer.
+        Then, remove the GPE tunnel, send the same packet, and verify that
+        the packet is received without its VXLAN layer.
+    """
+
+    def set_up_suite(self) -> None:
+        """Set up the test suite.
+
+        Setup:
+            Verify that we have at least 2 port links in the current test run.
+        """
+        self.verify(
+            len(self._port_links) > 1,
+            "There must be at least two port links to run the scatter test suite",
+        )
+
+    def send_vxlan_packet_and_verify(self, udp_dport: int, should_receive_vxlan: bool) -> None:
+        """Generate a VXLAN GPE packet with the given UDP destination port, send and verify.
+
+        Args:
+            udp_dport: The destination UDP port to generate in the packet.
+            should_receive_vxlan: Indicates whether the packet should be
+                received by the traffic generator with its VXLAN layer.
+        """
+        packet = Ether() / IP() / UDP(dport=udp_dport) / VXLAN(flags=12) / IP() / Raw(load="xxxxx")
+        received = self.send_packet_and_capture(packet)
+        print(f"Received packets = {received}")
+        has_vxlan = any(
+            "VXLAN" in packet.summary() and "xxxxx" in str(packet.load) for packet in received
+        )
+        self.verify(
+            not (has_vxlan ^ should_receive_vxlan), "Expected packet did not match received packet."
+        )
+
+    def test_gpe_tunneling(self) -> None:
+        """Verifies expected behavior of VXLAN packets through a GPE tunnel."""
+        GPE_port = 4790
+        with TestPmdShell(node=self.sut_node) as testpmd:
+            testpmd.set_forward_mode(SimpleForwardingModes.io)
+            testpmd.set_verbose(level=1)
+            testpmd.start()
+            testpmd.udp_tunnel_port(port_id=0, add=True, udp_port=GPE_port, protocol="vxlan")
+            self.send_vxlan_packet_and_verify(udp_dport=GPE_port, should_receive_vxlan=True)
+            testpmd.udp_tunnel_port(port_id=0, add=False, udp_port=GPE_port, protocol="vxlan")
+            self.send_vxlan_packet_and_verify(udp_dport=GPE_port, should_receive_vxlan=False)
-- 
2.44.0

Re: [RFC v1 2/3] dts: VXLAN gpe support test suite

[Jeremy Spewock]

This all makes sense to me and looks good to me, I just had one
suggestion about verification below.

On Thu, Jul 25, 2024 at 12:23 PM Dean Marx <dmarx@iol.unh.edu> wrote:
>
> Test suite for verifying vxlan gpe support on NIC, as well as expected
> behavior while sending vxlan packets through tunnel
>
> Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
> ---
>  dts/tests/TestSuite_vxlan_gpe_support.py | 77 ++++++++++++++++++++++++
>  1 file changed, 77 insertions(+)
>  create mode 100644 dts/tests/TestSuite_vxlan_gpe_support.py
>
> diff --git a/dts/tests/TestSuite_vxlan_gpe_support.py b/dts/tests/TestSuite_vxlan_gpe_support.py
> new file mode 100644
> index 0000000000..981f878a4c
> --- /dev/null
> +++ b/dts/tests/TestSuite_vxlan_gpe_support.py
> @@ -0,0 +1,77 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 University of New Hampshire
> +
> +"""VXLAN-GPE support test suite.
> +
> +This suite verifies virtual extensible local area network packets
> +are only received in the same state when a UDP tunnel port for VXLAN tunneling
> +protocols is enabled. GPE is the Generic Protocol Extension for VXLAN,
> +which is used for configuring fields in the VXLAN header through GPE tunnels.
> +
> +If a GPE tunnel is configured for the corresponding UDP port within a sent packet,
> +that packet should be received with its VXLAN layer. If there is no GPE tunnel,
> +the packet should be received without its VXLAN layer.
> +
> +"""
> +
> +from scapy.layers.inet import IP, UDP  # type: ignore[import-untyped]
> +from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
> +from scapy.layers.vxlan import VXLAN  # type: ignore[import-untyped]
> +from scapy.packet import Raw  # type: ignore[import-untyped]
> +
> +from framework.params.testpmd import SimpleForwardingModes
> +from framework.remote_session.testpmd_shell import TestPmdShell
> +from framework.test_suite import TestSuite
> +
> +
> +class TestVxlanGpeSupport(TestSuite):
> +    """DPDK VXLAN-GPE test suite.
> +
> +    This suite consists of one test case (Port 4790 is designated for VXLAN-GPE streams):
> +    1. VXLAN-GPE ipv4 packet detect - configures a GPE tunnel on port 4790
> +        and sends packets with a matching UDP destination port. This packet
> +        should be received by the traffic generator with its VXLAN layer.
> +        Then, remove the GPE tunnel, send the same packet, and verify that
> +        the packet is received without its VXLAN layer.
> +    """
> +
> +    def set_up_suite(self) -> None:
> +        """Set up the test suite.
> +
> +        Setup:
> +            Verify that we have at least 2 port links in the current test run.
> +        """
> +        self.verify(
> +            len(self._port_links) > 1,
> +            "There must be at least two port links to run the scatter test suite",
> +        )
> +
> +    def send_vxlan_packet_and_verify(self, udp_dport: int, should_receive_vxlan: bool) -> None:
> +        """Generate a VXLAN GPE packet with the given UDP destination port, send and verify.
> +
> +        Args:
> +            udp_dport: The destination UDP port to generate in the packet.
> +            should_receive_vxlan: Indicates whether the packet should be
> +                received by the traffic generator with its VXLAN layer.
> +        """
> +        packet = Ether() / IP() / UDP(dport=udp_dport) / VXLAN(flags=12) / IP() / Raw(load="xxxxx")
> +        received = self.send_packet_and_capture(packet)
> +        print(f"Received packets = {received}")
> +        has_vxlan = any(
> +            "VXLAN" in packet.summary() and "xxxxx" in str(packet.load) for packet in received

Scapy actually allows for checking if a layer exists in a packet using
the real types in a few different ways. You could use
packet.haslayer(VXLAN), or you could do basically the same as what you
have without the string comparison if you do `VXLAN in packet`. This
might end up being a little shorter and it saves you from having to
deal with the string provided from the packet summary.


> +        )
> +        self.verify(
> +            not (has_vxlan ^ should_receive_vxlan), "Expected packet did not match received packet."
> +        )
> +
> +    def test_gpe_tunneling(self) -> None:
> +        """Verifies expected behavior of VXLAN packets through a GPE tunnel."""
> +        GPE_port = 4790
> +        with TestPmdShell(node=self.sut_node) as testpmd:
> +            testpmd.set_forward_mode(SimpleForwardingModes.io)
> +            testpmd.set_verbose(level=1)
> +            testpmd.start()
> +            testpmd.udp_tunnel_port(port_id=0, add=True, udp_port=GPE_port, protocol="vxlan")
> +            self.send_vxlan_packet_and_verify(udp_dport=GPE_port, should_receive_vxlan=True)
> +            testpmd.udp_tunnel_port(port_id=0, add=False, udp_port=GPE_port, protocol="vxlan")
> +            self.send_vxlan_packet_and_verify(udp_dport=GPE_port, should_receive_vxlan=False)
> --
> 2.44.0
>

[RFC v1 3/3] dts: conf schema VXLAN gpe support

[Dean Marx]

Configuration schema to run vxlan gpe support test suite

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..06c8978103 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "vxlan_gpe_support"
       ]
     },
     "test_target": {
-- 
2.44.0