dts: dynamic config test suite implementation

dts: dynamic config test suite implementation

[Dean Marx]

Dynamic Configuration test suite for ensuring Poll Mode Driver's ability
to enable/disable promiscuous and allmulticast mode, and verify the
expected behavior in the following four test cases:

1. Default mode - verifies that promiscuous mode is enabled by default,
and packets with any destination MAC address are received and forwarded.
2. Disable promisc - turns off promiscuous mode and verifies that
packets with a destination MAC address matching that of the Rx port are
forwarded, while unknown MAC addresses are dropped.
3. Disable promisc broadcast - turns off promiscuous mode and verifies
that packets with a matching or broadcast destination MAC address are
forwarded.
4. Disable promisc multicast - turns off promiscuous mode and verifies
that packets with a multicast destination MAC address are dropped when
allmulticast mode is turned off, and forwarded when it is turned on.

Dean Marx (4):
  dts: add multicast set function to shell
  dts: add toggle option to send and capture
  dts: dynamic config test suite
  dts: dynamic config conf schema

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py |  46 ++++++
 dts/framework/test_suite.py                   |   7 +-
 dts/tests/TestSuite_dynamic_config.py         | 149 ++++++++++++++++++
 4 files changed, 203 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_dynamic_config.py

-- 
2.44.0

dts: add multicast set function to shell

[Dean Marx]

added set multicast function for changing allmulticast mode within testpmd.

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 46 +++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index ec22f72221..a0be0bd09d 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -806,6 +806,52 @@ def show_port_stats(self, port_id: int) -> TestPmdPortStats:
 
         return TestPmdPortStats.parse(output)
 
+    def set_promisc(self, port: int, on: bool, verify: bool = True):
+        """Turns promiscuous mode on/off for the specified port.
+
+        Args:
+            port: Port number to use, should be within 0-32.
+            on: If :data:`True`, turn promisc mode on, otherwise turn off.
+            verify: If :data:`True` an additional command will be sent to verify that promisc mode
+                is properly set. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and promisc mode
+                is not correctly set.
+        """
+        promisc_output = self.send_command(f"set promisc {port} {'on' if on else 'off'}")
+        if verify:
+            stats = self.show_port_info(port_id=port)
+            if on ^ stats.is_promiscuous_mode_enabled:
+                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set promisc mode on port {port}."
+                )
+
+    def set_multicast_all(self, on: bool, verify: bool = True):
+        """Turns multicast mode on/off for the specified port.
+
+        Args:
+            on: If :data:`True`, turns multicast mode on, otherwise turns off.
+            verify: If :data:`True` an additional command will be sent to verify
+            that multicast mode is properly set. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and multicast
+                mode is not properly set.
+        """
+        multicast_output = self.send_command(f"set allmulti all {'on' if on else 'off'}")
+        if verify:
+            stats0 = self.show_port_info(port_id=0)
+            stats1 = self.show_port_info(port_id=1)
+            if on ^ (stats0.is_allmulticast_mode_enabled and stats1.is_allmulticast_mode_enabled):
+                self._logger.debug(
+                    f"Failed to set multicast mode on all ports.: \n{multicast_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    "Testpmd failed to set multicast mode on all ports."
+                )
+
     def close(self) -> None:
         """Overrides :meth:`~.interactive_shell.close`."""
         self.send_command("quit", "")
-- 
2.44.0

Re: dts: add multicast set function to shell

[Luca Vizzarro]

Hi Dean, thank you for your work.

On 08/07/2024 20:08, Dean Marx wrote:
> +    def set_multicast_all(self, on: bool, verify: bool = True):
> +        """Turns multicast mode on/off for the specified port.
> +
> +        Args:
> +            on: If :data:`True`, turns multicast mode on, otherwise turns off.
> +            verify: If :data:`True` an additional command will be sent to verify
> +            that multicast mode is properly set. Defaults to :data:`True`.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If `verify` is :data:`True` and multicast
> +                mode is not properly set.
> +        """
> +        multicast_output = self.send_command(f"set allmulti all {'on' if on else 'off'}")
> +        if verify:
> +            stats0 = self.show_port_info(port_id=0)
> +            stats1 = self.show_port_info(port_id=1)
This assumes that we have port 0 and 1, but *technically* we shouldn't 
be making assumptions about the environment in the framework. I'd rather 
use show_port_info_all and sample from there what's available.
> +            if on ^ (stats0.is_allmulticast_mode_enabled and stats1.is_allmulticast_mode_enabled):
> +                self._logger.debug(
> +                    f"Failed to set multicast mode on all ports.: \n{multicast_output}"
> +                )
> +                raise InteractiveCommandExecutionError(
> +                    "Testpmd failed to set multicast mode on all ports."
> +                )

dts: add toggle option to send and capture

[Dean Marx]

add option to skip _adjust_addresses method in send_packet_and_capture
when test cases involve sending packets with a preset MAC address.

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/framework/test_suite.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 694b2eba65..551a587525 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -185,6 +185,7 @@ def send_packet_and_capture(
         packet: Packet,
         filter_config: PacketFilteringConfig = PacketFilteringConfig(),
         duration: float = 1,
+        adjust_addresses: bool = True,
     ) -> list[Packet]:
         """Send and receive `packet` using the associated TG.
 
@@ -195,11 +196,15 @@ def send_packet_and_capture(
             packet: The packet to send.
             filter_config: The filter to use when capturing packets.
             duration: Capture traffic for this amount of time after sending `packet`.
+            adjust_addresses: If :data:'True', adjust addresses of the egressing packet with
+                a default addressing scheme. If :data:'False', do not adjust the addresses of
+                egressing packet.
 
         Returns:
             A list of received packets.
         """
-        packet = self._adjust_addresses(packet)
+        if adjust_addresses:
+            packet = self._adjust_addresses(packet)
         return self.tg_node.send_packet_and_capture(
             packet,
             self._tg_port_egress,
-- 
2.44.0

Re: dts: add toggle option to send and capture

[Luca Vizzarro]

Hi Dean,

this patch is awfully identical to Nicholas' older patch:

http://inbox.dpdk.org/dev/20240702192422.2480-3-npratte@iol.unh.edu/

Can you explain what's going on here? As it's all very confusing. It 
seems that maybe Nicholas dropped it in his late versions.

Best,
Luca

dts: dynamic config test suite

[Dean Marx]

Suite for testing ability of Poll Mode Driver to turn promiscuous
mode on/off, allmulticast mode on/off, and show expected behavior
when sending packets with known, unknown, broadcast, and multicast
destination MAC addresses.

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/tests/TestSuite_dynamic_config.py | 149 ++++++++++++++++++++++++++
 1 file changed, 149 insertions(+)
 create mode 100644 dts/tests/TestSuite_dynamic_config.py

diff --git a/dts/tests/TestSuite_dynamic_config.py b/dts/tests/TestSuite_dynamic_config.py
new file mode 100644
index 0000000000..326a57d60f
--- /dev/null
+++ b/dts/tests/TestSuite_dynamic_config.py
@@ -0,0 +1,149 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 University of New Hampshire
+
+"""Dynamic configuration capabilities test suite.
+
+This suite checks that it is possible to change the configuration of a port
+dynamically. The Poll Mode Driver should be able to enable and disable
+promiscuous mode on each port, as well as check the Rx and Tx packets of
+each port.
+
+If packets should be received and forwarded, or received and not forwarded,
+depending on the configuration, the port info should match the expected behavior.
+"""
+
+from time import sleep
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.params.testpmd import SimpleForwardingModes
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+
+class TestDynamicConfig(TestSuite):
+    """Dynamic config suite.
+
+    Use the show port commands to see the MAC address and promisc mode status
+    of the Rx port on the DUT. The suite will check the Rx and Tx packets
+    of each port after configuring promiscuous, multicast, and default mode
+    on the DUT to verify the expected behavior. It consists of four test cases:
+
+    1. Default mode: verify packets are received and forwarded.
+    2. Disable promiscuous mode: verfiy that packets are received
+    only for the packet with destination address matching the port address.
+    3. Disable promiscuous mode broadcast: verify that packets with destination
+    MAC address not matching the port are received and not forwarded, and verify
+    that broadcast packets are received and forwarded.
+    4. Disable promiscuous mode multicast: verify that packets with destination
+    MAC address not matching the port are received and not forwarded, and verify
+    that multicast packets are received and forwarded.
+    """
+
+    def set_up_suite(self) -> None:
+        """Set up the test suite.
+
+        Setup:
+            Verify that at least two ports are open for session.
+        """
+        self.verify(len(self._port_links) > 1, "Not enough ports")
+
+    def send_packet_and_verify(self, should_receive: bool, mac_address: str) -> None:
+        """Generate, send and verify packets.
+
+        Generate a packet and send to the DUT, verify that packet is forwarded from DUT to
+        traffic generator if that behavior is expected.
+
+        Args:
+            should_receive: Indicate whether the packet should be received.
+            mac_address: Destination MAC address to generate in packet.
+        """
+        packet = Ether(dst=mac_address) / IP() / Raw(load="xxxxx")
+        received = self.send_packet_and_capture(packet=packet, adjust_addresses=False)
+        contains_packet = any(
+            packet.haslayer(Raw) and b"xxxxx" in packet.load for packet in received
+        )
+        self.verify(
+            should_receive == contains_packet,
+            f"Packet was {'dropped' if should_receive else 'received'}",
+        )
+
+    def disable_promisc_setup(self, port_id: int) -> TestPmdShell:
+        """Sets up testpmd shell config for cases where promisc mode is disabled.
+
+        Args:
+            port_id: Port number to disable promisc mode on.
+
+        Returns:
+            shell: interactive testpmd shell object.
+        """
+        shell = TestPmdShell(node=self.sut_node)
+        shell.start()
+        shell.set_promisc(port=port_id, on=False)
+        shell.set_forward_mode(SimpleForwardingModes.io)
+        return shell
+
+    def test_default_mode(self) -> None:
+        """Tests default configuration.
+
+        Creates a testpmd shell, verifies that promiscuous mode is enabled by default,
+        and sends two packets; one matching source MAC address and one unknown.
+        Verifies that both are received.
+        """
+        testpmd = TestPmdShell(node=self.sut_node)
+        isPromisc = testpmd.show_port_info(0).is_promiscuous_mode_enabled
+        self.verify(isPromisc, "Promiscuous mode was not enabled by default.")
+        testpmd.start()
+        mac = testpmd.show_port_info(0).mac_address
+        # send a packet with Rx port mac address
+        self.send_packet_and_verify(should_receive=True, mac_address=str(mac))
+        # send a packet with mismatched mac address
+        self.send_packet_and_verify(should_receive=True, mac_address="00:00:00:00:00:00")
+        testpmd.close()
+        sleep(6)
+
+    def test_disable_promisc(self) -> None:
+        """Tests disabled promiscuous mode configuration.
+
+        Creates an interactive testpmd shell, disables promiscuous mode,
+        and sends two packets; one matching source MAC address and one unknown.
+        Verifies that only the matching address packet is received.
+        """
+        testpmd = self.disable_promisc_setup(port_id=0)
+        mac = testpmd.show_port_info(0).mac_address
+        self.send_packet_and_verify(should_receive=True, mac_address=str(mac))
+        self.send_packet_and_verify(should_receive=False, mac_address="00:00:00:00:00:00")
+        testpmd.close()
+        sleep(6)
+
+    def test_disable_promisc_broadcast(self) -> None:
+        """Tests broadcast reception with disabled promisc mode config.
+
+        Creates an interactive testpmd shell, disables promiscuous mode,
+        and sends two packets; one matching source MAC address and one broadcast.
+        Verifies that both packets are received.
+        """
+        testpmd = self.disable_promisc_setup(port_id=0)
+        mac = testpmd.show_port_info(0).mac_address
+        self.send_packet_and_verify(should_receive=True, mac_address=str(mac))
+        self.send_packet_and_verify(should_receive=True, mac_address="ff:ff:ff:ff:ff:ff")
+        testpmd.close()
+        sleep(6)
+
+    def test_disable_promisc_multicast(self) -> None:
+        """Tests allmulticast mode with disabled promisc config.
+
+        Creates an interactive testpmd shell, disables promiscuous mode,
+        and sends two packets; one matching source MAC address and one multicast.
+        Verifies that the multicast packet is only received once allmulticast mode is enabled.
+        """
+        testpmd = self.disable_promisc_setup(port_id=0)
+        testpmd.set_multicast_all(on=False)
+        # 01:00:5E:00:00:01 is the first of the multicast MAC range of addresses
+        self.send_packet_and_verify(should_receive=False, mac_address="01:00:5E:00:00:01")
+        testpmd.set_multicast_all(on=True)
+        self.send_packet_and_verify(should_receive=True, mac_address="01:00:05E:00:00:01")
+        testpmd.close()
+        sleep(6)
-- 
2.44.0

Re: dts: dynamic config test suite

[Luca Vizzarro]

Can you please rebase this on the latest changes, as the code will look 
different for sure?

Three comments in the meantime:

  * Is there a reason for sleeping for 6 seconds? If so could you please
    explain it?
  * Please follow naming standards consistently, variables should be in
    snake_case, referring to isPromisc -> is_promisc
  * Please follow contributing guidelines when writing your commit
    subject and body[1]
      o Example I can spot are capitalisation of wording and the
        non-imperative subject

Thanks,

Luca

[1] 
https://doc.dpdk.org/guides/contributing/patches.html#commit-messages-subject-line

dts: dynamic config conf schema

[Dean Marx]

configuration schema to run dynamic configuration test suite.

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..d7b4afed7d 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "dynamic_config"
       ]
     },
     "test_target": {
-- 
2.44.0

Re: dts: dynamic config conf schema

[Luca Vizzarro]

Similar to the previous commit, please follow contributing guidelines:

https://doc.dpdk.org/guides/contributing/patches.html#commit-messages-subject-line

Is there a reason to split this from the test suite commit?

Re: dts: dynamic config test suite implementation

[Luca Vizzarro]

Hi Dean,

I've just realised that you sent a new version but it was sent a 
separate thread. Sorry about that! I'll try to send a review to that 
one. Can you please make sure to reply to the original thread starting 
email when you send new versions as per contributing guidelines[1]?

Thank you,
Luca

[1] https://core.dpdk.org/contribute/

On 08/07/2024 20:08, Dean Marx wrote:
> Dynamic Configuration test suite for ensuring Poll Mode Driver's ability
> to enable/disable promiscuous and allmulticast mode, and verify the
> expected behavior in the following four test cases:
>
> 1. Default mode - verifies that promiscuous mode is enabled by default,
> and packets with any destination MAC address are received and forwarded.
> 2. Disable promisc - turns off promiscuous mode and verifies that
> packets with a destination MAC address matching that of the Rx port are
> forwarded, while unknown MAC addresses are dropped.
> 3. Disable promisc broadcast - turns off promiscuous mode and verifies
> that packets with a matching or broadcast destination MAC address are
> forwarded.
> 4. Disable promisc multicast - turns off promiscuous mode and verifies
> that packets with a multicast destination MAC address are dropped when
> allmulticast mode is turned off, and forwarded when it is turned on.
>
> Dean Marx (4):
>    dts: add multicast set function to shell
>    dts: add toggle option to send and capture
>    dts: dynamic config test suite
>    dts: dynamic config conf schema
>
>   dts/framework/config/conf_yaml_schema.json    |   3 +-
>   dts/framework/remote_session/testpmd_shell.py |  46 ++++++
>   dts/framework/test_suite.py                   |   7 +-
>   dts/tests/TestSuite_dynamic_config.py         | 149 ++++++++++++++++++
>   4 files changed, 203 insertions(+), 2 deletions(-)
>   create mode 100644 dts/tests/TestSuite_dynamic_config.py
>