[PATCH 0/3] Mac Filter Port to New DTS

[PATCH 0/3] Mac Filter Port to New DTS

[Nicholas Pratte]

This series incorporates the test suite itself, in addition to framework
changes needed to complete the suite's stipulated test cases. The test
suite consists of three test cases: one testing basically mac filtering
functionality, the other testing the PMD's mac address pool behavior
within a TestPMD shell, and the third testing filtering functionality
of multicast mac addresses.

The port from old DTS has removed some potentially important details
regarding EAL and TestPMD parameters, and these can be found within the
test suites corresponding commit messages. Moreover, unlike old DTS,
instead of validating packet behavior using TestPMD, this test suite
validates behavior by sending, and analysing, a list of received packets.

Nicholas Pratte (3):
  dts: add boolean to adjust addresses
  dts: add testpmd methods for test suite
  dts: mac filter test suite refactored for new dts

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py | 209 +++++++++++++++++
 dts/framework/test_suite.py                   |   4 +-
 dts/tests/TestSuite_mac_filter.py             | 212 ++++++++++++++++++
 4 files changed, 426 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

-- 
2.44.0

[PATCH 1/3] dts: add boolean to adjust addresses

[Nicholas Pratte]

Various test cases in the mac filter test suite called for granular
manipulation of destination mac addresses to properly test mac address
filtering functionality. To compensate, there is now an
adjust_addresses boolean which the user can toggle if they wish to send
their own addressing; the boolean is true by default.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/test_suite.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 694b2eba65..5044d5f9bb 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -185,6 +185,7 @@ def send_packet_and_capture(
         packet: Packet,
         filter_config: PacketFilteringConfig = PacketFilteringConfig(),
         duration: float = 1,
+        adjust_addresses: bool = True,
     ) -> list[Packet]:
         """Send and receive `packet` using the associated TG.
 
@@ -199,7 +200,8 @@ def send_packet_and_capture(
         Returns:
             A list of received packets.
         """
-        packet = self._adjust_addresses(packet)
+        if adjust_addresses:
+            packet = self._adjust_addresses(packet)
         return self.tg_node.send_packet_and_capture(
             packet,
             self._tg_port_egress,
-- 
2.44.0

Re: [PATCH 1/3] dts: add boolean to adjust addresses

[Jeremy Spewock]

This is funny because I actually ended up trying to solve the same
problem when writing the dynamic queue test suite. We ended up taking
different approaches, so we should probably have a discussion about
the best way to handle this. Now that we have a few use cases for why
this fix is needed, it will probably make the discussion easier since
there is less speculation.

On Fri, Jun 21, 2024 at 1:22 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
>
> Various test cases in the mac filter test suite called for granular
> manipulation of destination mac addresses to properly test mac address
> filtering functionality. To compensate, there is now an
> adjust_addresses boolean which the user can toggle if they wish to send
> their own addressing; the boolean is true by default.
>
> Bugzilla ID: 1454
> Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
> ---
>  dts/framework/test_suite.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> index 694b2eba65..5044d5f9bb 100644
> --- a/dts/framework/test_suite.py
> +++ b/dts/framework/test_suite.py
> @@ -185,6 +185,7 @@ def send_packet_and_capture(
>          packet: Packet,
>          filter_config: PacketFilteringConfig = PacketFilteringConfig(),
>          duration: float = 1,
> +        adjust_addresses: bool = True,

This should probably get added to the Args section of this doc-string
since it's a public method.



>      ) -> list[Packet]:
>          """Send and receive `packet` using the associated TG.
>
> @@ -199,7 +200,8 @@ def send_packet_and_capture(
>          Returns:
>              A list of received packets.
>          """
> -        packet = self._adjust_addresses(packet)
> +        if adjust_addresses:
> +            packet = self._adjust_addresses(packet)
>          return self.tg_node.send_packet_and_capture(
>              packet,
>              self._tg_port_egress,
> --
> 2.44.0
>

Re: [PATCH 1/3] dts: add boolean to adjust addresses

[Nicholas Pratte]

Yeah. I vaguely remember a conversation regarding the need and
validity of the 'adjust_addresses' functionality within DTS, going as
far as discussing whether it is needed or not, but maybe I'm wrong?
I'm honestly not sure.

I'll add the argument to the doc-string.

On Wed, Jun 26, 2024 at 11:49 AM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
> This is funny because I actually ended up trying to solve the same
> problem when writing the dynamic queue test suite. We ended up taking
> different approaches, so we should probably have a discussion about
> the best way to handle this. Now that we have a few use cases for why
> this fix is needed, it will probably make the discussion easier since
> there is less speculation.
>
> On Fri, Jun 21, 2024 at 1:22 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
> >
> > Various test cases in the mac filter test suite called for granular
> > manipulation of destination mac addresses to properly test mac address
> > filtering functionality. To compensate, there is now an
> > adjust_addresses boolean which the user can toggle if they wish to send
> > their own addressing; the boolean is true by default.
> >
> > Bugzilla ID: 1454
> > Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
> > ---
> >  dts/framework/test_suite.py | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
> > index 694b2eba65..5044d5f9bb 100644
> > --- a/dts/framework/test_suite.py
> > +++ b/dts/framework/test_suite.py
> > @@ -185,6 +185,7 @@ def send_packet_and_capture(
> >          packet: Packet,
> >          filter_config: PacketFilteringConfig = PacketFilteringConfig(),
> >          duration: float = 1,
> > +        adjust_addresses: bool = True,
>
> This should probably get added to the Args section of this doc-string
> since it's a public method.
>
>
>
> >      ) -> list[Packet]:
> >          """Send and receive `packet` using the associated TG.
> >
> > @@ -199,7 +200,8 @@ def send_packet_and_capture(
> >          Returns:
> >              A list of received packets.
> >          """
> > -        packet = self._adjust_addresses(packet)
> > +        if adjust_addresses:
> > +            packet = self._adjust_addresses(packet)
> >          return self.tg_node.send_packet_and_capture(
> >              packet,
> >              self._tg_port_egress,
> > --
> > 2.44.0
> >

[PATCH 2/3] dts: add testpmd methods for test suite

[Nicholas Pratte]

Several methods are either imported from currently in-develoment test
suites, or they have been created specifically for this test suite.

Methods here that can be found in other test suites include vlan
filtering, adding and removing vlan tags, and setting promiscuous mode.

Methods that are specific to this test suite include adding or removing
mac addresses, and adding or removing multicast mac addresses.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 209 ++++++++++++++++++
 1 file changed, 209 insertions(+)

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index ec22f72221..8a7d5905b3 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -767,6 +767,215 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
 
         return TestPmdPort.parse(output)
 
+    def set_mac_addr(self, port_id: int, mac_address: str, add: bool = True, verify: bool = True):
+        """Add or remove a mac address on a given port.
+
+        Args:
+            port_id: The port ID the mac address is set on.
+            mac_address: The mac address to be added or removed to the specified port.
+            add: If :data:`True`, add the specified mac address. If :data:`False`, remove specified
+                mac address.
+            verify: If :data:'True', assert that the 'mac_addr' operation was successful. If
+                :data:'False', run the command and skip this assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fail.
+        """
+        if add:
+            output = self.send_command(f"mac_addr add {port_id} {mac_address}")
+        else:
+            output = self.send_command(f"mac_addr remove {port_id} {mac_address}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid argument provided to mac_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if add and "mac_addr_cmd error:" in output:
+                self._logger.debug(f"Failed to add {mac_address} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to add {mac_address} on port {port_id} \n{output}"
+                )
+            if not add and "mac_addr_cmd error" in output:
+                self._logger.debug(f"Failed to remove {mac_address} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to remove {mac_address} on port {port_id} \n{output}"
+                )
+
+    def set_multicast_mac_addr(
+        self, port_id: int, multi_addr: str, add: bool = True, verify: bool = True
+    ):
+        """Add or remove multicast mac address to a specified port filter.
+
+        Args:
+            port_id: The port ID the multicast address is set on.
+            multi_addr: The multicast address to be added to the filter.
+            add: If :data:'True', add the specified multicast address to the port filter.
+                If :data:'False', remove the specified multicast address from the port filter.
+            verify: If :data:'True', assert that the 'mcast_addr' operations was successful.
+                If :data:'False', execute the 'mcast_addr' operation and skip the assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fails.
+        """
+        if add:
+            output = self.send_command(f"mcast_addr add {port_id} {multi_addr}")
+        else:
+            output = self.send_command(f"mcast_addr remove {port_id} {multi_addr}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid arguments provided to mcast_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if (
+                add
+                and "Invalid multicast_addr"
+                or "multicast address already filtered by port" in output
+            ):
+                self._logger.debug(f"Failed to add {multi_addr} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to add {multi_addr} on port {port_id} \n{output}"
+                )
+            if (
+                not add
+                and "Invalid multicast addr"
+                or "multicast address not filtered by port" in output
+            ):
+                self._logger.debug(f"Failed to remove {multi_addr} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to remove {multi_addr} on port {port_id} \n{output}"
+                )
+
+    def rx_vlan_add(self, vlan: int, port: int, verify: bool = True):
+        """Add specified vlan tag to the filter list on a port.
+
+        Args:
+            vlan: The vlan tag to add, should be within 1-1005, 1-4094 extended.
+            port: The port number to add the tag on, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was added to the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+            is not added.
+        """
+        vlan_add_output = self.send_command(f"rx_vlan add {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_add_output or "Invalid vlan_id" in vlan_add_output:
+                self._logger.debug(
+                    f"Failed to add vlan tag {vlan} on port {port}: \n{vlan_add_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to add vlan tag {vlan} on port {port}."
+                )
+
+    def rx_vlan_rm(self, vlan: int, port: int, verify: bool = True):
+        """Remove specified vlan tag from filter list on a port.
+
+        Args:
+            vlan: The vlan tag to remove, should be within 1-4094.
+            port: The port number to remove the tag from, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was removed from the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+            is not removed.
+        """
+        vlan_rm_output = self.send_command(f"rx_vlan rm {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_rm_output or "Invalid vlan_id" in vlan_rm_output:
+                self._logger.debug(
+                    f"Failed to remove vlan tag {vlan} on port {port}: \n{vlan_rm_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to remove vlan tag {vlan} on port {port}."
+                )
+
+    def vlan_filter_set_on(self, port: int, verify: bool = True):
+        """Set vlan filter on.
+
+        Args:
+            port: The port number to use, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                vlan filtering was enabled successfully. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter on {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter: on" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to enable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to enable vlan filter on port {port}."
+                )
+
+    def vlan_filter_set_off(self, port: int, verify: bool = True):
+        """Set vlan filter off.
+
+        Args:
+            port: The port number to use, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                vlan filtering was disabled successfully. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter off {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter: off" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to disable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to disable vlan filter on port {port}."
+                )
+
+    def set_promisc(self, port: int, on: bool, verify: bool = True):
+        """Turns promiscuous mode on/off for the specified port.
+
+        Args:
+            port: port number to use, should be within 0-32.
+            on: if :data:`True`, turn promisc mode on, otherwise turn off.
+            verify: if :data:`True` an additional command will be sent to verify that promisc mode
+                is properly set. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and promisc mode
+            is not correctly set.
+        """
+        if on:
+            promisc_output = self.send_command(f"set promisc {port} on")
+        else:
+            promisc_output = self.send_command(f"set promisc {port} off")
+        if verify:
+            if on and "Promiscuous mode: enabled" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set promisc mode on port {port}."
+                )
+            elif not on and "Promiscuous mode: disabled" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set promisc mode on port {port}."
+                )
+
     def show_port_stats_all(self) -> list[TestPmdPortStats]:
         """Returns the statistics of all the ports.
 
-- 
2.44.0

Re: [PATCH 2/3] dts: add testpmd methods for test suite

[Jeremy Spewock]

I think the subject line of this commit makes sense in the context of
this series, but might be less helpful when the commit is applied to
the git tree. For this reason I might favor changing it to something
that briefly says what the added testpmd commands actually do.

On Fri, Jun 21, 2024 at 1:23 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
>
> Several methods are either imported from currently in-develoment test
> suites, or they have been created specifically for this test suite.
>
> Methods here that can be found in other test suites include vlan
> filtering, adding and removing vlan tags, and setting promiscuous mode.
>
> Methods that are specific to this test suite include adding or removing
> mac addresses, and adding or removing multicast mac addresses.

You probably could make the subject about the adding/modification of
mac addresses judging by the body.

>
> Bugzilla ID: 1454
> Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
> ---
>  dts/framework/remote_session/testpmd_shell.py | 209 ++++++++++++++++++
>  1 file changed, 209 insertions(+)
>
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index ec22f72221..8a7d5905b3 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -767,6 +767,215 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
>
>          return TestPmdPort.parse(output)
>
> +    def set_mac_addr(self, port_id: int, mac_address: str, add: bool = True, verify: bool = True):

I'm not sure it makes sense to default the `add` parameter in this
method or the following. It seems to me like this method would likely
be split pretty evenly between adding and removing and I can't think
of a reason why one would be done in the general case over the other,
so I think it would be more clear if we didn't default it.

> +        """Add or remove a mac address on a given port.
> +

Is it worth mentioning that we are adding the MAC address to the
allowlist of the port?

> +        Args:
> +            port_id: The port ID the mac address is set on.
> +            mac_address: The mac address to be added or removed to the specified port.
> +            add: If :data:`True`, add the specified mac address. If :data:`False`, remove specified
> +                mac address.
> +            verify: If :data:'True', assert that the 'mac_addr' operation was successful. If
> +                :data:'False', run the command and skip this assertion.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fail.

This error documentation confuses me a little because it sounds like
both `add` and `remove` operations are happening in this method when
really either one happens or the other happens. Maybe changing this to
just saying the "mac address operation" failed would fix this.

> +        """

I think something that could save you a lot of space in this method
(and the next one) is creating a variable for what command to use if
`add` is true or not.
You could do something like this:

mac_cmd = "add" if add else "remove"

> +        if add:
> +            output = self.send_command(f"mac_addr add {port_id} {mac_address}")
> +        else:
> +            output = self.send_command(f"mac_addr remove {port_id} {mac_address}")

Then you don't need this if-statement because this just becomes:
output = self.send_command(f"mac_addr {mac_cmd} {port_id} {mac_address}")

> +        if "Bad arguments" in output:
> +            self._logger.debug("Invalid argument provided to mac_addr")
> +            raise InteractiveCommandExecutionError("Invalid argument provided")
> +
> +        if verify:

If you have the variable you can also remove this if-statement about
which output/error to provide. Since it's the same error in either
case you can just do:

if "mac_addr_cmd error" in output:
    self._logger.debug(f"Failed to {mac_cmd} ...

> +            if add and "mac_addr_cmd error:" in output:
> +                self._logger.debug(f"Failed to add {mac_address} on port {port_id}")
> +                raise InteractiveCommandExecutionError(
> +                    f"Failed to add {mac_address} on port {port_id} \n{output}"
> +                )
> +            if not add and "mac_addr_cmd error" in output:
> +                self._logger.debug(f"Failed to remove {mac_address} on port {port_id}")
> +                raise InteractiveCommandExecutionError(
> +                    f"Failed to remove {mac_address} on port {port_id} \n{output}"
> +                )
> +
> +    def set_multicast_mac_addr(
> +        self, port_id: int, multi_addr: str, add: bool = True, verify: bool = True
> +    ):
> +        """Add or remove multicast mac address to a specified port filter.
> +
> +        Args:
> +            port_id: The port ID the multicast address is set on.
> +            multi_addr: The multicast address to be added to the filter.
> +            add: If :data:'True', add the specified multicast address to the port filter.
> +                If :data:'False', remove the specified multicast address from the port filter.
> +            verify: If :data:'True', assert that the 'mcast_addr' operations was successful.
> +                If :data:'False', execute the 'mcast_addr' operation and skip the assertion.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fails.
> +        """
> +        if add:
> +            output = self.send_command(f"mcast_addr add {port_id} {multi_addr}")
> +        else:
> +            output = self.send_command(f"mcast_addr remove {port_id} {multi_addr}")
> +        if "Bad arguments" in output:
> +            self._logger.debug("Invalid arguments provided to mcast_addr")
> +            raise InteractiveCommandExecutionError("Invalid argument provided")
> +
> +        if verify:
> +            if (
> +                add
> +                and "Invalid multicast_addr"
> +                or "multicast address already filtered by port" in output

If you use the variable here as well, this is a little trickier
because this last line in the condition is different. Still can be
solved however by making it:
f"multicast address {'already' if add else 'not'} filtered by port"

> +            ):
> +                self._logger.debug(f"Failed to add {multi_addr} on port {port_id}")
> +                raise InteractiveCommandExecutionError(
> +                    f"Failed to add {multi_addr} on port {port_id} \n{output}"
> +                )
> +            if (
> +                not add
> +                and "Invalid multicast addr"

Is this a typo or does testpmd remove the underscore when you're removing?

> +                or "multicast address not filtered by port" in output
> +            ):
> +                self._logger.debug(f"Failed to remove {multi_addr} on port {port_id}")
> +                raise InteractiveCommandExecutionError(
> +                    f"Failed to remove {multi_addr} on port {port_id} \n{output}"
> +                )
> +
<snip>
> +
> +    def rx_vlan_rm(self, vlan: int, port: int, verify: bool = True):
> +        """Remove specified vlan tag from filter list on a port.
> +
> +        Args:
> +            vlan: The vlan tag to remove, should be within 1-4094.
> +            port: The port number to remove the tag from, should be within 0-32.
> +            verify: If :data:`True`, the output of the command is scanned to verify that
> +                the vlan tag was removed from the filter list on the specified port. If not, it is
> +                considered an error.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
> +            is not removed.

Other methods have this second line indented. I'm not sure if I
addressed this on the VLAN suite already or not but this should
probably do the same.

> +        """
<snip>
> +
> +    def set_promisc(self, port: int, on: bool, verify: bool = True):
> +        """Turns promiscuous mode on/off for the specified port.
> +
> +        Args:
> +            port: port number to use, should be within 0-32.
> +            on: if :data:`True`, turn promisc mode on, otherwise turn off.
> +            verify: if :data:`True` an additional command will be sent to verify that promisc mode
> +                is properly set. Defaults to :data:`True`.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If `verify` is :data:`True` and promisc mode
> +            is not correctly set.
> +        """
> +        if on:
> +            promisc_output = self.send_command(f"set promisc {port} on")
> +        else:
> +            promisc_output = self.send_command(f"set promisc {port} off")

This can also be done with a variable, or you can inline it by doing
{'on' if on else 'off'}.

> +        if verify:
> +            if on and "Promiscuous mode: enabled" not in self.send_command(
> +                f"show port info {port}"
> +            ):
> +                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
> +                raise InteractiveCommandExecutionError(
> +                    f"Testpmd failed to set promisc mode on port {port}."
> +                )
> +            elif not on and "Promiscuous mode: disabled" not in self.send_command(
> +                f"show port info {port}"
> +            ):
> +                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
> +                raise InteractiveCommandExecutionError(
> +                    f"Testpmd failed to set promisc mode on port {port}."
> +                )

The logger output and the error seem to be exactly the same here so we
should avoid duplicating them. There are a few ways to go about
combining these two cases in the conditional but I would probably just
use an f-string to conditionally look for "enabled" vs "disabled".

I wonder if this check would be easier using the dataclass for port
info since it will be a boolean inside of the dataclass rather than
just searching for a string. I think if you had a boolean for if
promisc mode was on you could use an XOR of add and is_promisc_mode
and it would have the same effect. Normally I avoid using the
dataclass if the check is simple without it just because I think it is
slightly faster that way, but if there is a good use-case for it (like
there is here) then I think we might as well use it.

I think using the testpmd.show_port_info method would make for a
cleaner approach, so I slightly favor that one.


> +
>      def show_port_stats_all(self) -> list[TestPmdPortStats]:
>          """Returns the statistics of all the ports.
>
> --
> 2.44.0
>

Re: [PATCH 2/3] dts: add testpmd methods for test suite

[Nicholas Pratte]

On Wed, Jun 26, 2024 at 11:50 AM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
> I think the subject line of this commit makes sense in the context of
> this series, but might be less helpful when the commit is applied to
> the git tree. For this reason I might favor changing it to something
> that briefly says what the added testpmd commands actually do.
>
> On Fri, Jun 21, 2024 at 1:23 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
> >
> > Several methods are either imported from currently in-develoment test
> > suites, or they have been created specifically for this test suite.
> >
> > Methods here that can be found in other test suites include vlan
> > filtering, adding and removing vlan tags, and setting promiscuous mode.
> >
> > Methods that are specific to this test suite include adding or removing
> > mac addresses, and adding or removing multicast mac addresses.
>
> You probably could make the subject about the adding/modification of
> mac addresses judging by the body.
>

Good point. I'll make adjustments to the commit subject as well as the
commit messages.

> >
> > Bugzilla ID: 1454
> > Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
> > ---
> >  dts/framework/remote_session/testpmd_shell.py | 209 ++++++++++++++++++
> >  1 file changed, 209 insertions(+)
> >
> > diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> > index ec22f72221..8a7d5905b3 100644
> > --- a/dts/framework/remote_session/testpmd_shell.py
> > +++ b/dts/framework/remote_session/testpmd_shell.py
> > @@ -767,6 +767,215 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
> >
> >          return TestPmdPort.parse(output)
> >
> > +    def set_mac_addr(self, port_id: int, mac_address: str, add: bool = True, verify: bool = True):
>
> I'm not sure it makes sense to default the `add` parameter in this
> method or the following. It seems to me like this method would likely
> be split pretty evenly between adding and removing and I can't think
> of a reason why one would be done in the general case over the other,
> so I think it would be more clear if we didn't default it.
>

Good point. It made sense for me to do it this way in a vacuum (since
I'm mostly adding addresses in each test case), but there's no good
reason it should be this way.

> > +        """Add or remove a mac address on a given port.
> > +
>
> Is it worth mentioning that we are adding the MAC address to the
> allowlist of the port?

Yes, I made a subtle change to the docstring.

>
> > +        Args:
> > +            port_id: The port ID the mac address is set on.
> > +            mac_address: The mac address to be added or removed to the specified port.
> > +            add: If :data:`True`, add the specified mac address. If :data:`False`, remove specified
> > +                mac address.
> > +            verify: If :data:'True', assert that the 'mac_addr' operation was successful. If
> > +                :data:'False', run the command and skip this assertion.
> > +
> > +        Raises:
> > +            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fail.
>
> This error documentation confuses me a little because it sounds like
> both `add` and `remove` operations are happening in this method when
> really either one happens or the other happens. Maybe changing this to
> just saying the "mac address operation" failed would fix this.
>

Agreed. I made the change.

> > +        """
>
> I think something that could save you a lot of space in this method
> (and the next one) is creating a variable for what command to use if
> `add` is true or not.
> You could do something like this:
>
> mac_cmd = "add" if add else "remove"
>
> > +        if add:
> > +            output = self.send_command(f"mac_addr add {port_id} {mac_address}")
> > +        else:
> > +            output = self.send_command(f"mac_addr remove {port_id} {mac_address}")
>
> Then you don't need this if-statement because this just becomes:
> output = self.send_command(f"mac_addr {mac_cmd} {port_id} {mac_address}")
>

I love this! Good thinking. I'll make the changes.

<snip>
> > +            ):
> > +                self._logger.debug(f"Failed to add {multi_addr} on port {port_id}")
> > +                raise InteractiveCommandExecutionError(
> > +                    f"Failed to add {multi_addr} on port {port_id} \n{output}"
> > +                )
> > +            if (
> > +                not add
> > +                and "Invalid multicast addr"
>
> Is this a typo or does testpmd remove the underscore when you're removing?

Not a typo. Good attention to detail though.

<snip>
> > +
> > +    def rx_vlan_rm(self, vlan: int, port: int, verify: bool = True):
> > +        """Remove specified vlan tag from filter list on a port.
> > +
> > +        Args:
> > +            vlan: The vlan tag to remove, should be within 1-4094.
> > +            port: The port number to remove the tag from, should be within 0-32.
> > +            verify: If :data:`True`, the output of the command is scanned to verify that
> > +                the vlan tag was removed from the filter list on the specified port. If not, it is
> > +                considered an error.
> > +
> > +        Raises:
> > +            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
> > +            is not removed.
>
> Other methods have this second line indented. I'm not sure if I
> addressed this on the VLAN suite already or not but this should
> probably do the same.

All of the remaining methods were ripped from the VLAN suite, and I'll
update these as that series progresses. I've been doing it this way to
hopefully make it less confusing come the time these patches series
are ready to be merged.

[PATCH 3/3] dts: mac filter test suite refactored for new dts

[Nicholas Pratte]

The mac address filter test suite, whose test cases are based on old
DTS's test cases, has been refactored to interface with the new DTS
framework.

In porting over this test suite into the new framework, some
adjustments were made, namely in the EAL and TestPMD parameter provided
before executing the application. While the original test plan was
referenced, by and large, only for the individual test cases, I'll leave
the parameters the original test plan was asking for below for the sake
of discussion:

--burst=1 --rxpt=0 --rxht=0 --rxwt=0 --txpt=36 --txht=0 --txwt=0
--txfreet=32 --rxfreet=64 --mbcache=250 --portmask=0x3

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_mac_filter.py          | 212 +++++++++++++++++++++
 2 files changed, 214 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..ad1f3757f7 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "mac_filter"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
new file mode 100644
index 0000000000..2b97108e0e
--- /dev/null
+++ b/dts/tests/TestSuite_mac_filter.py
@@ -0,0 +1,212 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023-2024 University of New Hampshire
+"""Mac address filtering test suite.
+
+This test suite ensures proper and expected behavior of Allowlist filtering via mac
+addresses on devices bound to the Poll Mode Driver. If a packet received on a device
+contains a mac address not contained with its mac address pool, the packet should
+be dropped. Alternatively, if a packet is received that contains a destination mac
+within the devices address pool, the packet should be accepted and forwarded. This
+behavior should remain consistent across all packets, namely those containing dot1q
+tags or otherwise.
+
+The following test suite assesses behaviors based on the aforementioned logic.
+Additionally, testing is done within the PMD itself to ensure that the mac address
+allow list is behaving as expected.
+"""
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Dot1Q, Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.exception import InteractiveCommandExecutionError
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+
+class TestMacFilter(TestSuite):
+    """Mac address allowlist filtering test suite.
+
+    Configure mac address filtering on a given port, and test the port's filtering behavior
+    using both a given port's hardware address as well as dummy addresses. If a port accepts
+    a packet that is not contained within its mac address allowlist, then a given test case
+    fails. Alternatively, if a port drops a packet that is designated within its mac address
+    allowlist, a given test case will fail.
+
+    Moreover, a given port should demonstrate proper behavior when bound to the Poll Mode
+    Driver. A port should not have an mac address allowlist that exceeds its designated size.
+    A port's default hardware address should not be removed from its address pool, and invalid
+    addresses should not be included in the allowlist. If a port abides by the above rules, the
+    test case passes.
+    """
+
+    def send_packet_and_verify(
+        self,
+        mac_address: str,
+        add_vlan: bool = False,
+        should_receive: bool = True,
+    ) -> None:
+        """Generate, send, and verify a packet based on specified parameters.
+
+        Test cases within this suite utilize this method to create, send, and verify
+        packets based on criteria relating to the packet's destination mac address,
+        vlan tag, and whether or not the packet should be received or not. Packets
+        are verified using an inserted payload. If the list of received packets
+        contains this payload within any of its packets, the test case passes. This
+        implementation does not require a quiet wire, each call with this method
+        sends exactly one packet.
+
+        Args:
+            mac_address: The destination mac address of the packet being sent.
+            add_vlan: Add a vlan tag to the packet being sent. :data:'2' if the packet
+                should be received, :data:'1' if the packet should not be received but
+                requires a vlan tag, and None for any other condition.
+            should_receive: If :data:'True', assert whether or not the sent packet
+                has been received. If :data:'False', assert that the send packet was not
+                received. :data:'True' by default
+        """
+        packet = (
+            Ether()
+            / Dot1Q(vlan=2 if add_vlan and should_receive else 1 if add_vlan else None)
+            / IP()
+            / Raw(load="P" * 22)
+        )
+        packet.dst = mac_address
+        received_packets = [
+            packets
+            for packets in self.send_packet_and_capture(packet, adjust_addresses=False)
+            if hasattr(packets, "load") and "P" * 22 in str(packets.load)
+        ]
+        if should_receive:
+            self.verify(len(received_packets) == 1, "Expected packet not received")
+        else:
+            self.verify(len(received_packets) == 0, "Expected packet received")
+
+    def test_add_remove_mac_addresses(self) -> None:
+        """Assess basic mac addressing filtering functionalities.
+
+        This test cases validates for proper behavior of mac address filtering with both
+        a port's default, burned-in mac address, as well as additional mac addresses
+        added to the PMD. Packets should either be received or not received depending on
+        the properties applied to the PMD at any given time.
+
+        Test:
+            Start TestPMD with promiscuous mode.
+            Send a packet with the port's default mac address. (Should receive)
+            Send a packet with fake mac address. (Should not receive)
+            Add fake mac address to the PMD's address pool.
+            Send a packet with the fake mac address to the PMD. (Should receive)
+            Remove the fake mac address from the PMD's address pool.
+            Sent a packet with the fake mac address to the PMD. (Should not receive)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.set_promisc(0, on=False, verify=False)
+        testpmd.start()
+        mac_address = self._sut_port_ingress.mac_address
+
+        # Send a packet with NIC default mac address
+        self.send_packet_and_verify(mac_address=mac_address, should_receive=True)
+        # Send a packet with different mac address
+        fake_address = "00:00:00:00:00:01"
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
+
+        # Add mac address to pool and rerun tests
+        testpmd.set_mac_addr(0, mac_address=fake_address, verify=True)
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=True)
+        testpmd.set_mac_addr(0, mac_address=fake_address, add=False)
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
+
+    def test_invalid_address(self) -> None:
+        """Assess the behavior of a NIC mac address pool while binded to the PMD.
+
+        An assessment of a NIC's behavior when mounted to a PMD as it relates to mac addresses
+        and address pooling. Devices should not be able to use invalid mac addresses, remove their
+        built-in hardware address, or exceed their address pools.
+
+        Test:
+            Start TestPMD.
+            Attempt to set add an invalid mac address. (Should fail)
+            Attempt to remove the device's hardware address. (Should fail)
+            Add a fake mac address to the pool twice in succession. (Should not create any errors)
+            Attempt to remove the device's hardware address. (Should fail)
+            Determine the device's mac address pool size, and fill the pool with fake addresses.
+            Attempt to add another fake mac address, overloading the address pool. (Should not work)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.start()
+        mac_address = self._sut_port_ingress.mac_address
+        try:
+            testpmd.set_mac_addr(0, "00:00:00:00:00:00")
+            self.verify(False, "Invalid mac address added.")
+        except InteractiveCommandExecutionError:
+            pass
+        try:
+            testpmd.set_mac_addr(0, mac_address, False)
+            self.verify(False, "Default mac address removed.")
+        except InteractiveCommandExecutionError:
+            pass
+        # Should be no errors adding this twice
+        testpmd.set_mac_addr(0, f"'1' + {mac_address[1:]}")
+        testpmd.set_mac_addr(0, f"'1' + {mac_address[1:]}")
+        # Double check to see if default mac address can be removed
+        try:
+            testpmd.set_mac_addr(0, mac_address, False)
+            self.verify(False, "Default mac address removed.")
+        except InteractiveCommandExecutionError:
+            pass
+
+        for i in range(testpmd.show_port_info(0).max_mac_addresses_num - 1):
+            number = str(hex(i)[2:].zfill(4))
+            fake_address = f'{number[:2] + ":" + number[2:] + mac_address[5:]}'.upper()
+            testpmd.set_mac_addr(0, fake_address, verify=False)
+        try:
+            testpmd.set_mac_addr(0, f'{mac_address[:-1] + "1"}')
+            self.verify(False, "Mac address limit exceeded.")
+        except InteractiveCommandExecutionError:
+            pass
+        testpmd.close()
+
+    def test_multicast_filter(self) -> None:
+        """Assess basic multicast address filtering functionalities.
+
+        Ensure that multicast filtering performs as intended when a given device is bound
+        to the PMD, with and without dot1q vlan tagging.
+
+        Test:
+            Start TestPMD with promiscuous mode.
+            Add a fake multicast address to the PMD's multicast address pool.
+            Send a packet with the fake multicast address to the PMD. (Should receive)
+            Set vlan filtering on the PMD, and add vlan ID to the PMD.
+            Send a packet with the fake multicast address and vlan ID to the PMD. (Should receive)
+            Send a packet with the fake multicast address and a different vlan ID to the PMD.
+                (Should not receive)
+            Remove the vlan tag from the PMD, and turn vlan filtering off on the PMD.
+            Send a packet with the fake multicast address and no vlan tag to the PMD.
+                (Should receive)
+            Remove the fake multicast address from the PMDs multicast address filter.
+            Send a packet with the fake multicast address to the PMD. (Should not receive)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.start()
+        testpmd.set_promisc(0, on=False, verify=True)
+        multicast_address = "01:00:5E:00:00:00"
+        vlan_id = 2
+
+        testpmd.set_multicast_mac_addr(0, multi_addr=multicast_address, verify=False)
+        self.send_packet_and_verify(multicast_address, should_receive=True)
+
+        # Test vlan filtering on multicast addressing.
+        testpmd.vlan_filter_set_on(0, verify=False)
+        testpmd.rx_vlan_add(2, 0, verify=False)
+        self.send_packet_and_verify(multicast_address, should_receive=True, add_vlan=True)
+        self.send_packet_and_verify(multicast_address, should_receive=False, add_vlan=True)
+
+        # Remove vlan tag and filtering and run basic multicast addr test.
+        testpmd.rx_vlan_rm(vlan_id, 0, verify=False)
+        testpmd.vlan_filter_set_off(0, verify=False)
+        self.send_packet_and_verify(multicast_address, should_receive=True)
+
+        # Remove multicast filter and verify the packet was not received.
+        testpmd.set_multicast_mac_addr(0, multicast_address, add=False, verify=False)
+        self.send_packet_and_verify(multicast_address, should_receive=False)
+        testpmd.close()
-- 
2.44.0

Re: [PATCH 3/3] dts: mac filter test suite refactored for new dts

[Jeremy Spewock]

On Fri, Jun 21, 2024 at 1:24 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
>
> The mac address filter test suite, whose test cases are based on old
> DTS's test cases, has been refactored to interface with the new DTS
> framework.
>
> In porting over this test suite into the new framework, some
> adjustments were made, namely in the EAL and TestPMD parameter provided
> before executing the application. While the original test plan was
> referenced, by and large, only for the individual test cases, I'll leave
> the parameters the original test plan was asking for below for the sake
> of discussion:
>
> --burst=1 --rxpt=0 --rxht=0 --rxwt=0 --txpt=36 --txht=0 --txwt=0
> --txfreet=32 --rxfreet=64 --mbcache=250 --portmask=0x3
>
> Bugzilla ID: 1454
> Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
> ---
<snip>
> +    def send_packet_and_verify(
> +        self,
> +        mac_address: str,
> +        add_vlan: bool = False,
> +        should_receive: bool = True,
> +    ) -> None:
> +        """Generate, send, and verify a packet based on specified parameters.
> +
> +        Test cases within this suite utilize this method to create, send, and verify
> +        packets based on criteria relating to the packet's destination mac address,
> +        vlan tag, and whether or not the packet should be received or not. Packets
> +        are verified using an inserted payload. If the list of received packets
> +        contains this payload within any of its packets, the test case passes. This
> +        implementation does not require a quiet wire, each call with this method

I don't think we need to mention that it doesn't require a quiet wire,
in general this is something we are writing all test cases to assume.

Looking at this though, it would probably be a great thing to add to
the documentation for writing test suites. We never really mention
anywhere that you must write test cases without this assumption, it's
sort of just something we decided. Of course out of scope of this
patch, but something for the future.

> +        sends exactly one packet.
> +
> +        Args:
> +            mac_address: The destination mac address of the packet being sent.
> +            add_vlan: Add a vlan tag to the packet being sent. :data:'2' if the packet
> +                should be received, :data:'1' if the packet should not be received but
> +                requires a vlan tag, and None for any other condition.

This confused me a little because I thought you were saying this
parameter would be 2 if the packet should be received or it would be 1
in another case. I think this parameter is really just denoting if we
should add a VLAN tag at all, and then the should_receive specifies
what the tag will be. We should probably move the information about
either being 2 or 1 to the `should_receive` documentation and specify
that the VLAN tag will be that value based on if it should be
received.

This is a little hard because it relies on both conditions, but you
can just say in the `should_recieve` parameter that, if there is a
vlan tag, this is what it will be.

> +            should_receive: If :data:'True', assert whether or not the sent packet
> +                has been received. If :data:'False', assert that the send packet was not
> +                received. :data:'True' by default
> +        """
> +        packet = (
> +            Ether()
> +            / Dot1Q(vlan=2 if add_vlan and should_receive else 1 if add_vlan else None)

I haven't used the Dot1Q class in scapy very much myself, but it seems
like just setting vlan=None is strange. Does this just make it
completely omit the VLAN tag? Also, if we don't have a VLAN tag,
should we just not include the Dot1Q layer at all? It feels like it
doesn't really do anything if you don't have a VLAN.

> +            / IP()
> +            / Raw(load="P" * 22)
> +        )
> +        packet.dst = mac_address
> +        received_packets = [
> +            packets
> +            for packets in self.send_packet_and_capture(packet, adjust_addresses=False)
> +            if hasattr(packets, "load") and "P" * 22 in str(packets.load)
> +        ]

i think this would be a little easier using the filter() built-in
Python function. It would instead be:
recieved_packets = [filter(lambda x: if hasattr(x, "load") and "P" *
22 in str(x.load),
 self.send_packet_and_capture(packet, adjust_addresses=False)]

It doesn't really save you that much verbosity now that I've written
it out, you kind of trade the "for x in ..." for "filter(lambda x:
..." so maybe it isn't any more valuable. However, this function for
if a packet has a certain payload is becoming more and more common, I
wonder if we could just add a method to test_suite called
packet_has_payload(p: Packet, payload: str) that just checks does the
packet have the load attribute and does that load attribute match the
desired payload. Or, even better, if we made it a high-order function
that just accepts a payload and then returns a function that validates
if a packet has that payload. Something like this:

def packet_has_payload(payload: str) -> Callable[[Packet], bool]:
    def wrap(p: Packet) -> bool:
        return if hasattr(p, "load") and payload in str(p.load)
    return wrap

If we had this high-order function then filtering becomes super easy.
The filter would just become:
recieved_packets = [filter(self.packet_has_payload("P" * 22),
 self.send_packet_and_capture(packet, adjust_addresses=False)]

Maybe this is something that should be addressed in the bugzilla
ticket 1372 however.

> +        if should_receive:
> +            self.verify(len(received_packets) == 1, "Expected packet not received")
> +        else:
> +            self.verify(len(received_packets) == 0, "Expected packet received")
> +
> +    def test_add_remove_mac_addresses(self) -> None:
> +        """Assess basic mac addressing filtering functionalities.
> +
> +        This test cases validates for proper behavior of mac address filtering with both
> +        a port's default, burned-in mac address, as well as additional mac addresses
> +        added to the PMD. Packets should either be received or not received depending on
> +        the properties applied to the PMD at any given time.
> +
> +        Test:
> +            Start TestPMD with promiscuous mode.
> +            Send a packet with the port's default mac address. (Should receive)
> +            Send a packet with fake mac address. (Should not receive)
> +            Add fake mac address to the PMD's address pool.
> +            Send a packet with the fake mac address to the PMD. (Should receive)
> +            Remove the fake mac address from the PMD's address pool.
> +            Sent a packet with the fake mac address to the PMD. (Should not receive)

Typo: Sent a packet.

> +        """
> +        testpmd = TestPmdShell(self.sut_node)
> +        testpmd.set_promisc(0, on=False, verify=False)

Why don't we want to verify this command? If there is a reason we
should probably leave a comment explaining it.

> +        testpmd.start()
> +        mac_address = self._sut_port_ingress.mac_address
> +
> +        # Send a packet with NIC default mac address
> +        self.send_packet_and_verify(mac_address=mac_address, should_receive=True)
> +        # Send a packet with different mac address
> +        fake_address = "00:00:00:00:00:01"
> +        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
> +
> +        # Add mac address to pool and rerun tests
> +        testpmd.set_mac_addr(0, mac_address=fake_address, verify=True)

verify=True if the default and we generally omit this, we should
probably omit it here as well.

> +        self.send_packet_and_verify(mac_address=fake_address, should_receive=True)
> +        testpmd.set_mac_addr(0, mac_address=fake_address, add=False)
> +        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
> +
> +    def test_invalid_address(self) -> None:
> +        """Assess the behavior of a NIC mac address pool while binded to the PMD.

Should this be binded or bound? I think it's bound, but I'm not sure
if it's different when referring to drivers.

> +
> +        An assessment of a NIC's behavior when mounted to a PMD as it relates to mac addresses
> +        and address pooling. Devices should not be able to use invalid mac addresses, remove their
> +        built-in hardware address, or exceed their address pools.
> +
> +        Test:
> +            Start TestPMD.
> +            Attempt to set add an invalid mac address. (Should fail)

Typo: Attempt to set add an invalid address.

> +            Attempt to remove the device's hardware address. (Should fail)
> +            Add a fake mac address to the pool twice in succession. (Should not create any errors)
> +            Attempt to remove the device's hardware address. (Should fail)

I assume the reason this is checked again is making sure it still
fails when the device mac is not the only one in the pool, but we
should probably explain that.

> +            Determine the device's mac address pool size, and fill the pool with fake addresses.
> +            Attempt to add another fake mac address, overloading the address pool. (Should not work)

We say "should fail" everywhere else. Doesn't really change anything,
but it would probably flow a little better to keep this consistent.

> +        """
> +        testpmd = TestPmdShell(self.sut_node)
> +        testpmd.start()
> +        mac_address = self._sut_port_ingress.mac_address
> +        try:
> +            testpmd.set_mac_addr(0, "00:00:00:00:00:00")
> +            self.verify(False, "Invalid mac address added.")
> +        except InteractiveCommandExecutionError:
> +            pass
> +        try:
> +            testpmd.set_mac_addr(0, mac_address, False)
> +            self.verify(False, "Default mac address removed.")
> +        except InteractiveCommandExecutionError:
> +            pass
> +        # Should be no errors adding this twice
> +        testpmd.set_mac_addr(0, f"'1' + {mac_address[1:]}")
> +        testpmd.set_mac_addr(0, f"'1' + {mac_address[1:]}")
> +        # Double check to see if default mac address can be removed
> +        try:
> +            testpmd.set_mac_addr(0, mac_address, False)
> +            self.verify(False, "Default mac address removed.")
> +        except InteractiveCommandExecutionError:
> +            pass
> +
> +        for i in range(testpmd.show_port_info(0).max_mac_addresses_num - 1):
> +            number = str(hex(i)[2:].zfill(4))
> +            fake_address = f'{number[:2] + ":" + number[2:] + mac_address[5:]}'.upper()

It's not immediately obvious to me what this is doing, maybe a comment
of an example about what this would look like for iteration 1 vs
iteration 2 would help make it clear.

> +            testpmd.set_mac_addr(0, fake_address, verify=False)
> +        try:
> +            testpmd.set_mac_addr(0, f'{mac_address[:-1] + "1"}')

What happens here if the last character in the MAC address already is
1? This would fail the test case right?

> +            self.verify(False, "Mac address limit exceeded.")
> +        except InteractiveCommandExecutionError:
> +            pass
> +        testpmd.close()
> +
> +    def test_multicast_filter(self) -> None:
> +        """Assess basic multicast address filtering functionalities.
> +
> +        Ensure that multicast filtering performs as intended when a given device is bound
> +        to the PMD, with and without dot1q vlan tagging.
> +
> +        Test:
> +            Start TestPMD with promiscuous mode.
> +            Add a fake multicast address to the PMD's multicast address pool.
> +            Send a packet with the fake multicast address to the PMD. (Should receive)
> +            Set vlan filtering on the PMD, and add vlan ID to the PMD.

This test case seems like it is trying to test the multicast mac
filtering, do we need to worry about VLAN filtering here as well, or
just simply test when the VLAN tag is present? If possible it's
probably better to avoid the testing of VLAN filtering since that is a
different suite and I would assume that the functionality of filtering
based on VLANs isn't directly related to filtering based on MAC
addresses but I could be wrong.

> +            Send a packet with the fake multicast address and vlan ID to the PMD. (Should receive)
> +            Send a packet with the fake multicast address and a different vlan ID to the PMD.
> +                (Should not receive)
> +            Remove the vlan tag from the PMD, and turn vlan filtering off on the PMD.
> +            Send a packet with the fake multicast address and no vlan tag to the PMD.
> +                (Should receive)
> +            Remove the fake multicast address from the PMDs multicast address filter.
> +            Send a packet with the fake multicast address to the PMD. (Should not receive)
> +        """
<snip>

> --
> 2.44.0
>

Re: [PATCH 3/3] dts: mac filter test suite refactored for new dts

[Nicholas Pratte]

<snip>
> <snip>
> > +    def send_packet_and_verify(
> > +        self,
> > +        mac_address: str,
> > +        add_vlan: bool = False,
> > +        should_receive: bool = True,
> > +    ) -> None:
> > +        """Generate, send, and verify a packet based on specified parameters.
> > +
> > +        Test cases within this suite utilize this method to create, send, and verify
> > +        packets based on criteria relating to the packet's destination mac address,
> > +        vlan tag, and whether or not the packet should be received or not. Packets
> > +        are verified using an inserted payload. If the list of received packets
> > +        contains this payload within any of its packets, the test case passes. This
> > +        implementation does not require a quiet wire, each call with this method
>
> I don't think we need to mention that it doesn't require a quiet wire,
> in general this is something we are writing all test cases to assume.
>
> Looking at this though, it would probably be a great thing to add to
> the documentation for writing test suites. We never really mention
> anywhere that you must write test cases without this assumption, it's
> sort of just something we decided. Of course out of scope of this
> patch, but something for the future.

Noted. I'll get rid of this. I also wouldn't mind making these changes
in documentation at some point if others think it is a good idea.

>
> > +        sends exactly one packet.
> > +
> > +        Args:
> > +            mac_address: The destination mac address of the packet being sent.
> > +            add_vlan: Add a vlan tag to the packet being sent. :data:'2' if the packet
> > +                should be received, :data:'1' if the packet should not be received but
> > +                requires a vlan tag, and None for any other condition.
>
> This confused me a little because I thought you were saying this
> parameter would be 2 if the packet should be received or it would be 1
> in another case. I think this parameter is really just denoting if we
> should add a VLAN tag at all, and then the should_receive specifies
> what the tag will be. We should probably move the information about
> either being 2 or 1 to the `should_receive` documentation and specify
> that the VLAN tag will be that value based on if it should be
> received.
>
> This is a little hard because it relies on both conditions, but you
> can just say in the `should_recieve` parameter that, if there is a
> vlan tag, this is what it will be.

I think you make a good point here, it's probably easier to read if I
move the information like you said.

>
> > +            should_receive: If :data:'True', assert whether or not the sent packet
> > +                has been received. If :data:'False', assert that the send packet was not
> > +                received. :data:'True' by default
> > +        """
> > +        packet = (
> > +            Ether()
> > +            / Dot1Q(vlan=2 if add_vlan and should_receive else 1 if add_vlan else None)
>
> I haven't used the Dot1Q class in scapy very much myself, but it seems
> like just setting vlan=None is strange. Does this just make it
> completely omit the VLAN tag? Also, if we don't have a VLAN tag,

Yeah, it's funny you say this because, in doing some refactoring on
Jumbo Frames, I put two and two together that None is not technically
an acceptable parameter, although it tested fine, and I didn't get any
warnings, formatting errors, or otherwise when I was was publishing
the suite to the mailing list.

> should we just not include the Dot1Q layer at all? It feels like it
> doesn't really do anything if you don't have a VLAN.

This is the solution to the problem, yes. It shouldn't take any work
at all to rework this conditional. I guess my initial mindset was that
Scapy might be 'smart' enough to understand that a 'None' input in the
vlan attribute implies mitigating the vlan header entirely. In
retrospect it doesn't really make any sense, but when everything
tested fine, I think I was wrongfully convinced that I was right.
Making this change would effectively have the same output for the test
suite.

>
> > +            / IP()
> > +            / Raw(load="P" * 22)
> > +        )
> > +        packet.dst = mac_address
> > +        received_packets = [
> > +            packets
> > +            for packets in self.send_packet_and_capture(packet, adjust_addresses=False)
> > +            if hasattr(packets, "load") and "P" * 22 in str(packets.load)
> > +        ]
>
> i think this would be a little easier using the filter() built-in
> Python function. It would instead be:
> recieved_packets = [filter(lambda x: if hasattr(x, "load") and "P" *
> 22 in str(x.load),
>  self.send_packet_and_capture(packet, adjust_addresses=False)]
>
> It doesn't really save you that much verbosity now that I've written
> it out, you kind of trade the "for x in ..." for "filter(lambda x:
> ..." so maybe it isn't any more valuable. However, this function for
> if a packet has a certain payload is becoming more and more common, I
> wonder if we could just add a method to test_suite called
> packet_has_payload(p: Packet, payload: str) that just checks does the
> packet have the load attribute and does that load attribute match the
> desired payload. Or, even better, if we made it a high-order function
> that just accepts a payload and then returns a function that validates
> if a packet has that payload. Something like this:
>
> def packet_has_payload(payload: str) -> Callable[[Packet], bool]:
>     def wrap(p: Packet) -> bool:
>         return if hasattr(p, "load") and payload in str(p.load)
>     return wrap
>
> If we had this high-order function then filtering becomes super easy.
> The filter would just become:
> recieved_packets = [filter(self.packet_has_payload("P" * 22),
>  self.send_packet_and_capture(packet, adjust_addresses=False)]
>
> Maybe this is something that should be addressed in the bugzilla
> ticket 1372 however.

Took a lot of trial and error to get this big if statement to work
properly, I was wondering how long it would take before someone
mentions it. I agree that the ticket is probably the best spot to
discuss this, but there is definitely a better way of doing this. In
any case, your filter solution is more graceful than mine, but
ultimately I agree that the best solution is to just abstract this
component away from the test suite development process; it will be
much easier for developers to work on test suites with this out of
sight and out of mind.

>
> > +        if should_receive:
> > +            self.verify(len(received_packets) == 1, "Expected packet not received")
> > +        else:
> > +            self.verify(len(received_packets) == 0, "Expected packet received")
> > +
> > +    def test_add_remove_mac_addresses(self) -> None:
> > +        """Assess basic mac addressing filtering functionalities.
> > +
> > +        This test cases validates for proper behavior of mac address filtering with both
> > +        a port's default, burned-in mac address, as well as additional mac addresses
> > +        added to the PMD. Packets should either be received or not received depending on
> > +        the properties applied to the PMD at any given time.
> > +
> > +        Test:
> > +            Start TestPMD with promiscuous mode.
> > +            Send a packet with the port's default mac address. (Should receive)
> > +            Send a packet with fake mac address. (Should not receive)
> > +            Add fake mac address to the PMD's address pool.
> > +            Send a packet with the fake mac address to the PMD. (Should receive)
> > +            Remove the fake mac address from the PMD's address pool.
> > +            Sent a packet with the fake mac address to the PMD. (Should not receive)
>
> Typo: Sent a packet.
>
> > +        """
> > +        testpmd = TestPmdShell(self.sut_node)
> > +        testpmd.set_promisc(0, on=False, verify=False)
>
> Why don't we want to verify this command? If there is a reason we
> should probably leave a comment explaining it.

My idea at the time was that promiscuous mode is technically not
something that the test case is testing for, which seems a bit silly
from a debugging perspective. I'll make the change.

>
> > +        testpmd.start()
> > +        mac_address = self._sut_port_ingress.mac_address
> > +
> > +        # Send a packet with NIC default mac address
> > +        self.send_packet_and_verify(mac_address=mac_address, should_receive=True)
> > +        # Send a packet with different mac address
> > +        fake_address = "00:00:00:00:00:01"
> > +        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
> > +
> > +        # Add mac address to pool and rerun tests
> > +        testpmd.set_mac_addr(0, mac_address=fake_address, verify=True)
>
> verify=True if the default and we generally omit this, we should
> probably omit it here as well.

Noted.

>
> > +        self.send_packet_and_verify(mac_address=fake_address, should_receive=True)
> > +        testpmd.set_mac_addr(0, mac_address=fake_address, add=False)
> > +        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
> > +
> > +    def test_invalid_address(self) -> None:
> > +        """Assess the behavior of a NIC mac address pool while binded to the PMD.
>
> Should this be binded or bound? I think it's bound, but I'm not sure
> if it's different when referring to drivers.

I don't think binded is a word... I'll get rid of that. That's a good catch!
>
> > +
> > +        An assessment of a NIC's behavior when mounted to a PMD as it relates to mac addresses
> > +        and address pooling. Devices should not be able to use invalid mac addresses, remove their
> > +        built-in hardware address, or exceed their address pools.
> > +
> > +        Test:
> > +            Start TestPMD.
> > +            Attempt to set add an invalid mac address. (Should fail)
>
> Typo: Attempt to set add an invalid address.
>
> > +            Attempt to remove the device's hardware address. (Should fail)

I'll add an additional explanation here, with respect to your comment below.

> > +            Add a fake mac address to the pool twice in succession. (Should not create any errors)
> > +            Attempt to remove the device's hardware address. (Should fail)
>
> I assume the reason this is checked again is making sure it still
> fails when the device mac is not the only one in the pool, but we
> should probably explain that.

Good point! I'll add some additional explanation for this line and the
one above.

>
> > +            Determine the device's mac address pool size, and fill the pool with fake addresses.
> > +            Attempt to add another fake mac address, overloading the address pool. (Should not work)
>
> We say "should fail" everywhere else. Doesn't really change anything,
> but it would probably flow a little better to keep this consistent.
>
> > +        """
> > +        testpmd = TestPmdShell(self.sut_node)
> > +        testpmd.start()
> > +        mac_address = self._sut_port_ingress.mac_address
> > +        try:
> > +            testpmd.set_mac_addr(0, "00:00:00:00:00:00")
> > +            self.verify(False, "Invalid mac address added.")
> > +        except InteractiveCommandExecutionError:
> > +            pass
> > +        try:
> > +            testpmd.set_mac_addr(0, mac_address, False)
> > +            self.verify(False, "Default mac address removed.")
> > +        except InteractiveCommandExecutionError:
> > +            pass
> > +        # Should be no errors adding this twice
> > +        testpmd.set_mac_addr(0, f"'1' + {mac_address[1:]}")
> > +        testpmd.set_mac_addr(0, f"'1' + {mac_address[1:]}")
> > +        # Double check to see if default mac address can be removed
> > +        try:
> > +            testpmd.set_mac_addr(0, mac_address, False)
> > +            self.verify(False, "Default mac address removed.")
> > +        except InteractiveCommandExecutionError:
> > +            pass
> > +
> > +        for i in range(testpmd.show_port_info(0).max_mac_addresses_num - 1):
> > +            number = str(hex(i)[2:].zfill(4))
> > +            fake_address = f'{number[:2] + ":" + number[2:] + mac_address[5:]}'.upper()
>
> It's not immediately obvious to me what this is doing, maybe a comment
> of an example about what this would look like for iteration 1 vs
> iteration 2 would help make it clear.

Sure, that seems reasonable. For the record, this adds a bunch of fake
mac addresses to fill the entire mac address pool on a given NIC. I'm
actually going to refactor this to make it easier to read.

>
> > +            testpmd.set_mac_addr(0, fake_address, verify=False)
> > +        try:
> > +            testpmd.set_mac_addr(0, f'{mac_address[:-1] + "1"}')
>
> What happens here if the last character in the MAC address already is
> 1? This would fail the test case right?

Given the way it was designed above, only the first couple of octets
would be manipulated to create fake addresses. So there should never
be a circumstance where this address is used.

>
> > +            self.verify(False, "Mac address limit exceeded.")
> > +        except InteractiveCommandExecutionError:
> > +            pass
> > +        testpmd.close()
> > +
> > +    def test_multicast_filter(self) -> None:
> > +        """Assess basic multicast address filtering functionalities.
> > +
> > +        Ensure that multicast filtering performs as intended when a given device is bound
> > +        to the PMD, with and without dot1q vlan tagging.
> > +
> > +        Test:
> > +            Start TestPMD with promiscuous mode.
> > +            Add a fake multicast address to the PMD's multicast address pool.
> > +            Send a packet with the fake multicast address to the PMD. (Should receive)
> > +            Set vlan filtering on the PMD, and add vlan ID to the PMD.
>
> This test case seems like it is trying to test the multicast mac
> filtering, do we need to worry about VLAN filtering here as well, or
> just simply test when the VLAN tag is present? If possible it's
> probably better to avoid the testing of VLAN filtering since that is a
> different suite and I would assume that the functionality of filtering
> based on VLANs isn't directly related to filtering based on MAC
> addresses but I could be wrong.

It's a good point, and it was kind of a head scratcher for me as well,
but I threw it in the test suite since vlan filtering on multicast was
also conducted in the old DTS test suite. I'd be willing to remove
this if people generally agree it's not needed because I also agree
that it's seemingly tangential from the rest of the test suite.
<snip>

[PATCH v2 0/3] dts: mac filter port to new dts

[Nicholas Pratte]

v2:
  - refactored mac filter suite and cleared out hidden bugs.
  - add/remove mac and multicast addresses code semantics
      improvements.
  - minor documentation changes based on suggestions.

Nicholas Pratte (3):
  dts: add boolean to adjust addresses
  dts: add methods for setting mac and multicast addresses
  dts: mac filter test suite refactored for new dts

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++
 dts/framework/test_suite.py                   |   7 +-
 dts/tests/TestSuite_mac_filter.py             | 220 ++++++++++++++++++
 4 files changed, 405 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

-- 
2.44.0

[PATCH v2 1/3] dts: add boolean to adjust addresses

[Nicholas Pratte]

Various test cases in the mac filter test suite called for granular
manipulation of destination mac addresses to properly test mac address
filtering functionality. To compensate, there is now an
adjust_addresses boolean which the user can toggle if they wish to send
their own addressing; the boolean is true by default.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/test_suite.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 694b2eba65..551a587525 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -185,6 +185,7 @@ def send_packet_and_capture(
         packet: Packet,
         filter_config: PacketFilteringConfig = PacketFilteringConfig(),
         duration: float = 1,
+        adjust_addresses: bool = True,
     ) -> list[Packet]:
         """Send and receive `packet` using the associated TG.
 
@@ -195,11 +196,15 @@ def send_packet_and_capture(
             packet: The packet to send.
             filter_config: The filter to use when capturing packets.
             duration: Capture traffic for this amount of time after sending `packet`.
+            adjust_addresses: If :data:'True', adjust addresses of the egressing packet with
+                a default addressing scheme. If :data:'False', do not adjust the addresses of
+                egressing packet.
 
         Returns:
             A list of received packets.
         """
-        packet = self._adjust_addresses(packet)
+        if adjust_addresses:
+            packet = self._adjust_addresses(packet)
         return self.tg_node.send_packet_and_capture(
             packet,
             self._tg_port_egress,
-- 
2.44.0

[PATCH v2 2/3] dts: add methods for setting mac and multicast addresses

[Nicholas Pratte]

Several new methods have been added to TestPMDShell in order to produce
the mac filter's individual test cases:
 - set_mac_addr
 - set_multicast_mac_addr
 - rx_vlan_add
 - rx_vlan_rm
 - vlan_filter_set_on
 - vlan_filter_set_off
 - set_promisc

set_mac_addr and set_multicast_addr were created for the mac filter test
suite, enabling users to both add or remove mac and multicast
addresses based on a booling 'add or remove' parameter. The success or
failure of each call can be verified if a user deems it necessary.

The other methods listed are implemented in other respective test
suites, and their implementations have been copied, but are subject to
change; they are not the focus of this patch.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++++++
 dts/tests/TestSuite_mac_filter.py             |   0
 2 files changed, 177 insertions(+)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index ec22f72221..0be1fb8754 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -767,6 +767,183 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
 
         return TestPmdPort.parse(output)
 
+    def set_mac_addr(self, port_id: int, mac_address: str, add: bool, verify: bool = True):
+        """Add or remove a mac address on a given port's Allowlist.
+
+        Args:
+            port_id: The port ID the mac address is set on.
+            mac_address: The mac address to be added or removed to the specified port.
+            add: If :data:`True`, add the specified mac address. If :data:`False`, remove specified
+                mac address.
+            verify: If :data:'True', assert that the 'mac_addr' operation was successful. If
+                :data:'False', run the command and skip this assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If the set mac address operation fails.
+        """
+        mac_cmd = "add" if add else "remove"
+        output = self.send_command(f"mac_addr {mac_cmd} {port_id} {mac_address}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid argument provided to mac_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if "mac_addr_cmd error:" in output:
+                self._logger.debug(f"Failed to {mac_cmd} {mac_address} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mac_cmd} {mac_address} on port {port_id} \n{output}"
+                )
+
+    def set_multicast_mac_addr(self, port_id: int, multi_addr: str, add: bool, verify: bool = True):
+        """Add or remove multicast mac address to a specified port filter.
+
+        Args:
+            port_id: The port ID the multicast address is set on.
+            multi_addr: The multicast address to be added to the filter.
+            add: If :data:'True', add the specified multicast address to the port filter.
+                If :data:'False', remove the specified multicast address from the port filter.
+            verify: If :data:'True', assert that the 'mcast_addr' operations was successful.
+                If :data:'False', execute the 'mcast_addr' operation and skip the assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fails.
+        """
+        mcast_cmd = "add" if add else "remove"
+        output = self.send_command(f"mcast_addr {mcast_cmd} {port_id} {multi_addr}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid arguments provided to mcast_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if (
+                "Invalid multicast_addr" in output
+                or f'multicast address {"already" if add else "not"} filtered by port' in output
+            ):
+                self._logger.debug(f"Failed to {mcast_cmd} {multi_addr} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mcast_cmd} {multi_addr} on port {port_id} \n{output}"
+                )
+
+    def rx_vlan_add(self, vlan: int, port: int, verify: bool = True):
+        """Add specified vlan tag to the filter list on a port.
+
+        Args:
+            vlan: The vlan tag to add, should be within 1-1005, 1-4094 extended.
+            port: The port number to add the tag on, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was added to the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not added.
+        """
+        vlan_add_output = self.send_command(f"rx_vlan add {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_add_output or "Invalid vlan_id" in vlan_add_output:
+                self._logger.debug(
+                    f"Failed to add vlan tag {vlan} on port {port}: \n{vlan_add_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to add vlan tag {vlan} on port {port}."
+                )
+
+    def rx_vlan_rm(self, vlan: int, port: int, verify: bool = True):
+        """Remove specified vlan tag from filter list on a port.
+
+        Args:
+            vlan: The vlan tag to remove, should be within 1-4094.
+            port: The port number to remove the tag from, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was removed from the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not removed.
+        """
+        vlan_rm_output = self.send_command(f"rx_vlan rm {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_rm_output or "Invalid vlan_id" in vlan_rm_output:
+                self._logger.debug(
+                    f"Failed to remove vlan tag {vlan} on port {port}: \n{vlan_rm_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to remove vlan tag {vlan} on port {port}."
+                )
+
+    def vlan_filter_set_on(self, port: int, verify: bool = True):
+        """Set vlan filter on.
+
+        Args:
+            port: The port number to enable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was enabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter on {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter on" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to enable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to enable vlan filter on port {port}."
+                )
+
+    def vlan_filter_set_off(self, port: int, verify: bool = True):
+        """Set vlan filter off.
+
+        Args:
+            port: The port number to disable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was disabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter off {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter off" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to disable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to disable vlan filter on port {port}."
+                )
+
+    def set_promisc(self, port: int, on: bool, verify: bool = True):
+        """Turns promiscuous mode on/off for the specified port.
+
+        Args:
+            port: Port number to use, should be within 0-32.
+            on: If :data:`True`, turn promisc mode on, otherwise turn off.
+            verify: If :data:`True` an additional command will be sent to verify that promisc mode
+                is properly set. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and promisc mode
+                is not correctly set.
+        """
+        promisc_output = self.send_command(f"set promisc {port} {'on' if on else 'off'}")
+        if verify:
+            stats = self.show_port_info(port_id=port)
+            if on ^ stats.is_promiscuous_mode_enabled:
+                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set promisc mode on port {port}."
+                )
+
     def show_port_stats_all(self) -> list[TestPmdPortStats]:
         """Returns the statistics of all the ports.
 
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
new file mode 100644
index 0000000000..e69de29bb2
-- 
2.44.0

[PATCH v2 3/3] dts: mac filter test suite refactored for new dts

[Nicholas Pratte]

The mac address filter test suite, whose test cases are based on old
DTS's test cases, has been refactored to interface with the new DTS
framework.

In porting over this test suite into the new framework, some
adjustments were made, namely in the EAL and TestPMD parameter provided
before executing the application. While the original test plan was
referenced, by and large, only for the individual test cases, I'll leave
the parameters the original test plan was asking for below for the sake
of discussion:

--burst=1 --rxpt=0 --rxht=0 --rxwt=0 --txpt=36 --txht=0 --txwt=0
--txfreet=32 --rxfreet=64 --mbcache=250 --portmask=0x3

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>

---
v2:
 * Refactored the address pool capacity tests to use all available
   octets in the mac address.
 * Change the payload to 'X' characters instead of 'P' characters.
---
 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_mac_filter.py          | 220 +++++++++++++++++++++
 2 files changed, 222 insertions(+), 1 deletion(-)

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..ad1f3757f7 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "mac_filter"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
index e69de29bb2..c960d6b36c 100644
--- a/dts/tests/TestSuite_mac_filter.py
+++ b/dts/tests/TestSuite_mac_filter.py
@@ -0,0 +1,220 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023-2024 University of New Hampshire
+"""Mac address filtering test suite.
+
+This test suite ensures proper and expected behavior of Allowlist filtering via mac
+addresses on devices bound to the Poll Mode Driver. If a packet received on a device
+contains a mac address not contained with its mac address pool, the packet should
+be dropped. Alternatively, if a packet is received that contains a destination mac
+within the devices address pool, the packet should be accepted and forwarded. This
+behavior should remain consistent across all packets, namely those containing dot1q
+tags or otherwise.
+
+The following test suite assesses behaviors based on the aforementioned logic.
+Additionally, testing is done within the PMD itself to ensure that the mac address
+allow list is behaving as expected.
+"""
+
+from time import sleep
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Dot1Q, Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.exception import InteractiveCommandExecutionError
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+
+class TestMacFilter(TestSuite):
+    """Mac address allowlist filtering test suite.
+
+    Configure mac address filtering on a given port, and test the port's filtering behavior
+    using both a given port's hardware address as well as dummy addresses. If a port accepts
+    a packet that is not contained within its mac address allowlist, then a given test case
+    fails. Alternatively, if a port drops a packet that is designated within its mac address
+    allowlist, a given test case will fail.
+
+    Moreover, a given port should demonstrate proper behavior when bound to the Poll Mode
+    Driver. A port should not have an mac address allowlist that exceeds its designated size.
+    A port's default hardware address should not be removed from its address pool, and invalid
+    addresses should not be included in the allowlist. If a port abides by the above rules, the
+    test case passes.
+    """
+
+    def send_packet_and_verify(
+        self,
+        mac_address: str,
+        add_vlan: bool = False,
+        should_receive: bool = True,
+    ) -> None:
+        """Generate, send, and verify a packet based on specified parameters.
+
+        Test cases within this suite utilize this method to create, send, and verify
+        packets based on criteria relating to the packet's destination mac address,
+        vlan tag, and whether or not the packet should be received or not. Packets
+        are verified using an inserted payload. If the list of received packets
+        contains this payload within any of its packets, the test case passes. Each
+        call with this method sends exactly one packet.
+
+        Args:
+            mac_address: The destination mac address of the packet being sent.
+            add_vlan: Add a vlan tag to the packet being sent. :data:'2' if the packet
+                should be received, :data:'1' if the packet should not be received but
+                requires a vlan tag, and None for any other condition.
+            should_receive: If :data:'True', assert whether or not the sent packet
+                has been received. If :data:'False', assert that the send packet was not
+                received. :data:'True' by default
+        """
+        if add_vlan:
+            packet = Ether() / Dot1Q(vlan=2 if should_receive else 1) / IP() / Raw(load="X" * 22)
+        else:
+            packet = Ether() / IP() / Raw(load="X" * 22)
+        packet.dst = mac_address
+        received_packets = [
+            packets
+            for packets in self.send_packet_and_capture(packet, adjust_addresses=False)
+            if hasattr(packets, "load") and "X" * 22 in str(packets.load)
+        ]
+        if should_receive:
+            self.verify(len(received_packets) == 1, "Expected packet not received")
+        else:
+            self.verify(len(received_packets) == 0, "Expected packet received")
+
+    def test_add_remove_mac_addresses(self) -> None:
+        """Assess basic mac addressing filtering functionalities.
+
+        This test cases validates for proper behavior of mac address filtering with both
+        a port's default, burned-in mac address, as well as additional mac addresses
+        added to the PMD. Packets should either be received or not received depending on
+        the properties applied to the PMD at any given time.
+
+        Test:
+            Start TestPMD with promiscuous mode.
+            Send a packet with the port's default mac address. (Should receive)
+            Send a packet with fake mac address. (Should not receive)
+            Add fake mac address to the PMD's address pool.
+            Send a packet with the fake mac address to the PMD. (Should receive)
+            Remove the fake mac address from the PMD's address pool.
+            Sent a packet with the fake mac address to the PMD. (Should not receive)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.set_promisc(0, on=False)
+        testpmd.start()
+        mac_address = self._sut_port_ingress.mac_address
+
+        # Send a packet with NIC default mac address
+        self.send_packet_and_verify(mac_address=mac_address, should_receive=True)
+        # Send a packet with different mac address
+        fake_address = "00:00:00:00:00:01"
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
+
+        # Add mac address to pool and rerun tests
+        testpmd.set_mac_addr(0, mac_address=fake_address, add=True)
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=True)
+        testpmd.set_mac_addr(0, mac_address=fake_address, add=False)
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
+        testpmd.close()
+        sleep(6)
+
+    def test_invalid_address(self) -> None:
+        """Assess the behavior of a NIC mac address pool while bound to the PMD.
+
+        An assessment of a NIC's behavior when mounted to a PMD as it relates to mac addresses
+        and address pooling. Devices should not be able to use invalid mac addresses, remove their
+        built-in hardware address, or exceed their address pools.
+
+        Test:
+            Start TestPMD.
+            Attempt to add an invalid mac address. (Should fail)
+            Attempt to remove the device's hardware address with no additional addresses in the
+                address pool. (Should fail)
+            Add a fake mac address to the pool twice in succession. (Should not create any errors)
+            Attempt to remove the device's hardware address with other addresses in the address
+                pool. (Should fail)
+            Determine the device's mac address pool size, and fill the pool with fake addresses.
+            Attempt to add another fake mac address, overloading the address pool. (Should fail)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.start()
+        mac_address = self._sut_port_ingress.mac_address
+        try:
+            testpmd.set_mac_addr(0, "00:00:00:00:00:00", add=True)
+            self.verify(False, "Invalid mac address added.")
+        except InteractiveCommandExecutionError:
+            pass
+        try:
+            testpmd.set_mac_addr(0, mac_address, add=False)
+            self.verify(False, "Default mac address removed.")
+        except InteractiveCommandExecutionError:
+            pass
+        # Should be no errors adding this twice
+        testpmd.set_mac_addr(0, "1" + mac_address[1:], add=True)
+        testpmd.set_mac_addr(0, "1" + mac_address[1:], add=True)
+        # Double check to see if default mac address can be removed
+        try:
+            testpmd.set_mac_addr(0, mac_address, add=False)
+            self.verify(False, "Default mac address removed.")
+        except InteractiveCommandExecutionError:
+            pass
+
+        for i in range(testpmd.show_port_info(0).max_mac_addresses_num - 1):
+            # A0 fake address based on the index 'i'.
+            fake_address = str(hex(i)[2:].zfill(12))
+            # Insert ':' characters every two indexes to create a fake mac address.
+            fake_address = ":".join(fake_address[x : x + 2] for x in range(0, len(fake_address), 2))
+            testpmd.set_mac_addr(0, fake_address, add=True, verify=False)
+        try:
+            testpmd.set_mac_addr(0, "F" + mac_address[1:], add=True)
+            self.verify(False, "Mac address limit exceeded.")
+        except InteractiveCommandExecutionError:
+            pass
+        testpmd.close()
+        sleep(6)
+
+    def test_multicast_filter(self) -> None:
+        """Assess basic multicast address filtering functionalities.
+
+        Ensure that multicast filtering performs as intended when a given device is bound
+        to the PMD, with and without dot1q vlan tagging.
+
+        Test:
+            Start TestPMD with promiscuous mode.
+            Add a fake multicast address to the PMD's multicast address pool.
+            Send a packet with the fake multicast address to the PMD. (Should receive)
+            Set vlan filtering on the PMD, and add vlan ID to the PMD.
+            Send a packet with the fake multicast address and vlan ID to the PMD. (Should receive)
+            Send a packet with the fake multicast address and a different vlan ID to the PMD.
+                (Should not receive)
+            Remove the vlan tag from the PMD, and turn vlan filtering off on the PMD.
+            Send a packet with the fake multicast address and no vlan tag to the PMD.
+                (Should receive)
+            Remove the fake multicast address from the PMDs multicast address filter.
+            Send a packet with the fake multicast address to the PMD. (Should not receive)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.start()
+        testpmd.set_promisc(0, on=False)
+        multicast_address = "01:00:5E:00:00:00"
+        vlan_id = 2
+
+        testpmd.set_multicast_mac_addr(0, multi_addr=multicast_address, add=True)
+        self.send_packet_and_verify(multicast_address, should_receive=True)
+
+        # Test vlan filtering on multicast addressing.
+        # Verify vlan functionality for debugging purposes.
+        testpmd.vlan_filter_set_on(port=0)
+        testpmd.rx_vlan_add(vlan_id, 0)
+        self.send_packet_and_verify(multicast_address, should_receive=True, add_vlan=True)
+        self.send_packet_and_verify(multicast_address, should_receive=False, add_vlan=True)
+
+        # Remove vlan tag and filtering and run basic multicast addr test.
+        testpmd.rx_vlan_rm(vlan_id, 0)
+        testpmd.vlan_filter_set_off(port=0)
+        self.send_packet_and_verify(multicast_address, should_receive=True)
+
+        # Remove multicast filter and verify the packet was not received.
+        testpmd.set_multicast_mac_addr(0, multicast_address, add=False)
+        self.send_packet_and_verify(multicast_address, should_receive=False)
+        testpmd.close()
+        sleep(6)
-- 
2.44.0

[PATCH v2 0/3] dts: mac filter port to new dts

[Nicholas Pratte]

v2:
  - refactored mac filter suite and cleared out hidden bugs.
  - add/remove mac and multicast addresses code semantics
      improvements.
  - minor documentation changes based on suggestions.

Nicholas Pratte (3):
  dts: add boolean to adjust addresses
  dts: add methods for setting mac and multicast addresses
  dts: mac filter test suite refactored for new dts

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++
 dts/framework/test_suite.py                   |   7 +-
 dts/tests/TestSuite_mac_filter.py             | 220 ++++++++++++++++++
 4 files changed, 405 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

-- 
2.44.0

[PATCH v2 1/3] dts: add boolean to adjust addresses

[Nicholas Pratte]

Various test cases in the mac filter test suite called for granular
manipulation of destination mac addresses to properly test mac address
filtering functionality. To compensate, there is now an
adjust_addresses boolean which the user can toggle if they wish to send
their own addressing; the boolean is true by default.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/test_suite.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 694b2eba65..551a587525 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -185,6 +185,7 @@ def send_packet_and_capture(
         packet: Packet,
         filter_config: PacketFilteringConfig = PacketFilteringConfig(),
         duration: float = 1,
+        adjust_addresses: bool = True,
     ) -> list[Packet]:
         """Send and receive `packet` using the associated TG.
 
@@ -195,11 +196,15 @@ def send_packet_and_capture(
             packet: The packet to send.
             filter_config: The filter to use when capturing packets.
             duration: Capture traffic for this amount of time after sending `packet`.
+            adjust_addresses: If :data:'True', adjust addresses of the egressing packet with
+                a default addressing scheme. If :data:'False', do not adjust the addresses of
+                egressing packet.
 
         Returns:
             A list of received packets.
         """
-        packet = self._adjust_addresses(packet)
+        if adjust_addresses:
+            packet = self._adjust_addresses(packet)
         return self.tg_node.send_packet_and_capture(
             packet,
             self._tg_port_egress,
-- 
2.44.0

[PATCH v2 2/3] dts: add methods for setting mac and multicast addresses

[Nicholas Pratte]

Several new methods have been added to TestPMDShell in order to produce
the mac filter's individual test cases:
 - set_mac_addr
 - set_multicast_mac_addr
 - rx_vlan_add
 - rx_vlan_rm
 - vlan_filter_set_on
 - vlan_filter_set_off
 - set_promisc

set_mac_addr and set_multicast_addr were created for the mac filter test
suite, enabling users to both add or remove mac and multicast
addresses based on a booling 'add or remove' parameter. The success or
failure of each call can be verified if a user deems it necessary.

The other methods listed are implemented in other respective test
suites, and their implementations have been copied, but are subject to
change; they are not the focus of this patch.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++++++
 dts/tests/TestSuite_mac_filter.py             |   0
 2 files changed, 177 insertions(+)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index ec22f72221..0be1fb8754 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -767,6 +767,183 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
 
         return TestPmdPort.parse(output)
 
+    def set_mac_addr(self, port_id: int, mac_address: str, add: bool, verify: bool = True):
+        """Add or remove a mac address on a given port's Allowlist.
+
+        Args:
+            port_id: The port ID the mac address is set on.
+            mac_address: The mac address to be added or removed to the specified port.
+            add: If :data:`True`, add the specified mac address. If :data:`False`, remove specified
+                mac address.
+            verify: If :data:'True', assert that the 'mac_addr' operation was successful. If
+                :data:'False', run the command and skip this assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If the set mac address operation fails.
+        """
+        mac_cmd = "add" if add else "remove"
+        output = self.send_command(f"mac_addr {mac_cmd} {port_id} {mac_address}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid argument provided to mac_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if "mac_addr_cmd error:" in output:
+                self._logger.debug(f"Failed to {mac_cmd} {mac_address} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mac_cmd} {mac_address} on port {port_id} \n{output}"
+                )
+
+    def set_multicast_mac_addr(self, port_id: int, multi_addr: str, add: bool, verify: bool = True):
+        """Add or remove multicast mac address to a specified port filter.
+
+        Args:
+            port_id: The port ID the multicast address is set on.
+            multi_addr: The multicast address to be added to the filter.
+            add: If :data:'True', add the specified multicast address to the port filter.
+                If :data:'False', remove the specified multicast address from the port filter.
+            verify: If :data:'True', assert that the 'mcast_addr' operations was successful.
+                If :data:'False', execute the 'mcast_addr' operation and skip the assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fails.
+        """
+        mcast_cmd = "add" if add else "remove"
+        output = self.send_command(f"mcast_addr {mcast_cmd} {port_id} {multi_addr}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid arguments provided to mcast_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if (
+                "Invalid multicast_addr" in output
+                or f'multicast address {"already" if add else "not"} filtered by port' in output
+            ):
+                self._logger.debug(f"Failed to {mcast_cmd} {multi_addr} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mcast_cmd} {multi_addr} on port {port_id} \n{output}"
+                )
+
+    def rx_vlan_add(self, vlan: int, port: int, verify: bool = True):
+        """Add specified vlan tag to the filter list on a port.
+
+        Args:
+            vlan: The vlan tag to add, should be within 1-1005, 1-4094 extended.
+            port: The port number to add the tag on, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was added to the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not added.
+        """
+        vlan_add_output = self.send_command(f"rx_vlan add {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_add_output or "Invalid vlan_id" in vlan_add_output:
+                self._logger.debug(
+                    f"Failed to add vlan tag {vlan} on port {port}: \n{vlan_add_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to add vlan tag {vlan} on port {port}."
+                )
+
+    def rx_vlan_rm(self, vlan: int, port: int, verify: bool = True):
+        """Remove specified vlan tag from filter list on a port.
+
+        Args:
+            vlan: The vlan tag to remove, should be within 1-4094.
+            port: The port number to remove the tag from, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was removed from the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not removed.
+        """
+        vlan_rm_output = self.send_command(f"rx_vlan rm {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_rm_output or "Invalid vlan_id" in vlan_rm_output:
+                self._logger.debug(
+                    f"Failed to remove vlan tag {vlan} on port {port}: \n{vlan_rm_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to remove vlan tag {vlan} on port {port}."
+                )
+
+    def vlan_filter_set_on(self, port: int, verify: bool = True):
+        """Set vlan filter on.
+
+        Args:
+            port: The port number to enable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was enabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter on {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter on" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to enable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to enable vlan filter on port {port}."
+                )
+
+    def vlan_filter_set_off(self, port: int, verify: bool = True):
+        """Set vlan filter off.
+
+        Args:
+            port: The port number to disable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was disabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter off {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter off" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to disable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to disable vlan filter on port {port}."
+                )
+
+    def set_promisc(self, port: int, on: bool, verify: bool = True):
+        """Turns promiscuous mode on/off for the specified port.
+
+        Args:
+            port: Port number to use, should be within 0-32.
+            on: If :data:`True`, turn promisc mode on, otherwise turn off.
+            verify: If :data:`True` an additional command will be sent to verify that promisc mode
+                is properly set. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and promisc mode
+                is not correctly set.
+        """
+        promisc_output = self.send_command(f"set promisc {port} {'on' if on else 'off'}")
+        if verify:
+            stats = self.show_port_info(port_id=port)
+            if on ^ stats.is_promiscuous_mode_enabled:
+                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set promisc mode on port {port}."
+                )
+
     def show_port_stats_all(self) -> list[TestPmdPortStats]:
         """Returns the statistics of all the ports.
 
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
new file mode 100644
index 0000000000..e69de29bb2
-- 
2.44.0

[PATCH v2 0/3] Mac Filter Port to New DTS

[Nicholas Pratte]

v2:
  - refactored mac filter suite and cleared out hidden bugs.
  - add/remove mac and multicast addresses code semantics
      improvements.
  - minor documentation changes based on suggestions.

Nicholas Pratte (3):
  dts: add boolean to adjust addresses
  dts: add methods for setting mac and multicast addresses
  dts: mac filter test suite refactored for new dts

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++
 dts/framework/test_suite.py                   |   7 +-
 dts/tests/TestSuite_mac_filter.py             | 220 ++++++++++++++++++
 4 files changed, 405 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

-- 
2.44.0

[PATCH v2 1/3] dts: add boolean to adjust addresses

[Nicholas Pratte]

Various test cases in the mac filter test suite called for granular
manipulation of destination mac addresses to properly test mac address
filtering functionality. To compensate, there is now an
adjust_addresses boolean which the user can toggle if they wish to send
their own addressing; the boolean is true by default.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/test_suite.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 694b2eba65..551a587525 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -185,6 +185,7 @@ def send_packet_and_capture(
         packet: Packet,
         filter_config: PacketFilteringConfig = PacketFilteringConfig(),
         duration: float = 1,
+        adjust_addresses: bool = True,
     ) -> list[Packet]:
         """Send and receive `packet` using the associated TG.
 
@@ -195,11 +196,15 @@ def send_packet_and_capture(
             packet: The packet to send.
             filter_config: The filter to use when capturing packets.
             duration: Capture traffic for this amount of time after sending `packet`.
+            adjust_addresses: If :data:'True', adjust addresses of the egressing packet with
+                a default addressing scheme. If :data:'False', do not adjust the addresses of
+                egressing packet.
 
         Returns:
             A list of received packets.
         """
-        packet = self._adjust_addresses(packet)
+        if adjust_addresses:
+            packet = self._adjust_addresses(packet)
         return self.tg_node.send_packet_and_capture(
             packet,
             self._tg_port_egress,
-- 
2.44.0

[PATCH v2 2/3] dts: add methods for setting mac and multicast addresses

[Nicholas Pratte]

Several new methods have been added to TestPMDShell in order to produce
the mac filter's individual test cases:
 - set_mac_addr
 - set_multicast_mac_addr
 - rx_vlan_add
 - rx_vlan_rm
 - vlan_filter_set_on
 - vlan_filter_set_off
 - set_promisc

set_mac_addr and set_multicast_addr were created for the mac filter test
suite, enabling users to both add or remove mac and multicast
addresses based on a booling 'add or remove' parameter. The success or
failure of each call can be verified if a user deems it necessary.

The other methods listed are implemented in other respective test
suites, and their implementations have been copied, but are subject to
change; they are not the focus of this patch.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++++++
 dts/tests/TestSuite_mac_filter.py             |   0
 2 files changed, 177 insertions(+)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index ec22f72221..0be1fb8754 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -767,6 +767,183 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
 
         return TestPmdPort.parse(output)
 
+    def set_mac_addr(self, port_id: int, mac_address: str, add: bool, verify: bool = True):
+        """Add or remove a mac address on a given port's Allowlist.
+
+        Args:
+            port_id: The port ID the mac address is set on.
+            mac_address: The mac address to be added or removed to the specified port.
+            add: If :data:`True`, add the specified mac address. If :data:`False`, remove specified
+                mac address.
+            verify: If :data:'True', assert that the 'mac_addr' operation was successful. If
+                :data:'False', run the command and skip this assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If the set mac address operation fails.
+        """
+        mac_cmd = "add" if add else "remove"
+        output = self.send_command(f"mac_addr {mac_cmd} {port_id} {mac_address}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid argument provided to mac_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if "mac_addr_cmd error:" in output:
+                self._logger.debug(f"Failed to {mac_cmd} {mac_address} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mac_cmd} {mac_address} on port {port_id} \n{output}"
+                )
+
+    def set_multicast_mac_addr(self, port_id: int, multi_addr: str, add: bool, verify: bool = True):
+        """Add or remove multicast mac address to a specified port filter.
+
+        Args:
+            port_id: The port ID the multicast address is set on.
+            multi_addr: The multicast address to be added to the filter.
+            add: If :data:'True', add the specified multicast address to the port filter.
+                If :data:'False', remove the specified multicast address from the port filter.
+            verify: If :data:'True', assert that the 'mcast_addr' operations was successful.
+                If :data:'False', execute the 'mcast_addr' operation and skip the assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fails.
+        """
+        mcast_cmd = "add" if add else "remove"
+        output = self.send_command(f"mcast_addr {mcast_cmd} {port_id} {multi_addr}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid arguments provided to mcast_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if (
+                "Invalid multicast_addr" in output
+                or f'multicast address {"already" if add else "not"} filtered by port' in output
+            ):
+                self._logger.debug(f"Failed to {mcast_cmd} {multi_addr} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mcast_cmd} {multi_addr} on port {port_id} \n{output}"
+                )
+
+    def rx_vlan_add(self, vlan: int, port: int, verify: bool = True):
+        """Add specified vlan tag to the filter list on a port.
+
+        Args:
+            vlan: The vlan tag to add, should be within 1-1005, 1-4094 extended.
+            port: The port number to add the tag on, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was added to the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not added.
+        """
+        vlan_add_output = self.send_command(f"rx_vlan add {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_add_output or "Invalid vlan_id" in vlan_add_output:
+                self._logger.debug(
+                    f"Failed to add vlan tag {vlan} on port {port}: \n{vlan_add_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to add vlan tag {vlan} on port {port}."
+                )
+
+    def rx_vlan_rm(self, vlan: int, port: int, verify: bool = True):
+        """Remove specified vlan tag from filter list on a port.
+
+        Args:
+            vlan: The vlan tag to remove, should be within 1-4094.
+            port: The port number to remove the tag from, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was removed from the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not removed.
+        """
+        vlan_rm_output = self.send_command(f"rx_vlan rm {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_rm_output or "Invalid vlan_id" in vlan_rm_output:
+                self._logger.debug(
+                    f"Failed to remove vlan tag {vlan} on port {port}: \n{vlan_rm_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to remove vlan tag {vlan} on port {port}."
+                )
+
+    def vlan_filter_set_on(self, port: int, verify: bool = True):
+        """Set vlan filter on.
+
+        Args:
+            port: The port number to enable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was enabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter on {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter on" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to enable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to enable vlan filter on port {port}."
+                )
+
+    def vlan_filter_set_off(self, port: int, verify: bool = True):
+        """Set vlan filter off.
+
+        Args:
+            port: The port number to disable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was disabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter off {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter off" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to disable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to disable vlan filter on port {port}."
+                )
+
+    def set_promisc(self, port: int, on: bool, verify: bool = True):
+        """Turns promiscuous mode on/off for the specified port.
+
+        Args:
+            port: Port number to use, should be within 0-32.
+            on: If :data:`True`, turn promisc mode on, otherwise turn off.
+            verify: If :data:`True` an additional command will be sent to verify that promisc mode
+                is properly set. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and promisc mode
+                is not correctly set.
+        """
+        promisc_output = self.send_command(f"set promisc {port} {'on' if on else 'off'}")
+        if verify:
+            stats = self.show_port_info(port_id=port)
+            if on ^ stats.is_promiscuous_mode_enabled:
+                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set promisc mode on port {port}."
+                )
+
     def show_port_stats_all(self) -> list[TestPmdPortStats]:
         """Returns the statistics of all the ports.
 
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
new file mode 100644
index 0000000000..e69de29bb2
-- 
2.44.0

[PATCH v2 3/3] dts: mac filter test suite refactored for new dts

[Nicholas Pratte]

The mac address filter test suite, whose test cases are based on old
DTS's test cases, has been refactored to interface with the new DTS
framework.

In porting over this test suite into the new framework, some
adjustments were made, namely in the EAL and TestPMD parameter provided
before executing the application. While the original test plan was
referenced, by and large, only for the individual test cases, I'll leave
the parameters the original test plan was asking for below for the sake
of discussion:

--burst=1 --rxpt=0 --rxht=0 --rxwt=0 --txpt=36 --txht=0 --txwt=0
--txfreet=32 --rxfreet=64 --mbcache=250 --portmask=0x3

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>

---
v2:
 * Refactored the address pool capacity tests to use all available
   octets in the mac address.
 * Change the payload to 'X' characters instead of 'P' characters.
---
 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_mac_filter.py          | 220 +++++++++++++++++++++
 2 files changed, 222 insertions(+), 1 deletion(-)

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..ad1f3757f7 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "mac_filter"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
index e69de29bb2..c960d6b36c 100644
--- a/dts/tests/TestSuite_mac_filter.py
+++ b/dts/tests/TestSuite_mac_filter.py
@@ -0,0 +1,220 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023-2024 University of New Hampshire
+"""Mac address filtering test suite.
+
+This test suite ensures proper and expected behavior of Allowlist filtering via mac
+addresses on devices bound to the Poll Mode Driver. If a packet received on a device
+contains a mac address not contained with its mac address pool, the packet should
+be dropped. Alternatively, if a packet is received that contains a destination mac
+within the devices address pool, the packet should be accepted and forwarded. This
+behavior should remain consistent across all packets, namely those containing dot1q
+tags or otherwise.
+
+The following test suite assesses behaviors based on the aforementioned logic.
+Additionally, testing is done within the PMD itself to ensure that the mac address
+allow list is behaving as expected.
+"""
+
+from time import sleep
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Dot1Q, Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.exception import InteractiveCommandExecutionError
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+
+class TestMacFilter(TestSuite):
+    """Mac address allowlist filtering test suite.
+
+    Configure mac address filtering on a given port, and test the port's filtering behavior
+    using both a given port's hardware address as well as dummy addresses. If a port accepts
+    a packet that is not contained within its mac address allowlist, then a given test case
+    fails. Alternatively, if a port drops a packet that is designated within its mac address
+    allowlist, a given test case will fail.
+
+    Moreover, a given port should demonstrate proper behavior when bound to the Poll Mode
+    Driver. A port should not have an mac address allowlist that exceeds its designated size.
+    A port's default hardware address should not be removed from its address pool, and invalid
+    addresses should not be included in the allowlist. If a port abides by the above rules, the
+    test case passes.
+    """
+
+    def send_packet_and_verify(
+        self,
+        mac_address: str,
+        add_vlan: bool = False,
+        should_receive: bool = True,
+    ) -> None:
+        """Generate, send, and verify a packet based on specified parameters.
+
+        Test cases within this suite utilize this method to create, send, and verify
+        packets based on criteria relating to the packet's destination mac address,
+        vlan tag, and whether or not the packet should be received or not. Packets
+        are verified using an inserted payload. If the list of received packets
+        contains this payload within any of its packets, the test case passes. Each
+        call with this method sends exactly one packet.
+
+        Args:
+            mac_address: The destination mac address of the packet being sent.
+            add_vlan: Add a vlan tag to the packet being sent. :data:'2' if the packet
+                should be received, :data:'1' if the packet should not be received but
+                requires a vlan tag, and None for any other condition.
+            should_receive: If :data:'True', assert whether or not the sent packet
+                has been received. If :data:'False', assert that the send packet was not
+                received. :data:'True' by default
+        """
+        if add_vlan:
+            packet = Ether() / Dot1Q(vlan=2 if should_receive else 1) / IP() / Raw(load="X" * 22)
+        else:
+            packet = Ether() / IP() / Raw(load="X" * 22)
+        packet.dst = mac_address
+        received_packets = [
+            packets
+            for packets in self.send_packet_and_capture(packet, adjust_addresses=False)
+            if hasattr(packets, "load") and "X" * 22 in str(packets.load)
+        ]
+        if should_receive:
+            self.verify(len(received_packets) == 1, "Expected packet not received")
+        else:
+            self.verify(len(received_packets) == 0, "Expected packet received")
+
+    def test_add_remove_mac_addresses(self) -> None:
+        """Assess basic mac addressing filtering functionalities.
+
+        This test cases validates for proper behavior of mac address filtering with both
+        a port's default, burned-in mac address, as well as additional mac addresses
+        added to the PMD. Packets should either be received or not received depending on
+        the properties applied to the PMD at any given time.
+
+        Test:
+            Start TestPMD with promiscuous mode.
+            Send a packet with the port's default mac address. (Should receive)
+            Send a packet with fake mac address. (Should not receive)
+            Add fake mac address to the PMD's address pool.
+            Send a packet with the fake mac address to the PMD. (Should receive)
+            Remove the fake mac address from the PMD's address pool.
+            Sent a packet with the fake mac address to the PMD. (Should not receive)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.set_promisc(0, on=False)
+        testpmd.start()
+        mac_address = self._sut_port_ingress.mac_address
+
+        # Send a packet with NIC default mac address
+        self.send_packet_and_verify(mac_address=mac_address, should_receive=True)
+        # Send a packet with different mac address
+        fake_address = "00:00:00:00:00:01"
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
+
+        # Add mac address to pool and rerun tests
+        testpmd.set_mac_addr(0, mac_address=fake_address, add=True)
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=True)
+        testpmd.set_mac_addr(0, mac_address=fake_address, add=False)
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
+        testpmd.close()
+        sleep(6)
+
+    def test_invalid_address(self) -> None:
+        """Assess the behavior of a NIC mac address pool while bound to the PMD.
+
+        An assessment of a NIC's behavior when mounted to a PMD as it relates to mac addresses
+        and address pooling. Devices should not be able to use invalid mac addresses, remove their
+        built-in hardware address, or exceed their address pools.
+
+        Test:
+            Start TestPMD.
+            Attempt to add an invalid mac address. (Should fail)
+            Attempt to remove the device's hardware address with no additional addresses in the
+                address pool. (Should fail)
+            Add a fake mac address to the pool twice in succession. (Should not create any errors)
+            Attempt to remove the device's hardware address with other addresses in the address
+                pool. (Should fail)
+            Determine the device's mac address pool size, and fill the pool with fake addresses.
+            Attempt to add another fake mac address, overloading the address pool. (Should fail)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.start()
+        mac_address = self._sut_port_ingress.mac_address
+        try:
+            testpmd.set_mac_addr(0, "00:00:00:00:00:00", add=True)
+            self.verify(False, "Invalid mac address added.")
+        except InteractiveCommandExecutionError:
+            pass
+        try:
+            testpmd.set_mac_addr(0, mac_address, add=False)
+            self.verify(False, "Default mac address removed.")
+        except InteractiveCommandExecutionError:
+            pass
+        # Should be no errors adding this twice
+        testpmd.set_mac_addr(0, "1" + mac_address[1:], add=True)
+        testpmd.set_mac_addr(0, "1" + mac_address[1:], add=True)
+        # Double check to see if default mac address can be removed
+        try:
+            testpmd.set_mac_addr(0, mac_address, add=False)
+            self.verify(False, "Default mac address removed.")
+        except InteractiveCommandExecutionError:
+            pass
+
+        for i in range(testpmd.show_port_info(0).max_mac_addresses_num - 1):
+            # A0 fake address based on the index 'i'.
+            fake_address = str(hex(i)[2:].zfill(12))
+            # Insert ':' characters every two indexes to create a fake mac address.
+            fake_address = ":".join(fake_address[x : x + 2] for x in range(0, len(fake_address), 2))
+            testpmd.set_mac_addr(0, fake_address, add=True, verify=False)
+        try:
+            testpmd.set_mac_addr(0, "F" + mac_address[1:], add=True)
+            self.verify(False, "Mac address limit exceeded.")
+        except InteractiveCommandExecutionError:
+            pass
+        testpmd.close()
+        sleep(6)
+
+    def test_multicast_filter(self) -> None:
+        """Assess basic multicast address filtering functionalities.
+
+        Ensure that multicast filtering performs as intended when a given device is bound
+        to the PMD, with and without dot1q vlan tagging.
+
+        Test:
+            Start TestPMD with promiscuous mode.
+            Add a fake multicast address to the PMD's multicast address pool.
+            Send a packet with the fake multicast address to the PMD. (Should receive)
+            Set vlan filtering on the PMD, and add vlan ID to the PMD.
+            Send a packet with the fake multicast address and vlan ID to the PMD. (Should receive)
+            Send a packet with the fake multicast address and a different vlan ID to the PMD.
+                (Should not receive)
+            Remove the vlan tag from the PMD, and turn vlan filtering off on the PMD.
+            Send a packet with the fake multicast address and no vlan tag to the PMD.
+                (Should receive)
+            Remove the fake multicast address from the PMDs multicast address filter.
+            Send a packet with the fake multicast address to the PMD. (Should not receive)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.start()
+        testpmd.set_promisc(0, on=False)
+        multicast_address = "01:00:5E:00:00:00"
+        vlan_id = 2
+
+        testpmd.set_multicast_mac_addr(0, multi_addr=multicast_address, add=True)
+        self.send_packet_and_verify(multicast_address, should_receive=True)
+
+        # Test vlan filtering on multicast addressing.
+        # Verify vlan functionality for debugging purposes.
+        testpmd.vlan_filter_set_on(port=0)
+        testpmd.rx_vlan_add(vlan_id, 0)
+        self.send_packet_and_verify(multicast_address, should_receive=True, add_vlan=True)
+        self.send_packet_and_verify(multicast_address, should_receive=False, add_vlan=True)
+
+        # Remove vlan tag and filtering and run basic multicast addr test.
+        testpmd.rx_vlan_rm(vlan_id, 0)
+        testpmd.vlan_filter_set_off(port=0)
+        self.send_packet_and_verify(multicast_address, should_receive=True)
+
+        # Remove multicast filter and verify the packet was not received.
+        testpmd.set_multicast_mac_addr(0, multicast_address, add=False)
+        self.send_packet_and_verify(multicast_address, should_receive=False)
+        testpmd.close()
+        sleep(6)
-- 
2.44.0

[PATCH v2 0/3] Mac Filter Port to New DTS

[Nicholas Pratte]

v2:
  - refactored mac filter suite and cleared out hidden bugs.
  - add/remove mac and multicast addresses code semantics
      improvements.
  - minor documentation changes based on suggestions.

Nicholas Pratte (3):
  dts: add boolean to adjust addresses
  dts: add methods for setting mac and multicast addresses
  dts: mac filter test suite refactored for new dts

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++
 dts/framework/test_suite.py                   |   7 +-
 dts/tests/TestSuite_mac_filter.py             | 220 ++++++++++++++++++
 4 files changed, 405 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

-- 
2.44.0

[PATCH v2 1/3] dts: add boolean to adjust addresses

[Nicholas Pratte]

Various test cases in the mac filter test suite called for granular
manipulation of destination mac addresses to properly test mac address
filtering functionality. To compensate, there is now an
adjust_addresses boolean which the user can toggle if they wish to send
their own addressing; the boolean is true by default.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/test_suite.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py
index 694b2eba65..551a587525 100644
--- a/dts/framework/test_suite.py
+++ b/dts/framework/test_suite.py
@@ -185,6 +185,7 @@ def send_packet_and_capture(
         packet: Packet,
         filter_config: PacketFilteringConfig = PacketFilteringConfig(),
         duration: float = 1,
+        adjust_addresses: bool = True,
     ) -> list[Packet]:
         """Send and receive `packet` using the associated TG.
 
@@ -195,11 +196,15 @@ def send_packet_and_capture(
             packet: The packet to send.
             filter_config: The filter to use when capturing packets.
             duration: Capture traffic for this amount of time after sending `packet`.
+            adjust_addresses: If :data:'True', adjust addresses of the egressing packet with
+                a default addressing scheme. If :data:'False', do not adjust the addresses of
+                egressing packet.
 
         Returns:
             A list of received packets.
         """
-        packet = self._adjust_addresses(packet)
+        if adjust_addresses:
+            packet = self._adjust_addresses(packet)
         return self.tg_node.send_packet_and_capture(
             packet,
             self._tg_port_egress,
-- 
2.44.0

[PATCH v2 2/3] dts: add testpmd methods for test suite

[Nicholas Pratte]

Several new methods have been added to TestPMDShell in order to produce
the mac filter's individual test cases:
 - set_mac_addr
 - set_multicast_mac_addr
 - rx_vlan_add
 - rx_vlan_rm
 - vlan_filter_set_on
 - vlan_filter_set_off
 - set_promisc

set_mac_addr and set_multicast_addr were created for the mac filter test
suite, enabling users to both add or remove mac and multicast
addresses based on a booling 'add or remove' parameter. The success or
failure of each call can be verified if a user deems it necessary.

The other methods listed are implemented in other respective test
suites, and their implementations have been copied, but are subject to
change; they are not the focus of this patch.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++++++
 dts/tests/TestSuite_mac_filter.py             |   0
 2 files changed, 177 insertions(+)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index ec22f72221..0be1fb8754 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -767,6 +767,183 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
 
         return TestPmdPort.parse(output)
 
+    def set_mac_addr(self, port_id: int, mac_address: str, add: bool, verify: bool = True):
+        """Add or remove a mac address on a given port's Allowlist.
+
+        Args:
+            port_id: The port ID the mac address is set on.
+            mac_address: The mac address to be added or removed to the specified port.
+            add: If :data:`True`, add the specified mac address. If :data:`False`, remove specified
+                mac address.
+            verify: If :data:'True', assert that the 'mac_addr' operation was successful. If
+                :data:'False', run the command and skip this assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If the set mac address operation fails.
+        """
+        mac_cmd = "add" if add else "remove"
+        output = self.send_command(f"mac_addr {mac_cmd} {port_id} {mac_address}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid argument provided to mac_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if "mac_addr_cmd error:" in output:
+                self._logger.debug(f"Failed to {mac_cmd} {mac_address} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mac_cmd} {mac_address} on port {port_id} \n{output}"
+                )
+
+    def set_multicast_mac_addr(self, port_id: int, multi_addr: str, add: bool, verify: bool = True):
+        """Add or remove multicast mac address to a specified port filter.
+
+        Args:
+            port_id: The port ID the multicast address is set on.
+            multi_addr: The multicast address to be added to the filter.
+            add: If :data:'True', add the specified multicast address to the port filter.
+                If :data:'False', remove the specified multicast address from the port filter.
+            verify: If :data:'True', assert that the 'mcast_addr' operations was successful.
+                If :data:'False', execute the 'mcast_addr' operation and skip the assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fails.
+        """
+        mcast_cmd = "add" if add else "remove"
+        output = self.send_command(f"mcast_addr {mcast_cmd} {port_id} {multi_addr}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid arguments provided to mcast_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if (
+                "Invalid multicast_addr" in output
+                or f'multicast address {"already" if add else "not"} filtered by port' in output
+            ):
+                self._logger.debug(f"Failed to {mcast_cmd} {multi_addr} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mcast_cmd} {multi_addr} on port {port_id} \n{output}"
+                )
+
+    def rx_vlan_add(self, vlan: int, port: int, verify: bool = True):
+        """Add specified vlan tag to the filter list on a port.
+
+        Args:
+            vlan: The vlan tag to add, should be within 1-1005, 1-4094 extended.
+            port: The port number to add the tag on, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was added to the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not added.
+        """
+        vlan_add_output = self.send_command(f"rx_vlan add {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_add_output or "Invalid vlan_id" in vlan_add_output:
+                self._logger.debug(
+                    f"Failed to add vlan tag {vlan} on port {port}: \n{vlan_add_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to add vlan tag {vlan} on port {port}."
+                )
+
+    def rx_vlan_rm(self, vlan: int, port: int, verify: bool = True):
+        """Remove specified vlan tag from filter list on a port.
+
+        Args:
+            vlan: The vlan tag to remove, should be within 1-4094.
+            port: The port number to remove the tag from, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was removed from the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not removed.
+        """
+        vlan_rm_output = self.send_command(f"rx_vlan rm {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_rm_output or "Invalid vlan_id" in vlan_rm_output:
+                self._logger.debug(
+                    f"Failed to remove vlan tag {vlan} on port {port}: \n{vlan_rm_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to remove vlan tag {vlan} on port {port}."
+                )
+
+    def vlan_filter_set_on(self, port: int, verify: bool = True):
+        """Set vlan filter on.
+
+        Args:
+            port: The port number to enable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was enabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter on {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter on" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to enable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to enable vlan filter on port {port}."
+                )
+
+    def vlan_filter_set_off(self, port: int, verify: bool = True):
+        """Set vlan filter off.
+
+        Args:
+            port: The port number to disable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was disabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter off {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter off" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to disable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to disable vlan filter on port {port}."
+                )
+
+    def set_promisc(self, port: int, on: bool, verify: bool = True):
+        """Turns promiscuous mode on/off for the specified port.
+
+        Args:
+            port: Port number to use, should be within 0-32.
+            on: If :data:`True`, turn promisc mode on, otherwise turn off.
+            verify: If :data:`True` an additional command will be sent to verify that promisc mode
+                is properly set. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and promisc mode
+                is not correctly set.
+        """
+        promisc_output = self.send_command(f"set promisc {port} {'on' if on else 'off'}")
+        if verify:
+            stats = self.show_port_info(port_id=port)
+            if on ^ stats.is_promiscuous_mode_enabled:
+                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set promisc mode on port {port}."
+                )
+
     def show_port_stats_all(self) -> list[TestPmdPortStats]:
         """Returns the statistics of all the ports.
 
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
new file mode 100644
index 0000000000..e69de29bb2
-- 
2.44.0

[PATCH v2 3/3] dts: mac filter test suite refactored for new dts

[Nicholas Pratte]

The mac address filter test suite, whose test cases are based on old
DTS's test cases, has been refactored to interface with the new DTS
framework.

In porting over this test suite into the new framework, some
adjustments were made, namely in the EAL and TestPMD parameter provided
before executing the application. While the original test plan was
referenced, by and large, only for the individual test cases, I'll leave
the parameters the original test plan was asking for below for the sake
of discussion:

--burst=1 --rxpt=0 --rxht=0 --rxwt=0 --txpt=36 --txht=0 --txwt=0
--txfreet=32 --rxfreet=64 --mbcache=250 --portmask=0x3

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>

---
v2:
 * Refactored the address pool capacity tests to use all available
   octets in the mac address.
 * Change the payload to 'X' characters instead of 'P' characters.
---
 dts/framework/config/conf_yaml_schema.json |   3 +-
 dts/tests/TestSuite_mac_filter.py          | 220 +++++++++++++++++++++
 2 files changed, 222 insertions(+), 1 deletion(-)

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..ad1f3757f7 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "mac_filter"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
index e69de29bb2..c960d6b36c 100644
--- a/dts/tests/TestSuite_mac_filter.py
+++ b/dts/tests/TestSuite_mac_filter.py
@@ -0,0 +1,220 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023-2024 University of New Hampshire
+"""Mac address filtering test suite.
+
+This test suite ensures proper and expected behavior of Allowlist filtering via mac
+addresses on devices bound to the Poll Mode Driver. If a packet received on a device
+contains a mac address not contained with its mac address pool, the packet should
+be dropped. Alternatively, if a packet is received that contains a destination mac
+within the devices address pool, the packet should be accepted and forwarded. This
+behavior should remain consistent across all packets, namely those containing dot1q
+tags or otherwise.
+
+The following test suite assesses behaviors based on the aforementioned logic.
+Additionally, testing is done within the PMD itself to ensure that the mac address
+allow list is behaving as expected.
+"""
+
+from time import sleep
+
+from scapy.layers.inet import IP  # type: ignore[import-untyped]
+from scapy.layers.l2 import Dot1Q, Ether  # type: ignore[import-untyped]
+from scapy.packet import Raw  # type: ignore[import-untyped]
+
+from framework.exception import InteractiveCommandExecutionError
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+
+
+class TestMacFilter(TestSuite):
+    """Mac address allowlist filtering test suite.
+
+    Configure mac address filtering on a given port, and test the port's filtering behavior
+    using both a given port's hardware address as well as dummy addresses. If a port accepts
+    a packet that is not contained within its mac address allowlist, then a given test case
+    fails. Alternatively, if a port drops a packet that is designated within its mac address
+    allowlist, a given test case will fail.
+
+    Moreover, a given port should demonstrate proper behavior when bound to the Poll Mode
+    Driver. A port should not have an mac address allowlist that exceeds its designated size.
+    A port's default hardware address should not be removed from its address pool, and invalid
+    addresses should not be included in the allowlist. If a port abides by the above rules, the
+    test case passes.
+    """
+
+    def send_packet_and_verify(
+        self,
+        mac_address: str,
+        add_vlan: bool = False,
+        should_receive: bool = True,
+    ) -> None:
+        """Generate, send, and verify a packet based on specified parameters.
+
+        Test cases within this suite utilize this method to create, send, and verify
+        packets based on criteria relating to the packet's destination mac address,
+        vlan tag, and whether or not the packet should be received or not. Packets
+        are verified using an inserted payload. If the list of received packets
+        contains this payload within any of its packets, the test case passes. Each
+        call with this method sends exactly one packet.
+
+        Args:
+            mac_address: The destination mac address of the packet being sent.
+            add_vlan: Add a vlan tag to the packet being sent. :data:'2' if the packet
+                should be received, :data:'1' if the packet should not be received but
+                requires a vlan tag, and None for any other condition.
+            should_receive: If :data:'True', assert whether or not the sent packet
+                has been received. If :data:'False', assert that the send packet was not
+                received. :data:'True' by default
+        """
+        if add_vlan:
+            packet = Ether() / Dot1Q(vlan=2 if should_receive else 1) / IP() / Raw(load="X" * 22)
+        else:
+            packet = Ether() / IP() / Raw(load="X" * 22)
+        packet.dst = mac_address
+        received_packets = [
+            packets
+            for packets in self.send_packet_and_capture(packet, adjust_addresses=False)
+            if hasattr(packets, "load") and "X" * 22 in str(packets.load)
+        ]
+        if should_receive:
+            self.verify(len(received_packets) == 1, "Expected packet not received")
+        else:
+            self.verify(len(received_packets) == 0, "Expected packet received")
+
+    def test_add_remove_mac_addresses(self) -> None:
+        """Assess basic mac addressing filtering functionalities.
+
+        This test cases validates for proper behavior of mac address filtering with both
+        a port's default, burned-in mac address, as well as additional mac addresses
+        added to the PMD. Packets should either be received or not received depending on
+        the properties applied to the PMD at any given time.
+
+        Test:
+            Start TestPMD with promiscuous mode.
+            Send a packet with the port's default mac address. (Should receive)
+            Send a packet with fake mac address. (Should not receive)
+            Add fake mac address to the PMD's address pool.
+            Send a packet with the fake mac address to the PMD. (Should receive)
+            Remove the fake mac address from the PMD's address pool.
+            Sent a packet with the fake mac address to the PMD. (Should not receive)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.set_promisc(0, on=False)
+        testpmd.start()
+        mac_address = self._sut_port_ingress.mac_address
+
+        # Send a packet with NIC default mac address
+        self.send_packet_and_verify(mac_address=mac_address, should_receive=True)
+        # Send a packet with different mac address
+        fake_address = "00:00:00:00:00:01"
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
+
+        # Add mac address to pool and rerun tests
+        testpmd.set_mac_addr(0, mac_address=fake_address, add=True)
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=True)
+        testpmd.set_mac_addr(0, mac_address=fake_address, add=False)
+        self.send_packet_and_verify(mac_address=fake_address, should_receive=False)
+        testpmd.close()
+        sleep(6)
+
+    def test_invalid_address(self) -> None:
+        """Assess the behavior of a NIC mac address pool while bound to the PMD.
+
+        An assessment of a NIC's behavior when mounted to a PMD as it relates to mac addresses
+        and address pooling. Devices should not be able to use invalid mac addresses, remove their
+        built-in hardware address, or exceed their address pools.
+
+        Test:
+            Start TestPMD.
+            Attempt to add an invalid mac address. (Should fail)
+            Attempt to remove the device's hardware address with no additional addresses in the
+                address pool. (Should fail)
+            Add a fake mac address to the pool twice in succession. (Should not create any errors)
+            Attempt to remove the device's hardware address with other addresses in the address
+                pool. (Should fail)
+            Determine the device's mac address pool size, and fill the pool with fake addresses.
+            Attempt to add another fake mac address, overloading the address pool. (Should fail)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.start()
+        mac_address = self._sut_port_ingress.mac_address
+        try:
+            testpmd.set_mac_addr(0, "00:00:00:00:00:00", add=True)
+            self.verify(False, "Invalid mac address added.")
+        except InteractiveCommandExecutionError:
+            pass
+        try:
+            testpmd.set_mac_addr(0, mac_address, add=False)
+            self.verify(False, "Default mac address removed.")
+        except InteractiveCommandExecutionError:
+            pass
+        # Should be no errors adding this twice
+        testpmd.set_mac_addr(0, "1" + mac_address[1:], add=True)
+        testpmd.set_mac_addr(0, "1" + mac_address[1:], add=True)
+        # Double check to see if default mac address can be removed
+        try:
+            testpmd.set_mac_addr(0, mac_address, add=False)
+            self.verify(False, "Default mac address removed.")
+        except InteractiveCommandExecutionError:
+            pass
+
+        for i in range(testpmd.show_port_info(0).max_mac_addresses_num - 1):
+            # A0 fake address based on the index 'i'.
+            fake_address = str(hex(i)[2:].zfill(12))
+            # Insert ':' characters every two indexes to create a fake mac address.
+            fake_address = ":".join(fake_address[x : x + 2] for x in range(0, len(fake_address), 2))
+            testpmd.set_mac_addr(0, fake_address, add=True, verify=False)
+        try:
+            testpmd.set_mac_addr(0, "F" + mac_address[1:], add=True)
+            self.verify(False, "Mac address limit exceeded.")
+        except InteractiveCommandExecutionError:
+            pass
+        testpmd.close()
+        sleep(6)
+
+    def test_multicast_filter(self) -> None:
+        """Assess basic multicast address filtering functionalities.
+
+        Ensure that multicast filtering performs as intended when a given device is bound
+        to the PMD, with and without dot1q vlan tagging.
+
+        Test:
+            Start TestPMD with promiscuous mode.
+            Add a fake multicast address to the PMD's multicast address pool.
+            Send a packet with the fake multicast address to the PMD. (Should receive)
+            Set vlan filtering on the PMD, and add vlan ID to the PMD.
+            Send a packet with the fake multicast address and vlan ID to the PMD. (Should receive)
+            Send a packet with the fake multicast address and a different vlan ID to the PMD.
+                (Should not receive)
+            Remove the vlan tag from the PMD, and turn vlan filtering off on the PMD.
+            Send a packet with the fake multicast address and no vlan tag to the PMD.
+                (Should receive)
+            Remove the fake multicast address from the PMDs multicast address filter.
+            Send a packet with the fake multicast address to the PMD. (Should not receive)
+        """
+        testpmd = TestPmdShell(self.sut_node)
+        testpmd.start()
+        testpmd.set_promisc(0, on=False)
+        multicast_address = "01:00:5E:00:00:00"
+        vlan_id = 2
+
+        testpmd.set_multicast_mac_addr(0, multi_addr=multicast_address, add=True)
+        self.send_packet_and_verify(multicast_address, should_receive=True)
+
+        # Test vlan filtering on multicast addressing.
+        # Verify vlan functionality for debugging purposes.
+        testpmd.vlan_filter_set_on(port=0)
+        testpmd.rx_vlan_add(vlan_id, 0)
+        self.send_packet_and_verify(multicast_address, should_receive=True, add_vlan=True)
+        self.send_packet_and_verify(multicast_address, should_receive=False, add_vlan=True)
+
+        # Remove vlan tag and filtering and run basic multicast addr test.
+        testpmd.rx_vlan_rm(vlan_id, 0)
+        testpmd.vlan_filter_set_off(port=0)
+        self.send_packet_and_verify(multicast_address, should_receive=True)
+
+        # Remove multicast filter and verify the packet was not received.
+        testpmd.set_multicast_mac_addr(0, multicast_address, add=False)
+        self.send_packet_and_verify(multicast_address, should_receive=False)
+        testpmd.close()
+        sleep(6)
-- 
2.44.0

[PATCH v2 0/3] Mac Filter Port to New DTS

[Nicholas Pratte]

This series incorporates the test suite itself, in addition to framwork
changes needed to complete the suite's stipulated test cases. The test
suite consists of three test cases: one testing basically mac filtering
functionality, the other testing the PMD's mac address pool behavior
within a TestPMD shell, and the third testing filtering functionality
of multicast mac addresses.

The port from old DTS has removed some potentially important details
regarding EAL and TestPMD parameters, and these can be found within the
test suites corresponding commit messages. Moreover, unlike old DTS,
instead of validating packet behavior using TestPMD, this test suite
validates behavior by sending, and analysing,a list of received packets.

Nicholas Pratte (3):
  dts: add boolean to adjust addresses
  dts: add methods for setting mac and multicast addresses
  dts: mac filter test suite refactored for new dts

 dts/framework/config/conf_yaml_schema.json    |   3 +-
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++
 dts/framework/test_suite.py                   |   7 +-
 dts/tests/TestSuite_mac_filter.py             | 220 ++++++++++++++++++
 4 files changed, 405 insertions(+), 2 deletions(-)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

-- 
2.44.0

[PATCH v2 2/3] dts: add testpmd methods for test suite

[Nicholas Pratte]

Several new methods have been added to TestPMDShell in order to produce
the mac filter's individual test cases:
 - set_mac_addr
 - set_multicast_mac_addr
 - rx_vlan_add
 - rx_vlan_rm
 - vlan_filter_set_on
 - vlan_filter_set_off
 - set_promisc

set_mac_addr and set_multicast_addr were created for the mac filter test
suite, enabling users to both add or remove mac and multicast
addresses based on a booling 'add or remove' parameter. The success or
failure of each call can be verified if a user deems it necessary.

The other methods listed are implemented in other respective test
suites, and their implementations have been copied, but are subject to
change; they are not the focus of this patch.

Bugzilla ID: 1454
Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
---
 dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++++++
 dts/tests/TestSuite_mac_filter.py             |   0
 2 files changed, 177 insertions(+)
 create mode 100644 dts/tests/TestSuite_mac_filter.py

diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index ec22f72221..0be1fb8754 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -767,6 +767,183 @@ def show_port_info(self, port_id: int) -> TestPmdPort:
 
         return TestPmdPort.parse(output)
 
+    def set_mac_addr(self, port_id: int, mac_address: str, add: bool, verify: bool = True):
+        """Add or remove a mac address on a given port's Allowlist.
+
+        Args:
+            port_id: The port ID the mac address is set on.
+            mac_address: The mac address to be added or removed to the specified port.
+            add: If :data:`True`, add the specified mac address. If :data:`False`, remove specified
+                mac address.
+            verify: If :data:'True', assert that the 'mac_addr' operation was successful. If
+                :data:'False', run the command and skip this assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If the set mac address operation fails.
+        """
+        mac_cmd = "add" if add else "remove"
+        output = self.send_command(f"mac_addr {mac_cmd} {port_id} {mac_address}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid argument provided to mac_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if "mac_addr_cmd error:" in output:
+                self._logger.debug(f"Failed to {mac_cmd} {mac_address} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mac_cmd} {mac_address} on port {port_id} \n{output}"
+                )
+
+    def set_multicast_mac_addr(self, port_id: int, multi_addr: str, add: bool, verify: bool = True):
+        """Add or remove multicast mac address to a specified port filter.
+
+        Args:
+            port_id: The port ID the multicast address is set on.
+            multi_addr: The multicast address to be added to the filter.
+            add: If :data:'True', add the specified multicast address to the port filter.
+                If :data:'False', remove the specified multicast address from the port filter.
+            verify: If :data:'True', assert that the 'mcast_addr' operations was successful.
+                If :data:'False', execute the 'mcast_addr' operation and skip the assertion.
+
+        Raises:
+            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fails.
+        """
+        mcast_cmd = "add" if add else "remove"
+        output = self.send_command(f"mcast_addr {mcast_cmd} {port_id} {multi_addr}")
+        if "Bad arguments" in output:
+            self._logger.debug("Invalid arguments provided to mcast_addr")
+            raise InteractiveCommandExecutionError("Invalid argument provided")
+
+        if verify:
+            if (
+                "Invalid multicast_addr" in output
+                or f'multicast address {"already" if add else "not"} filtered by port' in output
+            ):
+                self._logger.debug(f"Failed to {mcast_cmd} {multi_addr} on port {port_id}")
+                raise InteractiveCommandExecutionError(
+                    f"Failed to {mcast_cmd} {multi_addr} on port {port_id} \n{output}"
+                )
+
+    def rx_vlan_add(self, vlan: int, port: int, verify: bool = True):
+        """Add specified vlan tag to the filter list on a port.
+
+        Args:
+            vlan: The vlan tag to add, should be within 1-1005, 1-4094 extended.
+            port: The port number to add the tag on, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was added to the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not added.
+        """
+        vlan_add_output = self.send_command(f"rx_vlan add {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_add_output or "Invalid vlan_id" in vlan_add_output:
+                self._logger.debug(
+                    f"Failed to add vlan tag {vlan} on port {port}: \n{vlan_add_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to add vlan tag {vlan} on port {port}."
+                )
+
+    def rx_vlan_rm(self, vlan: int, port: int, verify: bool = True):
+        """Remove specified vlan tag from filter list on a port.
+
+        Args:
+            vlan: The vlan tag to remove, should be within 1-4094.
+            port: The port number to remove the tag from, should be within 0-32.
+            verify: If :data:`True`, the output of the command is scanned to verify that
+                the vlan tag was removed from the filter list on the specified port. If not, it is
+                considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the tag
+                is not removed.
+        """
+        vlan_rm_output = self.send_command(f"rx_vlan rm {vlan} {port}")
+        if verify:
+            if "VLAN-filtering disabled" in vlan_rm_output or "Invalid vlan_id" in vlan_rm_output:
+                self._logger.debug(
+                    f"Failed to remove vlan tag {vlan} on port {port}: \n{vlan_rm_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to remove vlan tag {vlan} on port {port}."
+                )
+
+    def vlan_filter_set_on(self, port: int, verify: bool = True):
+        """Set vlan filter on.
+
+        Args:
+            port: The port number to enable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was enabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter on {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter on" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to enable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to enable vlan filter on port {port}."
+                )
+
+    def vlan_filter_set_off(self, port: int, verify: bool = True):
+        """Set vlan filter off.
+
+        Args:
+            port: The port number to disable VLAN filter on, should be within 0-32.
+            verify: If :data:`True`, the output of the command and show port info
+                is scanned to verify that vlan filtering was disabled successfully.
+                If not, it is considered an error.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and the filter
+                fails to update.
+        """
+        filter_cmd_output = self.send_command(f"vlan set filter off {port}")
+        if verify:
+            if "Invalid port" in filter_cmd_output or "filter off" not in self.send_command(
+                f"show port info {port}"
+            ):
+                self._logger.debug(
+                    f"Failed to disable vlan filter on port {port}: \n{filter_cmd_output}"
+                )
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to disable vlan filter on port {port}."
+                )
+
+    def set_promisc(self, port: int, on: bool, verify: bool = True):
+        """Turns promiscuous mode on/off for the specified port.
+
+        Args:
+            port: Port number to use, should be within 0-32.
+            on: If :data:`True`, turn promisc mode on, otherwise turn off.
+            verify: If :data:`True` an additional command will be sent to verify that promisc mode
+                is properly set. Defaults to :data:`True`.
+
+        Raises:
+            InteractiveCommandExecutionError: If `verify` is :data:`True` and promisc mode
+                is not correctly set.
+        """
+        promisc_output = self.send_command(f"set promisc {port} {'on' if on else 'off'}")
+        if verify:
+            stats = self.show_port_info(port_id=port)
+            if on ^ stats.is_promiscuous_mode_enabled:
+                self._logger.debug(f"Failed to set promisc mode on port {port}: \n{promisc_output}")
+                raise InteractiveCommandExecutionError(
+                    f"Testpmd failed to set promisc mode on port {port}."
+                )
+
     def show_port_stats_all(self) -> list[TestPmdPortStats]:
         """Returns the statistics of all the ports.
 
diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
new file mode 100644
index 0000000000..e69de29bb2
-- 
2.44.0

Re: [PATCH v2 2/3] dts: add testpmd methods for test suite

[Jeremy Spewock]

There were a few emails that came through for this series but this was
the most recent one so I went with reviewing this one, but it looks
like the more descriptive commit subject got lost in this iteration.
It would probably be worth adding that back.

Additionally, it looks like the functions you added here (including
the ones added from the VLAN suite, but I believe those ones are
already fixed in the latest version ) are all missing their return
types. Of course they don't return anything, but that is still useful
to note by annotating it as "None".

On Tue, Jul 2, 2024 at 3:25 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
>
> Several new methods have been added to TestPMDShell in order to produce
> the mac filter's individual test cases:
>  - set_mac_addr
>  - set_multicast_mac_addr
>  - rx_vlan_add
>  - rx_vlan_rm
>  - vlan_filter_set_on
>  - vlan_filter_set_off
>  - set_promisc
>
> set_mac_addr and set_multicast_addr were created for the mac filter test
> suite, enabling users to both add or remove mac and multicast
> addresses based on a booling 'add or remove' parameter. The success or

I think this is a typo and "booling" should be "boolean" but I could be wrong.

> failure of each call can be verified if a user deems it necessary.
>
> The other methods listed are implemented in other respective test
> suites, and their implementations have been copied, but are subject to
> change; they are not the focus of this patch.
>
> Bugzilla ID: 1454
> Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
> ---
>  dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++++++
>  dts/tests/TestSuite_mac_filter.py             |   0

It looks like creating the file somehow snuck into the diff for this
commit instead of the other one that populates it.

>  2 files changed, 177 insertions(+)
>  create mode 100644 dts/tests/TestSuite_mac_filter.py
>
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index ec22f72221..0be1fb8754 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
<snip>
> +    def set_multicast_mac_addr(self, port_id: int, multi_addr: str, add: bool, verify: bool = True):
> +        """Add or remove multicast mac address to a specified port filter.

Just to make this more clear that you specify the port and not the
port filter, it might be helpful here to show that the port possesses
the filter by saying "a specified port's filter."

> +
> +        Args:
> +            port_id: The port ID the multicast address is set on.
> +            multi_addr: The multicast address to be added to the filter.
> +            add: If :data:'True', add the specified multicast address to the port filter.
> +                If :data:'False', remove the specified multicast address from the port filter.
> +            verify: If :data:'True', assert that the 'mcast_addr' operations was successful.
> +                If :data:'False', execute the 'mcast_addr' operation and skip the assertion.
> +
> +        Raises:
> +            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fails.
> +        """
> +        mcast_cmd = "add" if add else "remove"
> +        output = self.send_command(f"mcast_addr {mcast_cmd} {port_id} {multi_addr}")
> +        if "Bad arguments" in output:
> +            self._logger.debug("Invalid arguments provided to mcast_addr")
> +            raise InteractiveCommandExecutionError("Invalid argument provided")
> +
> +        if verify:
> +            if (
> +                "Invalid multicast_addr" in output
> +                or f'multicast address {"already" if add else "not"} filtered by port' in output
> +            ):
> +                self._logger.debug(f"Failed to {mcast_cmd} {multi_addr} on port {port_id}")
> +                raise InteractiveCommandExecutionError(
> +                    f"Failed to {mcast_cmd} {multi_addr} on port {port_id} \n{output}"
> +                )
<snip>

>      def show_port_stats_all(self) -> list[TestPmdPortStats]:
>          """Returns the statistics of all the ports.
>
> diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
> new file mode 100644
> index 0000000000..e69de29bb2
> --
> 2.44.0
>

Re: [PATCH v2 2/3] dts: add testpmd methods for test suite

[Nicholas Pratte]

On Thu, Jul 11, 2024 at 3:33 PM Jeremy Spewock <jspewock@iol.unh.edu> wrote:
>
> There were a few emails that came through for this series but this was
> the most recent one so I went with reviewing this one, but it looks
> like the more descriptive commit subject got lost in this iteration.
> It would probably be worth adding that back.
>
> Additionally, it looks like the functions you added here (including
> the ones added from the VLAN suite, but I believe those ones are
> already fixed in the latest version ) are all missing their return
> types. Of course they don't return anything, but that is still useful
> to note by annotating it as "None".

Surprised I never saw that. I'll fix this.


> On Tue, Jul 2, 2024 at 3:25 PM Nicholas Pratte <npratte@iol.unh.edu> wrote:
> >
> > Several new methods have been added to TestPMDShell in order to produce
> > the mac filter's individual test cases:
> >  - set_mac_addr
> >  - set_multicast_mac_addr
> >  - rx_vlan_add
> >  - rx_vlan_rm
> >  - vlan_filter_set_on
> >  - vlan_filter_set_off
> >  - set_promisc
> >
> > set_mac_addr and set_multicast_addr were created for the mac filter test
> > suite, enabling users to both add or remove mac and multicast
> > addresses based on a booling 'add or remove' parameter. The success or
>
> I think this is a typo and "booling" should be "boolean" but I could be wrong.

Good catch!

>
> > failure of each call can be verified if a user deems it necessary.
> >
> > The other methods listed are implemented in other respective test
> > suites, and their implementations have been copied, but are subject to
> > change; they are not the focus of this patch.
> >
> > Bugzilla ID: 1454
> > Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
> > ---
> >  dts/framework/remote_session/testpmd_shell.py | 177 ++++++++++++++++++
> >  dts/tests/TestSuite_mac_filter.py             |   0
>
> It looks like creating the file somehow snuck into the diff for this
> commit instead of the other one that populates it.

Strange. I can fix that real quick.

>
> >  2 files changed, 177 insertions(+)
> >  create mode 100644 dts/tests/TestSuite_mac_filter.py
> >
> > diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> > index ec22f72221..0be1fb8754 100644
> > --- a/dts/framework/remote_session/testpmd_shell.py
> > +++ b/dts/framework/remote_session/testpmd_shell.py
> <snip>
> > +    def set_multicast_mac_addr(self, port_id: int, multi_addr: str, add: bool, verify: bool = True):
> > +        """Add or remove multicast mac address to a specified port filter.
>
> Just to make this more clear that you specify the port and not the
> port filter, it might be helpful here to show that the port possesses
> the filter by saying "a specified port's filter."

Yes. This was just a typo. What you have here is what I was going for.

>
> > +
> > +        Args:
> > +            port_id: The port ID the multicast address is set on.
> > +            multi_addr: The multicast address to be added to the filter.
> > +            add: If :data:'True', add the specified multicast address to the port filter.
> > +                If :data:'False', remove the specified multicast address from the port filter.
> > +            verify: If :data:'True', assert that the 'mcast_addr' operations was successful.
> > +                If :data:'False', execute the 'mcast_addr' operation and skip the assertion.
> > +
> > +        Raises:
> > +            InteractiveCommandExecutionError: If either the 'add' or 'remove' operations fails.
> > +        """
> > +        mcast_cmd = "add" if add else "remove"
> > +        output = self.send_command(f"mcast_addr {mcast_cmd} {port_id} {multi_addr}")
> > +        if "Bad arguments" in output:
> > +            self._logger.debug("Invalid arguments provided to mcast_addr")
> > +            raise InteractiveCommandExecutionError("Invalid argument provided")
> > +
> > +        if verify:
> > +            if (
> > +                "Invalid multicast_addr" in output
> > +                or f'multicast address {"already" if add else "not"} filtered by port' in output
> > +            ):
> > +                self._logger.debug(f"Failed to {mcast_cmd} {multi_addr} on port {port_id}")
> > +                raise InteractiveCommandExecutionError(
> > +                    f"Failed to {mcast_cmd} {multi_addr} on port {port_id} \n{output}"
> > +                )
> <snip>
>
> >      def show_port_stats_all(self) -> list[TestPmdPortStats]:
> >          """Returns the statistics of all the ports.
> >
> > diff --git a/dts/tests/TestSuite_mac_filter.py b/dts/tests/TestSuite_mac_filter.py
> > new file mode 100644
> > index 0000000000..e69de29bb2
> > --
> > 2.44.0
> >