[PATCH 0/2] dts: add blocklist test suite

[PATCH 0/2] dts: add blocklist test suite

[Luca Vizzarro]

Hello,

sending in an re-implementation of the blocklist test suite.
This new version slightly differs in the way it verifies if a
NIC was blocked or not, but should be equivalent to the original from
old DTS.

Best,

Luca

Luca Vizzarro (2):
  dts: add blocked ports to EalParams
  dts: add blocklist test suite

 dts/framework/config/conf_yaml_schema.json    |  3 +-
 dts/framework/params/eal.py                   |  6 +-
 dts/framework/params/types.py                 |  3 +-
 dts/framework/remote_session/dpdk_shell.py    |  4 +-
 dts/framework/remote_session/testpmd_shell.py |  2 +-
 dts/tests/TestSuite_blocklist.py              | 68 +++++++++++++++++++
 6 files changed, 80 insertions(+), 6 deletions(-)
 create mode 100644 dts/tests/TestSuite_blocklist.py

-- 
2.34.1

[PATCH 1/2] dts: add blocked ports to EalParams

[Luca Vizzarro]

Make the "ports" attribute of EalParams as it actually is to
"allowed_ports", and add "blocked_ports".

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/params/eal.py                   | 6 +++++-
 dts/framework/params/types.py                 | 3 ++-
 dts/framework/remote_session/dpdk_shell.py    | 4 ++--
 dts/framework/remote_session/testpmd_shell.py | 2 +-
 4 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/dts/framework/params/eal.py b/dts/framework/params/eal.py
index 8d7766fefc..b893d874a7 100644
--- a/dts/framework/params/eal.py
+++ b/dts/framework/params/eal.py
@@ -42,9 +42,13 @@ class EalParams(Params):
     vdevs: list[VirtualDevice] | None = field(
         default=None, metadata=Params.multiple() | Params.long("vdev")
     )
-    ports: list[Port] | None = field(
+    allowed_ports: list[Port] | None = field(
         default=None,
         metadata=Params.convert_value(_port_to_pci) | Params.multiple() | Params.short("a"),
     )
+    blocked_ports: list[Port] | None = field(
+        default=None,
+        metadata=Params.convert_value(_port_to_pci) | Params.multiple() | Params.short("b"),
+    )
     other_eal_param: Params | None = None
     _separator: Literal[True] = field(default=True, init=False, metadata=Params.short("-"))
diff --git a/dts/framework/params/types.py b/dts/framework/params/types.py
index e668f658d8..0a4179fdd0 100644
--- a/dts/framework/params/types.py
+++ b/dts/framework/params/types.py
@@ -49,7 +49,8 @@ class EalParamsDict(TypedDict, total=False):
     prefix: str
     no_pci: Switch
     vdevs: list[VirtualDevice] | None
-    ports: list[Port] | None
+    allowed_ports: list[Port] | None
+    blocked_ports: list[Port] | None
     other_eal_param: Params | None
 
 
diff --git a/dts/framework/remote_session/dpdk_shell.py b/dts/framework/remote_session/dpdk_shell.py
index 296639f37d..6b4cec2466 100644
--- a/dts/framework/remote_session/dpdk_shell.py
+++ b/dts/framework/remote_session/dpdk_shell.py
@@ -54,8 +54,8 @@ def compute_eal_params(
         sut_node.dpdk_prefix_list.append(prefix)
     params.prefix = prefix
 
-    if params.ports is None:
-        params.ports = sut_node.ports
+    if params.allowed_ports is None:
+        params.allowed_ports = sut_node.ports
 
     return params
 
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index ec22f72221..baa773ed70 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -638,7 +638,7 @@ def start(self, verify: bool = True) -> None:
                 self._logger.debug(f"Failed to start packet forwarding: \n{start_cmd_output}")
                 raise InteractiveCommandExecutionError("Testpmd failed to start packet forwarding.")
 
-            number_of_ports = len(self._app_params.ports or [])
+            number_of_ports = len(self._app_params.allowed_ports or [])
             for port_id in range(number_of_ports):
                 if not self.wait_link_status_up(port_id):
                     raise InteractiveCommandExecutionError(
-- 
2.34.1

Re: [PATCH 1/2] dts: add blocked ports to EalParams

[Dean Marx]

[-- Attachment #1: Type: text/plain, Size: 357 bytes --]

On Tue, Jun 25, 2024 at 8:38 AM Luca Vizzarro <luca.vizzarro@arm.com> wrote:

> Make the "ports" attribute of EalParams as it actually is to
> "allowed_ports", and add "blocked_ports".
>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
>

Reviewed-by: Dean Marx <dmarx@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 852 bytes --]

Re: [PATCH 1/2] dts: add blocked ports to EalParams

[Patrick Robb]

Reviewed-by: Patrick Robb <probb@iol.unh.edu>

[PATCH 2/2] dts: add blocklist test suite

[Luca Vizzarro]

This test suite tests the port blocklisting functionality built in
testpmd.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/config/conf_yaml_schema.json |  3 +-
 dts/tests/TestSuite_blocklist.py           | 68 ++++++++++++++++++++++
 2 files changed, 70 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_blocklist.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index f02a310bb5..e9414ebff1 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -187,7 +187,8 @@
       "enum": [
         "hello_world",
         "os_udp",
-        "pmd_buffer_scatter"
+        "pmd_buffer_scatter",
+        "blocklist"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_blocklist.py b/dts/tests/TestSuite_blocklist.py
new file mode 100644
index 0000000000..7d7f323e36
--- /dev/null
+++ b/dts/tests/TestSuite_blocklist.py
@@ -0,0 +1,68 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""The DPDK device blocklisting test suite.
+
+This testing suite ensures tests the port blocklisting functionality of testpmd.
+"""
+
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite
+from framework.testbed_model.port import Port
+
+
+class TestBlocklist(TestSuite):
+    """DPDK device blocklisting test suite.
+
+    For this test suite to work at least 2 ports need to be configured for the SUT node.
+    """
+
+    def set_up_suite(self) -> None:
+        """Verify setup."""
+        self.verify(len(self.sut_node.ports) >= 2, "At least two ports are required for this test")
+
+    def verify_blocklisted_ports(self, ports_to_block: list[Port]):
+        """Runs testpmd with the given ports blocklisted and verifies the ports."""
+        testpmd = TestPmdShell(self.sut_node, allowed_ports=[], blocked_ports=ports_to_block)
+
+        allowlisted_ports = {port.device_name for port in testpmd.show_port_info_all()}
+        blocklisted_ports = {port.pci for port in ports_to_block}
+
+        # sanity check
+        allowed_len = len(allowlisted_ports - blocklisted_ports)
+        self.verify(allowed_len > 0, "At least one port should have been allowed")
+
+        blocked = not allowlisted_ports & blocklisted_ports
+        self.verify(blocked, "At least one port was not blocklisted")
+
+        testpmd.close()
+
+    def test_bl_no_blocklisted(self):
+        """Run testpmd with no blocklisted device.
+
+        Steps:
+            Run testpmd without specifying allowed or blocked ports.
+        Verify:
+            That no ports were blocked.
+        """
+        self.verify_blocklisted_ports([])
+
+    def test_bl_one_port_blocklisted(self):
+        """Run testpmd with one blocklisted port.
+
+        Steps:
+            Run testpmd with one only one blocklisted port and allowing all the other ones.
+        Verify:
+            That the port was successfully blocklisted.
+        """
+        self.verify_blocklisted_ports(self.sut_node.ports[:1])
+
+    def test_bl_all_but_one_port_blocklisted(self):
+        """Run testpmd with all but one blocklisted port.
+
+        Steps:
+            Run testpmd with only one allowed port, blocking all the other ones.
+        Verify:
+            That all specified ports were successfully blocklisted.
+        """
+        self.verify_blocklisted_ports(self.sut_node.ports[:-1])
-- 
2.34.1

Re: [PATCH 2/2] dts: add blocklist test suite

[Dean Marx]

[-- Attachment #1: Type: text/plain, Size: 330 bytes --]

On Tue, Jun 25, 2024 at 8:38 AM Luca Vizzarro <luca.vizzarro@arm.com> wrote:

> This test suite tests the port blocklisting functionality built in
> testpmd.
>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
>

Reviewed-by: Dean Marx <dmarx@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 795 bytes --]

Re: [PATCH 2/2] dts: add blocklist test suite

[Patrick Robb]

Barring a couple updates needed due to framework changes, this looks
good and runs on a Broadcom 975908 25G NIC I used for testing.

It runs fine after switching to invoking the testpmd shell via the
context manager, and removing the use of "testpmd.close()." Once a new
version is submitted with these changes, in my opinion it makes sense
to move this series to next-dts. Thanks!

Reviewed-by: Patrick Robb <probb@iol.unh.edu>
Tested-by: Patrick Robb <probb@iol.unh.edu>

[PATCH v2 0/2] dts: add blocklist test suite

[Luca Vizzarro]

Hi there,

sending v2.

v2:
- rebased

Best,
Luca

Luca Vizzarro (2):
  dts: add blocked ports to EalParams
  dts: add blocklist test suite

 dts/framework/config/conf_yaml_schema.json    |  3 +-
 dts/framework/params/eal.py                   |  6 +-
 dts/framework/params/types.py                 |  3 +-
 dts/framework/remote_session/dpdk_shell.py    |  4 +-
 dts/framework/remote_session/testpmd_shell.py |  2 +-
 dts/tests/TestSuite_blocklist.py              | 63 +++++++++++++++++++
 6 files changed, 75 insertions(+), 6 deletions(-)
 create mode 100644 dts/tests/TestSuite_blocklist.py

-- 
2.43.0

[PATCH v2 1/2] dts: add blocked ports to EalParams

[Luca Vizzarro]

Make the "ports" attribute of EalParams as it actually is to
"allowed_ports", and add "blocked_ports".

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/params/eal.py                   | 6 +++++-
 dts/framework/params/types.py                 | 3 ++-
 dts/framework/remote_session/dpdk_shell.py    | 4 ++--
 dts/framework/remote_session/testpmd_shell.py | 2 +-
 4 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/dts/framework/params/eal.py b/dts/framework/params/eal.py
index cf1594353a..71bc781eab 100644
--- a/dts/framework/params/eal.py
+++ b/dts/framework/params/eal.py
@@ -45,9 +45,13 @@ class EalParams(Params):
     vdevs: list[VirtualDevice] | None = field(
         default=None, metadata=Params.multiple() | Params.long("vdev")
     )
-    ports: list[Port] | None = field(
+    allowed_ports: list[Port] | None = field(
         default=None,
         metadata=Params.convert_value(_port_to_pci) | Params.multiple() | Params.short("a"),
     )
+    blocked_ports: list[Port] | None = field(
+        default=None,
+        metadata=Params.convert_value(_port_to_pci) | Params.multiple() | Params.short("b"),
+    )
     other_eal_param: Params | None = None
     _separator: Literal[True] = field(default=True, init=False, metadata=Params.short("-"))
diff --git a/dts/framework/params/types.py b/dts/framework/params/types.py
index d77c4625fb..87d11502e8 100644
--- a/dts/framework/params/types.py
+++ b/dts/framework/params/types.py
@@ -50,7 +50,8 @@ class EalParamsDict(TypedDict, total=False):
     prefix: str
     no_pci: Switch
     vdevs: list[VirtualDevice] | None
-    ports: list[Port] | None
+    allowed_ports: list[Port] | None
+    blocked_ports: list[Port] | None
     other_eal_param: Params | None
 
 
diff --git a/dts/framework/remote_session/dpdk_shell.py b/dts/framework/remote_session/dpdk_shell.py
index b39132cc42..82fa4755f0 100644
--- a/dts/framework/remote_session/dpdk_shell.py
+++ b/dts/framework/remote_session/dpdk_shell.py
@@ -56,8 +56,8 @@ def compute_eal_params(
         sut_node.dpdk_prefix_list.append(prefix)
     params.prefix = prefix
 
-    if params.ports is None:
-        params.ports = sut_node.ports
+    if params.allowed_ports is None:
+        params.allowed_ports = sut_node.ports
 
     return params
 
diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
index 8a45a5231b..221465f6fb 100644
--- a/dts/framework/remote_session/testpmd_shell.py
+++ b/dts/framework/remote_session/testpmd_shell.py
@@ -1465,7 +1465,7 @@ def start(self, verify: bool = True) -> None:
                 self._logger.debug(f"Failed to start packet forwarding: \n{start_cmd_output}")
                 raise InteractiveCommandExecutionError("Testpmd failed to start packet forwarding.")
 
-            number_of_ports = len(self._app_params.ports or [])
+            number_of_ports = len(self._app_params.allowed_ports or [])
             for port_id in range(number_of_ports):
                 if not self.wait_link_status_up(port_id):
                     raise InteractiveCommandExecutionError(
-- 
2.43.0

Re: [PATCH v2 1/2] dts: add blocked ports to EalParams

[Nicholas Pratte]

Hi Luca, this looks good to me! The only thing that needs to be fixed
is the attributes component of the docstring for EalParams in eal.py.
Aside from that:

Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>

On Thu, Nov 7, 2024 at 9:00 AM Luca Vizzarro <luca.vizzarro@arm.com> wrote:
>
> Make the "ports" attribute of EalParams as it actually is to
> "allowed_ports", and add "blocked_ports".
>
> Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
> Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
> ---
>  dts/framework/params/eal.py                   | 6 +++++-
>  dts/framework/params/types.py                 | 3 ++-
>  dts/framework/remote_session/dpdk_shell.py    | 4 ++--
>  dts/framework/remote_session/testpmd_shell.py | 2 +-
>  4 files changed, 10 insertions(+), 5 deletions(-)
>
> diff --git a/dts/framework/params/eal.py b/dts/framework/params/eal.py
> index cf1594353a..71bc781eab 100644
> --- a/dts/framework/params/eal.py
> +++ b/dts/framework/params/eal.py
> @@ -45,9 +45,13 @@ class EalParams(Params):
>      vdevs: list[VirtualDevice] | None = field(
>          default=None, metadata=Params.multiple() | Params.long("vdev")
>      )
> -    ports: list[Port] | None = field(
> +    allowed_ports: list[Port] | None = field(
>          default=None,
>          metadata=Params.convert_value(_port_to_pci) | Params.multiple() | Params.short("a"),
>      )
> +    blocked_ports: list[Port] | None = field(
> +        default=None,
> +        metadata=Params.convert_value(_port_to_pci) | Params.multiple() | Params.short("b"),
> +    )
>      other_eal_param: Params | None = None
>      _separator: Literal[True] = field(default=True, init=False, metadata=Params.short("-"))
> diff --git a/dts/framework/params/types.py b/dts/framework/params/types.py
> index d77c4625fb..87d11502e8 100644
> --- a/dts/framework/params/types.py
> +++ b/dts/framework/params/types.py
> @@ -50,7 +50,8 @@ class EalParamsDict(TypedDict, total=False):
>      prefix: str
>      no_pci: Switch
>      vdevs: list[VirtualDevice] | None
> -    ports: list[Port] | None
> +    allowed_ports: list[Port] | None
> +    blocked_ports: list[Port] | None
>      other_eal_param: Params | None
>
>
> diff --git a/dts/framework/remote_session/dpdk_shell.py b/dts/framework/remote_session/dpdk_shell.py
> index b39132cc42..82fa4755f0 100644
> --- a/dts/framework/remote_session/dpdk_shell.py
> +++ b/dts/framework/remote_session/dpdk_shell.py
> @@ -56,8 +56,8 @@ def compute_eal_params(
>          sut_node.dpdk_prefix_list.append(prefix)
>      params.prefix = prefix
>
> -    if params.ports is None:
> -        params.ports = sut_node.ports
> +    if params.allowed_ports is None:
> +        params.allowed_ports = sut_node.ports
>
>      return params
>
> diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py
> index 8a45a5231b..221465f6fb 100644
> --- a/dts/framework/remote_session/testpmd_shell.py
> +++ b/dts/framework/remote_session/testpmd_shell.py
> @@ -1465,7 +1465,7 @@ def start(self, verify: bool = True) -> None:
>                  self._logger.debug(f"Failed to start packet forwarding: \n{start_cmd_output}")
>                  raise InteractiveCommandExecutionError("Testpmd failed to start packet forwarding.")
>
> -            number_of_ports = len(self._app_params.ports or [])
> +            number_of_ports = len(self._app_params.allowed_ports or [])
>              for port_id in range(number_of_ports):
>                  if not self.wait_link_status_up(port_id):
>                      raise InteractiveCommandExecutionError(
> --
> 2.43.0
>

Re: [PATCH v2 1/2] dts: add blocked ports to EalParams

[Luca Vizzarro]

On 07/11/2024 19:04, Nicholas Pratte wrote:
> Hi Luca, this looks good to me! The only thing that needs to be fixed
> is the attributes component of the docstring for EalParams in eal.py.
> Aside from that:
> 
> Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>

Great, catch! Thanks!

[PATCH v2 2/2] dts: add blocklist test suite

[Luca Vizzarro]

This test suite tests the port blocklisting functionality built in
testpmd.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
---
 dts/framework/config/conf_yaml_schema.json |  3 +-
 dts/tests/TestSuite_blocklist.py           | 63 ++++++++++++++++++++++
 2 files changed, 65 insertions(+), 1 deletion(-)
 create mode 100644 dts/tests/TestSuite_blocklist.py

diff --git a/dts/framework/config/conf_yaml_schema.json b/dts/framework/config/conf_yaml_schema.json
index cc3e78cef5..65d0991686 100644
--- a/dts/framework/config/conf_yaml_schema.json
+++ b/dts/framework/config/conf_yaml_schema.json
@@ -244,7 +244,8 @@
         "hello_world",
         "os_udp",
         "pmd_buffer_scatter",
-        "vlan"
+        "vlan",
+        "blocklist"
       ]
     },
     "test_target": {
diff --git a/dts/tests/TestSuite_blocklist.py b/dts/tests/TestSuite_blocklist.py
new file mode 100644
index 0000000000..b9e9cd1d1a
--- /dev/null
+++ b/dts/tests/TestSuite_blocklist.py
@@ -0,0 +1,63 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Arm Limited
+
+"""The DPDK device blocklisting test suite.
+
+This testing suite ensures tests the port blocklisting functionality of testpmd.
+"""
+
+from framework.remote_session.testpmd_shell import TestPmdShell
+from framework.test_suite import TestSuite, func_test
+from framework.testbed_model.capability import TopologyType, requires
+from framework.testbed_model.port import Port
+
+
+@requires(topology_type=TopologyType.two_links)
+class TestBlocklist(TestSuite):
+    """DPDK device blocklisting test suite."""
+
+    def verify_blocklisted_ports(self, ports_to_block: list[Port]):
+        """Runs testpmd with the given ports blocklisted and verifies the ports."""
+        with TestPmdShell(self.sut_node, allowed_ports=[], blocked_ports=ports_to_block) as testpmd:
+            allowlisted_ports = {port.device_name for port in testpmd.show_port_info_all()}
+            blocklisted_ports = {port.pci for port in ports_to_block}
+
+            # sanity check
+            allowed_len = len(allowlisted_ports - blocklisted_ports)
+            self.verify(allowed_len > 0, "At least one port should have been allowed")
+
+            blocked = not allowlisted_ports & blocklisted_ports
+            self.verify(blocked, "At least one port was not blocklisted")
+
+    @func_test
+    def no_blocklisted(self):
+        """Run testpmd with no blocklisted device.
+
+        Steps:
+            Run testpmd without specifying allowed or blocked ports.
+        Verify:
+            That no ports were blocked.
+        """
+        self.verify_blocklisted_ports([])
+
+    @func_test
+    def one_port_blocklisted(self):
+        """Run testpmd with one blocklisted port.
+
+        Steps:
+            Run testpmd with one only one blocklisted port and allowing all the other ones.
+        Verify:
+            That the port was successfully blocklisted.
+        """
+        self.verify_blocklisted_ports(self.sut_node.ports[:1])
+
+    @func_test
+    def all_but_one_port_blocklisted(self):
+        """Run testpmd with all but one blocklisted port.
+
+        Steps:
+            Run testpmd with only one allowed port, blocking all the other ones.
+        Verify:
+            That all specified ports were successfully blocklisted.
+        """
+        self.verify_blocklisted_ports(self.sut_node.ports[:-1])
-- 
2.43.0

Re: [PATCH v2 2/2] dts: add blocklist test suite

[Nicholas Pratte]

Hi Luca, nice work!

See my small comment below/ I ran some local tests and everything
seems to be working fine, so other than that minor fix I pointed out,
the rest should be all set!

<snip>
> +"""The DPDK device blocklisting test suite.
> +
> +This testing suite ensures tests the port blocklisting functionality of testpmd.

I would remove the 'tests' and just write 'testing suite ensures the port...'

> +"""
> +
<snip>

Re: [PATCH v2 2/2] dts: add blocklist test suite

[Luca Vizzarro]

On 07/11/2024 20:33, Nicholas Pratte wrote:
> Hi Luca, nice work!
> 
> See my small comment below/ I ran some local tests and everything
> seems to be working fine, so other than that minor fix I pointed out,
> the rest should be all set!


Good suggestion, I've changed it a little bit now. Thanks!