Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver)

Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver)

[Tao Li]

[-- Attachment #1: Type: text/plain, Size: 5018 bytes --]

Hi All,

I am currently experimenting with a feature newly supported by DPDK 23.11, known as "multiport e-switch<https://doc.dpdk.org/guides/nics/mlx5.html#multiport-e-switch>" to improve communication reliability on the server side. During the trials, I encountered an issue in which activating multiport e-switch mode on the NIC disrupts the hypervisor’s software running on the second PF interface (PF1). More specifically, packets coming from the second PF (PF1) cannot be delivered to hypervisor’s kernel network stack, right after setting the multiport e-switch mode for the NIC as guided in documentation. A snapshot of the packet trace comparison on the second PF (PF1, ens2f1np1) before and after setting the multiport e-switch mode is attached here.  Packets marked with the gray color/italic in the second trace are missing under the multiport e-switch mode.

----<test environment>-----
ConnectX-6 Dx with firmware version 22.39.1002
Linux kernel version: 6.6.16
DPDK: 23.11
----</test environment>------

----<packet trace after setting multiport e-switch mode>------
14:37:24.835716 04:3f:72:e8:cf:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: fe80::63f:72ff:fee8:cfcb > ff02::1: ICMP6, router advertisement, length 24

14:37:28.527829 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advertisement, length 24

14:37:28.528359 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS val 1610632473 ecr 0,nop,wscale 7], length 0 // link-local addresses are used

14:37:29.559918 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS val 1610633505 ecr 0,nop,wscale 7], length 0

14:37:30.583925 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS val 1610634529 ecr 0,nop,wscale 7], length 0
----</packet trace after setting multiport e-switch mode>------

----<packet trace before setting multiport e-switch mode> ------
16:09:40.375865 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advertisement, length 24

16:09:40.376473 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 3409227589, win 33120, options [mss 1440,sackOK,TS val 2302010436 ecr 0,nop,wscale 7], length 0

16:09:40.376692 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x86dd), length 94: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9.36168: Flags [S.], seq 3495571820, ack 3409227590, win 63196, options [mss 9040,sackOK,TS val 1054058675 ecr 2302010436,nop,wscale 9], length 0

16:09:40.376711 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 86: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83fb.179: Flags [.], ack 1, win 259, options [nop,nop,TS val 2302010436 ecr 1054058675], length 0

16:09:40.376865 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 193: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83fb.179: Flags [P.], seq 1:108, ack 1, win 259, options [nop,nop,TS val 2302010436 ecr 1054058675], length 107: BGP

16:09:40.376986 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x86dd), length 86: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9.36168: Flags [.], ack 108, win 124, options [nop,nop,TS val 1054058676 ecr 2302010436], length 0
----</packet trace before setting multiport e-switch mode> ------

Attempts to ping from another directly connected host to this hypervisor also resulted in incoming ICMP packets not being captured, which is reproducible in another testing environment setup. In the end, I was able to restore communication on the second PF by using a vdev TAP device and performing packet forwarding between the TAP device and PF1, as shown in our public examplary code<https://github.com/byteocean/multiport-eswitch-example>.

Enabling the isolation mode on PF1 by starting testpmd or programmably using `rte_flow_isolate()` leads to no change from the behavior as described above, but only affects whether packets can be captured and processed by the DPDK application.
----<command to start testpmd> ------
sudo ./dpdk-testpmd -a 3b:00.0,dv_flow_en=2,dv_esw_en=1,fdb_def_rule_en=1,representor=pf0-1vf0 -- -i --rxq=1 --txq=1 --flow-isolate-all
----</command to start testpmd> ------

Any experience sharing or comment on the above described issue is very appreciated. Thanks a lot in advance.

Best regards,
Tao Li




[-- Attachment #2: Type: text/html, Size: 21588 bytes --]

RE: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver)

[Dariusz Sosnowski]

Hi,

I could not reproduce the issue locally with testpmd, with flow isolation enabled. I can see ICMP packets passing both ways to kernel interfaces of PF0 and PF1.
Without flow isolation, it is expected that traffic coming to the host will be hijacked by DPDK (depending on the MAC address, multicast config and promiscuous mode).

Could you please run testpmd with the following command line parameters and execute the following commands?

Testpmd command line:
	dpdk-testpmd -a 3b:00.0,dv_flow_en=2,representor=pf0-1vf0 -- --flow-isolate-all -i

Testpmd commands:
	port stop all
	flow configure 0 queues_number 4 queues_size 64
	flow configure 1 queues_number 4 queues_size 64
	flow configure 2 queues_number 4 queues_size 64
	flow configure 3 queues_number 4 queues_size 64
	port start 0
	port start 1
	port start 2
	port start 3
	set verbose 1
	set fwd rxonly
	start

With this testpmd running, could you please test if both PF0 and PF1 kernel interfaces are reachable and all packets pass?

Best regards,
Dariusz Sosnowski

> From: Tao Li <byteocean@hotmail.com> 
> Sent: Wednesday, April 10, 2024 10:18
> To: users@dpdk.org
> Cc: tao.li06@sap.com
> Subject: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver)
> 
> External email: Use caution opening links or attachments 
> 
> Hi All,
>  
> I am currently experimenting with a feature newly supported by DPDK 23.11, known as "https://doc.dpdk.org/guides/nics/mlx5.html#multiport-e-switch" to improve communication reliability on the server side. During the trials, I encountered an issue in which activating multiport e-switch mode on the NIC disrupts the hypervisor's software running on the second PF interface (PF1). More specifically, packets coming from the second PF (PF1) cannot be delivered to hypervisor's kernel network stack, right after setting the multiport e-switch mode for the NIC as guided in documentation. A snapshot of the packet trace comparison on the second PF (PF1, ens2f1np1) before and after setting the multiport e-switch mode is attached here.  Packets marked with the gray color/italic in the second trace are missing under the multiport e-switch mode.
>  
> ----<test environment>-----
> ConnectX-6 Dx with firmware version 22.39.1002
> Linux kernel version: 6.6.16
> DPDK: 23.11
> ----</test environment>------
>  
> ----<packet trace after setting multiport e-switch mode>------
> 14:37:24.835716 04:3f:72:e8:cf:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: fe80::63f:72ff:fee8:cfcb > ff02::1: ICMP6, router advertisement, length 24
>  
> 14:37:28.527829 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advertisement, length 24
>  
> 14:37:28.528359 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS val 1610632473 ecr 0,nop,wscale 7], length 0 // link-local addresses are used
>  
> 14:37:29.559918 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS val 1610633505 ecr 0,nop,wscale 7], length 0
>  
> 14:37:30.583925 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS val 1610634529 ecr 0,nop,wscale 7], length 0
> ----</packet trace after setting multiport e-switch mode>------
>  
> ----<packet trace before setting multiport e-switch mode> ------
> 16:09:40.375865 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advertisement, length 24
>  
> 16:09:40.376473 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 3409227589, win 33120, options [mss 1440,sackOK,TS val 2302010436 ecr 0,nop,wscale 7], length 0
>  
> 16:09:40.376692 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x86dd), length 94: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9.36168: Flags [S.], seq 3495571820, ack 3409227590, win 63196, options [mss 9040,sackOK,TS val 1054058675 ecr 2302010436,nop,wscale 9], length 0
>  
> 16:09:40.376711 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 86: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83fb.179: Flags [.], ack 1, win 259, options [nop,nop,TS val 2302010436 ecr 1054058675], length 0
>  
> 16:09:40.376865 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 193: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83fb.179: Flags [P.], seq 1:108, ack 1, win 259, options [nop,nop,TS val 2302010436 ecr 1054058675], length 107: BGP
>  
> 16:09:40.376986 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x86dd), length 86: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9.36168: Flags [.], ack 108, win 124, options [nop,nop,TS val 1054058676 ecr 2302010436], length 0
> ----</packet trace before setting multiport e-switch mode> ------
>  
> Attempts to ping from another directly connected host to this hypervisor also resulted in incoming ICMP packets not being captured, which is reproducible in another testing environment setup. In the end, I was able to restore communication on the second PF by using a vdev TAP device and performing packet forwarding between the TAP device and PF1, as shown in our public examplary https://github.com/byteocean/multiport-eswitch-example.
>  
> Enabling the isolation mode on PF1 by starting testpmd or programmably using `rte_flow_isolate()` leads to no change from the behavior as described above, but only affects whether packets can be captured and processed by the DPDK application.
> ----<command to start testpmd> ------
> sudo ./dpdk-testpmd -a 3b:00.0,dv_flow_en=2,dv_esw_en=1,fdb_def_rule_en=1,representor=pf0-1vf0 -- -i --rxq=1 --txq=1 --flow-isolate-all
> ----</command to start testpmd> ------
>  
> Any experience sharing or comment on the above described issue is very appreciated. Thanks a lot in advance.
>  
> Best regards,
> Tao Li

Re: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver)

[Tao Li]

[-- Attachment #1: Type: text/plain, Size: 8273 bytes --]

Hi Dariusz,

It is very appreciated that you took a look at the issue and provided suggestions. This time, we again performed tests using two directly connected machines and focused on ICMP (IPv4) packets in addition to ICMPv6 packets mentioned in the original problem description. The issue remains the same. I would like to highlight two points in our setup:


  1.  ICMP packets immediately cannot be captured on PF1 right after setting the nic into the multiport eswitch mode. And if I switch off the multiport eswitch mode by using following two commands, ICMP communication is resumed immediately, which shall prove that configs, such as firewall, on the system are correct. I would also assume it has little to do with a running DPDK application, as communication is already broken before starting an application like testpmd.

sudo devlink dev param set pci/0000:3b:00.0 name esw_multiport value false cmode runtime

sudo devlink dev param set pci/0000:3b:00.1 name esw_multiport value false cmode runtime


  1.  In this setup, we do not use MLNX_OFED drivers but rely on the upstream Mellanox drivers from Linux kernel 6.5.0 (which is greater than the suggested kernel version 6.3). Would that make a difference? Could you share some more detailed information regarding the environment setup on your side? The firmware version we are using for Mellanox ConnectX-6 is 22.39.1002.

Looking forward to your further reply. Thanks in advance.

Best regards,
Tao Li

From: Dariusz Sosnowski <dsosnowski@nvidia.com>
Date: Friday, 19. April 2024 at 19:30
To: Tao Li <byteocean@hotmail.com>, users@dpdk.org <users@dpdk.org>
Cc: tao.li06@sap.com <tao.li06@sap.com>
Subject: RE: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver)
Hi,

I could not reproduce the issue locally with testpmd, with flow isolation enabled. I can see ICMP packets passing both ways to kernel interfaces of PF0 and PF1.
Without flow isolation, it is expected that traffic coming to the host will be hijacked by DPDK (depending on the MAC address, multicast config and promiscuous mode).

Could you please run testpmd with the following command line parameters and execute the following commands?

Testpmd command line:
        dpdk-testpmd -a 3b:00.0,dv_flow_en=2,representor=pf0-1vf0 -- --flow-isolate-all -i

Testpmd commands:
        port stop all
        flow configure 0 queues_number 4 queues_size 64
        flow configure 1 queues_number 4 queues_size 64
        flow configure 2 queues_number 4 queues_size 64
        flow configure 3 queues_number 4 queues_size 64
        port start 0
        port start 1
        port start 2
        port start 3
        set verbose 1
        set fwd rxonly
        start

With this testpmd running, could you please test if both PF0 and PF1 kernel interfaces are reachable and all packets pass?

Best regards,
Dariusz Sosnowski

> From: Tao Li <byteocean@hotmail.com>
> Sent: Wednesday, April 10, 2024 10:18
> To: users@dpdk.org
> Cc: tao.li06@sap.com
> Subject: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver)
>
> External email: Use caution opening links or attachments
>
> Hi All,
>
> I am currently experimenting with a feature newly supported by DPDK 23.11, known as "https://doc.dpdk.org/guides/nics/mlx5.html#multiport-e-switch" to improve communication reliability on the server side. During the trials, I encountered an issue in which activating multiport e-switch mode on the NIC disrupts the hypervisor's software running on the second PF interface (PF1). More specifically, packets coming from the second PF (PF1) cannot be delivered to hypervisor's kernel network stack, right after setting the multiport e-switch mode for the NIC as guided in documentation. A snapshot of the packet trace comparison on the second PF (PF1, ens2f1np1) before and after setting the multiport e-switch mode is attached here.  Packets marked with the gray color/italic in the second trace are missing under the multiport e-switch mode.
>
> ----<test environment>-----
> ConnectX-6 Dx with firmware version 22.39.1002
> Linux kernel version: 6.6.16
> DPDK: 23.11
> ----</test environment>------
>
> ----<packet trace after setting multiport e-switch mode>------
> 14:37:24.835716 04:3f:72:e8:cf:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: fe80::63f:72ff:fee8:cfcb > ff02::1: ICMP6, router advertisement, length 24
>
> 14:37:28.527829 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advertisement, length 24
>
> 14:37:28.528359 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS val 1610632473 ecr 0,nop,wscale 7], length 0 // link-local addresses are used
>
> 14:37:29.559918 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS val 1610633505 ecr 0,nop,wscale 7], length 0
>
> 14:37:30.583925 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS val 1610634529 ecr 0,nop,wscale 7], length 0
> ----</packet trace after setting multiport e-switch mode>------
>
> ----<packet trace before setting multiport e-switch mode> ------
> 16:09:40.375865 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advertisement, length 24
>
> 16:09:40.376473 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 94: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83fb.179: Flags [S], seq 3409227589, win 33120, options [mss 1440,sackOK,TS val 2302010436 ecr 0,nop,wscale 7], length 0
>
> 16:09:40.376692 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x86dd), length 94: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9.36168: Flags [S.], seq 3495571820, ack 3409227590, win 63196, options [mss 9040,sackOK,TS val 1054058675 ecr 2302010436,nop,wscale 9], length 0
>
> 16:09:40.376711 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 86: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83fb.179: Flags [.], ack 1, win 259, options [nop,nop,TS val 2302010436 ecr 1054058675], length 0
>
> 16:09:40.376865 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x86dd), length 193: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83fb.179: Flags [P.], seq 1:108, ack 1, win 259, options [nop,nop,TS val 2302010436 ecr 1054058675], length 107: BGP
>
> 16:09:40.376986 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x86dd), length 86: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9.36168: Flags [.], ack 108, win 124, options [nop,nop,TS val 1054058676 ecr 2302010436], length 0
> ----</packet trace before setting multiport e-switch mode> ------
>
> Attempts to ping from another directly connected host to this hypervisor also resulted in incoming ICMP packets not being captured, which is reproducible in another testing environment setup. In the end, I was able to restore communication on the second PF by using a vdev TAP device and performing packet forwarding between the TAP device and PF1, as shown in our public examplary https://github.com/byteocean/multiport-eswitch-example.
>
> Enabling the isolation mode on PF1 by starting testpmd or programmably using `rte_flow_isolate()` leads to no change from the behavior as described above, but only affects whether packets can be captured and processed by the DPDK application.
> ----<command to start testpmd> ------
> sudo ./dpdk-testpmd -a 3b:00.0,dv_flow_en=2,dv_esw_en=1,fdb_def_rule_en=1,representor=pf0-1vf0 -- -i --rxq=1 --txq=1 --flow-isolate-all
> ----</command to start testpmd> ------
>
> Any experience sharing or comment on the above described issue is very appreciated. Thanks a lot in advance.
>
> Best regards,
> Tao Li

[-- Attachment #2: Type: text/html, Size: 14623 bytes --]