MLX5 VF stops transmitting when the Physical Function is added to a Linux bridge

MLX5 VF stops transmitting when the Physical Function is added to a Linux bridge

[Antonio Di Bacco]

I have a Connect X5 card (PF ens1f0np0) directly connected to another server:

1) create VF on PF on both servers
2) change mac address of VFs to my own addressing
3) start testpmd on server 1 in txonly mode to transmit to server 0
4) start testpmd on server 0 in rxonly mode to receive
5) everything is fine, I keep receiving packets on node-0

Now, on server 1 I add the PF to a linux bridge, and everything's still fine.

If I add another interface (a simple 1Gbps with no VF, ens5f0)  to the
linux bridge, then, I don't receive anymore packets on node-0

If I remove the ens5f0 from the bridge or I put down the ens5f0 the
traffic flow restarts.

I understand that DPDK uses the VF directly with no dependencies on
the kernel. How can operations on the kernel side (like adding an
interface to bridge) can affect the VF?


Best regards,
Antonio.

Re: MLX5 VF stops transmitting when the Physical Function is added to a Linux bridge

[Stephen Hemminger]

On Mon, 25 Mar 2024 15:59:36 +0100
Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:

> I have a Connect X5 card (PF ens1f0np0) directly connected to another server:
> 
> 1) create VF on PF on both servers
> 2) change mac address of VFs to my own addressing
> 3) start testpmd on server 1 in txonly mode to transmit to server 0
> 4) start testpmd on server 0 in rxonly mode to receive
> 5) everything is fine, I keep receiving packets on node-0
> 
> Now, on server 1 I add the PF to a linux bridge, and everything's still fine.
> 
> If I add another interface (a simple 1Gbps with no VF, ens5f0)  to the
> linux bridge, then, I don't receive anymore packets on node-0
> 
> If I remove the ens5f0 from the bridge or I put down the ens5f0 the
> traffic flow restarts.
> 
> I understand that DPDK uses the VF directly with no dependencies on
> the kernel. How can operations on the kernel side (like adding an
> interface to bridge) can affect the VF?
> 
> 
> Best regards,
> Antonio.

Mellanox is bifurcated driver, so kernel and DPDK interact.
Adding device to bridge will change MAC address.

Re: MLX5 VF stops transmitting when the Physical Function is added to a Linux bridge

[Antonio Di Bacco]

Hi Stephen,

I know that the mlx5_core is bifurcated, I did another test,  I just
added one of the VF to the bridge but I see that testpmd using other
VFs suffer the same problem.
When I add to the bridge another interface (not the one with VF but a
simple 1Gbps named ens5f0) I see that VFs used by testpmd stop
working.

If I give the command ifconfig ens5f0 down, then, the testpmd VF
resumes transmission.

Best regards,
Antonio.


On Mon, Mar 25, 2024 at 6:14 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Mon, 25 Mar 2024 15:59:36 +0100
> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>
> > I have a Connect X5 card (PF ens1f0np0) directly connected to another server:
> >
> > 1) create VF on PF on both servers
> > 2) change mac address of VFs to my own addressing
> > 3) start testpmd on server 1 in txonly mode to transmit to server 0
> > 4) start testpmd on server 0 in rxonly mode to receive
> > 5) everything is fine, I keep receiving packets on node-0
> >
> > Now, on server 1 I add the PF to a linux bridge, and everything's still fine.
> >
> > If I add another interface (a simple 1Gbps with no VF, ens5f0)  to the
> > linux bridge, then, I don't receive anymore packets on node-0
> >
> > If I remove the ens5f0 from the bridge or I put down the ens5f0 the
> > traffic flow restarts.
> >
> > I understand that DPDK uses the VF directly with no dependencies on
> > the kernel. How can operations on the kernel side (like adding an
> > interface to bridge) can affect the VF?
> >
> >
> > Best regards,
> > Antonio.
>
> Mellanox is bifurcated driver, so kernel and DPDK interact.
> Adding device to bridge will change MAC address.