Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[CJ Sculti <cj@cj.gy>]

[-- Attachment #1: Type: text/plain, Size: 5446 bytes --]

Thank you for the advice on that. I have removed the reset, and I also
fixed the rte_eth_dev_get_supported_ptypes(). I moved
my rte_eth_dev_get_supported_ptypes() calls after the bypass interface was
configured, and now they're returning expected values.

My next issue I'm running into is the bonding issue that I talked about in
my initial post. It seems that when part of a kernel bond, the mlx5_core
driver combines both ports into a single 'verbs' device named
'mlx5_bond_0'.  On my old setup with igb_uio, it worked like this:

- At Linux boot, the 2x Intel ports were configured as a bond on the kernel
via /etc/network/interfaces file.
- Before starting DPDK software, both ports bound to igb_uio driver.
- Software started, ports are setup by my software as kernel bypass.
- Enslaved the 2x 'new' bypass interfaces onto the same bond.
- Software reinjected LACP packets into kernel, to let kernel handle LACP
protocol.

This behavior where both ports are 'combined' into a single verbs device is
strange to me. How should I handle this? Is there any way to disable it and
just have the 2 ports be separate interfaces?

   1.
   2.

   root@DDoSMitigation:~/anubis/engine/bin# ibv_devinfo
   hca_id:    mlx5_bond_0
       transport:            InfiniBand (0)
       fw_ver:                16.35.4030
       node_guid:            506b:4b03:00b6:76ec
       sys_image_guid:            506b:4b03:00b6:76ec
       vendor_id:            0x02c9
       vendor_part_id:            4119
       hw_ver:                0x0
       board_id:            MT_0000000090
       phys_port_cnt:            1
           port:    1
               state:            PORT_ACTIVE (4)
               max_mtu:        4096 (5)
               active_mtu:        1024 (3)
               sm_lid:            0
               port_lid:        0
               port_lmc:        0x00
               link_layer:        Ethernet

   3. [12:58 PM]



On Fri, Nov 15, 2024 at 5:12 AM Yasuhiro Ohara <yasu1976@gmail.com> wrote:

> > I think you should try a bit more, we are here to help.
>
> I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK,
> and I think you can make it work in only a few more steps.
>
> I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP
> might have made the whole NIC functions stopped.
> IIRC the ptype was working fine on CX5 in my app too.
> Can you comment out the rte_eth_dev_reset() ?
> (I think it's worth to try.)
>
> I don't think it is so uphill, but I don't disagree with you about
> purchasing another
> Intel 40G NICs. It'll also work perfectly fine.
>
> 2024年11月15日(金) 4:16 Thomas Monjalon <thomas@monjalon.net>:
> >
> > 14/11/2024 17:10, CJ Sculti:
> > > I figured out the initial issue. For some reason, having both devices
> in a
> > > bond on the kernel results in only 1 of the two ports being exposed as
> > > 'verb' devices. Previously, ibv_devinfo returned only one port. After
> > > removing both from the bond, ibv_devinfo returns both ports, and the
> DPDK
> > > application successfully takes both in. I'm still having some weird
> > > behavior trying to create a bypass interface with these ports though.
> I'm
> > > using the same code that I've been using on my Intel NICs with igb_uio
> for
> > > years, but seeing weird behavior. The ports are connected to our 40Gbps
> > > Ethernet switch, and set to link_layer: Ethernet.
> >
> > You should be able to make it work with kernel bonding,
> > but I'm not sure what's wrong to do that.
> > And it looks not a priority for you. Let's focus on the other parts.
> >
> >
> > > The first thing I noticed is that rte_eth_dev_reset() fails on these
> > > interfaces with "ENOTSUP: hardware doesn't support reset".
> >
> > You don't need the reset procedure with mlx5,
> > so you can make this code optional.
> >
> >
> > > Secondly, when checking ptypes, I noticed my code says these NICs are
> > > unable to support any sort of packet detection capability (code below,
> all
> > > return false). The MLX5 docs do say that all of these ptypes used here
> are
> > > supported by MLX5.
> >
> > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get()
> code.
> > I don't understand why it does not work for you.
> >
> >
> > > I'm just picking up a project that was left off by an older dev. It
> hasn't
> > > been touched in years, but has been working fine with our Intel NICs.
> All
> > > I'm trying to do is update DPDK (which is done, updated from dpdk
> 19.05 to
> > > DPDK 22.11, latest version with KNI support),
> >
> > You don't need KNI with mlx5.
> > That's a big benefit of mlx5 design, it is natively bifurcated:
> >         https://doc.dpdk.org/guides/howto/flow_bifurcation.html
> >
> >
> > > and get it to work with our Mellanox CX5 NICs.
> > > This is my first time working with DPDK and I'm not very
> > > familiar. Should I expect to be able to do this without having to make
> a
> > > ton of code changes, or is this going to be an uphill battle for me? If
> > > it's the latter, I will likely just go purchase Intel NICs and give up
> on
> > > this.
> >
> > The NICs have difference that DPDK is trying to hide.
> > If something is not compatible you may consider it as a bug or a
> limitation.
> > I think you should try a bit more, we are here to help.
> >
> >
>

[-- Attachment #2: Type: text/html, Size: 16359 bytes --]