DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[CJ Sculti]

[-- Attachment #1: Type: text/plain, Size: 2004 bytes --]

I've been running my application for years on igb_uio with Intel NICs. I
recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated
the DPDK version my application uses, and compiled with support for mlx5
PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode,
not Infiniband mode. However, I'm getting complaints when I start my
application about trying to load 'mlx5_eth'? Both are bound to mlx5_core
driver at the moment. When I bind them to vfio-pci, or uio_pci_generic, my
application fails to recognize them at all as valid DPDK devices. Anyone
have any ideas? Also, strange that it only complains about one? I have them
configured in a bond on the kernel, as my application requires that.

Network devices using kernel driver

===================================

0000:2b:00.0 'MT27800 Family [ConnectX-5] 1017' if=enp43s0f0np0
drv=mlx5_core unused=vfio-pci

0000:2b:00.1 'MT27800 Family [ConnectX-5] 1017' if=enp43s0f1np1
drv=mlx5_core unused=vfio-pci

root@DDoSMitigation:~/anubis/engine/bin# ./anubis-engine


  Electric Fence 2.2 Copyright (C) 1987-1999 Bruce Perens <bruce@perens.com>

EAL: Detected CPU lcores: 12

EAL: Detected NUMA nodes: 1

EAL: Detected static linkage of DPDK

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'VA'

EAL: VFIO support initialized

EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:2b:00.0 (socket -1)

EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:2b:00.1 (socket -1)

mlx5_net: PF 0 doesn't have Verbs device matches PCI device 0000:2b:00.1,
are kernel drivers loaded?

mlx5_common: Failed to load driver mlx5_eth

EAL: Requested device 0000:2b:00.1 cannot be used

TELEMETRY: No legacy callbacks, legacy socket not created

USER1: Anubis build master/.

USER1: We will run on 12 logical cores.

USER1: Enabled lcores not a power of 2!

This could have performance issues.

KNI: WARNING: KNI is deprecated and will be removed in DPDK 23.11

USER1: Failed to reset link fe0.

[-- Attachment #2: Type: text/html, Size: 11554 bytes --]

Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[Thomas Monjalon]

13/11/2024 21:10, CJ Sculti:
> I've been running my application for years on igb_uio with Intel NICs. I
> recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated
> the DPDK version my application uses, and compiled with support for mlx5
> PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode,
> not Infiniband mode. However, I'm getting complaints when I start my
> application about trying to load 'mlx5_eth'? Both are bound to mlx5_core
> driver at the moment. When I bind them to vfio-pci, or uio_pci_generic, my
> application fails to recognize them at all as valid DPDK devices. Anyone
> have any ideas? Also, strange that it only complains about one? I have them
> configured in a bond on the kernel, as my application requires that.

You must not bind mlx5 devices with VFIO.
I recommend reading documentation.
You can start here:
https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver
then
https://doc.dpdk.org/guides/platform/mlx5.html#design

Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[CJ Sculti]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

I'm not using vfio, I just bound interfaces on there one time to test.
Shouldn't I be able to just use the default mlx5_core driver, without
binding to uio_pci_generic?


On Wed, Nov 13, 2024 at 4:26 PM Thomas Monjalon <thomas@monjalon.net> wrote:

> 13/11/2024 21:10, CJ Sculti:
> > I've been running my application for years on igb_uio with Intel NICs. I
> > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated
> > the DPDK version my application uses, and compiled with support for mlx5
> > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode,
> > not Infiniband mode. However, I'm getting complaints when I start my
> > application about trying to load 'mlx5_eth'? Both are bound to mlx5_core
> > driver at the moment. When I bind them to vfio-pci, or uio_pci_generic,
> my
> > application fails to recognize them at all as valid DPDK devices. Anyone
> > have any ideas? Also, strange that it only complains about one? I have
> them
> > configured in a bond on the kernel, as my application requires that.
>
> You must not bind mlx5 devices with VFIO.
> I recommend reading documentation.
> You can start here:
> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver
> then
> https://doc.dpdk.org/guides/platform/mlx5.html#design
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 2098 bytes --]

Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[Thomas Monjalon]

13/11/2024 22:43, CJ Sculti:
> I'm not using vfio, I just bound interfaces on there one time to test.
> Shouldn't I be able to just use the default mlx5_core driver, without
> binding to uio_pci_generic?

Yes you must not bind to anything else than mlx5 kernel driver.
Please try to follow the documentation and tell us what you find.


> On Wed, Nov 13, 2024 at 4:26 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> > 13/11/2024 21:10, CJ Sculti:
> > > I've been running my application for years on igb_uio with Intel NICs. I
> > > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated
> > > the DPDK version my application uses, and compiled with support for mlx5
> > > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode,
> > > not Infiniband mode. However, I'm getting complaints when I start my
> > > application about trying to load 'mlx5_eth'? Both are bound to mlx5_core
> > > driver at the moment. When I bind them to vfio-pci, or uio_pci_generic,
> > my
> > > application fails to recognize them at all as valid DPDK devices. Anyone
> > > have any ideas? Also, strange that it only complains about one? I have
> > them
> > > configured in a bond on the kernel, as my application requires that.
> >
> > You must not bind mlx5 devices with VFIO.
> > I recommend reading documentation.
> > You can start here:
> > https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver
> > then
> > https://doc.dpdk.org/guides/platform/mlx5.html#design

Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[Yasuhiro Ohara]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

I would suggest re-installation of MELLANOX OFED, and/or upgrading NIC
firmware (can be done using OFED tools).

Yes, warning on only 1 port is odd. I suspect some kind of mismatch, like
older version of the NIC had only 1 port versions and the soft assumed it,
for example.


2024年11月14日(木) 5:43 CJ Sculti <cj@cj.gy>:

> I'm not using vfio, I just bound interfaces on there one time to test.
> Shouldn't I be able to just use the default mlx5_core driver, without
> binding to uio_pci_generic?
>
>
> On Wed, Nov 13, 2024 at 4:26 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
>
>> 13/11/2024 21:10, CJ Sculti:
>> > I've been running my application for years on igb_uio with Intel NICs. I
>> > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated
>> > the DPDK version my application uses, and compiled with support for mlx5
>> > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode,
>> > not Infiniband mode. However, I'm getting complaints when I start my
>> > application about trying to load 'mlx5_eth'? Both are bound to mlx5_core
>> > driver at the moment. When I bind them to vfio-pci, or uio_pci_generic,
>> my
>> > application fails to recognize them at all as valid DPDK devices. Anyone
>> > have any ideas? Also, strange that it only complains about one? I have
>> them
>> > configured in a bond on the kernel, as my application requires that.
>>
>> You must not bind mlx5 devices with VFIO.
>> I recommend reading documentation.
>> You can start here:
>> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver
>> then
>> https://doc.dpdk.org/guides/platform/mlx5.html#design
>>
>>
>>
>>

[-- Attachment #2: Type: text/html, Size: 2815 bytes --]

Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[CJ Sculti]

[-- Attachment #1: Type: text/plain, Size: 6603 bytes --]

I figured out the initial issue. For some reason, having both devices in a
bond on the kernel results in only 1 of the two ports being exposed as
'verb' devices. Previously, ibv_devinfo returned only one port. After
removing both from the bond, ibv_devinfo returns both ports, and the DPDK
application successfully takes both in. I'm still having some weird
behavior trying to create a bypass interface with these ports though. I'm
using the same code that I've been using on my Intel NICs with igb_uio for
years, but seeing weird behavior. The ports are connected to our 40Gbps
Ethernet switch, and set to link_layer: Ethernet.

The first thing I noticed is that rte_eth_dev_reset() fails on these
interfaces with "ENOTSUP: hardware doesn't support reset".

Secondly, when checking ptypes, I noticed my code says these NICs are
unable to support any sort of packet detection capability (code below, all
return false). The MLX5 docs do say that all of these ptypes used here are
supported by MLX5.

I'm just picking up a project that was left off by an older dev. It hasn't
been touched in years, but has been working fine with our Intel NICs. All
I'm trying to do is update DPDK (which is done, updated from dpdk 19.05 to
DPDK 22.11, latest version with KNI support), and get it to work with our
Mellanox CX5 NICs. This is my first time working with DPDK and I'm not very
familiar. Should I expect to be able to do this without having to make a
ton of code changes, or is this going to be an uphill battle for me? If
it's the latter, I will likely just go purchase Intel NICs and give up on
this.


bool Manager::HasPacketTypeDetectionCapabilities(uint16_t port_id) {

  // Detect L2 packet type detection capability

  int num_ptypes = rte_eth_dev_get_supported_ptypes(
      port_id, RTE_PTYPE_ALL_MASK, nullptr, 0);

  if (num_ptypes == -1) {
    RTE_LOG(WARNING, USER1,
        "Driver does not report supported ptypes; enabling software
parsing.\n");
    // Proceed with software-based packet type parsing
    return true;
  }

  bool detect_ether = false, detect_arp = false;
  auto num_l2_dtx = rte_eth_dev_get_supported_ptypes(port_id,
      RTE_PTYPE_L2_MASK, nullptr, 0);
  if (num_l2_dtx == -1) {
    RTE_LOG(ERR, USER1, "Failed to detect L2 detection capabilities.\n");
    return false;
  }
  uint32_t l2_ptypes[num_l2_dtx];
  num_l2_dtx = rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L2_MASK,
      l2_ptypes, num_l2_dtx);
  for (auto i = 0; i < num_l2_dtx; ++i) {
    if (l2_ptypes[i] & RTE_PTYPE_L2_ETHER)
      detect_ether = true;
  }

  // Detect L3 packet type detection capability
  bool detect_ipv4 = false, detect_ipv6 = false;
  auto num_l3_dtx = rte_eth_dev_get_supported_ptypes(port_id,
      RTE_PTYPE_L3_MASK, nullptr, 0);
  if (num_l3_dtx == -1) {
    RTE_LOG(ERR, USER1, "Failed to detect L3 detection capabilities.\n");
    return false;
  }
  uint32_t l3_ptypes[num_l3_dtx];
  num_l3_dtx = rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L3_MASK,
      l3_ptypes, num_l3_dtx);
  for (auto i = 0; i < num_l3_dtx; ++i) {
    if (l3_ptypes[i] & RTE_PTYPE_L3_IPV4)
      detect_ipv4 = true;
    if (l3_ptypes[i] & RTE_PTYPE_L3_IPV6)
      detect_ipv6 = true;
  }

  // Detect L4 packet type detection capability
  bool detect_tcp = false, detect_udp = false, detect_icmp = false;
  auto num_l4_dtx = rte_eth_dev_get_supported_ptypes(port_id,
      RTE_PTYPE_L4_MASK, nullptr, 0);
  if (num_l4_dtx == -1) {
    RTE_LOG(ERR, USER1, "Failed to detect L4 detection capabilities.\n");
    return false;
  }
  uint32_t l4_ptypes[num_l4_dtx];
  num_l4_dtx = rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L4_MASK,
      l4_ptypes,  num_l4_dtx);
  for (auto i = 0; i < num_l4_dtx; ++i) {
    if (l4_ptypes[i] & RTE_PTYPE_L4_TCP)
      detect_tcp = true;
    if (l4_ptypes[i] & RTE_PTYPE_L4_UDP)
      detect_udp = true;
    if (l4_ptypes[i] & RTE_PTYPE_L4_ICMP)
      detect_icmp = true;
  }

  if (!detect_ether || !detect_arp
      || !detect_ipv4 || !detect_ipv6
      || !detect_tcp || !detect_udp || !detect_icmp) {
    RTE_LOG(ERR, USER1, "Supported Detection Modes:\n"
        "L2 Ether: %s\n"
        "L2 Arp  : %s\n"
        "L3 IPv4 : %s\n"
        "L3 IPv6 : %s\n"
        "L4 TCP  : %s\n"
        "L4 UDP  : %s\n"
        "L4 ICMP : %s\n",
        detect_ether ? "True" : "False",
        detect_arp ? "True": "False",
        detect_ipv4 ? "True": "False",
        detect_ipv6 ? "True": "False",
        detect_tcp ? "True" : "False",
        detect_udp ? "True" : "False",
        detect_icmp ? "True" : "False");
    return false;
  }

  return true;
}

On Wed, Nov 13, 2024 at 9:10 PM Yasuhiro Ohara <yasu1976@gmail.com> wrote:

> I would suggest re-installation of MELLANOX OFED, and/or upgrading NIC
> firmware (can be done using OFED tools).
>
> Yes, warning on only 1 port is odd. I suspect some kind of mismatch, like
> older version of the NIC had only 1 port versions and the soft assumed it,
> for example.
>
>
> 2024年11月14日(木) 5:43 CJ Sculti <cj@cj.gy>:
>
>> I'm not using vfio, I just bound interfaces on there one time to test.
>> Shouldn't I be able to just use the default mlx5_core driver, without
>> binding to uio_pci_generic?
>>
>>
>> On Wed, Nov 13, 2024 at 4:26 PM Thomas Monjalon <thomas@monjalon.net>
>> wrote:
>>
>>> 13/11/2024 21:10, CJ Sculti:
>>> > I've been running my application for years on igb_uio with Intel NICs.
>>> I
>>> > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC,
>>> updated
>>> > the DPDK version my application uses, and compiled with support for
>>> mlx5
>>> > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet
>>> mode,
>>> > not Infiniband mode. However, I'm getting complaints when I start my
>>> > application about trying to load 'mlx5_eth'? Both are bound to
>>> mlx5_core
>>> > driver at the moment. When I bind them to vfio-pci, or
>>> uio_pci_generic, my
>>> > application fails to recognize them at all as valid DPDK devices.
>>> Anyone
>>> > have any ideas? Also, strange that it only complains about one? I have
>>> them
>>> > configured in a bond on the kernel, as my application requires that.
>>>
>>> You must not bind mlx5 devices with VFIO.
>>> I recommend reading documentation.
>>> You can start here:
>>>
>>> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver
>>> then
>>> https://doc.dpdk.org/guides/platform/mlx5.html#design
>>>
>>>
>>>
>>>

[-- Attachment #2: Type: text/html, Size: 8649 bytes --]

Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[Thomas Monjalon]

14/11/2024 17:10, CJ Sculti:
> I figured out the initial issue. For some reason, having both devices in a
> bond on the kernel results in only 1 of the two ports being exposed as
> 'verb' devices. Previously, ibv_devinfo returned only one port. After
> removing both from the bond, ibv_devinfo returns both ports, and the DPDK
> application successfully takes both in. I'm still having some weird
> behavior trying to create a bypass interface with these ports though. I'm
> using the same code that I've been using on my Intel NICs with igb_uio for
> years, but seeing weird behavior. The ports are connected to our 40Gbps
> Ethernet switch, and set to link_layer: Ethernet.

You should be able to make it work with kernel bonding,
but I'm not sure what's wrong to do that.
And it looks not a priority for you. Let's focus on the other parts.


> The first thing I noticed is that rte_eth_dev_reset() fails on these
> interfaces with "ENOTSUP: hardware doesn't support reset".

You don't need the reset procedure with mlx5,
so you can make this code optional.


> Secondly, when checking ptypes, I noticed my code says these NICs are
> unable to support any sort of packet detection capability (code below, all
> return false). The MLX5 docs do say that all of these ptypes used here are
> supported by MLX5.

The supported ptypes can be checked in mlx5_dev_supported_ptypes_get() code.
I don't understand why it does not work for you.


> I'm just picking up a project that was left off by an older dev. It hasn't
> been touched in years, but has been working fine with our Intel NICs. All
> I'm trying to do is update DPDK (which is done, updated from dpdk 19.05 to
> DPDK 22.11, latest version with KNI support),

You don't need KNI with mlx5.
That's a big benefit of mlx5 design, it is natively bifurcated:
	https://doc.dpdk.org/guides/howto/flow_bifurcation.html


> and get it to work with our Mellanox CX5 NICs.
> This is my first time working with DPDK and I'm not very
> familiar. Should I expect to be able to do this without having to make a
> ton of code changes, or is this going to be an uphill battle for me? If
> it's the latter, I will likely just go purchase Intel NICs and give up on
> this.

The NICs have difference that DPDK is trying to hide.
If something is not compatible you may consider it as a bug or a limitation.
I think you should try a bit more, we are here to help.

Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[Yasuhiro Ohara]

> I think you should try a bit more, we are here to help.

I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK,
and I think you can make it work in only a few more steps.

I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP
might have made the whole NIC functions stopped.
IIRC the ptype was working fine on CX5 in my app too.
Can you comment out the rte_eth_dev_reset() ?
(I think it's worth to try.)

I don't think it is so uphill, but I don't disagree with you about
purchasing another
Intel 40G NICs. It'll also work perfectly fine.

2024年11月15日(金) 4:16 Thomas Monjalon <thomas@monjalon.net>:
>
> 14/11/2024 17:10, CJ Sculti:
> > I figured out the initial issue. For some reason, having both devices in a
> > bond on the kernel results in only 1 of the two ports being exposed as
> > 'verb' devices. Previously, ibv_devinfo returned only one port. After
> > removing both from the bond, ibv_devinfo returns both ports, and the DPDK
> > application successfully takes both in. I'm still having some weird
> > behavior trying to create a bypass interface with these ports though. I'm
> > using the same code that I've been using on my Intel NICs with igb_uio for
> > years, but seeing weird behavior. The ports are connected to our 40Gbps
> > Ethernet switch, and set to link_layer: Ethernet.
>
> You should be able to make it work with kernel bonding,
> but I'm not sure what's wrong to do that.
> And it looks not a priority for you. Let's focus on the other parts.
>
>
> > The first thing I noticed is that rte_eth_dev_reset() fails on these
> > interfaces with "ENOTSUP: hardware doesn't support reset".
>
> You don't need the reset procedure with mlx5,
> so you can make this code optional.
>
>
> > Secondly, when checking ptypes, I noticed my code says these NICs are
> > unable to support any sort of packet detection capability (code below, all
> > return false). The MLX5 docs do say that all of these ptypes used here are
> > supported by MLX5.
>
> The supported ptypes can be checked in mlx5_dev_supported_ptypes_get() code.
> I don't understand why it does not work for you.
>
>
> > I'm just picking up a project that was left off by an older dev. It hasn't
> > been touched in years, but has been working fine with our Intel NICs. All
> > I'm trying to do is update DPDK (which is done, updated from dpdk 19.05 to
> > DPDK 22.11, latest version with KNI support),
>
> You don't need KNI with mlx5.
> That's a big benefit of mlx5 design, it is natively bifurcated:
>         https://doc.dpdk.org/guides/howto/flow_bifurcation.html
>
>
> > and get it to work with our Mellanox CX5 NICs.
> > This is my first time working with DPDK and I'm not very
> > familiar. Should I expect to be able to do this without having to make a
> > ton of code changes, or is this going to be an uphill battle for me? If
> > it's the latter, I will likely just go purchase Intel NICs and give up on
> > this.
>
> The NICs have difference that DPDK is trying to hide.
> If something is not compatible you may consider it as a bug or a limitation.
> I think you should try a bit more, we are here to help.
>
>

Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[CJ Sculti]

[-- Attachment #1: Type: text/plain, Size: 5446 bytes --]

Thank you for the advice on that. I have removed the reset, and I also
fixed the rte_eth_dev_get_supported_ptypes(). I moved
my rte_eth_dev_get_supported_ptypes() calls after the bypass interface was
configured, and now they're returning expected values.

My next issue I'm running into is the bonding issue that I talked about in
my initial post. It seems that when part of a kernel bond, the mlx5_core
driver combines both ports into a single 'verbs' device named
'mlx5_bond_0'.  On my old setup with igb_uio, it worked like this:

- At Linux boot, the 2x Intel ports were configured as a bond on the kernel
via /etc/network/interfaces file.
- Before starting DPDK software, both ports bound to igb_uio driver.
- Software started, ports are setup by my software as kernel bypass.
- Enslaved the 2x 'new' bypass interfaces onto the same bond.
- Software reinjected LACP packets into kernel, to let kernel handle LACP
protocol.

This behavior where both ports are 'combined' into a single verbs device is
strange to me. How should I handle this? Is there any way to disable it and
just have the 2 ports be separate interfaces?

   1.
   2.

   root@DDoSMitigation:~/anubis/engine/bin# ibv_devinfo
   hca_id:    mlx5_bond_0
       transport:            InfiniBand (0)
       fw_ver:                16.35.4030
       node_guid:            506b:4b03:00b6:76ec
       sys_image_guid:            506b:4b03:00b6:76ec
       vendor_id:            0x02c9
       vendor_part_id:            4119
       hw_ver:                0x0
       board_id:            MT_0000000090
       phys_port_cnt:            1
           port:    1
               state:            PORT_ACTIVE (4)
               max_mtu:        4096 (5)
               active_mtu:        1024 (3)
               sm_lid:            0
               port_lid:        0
               port_lmc:        0x00
               link_layer:        Ethernet

   3. [12:58 PM]



On Fri, Nov 15, 2024 at 5:12 AM Yasuhiro Ohara <yasu1976@gmail.com> wrote:

> > I think you should try a bit more, we are here to help.
>
> I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK,
> and I think you can make it work in only a few more steps.
>
> I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP
> might have made the whole NIC functions stopped.
> IIRC the ptype was working fine on CX5 in my app too.
> Can you comment out the rte_eth_dev_reset() ?
> (I think it's worth to try.)
>
> I don't think it is so uphill, but I don't disagree with you about
> purchasing another
> Intel 40G NICs. It'll also work perfectly fine.
>
> 2024年11月15日(金) 4:16 Thomas Monjalon <thomas@monjalon.net>:
> >
> > 14/11/2024 17:10, CJ Sculti:
> > > I figured out the initial issue. For some reason, having both devices
> in a
> > > bond on the kernel results in only 1 of the two ports being exposed as
> > > 'verb' devices. Previously, ibv_devinfo returned only one port. After
> > > removing both from the bond, ibv_devinfo returns both ports, and the
> DPDK
> > > application successfully takes both in. I'm still having some weird
> > > behavior trying to create a bypass interface with these ports though.
> I'm
> > > using the same code that I've been using on my Intel NICs with igb_uio
> for
> > > years, but seeing weird behavior. The ports are connected to our 40Gbps
> > > Ethernet switch, and set to link_layer: Ethernet.
> >
> > You should be able to make it work with kernel bonding,
> > but I'm not sure what's wrong to do that.
> > And it looks not a priority for you. Let's focus on the other parts.
> >
> >
> > > The first thing I noticed is that rte_eth_dev_reset() fails on these
> > > interfaces with "ENOTSUP: hardware doesn't support reset".
> >
> > You don't need the reset procedure with mlx5,
> > so you can make this code optional.
> >
> >
> > > Secondly, when checking ptypes, I noticed my code says these NICs are
> > > unable to support any sort of packet detection capability (code below,
> all
> > > return false). The MLX5 docs do say that all of these ptypes used here
> are
> > > supported by MLX5.
> >
> > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get()
> code.
> > I don't understand why it does not work for you.
> >
> >
> > > I'm just picking up a project that was left off by an older dev. It
> hasn't
> > > been touched in years, but has been working fine with our Intel NICs.
> All
> > > I'm trying to do is update DPDK (which is done, updated from dpdk
> 19.05 to
> > > DPDK 22.11, latest version with KNI support),
> >
> > You don't need KNI with mlx5.
> > That's a big benefit of mlx5 design, it is natively bifurcated:
> >         https://doc.dpdk.org/guides/howto/flow_bifurcation.html
> >
> >
> > > and get it to work with our Mellanox CX5 NICs.
> > > This is my first time working with DPDK and I'm not very
> > > familiar. Should I expect to be able to do this without having to make
> a
> > > ton of code changes, or is this going to be an uphill battle for me? If
> > > it's the latter, I will likely just go purchase Intel NICs and give up
> on
> > > this.
> >
> > The NICs have difference that DPDK is trying to hide.
> > If something is not compatible you may consider it as a bug or a
> limitation.
> > I think you should try a bit more, we are here to help.
> >
> >
>

[-- Attachment #2: Type: text/html, Size: 16359 bytes --]

Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth?

[Thomas Monjalon]

I'm not sure to understand your need.
You want ports being bonded in the kernel but use them separately in userspace?

About "reinjecting" in the kernel, you don't need to do that with mlx5,
as I said before, you can enjoy the bifurcated model to make some flows
being processed in the kernel directly: you choose which flows go to the kernel or to userspace.

Also, you can find this in mlx5 doc:
"
- ``lacp_by_user`` parameter [int]

  A nonzero value enables the control of LACP traffic by the user application.
  When a bond exists in the driver, by default it should be managed by the
  kernel and therefore LACP traffic should be steered to the kernel.
  If this devarg is set to 1 it will allow the user to manage the bond by
  itself and not steer LACP traffic to the kernel.

  Disabled by default (set to 0).
"

18/11/2024 18:33, CJ Sculti:
> Thank you for the advice on that. I have removed the reset, and I also
> fixed the rte_eth_dev_get_supported_ptypes(). I moved
> my rte_eth_dev_get_supported_ptypes() calls after the bypass interface was
> configured, and now they're returning expected values.
> 
> My next issue I'm running into is the bonding issue that I talked about in
> my initial post. It seems that when part of a kernel bond, the mlx5_core
> driver combines both ports into a single 'verbs' device named
> 'mlx5_bond_0'.  On my old setup with igb_uio, it worked like this:
> 
> - At Linux boot, the 2x Intel ports were configured as a bond on the kernel
> via /etc/network/interfaces file.
> - Before starting DPDK software, both ports bound to igb_uio driver.
> - Software started, ports are setup by my software as kernel bypass.
> - Enslaved the 2x 'new' bypass interfaces onto the same bond.
> - Software reinjected LACP packets into kernel, to let kernel handle LACP
> protocol.
> 
> This behavior where both ports are 'combined' into a single verbs device is
> strange to me. How should I handle this? Is there any way to disable it and
> just have the 2 ports be separate interfaces?
> 
>    1.
>    2.
> 
>    root@DDoSMitigation:~/anubis/engine/bin# ibv_devinfo
>    hca_id:    mlx5_bond_0
>        transport:            InfiniBand (0)
>        fw_ver:                16.35.4030
>        node_guid:            506b:4b03:00b6:76ec
>        sys_image_guid:            506b:4b03:00b6:76ec
>        vendor_id:            0x02c9
>        vendor_part_id:            4119
>        hw_ver:                0x0
>        board_id:            MT_0000000090
>        phys_port_cnt:            1
>            port:    1
>                state:            PORT_ACTIVE (4)
>                max_mtu:        4096 (5)
>                active_mtu:        1024 (3)
>                sm_lid:            0
>                port_lid:        0
>                port_lmc:        0x00
>                link_layer:        Ethernet
> 
>    3. [12:58 PM]
> 
> 
> 
> On Fri, Nov 15, 2024 at 5:12 AM Yasuhiro Ohara <yasu1976@gmail.com> wrote:
> 
> > > I think you should try a bit more, we are here to help.
> >
> > I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK,
> > and I think you can make it work in only a few more steps.
> >
> > I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP
> > might have made the whole NIC functions stopped.
> > IIRC the ptype was working fine on CX5 in my app too.
> > Can you comment out the rte_eth_dev_reset() ?
> > (I think it's worth to try.)
> >
> > I don't think it is so uphill, but I don't disagree with you about
> > purchasing another
> > Intel 40G NICs. It'll also work perfectly fine.
> >
> > 2024年11月15日(金) 4:16 Thomas Monjalon <thomas@monjalon.net>:
> > >
> > > 14/11/2024 17:10, CJ Sculti:
> > > > I figured out the initial issue. For some reason, having both devices
> > in a
> > > > bond on the kernel results in only 1 of the two ports being exposed as
> > > > 'verb' devices. Previously, ibv_devinfo returned only one port. After
> > > > removing both from the bond, ibv_devinfo returns both ports, and the
> > DPDK
> > > > application successfully takes both in. I'm still having some weird
> > > > behavior trying to create a bypass interface with these ports though.
> > I'm
> > > > using the same code that I've been using on my Intel NICs with igb_uio
> > for
> > > > years, but seeing weird behavior. The ports are connected to our 40Gbps
> > > > Ethernet switch, and set to link_layer: Ethernet.
> > >
> > > You should be able to make it work with kernel bonding,
> > > but I'm not sure what's wrong to do that.
> > > And it looks not a priority for you. Let's focus on the other parts.
> > >
> > >
> > > > The first thing I noticed is that rte_eth_dev_reset() fails on these
> > > > interfaces with "ENOTSUP: hardware doesn't support reset".
> > >
> > > You don't need the reset procedure with mlx5,
> > > so you can make this code optional.
> > >
> > >
> > > > Secondly, when checking ptypes, I noticed my code says these NICs are
> > > > unable to support any sort of packet detection capability (code below,
> > all
> > > > return false). The MLX5 docs do say that all of these ptypes used here
> > are
> > > > supported by MLX5.
> > >
> > > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get()
> > code.
> > > I don't understand why it does not work for you.
> > >
> > >
> > > > I'm just picking up a project that was left off by an older dev. It
> > hasn't
> > > > been touched in years, but has been working fine with our Intel NICs.
> > All
> > > > I'm trying to do is update DPDK (which is done, updated from dpdk
> > 19.05 to
> > > > DPDK 22.11, latest version with KNI support),
> > >
> > > You don't need KNI with mlx5.
> > > That's a big benefit of mlx5 design, it is natively bifurcated:
> > >         https://doc.dpdk.org/guides/howto/flow_bifurcation.html
> > >
> > >
> > > > and get it to work with our Mellanox CX5 NICs.
> > > > This is my first time working with DPDK and I'm not very
> > > > familiar. Should I expect to be able to do this without having to make
> > a
> > > > ton of code changes, or is this going to be an uphill battle for me? If
> > > > it's the latter, I will likely just go purchase Intel NICs and give up
> > on
> > > > this.
> > >
> > > The NICs have difference that DPDK is trying to hide.
> > > If something is not compatible you may consider it as a bug or a
> > limitation.
> > > I think you should try a bit more, we are here to help.
> > >
> > >
> >
>