* DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? @ 2024-11-13 20:10 CJ Sculti 2024-11-13 21:26 ` Thomas Monjalon 0 siblings, 1 reply; 10+ messages in thread From: CJ Sculti @ 2024-11-13 20:10 UTC (permalink / raw) To: users [-- Attachment #1: Type: text/plain, Size: 2004 bytes --] I've been running my application for years on igb_uio with Intel NICs. I recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated the DPDK version my application uses, and compiled with support for mlx5 PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode, not Infiniband mode. However, I'm getting complaints when I start my application about trying to load 'mlx5_eth'? Both are bound to mlx5_core driver at the moment. When I bind them to vfio-pci, or uio_pci_generic, my application fails to recognize them at all as valid DPDK devices. Anyone have any ideas? Also, strange that it only complains about one? I have them configured in a bond on the kernel, as my application requires that. Network devices using kernel driver =================================== 0000:2b:00.0 'MT27800 Family [ConnectX-5] 1017' if=enp43s0f0np0 drv=mlx5_core unused=vfio-pci 0000:2b:00.1 'MT27800 Family [ConnectX-5] 1017' if=enp43s0f1np1 drv=mlx5_core unused=vfio-pci root@DDoSMitigation:~/anubis/engine/bin# ./anubis-engine Electric Fence 2.2 Copyright (C) 1987-1999 Bruce Perens <bruce@perens.com> EAL: Detected CPU lcores: 12 EAL: Detected NUMA nodes: 1 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: VFIO support initialized EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:2b:00.0 (socket -1) EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:2b:00.1 (socket -1) mlx5_net: PF 0 doesn't have Verbs device matches PCI device 0000:2b:00.1, are kernel drivers loaded? mlx5_common: Failed to load driver mlx5_eth EAL: Requested device 0000:2b:00.1 cannot be used TELEMETRY: No legacy callbacks, legacy socket not created USER1: Anubis build master/. USER1: We will run on 12 logical cores. USER1: Enabled lcores not a power of 2! This could have performance issues. KNI: WARNING: KNI is deprecated and will be removed in DPDK 23.11 USER1: Failed to reset link fe0. [-- Attachment #2: Type: text/html, Size: 11554 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? 2024-11-13 20:10 DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? CJ Sculti @ 2024-11-13 21:26 ` Thomas Monjalon 2024-11-13 21:43 ` CJ Sculti 0 siblings, 1 reply; 10+ messages in thread From: Thomas Monjalon @ 2024-11-13 21:26 UTC (permalink / raw) To: CJ Sculti; +Cc: users, Dariusz Sosnowski 13/11/2024 21:10, CJ Sculti: > I've been running my application for years on igb_uio with Intel NICs. I > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated > the DPDK version my application uses, and compiled with support for mlx5 > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode, > not Infiniband mode. However, I'm getting complaints when I start my > application about trying to load 'mlx5_eth'? Both are bound to mlx5_core > driver at the moment. When I bind them to vfio-pci, or uio_pci_generic, my > application fails to recognize them at all as valid DPDK devices. Anyone > have any ideas? Also, strange that it only complains about one? I have them > configured in a bond on the kernel, as my application requires that. You must not bind mlx5 devices with VFIO. I recommend reading documentation. You can start here: https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver then https://doc.dpdk.org/guides/platform/mlx5.html#design ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? 2024-11-13 21:26 ` Thomas Monjalon @ 2024-11-13 21:43 ` CJ Sculti 2024-11-13 22:43 ` Thomas Monjalon 2024-11-14 2:10 ` Yasuhiro Ohara 0 siblings, 2 replies; 10+ messages in thread From: CJ Sculti @ 2024-11-13 21:43 UTC (permalink / raw) To: Thomas Monjalon; +Cc: users, Dariusz Sosnowski [-- Attachment #1: Type: text/plain, Size: 1337 bytes --] I'm not using vfio, I just bound interfaces on there one time to test. Shouldn't I be able to just use the default mlx5_core driver, without binding to uio_pci_generic? On Wed, Nov 13, 2024 at 4:26 PM Thomas Monjalon <thomas@monjalon.net> wrote: > 13/11/2024 21:10, CJ Sculti: > > I've been running my application for years on igb_uio with Intel NICs. I > > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated > > the DPDK version my application uses, and compiled with support for mlx5 > > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode, > > not Infiniband mode. However, I'm getting complaints when I start my > > application about trying to load 'mlx5_eth'? Both are bound to mlx5_core > > driver at the moment. When I bind them to vfio-pci, or uio_pci_generic, > my > > application fails to recognize them at all as valid DPDK devices. Anyone > > have any ideas? Also, strange that it only complains about one? I have > them > > configured in a bond on the kernel, as my application requires that. > > You must not bind mlx5 devices with VFIO. > I recommend reading documentation. > You can start here: > https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver > then > https://doc.dpdk.org/guides/platform/mlx5.html#design > > > > [-- Attachment #2: Type: text/html, Size: 2098 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? 2024-11-13 21:43 ` CJ Sculti @ 2024-11-13 22:43 ` Thomas Monjalon 2024-11-14 2:10 ` Yasuhiro Ohara 1 sibling, 0 replies; 10+ messages in thread From: Thomas Monjalon @ 2024-11-13 22:43 UTC (permalink / raw) To: CJ Sculti; +Cc: users, Dariusz Sosnowski 13/11/2024 22:43, CJ Sculti: > I'm not using vfio, I just bound interfaces on there one time to test. > Shouldn't I be able to just use the default mlx5_core driver, without > binding to uio_pci_generic? Yes you must not bind to anything else than mlx5 kernel driver. Please try to follow the documentation and tell us what you find. > On Wed, Nov 13, 2024 at 4:26 PM Thomas Monjalon <thomas@monjalon.net> wrote: > > > 13/11/2024 21:10, CJ Sculti: > > > I've been running my application for years on igb_uio with Intel NICs. I > > > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated > > > the DPDK version my application uses, and compiled with support for mlx5 > > > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode, > > > not Infiniband mode. However, I'm getting complaints when I start my > > > application about trying to load 'mlx5_eth'? Both are bound to mlx5_core > > > driver at the moment. When I bind them to vfio-pci, or uio_pci_generic, > > my > > > application fails to recognize them at all as valid DPDK devices. Anyone > > > have any ideas? Also, strange that it only complains about one? I have > > them > > > configured in a bond on the kernel, as my application requires that. > > > > You must not bind mlx5 devices with VFIO. > > I recommend reading documentation. > > You can start here: > > https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver > > then > > https://doc.dpdk.org/guides/platform/mlx5.html#design ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? 2024-11-13 21:43 ` CJ Sculti 2024-11-13 22:43 ` Thomas Monjalon @ 2024-11-14 2:10 ` Yasuhiro Ohara 2024-11-14 16:10 ` CJ Sculti 1 sibling, 1 reply; 10+ messages in thread From: Yasuhiro Ohara @ 2024-11-14 2:10 UTC (permalink / raw) To: CJ Sculti; +Cc: Thomas Monjalon, users, Dariusz Sosnowski [-- Attachment #1: Type: text/plain, Size: 1713 bytes --] I would suggest re-installation of MELLANOX OFED, and/or upgrading NIC firmware (can be done using OFED tools). Yes, warning on only 1 port is odd. I suspect some kind of mismatch, like older version of the NIC had only 1 port versions and the soft assumed it, for example. 2024年11月14日(木) 5:43 CJ Sculti <cj@cj.gy>: > I'm not using vfio, I just bound interfaces on there one time to test. > Shouldn't I be able to just use the default mlx5_core driver, without > binding to uio_pci_generic? > > > On Wed, Nov 13, 2024 at 4:26 PM Thomas Monjalon <thomas@monjalon.net> > wrote: > >> 13/11/2024 21:10, CJ Sculti: >> > I've been running my application for years on igb_uio with Intel NICs. I >> > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updated >> > the DPDK version my application uses, and compiled with support for mlx5 >> > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mode, >> > not Infiniband mode. However, I'm getting complaints when I start my >> > application about trying to load 'mlx5_eth'? Both are bound to mlx5_core >> > driver at the moment. When I bind them to vfio-pci, or uio_pci_generic, >> my >> > application fails to recognize them at all as valid DPDK devices. Anyone >> > have any ideas? Also, strange that it only complains about one? I have >> them >> > configured in a bond on the kernel, as my application requires that. >> >> You must not bind mlx5 devices with VFIO. >> I recommend reading documentation. >> You can start here: >> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver >> then >> https://doc.dpdk.org/guides/platform/mlx5.html#design >> >> >> >> [-- Attachment #2: Type: text/html, Size: 2815 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? 2024-11-14 2:10 ` Yasuhiro Ohara @ 2024-11-14 16:10 ` CJ Sculti 2024-11-14 20:16 ` Thomas Monjalon 0 siblings, 1 reply; 10+ messages in thread From: CJ Sculti @ 2024-11-14 16:10 UTC (permalink / raw) To: Yasuhiro Ohara; +Cc: Thomas Monjalon, users, Dariusz Sosnowski [-- Attachment #1: Type: text/plain, Size: 6603 bytes --] I figured out the initial issue. For some reason, having both devices in a bond on the kernel results in only 1 of the two ports being exposed as 'verb' devices. Previously, ibv_devinfo returned only one port. After removing both from the bond, ibv_devinfo returns both ports, and the DPDK application successfully takes both in. I'm still having some weird behavior trying to create a bypass interface with these ports though. I'm using the same code that I've been using on my Intel NICs with igb_uio for years, but seeing weird behavior. The ports are connected to our 40Gbps Ethernet switch, and set to link_layer: Ethernet. The first thing I noticed is that rte_eth_dev_reset() fails on these interfaces with "ENOTSUP: hardware doesn't support reset". Secondly, when checking ptypes, I noticed my code says these NICs are unable to support any sort of packet detection capability (code below, all return false). The MLX5 docs do say that all of these ptypes used here are supported by MLX5. I'm just picking up a project that was left off by an older dev. It hasn't been touched in years, but has been working fine with our Intel NICs. All I'm trying to do is update DPDK (which is done, updated from dpdk 19.05 to DPDK 22.11, latest version with KNI support), and get it to work with our Mellanox CX5 NICs. This is my first time working with DPDK and I'm not very familiar. Should I expect to be able to do this without having to make a ton of code changes, or is this going to be an uphill battle for me? If it's the latter, I will likely just go purchase Intel NICs and give up on this. bool Manager::HasPacketTypeDetectionCapabilities(uint16_t port_id) { // Detect L2 packet type detection capability int num_ptypes = rte_eth_dev_get_supported_ptypes( port_id, RTE_PTYPE_ALL_MASK, nullptr, 0); if (num_ptypes == -1) { RTE_LOG(WARNING, USER1, "Driver does not report supported ptypes; enabling software parsing.\n"); // Proceed with software-based packet type parsing return true; } bool detect_ether = false, detect_arp = false; auto num_l2_dtx = rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L2_MASK, nullptr, 0); if (num_l2_dtx == -1) { RTE_LOG(ERR, USER1, "Failed to detect L2 detection capabilities.\n"); return false; } uint32_t l2_ptypes[num_l2_dtx]; num_l2_dtx = rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L2_MASK, l2_ptypes, num_l2_dtx); for (auto i = 0; i < num_l2_dtx; ++i) { if (l2_ptypes[i] & RTE_PTYPE_L2_ETHER) detect_ether = true; } // Detect L3 packet type detection capability bool detect_ipv4 = false, detect_ipv6 = false; auto num_l3_dtx = rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L3_MASK, nullptr, 0); if (num_l3_dtx == -1) { RTE_LOG(ERR, USER1, "Failed to detect L3 detection capabilities.\n"); return false; } uint32_t l3_ptypes[num_l3_dtx]; num_l3_dtx = rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L3_MASK, l3_ptypes, num_l3_dtx); for (auto i = 0; i < num_l3_dtx; ++i) { if (l3_ptypes[i] & RTE_PTYPE_L3_IPV4) detect_ipv4 = true; if (l3_ptypes[i] & RTE_PTYPE_L3_IPV6) detect_ipv6 = true; } // Detect L4 packet type detection capability bool detect_tcp = false, detect_udp = false, detect_icmp = false; auto num_l4_dtx = rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L4_MASK, nullptr, 0); if (num_l4_dtx == -1) { RTE_LOG(ERR, USER1, "Failed to detect L4 detection capabilities.\n"); return false; } uint32_t l4_ptypes[num_l4_dtx]; num_l4_dtx = rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L4_MASK, l4_ptypes, num_l4_dtx); for (auto i = 0; i < num_l4_dtx; ++i) { if (l4_ptypes[i] & RTE_PTYPE_L4_TCP) detect_tcp = true; if (l4_ptypes[i] & RTE_PTYPE_L4_UDP) detect_udp = true; if (l4_ptypes[i] & RTE_PTYPE_L4_ICMP) detect_icmp = true; } if (!detect_ether || !detect_arp || !detect_ipv4 || !detect_ipv6 || !detect_tcp || !detect_udp || !detect_icmp) { RTE_LOG(ERR, USER1, "Supported Detection Modes:\n" "L2 Ether: %s\n" "L2 Arp : %s\n" "L3 IPv4 : %s\n" "L3 IPv6 : %s\n" "L4 TCP : %s\n" "L4 UDP : %s\n" "L4 ICMP : %s\n", detect_ether ? "True" : "False", detect_arp ? "True": "False", detect_ipv4 ? "True": "False", detect_ipv6 ? "True": "False", detect_tcp ? "True" : "False", detect_udp ? "True" : "False", detect_icmp ? "True" : "False"); return false; } return true; } On Wed, Nov 13, 2024 at 9:10 PM Yasuhiro Ohara <yasu1976@gmail.com> wrote: > I would suggest re-installation of MELLANOX OFED, and/or upgrading NIC > firmware (can be done using OFED tools). > > Yes, warning on only 1 port is odd. I suspect some kind of mismatch, like > older version of the NIC had only 1 port versions and the soft assumed it, > for example. > > > 2024年11月14日(木) 5:43 CJ Sculti <cj@cj.gy>: > >> I'm not using vfio, I just bound interfaces on there one time to test. >> Shouldn't I be able to just use the default mlx5_core driver, without >> binding to uio_pci_generic? >> >> >> On Wed, Nov 13, 2024 at 4:26 PM Thomas Monjalon <thomas@monjalon.net> >> wrote: >> >>> 13/11/2024 21:10, CJ Sculti: >>> > I've been running my application for years on igb_uio with Intel NICs. >>> I >>> > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, >>> updated >>> > the DPDK version my application uses, and compiled with support for >>> mlx5 >>> > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet >>> mode, >>> > not Infiniband mode. However, I'm getting complaints when I start my >>> > application about trying to load 'mlx5_eth'? Both are bound to >>> mlx5_core >>> > driver at the moment. When I bind them to vfio-pci, or >>> uio_pci_generic, my >>> > application fails to recognize them at all as valid DPDK devices. >>> Anyone >>> > have any ideas? Also, strange that it only complains about one? I have >>> them >>> > configured in a bond on the kernel, as my application requires that. >>> >>> You must not bind mlx5 devices with VFIO. >>> I recommend reading documentation. >>> You can start here: >>> >>> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-driver >>> then >>> https://doc.dpdk.org/guides/platform/mlx5.html#design >>> >>> >>> >>> [-- Attachment #2: Type: text/html, Size: 8649 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? 2024-11-14 16:10 ` CJ Sculti @ 2024-11-14 20:16 ` Thomas Monjalon 2024-11-15 10:11 ` Yasuhiro Ohara 0 siblings, 1 reply; 10+ messages in thread From: Thomas Monjalon @ 2024-11-14 20:16 UTC (permalink / raw) To: CJ Sculti; +Cc: Yasuhiro Ohara, users, Dariusz Sosnowski 14/11/2024 17:10, CJ Sculti: > I figured out the initial issue. For some reason, having both devices in a > bond on the kernel results in only 1 of the two ports being exposed as > 'verb' devices. Previously, ibv_devinfo returned only one port. After > removing both from the bond, ibv_devinfo returns both ports, and the DPDK > application successfully takes both in. I'm still having some weird > behavior trying to create a bypass interface with these ports though. I'm > using the same code that I've been using on my Intel NICs with igb_uio for > years, but seeing weird behavior. The ports are connected to our 40Gbps > Ethernet switch, and set to link_layer: Ethernet. You should be able to make it work with kernel bonding, but I'm not sure what's wrong to do that. And it looks not a priority for you. Let's focus on the other parts. > The first thing I noticed is that rte_eth_dev_reset() fails on these > interfaces with "ENOTSUP: hardware doesn't support reset". You don't need the reset procedure with mlx5, so you can make this code optional. > Secondly, when checking ptypes, I noticed my code says these NICs are > unable to support any sort of packet detection capability (code below, all > return false). The MLX5 docs do say that all of these ptypes used here are > supported by MLX5. The supported ptypes can be checked in mlx5_dev_supported_ptypes_get() code. I don't understand why it does not work for you. > I'm just picking up a project that was left off by an older dev. It hasn't > been touched in years, but has been working fine with our Intel NICs. All > I'm trying to do is update DPDK (which is done, updated from dpdk 19.05 to > DPDK 22.11, latest version with KNI support), You don't need KNI with mlx5. That's a big benefit of mlx5 design, it is natively bifurcated: https://doc.dpdk.org/guides/howto/flow_bifurcation.html > and get it to work with our Mellanox CX5 NICs. > This is my first time working with DPDK and I'm not very > familiar. Should I expect to be able to do this without having to make a > ton of code changes, or is this going to be an uphill battle for me? If > it's the latter, I will likely just go purchase Intel NICs and give up on > this. The NICs have difference that DPDK is trying to hide. If something is not compatible you may consider it as a bug or a limitation. I think you should try a bit more, we are here to help. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? 2024-11-14 20:16 ` Thomas Monjalon @ 2024-11-15 10:11 ` Yasuhiro Ohara 2024-11-18 17:33 ` CJ Sculti 0 siblings, 1 reply; 10+ messages in thread From: Yasuhiro Ohara @ 2024-11-15 10:11 UTC (permalink / raw) To: Thomas Monjalon; +Cc: CJ Sculti, users, Dariusz Sosnowski > I think you should try a bit more, we are here to help. I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK, and I think you can make it work in only a few more steps. I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP might have made the whole NIC functions stopped. IIRC the ptype was working fine on CX5 in my app too. Can you comment out the rte_eth_dev_reset() ? (I think it's worth to try.) I don't think it is so uphill, but I don't disagree with you about purchasing another Intel 40G NICs. It'll also work perfectly fine. 2024年11月15日(金) 4:16 Thomas Monjalon <thomas@monjalon.net>: > > 14/11/2024 17:10, CJ Sculti: > > I figured out the initial issue. For some reason, having both devices in a > > bond on the kernel results in only 1 of the two ports being exposed as > > 'verb' devices. Previously, ibv_devinfo returned only one port. After > > removing both from the bond, ibv_devinfo returns both ports, and the DPDK > > application successfully takes both in. I'm still having some weird > > behavior trying to create a bypass interface with these ports though. I'm > > using the same code that I've been using on my Intel NICs with igb_uio for > > years, but seeing weird behavior. The ports are connected to our 40Gbps > > Ethernet switch, and set to link_layer: Ethernet. > > You should be able to make it work with kernel bonding, > but I'm not sure what's wrong to do that. > And it looks not a priority for you. Let's focus on the other parts. > > > > The first thing I noticed is that rte_eth_dev_reset() fails on these > > interfaces with "ENOTSUP: hardware doesn't support reset". > > You don't need the reset procedure with mlx5, > so you can make this code optional. > > > > Secondly, when checking ptypes, I noticed my code says these NICs are > > unable to support any sort of packet detection capability (code below, all > > return false). The MLX5 docs do say that all of these ptypes used here are > > supported by MLX5. > > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get() code. > I don't understand why it does not work for you. > > > > I'm just picking up a project that was left off by an older dev. It hasn't > > been touched in years, but has been working fine with our Intel NICs. All > > I'm trying to do is update DPDK (which is done, updated from dpdk 19.05 to > > DPDK 22.11, latest version with KNI support), > > You don't need KNI with mlx5. > That's a big benefit of mlx5 design, it is natively bifurcated: > https://doc.dpdk.org/guides/howto/flow_bifurcation.html > > > > and get it to work with our Mellanox CX5 NICs. > > This is my first time working with DPDK and I'm not very > > familiar. Should I expect to be able to do this without having to make a > > ton of code changes, or is this going to be an uphill battle for me? If > > it's the latter, I will likely just go purchase Intel NICs and give up on > > this. > > The NICs have difference that DPDK is trying to hide. > If something is not compatible you may consider it as a bug or a limitation. > I think you should try a bit more, we are here to help. > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? 2024-11-15 10:11 ` Yasuhiro Ohara @ 2024-11-18 17:33 ` CJ Sculti 2024-11-18 17:46 ` Thomas Monjalon 0 siblings, 1 reply; 10+ messages in thread From: CJ Sculti @ 2024-11-18 17:33 UTC (permalink / raw) To: Yasuhiro Ohara; +Cc: Thomas Monjalon, users, Dariusz Sosnowski [-- Attachment #1: Type: text/plain, Size: 5446 bytes --] Thank you for the advice on that. I have removed the reset, and I also fixed the rte_eth_dev_get_supported_ptypes(). I moved my rte_eth_dev_get_supported_ptypes() calls after the bypass interface was configured, and now they're returning expected values. My next issue I'm running into is the bonding issue that I talked about in my initial post. It seems that when part of a kernel bond, the mlx5_core driver combines both ports into a single 'verbs' device named 'mlx5_bond_0'. On my old setup with igb_uio, it worked like this: - At Linux boot, the 2x Intel ports were configured as a bond on the kernel via /etc/network/interfaces file. - Before starting DPDK software, both ports bound to igb_uio driver. - Software started, ports are setup by my software as kernel bypass. - Enslaved the 2x 'new' bypass interfaces onto the same bond. - Software reinjected LACP packets into kernel, to let kernel handle LACP protocol. This behavior where both ports are 'combined' into a single verbs device is strange to me. How should I handle this? Is there any way to disable it and just have the 2 ports be separate interfaces? 1. 2. root@DDoSMitigation:~/anubis/engine/bin# ibv_devinfo hca_id: mlx5_bond_0 transport: InfiniBand (0) fw_ver: 16.35.4030 node_guid: 506b:4b03:00b6:76ec sys_image_guid: 506b:4b03:00b6:76ec vendor_id: 0x02c9 vendor_part_id: 4119 hw_ver: 0x0 board_id: MT_0000000090 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet 3. [12:58 PM] On Fri, Nov 15, 2024 at 5:12 AM Yasuhiro Ohara <yasu1976@gmail.com> wrote: > > I think you should try a bit more, we are here to help. > > I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK, > and I think you can make it work in only a few more steps. > > I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP > might have made the whole NIC functions stopped. > IIRC the ptype was working fine on CX5 in my app too. > Can you comment out the rte_eth_dev_reset() ? > (I think it's worth to try.) > > I don't think it is so uphill, but I don't disagree with you about > purchasing another > Intel 40G NICs. It'll also work perfectly fine. > > 2024年11月15日(金) 4:16 Thomas Monjalon <thomas@monjalon.net>: > > > > 14/11/2024 17:10, CJ Sculti: > > > I figured out the initial issue. For some reason, having both devices > in a > > > bond on the kernel results in only 1 of the two ports being exposed as > > > 'verb' devices. Previously, ibv_devinfo returned only one port. After > > > removing both from the bond, ibv_devinfo returns both ports, and the > DPDK > > > application successfully takes both in. I'm still having some weird > > > behavior trying to create a bypass interface with these ports though. > I'm > > > using the same code that I've been using on my Intel NICs with igb_uio > for > > > years, but seeing weird behavior. The ports are connected to our 40Gbps > > > Ethernet switch, and set to link_layer: Ethernet. > > > > You should be able to make it work with kernel bonding, > > but I'm not sure what's wrong to do that. > > And it looks not a priority for you. Let's focus on the other parts. > > > > > > > The first thing I noticed is that rte_eth_dev_reset() fails on these > > > interfaces with "ENOTSUP: hardware doesn't support reset". > > > > You don't need the reset procedure with mlx5, > > so you can make this code optional. > > > > > > > Secondly, when checking ptypes, I noticed my code says these NICs are > > > unable to support any sort of packet detection capability (code below, > all > > > return false). The MLX5 docs do say that all of these ptypes used here > are > > > supported by MLX5. > > > > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get() > code. > > I don't understand why it does not work for you. > > > > > > > I'm just picking up a project that was left off by an older dev. It > hasn't > > > been touched in years, but has been working fine with our Intel NICs. > All > > > I'm trying to do is update DPDK (which is done, updated from dpdk > 19.05 to > > > DPDK 22.11, latest version with KNI support), > > > > You don't need KNI with mlx5. > > That's a big benefit of mlx5 design, it is natively bifurcated: > > https://doc.dpdk.org/guides/howto/flow_bifurcation.html > > > > > > > and get it to work with our Mellanox CX5 NICs. > > > This is my first time working with DPDK and I'm not very > > > familiar. Should I expect to be able to do this without having to make > a > > > ton of code changes, or is this going to be an uphill battle for me? If > > > it's the latter, I will likely just go purchase Intel NICs and give up > on > > > this. > > > > The NICs have difference that DPDK is trying to hide. > > If something is not compatible you may consider it as a bug or a > limitation. > > I think you should try a bit more, we are here to help. > > > > > [-- Attachment #2: Type: text/html, Size: 16359 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? 2024-11-18 17:33 ` CJ Sculti @ 2024-11-18 17:46 ` Thomas Monjalon 0 siblings, 0 replies; 10+ messages in thread From: Thomas Monjalon @ 2024-11-18 17:46 UTC (permalink / raw) To: CJ Sculti; +Cc: Yasuhiro Ohara, users, Dariusz Sosnowski I'm not sure to understand your need. You want ports being bonded in the kernel but use them separately in userspace? About "reinjecting" in the kernel, you don't need to do that with mlx5, as I said before, you can enjoy the bifurcated model to make some flows being processed in the kernel directly: you choose which flows go to the kernel or to userspace. Also, you can find this in mlx5 doc: " - ``lacp_by_user`` parameter [int] A nonzero value enables the control of LACP traffic by the user application. When a bond exists in the driver, by default it should be managed by the kernel and therefore LACP traffic should be steered to the kernel. If this devarg is set to 1 it will allow the user to manage the bond by itself and not steer LACP traffic to the kernel. Disabled by default (set to 0). " 18/11/2024 18:33, CJ Sculti: > Thank you for the advice on that. I have removed the reset, and I also > fixed the rte_eth_dev_get_supported_ptypes(). I moved > my rte_eth_dev_get_supported_ptypes() calls after the bypass interface was > configured, and now they're returning expected values. > > My next issue I'm running into is the bonding issue that I talked about in > my initial post. It seems that when part of a kernel bond, the mlx5_core > driver combines both ports into a single 'verbs' device named > 'mlx5_bond_0'. On my old setup with igb_uio, it worked like this: > > - At Linux boot, the 2x Intel ports were configured as a bond on the kernel > via /etc/network/interfaces file. > - Before starting DPDK software, both ports bound to igb_uio driver. > - Software started, ports are setup by my software as kernel bypass. > - Enslaved the 2x 'new' bypass interfaces onto the same bond. > - Software reinjected LACP packets into kernel, to let kernel handle LACP > protocol. > > This behavior where both ports are 'combined' into a single verbs device is > strange to me. How should I handle this? Is there any way to disable it and > just have the 2 ports be separate interfaces? > > 1. > 2. > > root@DDoSMitigation:~/anubis/engine/bin# ibv_devinfo > hca_id: mlx5_bond_0 > transport: InfiniBand (0) > fw_ver: 16.35.4030 > node_guid: 506b:4b03:00b6:76ec > sys_image_guid: 506b:4b03:00b6:76ec > vendor_id: 0x02c9 > vendor_part_id: 4119 > hw_ver: 0x0 > board_id: MT_0000000090 > phys_port_cnt: 1 > port: 1 > state: PORT_ACTIVE (4) > max_mtu: 4096 (5) > active_mtu: 1024 (3) > sm_lid: 0 > port_lid: 0 > port_lmc: 0x00 > link_layer: Ethernet > > 3. [12:58 PM] > > > > On Fri, Nov 15, 2024 at 5:12 AM Yasuhiro Ohara <yasu1976@gmail.com> wrote: > > > > I think you should try a bit more, we are here to help. > > > > I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK, > > and I think you can make it work in only a few more steps. > > > > I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP > > might have made the whole NIC functions stopped. > > IIRC the ptype was working fine on CX5 in my app too. > > Can you comment out the rte_eth_dev_reset() ? > > (I think it's worth to try.) > > > > I don't think it is so uphill, but I don't disagree with you about > > purchasing another > > Intel 40G NICs. It'll also work perfectly fine. > > > > 2024年11月15日(金) 4:16 Thomas Monjalon <thomas@monjalon.net>: > > > > > > 14/11/2024 17:10, CJ Sculti: > > > > I figured out the initial issue. For some reason, having both devices > > in a > > > > bond on the kernel results in only 1 of the two ports being exposed as > > > > 'verb' devices. Previously, ibv_devinfo returned only one port. After > > > > removing both from the bond, ibv_devinfo returns both ports, and the > > DPDK > > > > application successfully takes both in. I'm still having some weird > > > > behavior trying to create a bypass interface with these ports though. > > I'm > > > > using the same code that I've been using on my Intel NICs with igb_uio > > for > > > > years, but seeing weird behavior. The ports are connected to our 40Gbps > > > > Ethernet switch, and set to link_layer: Ethernet. > > > > > > You should be able to make it work with kernel bonding, > > > but I'm not sure what's wrong to do that. > > > And it looks not a priority for you. Let's focus on the other parts. > > > > > > > > > > The first thing I noticed is that rte_eth_dev_reset() fails on these > > > > interfaces with "ENOTSUP: hardware doesn't support reset". > > > > > > You don't need the reset procedure with mlx5, > > > so you can make this code optional. > > > > > > > > > > Secondly, when checking ptypes, I noticed my code says these NICs are > > > > unable to support any sort of packet detection capability (code below, > > all > > > > return false). The MLX5 docs do say that all of these ptypes used here > > are > > > > supported by MLX5. > > > > > > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get() > > code. > > > I don't understand why it does not work for you. > > > > > > > > > > I'm just picking up a project that was left off by an older dev. It > > hasn't > > > > been touched in years, but has been working fine with our Intel NICs. > > All > > > > I'm trying to do is update DPDK (which is done, updated from dpdk > > 19.05 to > > > > DPDK 22.11, latest version with KNI support), > > > > > > You don't need KNI with mlx5. > > > That's a big benefit of mlx5 design, it is natively bifurcated: > > > https://doc.dpdk.org/guides/howto/flow_bifurcation.html > > > > > > > > > > and get it to work with our Mellanox CX5 NICs. > > > > This is my first time working with DPDK and I'm not very > > > > familiar. Should I expect to be able to do this without having to make > > a > > > > ton of code changes, or is this going to be an uphill battle for me? If > > > > it's the latter, I will likely just go purchase Intel NICs and give up > > on > > > > this. > > > > > > The NICs have difference that DPDK is trying to hide. > > > If something is not compatible you may consider it as a bug or a > > limitation. > > > I think you should try a bit more, we are here to help. > > > > > > > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-11-18 17:46 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-11-13 20:10 DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? CJ Sculti 2024-11-13 21:26 ` Thomas Monjalon 2024-11-13 21:43 ` CJ Sculti 2024-11-13 22:43 ` Thomas Monjalon 2024-11-14 2:10 ` Yasuhiro Ohara 2024-11-14 16:10 ` CJ Sculti 2024-11-14 20:16 ` Thomas Monjalon 2024-11-15 10:11 ` Yasuhiro Ohara 2024-11-18 17:33 ` CJ Sculti 2024-11-18 17:46 ` Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).