From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5BEDE45D06 for ; Thu, 14 Nov 2024 17:11:10 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3DC514025D; Thu, 14 Nov 2024 17:11:10 +0100 (CET) Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) by mails.dpdk.org (Postfix) with ESMTP id A95284021F for ; Thu, 14 Nov 2024 17:11:08 +0100 (CET) Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-6ecaa45af7bso9251677b3.3 for ; Thu, 14 Nov 2024 08:11:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cj-gy.20230601.gappssmtp.com; s=20230601; t=1731600668; x=1732205468; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=31ggZNAo7a30iIAbbNgiUminJSEmnALd2jzpMhPjE4g=; b=Sm8VkHPI3NqetoUTsHRPANtDP97NEhL0QodPfQqcu07hPGGHYA6Eb0INUXeWkjYkto +14d1iHiyyUoPtKhtl88XDhJ7bk7opFehhpcyp/uu5ahGfSR4OQCd++v7nZA19EQtQIf qc8xjlopaEPjcNSSEp5K3i96RNig/UcUOPjSeFX0041Qk/slZLbWR6/+r6b9/6QC4qzG Rwn+RwvKIv+cXy1KkyM1vudOSyVkNhtuDHSfznPw+29tmYNiqy6osabWrLMprqryddLS LWAq06eTl5QyZkB5qeoN7rNSzLwxxGy/FxwDDXWMrpDNhKF09iy/Tz9pCI27Ph8EjVFk TduQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731600668; x=1732205468; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=31ggZNAo7a30iIAbbNgiUminJSEmnALd2jzpMhPjE4g=; b=cBHHyPVB9pQvDHm5B2W6XSWMjTmGB7TA2Jo54EFPMjq+4MQ1p/UDyqtQpBbaxRfrB1 lXvOPaOJhg05vljPw6w6453yhqoMcQ8cQroGW4CsUIeJc42N0IWi7zdIl8sMxjYFAIMN WxFvmkTfUHcEAdv2qQVpTHNPq2olwDsVgxDeh4Cu5kVwnpa0G2ICWtzrJKjJ2nKLO0s4 2xYwdrEUGGgixW59blDq6/bUNL+TWYJL+XNI1eaOX93DXpE86vn6sg0q8qhHrV+lxzuY TCBvU5rPn5nN64se8+oki/oOpq9wsAIp4oEgcJK4sMy7TmgE5jyVZPrCN7O31DiowXD+ 6akw== X-Forwarded-Encrypted: i=1; AJvYcCU7GbWow1k+lDoxUuSZ3mvk3wBWUS/OkTe29Zk+o2mG2C9srlj/o/eLs4UldGBP6WZz7IjALA==@dpdk.org X-Gm-Message-State: AOJu0Yy6af9/hxEvnoxvikMmw5Fuvorwki6UCxdUarhKnnkcq67ghfLQ sDzfEXzxMrLJDnZ5mp6TEy8m8+b1hNaWT2uotB3riilrv28y7m2CynbMdbkSSF7euu7kDnkEgCJ BLDn/vSL0WJhBtCckEwMyrWJxHTkGp8gAcsLFQg== X-Google-Smtp-Source: AGHT+IH/H4UaleaPNWbKDPA4n0TlCo5/fQGDRPjVAb4dcC3KDUpedP+RIpI67mmktXTlH0oqae0ti5aiax7y+rTqM4g= X-Received: by 2002:a81:f912:0:b0:6e2:413:f19 with SMTP id 00721157ae682-6ecb347fa3emr60809737b3.27.1731600667244; Thu, 14 Nov 2024 08:11:07 -0800 (PST) MIME-Version: 1.0 References: <9487809.CDJkKcVGEf@thomas> In-Reply-To: From: CJ Sculti Date: Thu, 14 Nov 2024 11:10:55 -0500 Message-ID: Subject: Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? To: Yasuhiro Ohara Cc: Thomas Monjalon , users@dpdk.org, Dariusz Sosnowski Content-Type: multipart/alternative; boundary="000000000000aa1c900626e1b242" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000aa1c900626e1b242 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I figured out the initial issue. For some reason, having both devices in a bond on the kernel results in only 1 of the two ports being exposed as 'verb' devices. Previously, ibv_devinfo returned only one port. After removing both from the bond, ibv_devinfo returns both ports, and the DPDK application successfully takes both in. I'm still having some weird behavior trying to create a bypass interface with these ports though. I'm using the same code that I've been using on my Intel NICs with igb_uio for years, but seeing weird behavior. The ports are connected to our 40Gbps Ethernet switch, and set to link_layer: Ethernet. The first thing I noticed is that rte_eth_dev_reset() fails on these interfaces with "ENOTSUP: hardware doesn't support reset". Secondly, when checking ptypes, I noticed my code says these NICs are unable to support any sort of packet detection capability (code below, all return false). The MLX5 docs do say that all of these ptypes used here are supported by MLX5. I'm just picking up a project that was left off by an older dev. It hasn't been touched in years, but has been working fine with our Intel NICs. All I'm trying to do is update DPDK (which is done, updated from dpdk 19.05 to DPDK 22.11, latest version with KNI support), and get it to work with our Mellanox CX5 NICs. This is my first time working with DPDK and I'm not very familiar. Should I expect to be able to do this without having to make a ton of code changes, or is this going to be an uphill battle for me? If it's the latter, I will likely just go purchase Intel NICs and give up on this. bool Manager::HasPacketTypeDetectionCapabilities(uint16_t port_id) { // Detect L2 packet type detection capability int num_ptypes =3D rte_eth_dev_get_supported_ptypes( port_id, RTE_PTYPE_ALL_MASK, nullptr, 0); if (num_ptypes =3D=3D -1) { RTE_LOG(WARNING, USER1, "Driver does not report supported ptypes; enabling software parsing.\n"); // Proceed with software-based packet type parsing return true; } bool detect_ether =3D false, detect_arp =3D false; auto num_l2_dtx =3D rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L2_MASK, nullptr, 0); if (num_l2_dtx =3D=3D -1) { RTE_LOG(ERR, USER1, "Failed to detect L2 detection capabilities.\n"); return false; } uint32_t l2_ptypes[num_l2_dtx]; num_l2_dtx =3D rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L2_MAS= K, l2_ptypes, num_l2_dtx); for (auto i =3D 0; i < num_l2_dtx; ++i) { if (l2_ptypes[i] & RTE_PTYPE_L2_ETHER) detect_ether =3D true; } // Detect L3 packet type detection capability bool detect_ipv4 =3D false, detect_ipv6 =3D false; auto num_l3_dtx =3D rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L3_MASK, nullptr, 0); if (num_l3_dtx =3D=3D -1) { RTE_LOG(ERR, USER1, "Failed to detect L3 detection capabilities.\n"); return false; } uint32_t l3_ptypes[num_l3_dtx]; num_l3_dtx =3D rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L3_MAS= K, l3_ptypes, num_l3_dtx); for (auto i =3D 0; i < num_l3_dtx; ++i) { if (l3_ptypes[i] & RTE_PTYPE_L3_IPV4) detect_ipv4 =3D true; if (l3_ptypes[i] & RTE_PTYPE_L3_IPV6) detect_ipv6 =3D true; } // Detect L4 packet type detection capability bool detect_tcp =3D false, detect_udp =3D false, detect_icmp =3D false; auto num_l4_dtx =3D rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L4_MASK, nullptr, 0); if (num_l4_dtx =3D=3D -1) { RTE_LOG(ERR, USER1, "Failed to detect L4 detection capabilities.\n"); return false; } uint32_t l4_ptypes[num_l4_dtx]; num_l4_dtx =3D rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L4_MAS= K, l4_ptypes, num_l4_dtx); for (auto i =3D 0; i < num_l4_dtx; ++i) { if (l4_ptypes[i] & RTE_PTYPE_L4_TCP) detect_tcp =3D true; if (l4_ptypes[i] & RTE_PTYPE_L4_UDP) detect_udp =3D true; if (l4_ptypes[i] & RTE_PTYPE_L4_ICMP) detect_icmp =3D true; } if (!detect_ether || !detect_arp || !detect_ipv4 || !detect_ipv6 || !detect_tcp || !detect_udp || !detect_icmp) { RTE_LOG(ERR, USER1, "Supported Detection Modes:\n" "L2 Ether: %s\n" "L2 Arp : %s\n" "L3 IPv4 : %s\n" "L3 IPv6 : %s\n" "L4 TCP : %s\n" "L4 UDP : %s\n" "L4 ICMP : %s\n", detect_ether ? "True" : "False", detect_arp ? "True": "False", detect_ipv4 ? "True": "False", detect_ipv6 ? "True": "False", detect_tcp ? "True" : "False", detect_udp ? "True" : "False", detect_icmp ? "True" : "False"); return false; } return true; } On Wed, Nov 13, 2024 at 9:10=E2=80=AFPM Yasuhiro Ohara = wrote: > I would suggest re-installation of MELLANOX OFED, and/or upgrading NIC > firmware (can be done using OFED tools). > > Yes, warning on only 1 port is odd. I suspect some kind of mismatch, like > older version of the NIC had only 1 port versions and the soft assumed it= , > for example. > > > 2024=E5=B9=B411=E6=9C=8814=E6=97=A5(=E6=9C=A8) 5:43 CJ Sculti : > >> I'm not using vfio, I just bound interfaces on there one time to test. >> Shouldn't I be able to just use the default mlx5_core driver, without >> binding to uio_pci_generic? >> >> >> On Wed, Nov 13, 2024 at 4:26=E2=80=AFPM Thomas Monjalon >> wrote: >> >>> 13/11/2024 21:10, CJ Sculti: >>> > I've been running my application for years on igb_uio with Intel NICs= . >>> I >>> > recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, >>> updated >>> > the DPDK version my application uses, and compiled with support for >>> mlx5 >>> > PMDs. Both 40Gbps ports are up with link, and both are in Ethernet >>> mode, >>> > not Infiniband mode. However, I'm getting complaints when I start my >>> > application about trying to load 'mlx5_eth'? Both are bound to >>> mlx5_core >>> > driver at the moment. When I bind them to vfio-pci, or >>> uio_pci_generic, my >>> > application fails to recognize them at all as valid DPDK devices. >>> Anyone >>> > have any ideas? Also, strange that it only complains about one? I hav= e >>> them >>> > configured in a bond on the kernel, as my application requires that. >>> >>> You must not bind mlx5 devices with VFIO. >>> I recommend reading documentation. >>> You can start here: >>> >>> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#bifurcated-dri= ver >>> then >>> https://doc.dpdk.org/guides/platform/mlx5.html#design >>> >>> >>> >>> --000000000000aa1c900626e1b242 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I figured out the initial issue. For some reason, having b= oth devices in a bond on the kernel results in only 1 of the two ports bein= g exposed as 'verb' devices. Previously,=C2=A0ibv_devinfo returned = only one port. After removing both from the bond,=C2=A0ibv_devinfo returns = both ports, and the DPDK application successfully=C2=A0takes both in. I'= ;m still having some weird behavior trying to create a bypass interface wit= h these ports though. I'm using the same code that I've been using = on my Intel NICs with igb_uio for years, but seeing weird behavior. The por= ts are connected to our 40Gbps Ethernet switch, and set to=C2=A0link_layer:= Ethernet.

The first thing I noticed is that=C2=A0rte_et= h_dev_reset() fails on these interfaces with "ENOTSUP: hardware doesn&= #39;t support reset".=C2=A0

Secondly, when ch= ecking ptypes, I noticed my code says these NICs are unable to support any = sort of packet detection capability=C2=A0(code below, all return false). Th= e MLX5 docs do say that all of these ptypes used here are supported by MLX5= .

I'm just picking up a project that was left = off by an older dev. It hasn't been touched in years, but has been work= ing fine with our Intel NICs. All I'm trying to do is update DPDK (whic= h is done, updated from dpdk 19.05 to DPDK 22.11, latest version with KNI s= upport), and get it to work with our Mellanox CX5 NICs. This is my first ti= me working with DPDK and I'm not very familiar. Should I expect to be a= ble to do this without having to make a ton of code changes, or is this goi= ng to be an uphill battle for me? If it's the latter, I will likely jus= t go purchase Intel NICs and give up on this.


=
bool Manager::HasPacketTypeDetectionCapabilities(uint16_t port_i= d) {

=C2=A0 // Detect L2 packet type detection capability

=C2= =A0 int num_ptypes =3D rte_eth_dev_get_supported_ptypes(
=C2=A0 =C2=A0 = =C2=A0 port_id, RTE_PTYPE_ALL_MASK, nullptr, 0);

=C2=A0 if (num_ptyp= es =3D=3D -1) {
=C2=A0 =C2=A0 RTE_LOG(WARNING, USER1,
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 "Driver does not report supported ptypes; enabling softw= are parsing.\n");
=C2=A0 =C2=A0 // Proceed with software-based pack= et type parsing
=C2=A0 =C2=A0 return true;
=C2=A0 }

=C2=A0 boo= l detect_ether =3D false, detect_arp =3D false;
=C2=A0 auto num_l2_dtx = =3D rte_eth_dev_get_supported_ptypes(port_id,
=C2=A0 =C2=A0 =C2=A0 RTE_P= TYPE_L2_MASK, nullptr, 0);
=C2=A0 if (num_l2_dtx =3D=3D -1) {
=C2=A0 = =C2=A0 RTE_LOG(ERR, USER1, "Failed to detect L2 detection capabilities= .\n");
=C2=A0 =C2=A0 return false;
=C2=A0 }
=C2=A0 uint32_t l= 2_ptypes[num_l2_dtx];
=C2=A0 num_l2_dtx =3D rte_eth_dev_get_supported_pt= ypes(port_id, RTE_PTYPE_L2_MASK,
=C2=A0 =C2=A0 =C2=A0 l2_ptypes, num_l2_= dtx);
=C2=A0 for (auto i =3D 0; i < num_l2_dtx; ++i) {
=C2=A0 =C2= =A0 if (l2_ptypes[i] & RTE_PTYPE_L2_ETHER)
=C2=A0 =C2=A0 =C2=A0 dete= ct_ether =3D true;
=C2=A0 }

=C2=A0 // Detect L3 packet type detec= tion capability
=C2=A0 bool detect_ipv4 =3D false, detect_ipv6 =3D false= ;
=C2=A0 auto num_l3_dtx =3D rte_eth_dev_get_supported_ptypes(port_id,=C2=A0 =C2=A0 =C2=A0 RTE_PTYPE_L3_MASK, nullptr, 0);
=C2=A0 if (num_l3= _dtx =3D=3D -1) {
=C2=A0 =C2=A0 RTE_LOG(ERR, USER1, "Failed to dete= ct L3 detection capabilities.\n");
=C2=A0 =C2=A0 return false;
= =C2=A0 }
=C2=A0 uint32_t l3_ptypes[num_l3_dtx];
=C2=A0 num_l3_dtx =3D= rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L3_MASK,
=C2=A0 =C2= =A0 =C2=A0 l3_ptypes, num_l3_dtx);
=C2=A0 for (auto i =3D 0; i < num_= l3_dtx; ++i) {
=C2=A0 =C2=A0 if (l3_ptypes[i] & RTE_PTYPE_L3_IPV4)=C2=A0 =C2=A0 =C2=A0 detect_ipv4 =3D true;
=C2=A0 =C2=A0 if (l3_ptypes= [i] & RTE_PTYPE_L3_IPV6)
=C2=A0 =C2=A0 =C2=A0 detect_ipv6 =3D true;<= br>=C2=A0 }

=C2=A0 // Detect L4 packet type detection capability
= =C2=A0 bool detect_tcp =3D false, detect_udp =3D false, detect_icmp =3D fal= se;
=C2=A0 auto num_l4_dtx =3D rte_eth_dev_get_supported_ptypes(port_id,=
=C2=A0 =C2=A0 =C2=A0 RTE_PTYPE_L4_MASK, nullptr, 0);
=C2=A0 if (num_= l4_dtx =3D=3D -1) {
=C2=A0 =C2=A0 RTE_LOG(ERR, USER1, "Failed to de= tect L4 detection capabilities.\n");
=C2=A0 =C2=A0 return false;=C2=A0 }
=C2=A0 uint32_t l4_ptypes[num_l4_dtx];
=C2=A0 num_l4_dtx = =3D rte_eth_dev_get_supported_ptypes(port_id, RTE_PTYPE_L4_MASK,
=C2=A0 = =C2=A0 =C2=A0 l4_ptypes, =C2=A0num_l4_dtx);
=C2=A0 for (auto i =3D 0; i = < num_l4_dtx; ++i) {
=C2=A0 =C2=A0 if (l4_ptypes[i] & RTE_PTYPE_L= 4_TCP)
=C2=A0 =C2=A0 =C2=A0 detect_tcp =3D true;
=C2=A0 =C2=A0 if (l4= _ptypes[i] & RTE_PTYPE_L4_UDP)
=C2=A0 =C2=A0 =C2=A0 detect_udp =3D t= rue;
=C2=A0 =C2=A0 if (l4_ptypes[i] & RTE_PTYPE_L4_ICMP)
=C2=A0 = =C2=A0 =C2=A0 detect_icmp =3D true;
=C2=A0 }

=C2=A0 if (!detect_e= ther || !detect_arp
=C2=A0 =C2=A0 =C2=A0 || !detect_ipv4 || !detect_ipv6=
=C2=A0 =C2=A0 =C2=A0 || !detect_tcp || !detect_udp || !detect_icmp) {=C2=A0 =C2=A0 RTE_LOG(ERR, USER1, "Supported Detection Modes:\n"= ;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 "L2 Ether: %s\n"
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 "L2 Arp =C2=A0: %s\n"
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 "L3 IPv4 : %s\n"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 "L3 I= Pv6 : %s\n"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 "L4 TCP =C2=A0: %s\n&q= uot;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 "L4 UDP =C2=A0: %s\n"
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 "L4 ICMP : %s\n",
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 detect_ether ? "True" : "False",
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 detect_arp ? "True": "False",
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 detect_ipv4 ? "True": "False&quo= t;,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 detect_ipv6 ? "True": "Fa= lse",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 detect_tcp ? "True" : &= quot;False",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 detect_udp ? "True&qu= ot; : "False",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 detect_icmp ? "= ;True" : "False");
=C2=A0 =C2=A0 return false;
=C2=A0 = }

=C2=A0 return true;
}

= On Wed, Nov 13, 2024 at 9:10=E2=80=AFPM Yasuhiro Ohara <yasu1976@gmail.com> wrote:
I would suggest re-installation of MELLANOX OFED, and/or upgrading NIC fir= mware (can be done using OFED tools).

Yes, warning on only 1 port is odd. I suspect some kind of mi= smatch, like older version of the NIC had only 1 port versions and the soft= assumed it, for example.


2024=E5=B9=B411= =E6=9C=8814=E6=97=A5(=E6=9C=A8) 5:43 CJ Sculti <cj@cj.gy>:
I'm not using= vfio, I just bound interfaces=C2=A0on there=C2=A0one time to test. Shouldn= 't I be able to just use the default mlx5_core driver, without binding = to uio_pci_generic?


On Wed, Nov 13, 2024 at 4:26=E2= =80=AFPM Thomas Monjalon <thomas@monjalon.net> wrote:
13/11/2024 21:10, CJ Sculti:
> I've been running my application for years on igb_uio with Intel N= ICs. I
> recently replaced them with a Mellanox ConnectX-5 2x 40gbps NIC, updat= ed
> the DPDK version my application uses, and compiled with support for ml= x5
> PMDs. Both 40Gbps ports are up with link, and both are in Ethernet mod= e,
> not Infiniband mode. However, I'm getting complaints when I start = my
> application about trying to load 'mlx5_eth'? Both are bound to= mlx5_core
> driver at the moment. When I bind them to vfio-pci, or uio_pci_generic= , my
> application fails to recognize them at all as valid DPDK devices. Anyo= ne
> have any ideas? Also, strange that it only complains about one? I have= them
> configured in a bond on the kernel, as my application requires that.
You must not bind mlx5 devices with VFIO.
I recommend reading documentation.
You can start here:
https://doc.dpdk.org/guide= s/linux_gsg/linux_drivers.html#bifurcated-driver
then
https://doc.dpdk.org/guides/platform/mlx5.html#= design



--000000000000aa1c900626e1b242--