From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6B43345D39 for ; Mon, 18 Nov 2024 18:46:35 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 48591402B9; Mon, 18 Nov 2024 18:46:35 +0100 (CET) Received: from fout-a4-smtp.messagingengine.com (fout-a4-smtp.messagingengine.com [103.168.172.147]) by mails.dpdk.org (Postfix) with ESMTP id CCB364025E for ; Mon, 18 Nov 2024 18:46:33 +0100 (CET) Received: from phl-compute-12.internal (phl-compute-12.phl.internal [10.202.2.52]) by mailfout.phl.internal (Postfix) with ESMTP id 65BE31380708; Mon, 18 Nov 2024 12:46:33 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-12.internal (MEProxy); Mon, 18 Nov 2024 12:46:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1731951993; x=1732038393; bh=zutOxgedXxLD+Breqss/Jns23SY9plF4ACOM+siRUq8=; b= rKxNyiRzTdNlLVJ8L93pTLVX9FQRlcOtA/MUhRODP8lViR/h+pkr1gtSOGINQcX9 DMrHXJFEtFbWnJtAmgZF9JXOD/1r3KDi/33joo0M4BPvou0A2heR/C/P5fOesCM4 Y5/A/SLMLE9QsiTGRmVTX0ZsrhTqQIT18KtbwXamijLHUPvohRPLrHuxP5809o4q wXK6Pgej4IIyniDVJB2jvV552eZmVhmAysPjKiKge2zMwu0/i475pb1Vml1rSVTL oVXj21N6+X6cNqlGvPVjgZ9qY03NAcKAL4tfZjhSe8/iiURBbFeuDHzcIvFQ2lny TFeH/NJzjq+c2bxTvFl3PQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1731951993; x= 1732038393; bh=zutOxgedXxLD+Breqss/Jns23SY9plF4ACOM+siRUq8=; b=N NZ2YOEIxIuJb2UHlSgKPfTjrL0foMcAjnDyFDCggOGULK5c42mx0AJfKnMubwb5b ZpY6bBjZ1fKQ+E0vfirMx3R/KL1V6rUYr+xKUx9EhGwug7/ylbDsvA3qPJqqF8IL MESK3UAKCjJ1kYIjI+3Tip0AkmrV3f0CCZW1tQghgyrJVlPI1FM8bba3J2q4m2/j 8TS2cXwow5PqT8WtloKZXZX7P8Tl2RfdVY2cqiwNT6RUJAx43fWXEscQ12VGseYx EOq339ueo7T1gbKjfiY1w+VUdY8RjmfcrlzYshmjIFuG23jM8qTcCncFIkOjLPc/ twgW13JMVtVSd76OqX56A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrfedtgddutddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnh htshculddquddttddmnecujfgurhephffvvefufffkjghfggfgtgesthhqredttddtjeen ucfhrhhomhepvfhhohhmrghsucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrg hlohhnrdhnvghtqeenucggtffrrghtthgvrhhnpefftdeuhfehvdekleelveffvdelhfel hedvgedtvddvudeuieevtdfgjedvudegfeenucffohhmrghinhepughpughkrdhorhhgne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhm rghssehmohhnjhgrlhhonhdrnhgvthdpnhgspghrtghpthhtohepgedpmhhouggvpehsmh htphhouhhtpdhrtghpthhtoheptghjsegtjhdrghihpdhrtghpthhtohephigrshhuudel jeeisehgmhgrihhlrdgtohhmpdhrtghpthhtohepuhhsvghrshesughpughkrdhorhhgpd hrtghpthhtohepughsohhsnhhofihskhhisehnvhhiughirgdrtghomh X-ME-Proxy: Feedback-ID: i47234305:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Nov 2024 12:46:32 -0500 (EST) From: Thomas Monjalon To: CJ Sculti Cc: Yasuhiro Ohara , users@dpdk.org, Dariusz Sosnowski Subject: Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? Date: Mon, 18 Nov 2024 18:46:31 +0100 Message-ID: <1835831.3VsfAaAtOV@thomas> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org I'm not sure to understand your need. You want ports being bonded in the kernel but use them separately in usersp= ace? About "reinjecting" in the kernel, you don't need to do that with mlx5, as I said before, you can enjoy the bifurcated model to make some flows being processed in the kernel directly: you choose which flows go to the ke= rnel or to userspace. Also, you can find this in mlx5 doc: " =2D ``lacp_by_user`` parameter [int] A nonzero value enables the control of LACP traffic by the user applicati= on. When a bond exists in the driver, by default it should be managed by the kernel and therefore LACP traffic should be steered to the kernel. If this devarg is set to 1 it will allow the user to manage the bond by itself and not steer LACP traffic to the kernel. Disabled by default (set to 0). " 18/11/2024 18:33, CJ Sculti: > Thank you for the advice on that. I have removed the reset, and I also > fixed the rte_eth_dev_get_supported_ptypes(). I moved > my rte_eth_dev_get_supported_ptypes() calls after the bypass interface was > configured, and now they're returning expected values. >=20 > My next issue I'm running into is the bonding issue that I talked about in > my initial post. It seems that when part of a kernel bond, the mlx5_core > driver combines both ports into a single 'verbs' device named > 'mlx5_bond_0'. On my old setup with igb_uio, it worked like this: >=20 > - At Linux boot, the 2x Intel ports were configured as a bond on the kern= el > via /etc/network/interfaces file. > - Before starting DPDK software, both ports bound to igb_uio driver. > - Software started, ports are setup by my software as kernel bypass. > - Enslaved the 2x 'new' bypass interfaces onto the same bond. > - Software reinjected LACP packets into kernel, to let kernel handle LACP > protocol. >=20 > This behavior where both ports are 'combined' into a single verbs device = is > strange to me. How should I handle this? Is there any way to disable it a= nd > just have the 2 ports be separate interfaces? >=20 > 1. > 2. >=20 > root@DDoSMitigation:~/anubis/engine/bin# ibv_devinfo > hca_id: mlx5_bond_0 > transport: InfiniBand (0) > fw_ver: 16.35.4030 > node_guid: 506b:4b03:00b6:76ec > sys_image_guid: 506b:4b03:00b6:76ec > vendor_id: 0x02c9 > vendor_part_id: 4119 > hw_ver: 0x0 > board_id: MT_0000000090 > phys_port_cnt: 1 > port: 1 > state: PORT_ACTIVE (4) > max_mtu: 4096 (5) > active_mtu: 1024 (3) > sm_lid: 0 > port_lid: 0 > port_lmc: 0x00 > link_layer: Ethernet >=20 > 3. [12:58 PM] >=20 >=20 >=20 > On Fri, Nov 15, 2024 at 5:12=E2=80=AFAM Yasuhiro Ohara wrote: >=20 > > > I think you should try a bit more, we are here to help. > > > > I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK, > > and I think you can make it work in only a few more steps. > > > > I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP > > might have made the whole NIC functions stopped. > > IIRC the ptype was working fine on CX5 in my app too. > > Can you comment out the rte_eth_dev_reset() ? > > (I think it's worth to try.) > > > > I don't think it is so uphill, but I don't disagree with you about > > purchasing another > > Intel 40G NICs. It'll also work perfectly fine. > > > > 2024=E5=B9=B411=E6=9C=8815=E6=97=A5(=E9=87=91) 4:16 Thomas Monjalon : > > > > > > 14/11/2024 17:10, CJ Sculti: > > > > I figured out the initial issue. For some reason, having both devic= es > > in a > > > > bond on the kernel results in only 1 of the two ports being exposed= as > > > > 'verb' devices. Previously, ibv_devinfo returned only one port. Aft= er > > > > removing both from the bond, ibv_devinfo returns both ports, and the > > DPDK > > > > application successfully takes both in. I'm still having some weird > > > > behavior trying to create a bypass interface with these ports thoug= h. > > I'm > > > > using the same code that I've been using on my Intel NICs with igb_= uio > > for > > > > years, but seeing weird behavior. The ports are connected to our 40= Gbps > > > > Ethernet switch, and set to link_layer: Ethernet. > > > > > > You should be able to make it work with kernel bonding, > > > but I'm not sure what's wrong to do that. > > > And it looks not a priority for you. Let's focus on the other parts. > > > > > > > > > > The first thing I noticed is that rte_eth_dev_reset() fails on these > > > > interfaces with "ENOTSUP: hardware doesn't support reset". > > > > > > You don't need the reset procedure with mlx5, > > > so you can make this code optional. > > > > > > > > > > Secondly, when checking ptypes, I noticed my code says these NICs a= re > > > > unable to support any sort of packet detection capability (code bel= ow, > > all > > > > return false). The MLX5 docs do say that all of these ptypes used h= ere > > are > > > > supported by MLX5. > > > > > > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get() > > code. > > > I don't understand why it does not work for you. > > > > > > > > > > I'm just picking up a project that was left off by an older dev. It > > hasn't > > > > been touched in years, but has been working fine with our Intel NIC= s. > > All > > > > I'm trying to do is update DPDK (which is done, updated from dpdk > > 19.05 to > > > > DPDK 22.11, latest version with KNI support), > > > > > > You don't need KNI with mlx5. > > > That's a big benefit of mlx5 design, it is natively bifurcated: > > > https://doc.dpdk.org/guides/howto/flow_bifurcation.html > > > > > > > > > > and get it to work with our Mellanox CX5 NICs. > > > > This is my first time working with DPDK and I'm not very > > > > familiar. Should I expect to be able to do this without having to m= ake > > a > > > > ton of code changes, or is this going to be an uphill battle for me= ? If > > > > it's the latter, I will likely just go purchase Intel NICs and give= up > > on > > > > this. > > > > > > The NICs have difference that DPDK is trying to hide. > > > If something is not compatible you may consider it as a bug or a > > limitation. > > > I think you should try a bit more, we are here to help. > > > > > > > > >=20