From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EA9FE45D39 for ; Mon, 18 Nov 2024 18:34:15 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2BB3D41153; Mon, 18 Nov 2024 18:34:00 +0100 (CET) Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) by mails.dpdk.org (Postfix) with ESMTP id 6EA9340F35 for ; Mon, 18 Nov 2024 18:33:54 +0100 (CET) Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-6ee7a48377cso16550347b3.3 for ; Mon, 18 Nov 2024 09:33:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cj-gy.20230601.gappssmtp.com; s=20230601; t=1731951234; x=1732556034; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ucJvGJwDi5JLCB4OM4iNHS1TDbmvc5TCx1Ll3/O5ISU=; b=lHW63HfbE8sxb59bjfRN0IChJDoa9dVk5ShPL6jg0U5MESfn7nQdwRQmNvJwONnTHv hMIwzwIpstmRIeS1/mi2bSI+PeDO3wHbgNy9Cm/zfPEQ1MBZnphxNpht2xkTLIts0MHa AXUXCAi8yeSbQGQ78pEh6zbzCV+rMOkSLFgVT3knyeeEofUDTQFCB/2xuj/8r77xUz0h kQaLAhl8bZfKu1mov57GcjW137scboBijesMPOaGItr1YIUeeQn/S+nwPpHwCGY/M2NF tjayVduHI0FZDQk+FTHXaFwaekUZXQ6HCNd3J8UioV+h2zafwG6SdX6xrsfRIGSjN9I1 NtjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731951234; x=1732556034; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ucJvGJwDi5JLCB4OM4iNHS1TDbmvc5TCx1Ll3/O5ISU=; b=lFdI3EcfUG8AsFo5Q5qJKIscUPOv5T8Da0M7Z9eeXla61V7MGt8h0Nd4zPln4o+HeD H8ioRxanoDQXkaB0wfWJqS7PrNDbwcACW9984+ayWWBEPTqMoiOxbrWN9MniRlDecFJi LC3AofJ2ycP+WFMXY4Xl+pkcRHgzZP/RzbYPWQ9yWl24K6VFcXhSqGY69rj8PE/mQrXF HNCVOAHzqMpP6oJuq5GuHFajLUGzHgj7TDTSVhuJQDlzNs9Ad2TgB5yOrBCxhBeb3vKN vpfPIJTwoo02WIlq8FMQC2u8L5awJ8tgMf9HhJlid6w7PkTo0uiW5Vl96vcnz1Wg7Zko Eogw== X-Forwarded-Encrypted: i=1; AJvYcCUdk2lKEYtWfi+7pZ5W6MwjmIJa26NYlOGHvR7Cyz4rLfYRsTM8joxsyNYUeXE9IpJ86fq/aA==@dpdk.org X-Gm-Message-State: AOJu0YxwzLriti6XDtXT5po6BRi/kv1F2bdPHX72rnHrDhhh+h/Xjn3f Qean7Uu63VuvseZAd677JjB3470QUPlcWTt30tNixpmAT/IkuHYsRxqvKJa54nh4rBgo6NsD5fu xPs7juqBZjKxotUT1PZZhw1Ic/nNHF+44EZDK9A== X-Google-Smtp-Source: AGHT+IGcH6PsenwdDho9xY/aW7tjkltXNjVjzFCFHdorxxq/6x818uDJ48S6Yf1AJ7c1HxfWm4GpTv4X4VP/FTMmlOc= X-Received: by 2002:a05:690c:9c11:b0:6ea:4b85:7a13 with SMTP id 00721157ae682-6ee55a4f605mr143877177b3.3.1731951233481; Mon, 18 Nov 2024 09:33:53 -0800 (PST) MIME-Version: 1.0 References: <2965041.e9J7NaK4W3@thomas> In-Reply-To: From: CJ Sculti Date: Mon, 18 Nov 2024 12:33:38 -0500 Message-ID: Subject: Re: DPDK with Mellanox ConnectX-5, complaining about mlx5_eth? To: Yasuhiro Ohara Cc: Thomas Monjalon , users@dpdk.org, Dariusz Sosnowski Content-Type: multipart/alternative; boundary="0000000000000a599306273352cb" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000000a599306273352cb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank you for the advice on that. I have removed the reset, and I also fixed the rte_eth_dev_get_supported_ptypes(). I moved my rte_eth_dev_get_supported_ptypes() calls after the bypass interface was configured, and now they're returning expected values. My next issue I'm running into is the bonding issue that I talked about in my initial post. It seems that when part of a kernel bond, the mlx5_core driver combines both ports into a single 'verbs' device named 'mlx5_bond_0'. On my old setup with igb_uio, it worked like this: - At Linux boot, the 2x Intel ports were configured as a bond on the kernel via /etc/network/interfaces file. - Before starting DPDK software, both ports bound to igb_uio driver. - Software started, ports are setup by my software as kernel bypass. - Enslaved the 2x 'new' bypass interfaces onto the same bond. - Software reinjected LACP packets into kernel, to let kernel handle LACP protocol. This behavior where both ports are 'combined' into a single verbs device is strange to me. How should I handle this? Is there any way to disable it and just have the 2 ports be separate interfaces? 1. 2. root@DDoSMitigation:~/anubis/engine/bin# ibv_devinfo hca_id: mlx5_bond_0 transport: InfiniBand (0) fw_ver: 16.35.4030 node_guid: 506b:4b03:00b6:76ec sys_image_guid: 506b:4b03:00b6:76ec vendor_id: 0x02c9 vendor_part_id: 4119 hw_ver: 0x0 board_id: MT_0000000090 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet 3. [12:58 PM] On Fri, Nov 15, 2024 at 5:12=E2=80=AFAM Yasuhiro Ohara = wrote: > > I think you should try a bit more, we are here to help. > > I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK, > and I think you can make it work in only a few more steps. > > I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP > might have made the whole NIC functions stopped. > IIRC the ptype was working fine on CX5 in my app too. > Can you comment out the rte_eth_dev_reset() ? > (I think it's worth to try.) > > I don't think it is so uphill, but I don't disagree with you about > purchasing another > Intel 40G NICs. It'll also work perfectly fine. > > 2024=E5=B9=B411=E6=9C=8815=E6=97=A5(=E9=87=91) 4:16 Thomas Monjalon : > > > > 14/11/2024 17:10, CJ Sculti: > > > I figured out the initial issue. For some reason, having both devices > in a > > > bond on the kernel results in only 1 of the two ports being exposed a= s > > > 'verb' devices. Previously, ibv_devinfo returned only one port. After > > > removing both from the bond, ibv_devinfo returns both ports, and the > DPDK > > > application successfully takes both in. I'm still having some weird > > > behavior trying to create a bypass interface with these ports though. > I'm > > > using the same code that I've been using on my Intel NICs with igb_ui= o > for > > > years, but seeing weird behavior. The ports are connected to our 40Gb= ps > > > Ethernet switch, and set to link_layer: Ethernet. > > > > You should be able to make it work with kernel bonding, > > but I'm not sure what's wrong to do that. > > And it looks not a priority for you. Let's focus on the other parts. > > > > > > > The first thing I noticed is that rte_eth_dev_reset() fails on these > > > interfaces with "ENOTSUP: hardware doesn't support reset". > > > > You don't need the reset procedure with mlx5, > > so you can make this code optional. > > > > > > > Secondly, when checking ptypes, I noticed my code says these NICs are > > > unable to support any sort of packet detection capability (code below= , > all > > > return false). The MLX5 docs do say that all of these ptypes used her= e > are > > > supported by MLX5. > > > > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get() > code. > > I don't understand why it does not work for you. > > > > > > > I'm just picking up a project that was left off by an older dev. It > hasn't > > > been touched in years, but has been working fine with our Intel NICs. > All > > > I'm trying to do is update DPDK (which is done, updated from dpdk > 19.05 to > > > DPDK 22.11, latest version with KNI support), > > > > You don't need KNI with mlx5. > > That's a big benefit of mlx5 design, it is natively bifurcated: > > https://doc.dpdk.org/guides/howto/flow_bifurcation.html > > > > > > > and get it to work with our Mellanox CX5 NICs. > > > This is my first time working with DPDK and I'm not very > > > familiar. Should I expect to be able to do this without having to mak= e > a > > > ton of code changes, or is this going to be an uphill battle for me? = If > > > it's the latter, I will likely just go purchase Intel NICs and give u= p > on > > > this. > > > > The NICs have difference that DPDK is trying to hide. > > If something is not compatible you may consider it as a bug or a > limitation. > > I think you should try a bit more, we are here to help. > > > > > --0000000000000a599306273352cb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thank you for the advice on that. I have removed the reset= , and I also fixed the=C2=A0rte_eth_dev_get_supported_ptypes().=C2=A0I move= d my=C2=A0rte_eth_dev_get_supported_ptypes() calls after the bypass interfa= ce was configured, and now they're returning=C2=A0expected values.
=
My next issue I'm running into is the bonding issue that= I talked about in my initial post. It seems that when=C2=A0part of a kerne= l bond, the mlx5_core driver combines both ports into a single 'verbs&#= 39; device named 'mlx5_bond_0'.=C2=A0 On my old setup with igb_uio,= it worked like this:

- At Linux boot, the 2x Inte= l ports were configured as a bond on the kernel via /etc/network/interfaces= file.
- Before starting DPDK software, both ports bound to igb_u= io driver.
- Software started, ports are setup by my software=C2= =A0as kernel bypass.
- Enslaved the 2x 'new' bypass inter= faces onto the same bond.
- Software reinjected LACP packets into= kernel, to let kernel handle LACP protocol.

This = behavior where both ports are 'combined' into a single verbs device= is strange to me. How should I handle this? Is there any way to disable it= and just have the 2 ports be separate interfaces?
  • root@DDoSMit=
igation:~/anubis/engine/bin# ibv_devinfo=20
hca_id:    mlx5_bond_0
    transport:            InfiniBand (0)
    fw_ver:                16.35.4030
    node_guid:            506b:4b03:00b6:76ec
    sys_image_guid:            506b:4b03:00b6:76ec
    vendor_id:            0x02c9
    vendor_part_id:            4119
    hw_ver:                0x0
    board_id:            MT_0000000090
    phys_port_cnt:            1
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:        1024 (3)
            sm_lid:            0
            port_lid:        0
            port_lmc:        0x00
            link_layer:        Ethernet
    • =
  • [12:58 PM]


    • On Fri, Nov 15, 2024 at 5:12=E2=80=AFAM Yasuhiro = Ohara <yasu1976@= gmail.com> wrote:
    > I think you should try a bit more, we are here to help.

    I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK, and I think you can make it work in only a few more steps.

    I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP might have made the whole NIC functions stopped.
    IIRC the ptype was working fine on CX5 in my app too.
    Can you comment out the rte_eth_dev_reset() ?
    (I think it's worth to try.)

    I don't think it is so uphill, but I don't disagree with you about<= br> purchasing another
    Intel 40G NICs. It'll also work perfectly fine.

    2024=E5=B9=B411=E6=9C=8815=E6=97=A5(=E9=87=91) 4:16 Thomas Monjalon <thomas@monjalon.net>:
    >
    > 14/11/2024 17:10, CJ Sculti:
    > > I figured out the initial issue. For some reason, having both dev= ices in a
    > > bond on the kernel results in only 1 of the two ports being expos= ed as
    > > 'verb' devices. Previously, ibv_devinfo returned only one= port. After
    > > removing both from the bond, ibv_devinfo returns both ports, and = the DPDK
    > > application successfully takes both in. I'm still having some= weird
    > > behavior trying to create a bypass interface with these ports tho= ugh. I'm
    > > using the same code that I've been using on my Intel NICs wit= h igb_uio for
    > > years, but seeing weird behavior. The ports are connected to our = 40Gbps
    > > Ethernet switch, and set to link_layer: Ethernet.
    >
    > You should be able to make it work with kernel bonding,
    > but I'm not sure what's wrong to do that.
    > And it looks not a priority for you. Let's focus on the other part= s.
    >
    >
    > > The first thing I noticed is that rte_eth_dev_reset() fails on th= ese
    > > interfaces with "ENOTSUP: hardware doesn't support reset= ".
    >
    > You don't need the reset procedure with mlx5,
    > so you can make this code optional.
    >
    >
    > > Secondly, when checking ptypes, I noticed my code says these NICs= are
    > > unable to support any sort of packet detection capability (code b= elow, all
    > > return false). The MLX5 docs do say that all of these ptypes used= here are
    > > supported by MLX5.
    >
    > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get()= code.
    > I don't understand why it does not work for you.
    >
    >
    > > I'm just picking up a project that was left off by an older d= ev. It hasn't
    > > been touched in years, but has been working fine with our Intel N= ICs. All
    > > I'm trying to do is update DPDK (which is done, updated from = dpdk 19.05 to
    > > DPDK 22.11, latest version with KNI support),
    >
    > You don't need KNI with mlx5.
    > That's a big benefit of mlx5 design, it is natively bifurcated: >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0    https:/= /doc.dpdk.org/guides/howto/flow_bifurcation.html
    >
    >
    > > and get it to work with our Mellanox CX5 NICs.
    > > This is my first time working with DPDK and I'm not very
    > > familiar. Should I expect to be able to do this without having to= make a
    > > ton of code changes, or is this going to be an uphill battle for = me? If
    > > it's the latter, I will likely just go purchase Intel NICs an= d give up on
    > > this.
    >
    > The NICs have difference that DPDK is trying to hide.
    > If something is not compatible you may consider it as a bug or a limit= ation.
    > I think you should try a bit more, we are here to help.
    >
    >
    --0000000000000a599306273352cb--