From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 15B8CA050B for ; Thu, 7 Apr 2022 10:22:54 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9A81D40689; Thu, 7 Apr 2022 10:22:53 +0200 (CEST) Received: from mail-lf1-f44.google.com (mail-lf1-f44.google.com [209.85.167.44]) by mails.dpdk.org (Postfix) with ESMTP id B49DD4014F for ; Thu, 7 Apr 2022 10:22:52 +0200 (CEST) Received: by mail-lf1-f44.google.com with SMTP id k21so8337961lfe.4 for ; Thu, 07 Apr 2022 01:22:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dTrCOI4lFwQc3mh9/NKKQYwUfRPFkU4bC5tHSmMFpFQ=; b=EShd0uQ8DplZebpVH9L6XQn//1BOuTXUIV3gM92B6bwfs6RkYvRszxEkSmHXuhdPaD ASoiRVW/RFX8wNqLtlAn9sMwGhciji1x/Ih5h77uMagSN6IRQawTFKNFCaRAubuebuBu z/RWtmyCICEx/s82ARsNA5Vbzq+rJFXfxEOuSnXjy23WF7llcjH9qahRtElnf7UxIO6N 8bQJ2iNrNBClipcncdHpbdiXOt8SWATpCtMn1xiMjyfTWUrZ4UWvDWvUxSqil64EV2sB YuupFSVaRvMIeRJlTj5ObdEZ475mW3a+JGvYACoS7vXA4QkDZkqb06D6jRXW0VmtEGaZ YOdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dTrCOI4lFwQc3mh9/NKKQYwUfRPFkU4bC5tHSmMFpFQ=; b=FrYzg7pzg3vBGhoBDOhhNUFzakG7fC0I4BxWoqziWCtDXjyYS3fClPpYgtcQAFBygk SYvQjlY9PPAEs50JepHiCt8WSCpHGZi4nmUtth47Wa8o4Bywe4zOiJhAXQDjCIsLMtpS vuzUbPfIMG9tr0DuyI/kjPH6HKlB6JKJJ8eXDi7f7LZ0J4O+sGygDiLYfEtgV7Kv4yje +7fdoxGgnnMZyRkf9dygkPtIBNiIsGqS5C3uwSJXhPVtoqvmUUgi7vIQ+KPlbEsludRr c4JEzBBrY70KXbdwMKONIMUTm0gJB0NhEHjFMrBGTvnzUGN2w+TtT1WtNu/d2JA7uNoE CL5g== X-Gm-Message-State: AOAM530uoCiZFJ3kG+9y2DeQfhoh7mccN9Ck17mPrGT8Q6EzYDrFOhYh 9EaKmcO3aMYUdsOwqpCz69NG2rANlAymBMGgGmk= X-Google-Smtp-Source: ABdhPJyDgJa2lDw3hbiuqImvgIZ8Hqw1wfUzUTORY9jUUS/bzHCenabEWg944DekPuPQf9wS+y25bCHR1OQWb8x97+I= X-Received: by 2002:a05:6512:689:b0:44a:31bd:3a6a with SMTP id t9-20020a056512068900b0044a31bd3a6amr8870170lfe.386.1649319771954; Thu, 07 Apr 2022 01:22:51 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Erez Ferber Date: Thu, 7 Apr 2022 11:22:40 +0300 Message-ID: Subject: Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1 To: fwefew 4t4tg <7532yahoo@gmail.com> Cc: users@dpdk.org Content-Type: multipart/alternative; boundary="00000000000020e4bb05dc0c2fd0" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --00000000000020e4bb05dc0c2fd0 Content-Type: text/plain; charset="UTF-8" I assume your tree assumes there's a ConnectX-3 device installed, while the kernel driver doesnt support it for quite a while I would suggest re-compiling while excluding mlx4 PMD support. Thanks, Erez On Tue, 5 Apr 2022 at 23:54, fwefew 4t4tg <7532yahoo@gmail.com> wrote: > Mr. Ferber, much appreciated. I knew this metal box came to me with two > mellanox NICs bonded. I used their util to unbond it, but alas it did not > do it all the way. The /etc/network/interfaces was bad. I fixed the config > and rebooted. > > Voila. Success. ibv_devinfo now shows two devices. And the DPDK app runs. > > Your point about bonding nudged me to double check. > > *Now, on a second machine where I believe I have done everything I did on > the first machine*, running the DPDK application continues to look for > the wrong driver: > > EAL: Detected CPU lcores: 16 > EAL: Detected NUMA nodes: 1 > EAL: Detected shared linkage of DPDK > EAL: libmlx4.so.1: cannot open shared object file: No such file or > directory > EAL: FATAL: Cannot init plugins > EAL: Cannot init plugins > > Somehow I fixed this on the first machine, but I cannot duplicate success > on the second machine. > > Both machines report two devices; both machines show similar ifconifgs > > device node GUID > > ------ ---------------- > > mlx5_0 0c42a103007ea9b8 > > mlx5_1 0c42a103007ea9b9 > > > device node GUID > > ------ ---------------- > > mlx5_0 0c42a103007ea3ec > > mlx5_1 0c42a103007ea3ed > > > root@server:~/Dev/reinvent/scripts# ibv_devinfo > > hca_id: mlx5_0 > > transport: InfiniBand (0) > > fw_ver: 14.32.1010 > > node_guid: 0c42:a103:007e:a3ec > > sys_image_guid: 0c42:a103:007e:a3ec > > vendor_id: 0x02c9 > > vendor_part_id: 4117 > > hw_ver: 0x0 > > board_id: MT_2420110034 > > phys_port_cnt: 1 > > port: 1 > > state: PORT_ACTIVE (4) > > max_mtu: 4096 (5) > > active_mtu: 1024 (3) > > sm_lid: 0 > > port_lid: 0 > > port_lmc: 0x00 > > link_layer: Ethernet > > > hca_id: mlx5_1 > > transport: InfiniBand (0) > > fw_ver: 14.32.1010 > > node_guid: 0c42:a103:007e:a3ed > > sys_image_guid: 0c42:a103:007e:a3ec > > vendor_id: 0x02c9 > > vendor_part_id: 4117 > > hw_ver: 0x0 > > board_id: MT_2420110034 > > phys_port_cnt: 1 > > port: 1 > > state: PORT_ACTIVE (4) > > max_mtu: 4096 (5) > > active_mtu: 1024 (3) > > sm_lid: 0 > > port_lid: 0 > > port_lmc: 0x00 > > link_layer: Ethernet > > > > > > ibv_devinfo > > hca_id: mlx5_0 > > transport: InfiniBand (0) > > fw_ver: 14.32.1010 > > node_guid: 0c42:a103:007e:a9b8 > > sys_image_guid: 0c42:a103:007e:a9b8 > > vendor_id: 0x02c9 > > vendor_part_id: 4117 > > hw_ver: 0x0 > > board_id: MT_2420110034 > > phys_port_cnt: 1 > > port: 1 > > state: PORT_ACTIVE (4) > > max_mtu: 4096 (5) > > active_mtu: 1024 (3) > > sm_lid: 0 > > port_lid: 0 > > port_lmc: 0x00 > > link_layer: Ethernet > > > hca_id: mlx5_1 > > transport: InfiniBand (0) > > fw_ver: 14.32.1010 > > node_guid: 0c42:a103:007e:a9b9 > > sys_image_guid: 0c42:a103:007e:a9b8 > > vendor_id: 0x02c9 > > vendor_part_id: 4117 > > hw_ver: 0x0 > > board_id: MT_2420110034 > > phys_port_cnt: 1 > > port: 1 > > state: PORT_ACTIVE (4) > > max_mtu: 4096 (5) > > active_mtu: 1024 (3) > > sm_lid: 0 > > port_lid: 0 > > port_lmc: 0x00 > > link_layer: Ethernet > > On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber wrote: > >> Hi, >> >> Based on your output, the ConnectX-4LX device is configured in LAG mode >> managed via the kernel bonding scripts. In this mode, both physical >> functions share a single port (mlx5_bond_0). You should only probe the >> first PCI BDF - 01:00.0, not the 2nd one. >> >> By the way, the --dpdk installation flag should not be necessary, it is >> an old flag keps for Mellanox OFED builds lower than 5.x. >> >> Regards, >> Erez >> >> On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532yahoo@gmail.com> wrote: >> >>> I built the current version of DPDK directly from dpdk.org after I >>> installed the current OFED Mellanox driver set: >>> >>> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso >>> >>> with ./install --dpdk >>> >>> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which >>> is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034 >>> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI >>> device 0000:01:00.1, are kernel drivers loaded?" >>> >>> EAL: Detected CPU lcores: 16 >>> EAL: Detected NUMA nodes: 1 >>> EAL: Detected static linkage of DPDK >>> EAL: Selected IOVA mode 'PA' >>> EAL: No free 2048 kB hugepages reported on node 0 >>> EAL: VFIO support initialized >>> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket >>> 0) >>> mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are kernel >>> drivers loaded? >>> mlx5_common: Verbs device not found: 01:00.1 >>> mlx5_common: Failed to initialize device context. >>> EAL: Requested device 0000:01:00.1 cannot be used >>> EAL: Bus (pci) probe failed. >>> >>> As far as I can see all the kernel modules are loaded: >>> >>> lsmod | egrep "(ib|mlx)" | sort >>> ib_cm 53248 2 rdma_cm,ib_ipoib >>> ib_core 368640 8 >>> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm >>> ib_ipoib 135168 0 >>> ib_umad 24576 0 >>> ib_uverbs 139264 2 rdma_ucm,mlx5_ib >>> libahci 36864 1 ahci >>> libcrc32c 16384 2 btrfs,raid456 >>> mlx5_core 1634304 1 mlx5_ib >>> mlx5_ib 397312 0 >>> mlx_compat 69632 11 >>> rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core >>> mlxdevm 172032 1 mlx5_core >>> mlxfw 32768 1 mlx5_core >>> pci_hyperv_intf 16384 1 mlx5_core >>> psample 20480 1 mlx5_core >>> tls 94208 2 bonding,mlx5_core >>> >>> root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v >>> MST modules: >>> ------------ >>> MST PCI module is not loaded >>> MST PCI configuration module loaded >>> PCI devices: >>> ------------ >>> DEVICE_TYPE MST PCI RDMA >>> NET NUMA >>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1 >>> mlx5_bond_0 net-bond0 -1 >>> >>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 >>> mlx5_bond_0 net-bond0 -1 >>> >> --00000000000020e4bb05dc0c2fd0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I assume your tree assumes there's a ConnectX-3 device= installed, while the kernel driver doesnt support it for quite a whileI would suggest re-compiling while excluding mlx4 PMD support.
<= br>
Thanks,
Erez

On Tue, 5 Apr 2022 at 23:54, fwef= ew 4t4tg <7532yahoo@gmail.com= > wrote:
Mr. Ferber, much appreciated. I knew this metal box came to me= with two mellanox NICs bonded. I used their util to unbond it, but alas it= did not do it all the way. The=C2=A0/etc/network/interfaces was bad. I fix= ed the config and rebooted.

Voila. Success. ibv_devinfo now shows tw= o devices. And the DPDK app runs.

Your point about bonding nudged m= e to double check.

Now, on a second machine where I believe I hav= e done everything I did on the first machine, running the=C2=A0DPDK app= lication continues to look for the wrong driver:

EAL: Detected CPU l= cores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of= DPDK
EAL: libmlx4.so.1: cannot open shared object file: No such file or= directory
EAL: FATAL: Cannot init plugins
EAL: Cannot init plugins

Somehow I fixed this on the first machine, but = I cannot duplicate success on the second machine.

= Both machines report two devices; both machines show similar ifconifgs
<= br>

=C2=A0 =C2=A0=C2=A0device= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 node GUID

=C2=A0 =C2=A0 ------=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <= /span>----------------

=C2=A0 =C2=A0 mlx5_0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <= /span>0c42a103007ea9b8

=C2=A0 =C2=A0 mlx5_1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <= /span>0c42a103007ea9b9


=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 no= de GUID

=C2=A0 =C2=A0 ------=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 ---------= -------

=C2=A0 =C2=A0 mlx5_0=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 0c42a1030= 07ea3ec

=C2=A0 =C2=A0=C2=A0mlx5_1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0c42a103007ea3ed


root@server:~/Dev/reinvent/scripts# ibv_devin= fo

hca_id: mlx5_0=

transport: InfiniBand (0)

fw_ver: 14.3= 2.1010

node_guid: 0c42:a103:007e:a3ec

sys_image_guid: <= /span> 0c42:a103:007e:a3ec

vendor_id: 0x02c9

vendor_part_id: 4117

hw_ver: 0x0

board_id: <= /span> MT_2420110034

phys_port_cnt: 1

port: 1

state: PORT_= ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

<= span style=3D"font-variant-ligatures:no-common-ligatures"> port_lid: 0=

port_lmc: 0x00

= link_layer: Ethernet

hca_id: mlx5_1

transport: InfiniBand (0)

fw_ver: 14.32.1010

node_guid: 0c42:a103:007e:a3ed

sys_image_guid: 0c42:a103:007e:a3ec

vendor_id: 0x02c9

vendor_part_id: 4117

hw_ver: 0x0

board_id: MT_2420110034

phys_port_cnt: 1

port: 1

sta= te: PORT_ACTIVE = (4)

max_= mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

= port_lmc: 0x00

link_layer: Ethernet=





ibv_devinfo=C2=A0=

hca_id: mlx5_0

transport: InfiniBand (0)

fw_ver: 14.32.1010

node_guid: 0c42:a103:007e:a9b8

sys_image_guid: 0c42:a103:007e:a9b8

vendor_id: 0x02c9

vendor_part_id: 4117

hw_ver: 0x0

board_id: MT_2420110034

phys_port_cnt: 1

port: 1

sta= te: PORT_ACTIVE = (4)

max_= mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

= port_lmc: 0x00

link_layer: Ethernet

=

<= p style=3D"margin:0px;font:11px Menlo;color:rgb(33,255,6);background-color:= rgb(0,0,0)">hca_= id: mlx5_1

transport: InfiniBand (0)

fw_ver: 14.32.1010=

node_guid: 0c42:a103:007e:a9b9

sys_image_guid: 0c42:a103:007e:a9b8

vendor_id: 0x02c9

vendor_part_id: <= span style=3D"white-space:pre-wrap"> 4117

hw_v= er: 0x0

board_id: MT_2420110034

= phys_port_cnt: 1=

port: 1<= /span>

state: PORT_ACTIVE (4)

max_mtu:<= span style=3D"white-space:pre-wrap"> 4096 (5)

= active_mtu: 1024 (3)

sm_lid: 0

port_lid: = 0

port_lmc: 0x00<= /span>

link_layer: Ethernet=


On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber <erezferber@gmail.com> wrote:=
Hi,

Based on your output, the ConnectX-4LX device is c= onfigured in LAG mode managed via the kernel bonding scripts. In this mode,= both physical functions share a single port (mlx5_bond_0). You should only= probe the first PCI BDF - 01:00.0, not the 2nd one.

By the way, the --dpdk installation flag should not be necessary, it is = an old flag=C2=A0keps for Mellanox OFED builds lower than 5.x.
Regards,
Erez

On Tue, 5 Apr 2022 at 19:17, fwef= ew 4t4tg <7532y= ahoo@gmail.com> wrote:
I built the current version of DPDK directly= from dpdk.org after I in= stalled the current OFED Mellanox driver set:

* MLNX_OFED_LINUX-5.5-= 1.0.3.2-ubuntu20.04-x86_64.iso

with ./install --dpdk

I am usi= ng a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which is Ethernet= only; there is no IB mode for this NIC. This is a MT_2420110034 board. How= ever, when I run dpdk-testpmd I see "No Verbs device matches PCI devic= e 0000:01:00.1, are kernel drivers loaded?"

EAL: Detected CPU l= cores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of= DPDK
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugep= ages reported on node 0
EAL: VFIO support initialized
EAL: Probe PCI = driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0)
mlx5_common= : No Verbs device matches PCI device 0000:01:00.1, are kernel drivers loade= d? =C2=A0
mlx5_common: Verbs device not found: 01:00.1
mlx5_common: F= ailed to initialize device context.
EAL: Requested device 0000:01:00.1 c= annot be used
EAL: Bus (pci) probe failed.

As far as I can see al= l the kernel modules are loaded:

lsmod | egrep "(ib|mlx)" = | sort
ib_cm =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A053248 =C2=A02 rdma_cm,ib_ipoib
ib_core =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 368640 =C2=A08 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm= ,ib_uverbs,mlx5_ib,ib_cm
ib_ipoib =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0135168 =C2=A00
ib_umad =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A024576 =C2=A00
ib_uverbs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 139264 =C2=A02 rdma_ucm,mlx5_ib
libahci =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A036864 =C2=A01 ahci
libcrc32c =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A016384 =C2=A02 btrfs,raid456
mlx= 5_core =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01634304 =C2=A01 mlx5_ib
= mlx5_ib =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 397312 =C2=A00
= mlx_compat =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 69632 =C2=A011 rdma_cm= ,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,ml= x5_core
mlxdevm =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 172032 = =C2=A01 mlx5_core
mlxfw =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A032768 =C2=A01 mlx5_core
pci_hyperv_intf =C2=A0 =C2=A0 =C2= =A0 =C2=A016384 =C2=A01 mlx5_core
psample =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A020480 =C2=A01 mlx5_core
tls =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A094208 =C2=A02 bonding,m= lx5_core

root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status = -v
MST modules:
------------
=C2=A0 =C2=A0 MST PCI module is not l= oaded
=C2=A0 =C2=A0 MST PCI configuration module loaded
PCI devices:<= br>------------
DEVICE_TYPE =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 MS= T =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 PCI =C2=A0 =C2=A0 =C2=A0 RDMA =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0NET =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 NUMA =C2=A0
ConnectX4LX(rev:0) =C2=A0 =C2=A0 =C2=A0= /dev/mst/mt4117_pciconf0.1 =C2=A0 =C2=A001:00.1 =C2=A0 mlx5_bond_0 =C2=A0 = =C2=A0 net-bond0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -1= =C2=A0 =C2=A0

ConnectX4LX(rev:0) =C2=A0 =C2=A0 =C2=A0/dev/mst/mt411= 7_pciconf0 =C2=A0 =C2=A0 =C2=A001:00.0 =C2=A0 mlx5_bond_0 =C2=A0 =C2=A0 net= -bond0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -1=C2=A0 =C2= =A0=C2=A0
--00000000000020e4bb05dc0c2fd0--