From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D87C7A0509 for ; Tue, 5 Apr 2022 22:54:49 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 658F640F35; Tue, 5 Apr 2022 22:54:49 +0200 (CEST) Received: from mail-vk1-f178.google.com (mail-vk1-f178.google.com [209.85.221.178]) by mails.dpdk.org (Postfix) with ESMTP id B322240DF6 for ; Tue, 5 Apr 2022 22:54:47 +0200 (CEST) Received: by mail-vk1-f178.google.com with SMTP id c4so181913vkq.9 for ; Tue, 05 Apr 2022 13:54:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=X9f9jHRlPD13+1Wp2LgtaS7WI/AKEExcpc/W3H/cPVI=; b=J9bcC2LTE/VY0eYcws2l8U4DgzFc2TNcx2GN4UrfJKfE3UOw2Ybkpy/PLoAqAx/abI 3J7lDyrDSRqKZOH6bWxkjgzCKFFCNy/biXCRU5tRUg9YoSBUyx1XRgHBC9gOCdFf9gLG EaO72burxjndGJ2W5w/CFFEMF2yFpxkjEqD3cAB3sHdGI/WHGn2u2YhA6De55APmYura Su9DqLcVZ7KIyaHatHns+8/P3L4Yp6rXZqMN+DJKMkum3hOGn9UxeIqAnFw5/alosEk+ Vjl8uHQDecBwPc9yhTAkaTNVDxHnHY1EuH2WSc3D+LvCam7y6psrK0twSGxxtBoehm+f CMyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=X9f9jHRlPD13+1Wp2LgtaS7WI/AKEExcpc/W3H/cPVI=; b=Df0t60LWMxENlexKClfJdhMpmamZbsVdQGgdHzs7CF5z/CXd86hKZA00YtjYCYNZN8 lDrr6BNffx4n0PfXlzsA6mI27EpIWQIOOSPa4WInvsMma28brsmM5mF2LQya/wWqOaVI 02EpUTWNlOMCzZmeTVhGlyI2yJ+cPFK63YP4AbHxEg9u44b57J+74452/iTRPhAqwEyH b4hTYReg+PpkwRctlNKnFGv6VIeiVC9ruwrgzEN+EeWhsYFcpZAi/uco20eGMYMn2aYU cuiIbnsollpK+VyPwFQ3E3mrzOBmTEnY69+uB2cdDKsG4vNd8ZdmnwxgUAsVqPZFR2wh 8ZbA== X-Gm-Message-State: AOAM530HjyLIzmWofaFqTvDND4X72XeEu0ZWdJAQR8Xmzq1OhlLYeZfV LfVPfGAg/k/DgTd18C8v37GhbDEmVe+VLWbq2KY= X-Google-Smtp-Source: ABdhPJzvkyVdrHyvF7L059FV9jnCd0mNaMgHLeLU50efCtGoODy4xQ5ImipWzDBXt7IkfdVAvaGtc+bdgJkKFuO917I= X-Received: by 2002:a05:6122:8d2:b0:344:541c:c8e5 with SMTP id 18-20020a05612208d200b00344541cc8e5mr1959841vkg.32.1649192086845; Tue, 05 Apr 2022 13:54:46 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: fwefew 4t4tg <7532yahoo@gmail.com> Date: Tue, 5 Apr 2022 16:54:34 -0400 Message-ID: Subject: Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1 To: Erez Ferber , users@dpdk.org Content-Type: multipart/alternative; boundary="00000000000080bc2b05dbee744c" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --00000000000080bc2b05dbee744c Content-Type: text/plain; charset="UTF-8" Mr. Ferber, much appreciated. I knew this metal box came to me with two mellanox NICs bonded. I used their util to unbond it, but alas it did not do it all the way. The /etc/network/interfaces was bad. I fixed the config and rebooted. Voila. Success. ibv_devinfo now shows two devices. And the DPDK app runs. Your point about bonding nudged me to double check. *Now, on a second machine where I believe I have done everything I did on the first machine*, running the DPDK application continues to look for the wrong driver: EAL: Detected CPU lcores: 16 EAL: Detected NUMA nodes: 1 EAL: Detected shared linkage of DPDK EAL: libmlx4.so.1: cannot open shared object file: No such file or directory EAL: FATAL: Cannot init plugins EAL: Cannot init plugins Somehow I fixed this on the first machine, but I cannot duplicate success on the second machine. Both machines report two devices; both machines show similar ifconifgs device node GUID ------ ---------------- mlx5_0 0c42a103007ea9b8 mlx5_1 0c42a103007ea9b9 device node GUID ------ ---------------- mlx5_0 0c42a103007ea3ec mlx5_1 0c42a103007ea3ed root@server:~/Dev/reinvent/scripts# ibv_devinfo hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:007e:a3ec sys_image_guid: 0c42:a103:007e:a3ec vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:007e:a3ed sys_image_guid: 0c42:a103:007e:a3ec vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet ibv_devinfo hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:007e:a9b8 sys_image_guid: 0c42:a103:007e:a9b8 vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 14.32.1010 node_guid: 0c42:a103:007e:a9b9 sys_image_guid: 0c42:a103:007e:a9b8 vendor_id: 0x02c9 vendor_part_id: 4117 hw_ver: 0x0 board_id: MT_2420110034 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber wrote: > Hi, > > Based on your output, the ConnectX-4LX device is configured in LAG mode > managed via the kernel bonding scripts. In this mode, both physical > functions share a single port (mlx5_bond_0). You should only probe the > first PCI BDF - 01:00.0, not the 2nd one. > > By the way, the --dpdk installation flag should not be necessary, it is an > old flag keps for Mellanox OFED builds lower than 5.x. > > Regards, > Erez > > On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532yahoo@gmail.com> wrote: > >> I built the current version of DPDK directly from dpdk.org after I >> installed the current OFED Mellanox driver set: >> >> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso >> >> with ./install --dpdk >> >> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which >> is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034 >> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI >> device 0000:01:00.1, are kernel drivers loaded?" >> >> EAL: Detected CPU lcores: 16 >> EAL: Detected NUMA nodes: 1 >> EAL: Detected static linkage of DPDK >> EAL: Selected IOVA mode 'PA' >> EAL: No free 2048 kB hugepages reported on node 0 >> EAL: VFIO support initialized >> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket >> 0) >> mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are kernel >> drivers loaded? >> mlx5_common: Verbs device not found: 01:00.1 >> mlx5_common: Failed to initialize device context. >> EAL: Requested device 0000:01:00.1 cannot be used >> EAL: Bus (pci) probe failed. >> >> As far as I can see all the kernel modules are loaded: >> >> lsmod | egrep "(ib|mlx)" | sort >> ib_cm 53248 2 rdma_cm,ib_ipoib >> ib_core 368640 8 >> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm >> ib_ipoib 135168 0 >> ib_umad 24576 0 >> ib_uverbs 139264 2 rdma_ucm,mlx5_ib >> libahci 36864 1 ahci >> libcrc32c 16384 2 btrfs,raid456 >> mlx5_core 1634304 1 mlx5_ib >> mlx5_ib 397312 0 >> mlx_compat 69632 11 >> rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core >> mlxdevm 172032 1 mlx5_core >> mlxfw 32768 1 mlx5_core >> pci_hyperv_intf 16384 1 mlx5_core >> psample 20480 1 mlx5_core >> tls 94208 2 bonding,mlx5_core >> >> root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v >> MST modules: >> ------------ >> MST PCI module is not loaded >> MST PCI configuration module loaded >> PCI devices: >> ------------ >> DEVICE_TYPE MST PCI RDMA >> NET NUMA >> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1 >> mlx5_bond_0 net-bond0 -1 >> >> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 >> mlx5_bond_0 net-bond0 -1 >> > --00000000000080bc2b05dbee744c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Mr. Ferber, much appreciated. I knew this metal box came t= o me with two mellanox NICs bonded. I used their util to unbond it, but ala= s it did not do it all the way. The=C2=A0/etc/network/interfaces was bad. I= fixed the config and rebooted.

Voila. Success. ibv_devinfo now show= s two devices. And the DPDK app runs.

Your point about bonding nudg= ed me to double check.

Now, on a second machine where I believe I= have done everything I did on the first machine, running the=C2=A0DPDK= application continues to look for the wrong driver:

EAL: Detected C= PU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkag= e of DPDK
EAL: libmlx4.so.1: cannot open shared object file: No such fil= e or directory
EAL: FATAL: Cannot init plugins
EAL: Cannot init plugi= ns

Somehow I fixed this on the first machine, = but I cannot duplicate success on the second machine.

<= div>Both machines report two devices; both machines show similar ifconifgs<= br>

= =C2=A0 =C2=A0=C2=A0devic= e =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 node GUID

= =C2=A0 =C2=A0 ------=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = ----------------

= =C2=A0 =C2=A0 mlx5_0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = 0c42a103007ea9b8

= =C2=A0 =C2=A0 mlx5_1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = 0c42a103007ea9b9


=C2=A0 =C2=A0 devic= e =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 node GUID

=C2=A0 =C2=A0 ------=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = ----------------

=C2=A0 =C2=A0 mlx5_0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0c42a103= 007ea3ec

=C2=A0 =C2=A0=C2=A0<= /span>mlx5_1=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 0c42a103007ea3ed


root@server:~/Dev/reinvent/scripts# ibv_devinfo

hca_id: mlx5_0

transport: InfiniBand (0)

= fw_v= er: <= span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> 14.32.1010

node_guid: 0c42:a103:007e:a3ec

sys_image_guid: 0c42:a103:007e:a3ec

vendor_id: 0x02c9

vendor_part_id:= 4117<= /p>

hw_ver: 0x0

<= span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> board= _id: = MT_24201= 10034

phys_port_cnt: 1

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet


hca_id: mlx5_1

transport: = InfiniBand (0)

fw_ver: 14.32.1010

node_guid: 0c42:a103:007e:a3ed

sys_image_guid: 0c42:a103:007e:a3ec

= vendor_id: 0x02c9

vendor_part_id: 4117

hw_ver: 0x0

board_id:= MT_242011003= 4

phys_port_cnt: 1

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 1024 (3)=

sm_lid: 0

port_lid: 0

port_lmc: 0x00

= l= ink_layer: <= /span> Ethernet





ibv_devinfo= =C2=A0

hca_id: mlx5_0=

transport: InfiniBand (0)

fw_ver: 14.32.1010

node_guid: 0c42:a103:007e:a9b8=

sys_image_guid: 0c42:a103:007e:a9b8

vendor_id: 0x02c9

vendor_part_id: = 4117

hw_ver: 0x0

board_id: MT_2420110034

phys_port= _cnt: 1

port: 1

state:<= span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> PORT_ACTIVE (= 4)

max_mtu: 4096 (5)

= active_mtu: <= /span>1024 (3)

sm_lid: 0

= port_lid:= 0

port_lmc: 0x00

= link_layer: <= /span>Ethernet


hca_id: mlx5_1

transport: InfiniBand (0)

fw_ver: 14.32.1010

node_guid: 0c42:a103:007e:a9b9

sys_image_guid: 0c42:a103:007e:a9b8

vendor_id: 0x02c9

vendor_part_id: = 4117

hw_ver: 0x0

board_id: MT_2420110034

phys_port= _cnt: 1

port: 1

state:<= span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> PORT_ACTIVE (= 4)

max_mtu: 4096 (5)

= active_mtu: <= /span>1024 (3)

sm_lid: 0

= port_lid:= 0

port_lmc: 0x00

= l= ink_layer: <= /span> Ethernet


On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber <<= a href=3D"mailto:erezferber@gmail.com">erezferber@gmail.com> wrote:<= br>
Hi,

Based on your output, the ConnectX-4LX device is co= nfigured in LAG mode managed via the kernel bonding scripts. In this mode, = both physical functions share a single port (mlx5_bond_0). You should only = probe the first PCI BDF - 01:00.0, not the 2nd one.

By the way, the --dpdk installation flag should not be necessary, it is a= n old flag=C2=A0keps for Mellanox OFED builds lower than 5.x.
Regards,
Erez

On Tue, 5 Apr 2022 at 19:17, fwefe= w 4t4tg <7532ya= hoo@gmail.com> wrote:
I built the current version of DPDK directly = from dpdk.org after I ins= talled the current OFED Mellanox driver set:

* MLNX_OFED_LINUX-5.5-1= .0.3.2-ubuntu20.04-x86_64.iso

with ./install --dpdk

I am usin= g a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which is Ethernet = only; there is no IB mode for this NIC. This is a MT_2420110034 board. Howe= ver, when I run dpdk-testpmd I see "No Verbs device matches PCI device= 0000:01:00.1, are kernel drivers loaded?"

EAL: Detected CPU lc= ores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of = DPDK
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugepa= ges reported on node 0
EAL: VFIO support initialized
EAL: Probe PCI d= river: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0)
mlx5_common:= No Verbs device matches PCI device 0000:01:00.1, are kernel drivers loaded= ? =C2=A0
mlx5_common: Verbs device not found: 01:00.1
mlx5_common: Fa= iled to initialize device context.
EAL: Requested device 0000:01:00.1 ca= nnot be used
EAL: Bus (pci) probe failed.

As far as I can see all= the kernel modules are loaded:

lsmod | egrep "(ib|mlx)" |= sort
ib_cm =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A053248 =C2=A02 rdma_cm,ib_ipoib
ib_core =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 368640 =C2=A08 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm= ,ib_uverbs,mlx5_ib,ib_cm
ib_ipoib =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0135168 =C2=A00
ib_umad =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A024576 =C2=A00
ib_uverbs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 139264 =C2=A02 rdma_ucm,mlx5_ib
libahci =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A036864 =C2=A01 ahci
libcrc32c =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A016384 =C2=A02 btrfs,raid456
mlx= 5_core =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01634304 =C2=A01 mlx5_ib
= mlx5_ib =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 397312 =C2=A00
= mlx_compat =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 69632 =C2=A011 rdma_cm= ,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,ml= x5_core
mlxdevm =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 172032 = =C2=A01 mlx5_core
mlxfw =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A032768 =C2=A01 mlx5_core
pci_hyperv_intf =C2=A0 =C2=A0 =C2= =A0 =C2=A016384 =C2=A01 mlx5_core
psample =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A020480 =C2=A01 mlx5_core
tls =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A094208 =C2=A02 bonding,m= lx5_core

root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status = -v
MST modules:
------------
=C2=A0 =C2=A0 MST PCI module is not l= oaded
=C2=A0 =C2=A0 MST PCI configuration module loaded
PCI devices:<= br>------------
DEVICE_TYPE =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 MS= T =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 PCI =C2=A0 =C2=A0 =C2=A0 RDMA =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0NET =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 NUMA =C2=A0
ConnectX4LX(rev:0) =C2=A0 =C2=A0 =C2=A0= /dev/mst/mt4117_pciconf0.1 =C2=A0 =C2=A001:00.1 =C2=A0 mlx5_bond_0 =C2=A0 = =C2=A0 net-bond0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -1= =C2=A0 =C2=A0

ConnectX4LX(rev:0) =C2=A0 =C2=A0 =C2=A0/dev/mst/mt411= 7_pciconf0 =C2=A0 =C2=A0 =C2=A001:00.0 =C2=A0 mlx5_bond_0 =C2=A0 =C2=A0 net= -bond0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -1=C2=A0 =C2= =A0=C2=A0
--00000000000080bc2b05dbee744c--