* mlx5_common: No Verbs device matches PCI device 0000:01:00.1
@ 2022-04-05 16:17 fwefew 4t4tg
2022-04-05 16:59 ` Erez Ferber
0 siblings, 1 reply; 5+ messages in thread
From: fwefew 4t4tg @ 2022-04-05 16:17 UTC (permalink / raw)
To: users
[-- Attachment #1: Type: text/plain, Size: 2363 bytes --]
I built the current version of DPDK directly from dpdk.org after I
installed the current OFED Mellanox driver set:
* MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso
with ./install --dpdk
I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which is
Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034
board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI
device 0000:01:00.1, are kernel drivers loaded?"
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0)
mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are kernel
drivers loaded?
mlx5_common: Verbs device not found: 01:00.1
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:01:00.1 cannot be used
EAL: Bus (pci) probe failed.
As far as I can see all the kernel modules are loaded:
lsmod | egrep "(ib|mlx)" | sort
ib_cm 53248 2 rdma_cm,ib_ipoib
ib_core 368640 8
rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
ib_ipoib 135168 0
ib_umad 24576 0
ib_uverbs 139264 2 rdma_ucm,mlx5_ib
libahci 36864 1 ahci
libcrc32c 16384 2 btrfs,raid456
mlx5_core 1634304 1 mlx5_ib
mlx5_ib 397312 0
mlx_compat 69632 11
rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
mlxdevm 172032 1 mlx5_core
mlxfw 32768 1 mlx5_core
pci_hyperv_intf 16384 1 mlx5_core
psample 20480 1 mlx5_core
tls 94208 2 bonding,mlx5_core
root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA
NET NUMA
ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1 mlx5_bond_0
net-bond0 -1
ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 mlx5_bond_0
net-bond0 -1
[-- Attachment #2: Type: text/html, Size: 2822 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1
2022-04-05 16:17 mlx5_common: No Verbs device matches PCI device 0000:01:00.1 fwefew 4t4tg
@ 2022-04-05 16:59 ` Erez Ferber
2022-04-05 20:54 ` fwefew 4t4tg
0 siblings, 1 reply; 5+ messages in thread
From: Erez Ferber @ 2022-04-05 16:59 UTC (permalink / raw)
To: fwefew 4t4tg; +Cc: users
[-- Attachment #1: Type: text/plain, Size: 2952 bytes --]
Hi,
Based on your output, the ConnectX-4LX device is configured in LAG mode
managed via the kernel bonding scripts. In this mode, both physical
functions share a single port (mlx5_bond_0). You should only probe the
first PCI BDF - 01:00.0, not the 2nd one.
By the way, the --dpdk installation flag should not be necessary, it is an
old flag keps for Mellanox OFED builds lower than 5.x.
Regards,
Erez
On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532yahoo@gmail.com> wrote:
> I built the current version of DPDK directly from dpdk.org after I
> installed the current OFED Mellanox driver set:
>
> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso
>
> with ./install --dpdk
>
> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which is
> Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034
> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI
> device 0000:01:00.1, are kernel drivers loaded?"
>
> EAL: Detected CPU lcores: 16
> EAL: Detected NUMA nodes: 1
> EAL: Detected static linkage of DPDK
> EAL: Selected IOVA mode 'PA'
> EAL: No free 2048 kB hugepages reported on node 0
> EAL: VFIO support initialized
> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0)
> mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are kernel
> drivers loaded?
> mlx5_common: Verbs device not found: 01:00.1
> mlx5_common: Failed to initialize device context.
> EAL: Requested device 0000:01:00.1 cannot be used
> EAL: Bus (pci) probe failed.
>
> As far as I can see all the kernel modules are loaded:
>
> lsmod | egrep "(ib|mlx)" | sort
> ib_cm 53248 2 rdma_cm,ib_ipoib
> ib_core 368640 8
> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
> ib_ipoib 135168 0
> ib_umad 24576 0
> ib_uverbs 139264 2 rdma_ucm,mlx5_ib
> libahci 36864 1 ahci
> libcrc32c 16384 2 btrfs,raid456
> mlx5_core 1634304 1 mlx5_ib
> mlx5_ib 397312 0
> mlx_compat 69632 11
> rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
> mlxdevm 172032 1 mlx5_core
> mlxfw 32768 1 mlx5_core
> pci_hyperv_intf 16384 1 mlx5_core
> psample 20480 1 mlx5_core
> tls 94208 2 bonding,mlx5_core
>
> root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v
> MST modules:
> ------------
> MST PCI module is not loaded
> MST PCI configuration module loaded
> PCI devices:
> ------------
> DEVICE_TYPE MST PCI RDMA
> NET NUMA
> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1
> mlx5_bond_0 net-bond0 -1
>
> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0
> mlx5_bond_0 net-bond0 -1
>
[-- Attachment #2: Type: text/html, Size: 3677 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1
2022-04-05 16:59 ` Erez Ferber
@ 2022-04-05 20:54 ` fwefew 4t4tg
2022-04-07 8:22 ` Erez Ferber
0 siblings, 1 reply; 5+ messages in thread
From: fwefew 4t4tg @ 2022-04-05 20:54 UTC (permalink / raw)
To: Erez Ferber, users
[-- Attachment #1: Type: text/plain, Size: 5873 bytes --]
Mr. Ferber, much appreciated. I knew this metal box came to me with two
mellanox NICs bonded. I used their util to unbond it, but alas it did not
do it all the way. The /etc/network/interfaces was bad. I fixed the config
and rebooted.
Voila. Success. ibv_devinfo now shows two devices. And the DPDK app runs.
Your point about bonding nudged me to double check.
*Now, on a second machine where I believe I have done everything I did on
the first machine*, running the DPDK application continues to look for the
wrong driver:
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: libmlx4.so.1: cannot open shared object file: No such file or directory
EAL: FATAL: Cannot init plugins
EAL: Cannot init plugins
Somehow I fixed this on the first machine, but I cannot duplicate success
on the second machine.
Both machines report two devices; both machines show similar ifconifgs
device node GUID
------ ----------------
mlx5_0 0c42a103007ea9b8
mlx5_1 0c42a103007ea9b9
device node GUID
------ ----------------
mlx5_0 0c42a103007ea3ec
mlx5_1 0c42a103007ea3ed
root@server:~/Dev/reinvent/scripts# ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 0c42:a103:007e:a3ec
sys_image_guid: 0c42:a103:007e:a3ec
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110034
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 0c42:a103:007e:a3ed
sys_image_guid: 0c42:a103:007e:a3ec
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110034
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 0c42:a103:007e:a9b8
sys_image_guid: 0c42:a103:007e:a9b8
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110034
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 0c42:a103:007e:a9b9
sys_image_guid: 0c42:a103:007e:a9b8
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110034
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber <erezferber@gmail.com> wrote:
> Hi,
>
> Based on your output, the ConnectX-4LX device is configured in LAG mode
> managed via the kernel bonding scripts. In this mode, both physical
> functions share a single port (mlx5_bond_0). You should only probe the
> first PCI BDF - 01:00.0, not the 2nd one.
>
> By the way, the --dpdk installation flag should not be necessary, it is an
> old flag keps for Mellanox OFED builds lower than 5.x.
>
> Regards,
> Erez
>
> On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532yahoo@gmail.com> wrote:
>
>> I built the current version of DPDK directly from dpdk.org after I
>> installed the current OFED Mellanox driver set:
>>
>> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso
>>
>> with ./install --dpdk
>>
>> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which
>> is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034
>> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI
>> device 0000:01:00.1, are kernel drivers loaded?"
>>
>> EAL: Detected CPU lcores: 16
>> EAL: Detected NUMA nodes: 1
>> EAL: Detected static linkage of DPDK
>> EAL: Selected IOVA mode 'PA'
>> EAL: No free 2048 kB hugepages reported on node 0
>> EAL: VFIO support initialized
>> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket
>> 0)
>> mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are kernel
>> drivers loaded?
>> mlx5_common: Verbs device not found: 01:00.1
>> mlx5_common: Failed to initialize device context.
>> EAL: Requested device 0000:01:00.1 cannot be used
>> EAL: Bus (pci) probe failed.
>>
>> As far as I can see all the kernel modules are loaded:
>>
>> lsmod | egrep "(ib|mlx)" | sort
>> ib_cm 53248 2 rdma_cm,ib_ipoib
>> ib_core 368640 8
>> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
>> ib_ipoib 135168 0
>> ib_umad 24576 0
>> ib_uverbs 139264 2 rdma_ucm,mlx5_ib
>> libahci 36864 1 ahci
>> libcrc32c 16384 2 btrfs,raid456
>> mlx5_core 1634304 1 mlx5_ib
>> mlx5_ib 397312 0
>> mlx_compat 69632 11
>> rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
>> mlxdevm 172032 1 mlx5_core
>> mlxfw 32768 1 mlx5_core
>> pci_hyperv_intf 16384 1 mlx5_core
>> psample 20480 1 mlx5_core
>> tls 94208 2 bonding,mlx5_core
>>
>> root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v
>> MST modules:
>> ------------
>> MST PCI module is not loaded
>> MST PCI configuration module loaded
>> PCI devices:
>> ------------
>> DEVICE_TYPE MST PCI RDMA
>> NET NUMA
>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1
>> mlx5_bond_0 net-bond0 -1
>>
>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0
>> mlx5_bond_0 net-bond0 -1
>>
>
[-- Attachment #2: Type: text/html, Size: 48262 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1
2022-04-05 20:54 ` fwefew 4t4tg
@ 2022-04-07 8:22 ` Erez Ferber
2022-04-07 22:58 ` fwefew 4t4tg
0 siblings, 1 reply; 5+ messages in thread
From: Erez Ferber @ 2022-04-07 8:22 UTC (permalink / raw)
To: fwefew 4t4tg; +Cc: users
[-- Attachment #1: Type: text/plain, Size: 6530 bytes --]
I assume your tree assumes there's a ConnectX-3 device installed, while the
kernel driver doesnt support it for quite a while
I would suggest re-compiling while excluding mlx4 PMD support.
Thanks,
Erez
On Tue, 5 Apr 2022 at 23:54, fwefew 4t4tg <7532yahoo@gmail.com> wrote:
> Mr. Ferber, much appreciated. I knew this metal box came to me with two
> mellanox NICs bonded. I used their util to unbond it, but alas it did not
> do it all the way. The /etc/network/interfaces was bad. I fixed the config
> and rebooted.
>
> Voila. Success. ibv_devinfo now shows two devices. And the DPDK app runs.
>
> Your point about bonding nudged me to double check.
>
> *Now, on a second machine where I believe I have done everything I did on
> the first machine*, running the DPDK application continues to look for
> the wrong driver:
>
> EAL: Detected CPU lcores: 16
> EAL: Detected NUMA nodes: 1
> EAL: Detected shared linkage of DPDK
> EAL: libmlx4.so.1: cannot open shared object file: No such file or
> directory
> EAL: FATAL: Cannot init plugins
> EAL: Cannot init plugins
>
> Somehow I fixed this on the first machine, but I cannot duplicate success
> on the second machine.
>
> Both machines report two devices; both machines show similar ifconifgs
>
> device node GUID
>
> ------ ----------------
>
> mlx5_0 0c42a103007ea9b8
>
> mlx5_1 0c42a103007ea9b9
>
>
> device node GUID
>
> ------ ----------------
>
> mlx5_0 0c42a103007ea3ec
>
> mlx5_1 0c42a103007ea3ed
>
>
> root@server:~/Dev/reinvent/scripts# ibv_devinfo
>
> hca_id: mlx5_0
>
> transport: InfiniBand (0)
>
> fw_ver: 14.32.1010
>
> node_guid: 0c42:a103:007e:a3ec
>
> sys_image_guid: 0c42:a103:007e:a3ec
>
> vendor_id: 0x02c9
>
> vendor_part_id: 4117
>
> hw_ver: 0x0
>
> board_id: MT_2420110034
>
> phys_port_cnt: 1
>
> port: 1
>
> state: PORT_ACTIVE (4)
>
> max_mtu: 4096 (5)
>
> active_mtu: 1024 (3)
>
> sm_lid: 0
>
> port_lid: 0
>
> port_lmc: 0x00
>
> link_layer: Ethernet
>
>
> hca_id: mlx5_1
>
> transport: InfiniBand (0)
>
> fw_ver: 14.32.1010
>
> node_guid: 0c42:a103:007e:a3ed
>
> sys_image_guid: 0c42:a103:007e:a3ec
>
> vendor_id: 0x02c9
>
> vendor_part_id: 4117
>
> hw_ver: 0x0
>
> board_id: MT_2420110034
>
> phys_port_cnt: 1
>
> port: 1
>
> state: PORT_ACTIVE (4)
>
> max_mtu: 4096 (5)
>
> active_mtu: 1024 (3)
>
> sm_lid: 0
>
> port_lid: 0
>
> port_lmc: 0x00
>
> link_layer: Ethernet
>
>
>
>
>
> ibv_devinfo
>
> hca_id: mlx5_0
>
> transport: InfiniBand (0)
>
> fw_ver: 14.32.1010
>
> node_guid: 0c42:a103:007e:a9b8
>
> sys_image_guid: 0c42:a103:007e:a9b8
>
> vendor_id: 0x02c9
>
> vendor_part_id: 4117
>
> hw_ver: 0x0
>
> board_id: MT_2420110034
>
> phys_port_cnt: 1
>
> port: 1
>
> state: PORT_ACTIVE (4)
>
> max_mtu: 4096 (5)
>
> active_mtu: 1024 (3)
>
> sm_lid: 0
>
> port_lid: 0
>
> port_lmc: 0x00
>
> link_layer: Ethernet
>
>
> hca_id: mlx5_1
>
> transport: InfiniBand (0)
>
> fw_ver: 14.32.1010
>
> node_guid: 0c42:a103:007e:a9b9
>
> sys_image_guid: 0c42:a103:007e:a9b8
>
> vendor_id: 0x02c9
>
> vendor_part_id: 4117
>
> hw_ver: 0x0
>
> board_id: MT_2420110034
>
> phys_port_cnt: 1
>
> port: 1
>
> state: PORT_ACTIVE (4)
>
> max_mtu: 4096 (5)
>
> active_mtu: 1024 (3)
>
> sm_lid: 0
>
> port_lid: 0
>
> port_lmc: 0x00
>
> link_layer: Ethernet
>
> On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber <erezferber@gmail.com> wrote:
>
>> Hi,
>>
>> Based on your output, the ConnectX-4LX device is configured in LAG mode
>> managed via the kernel bonding scripts. In this mode, both physical
>> functions share a single port (mlx5_bond_0). You should only probe the
>> first PCI BDF - 01:00.0, not the 2nd one.
>>
>> By the way, the --dpdk installation flag should not be necessary, it is
>> an old flag keps for Mellanox OFED builds lower than 5.x.
>>
>> Regards,
>> Erez
>>
>> On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532yahoo@gmail.com> wrote:
>>
>>> I built the current version of DPDK directly from dpdk.org after I
>>> installed the current OFED Mellanox driver set:
>>>
>>> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso
>>>
>>> with ./install --dpdk
>>>
>>> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which
>>> is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034
>>> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI
>>> device 0000:01:00.1, are kernel drivers loaded?"
>>>
>>> EAL: Detected CPU lcores: 16
>>> EAL: Detected NUMA nodes: 1
>>> EAL: Detected static linkage of DPDK
>>> EAL: Selected IOVA mode 'PA'
>>> EAL: No free 2048 kB hugepages reported on node 0
>>> EAL: VFIO support initialized
>>> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket
>>> 0)
>>> mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are kernel
>>> drivers loaded?
>>> mlx5_common: Verbs device not found: 01:00.1
>>> mlx5_common: Failed to initialize device context.
>>> EAL: Requested device 0000:01:00.1 cannot be used
>>> EAL: Bus (pci) probe failed.
>>>
>>> As far as I can see all the kernel modules are loaded:
>>>
>>> lsmod | egrep "(ib|mlx)" | sort
>>> ib_cm 53248 2 rdma_cm,ib_ipoib
>>> ib_core 368640 8
>>> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
>>> ib_ipoib 135168 0
>>> ib_umad 24576 0
>>> ib_uverbs 139264 2 rdma_ucm,mlx5_ib
>>> libahci 36864 1 ahci
>>> libcrc32c 16384 2 btrfs,raid456
>>> mlx5_core 1634304 1 mlx5_ib
>>> mlx5_ib 397312 0
>>> mlx_compat 69632 11
>>> rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
>>> mlxdevm 172032 1 mlx5_core
>>> mlxfw 32768 1 mlx5_core
>>> pci_hyperv_intf 16384 1 mlx5_core
>>> psample 20480 1 mlx5_core
>>> tls 94208 2 bonding,mlx5_core
>>>
>>> root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v
>>> MST modules:
>>> ------------
>>> MST PCI module is not loaded
>>> MST PCI configuration module loaded
>>> PCI devices:
>>> ------------
>>> DEVICE_TYPE MST PCI RDMA
>>> NET NUMA
>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1
>>> mlx5_bond_0 net-bond0 -1
>>>
>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0
>>> mlx5_bond_0 net-bond0 -1
>>>
>>
[-- Attachment #2: Type: text/html, Size: 37399 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1
2022-04-07 8:22 ` Erez Ferber
@ 2022-04-07 22:58 ` fwefew 4t4tg
0 siblings, 0 replies; 5+ messages in thread
From: fwefew 4t4tg @ 2022-04-07 22:58 UTC (permalink / raw)
To: Erez Ferber; +Cc: users
[-- Attachment #1: Type: text/plain, Size: 7750 bytes --]
This was my error. I linked the code incorrectly. For those new to DPDK the
correct linking methodology is, using cmake, follows:
- build DPDK from dpdk.org. For example see
https://github.com/rodgarrison/reinvent then see
https://github.com/rodgarrison/reinvent/blob/main/scripts/install#L58.
While this write-up is for DPDK on AWS ENA NICs, building DPDK is the same
no matter what NIC you are using.
- configure your linking libraries with the help of pkg-config. I do it
this way
https://github.com/rodgarrison/reinvent/blob/ubuntu/CMakeLists.txt#L27 to
setup the DPDK linking command defaulting to dynamic libraries. But static
link is also possible; see else subcommand. Then, in the actual,
application use this setup, for example,
https://github.com/rodgarrison/reinvent/blob/ubuntu/integration_tests/reinvent_dpdk_udp/CMakeLists.txt#L43
On Thu, Apr 7, 2022 at 4:22 AM Erez Ferber <erezferber@gmail.com> wrote:
> I assume your tree assumes there's a ConnectX-3 device installed, while
> the kernel driver doesnt support it for quite a while
> I would suggest re-compiling while excluding mlx4 PMD support.
>
> Thanks,
> Erez
>
> On Tue, 5 Apr 2022 at 23:54, fwefew 4t4tg <7532yahoo@gmail.com> wrote:
>
>> Mr. Ferber, much appreciated. I knew this metal box came to me with two
>> mellanox NICs bonded. I used their util to unbond it, but alas it did not
>> do it all the way. The /etc/network/interfaces was bad. I fixed the config
>> and rebooted.
>>
>> Voila. Success. ibv_devinfo now shows two devices. And the DPDK app runs.
>>
>> Your point about bonding nudged me to double check.
>>
>> *Now, on a second machine where I believe I have done everything I did on
>> the first machine*, running the DPDK application continues to look for
>> the wrong driver:
>>
>> EAL: Detected CPU lcores: 16
>> EAL: Detected NUMA nodes: 1
>> EAL: Detected shared linkage of DPDK
>> EAL: libmlx4.so.1: cannot open shared object file: No such file or
>> directory
>> EAL: FATAL: Cannot init plugins
>> EAL: Cannot init plugins
>>
>> Somehow I fixed this on the first machine, but I cannot duplicate success
>> on the second machine.
>>
>> Both machines report two devices; both machines show similar ifconifgs
>>
>> device node GUID
>>
>> ------ ----------------
>>
>> mlx5_0 0c42a103007ea9b8
>>
>> mlx5_1 0c42a103007ea9b9
>>
>>
>> device node GUID
>>
>> ------ ----------------
>>
>> mlx5_0 0c42a103007ea3ec
>>
>> mlx5_1 0c42a103007ea3ed
>>
>>
>> root@server:~/Dev/reinvent/scripts# ibv_devinfo
>>
>> hca_id: mlx5_0
>>
>> transport: InfiniBand (0)
>>
>> fw_ver: 14.32.1010
>>
>> node_guid: 0c42:a103:007e:a3ec
>>
>> sys_image_guid: 0c42:a103:007e:a3ec
>>
>> vendor_id: 0x02c9
>>
>> vendor_part_id: 4117
>>
>> hw_ver: 0x0
>>
>> board_id: MT_2420110034
>>
>> phys_port_cnt: 1
>>
>> port: 1
>>
>> state: PORT_ACTIVE (4)
>>
>> max_mtu: 4096 (5)
>>
>> active_mtu: 1024 (3)
>>
>> sm_lid: 0
>>
>> port_lid: 0
>>
>> port_lmc: 0x00
>>
>> link_layer: Ethernet
>>
>>
>> hca_id: mlx5_1
>>
>> transport: InfiniBand (0)
>>
>> fw_ver: 14.32.1010
>>
>> node_guid: 0c42:a103:007e:a3ed
>>
>> sys_image_guid: 0c42:a103:007e:a3ec
>>
>> vendor_id: 0x02c9
>>
>> vendor_part_id: 4117
>>
>> hw_ver: 0x0
>>
>> board_id: MT_2420110034
>>
>> phys_port_cnt: 1
>>
>> port: 1
>>
>> state: PORT_ACTIVE (4)
>>
>> max_mtu: 4096 (5)
>>
>> active_mtu: 1024 (3)
>>
>> sm_lid: 0
>>
>> port_lid: 0
>>
>> port_lmc: 0x00
>>
>> link_layer: Ethernet
>>
>>
>>
>>
>>
>> ibv_devinfo
>>
>> hca_id: mlx5_0
>>
>> transport: InfiniBand (0)
>>
>> fw_ver: 14.32.1010
>>
>> node_guid: 0c42:a103:007e:a9b8
>>
>> sys_image_guid: 0c42:a103:007e:a9b8
>>
>> vendor_id: 0x02c9
>>
>> vendor_part_id: 4117
>>
>> hw_ver: 0x0
>>
>> board_id: MT_2420110034
>>
>> phys_port_cnt: 1
>>
>> port: 1
>>
>> state: PORT_ACTIVE (4)
>>
>> max_mtu: 4096 (5)
>>
>> active_mtu: 1024 (3)
>>
>> sm_lid: 0
>>
>> port_lid: 0
>>
>> port_lmc: 0x00
>>
>> link_layer: Ethernet
>>
>>
>> hca_id: mlx5_1
>>
>> transport: InfiniBand (0)
>>
>> fw_ver: 14.32.1010
>>
>> node_guid: 0c42:a103:007e:a9b9
>>
>> sys_image_guid: 0c42:a103:007e:a9b8
>>
>> vendor_id: 0x02c9
>>
>> vendor_part_id: 4117
>>
>> hw_ver: 0x0
>>
>> board_id: MT_2420110034
>>
>> phys_port_cnt: 1
>>
>> port: 1
>>
>> state: PORT_ACTIVE (4)
>>
>> max_mtu: 4096 (5)
>>
>> active_mtu: 1024 (3)
>>
>> sm_lid: 0
>>
>> port_lid: 0
>>
>> port_lmc: 0x00
>>
>> link_layer: Ethernet
>>
>> On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber <erezferber@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Based on your output, the ConnectX-4LX device is configured in LAG mode
>>> managed via the kernel bonding scripts. In this mode, both physical
>>> functions share a single port (mlx5_bond_0). You should only probe the
>>> first PCI BDF - 01:00.0, not the 2nd one.
>>>
>>> By the way, the --dpdk installation flag should not be necessary, it is
>>> an old flag keps for Mellanox OFED builds lower than 5.x.
>>>
>>> Regards,
>>> Erez
>>>
>>> On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532yahoo@gmail.com> wrote:
>>>
>>>> I built the current version of DPDK directly from dpdk.org after I
>>>> installed the current OFED Mellanox driver set:
>>>>
>>>> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso
>>>>
>>>> with ./install --dpdk
>>>>
>>>> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which
>>>> is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034
>>>> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI
>>>> device 0000:01:00.1, are kernel drivers loaded?"
>>>>
>>>> EAL: Detected CPU lcores: 16
>>>> EAL: Detected NUMA nodes: 1
>>>> EAL: Detected static linkage of DPDK
>>>> EAL: Selected IOVA mode 'PA'
>>>> EAL: No free 2048 kB hugepages reported on node 0
>>>> EAL: VFIO support initialized
>>>> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1
>>>> (socket 0)
>>>> mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are
>>>> kernel drivers loaded?
>>>> mlx5_common: Verbs device not found: 01:00.1
>>>> mlx5_common: Failed to initialize device context.
>>>> EAL: Requested device 0000:01:00.1 cannot be used
>>>> EAL: Bus (pci) probe failed.
>>>>
>>>> As far as I can see all the kernel modules are loaded:
>>>>
>>>> lsmod | egrep "(ib|mlx)" | sort
>>>> ib_cm 53248 2 rdma_cm,ib_ipoib
>>>> ib_core 368640 8
>>>> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
>>>> ib_ipoib 135168 0
>>>> ib_umad 24576 0
>>>> ib_uverbs 139264 2 rdma_ucm,mlx5_ib
>>>> libahci 36864 1 ahci
>>>> libcrc32c 16384 2 btrfs,raid456
>>>> mlx5_core 1634304 1 mlx5_ib
>>>> mlx5_ib 397312 0
>>>> mlx_compat 69632 11
>>>> rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
>>>> mlxdevm 172032 1 mlx5_core
>>>> mlxfw 32768 1 mlx5_core
>>>> pci_hyperv_intf 16384 1 mlx5_core
>>>> psample 20480 1 mlx5_core
>>>> tls 94208 2 bonding,mlx5_core
>>>>
>>>> root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v
>>>> MST modules:
>>>> ------------
>>>> MST PCI module is not loaded
>>>> MST PCI configuration module loaded
>>>> PCI devices:
>>>> ------------
>>>> DEVICE_TYPE MST PCI RDMA
>>>> NET NUMA
>>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1
>>>> mlx5_bond_0 net-bond0 -1
>>>>
>>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0
>>>> mlx5_bond_0 net-bond0 -1
>>>>
>>>
[-- Attachment #2: Type: text/html, Size: 39018 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-04-07 22:58 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-05 16:17 mlx5_common: No Verbs device matches PCI device 0000:01:00.1 fwefew 4t4tg
2022-04-05 16:59 ` Erez Ferber
2022-04-05 20:54 ` fwefew 4t4tg
2022-04-07 8:22 ` Erez Ferber
2022-04-07 22:58 ` fwefew 4t4tg
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).