This was my error. I linked the code incorrectly. For those new to DPDK the correct linking methodology is, using cmake, follows: - build DPDK from dpdk.org. For example see https://github.com/rodgarrison/reinvent then see https://github.com/rodgarrison/reinvent/blob/main/scripts/install#L58. While this write-up is for DPDK on AWS ENA NICs, building DPDK is the same no matter what NIC you are using. - configure your linking libraries with the help of pkg-config. I do it this way https://github.com/rodgarrison/reinvent/blob/ubuntu/CMakeLists.txt#L27 to setup the DPDK linking command defaulting to dynamic libraries. But static link is also possible; see else subcommand. Then, in the actual, application use this setup, for example, https://github.com/rodgarrison/reinvent/blob/ubuntu/integration_tests/reinvent_dpdk_udp/CMakeLists.txt#L43 On Thu, Apr 7, 2022 at 4:22 AM Erez Ferber wrote: > I assume your tree assumes there's a ConnectX-3 device installed, while > the kernel driver doesnt support it for quite a while > I would suggest re-compiling while excluding mlx4 PMD support. > > Thanks, > Erez > > On Tue, 5 Apr 2022 at 23:54, fwefew 4t4tg <7532yahoo@gmail.com> wrote: > >> Mr. Ferber, much appreciated. I knew this metal box came to me with two >> mellanox NICs bonded. I used their util to unbond it, but alas it did not >> do it all the way. The /etc/network/interfaces was bad. I fixed the config >> and rebooted. >> >> Voila. Success. ibv_devinfo now shows two devices. And the DPDK app runs. >> >> Your point about bonding nudged me to double check. >> >> *Now, on a second machine where I believe I have done everything I did on >> the first machine*, running the DPDK application continues to look for >> the wrong driver: >> >> EAL: Detected CPU lcores: 16 >> EAL: Detected NUMA nodes: 1 >> EAL: Detected shared linkage of DPDK >> EAL: libmlx4.so.1: cannot open shared object file: No such file or >> directory >> EAL: FATAL: Cannot init plugins >> EAL: Cannot init plugins >> >> Somehow I fixed this on the first machine, but I cannot duplicate success >> on the second machine. >> >> Both machines report two devices; both machines show similar ifconifgs >> >> device node GUID >> >> ------ ---------------- >> >> mlx5_0 0c42a103007ea9b8 >> >> mlx5_1 0c42a103007ea9b9 >> >> >> device node GUID >> >> ------ ---------------- >> >> mlx5_0 0c42a103007ea3ec >> >> mlx5_1 0c42a103007ea3ed >> >> >> root@server:~/Dev/reinvent/scripts# ibv_devinfo >> >> hca_id: mlx5_0 >> >> transport: InfiniBand (0) >> >> fw_ver: 14.32.1010 >> >> node_guid: 0c42:a103:007e:a3ec >> >> sys_image_guid: 0c42:a103:007e:a3ec >> >> vendor_id: 0x02c9 >> >> vendor_part_id: 4117 >> >> hw_ver: 0x0 >> >> board_id: MT_2420110034 >> >> phys_port_cnt: 1 >> >> port: 1 >> >> state: PORT_ACTIVE (4) >> >> max_mtu: 4096 (5) >> >> active_mtu: 1024 (3) >> >> sm_lid: 0 >> >> port_lid: 0 >> >> port_lmc: 0x00 >> >> link_layer: Ethernet >> >> >> hca_id: mlx5_1 >> >> transport: InfiniBand (0) >> >> fw_ver: 14.32.1010 >> >> node_guid: 0c42:a103:007e:a3ed >> >> sys_image_guid: 0c42:a103:007e:a3ec >> >> vendor_id: 0x02c9 >> >> vendor_part_id: 4117 >> >> hw_ver: 0x0 >> >> board_id: MT_2420110034 >> >> phys_port_cnt: 1 >> >> port: 1 >> >> state: PORT_ACTIVE (4) >> >> max_mtu: 4096 (5) >> >> active_mtu: 1024 (3) >> >> sm_lid: 0 >> >> port_lid: 0 >> >> port_lmc: 0x00 >> >> link_layer: Ethernet >> >> >> >> >> >> ibv_devinfo >> >> hca_id: mlx5_0 >> >> transport: InfiniBand (0) >> >> fw_ver: 14.32.1010 >> >> node_guid: 0c42:a103:007e:a9b8 >> >> sys_image_guid: 0c42:a103:007e:a9b8 >> >> vendor_id: 0x02c9 >> >> vendor_part_id: 4117 >> >> hw_ver: 0x0 >> >> board_id: MT_2420110034 >> >> phys_port_cnt: 1 >> >> port: 1 >> >> state: PORT_ACTIVE (4) >> >> max_mtu: 4096 (5) >> >> active_mtu: 1024 (3) >> >> sm_lid: 0 >> >> port_lid: 0 >> >> port_lmc: 0x00 >> >> link_layer: Ethernet >> >> >> hca_id: mlx5_1 >> >> transport: InfiniBand (0) >> >> fw_ver: 14.32.1010 >> >> node_guid: 0c42:a103:007e:a9b9 >> >> sys_image_guid: 0c42:a103:007e:a9b8 >> >> vendor_id: 0x02c9 >> >> vendor_part_id: 4117 >> >> hw_ver: 0x0 >> >> board_id: MT_2420110034 >> >> phys_port_cnt: 1 >> >> port: 1 >> >> state: PORT_ACTIVE (4) >> >> max_mtu: 4096 (5) >> >> active_mtu: 1024 (3) >> >> sm_lid: 0 >> >> port_lid: 0 >> >> port_lmc: 0x00 >> >> link_layer: Ethernet >> >> On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber wrote: >> >>> Hi, >>> >>> Based on your output, the ConnectX-4LX device is configured in LAG mode >>> managed via the kernel bonding scripts. In this mode, both physical >>> functions share a single port (mlx5_bond_0). You should only probe the >>> first PCI BDF - 01:00.0, not the 2nd one. >>> >>> By the way, the --dpdk installation flag should not be necessary, it is >>> an old flag keps for Mellanox OFED builds lower than 5.x. >>> >>> Regards, >>> Erez >>> >>> On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532yahoo@gmail.com> wrote: >>> >>>> I built the current version of DPDK directly from dpdk.org after I >>>> installed the current OFED Mellanox driver set: >>>> >>>> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso >>>> >>>> with ./install --dpdk >>>> >>>> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which >>>> is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034 >>>> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI >>>> device 0000:01:00.1, are kernel drivers loaded?" >>>> >>>> EAL: Detected CPU lcores: 16 >>>> EAL: Detected NUMA nodes: 1 >>>> EAL: Detected static linkage of DPDK >>>> EAL: Selected IOVA mode 'PA' >>>> EAL: No free 2048 kB hugepages reported on node 0 >>>> EAL: VFIO support initialized >>>> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 >>>> (socket 0) >>>> mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are >>>> kernel drivers loaded? >>>> mlx5_common: Verbs device not found: 01:00.1 >>>> mlx5_common: Failed to initialize device context. >>>> EAL: Requested device 0000:01:00.1 cannot be used >>>> EAL: Bus (pci) probe failed. >>>> >>>> As far as I can see all the kernel modules are loaded: >>>> >>>> lsmod | egrep "(ib|mlx)" | sort >>>> ib_cm 53248 2 rdma_cm,ib_ipoib >>>> ib_core 368640 8 >>>> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm >>>> ib_ipoib 135168 0 >>>> ib_umad 24576 0 >>>> ib_uverbs 139264 2 rdma_ucm,mlx5_ib >>>> libahci 36864 1 ahci >>>> libcrc32c 16384 2 btrfs,raid456 >>>> mlx5_core 1634304 1 mlx5_ib >>>> mlx5_ib 397312 0 >>>> mlx_compat 69632 11 >>>> rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core >>>> mlxdevm 172032 1 mlx5_core >>>> mlxfw 32768 1 mlx5_core >>>> pci_hyperv_intf 16384 1 mlx5_core >>>> psample 20480 1 mlx5_core >>>> tls 94208 2 bonding,mlx5_core >>>> >>>> root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v >>>> MST modules: >>>> ------------ >>>> MST PCI module is not loaded >>>> MST PCI configuration module loaded >>>> PCI devices: >>>> ------------ >>>> DEVICE_TYPE MST PCI RDMA >>>> NET NUMA >>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1 >>>> mlx5_bond_0 net-bond0 -1 >>>> >>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 >>>> mlx5_bond_0 net-bond0 -1 >>>> >>>