We'll need to check, but how do you want to proceed? You either need 19.11 LTS or 20.11 LTS to work properly. Regards, Asaf Penso ________________________________ From: Rocio Dominguez Sent: Monday, January 31, 2022 2:01:43 PM To: Asaf Penso ; NBU-Contact-Thomas Monjalon (EXTERNAL) Cc: users@dpdk.org ; Matan Azrad ; Slava Ovsiienko ; Raslan Darawsheh Subject: RE: net_mlx5: unable to recognize master/representors on the multiple IB devices Hi Asaf, Yes, it seems that DPDK version 20.08 code is built-in with the VNF I’m deploying, so it is always using this version, which apparently doesn’t have the patch that overrides this error. I think the patch is the following: https://patches.dpdk.org/project/dpdk/patch/20200603150602.4686-7-ophirmu@mellanox.com/ and the code part that solves the error is: + if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) { + DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" + " driver."); + return 1; + } Could you please confirm? Thanks, Rocío From: Asaf Penso Sent: Monday, January 31, 2022 12:49 PM To: Rocio Dominguez ; NBU-Contact-Thomas Monjalon (EXTERNAL) Cc: users@dpdk.org; Matan Azrad ; Slava Ovsiienko ; Raslan Darawsheh Subject: RE: net_mlx5: unable to recognize master/representors on the multiple IB devices I see two differences below. First, in testpmd the version is 19.11.11, and in your application, it’s 20.08. See this print: {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] USER1: DPDK version: DPDK 20.08.0"} Second, in your application, I see the VFIO driver is not started properly: 20T19:19:16.637+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: cannot open VFIO container, error 2 (No such file or directory)"} Regards, Asaf Penso From: Rocio Dominguez > Sent: Thursday, January 20, 2022 9:49 PM To: Asaf Penso >; NBU-Contact-Thomas Monjalon (EXTERNAL) > Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: RE: net_mlx5: unable to recognize master/representors on the multiple IB devices Hi Asaf, I have manually compile and install the DPDK 19.11.11. Executing testpmd in the Mellanox NICs VFs where I want to run my app gives this result: pcgwpod009-c04:~/dpdk-stable-19.11.11 # ./x86_64-native-linux-gcc/app/testpmd -l 8-15 -n 4 -w d8:00.2 -w d8:00.3 -w d8:00.4 -w d8:00.5 -- --rxq=2 --txq=2 -i EAL: Detected 96 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:d8:00.2 on NUMA socket 1 EAL: probe driver: 15b3:1014 net_mlx5 EAL: PCI device 0000:d8:00.3 on NUMA socket 1 EAL: probe driver: 15b3:1014 net_mlx5 EAL: PCI device 0000:d8:00.4 on NUMA socket 1 EAL: probe driver: 15b3:1014 net_mlx5 EAL: PCI device 0000:d8:00.5 on NUMA socket 1 EAL: probe driver: 15b3:1014 net_mlx5 Interactive-mode selected testpmd: create a new mbuf pool : n=203456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc testpmd: create a new mbuf pool : n=203456, size=2176, socket=1 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 1) Port 0: 36:FE:F0:D2:90:27 Configuring Port 1 (socket 1) Port 1: 72:AC:33:BF:0A:FA Configuring Port 2 (socket 1) Port 2: 1E:8D:81:60:43:E0 Configuring Port 3 (socket 1) Port 3: C2:3C:EA:94:06:B4 Checking link statuses... Done testpmd> But when I run my Data Plane app, the result is {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.609+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"message":"[pktio_linux_packet_mmap_setup] block_size: 67108864, frame_size: 4096, block_nr: 1, frame_nr: 16384, mem_size: 67108864"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pktio_libpio_init] CTRL: pci devices added: 1, vhost user devices added: 0"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"message":"[add_pio_pci_devices_from_env_to_config] pci device from PCIDEVICE_MELLANOX_COM_MLNX_SRIOV_NETDEVICE=0000:d8:00.5 found"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pktio_libpio_init] CTRL: requesting 1024 MiB of hugepage memory for DPDK"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] USER1: DPDK version: DPDK 20.08.0"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] USER1: rte_eal_init() args: pio -m 1024 -n 4 --no-telemetry --file-prefix pio-0 --master-lcore=4 --lcores=4@(4) --pci-whitelist 0000:d8:00.5 --base-virtaddr=0x200000000 --iova-mode=va --legacy-mem --no-shconf "} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.618+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: Detected 96 lcore(s)"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.618+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: Detected 2 NUMA nodes"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.636+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: Selected IOVA mode 'VA'"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: Probing VFIO support..."} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: cannot open VFIO container, error 2 (No such file or directory)"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: VFIO support could not be initialized"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.567+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: Probe PCI driver: mlx5_pci (15b3:1014) device: 0000:d8:00.5 (socket 1)"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] net_mlx5: unable to recognize master/representors on the multiple IB devices"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] common_mlx5: Failed to load driver = net_mlx5."} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: Requested device 0000:d8:00.5 cannot be used"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] EAL: Bus (pci) probe failed."} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pio] USER1: ports init fail in DPDK, expect 1 ports, actual 0 ports."} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"error","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pktio_libpio_init] No network ports could be enabled!"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pktio_init_cpu] libpio packet module is NOT initialized"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pktio_init_cpu] pktsock packet module is NOT initialized"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pktio_init_cpu] linux packet module is initialized"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"message":"[pktio_init_cpu] tap packet module is NOT initialized"} Any idea on what could be the problem? Thanks, Rocío From: Asaf Penso > Sent: Thursday, January 20, 2022 8:17 AM To: Rocio Dominguez >; NBU-Contact-Thomas Monjalon (EXTERNAL) > Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: Re: net_mlx5: unable to recognize master/representors on the multiple IB devices Although inbox drivers come with a pre installed DPDK, you can manually download, compile, install, and work with whatever version you wish. Let us know the results, and we'll continue from there. Regards, Asaf Penso ________________________________ From: Rocio Dominguez > Sent: Monday, January 17, 2022 10:20:58 PM To: Asaf Penso >; NBU-Contact-Thomas Monjalon (EXTERNAL) > Cc: users@dpdk.org >; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: RE: net_mlx5: unable to recognize master/representors on the multiple IB devices Hi Asaf, Thanks for the prompt answer. I have checked that the latest 19.11 LTS is 19.11.11, but in OpenSUSE repositories the corresponding RPM package for SLES 15 SP2 is not available, the latest one is DPDK 19.11.10. I have installed it but the problem persists. It's probably solved in 19.11.11. There is a RPM package in SLES 15 SP3 for DPDK 20.11.3, which is LTS also, not sure if it could be a problem to install it in SLES 15 SP2. I will try it anyway. Also I will try to find other way to load the 19.11.11 in SLES 15 SP2 apart from using RPM or zipper, any suggestion is appreciated. Thanks, Rocío -----Original Message----- From: Asaf Penso > Sent: Sunday, January 16, 2022 4:31 PM To: NBU-Contact-Thomas Monjalon (EXTERNAL) >; Rocio Dominguez > Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: RE: net_mlx5: unable to recognize master/representors on the multiple IB devices Hello Rocio, IIRC, there was a fix in a recent stable version. Would you please try taking latest 19.11 LTS and tell whether you still see the issue? Regards, Asaf Penso >-----Original Message----- >From: Thomas Monjalon > >Sent: Sunday, January 16, 2022 3:24 PM >To: Rocio Dominguez > >Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >>; Raslan Darawsheh > >Subject: Re: net_mlx5: unable to recognize master/representors on the >multiple IB devices > >+Cc mlx5 experts > > >14/01/2022 11:10, Rocio Dominguez: >> Hi, >> >> I'm doing a setup with Mellanox ConnectX-4 (MCX416A-CCA) NICs. >> >> I'm using: >> >> OS SLES 15 SP2 >> DPDK 19.11.4 (the official supported version for SLES 15 SP2) >> MLNX_OFED_LINUX-5.5-1.0.3.2-sles15sp2-x86_64 (the latest one) >> Mellanox adapters firmware 12.28.2006 (corresponding to this >> MLNX_OFED version) kernel 5.3.18-24.34-default >> >> >> This is my SRIOV configuration for DPDK capable PCI slots: >> >> { >> "resourceName": "mlnx_sriov_netdevice", >> "resourcePrefix": "mellanox.com", >> "isRdma": true, >> "selectors": { >> "vendors": ["15b3"], >> "devices": ["1014"], >> "drivers": ["mlx5_core"], >> "pciAddresses": ["0000:d8:00.2", "0000:d8:00.3", >> "0000:d8:00.4", >"0000:d8:00.5"], >> "isRdma": true >> } >> >> The sriov device plugin starts without problems, the devices are >> correctly >allocated: >> >> { >> "cpu": "92", >> "ephemeral-storage": "419533922385", >> "hugepages-1Gi": "8Gi", >> "hugepages-2Mi": "4Gi", >> "intel.com/intel_sriov_dpdk": "0", >> "intel.com/sriov_cre": "3", >> "mellanox.com/mlnx_sriov_netdevice": "4", >> "mellanox.com/sriov_dp": "0", >> "memory": "183870336Ki", >> "pods": "110" >> } >> >> The Mellanox NICs are binded to the kernel driver mlx5_core: >> >> pcgwpod009-c04:~ # dpdk-devbind --status >> >> Network devices using kernel driver >> =================================== >> 0000:18:00.0 'Ethernet Controller 10G X550T 1563' if=em1 drv=ixgbe >> unused=vfio-pci >> 0000:18:00.1 'Ethernet Controller 10G X550T 1563' if=em2 drv=ixgbe >> unused=vfio-pci >> 0000:19:00.0 'Ethernet Controller 10G X550T 1563' if=em3 drv=ixgbe >> unused=vfio-pci >> 0000:19:00.1 'Ethernet Controller 10G X550T 1563' if=em4 drv=ixgbe >> unused=vfio-pci >> 0000:3b:00.0 'MT27700 Family [ConnectX-4] 1013' if=enp59s0f0 >> drv=mlx5_core unused=vfio-pci >> 0000:3b:00.1 'MT27700 Family [ConnectX-4] 1013' if=enp59s0f1 >> drv=mlx5_core unused=vfio-pci >> 0000:5e:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=p3p1 drv=ixgbe unused=vfio-pci >> 0000:5e:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=p3p2 drv=ixgbe unused=vfio-pci >> 0000:5e:10.0 '82599 Ethernet Controller Virtual Function 10ed' if= >> drv=ixgbevf unused=vfio-pci >> 0000:5e:10.2 '82599 Ethernet Controller Virtual Function 10ed' >> if=p3p1_1 drv=ixgbevf unused=vfio-pci >> 0000:5e:10.4 '82599 Ethernet Controller Virtual Function 10ed' if= >> drv=ixgbevf unused=vfio-pci >> 0000:5e:10.6 '82599 Ethernet Controller Virtual Function 10ed' >> if=p3p1_3 drv=ixgbevf unused=vfio-pci >> 0000:af:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=p4p1 drv=ixgbe unused=vfio-pci >> 0000:af:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=p4p2 drv=ixgbe unused=vfio-pci >> 0000:d8:00.0 'MT27700 Family [ConnectX-4] 1013' if=enp216s0f0 >> drv=mlx5_core unused=vfio-pci >> 0000:d8:00.1 'MT27700 Family [ConnectX-4] 1013' if=enp216s0f1 >> drv=mlx5_core unused=vfio-pci >> 0000:d8:00.2 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=enp216s0f2 drv=mlx5_core unused=vfio-pci >> 0000:d8:00.3 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=enp216s0f3 drv=mlx5_core unused=vfio-pci >> 0000:d8:00.4 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=enp216s0f4 drv=mlx5_core unused=vfio-pci >> 0000:d8:00.5 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=enp216s0f5 drv=mlx5_core unused=vfio-pci >> >> The interfaces are up: >> >> pcgwpod009-c04:~ # ibdev2netdev -v >> 0000:3b:00.0 mlx5_0 (MT4115 - MT1646K01301) CX416A - ConnectX-4 >QSFP28 >> fw 12.28.2006 port 1 (ACTIVE) ==> enp59s0f0 (Up) >> 0000:3b:00.1 mlx5_1 (MT4115 - MT1646K01301) CX416A - ConnectX-4 >QSFP28 >> fw 12.28.2006 port 1 (ACTIVE) ==> enp59s0f1 (Up) >> 0000:d8:00.0 mlx5_2 (MT4115 - MT1646K00538) CX416A - ConnectX-4 >QSFP28 >> fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f0 (Up) >> 0000:d8:00.1 mlx5_3 (MT4115 - MT1646K00538) CX416A - ConnectX-4 >QSFP28 >> fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f1 (Up) >> 0000:d8:00.2 mlx5_4 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) ==> >> enp216s0f2 (Up) >> 0000:d8:00.3 mlx5_5 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) ==> >> enp216s0f3 (Up) >> 0000:d8:00.4 mlx5_6 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) ==> >> enp216s0f4 (Up) >> 0000:d8:00.5 mlx5_7 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) ==> >> enp216s0f5 (Up) pcgwpod009-c04:~ # >> >> >> But when I run my application the Mellanox adapters are probed and I >obtain the following error: >> >> {"proc_id":"6"},"message":"[pio] EAL: Probe PCI driver: mlx5_pci >> (15b3:1014) device: 0000:d8:00.4 (socket 1)"} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] net_mlx5: unable to recognize >> master/representors on the multiple IB devices"} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] common_mlx5: Failed to load driver = >> net_mlx5."} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] EAL: Requested device 0000:d8:00.4 cannot be >> used"} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] EAL: Bus (pci) probe failed."} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] USER1: ports init fail in DPDK, expect 1 ports, >> actual 0 ports."} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","sever >> i >> ty":"error","service_id":"eric-pc-up-data-plane","metadata":{"proc_id" >> :"6"},"message":"[pktio_libpio_init] No network ports could be >> enabled!"} >> >> Could you please help me with this issue? >> >> >> Thanks, >> >> Rocío >> > > > >