RE: net_mlx5: unable to recognize master/representors on the multiple IB devices

[Rocio Dominguez <rocio.dominguez@ericsson.com>]

Hi Asaf,

Thanks for the prompt answer.

I have checked that the latest 19.11 LTS is 19.11.11, but in OpenSUSE repositories the corresponding RPM package for SLES 15 SP2 is not available, the latest one is DPDK 19.11.10.

I have installed it but the problem persists. It's probably solved in 19.11.11.

There is a RPM package in SLES 15 SP3 for DPDK 20.11.3, which is LTS also, not sure if it could be a problem to install it in SLES 15 SP2. I will try it anyway.

Also I will try to find other way to load the 19.11.11 in SLES 15 SP2 apart from using RPM or zipper, any suggestion is appreciated.

Thanks,

Rocío

-----Original Message-----
From: Asaf Penso <asafp@nvidia.com> 
Sent: Sunday, January 16, 2022 4:31 PM
To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>; Rocio Dominguez <rocio.dominguez@ericsson.com>
Cc: users@dpdk.org; Matan Azrad <matan@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>; Raslan Darawsheh <rasland@nvidia.com>
Subject: RE: net_mlx5: unable to recognize master/representors on the multiple IB devices

Hello Rocio,
IIRC, there was a fix in a recent stable version.
Would you please try taking latest 19.11 LTS and tell whether you still see the issue?

Regards,
Asaf Penso

>-----Original Message-----
>From: Thomas Monjalon <thomas@monjalon.net>
>Sent: Sunday, January 16, 2022 3:24 PM
>To: Rocio Dominguez <rocio.dominguez@ericsson.com>
>Cc: users@dpdk.org; Matan Azrad <matan@nvidia.com>; Slava Ovsiienko 
><viacheslavo@nvidia.com>; Raslan Darawsheh <rasland@nvidia.com>
>Subject: Re: net_mlx5: unable to recognize master/representors on the 
>multiple IB devices
>
>+Cc mlx5 experts
>
>
>14/01/2022 11:10, Rocio Dominguez:
>> Hi,
>>
>> I'm doing a setup with Mellanox ConnectX-4 (MCX416A-CCA) NICs.
>>
>> I'm using:
>>
>> OS SLES 15 SP2
>> DPDK 19.11.4 (the official supported version for SLES 15 SP2)
>> MLNX_OFED_LINUX-5.5-1.0.3.2-sles15sp2-x86_64 (the latest one) 
>> Mellanox adapters firmware 12.28.2006 (corresponding to this 
>> MLNX_OFED version) kernel 5.3.18-24.34-default
>>
>>
>> This is my SRIOV configuration for DPDK capable PCI slots:
>>
>>             {
>>                 "resourceName": "mlnx_sriov_netdevice",
>>                 "resourcePrefix": "mellanox.com",
>>                 "isRdma": true,
>>                 "selectors": {
>>                     "vendors": ["15b3"],
>>                     "devices": ["1014"],
>>                     "drivers": ["mlx5_core"],
>>                     "pciAddresses": ["0000:d8:00.2", "0000:d8:00.3", 
>> "0000:d8:00.4",
>"0000:d8:00.5"],
>>                     "isRdma": true
>>                 }
>>
>> The sriov device plugin starts without problems, the devices are 
>> correctly
>allocated:
>>
>> {
>>   "cpu": "92",
>>   "ephemeral-storage": "419533922385",
>>   "hugepages-1Gi": "8Gi",
>>   "hugepages-2Mi": "4Gi",
>>   "intel.com/intel_sriov_dpdk": "0",
>>   "intel.com/sriov_cre": "3",
>>   "mellanox.com/mlnx_sriov_netdevice": "4",
>>   "mellanox.com/sriov_dp": "0",
>>   "memory": "183870336Ki",
>>   "pods": "110"
>> }
>>
>> The Mellanox NICs are binded to the kernel driver mlx5_core:
>>
>> pcgwpod009-c04:~ # dpdk-devbind --status
>>
>> Network devices using kernel driver
>> ===================================
>> 0000:18:00.0 'Ethernet Controller 10G X550T 1563' if=em1 drv=ixgbe 
>> unused=vfio-pci
>> 0000:18:00.1 'Ethernet Controller 10G X550T 1563' if=em2 drv=ixgbe 
>> unused=vfio-pci
>> 0000:19:00.0 'Ethernet Controller 10G X550T 1563' if=em3 drv=ixgbe 
>> unused=vfio-pci
>> 0000:19:00.1 'Ethernet Controller 10G X550T 1563' if=em4 drv=ixgbe 
>> unused=vfio-pci
>> 0000:3b:00.0 'MT27700 Family [ConnectX-4] 1013' if=enp59s0f0 
>> drv=mlx5_core unused=vfio-pci
>> 0000:3b:00.1 'MT27700 Family [ConnectX-4] 1013' if=enp59s0f1 
>> drv=mlx5_core unused=vfio-pci
>> 0000:5e:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=p3p1 drv=ixgbe unused=vfio-pci
>> 0000:5e:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=p3p2 drv=ixgbe unused=vfio-pci
>> 0000:5e:10.0 '82599 Ethernet Controller Virtual Function 10ed' if= 
>> drv=ixgbevf unused=vfio-pci
>> 0000:5e:10.2 '82599 Ethernet Controller Virtual Function 10ed'
>> if=p3p1_1 drv=ixgbevf unused=vfio-pci
>> 0000:5e:10.4 '82599 Ethernet Controller Virtual Function 10ed' if= 
>> drv=ixgbevf unused=vfio-pci
>> 0000:5e:10.6 '82599 Ethernet Controller Virtual Function 10ed'
>> if=p3p1_3 drv=ixgbevf unused=vfio-pci
>> 0000:af:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=p4p1 drv=ixgbe unused=vfio-pci
>> 0000:af:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=p4p2 drv=ixgbe unused=vfio-pci
>> 0000:d8:00.0 'MT27700 Family [ConnectX-4] 1013' if=enp216s0f0 
>> drv=mlx5_core unused=vfio-pci
>> 0000:d8:00.1 'MT27700 Family [ConnectX-4] 1013' if=enp216s0f1 
>> drv=mlx5_core unused=vfio-pci
>> 0000:d8:00.2 'MT27700 Family [ConnectX-4 Virtual Function] 1014'
>> if=enp216s0f2 drv=mlx5_core unused=vfio-pci
>> 0000:d8:00.3 'MT27700 Family [ConnectX-4 Virtual Function] 1014'
>> if=enp216s0f3 drv=mlx5_core unused=vfio-pci
>> 0000:d8:00.4 'MT27700 Family [ConnectX-4 Virtual Function] 1014'
>> if=enp216s0f4 drv=mlx5_core unused=vfio-pci
>> 0000:d8:00.5 'MT27700 Family [ConnectX-4 Virtual Function] 1014'
>> if=enp216s0f5 drv=mlx5_core unused=vfio-pci
>>
>> The interfaces are up:
>>
>> pcgwpod009-c04:~ # ibdev2netdev -v
>> 0000:3b:00.0 mlx5_0 (MT4115 - MT1646K01301) CX416A - ConnectX-4
>QSFP28
>> fw 12.28.2006 port 1 (ACTIVE) ==> enp59s0f0 (Up)
>> 0000:3b:00.1 mlx5_1 (MT4115 - MT1646K01301) CX416A - ConnectX-4
>QSFP28
>> fw 12.28.2006 port 1 (ACTIVE) ==> enp59s0f1 (Up)
>> 0000:d8:00.0 mlx5_2 (MT4115 - MT1646K00538) CX416A - ConnectX-4
>QSFP28
>> fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f0 (Up)
>> 0000:d8:00.1 mlx5_3 (MT4115 - MT1646K00538) CX416A - ConnectX-4
>QSFP28
>> fw 12.28.2006 port 1 (ACTIVE) ==> enp216s0f1 (Up)
>> 0000:d8:00.2 mlx5_4 (MT4116 - NA)  fw 12.28.2006 port 1 (ACTIVE) ==>
>> enp216s0f2 (Up)
>> 0000:d8:00.3 mlx5_5 (MT4116 - NA)  fw 12.28.2006 port 1 (ACTIVE) ==>
>> enp216s0f3 (Up)
>> 0000:d8:00.4 mlx5_6 (MT4116 - NA)  fw 12.28.2006 port 1 (ACTIVE) ==>
>> enp216s0f4 (Up)
>> 0000:d8:00.5 mlx5_7 (MT4116 - NA)  fw 12.28.2006 port 1 (ACTIVE) ==>
>> enp216s0f5 (Up) pcgwpod009-c04:~ #
>>
>>
>> But when I run my application the Mellanox adapters are probed and I
>obtain the following error:
>>
>> {"proc_id":"6"},"message":"[pio] EAL: Probe PCI driver: mlx5_pci
>> (15b3:1014) device: 0000:d8:00.4 (socket 1)"} 
>> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] net_mlx5: unable to recognize 
>> master/representors on the multiple IB devices"} 
>> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] common_mlx5: Failed to load driver = 
>> net_mlx5."} 
>> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] EAL: Requested device 0000:d8:00.4 cannot be 
>> used"} 
>> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] EAL: Bus (pci) probe failed."} 
>> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] USER1: ports init fail in DPDK, expect 1 ports, 
>> actual 0 ports."} 
>> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","sever
>> i 
>> ty":"error","service_id":"eric-pc-up-data-plane","metadata":{"proc_id"
>> :"6"},"message":"[pktio_libpio_init] No network ports could be 
>> enabled!"}
>>
>> Could you please help me with this issue?
>>
>>
>> Thanks,
>>
>> Rocío
>>
>
>
>
>