Re: [dpdk-dev] Mellanox PMD failure w/DPDK-2.1.0 and MLNX_OFED-3.1-1.0.3

[Olga Shern <olgas@mellanox.com>]

Hi Bill,

Starting from DPDK 2.1 ConnectX-3 PMD is based on “accelerated verbs”,  ibv_exp_create_res_domain is coming from this new API.
Just to make sure I understand what you are doing:  you have enabled SRIOV and you are running DPDK on hypervisor on the probed VFs that you have created, right?
We did test this combination (dpdk2.1 and ofed3.1-3)on hypervisor on the PF and also on VM on VF, but in fact, I didn’t try to run DPDK on the VFs on hypervisor, I will check this.
Meanwhile, can you please send the output of the application on the start up.  Do you see any errors in dmesg?

Best Regards,
Olga

From: Bill O'Hara [mailto:billtohara@gmail.com]
Sent: Thursday, October 08, 2015 11:55 PM
To: Olga Shern
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Mellanox PMD failure w/DPDK-2.1.0 and MLNX_OFED-3.1-1.0.3

Olga

If it's all all helpful, linking our code against dpdk-2.0 and (statically) against the appropriate custom-built libibverbs that we used with it, works on those machines. There is of course no call to ibv_exp_create_res_domain() in that version of the library. But it at least confirms basic operation of the upgraded OFED and firmware on those boxes.

Is there anything else we can check or confirm for you?

thanks
bill


On Thu, Oct 8, 2015 at 9:06 AM, Bill O'Hara <billtohara@gmail.com<mailto:billtohara@gmail.com>> wrote:
Hi Olga

Firmware is version 2.35.5100. Configuration details below.

Thanks for any hints.
bill

root:~# cat /etc/modprobe.d/mlx4_core.conf
options mlx4_core port_type_array=2,2 num_vfs=16 probe_vf=4

root:~# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.35.5100
Hardware version: 1
Node GUID: 0xf4521403008f1680
System image GUID: 0xf4521403008f1683
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x0c010000
Port GUID: 0xf65214fffe8f1680
Link layer: Ethernet
CA 'mlx4_1'
CA type: MT4100
Number of ports: 1
Firmware version: 2.35.5100
Hardware version: 1
Node GUID: 0x00140500c2d3b05f
System image GUID: 0xf4521403008f1683
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x0c010000
Port GUID: 0xfc9739fffe1272c3
Link layer: Ethernet
CA 'mlx4_2'
CA type: MT4100
Number of ports: 1
Firmware version: 2.35.5100
Hardware version: 1
Node GUID: 0x00140500b90af10c
System image GUID: 0xf4521403008f1683
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x0c010000
Port GUID: 0x20ecbbfffeefb934
Link layer: Ethernet
CA 'mlx4_3'
CA type: MT4100
Number of ports: 1
Firmware version: 2.35.5100
Hardware version: 1
Node GUID: 0x001405009661e607
System image GUID: 0xf4521403008f1683
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x0c010000
Port GUID: 0xf4c8e6fffe5abc89
Link layer: Ethernet
CA 'mlx4_4'
CA type: MT4100
Number of ports: 1
Firmware version: 2.35.5100
Hardware version: 1
Node GUID: 0x00140500bd09e128
System image GUID: 0xf4521403008f1683
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x0c010000
Port GUID: 0x5828e1fffe34f919
Link layer: Ethernet

On Thu, Oct 8, 2015 at 2:03 AM, Olga Shern <olgas@mellanox.com<mailto:olgas@mellanox.com>> wrote:
Hi Bill,

Can you please check the fw version that is installed on your ConnectX3?

Thanks


Sent from Samsung Mobile.

-------- Original message --------
From: Olga Shern
Date:08/10/2015 7:55 AM (GMT+00:00)
To: Bill O'Hara ,dev@dpdk.org<mailto:dev@dpdk.org>
Subject: RE: [dpdk-dev] Mellanox PMD failure w/DPDK-2.1.0 and MLNX_OFED-3.1-1.0.3

Hi Bill,

There shouldn’t be any problem with what you are doing.
We are checking this now.

Best Regards,
Olga

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bill O'Hara
Sent: Thursday, October 08, 2015 6:05 AM
To: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: [dpdk-dev] Mellanox PMD failure w/DPDK-2.1.0 and MLNX_OFED-3.1-1.0.3

Hello

I wonder if anyone can suggest why previously working dpdk code may fail in the Mellanox pmd code in dpdk-2.1.0, seemingly due to failure to create a "resource domain" via ibv_exp_create_res_domain(). I must admit I haven't seen that verb before, and it appears to be returning null with no error message.

The DPDK log gives these hints:

PMD: librte_pmd_mlx4: 0xa4fc20: TX queues number update: 0 -> 1
PMD: librte_pmd_mlx4: 0xa4fc20: RX queues number update: 0 -> 1
PMD: librte_pmd_mlx4: 0xa4fc20: RD creation failure: Cannot allocate memory

I'm using dpdk-2.10.0 and  MLNX_OFED_LINUX-3.1-1.0.3 on ubuntu14.04 with a
connectx-3 card.

thanks
bill