Re: [dpdk-users] testpmd: [mlx4] rxq_add_flow(): flow configuration failed, errno=38: Function not implemented

[Ruslan Osmanov <rrosmanov@gmail.com>]

Hi,

it's a KVM guest.

We can't use newest firmware because we have SuperMicro blades with
ConnectX3 Pro ethernet cards. SuperMicro has their own version of
Mellanox firmware for that blade, and the last version on their
FTP is 2.35, not 2.36. Sadly, they have two different files depending on
the blade location in the chassis, which means that the stock Mellanox
firmware most likely won't work at all, or at least mess up blade boot
procedure (they all are network-booted in our case).

You probably meant mlx4_1, not mlx5_1.

raw_ethernet_bw --client -d mlx4_1  -l 1 -n 5 -s 64 \
--dest_mac 7C:FE:90:29:23:4F --source_mac 7C:FE:90:29:23:67
---------------------------------------------------------------------------------------
Send BW Test Dual-port       : OFF		Device         :
mlx4_1 Number of qps   : 1		Transport type : IB
 Connection type : RawEth		Using SRQ      : OFF
 TX depth        : 5
 CQ Moderation   : 5
 Mtu             : 1518[B]
 Link type       : Ethernet
 Gid index       : 0
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 Unable to create QP.
Failed to create QP.
 Couldn't create IB resources


On Mon, 4 Jul 2016 11:00:09 +0000
Olga Shern <olgas@mellanox.com> wrote:

> Hi Ruslan, 
> 
> According to your output you are using VF.
> Do you run testpmd on the VM?
> 
> Your FW is a little bit old, it is recommended to use FW that is
> coming with MLNX_OFED. Please upgrade to 2.36.5000.
> 
> After FW upgrade is you still have this issue
> 
> Please run raw_ethernet_bw example: 
> raw_ethernet_bw --client -d mlx5_1  -l 1 -n 5 -s 64 --dest_mac
> 7C:FE:90:29:23:4F --source_mac 7C:FE:90:29:23:67  
> 
> 
> > dpdk_nic_bind --status:
> > https://gist.github.com/2449c7b943eec70a2a3dc8da905148b2  
> 
> This script is not relevant for Mellanox NICs. 
> 
> Best Regards,
> Olga
> 
> 
> 
> -----Original Message-----
> From: Ruslan Osmanov [mailto:rrosmanov@gmail.com] 
> Sent: Monday, July 04, 2016 7:55 AM
> To: Olga Shern <olgas@mellanox.com>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] testpmd: [mlx4] rxq_add_flow(): flow
> configuration failed, errno=38: Function not implemented
> 
> Sorry, I forgot to mention that I had to add `#define _GNU_SOURCE` at
> the top of mlx4.c file, because it failed to compile otherwise.
> 
> diff -Naur \
> tmp/dpdk-16.04/drivers/net/mlx4/ \
> /opt/dpdk-16.04/drivers/net/mlx4/
> (the first is original)
> https://gist.github.com/ebe70bb62fafb30ab6bb01d164ea5c7f
> 
> And this is how the build fails without _GNU_SOURCE:
> https://gist.github.com/ed6fd3022b2cbf86d5bd30cf444765a4
> 
> 
> On Mon, 4 Jul 2016 11:30:30 +0700
> Ruslan Osmanov <rrosmanov@gmail.com> wrote:
> 
> > System information
> > -------------------
> > 
> > OS: Ubuntu 16.04 LTS
> > Arch: x86_64
> > Kernel: 4.4.0-28-generic
> > Kernel configuration:
> > https://gist.github.com/9b896efae7ad097b3f05581640f670d0 Installed
> > packages: https://gist.github.com/5d1fe6557a9a41cea9ce70b7269e35ac
> > ibv_devinfo:
> > https://gist.github.com/8c0ecd660aa1be16ddc6aa7b96ffcd11 ofed_info
> > -s: MLNX_OFED_LINUX-3.3-1.0.0.0: dpdk_nic_bind --status:
> > https://gist.github.com/2449c7b943eec70a2a3dc8da905148b2
> > 
> > 
> > testpmd without --no-huge --proc-type=auto
> > ------------------------------------------
> > 
> > sudo -E
> > ~/DPDK/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd \ -c
> > 0x3 --pci-blacklist 0000:00:06.0 -- -i --total-num-mbufs=2048
> > 
> > ###### OUTPUT #######
> > ...
> > mlx4.c:5464: mlx4_pci_devinit():
> > PCI information matches, using device "mlx4_1" (VF: true) ...
> > 
> > mlx4.c:2503: rxq_add_flow(): 0x7ffd6ca10050:
> > adding MAC address 8e:d5:86:2e:a3:02 index 0 (VLAN index 4294967295)
> > 
> > mlx4.c:2511: rxq_add_flow(): 0x7ffd6ca10050:
> > flow configuration failed, errno=38: Function not implemented
> > 
> > mlx4.c:3862: rxq_setup(): 0x921a20:
> > QP flow attachment failed: Function not implemented ...
> > ######### OUTPUT (END) ##########
> > 
> > Full output:
> > https://gist.github.com/3c83db9c0ba4a662be92fef958170f58
> > 
> > 
> > On Sun, 3 Jul 2016 12:24:11 +0000
> > Olga Shern <olgas@mellanox.com> wrote:
> >   
> > > Hello Ruslan,
> > > 
> > > Can you please send full details of your setup, please also
> > > include the following info: OS, FW, MLNX_OFED that you are using
> > > 
> > > The following commands can help to retrieve Mellanox specific
> > > info: ibv_devinfo
> > > ofed_info -s
> > > 
> > > Another q, I have notice you are using --no-huge option, this is
> > > not the issue here,  but I am not sure if this option is working
> > > at all. And why are you using  --proc-type=auto?
> > > 
> > > Please run as root user without the above flags (--no-huge and
> > > --proc-type)
> > > 
> > > Best Regards,
> > > Olga
> > > 
> > > -----Original Message-----
> > > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Ruslan 
> > > Osmanov Sent: Saturday, July 02, 2016 11:18 AM
> > > To: users@dpdk.org
> > > Subject: [dpdk-users] testpmd: [mlx4] rxq_add_flow(): flow 
> > > configuration failed, errno=38: Function not implemented
> > > 
> > > Hi,
> > > 
> > > I need help. DPDK applications fail to start with Mellanox
> > > ConnectX-3 NIC. All of them complain about 'not implemented'
> > > function when trying to setup the RX queue:
> > > 
> > > sudo -E
> > > ~/DPDK/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd \ -c
> > > 0x3 --pci-blacklist 0000:00:06.0 \ --no-huge --proc-type=auto \
> > > -- -i --total-num-mbufs=2048
> > > 
> > > ...
> > > 
> > > rxq_setup(): 0x921a20: disabling scattered packets support (128
> > > WRs) mlx4_mp2mr(): mempool 0x7f0c0b1eb000 area
> > > start=0x7f0c0b36e000 end=0x7f0c0bb6e000 size=8388608
> > > mlx4_mp2mr(): mempool 0x7f0c0b1eb000 using start=0x7f0c0b36e000
> > > end=0x7f0c0bb6e000 size=8388608 for MR
> > > rxq_setup(): priv->device_attr.max_qp_wr is 16351
> > > rxq_setup(): priv->device_attr.max_sge is 32
> > > rxq_add_flow(): 0x7ffd68781e10: adding MAC address
> > > 8e:d5:86:2e:a3:02 index 0 (VLAN index 4294967295) rxq_add_flow():
> > > 0x7ffd68781e10: flow configuration failed, errno=38: Function not
> > > implemented rxq_setup(): 0x921a20: QP flow attachment failed:
> > > Function not implemented ... Fail to configure port 0 rx queues
> > > EAL: Error - exiting with code: 1
> > >   Cause: Start ports failed
> > > 
> > > The whole output is in the attached dpdk-mlx4-testmpd.log file.
> > > If for some reason the mailing list doesn't support attachments,
> > > here is an online version:
> > > https://gist.github.com/7f740489da77ecd23cc162da5e7f864c
> > > 
> > > DPDK version: 16.04.
> > > dpdk_proc_info:
> > >  sudo -E
> > >  
> > > ~/DPDK/x86_64-native-linuxapp-gcc/build/app/proc_info/dpdk_proc_info
> > >  -c 0x3 --no-huge --proc-type=auto --pci-blacklist 0000:00:06.0
> > > -- -p 0x3 -m --stats --xstats (output is in the attached 
> > > dpdk-mlx4-dpdk_proc_info.log; online
> > > version: https://gist.github.com/e8a01484a02e147f9539e81638198747)
> > > 
> > > Please help me to figure out what's wrong with ... what? I even 
> > > don't know. Maybe the mlx4 driver is buggy, maybe dpdk, maybe my 
> > > configuration is wrong.
> > > 
> > > 
> > > --
> > > Ruslan Osmanov  
> > 
> >   
> 
> 
> --
> Ruslan Osmanov


-- 
Ruslan Osmanov