Re: [dpdk-users] testpmd: [mlx4] rxq_add_flow(): flow configuration failed, errno=38: Function not implemented

[Ruslan Osmanov <rrosmanov@gmail.com>]

Ahh, sorry.

sudo -E raw_ethernet_bw --client -d mlx4_1  -l 1 -n 5 -s 64 \
--dest_mac 7C:FE:90:29:23:4F --source_mac 7C:FE:90:29:23:67

https://gist.github.com/4310b258757a76a854a1610269207209

On Mon, 4 Jul 2016 13:40:32 +0000
Olga Shern <olgas@mellanox.com> wrote:

> You have to be user root to run raw_ethernet_bw 
> 
> -----Original Message-----
> From: Ruslan Osmanov [mailto:rrosmanov@gmail.com] 
> Sent: Monday, July 04, 2016 2:30 PM
> To: Olga Shern <olgas@mellanox.com>
> Cc: users@dpdk.org; Gideon Naim <gideonn@mellanox.com>
> Subject: Re: [dpdk-users] testpmd: [mlx4] rxq_add_flow(): flow
> configuration failed, errno=38: Function not implemented
> 
> Hi,
> 
> it's a KVM guest.
> 
> We can't use newest firmware because we have SuperMicro blades with
> ConnectX3 Pro ethernet cards. SuperMicro has their own version of
> Mellanox firmware for that blade, and the last version on their FTP
> is 2.35, not 2.36. Sadly, they have two different files depending on
> the blade location in the chassis, which means that the stock
> Mellanox firmware most likely won't work at all, or at least mess up
> blade boot procedure (they all are network-booted in our case).
> 
> You probably meant mlx4_1, not mlx5_1.
> 
> raw_ethernet_bw --client -d mlx4_1  -l 1 -n 5 -s 64 \ --dest_mac
> 7C:FE:90:29:23:4F --source_mac 7C:FE:90:29:23:67
> ---------------------------------------------------------------------------------------
> Send BW Test Dual-port       : OFF		Device         :
> mlx4_1 Number of qps   : 1		Transport type : IB
> Connection type : RawEth		Using SRQ      : OFF TX
> depth        : 5 CQ Moderation   : 5
>  Mtu             : 1518[B]
>  Link type       : Ethernet
>  Gid index       : 0
>  Max inline data : 0[B]
>  rdma_cm QPs	 : OFF
>  Data ex. method : Ethernet
> ---------------------------------------------------------------------------------------
>  Unable to create QP.
> Failed to create QP.
>  Couldn't create IB resources
> 
> 
> On Mon, 4 Jul 2016 11:00:09 +0000
> Olga Shern <olgas@mellanox.com> wrote:
> 
> > Hi Ruslan,
> > 
> > According to your output you are using VF.
> > Do you run testpmd on the VM?
> > 
> > Your FW is a little bit old, it is recommended to use FW that is 
> > coming with MLNX_OFED. Please upgrade to 2.36.5000.
> > 
> > After FW upgrade is you still have this issue
> > 
> > Please run raw_ethernet_bw example: 
> > raw_ethernet_bw --client -d mlx5_1  -l 1 -n 5 -s 64 --dest_mac 
> > 7C:FE:90:29:23:4F --source_mac 7C:FE:90:29:23:67
> > 
> >   
> > > dpdk_nic_bind --status:
> > > https://gist.github.com/2449c7b943eec70a2a3dc8da905148b2  
> > 
> > This script is not relevant for Mellanox NICs. 
> > 
> > Best Regards,
> > Olga
> > 
> > 
> > 
> > -----Original Message-----
> > From: Ruslan Osmanov [mailto:rrosmanov@gmail.com]
> > Sent: Monday, July 04, 2016 7:55 AM
> > To: Olga Shern <olgas@mellanox.com>
> > Cc: users@dpdk.org
> > Subject: Re: [dpdk-users] testpmd: [mlx4] rxq_add_flow(): flow 
> > configuration failed, errno=38: Function not implemented
> > 
> > Sorry, I forgot to mention that I had to add `#define _GNU_SOURCE`
> > at the top of mlx4.c file, because it failed to compile otherwise.
> > 
> > diff -Naur \
> > tmp/dpdk-16.04/drivers/net/mlx4/ \
> > /opt/dpdk-16.04/drivers/net/mlx4/
> > (the first is original)
> > https://gist.github.com/ebe70bb62fafb30ab6bb01d164ea5c7f
> > 
> > And this is how the build fails without _GNU_SOURCE:
> > https://gist.github.com/ed6fd3022b2cbf86d5bd30cf444765a4
> > 
> > 
> > On Mon, 4 Jul 2016 11:30:30 +0700
> > Ruslan Osmanov <rrosmanov@gmail.com> wrote:
> >   
> > > System information
> > > -------------------
> > > 
> > > OS: Ubuntu 16.04 LTS
> > > Arch: x86_64
> > > Kernel: 4.4.0-28-generic
> > > Kernel configuration:
> > > https://gist.github.com/9b896efae7ad097b3f05581640f670d0 Installed
> > > packages: https://gist.github.com/5d1fe6557a9a41cea9ce70b7269e35ac
> > > ibv_devinfo:
> > > https://gist.github.com/8c0ecd660aa1be16ddc6aa7b96ffcd11 ofed_info
> > > -s: MLNX_OFED_LINUX-3.3-1.0.0.0: dpdk_nic_bind --status:
> > > https://gist.github.com/2449c7b943eec70a2a3dc8da905148b2
> > > 
> > > 
> > > testpmd without --no-huge --proc-type=auto
> > > ------------------------------------------
> > > 
> > > sudo -E
> > > ~/DPDK/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd \ -c
> > > 0x3 --pci-blacklist 0000:00:06.0 -- -i --total-num-mbufs=2048
> > > 
> > > ###### OUTPUT #######
> > > ...
> > > mlx4.c:5464: mlx4_pci_devinit():
> > > PCI information matches, using device "mlx4_1" (VF: true) ...
> > > 
> > > mlx4.c:2503: rxq_add_flow(): 0x7ffd6ca10050:
> > > adding MAC address 8e:d5:86:2e:a3:02 index 0 (VLAN index
> > > 4294967295)
> > > 
> > > mlx4.c:2511: rxq_add_flow(): 0x7ffd6ca10050:
> > > flow configuration failed, errno=38: Function not implemented
> > > 
> > > mlx4.c:3862: rxq_setup(): 0x921a20:
> > > QP flow attachment failed: Function not implemented ...
> > > ######### OUTPUT (END) ##########
> > > 
> > > Full output:
> > > https://gist.github.com/3c83db9c0ba4a662be92fef958170f58
> > > 
> > > 
> > > On Sun, 3 Jul 2016 12:24:11 +0000
> > > Olga Shern <olgas@mellanox.com> wrote:
> > >     
> > > > Hello Ruslan,
> > > > 
> > > > Can you please send full details of your setup, please also 
> > > > include the following info: OS, FW, MLNX_OFED that you are using
> > > > 
> > > > The following commands can help to retrieve Mellanox specific
> > > > info: ibv_devinfo
> > > > ofed_info -s
> > > > 
> > > > Another q, I have notice you are using --no-huge option, this
> > > > is not the issue here,  but I am not sure if this option is
> > > > working at all. And why are you using  --proc-type=auto?
> > > > 
> > > > Please run as root user without the above flags (--no-huge and
> > > > --proc-type)
> > > > 
> > > > Best Regards,
> > > > Olga
> > > > 
> > > > -----Original Message-----
> > > > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Ruslan 
> > > > Osmanov Sent: Saturday, July 02, 2016 11:18 AM
> > > > To: users@dpdk.org
> > > > Subject: [dpdk-users] testpmd: [mlx4] rxq_add_flow(): flow 
> > > > configuration failed, errno=38: Function not implemented
> > > > 
> > > > Hi,
> > > > 
> > > > I need help. DPDK applications fail to start with Mellanox
> > > > ConnectX-3 NIC. All of them complain about 'not implemented'
> > > > function when trying to setup the RX queue:
> > > > 
> > > > sudo -E
> > > > ~/DPDK/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd \
> > > > -c 0x3 --pci-blacklist 0000:00:06.0 \ --no-huge
> > > > --proc-type=auto \ -- -i --total-num-mbufs=2048
> > > > 
> > > > ...
> > > > 
> > > > rxq_setup(): 0x921a20: disabling scattered packets support (128
> > > > WRs) mlx4_mp2mr(): mempool 0x7f0c0b1eb000 area
> > > > start=0x7f0c0b36e000 end=0x7f0c0bb6e000 size=8388608
> > > > mlx4_mp2mr(): mempool 0x7f0c0b1eb000 using start=0x7f0c0b36e000
> > > > end=0x7f0c0bb6e000 size=8388608 for MR
> > > > rxq_setup(): priv->device_attr.max_qp_wr is 16351
> > > > rxq_setup(): priv->device_attr.max_sge is 32
> > > > rxq_add_flow(): 0x7ffd68781e10: adding MAC address
> > > > 8e:d5:86:2e:a3:02 index 0 (VLAN index 4294967295)
> > > > rxq_add_flow(): 0x7ffd68781e10: flow configuration failed,
> > > > errno=38: Function not implemented rxq_setup(): 0x921a20: QP
> > > > flow attachment failed: Function not implemented ... Fail to
> > > > configure port 0 rx queues EAL: Error - exiting with code: 1
> > > >   Cause: Start ports failed
> > > > 
> > > > The whole output is in the attached dpdk-mlx4-testmpd.log file.
> > > > If for some reason the mailing list doesn't support
> > > > attachments, here is an online version:
> > > > https://gist.github.com/7f740489da77ecd23cc162da5e7f864c
> > > > 
> > > > DPDK version: 16.04.
> > > > dpdk_proc_info:
> > > >  sudo -E
> > > >  
> > > > ~/DPDK/x86_64-native-linuxapp-gcc/build/app/proc_info/dpdk_proc_in
> > > > fo  -c 0x3 --no-huge --proc-type=auto --pci-blacklist
> > > > 0000:00:06.0 -- -p 0x3 -m --stats --xstats (output is in the
> > > > attached dpdk-mlx4-dpdk_proc_info.log; online
> > > > version:
> > > > https://gist.github.com/e8a01484a02e147f9539e81638198747)
> > > > 
> > > > Please help me to figure out what's wrong with ... what? I even 
> > > > don't know. Maybe the mlx4 driver is buggy, maybe dpdk, maybe
> > > > my configuration is wrong.
> > > > 
> > > > 
> > > > --
> > > > Ruslan Osmanov  
> > > 
> > >     
> > 
> > 
> > --
> > Ruslan Osmanov  
> 
> 
> --
> Ruslan Osmanov


-- 
Ruslan Osmanov