[dpdk-users] Unable to launch testpmd with Mellanox ConnectX-3 Pro NIC

[dpdk-users] Unable to launch testpmd with Mellanox ConnectX-3 Pro NIC

[Timur Bogdanov]

Hi,

I'm trying to use dpdk with Mellanox ConnectX-3 Pro NIC.
I built PMD for it according to instructions on
"http://dpdk.org/doc/guides/nics/mlx4.html" and tried to launch
testpmd app but got following error:

[tim@centos]$ sudo ./testpmd -c 0x03 -n 2 -w 0000:b3:00.0 -- --rxq=2 --txq=2 -i

EAL: Detected 36 lcore(s)

EAL: Probing VFIO support...

EAL: PCI device 0000:b3:00.0 on NUMA socket 0

EAL:   probe driver: 15b3:1007 net_mlx4

PMD: net_mlx4: PCI information matches, using device "mlx4_0" (VF: false)

PMD: net_mlx4: 1 port(s) detected

PMD: net_mlx4: port 1 MAC address is ec:0d:9a:17:4a:00

Interactive-mode selected

USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456,
size=2176, socket=0

Configuring Port 0 (socket 0)

PMD: net_mlx4: 0xda13c0: cannot attach flow rules (code 95, "Operation
not supported"), flow error type 2, cause 0x7f2869cdab80, message:
flow rule rejected by device

Fail to start port 0

Please stop the ports first

Done

testpmd> quit


Additional information:

OS: CentOs 7.4
kernel: 4.14.13-1.el7.elrepo.x86_64
rdma-core v16 has been built from sources with rpmbuild and installed:

ibacm-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root   21680 янв 16 14:24 iwpmd-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root   11044 янв 16 14:24 libibcm-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root   18008 янв 16 14:24
libibumad-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root  220272 янв 16 14:24
libibverbs-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root   53096 янв 16 14:24
libibverbs-utils-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root   58020 янв 16 14:24
librdmacm-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root   78184 янв 16 14:24
librdmacm-utils-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root   55308 янв 16 14:24
rdma-core-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root 1883624 янв 16 14:24
rdma-core-debuginfo-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root  211728 янв 16 14:24
rdma-core-devel-16.1-1.el7.centos.x86_64.rpm

-rw-r--r-- 1 root root   35484 янв 16 14:24
srp_daemon-16.1-1.el7.centos.x86_64.rpm

--------------------
output of lspci command for mellanox:

*-network

       description: Ethernet interface

       product: MT27520 Family [ConnectX-3 Pro]

       vendor: Mellanox Technologies

       physical id: 0

       bus info: pci@0000:b3:00.0

       logical name: eth5

       version: 00

       serial: ec:0d:9a:17:4a:00

       size: 10Gbit/s

       width: 64 bits

       clock: 33MHz

       capabilities: bus_master cap_list rom ethernet physical fibre

       configuration: autonegotiation=off broadcast=yes driver=mlx4_en
driverversion=4.0-0 duplex=full firmware=2.40.5030 latency=0 link=yes
multicast=yes port=fibre slave=yes speed=10Gbit/s

       resources: iomemory:38ff0-38fef irq:38 memory:fbe00000-fbefffff
memory:38ffff800000-38ffffffffff memory:fbd00000-fbdfffff


What can be the reason of testpmd does not start correctly?

/Regards, Timur

Re: [dpdk-users] Unable to launch testpmd with Mellanox ConnectX-3 Pro NIC

[Adrien Mazarguil]

On Tue, Jan 16, 2018 at 06:59:31PM +0300, Timur Bogdanov wrote:
> Hi,
> 
> I'm trying to use dpdk with Mellanox ConnectX-3 Pro NIC.
> I built PMD for it according to instructions on
> "http://dpdk.org/doc/guides/nics/mlx4.html" and tried to launch
> testpmd app but got following error:
<snip>
> PMD: net_mlx4: 0xda13c0: cannot attach flow rules (code 95, "Operation
> not supported"), flow error type 2, cause 0x7f2869cdab80, message:
> flow rule rejected by device
> 
> Fail to start port 0
<snip>
> What can be the reason of testpmd does not start correctly?

Did you provide by any chance -7 instead of -1 as a value for mlx4_core.ko's
log_num_mgm_entry_size parameter (as recommended by the above
documentation)?

Timothy (CC'ed) reported that -7 does not quite work with the default flow
rules (too many limitations) and results in the above error message. -1
should be used, -7 is only for specific use cases and should probably not
even be documented in this file.

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-users] Unable to launch testpmd with Mellanox ConnectX-3 Pro NIC

[Timur Bogdanov]

Yes, the problem was in this parameter and after changing it to "-1"
testpmd application has been started successfully.

BUT, it started successfully only on server where Mellanox OFED is installed.

On  another server a rdma-core was installed from sources. (rpm's were
built and installed)
I also changed the value of log_num_mgm_entry_size parameter to "-1"
and restarted a server (since where is no openibd service on this
server) and got the same error.
Maybe mlx4_core kernel module parameter should be changed in some
other config file?

Anyway, I can also install and use OFED instead of RDMA but the RDMA
is pointed as recommended on instructions page
"http://dpdk.org/doc/guides/nics/mlx4.html" and it is interesting to
bring it to working state.
/Regards, Timur


On Thu, Jan 18, 2018 at 11:48 AM, Adrien Mazarguil
<adrien.mazarguil@6wind.com> wrote:
> On Tue, Jan 16, 2018 at 06:59:31PM +0300, Timur Bogdanov wrote:
>> Hi,
>>
>> I'm trying to use dpdk with Mellanox ConnectX-3 Pro NIC.
>> I built PMD for it according to instructions on
>> "http://dpdk.org/doc/guides/nics/mlx4.html" and tried to launch
>> testpmd app but got following error:
> <snip>
>> PMD: net_mlx4: 0xda13c0: cannot attach flow rules (code 95, "Operation
>> not supported"), flow error type 2, cause 0x7f2869cdab80, message:
>> flow rule rejected by device
>>
>> Fail to start port 0
> <snip>
>> What can be the reason of testpmd does not start correctly?
>
> Did you provide by any chance -7 instead of -1 as a value for mlx4_core.ko's
> log_num_mgm_entry_size parameter (as recommended by the above
> documentation)?
>
> Timothy (CC'ed) reported that -7 does not quite work with the default flow
> rules (too many limitations) and results in the above error message. -1
> should be used, -7 is only for specific use cases and should probably not
> even be documented in this file.
>
> --
> Adrien Mazarguil
> 6WIND

Re: [dpdk-users] Unable to launch testpmd with Mellanox ConnectX-3 Pro NIC

[Adrien Mazarguil]

On Thu, Jan 18, 2018 at 03:11:16PM +0300, Timur Bogdanov wrote:
> Yes, the problem was in this parameter and after changing it to "-1"
> testpmd application has been started successfully.
> 
> BUT, it started successfully only on server where Mellanox OFED is installed.
> 
> On  another server a rdma-core was installed from sources. (rpm's were
> built and installed)
> I also changed the value of log_num_mgm_entry_size parameter to "-1"
> and restarted a server (since where is no openibd service on this
> server) and got the same error.
> Maybe mlx4_core kernel module parameter should be changed in some
> other config file?

In case MLNX_OFED is *not* installed, do both rdma-core and kernel version
meet the minimum requirements?

- rdma-core v15
- Linux 4.14

Note while the above versions are fine for mlx5 (ConnectX-4, ConnectX-5),
they provide reduced performance and RSS capabilities for mlx4 (ConnectX-3),
for which I recommend to use at least:

- rdma-core v16
- Linux 4.15 (not released yet, 4.15-rc8 will do in the meantime)

> Anyway, I can also install and use OFED instead of RDMA but the RDMA
> is pointed as recommended on instructions page
> "http://dpdk.org/doc/guides/nics/mlx4.html" and it is interesting to
> bring it to working state.
> /Regards, Timur

rdma-core requires an up-to-date kernel. If this can't be satisfied,
MLNX_OFED is the only alternative (the mlnx-ofa_kernel package).

> On Thu, Jan 18, 2018 at 11:48 AM, Adrien Mazarguil
> <adrien.mazarguil@6wind.com> wrote:
> > On Tue, Jan 16, 2018 at 06:59:31PM +0300, Timur Bogdanov wrote:
> >> Hi,
> >>
> >> I'm trying to use dpdk with Mellanox ConnectX-3 Pro NIC.
> >> I built PMD for it according to instructions on
> >> "http://dpdk.org/doc/guides/nics/mlx4.html" and tried to launch
> >> testpmd app but got following error:
> > <snip>
> >> PMD: net_mlx4: 0xda13c0: cannot attach flow rules (code 95, "Operation
> >> not supported"), flow error type 2, cause 0x7f2869cdab80, message:
> >> flow rule rejected by device
> >>
> >> Fail to start port 0
> > <snip>
> >> What can be the reason of testpmd does not start correctly?
> >
> > Did you provide by any chance -7 instead of -1 as a value for mlx4_core.ko's
> > log_num_mgm_entry_size parameter (as recommended by the above
> > documentation)?
> >
> > Timothy (CC'ed) reported that -7 does not quite work with the default flow
> > rules (too many limitations) and results in the above error message. -1
> > should be used, -7 is only for specific use cases and should probably not
> > even be documented in this file.
> >
> > --
> > Adrien Mazarguil
> > 6WIND

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-users] Unable to launch testpmd with Mellanox ConnectX-3 Pro NIC

[Timur Bogdanov]

Server kernel is 4.14.13-1.el7.elrepo.x86_64.
The rdma-core is v16.
Parameter "log_num_mgm_entry_size" is set to "-1"
So I don't know why testpmd still runs with same error:

PMD: net_mlx4: 0xda13c0: cannot attach flow rules (code 95, "Operation
not supported"), flow error type 2, cause 0x7f2869cdab80, message:
flow rule rejected by device
/Regards, Timur


On Thu, Jan 18, 2018 at 6:05 PM, Adrien Mazarguil
<adrien.mazarguil@6wind.com> wrote:
> On Thu, Jan 18, 2018 at 03:11:16PM +0300, Timur Bogdanov wrote:
>> Yes, the problem was in this parameter and after changing it to "-1"
>> testpmd application has been started successfully.
>>
>> BUT, it started successfully only on server where Mellanox OFED is installed.
>>
>> On  another server a rdma-core was installed from sources. (rpm's were
>> built and installed)
>> I also changed the value of log_num_mgm_entry_size parameter to "-1"
>> and restarted a server (since where is no openibd service on this
>> server) and got the same error.
>> Maybe mlx4_core kernel module parameter should be changed in some
>> other config file?
>
> In case MLNX_OFED is *not* installed, do both rdma-core and kernel version
> meet the minimum requirements?
>
> - rdma-core v15
> - Linux 4.14
>
> Note while the above versions are fine for mlx5 (ConnectX-4, ConnectX-5),
> they provide reduced performance and RSS capabilities for mlx4 (ConnectX-3),
> for which I recommend to use at least:
>
> - rdma-core v16
> - Linux 4.15 (not released yet, 4.15-rc8 will do in the meantime)
>
>> Anyway, I can also install and use OFED instead of RDMA but the RDMA
>> is pointed as recommended on instructions page
>> "http://dpdk.org/doc/guides/nics/mlx4.html" and it is interesting to
>> bring it to working state.
>> /Regards, Timur
>
> rdma-core requires an up-to-date kernel. If this can't be satisfied,
> MLNX_OFED is the only alternative (the mlnx-ofa_kernel package).
>
>> On Thu, Jan 18, 2018 at 11:48 AM, Adrien Mazarguil
>> <adrien.mazarguil@6wind.com> wrote:
>> > On Tue, Jan 16, 2018 at 06:59:31PM +0300, Timur Bogdanov wrote:
>> >> Hi,
>> >>
>> >> I'm trying to use dpdk with Mellanox ConnectX-3 Pro NIC.
>> >> I built PMD for it according to instructions on
>> >> "http://dpdk.org/doc/guides/nics/mlx4.html" and tried to launch
>> >> testpmd app but got following error:
>> > <snip>
>> >> PMD: net_mlx4: 0xda13c0: cannot attach flow rules (code 95, "Operation
>> >> not supported"), flow error type 2, cause 0x7f2869cdab80, message:
>> >> flow rule rejected by device
>> >>
>> >> Fail to start port 0
>> > <snip>
>> >> What can be the reason of testpmd does not start correctly?
>> >
>> > Did you provide by any chance -7 instead of -1 as a value for mlx4_core.ko's
>> > log_num_mgm_entry_size parameter (as recommended by the above
>> > documentation)?
>> >
>> > Timothy (CC'ed) reported that -7 does not quite work with the default flow
>> > rules (too many limitations) and results in the above error message. -1
>> > should be used, -7 is only for specific use cases and should probably not
>> > even be documented in this file.
>> >
>> > --
>> > Adrien Mazarguil
>> > 6WIND
>
> --
> Adrien Mazarguil
> 6WIND

Re: [dpdk-users] Unable to launch testpmd with Mellanox ConnectX-3 Pro NIC

[Timur Bogdanov]

Finally I managed to run testpmd normally after updating a kernel to
4.14.14 and installing rdma-core_v17 from sources.

Tnx for the advice of setting parameter log_num_mgm_entry_size" to "-1"
/Regards, Timur


On Fri, Jan 19, 2018 at 2:10 PM, Timur Bogdanov <timurbogdanov@gmail.com> wrote:
> Server kernel is 4.14.13-1.el7.elrepo.x86_64.
> The rdma-core is v16.
> Parameter "log_num_mgm_entry_size" is set to "-1"
> So I don't know why testpmd still runs with same error:
>
> PMD: net_mlx4: 0xda13c0: cannot attach flow rules (code 95, "Operation
> not supported"), flow error type 2, cause 0x7f2869cdab80, message:
> flow rule rejected by device
> /Regards, Timur
>
>
> On Thu, Jan 18, 2018 at 6:05 PM, Adrien Mazarguil
> <adrien.mazarguil@6wind.com> wrote:
>> On Thu, Jan 18, 2018 at 03:11:16PM +0300, Timur Bogdanov wrote:
>>> Yes, the problem was in this parameter and after changing it to "-1"
>>> testpmd application has been started successfully.
>>>
>>> BUT, it started successfully only on server where Mellanox OFED is installed.
>>>
>>> On  another server a rdma-core was installed from sources. (rpm's were
>>> built and installed)
>>> I also changed the value of log_num_mgm_entry_size parameter to "-1"
>>> and restarted a server (since where is no openibd service on this
>>> server) and got the same error.
>>> Maybe mlx4_core kernel module parameter should be changed in some
>>> other config file?
>>
>> In case MLNX_OFED is *not* installed, do both rdma-core and kernel version
>> meet the minimum requirements?
>>
>> - rdma-core v15
>> - Linux 4.14
>>
>> Note while the above versions are fine for mlx5 (ConnectX-4, ConnectX-5),
>> they provide reduced performance and RSS capabilities for mlx4 (ConnectX-3),
>> for which I recommend to use at least:
>>
>> - rdma-core v16
>> - Linux 4.15 (not released yet, 4.15-rc8 will do in the meantime)
>>
>>> Anyway, I can also install and use OFED instead of RDMA but the RDMA
>>> is pointed as recommended on instructions page
>>> "http://dpdk.org/doc/guides/nics/mlx4.html" and it is interesting to
>>> bring it to working state.
>>> /Regards, Timur
>>
>> rdma-core requires an up-to-date kernel. If this can't be satisfied,
>> MLNX_OFED is the only alternative (the mlnx-ofa_kernel package).
>>
>>> On Thu, Jan 18, 2018 at 11:48 AM, Adrien Mazarguil
>>> <adrien.mazarguil@6wind.com> wrote:
>>> > On Tue, Jan 16, 2018 at 06:59:31PM +0300, Timur Bogdanov wrote:
>>> >> Hi,
>>> >>
>>> >> I'm trying to use dpdk with Mellanox ConnectX-3 Pro NIC.
>>> >> I built PMD for it according to instructions on
>>> >> "http://dpdk.org/doc/guides/nics/mlx4.html" and tried to launch
>>> >> testpmd app but got following error:
>>> > <snip>
>>> >> PMD: net_mlx4: 0xda13c0: cannot attach flow rules (code 95, "Operation
>>> >> not supported"), flow error type 2, cause 0x7f2869cdab80, message:
>>> >> flow rule rejected by device
>>> >>
>>> >> Fail to start port 0
>>> > <snip>
>>> >> What can be the reason of testpmd does not start correctly?
>>> >
>>> > Did you provide by any chance -7 instead of -1 as a value for mlx4_core.ko's
>>> > log_num_mgm_entry_size parameter (as recommended by the above
>>> > documentation)?
>>> >
>>> > Timothy (CC'ed) reported that -7 does not quite work with the default flow
>>> > rules (too many limitations) and results in the above error message. -1
>>> > should be used, -7 is only for specific use cases and should probably not
>>> > even be documented in this file.
>>> >
>>> > --
>>> > Adrien Mazarguil
>>> > 6WIND
>>
>> --
>> Adrien Mazarguil
>> 6WIND