[dpdk-users] mlx5 pmd + rdma-core 28 init failure

[dpdk-users] mlx5 pmd + rdma-core 28 init failure

[Benoit Ganne (bganne)]

Hi all,

I have troubles making DPDK v20.02 MLX5 PMD working with rdma-core v28.0: it looks like the flow initialization done while initializing the device fails in rdma-core providers/mlx5/dr_table.c:mlx5dv_dr_table_create() because of unsupported parameters.
The issue is the following test in rdma-core providers/mlx5/dr_table.c:mlx5dv_dr_table_create():
    if (level && !dmn->info.supp_sw_steering) {
        errno = EOPNOTSUPP;
        goto dec_ref;
    }
Where level == 65534 and dmn->info.supp_sw_steering == false, hence the test is true and the function returns EOPNOTSUPP.

Is this expected? It does work fine with ibv instead of dv API. Any help appreciated.

Here is what I do:
   1) checkout & compile rdma-core v28.0
~# git clone https://github.com/linux-rdma/rdma-core
~# cd rdma-core
~# git checkout v28.0
~# mkdir build
~# cd build
~# CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja ..
~# ninja

   2) checkout & compile dpdk v20.02
~# git clone git://dpdk.org/dpdk
~# cd dpdk
~# make config T=x86_64-native-linuxapp-gcc
~# sed -ri 's,(MLX5_PMD=).*,\1y,' build/.config
~# sed -ri 's,(IBVERBS_LINK_STATIC_PMD=).*,\1y,' build/.config
~# make EXTRA_CFLAGS=-I/home/bganne/src/rdma-core/build/include EXTRA_LDFLAGS=-L/home/bganne/src/rdma-core/build/lib PKG_CONFIG_PATH=/home/bganne/src/rdma-core/build/lib/pkgconfig
~# sudo gdb --args ./build/app/testpmd -w 0000:5e:00.0 -w 0000:5e:00.1 -l 4,11,35 -- -a --forward-mode=rxonly -i

The backtrace looks like this:
#0  mlx5dv_dr_table_create (dmn=0x555556c641b0, level=65534) at ../providers/mlx5/dr_table.c:183
#1  0x0000555555dfaeaa in flow_dv_tbl_resource_get (dev=<optimized out>, table_id=65534, egress=<optimized out>, transfer=<optimized out>, error=0x7fffffffdca0) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:6746
#2  0x0000555555e02b28 in __flow_dv_translate (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>, dev_flow=0x100388300, attr=<optimized out>, items=<optimized out>, actions=<optimized out>, error=<optimized out>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:7503
#3  0x0000555555e04954 in flow_dv_translate (dev=0x555556bbcdc0 <rte_eth_devices>, dev_flow=<optimized out>, attr=<optimized out>, items=<optimized out>, actions=<optimized out>, error=<optimized out>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:8841
#4  0x0000555555df152f in flow_drv_translate (error=0x7fffffffdca0, actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88, dev_flow=<optimized out>, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:2571
#5  flow_create_split_inner (error=0x7fffffffdca0, external=false, actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88, prefix_layers=0, sub_flow=0x0, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3490
#6  flow_create_split_metadata (error=0x7fffffffdca0, external=false, actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88, prefix_layers=0, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3865
#7  flow_create_split_meter (error=0x7fffffffdca0, external=false, actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4121
#8  flow_create_split_outer (error=0x7fffffffdca0, external=false, actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4178
#9  flow_list_create (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>, list=list@entry=0x0, attr=attr@entry=0x7fffffffdc94, items=items@entry=0x7fffffffdcc0, actions=actions@entry=0x7fffffffdce0, external=external@entry=false, error=0x7fffffffdca0) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4306
#10 0x0000555555df8587 in mlx5_flow_discover_mreg_c (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:5747
#11 0x0000555555d692a6 in mlx5_dev_spawn (config=..., spawn=0x1003e9e00, dpdk_dev=0x555556dd6fe0) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:2763
#12 mlx5_pci_probe (pci_drv=<optimized out>, pci_dev=<optimized out>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:3363
#13 0x0000555555a411c8 in pci_probe_all_drivers ()
#14 0x0000555555a412f8 in rte_pci_probe ()
#15 0x0000555555a083da in rte_bus_probe ()
#16 0x00005555559f204d in rte_eal_init ()
#17 0x00005555556a0d45 in main ()

Best
ben

Re: [dpdk-users] mlx5 pmd + rdma-core 28 init failure

[Benoit Ganne (bganne)]

Adding MLX5 PMD maintainers.

I also checked with latest rdma-core master and latest DPDK master and it is failing with the same issue.
Any recommendation?

Best
ben

> -----Original Message-----
> From: Benoit Ganne (bganne)
> Sent: mercredi 1 avril 2020 18:52
> To: users@dpdk.org
> Subject: mlx5 pmd + rdma-core 28 init failure
> 
> Hi all,
> 
> I have troubles making DPDK v20.02 MLX5 PMD working with rdma-core v28.0:
> it looks like the flow initialization done while initializing the device
> fails in rdma-core providers/mlx5/dr_table.c:mlx5dv_dr_table_create()
> because of unsupported parameters.
> The issue is the following test in rdma-core
> providers/mlx5/dr_table.c:mlx5dv_dr_table_create():
>     if (level && !dmn->info.supp_sw_steering) {
>         errno = EOPNOTSUPP;
>         goto dec_ref;
>     }
> Where level == 65534 and dmn->info.supp_sw_steering == false, hence the
> test is true and the function returns EOPNOTSUPP.
> 
> Is this expected? It does work fine with ibv instead of dv API. Any help
> appreciated.
> 
> Here is what I do:
>    1) checkout & compile rdma-core v28.0
> ~# git clone https://github.com/linux-rdma/rdma-core
> ~# cd rdma-core
> ~# git checkout v28.0
> ~# mkdir build
> ~# cd build
> ~# CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja ..
> ~# ninja
> 
>    2) checkout & compile dpdk v20.02
> ~# git clone git://dpdk.org/dpdk
> ~# cd dpdk
> ~# make config T=x86_64-native-linuxapp-gcc
> ~# sed -ri 's,(MLX5_PMD=).*,\1y,' build/.config
> ~# sed -ri 's,(IBVERBS_LINK_STATIC_PMD=).*,\1y,' build/.config
> ~# make EXTRA_CFLAGS=-I/home/bganne/src/rdma-core/build/include
> EXTRA_LDFLAGS=-L/home/bganne/src/rdma-core/build/lib
> PKG_CONFIG_PATH=/home/bganne/src/rdma-core/build/lib/pkgconfig
> ~# sudo gdb --args ./build/app/testpmd -w 0000:5e:00.0 -w 0000:5e:00.1 -l
> 4,11,35 -- -a --forward-mode=rxonly -i
> 
> The backtrace looks like this:
> #0  mlx5dv_dr_table_create (dmn=0x555556c641b0, level=65534) at
> ../providers/mlx5/dr_table.c:183
> #1  0x0000555555dfaeaa in flow_dv_tbl_resource_get (dev=<optimized out>,
> table_id=65534, egress=<optimized out>, transfer=<optimized out>,
> error=0x7fffffffdca0) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:6746
> #2  0x0000555555e02b28 in __flow_dv_translate
> (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>, dev_flow=0x100388300,
> attr=<optimized out>, items=<optimized out>, actions=<optimized out>,
> error=<optimized out>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:7503
> #3  0x0000555555e04954 in flow_dv_translate (dev=0x555556bbcdc0
> <rte_eth_devices>, dev_flow=<optimized out>, attr=<optimized out>,
> items=<optimized out>, actions=<optimized out>, error=<optimized out>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:8841
> #4  0x0000555555df152f in flow_drv_translate (error=0x7fffffffdca0,
> actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> dev_flow=<optimized out>, dev=0x555556bbcdc0 <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:2571
> #5  flow_create_split_inner (error=0x7fffffffdca0, external=false,
> actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> prefix_layers=0, sub_flow=0x0, flow=0x1003885c0, dev=0x555556bbcdc0
> <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3490
> #6  flow_create_split_metadata (error=0x7fffffffdca0, external=false,
> actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> prefix_layers=0, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>)
> at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3865
> #7  flow_create_split_meter (error=0x7fffffffdca0, external=false,
> actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94,
> flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4121
> #8  flow_create_split_outer (error=0x7fffffffdca0, external=false,
> actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94,
> flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4178
> #9  flow_list_create (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>,
> list=list@entry=0x0, attr=attr@entry=0x7fffffffdc94,
> items=items@entry=0x7fffffffdcc0, actions=actions@entry=0x7fffffffdce0,
> external=external@entry=false, error=0x7fffffffdca0) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4306
> #10 0x0000555555df8587 in mlx5_flow_discover_mreg_c
> (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:5747
> #11 0x0000555555d692a6 in mlx5_dev_spawn (config=..., spawn=0x1003e9e00,
> dpdk_dev=0x555556dd6fe0) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:2763
> #12 mlx5_pci_probe (pci_drv=<optimized out>, pci_dev=<optimized out>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:3363
> #13 0x0000555555a411c8 in pci_probe_all_drivers ()
> #14 0x0000555555a412f8 in rte_pci_probe ()
> #15 0x0000555555a083da in rte_bus_probe ()
> #16 0x00005555559f204d in rte_eal_init ()
> #17 0x00005555556a0d45 in main ()
> 
> Best
> ben

Re: [dpdk-users] mlx5 pmd + rdma-core 28 init failure

[Matan Azrad]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="iso-8859-8-i", Size: 5967 bytes --]

Can you run with log level debug and send us the log?

äùâ àú Outlook òáåø Android<https://aka.ms/ghei36>

________________________________
From: Benoit Ganne (bganne) <bganne@cisco.com>
Sent: Thursday, April 2, 2020 6:50:22 PM
To: users@dpdk.org <users@dpdk.org>
Cc: Matan Azrad <matan@mellanox.com>; Shahaf Shuler <shahafs@mellanox.com>; Slava Ovsiienko <viacheslavo@mellanox.com>
Subject: RE: mlx5 pmd + rdma-core 28 init failure

Adding MLX5 PMD maintainers.

I also checked with latest rdma-core master and latest DPDK master and it is failing with the same issue.
Any recommendation?

Best
ben

> -----Original Message-----
> From: Benoit Ganne (bganne)
> Sent: mercredi 1 avril 2020 18:52
> To: users@dpdk.org
> Subject: mlx5 pmd + rdma-core 28 init failure
>
> Hi all,
>
> I have troubles making DPDK v20.02 MLX5 PMD working with rdma-core v28.0:
> it looks like the flow initialization done while initializing the device
> fails in rdma-core providers/mlx5/dr_table.c:mlx5dv_dr_table_create()
> because of unsupported parameters.
> The issue is the following test in rdma-core
> providers/mlx5/dr_table.c:mlx5dv_dr_table_create():
>     if (level && !dmn->info.supp_sw_steering) {
>         errno = EOPNOTSUPP;
>         goto dec_ref;
>     }
> Where level == 65534 and dmn->info.supp_sw_steering == false, hence the
> test is true and the function returns EOPNOTSUPP.
>
> Is this expected? It does work fine with ibv instead of dv API. Any help
> appreciated.
>
> Here is what I do:
>    1) checkout & compile rdma-core v28.0
> ~# git clone https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flinux-rdma%2Frdma-core&amp;data=02%7C01%7Cmatan%40mellanox.com%7C7984449b13af4519d70608d7d71d9000%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637214394282618427&amp;sdata=DSLvWg3cWwY8prSFOm8BAFf1I2bbKnqPOKBPEiy4b44%3D&amp;reserved=0
> ~# cd rdma-core
> ~# git checkout v28.0
> ~# mkdir build
> ~# cd build
> ~# CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja ..
> ~# ninja
>
>    2) checkout & compile dpdk v20.02
> ~# git clone git://dpdk.org/dpdk
> ~# cd dpdk
> ~# make config T=x86_64-native-linuxapp-gcc
> ~# sed -ri 's,(MLX5_PMD=).*,\1y,' build/.config
> ~# sed -ri 's,(IBVERBS_LINK_STATIC_PMD=).*,\1y,' build/.config
> ~# make EXTRA_CFLAGS=-I/home/bganne/src/rdma-core/build/include
> EXTRA_LDFLAGS=-L/home/bganne/src/rdma-core/build/lib
> PKG_CONFIG_PATH=/home/bganne/src/rdma-core/build/lib/pkgconfig
> ~# sudo gdb --args ./build/app/testpmd -w 0000:5e:00.0 -w 0000:5e:00.1 -l
> 4,11,35 -- -a --forward-mode=rxonly -i
>
> The backtrace looks like this:
> #0  mlx5dv_dr_table_create (dmn=0x555556c641b0, level=65534) at
> ../providers/mlx5/dr_table.c:183
> #1  0x0000555555dfaeaa in flow_dv_tbl_resource_get (dev=<optimized out>,
> table_id=65534, egress=<optimized out>, transfer=<optimized out>,
> error=0x7fffffffdca0) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:6746
> #2  0x0000555555e02b28 in __flow_dv_translate
> (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>, dev_flow=0x100388300,
> attr=<optimized out>, items=<optimized out>, actions=<optimized out>,
> error=<optimized out>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:7503
> #3  0x0000555555e04954 in flow_dv_translate (dev=0x555556bbcdc0
> <rte_eth_devices>, dev_flow=<optimized out>, attr=<optimized out>,
> items=<optimized out>, actions=<optimized out>, error=<optimized out>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:8841
> #4  0x0000555555df152f in flow_drv_translate (error=0x7fffffffdca0,
> actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> dev_flow=<optimized out>, dev=0x555556bbcdc0 <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:2571
> #5  flow_create_split_inner (error=0x7fffffffdca0, external=false,
> actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> prefix_layers=0, sub_flow=0x0, flow=0x1003885c0, dev=0x555556bbcdc0
> <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3490
> #6  flow_create_split_metadata (error=0x7fffffffdca0, external=false,
> actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> prefix_layers=0, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>)
> at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3865
> #7  flow_create_split_meter (error=0x7fffffffdca0, external=false,
> actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94,
> flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4121
> #8  flow_create_split_outer (error=0x7fffffffdca0, external=false,
> actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94,
> flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4178
> #9  flow_list_create (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>,
> list=list@entry=0x0, attr=attr@entry=0x7fffffffdc94,
> items=items@entry=0x7fffffffdcc0, actions=actions@entry=0x7fffffffdce0,
> external=external@entry=false, error=0x7fffffffdca0) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4306
> #10 0x0000555555df8587 in mlx5_flow_discover_mreg_c
> (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:5747
> #11 0x0000555555d692a6 in mlx5_dev_spawn (config=..., spawn=0x1003e9e00,
> dpdk_dev=0x555556dd6fe0) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:2763
> #12 mlx5_pci_probe (pci_drv=<optimized out>, pci_dev=<optimized out>) at
> /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:3363
> #13 0x0000555555a411c8 in pci_probe_all_drivers ()
> #14 0x0000555555a412f8 in rte_pci_probe ()
> #15 0x0000555555a083da in rte_bus_probe ()
> #16 0x00005555559f204d in rte_eal_init ()
> #17 0x00005555556a0d45 in main ()
>
> Best
> ben

Re: [dpdk-users] mlx5 pmd + rdma-core 28 init failure

[Benoit Ganne (bganne)]

> Can you run with log level debug and send us the log?

Here it is:
~# sudo ./build/app/testpmd --log-level=8 --log-level=pmd.common.mlx5:8 --log-level=pmd.net.mlx5:8 -w 0000:5e:00.0 -w 0000:5e:00.1 -l 4,11,35 -- -a --forward-mode=rxonly
EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:5e:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1013 net_mlx5
net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_0"
net_mlx5: mlx5.c:3074: mlx5_pci_probe(): PCI information matches for device "mlx5_0"
net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_1"
net_mlx5: mlx5.c:3305: mlx5_pci_probe(): no E-Switch support detected
net_mlx5: mlx5.c:2178: mlx5_dev_spawn(): naming Ethernet device "0000:5e:00.0"
net_mlx5: mlx5.c:548: mlx5_alloc_shared_ibctx(): DevX is NOT supported
net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree 0x1003dcb20 with table 0x1003d9640
net_mlx5: mlx5.c:2251: mlx5_dev_spawn(): MPW isn't supported
net_mlx5: mlx5.c:2257: mlx5_dev_spawn(): SWP support: 7
net_mlx5: mlx5.c:2266: mlx5_dev_spawn(): 	min_single_stride_log_num_of_bytes: 0
net_mlx5: mlx5.c:2268: mlx5_dev_spawn(): 	max_single_stride_log_num_of_bytes: 0
net_mlx5: mlx5.c:2270: mlx5_dev_spawn(): 	min_single_wqe_log_num_of_strides: 0
net_mlx5: mlx5.c:2272: mlx5_dev_spawn(): 	max_single_wqe_log_num_of_strides: 0
net_mlx5: mlx5.c:2274: mlx5_dev_spawn(): 	supported_qpts: 0
net_mlx5: mlx5.c:2275: mlx5_dev_spawn(): device supports Multi-Packet RQ
net_mlx5: mlx5.c:2310: mlx5_dev_spawn(): tunnel offloading is supported
net_mlx5: mlx5.c:2322: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel offloading is not supported
net_mlx5: mlx5.c:2473: mlx5_dev_spawn(): checksum offloading is supported
net_mlx5: mlx5.c:2493: mlx5_dev_spawn(): maximum Rx indirection table size is 512
net_mlx5: mlx5.c:2497: mlx5_dev_spawn(): VLAN stripping is supported
net_mlx5: mlx5.c:2501: mlx5_dev_spawn(): FCS stripping configuration is supported
net_mlx5: mlx5.c:2531: mlx5_dev_spawn(): MPS is disabled
net_mlx5: mlx5.c:2656: mlx5_dev_spawn(): port 0 MAC address is 24:8a:07:5b:14:14
net_mlx5: mlx5.c:2663: mlx5_dev_spawn(): port 0 ifname is "enp94s0f0"
net_mlx5: mlx5.c:2676: mlx5_dev_spawn(): port 0 MTU is 9216
net_mlx5: mlx5.c:2703: mlx5_dev_spawn(): port 0 forcing Ethernet interface up
net_mlx5: mlx5.c:1836: mlx5_set_min_inline(): min tx inline configured: 18
net_mlx5: mlx5_utils.c:41: mlx5_hlist_create(): Hash list with mlx5_0_flow_table size 0x1000 is created.

net_mlx5: mlx5_utils.c:41: mlx5_hlist_create(): Hash list with mlx5_0_tags size 0x2000 is created.

net_mlx5: mlx5_flow.c:550: mlx5_flow_discover_priorities(): port 0 flow maximum priority: 3
net_mlx5: mlx5.c:1887: mlx5_set_metadata_mask(): metadata mode 0
net_mlx5: mlx5.c:1888: mlx5_set_metadata_mask(): metadata MARK mask 00FFFFFF
net_mlx5: mlx5.c:1889: mlx5_set_metadata_mask(): metadata META mask FFFFFFFF
net_mlx5: mlx5.c:1890: mlx5_set_metadata_mask(): metadata reg_c0 mask FFFFFFFF
net_mlx5: mlx5.c:2771: mlx5_dev_spawn(): port 0 extensive metadata register is not supported
EAL: PCI device 0000:5e:00.1 on NUMA socket 0
EAL:   probe driver: 15b3:1013 net_mlx5
net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_0"
net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_1"
net_mlx5: mlx5.c:3074: mlx5_pci_probe(): PCI information matches for device "mlx5_1"
net_mlx5: mlx5.c:3305: mlx5_pci_probe(): no E-Switch support detected
net_mlx5: mlx5.c:2178: mlx5_dev_spawn(): naming Ethernet device "0000:5e:00.1"
net_mlx5: mlx5.c:548: mlx5_alloc_shared_ibctx(): DevX is NOT supported
net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree 0x10037b420 with table 0x100377f40
net_mlx5: mlx5.c:2251: mlx5_dev_spawn(): MPW isn't supported
net_mlx5: mlx5.c:2257: mlx5_dev_spawn(): SWP support: 7
net_mlx5: mlx5.c:2266: mlx5_dev_spawn(): 	min_single_stride_log_num_of_bytes: 0
net_mlx5: mlx5.c:2268: mlx5_dev_spawn(): 	max_single_stride_log_num_of_bytes: 0
net_mlx5: mlx5.c:2270: mlx5_dev_spawn(): 	min_single_wqe_log_num_of_strides: 0
net_mlx5: mlx5.c:2272: mlx5_dev_spawn(): 	max_single_wqe_log_num_of_strides: 0
net_mlx5: mlx5.c:2274: mlx5_dev_spawn(): 	supported_qpts: 0
net_mlx5: mlx5.c:2275: mlx5_dev_spawn(): device supports Multi-Packet RQ
net_mlx5: mlx5.c:2310: mlx5_dev_spawn(): tunnel offloading is supported
net_mlx5: mlx5.c:2322: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel offloading is not supported
net_mlx5: mlx5.c:2473: mlx5_dev_spawn(): checksum offloading is supported
net_mlx5: mlx5.c:2493: mlx5_dev_spawn(): maximum Rx indirection table size is 512
net_mlx5: mlx5.c:2497: mlx5_dev_spawn(): VLAN stripping is supported
net_mlx5: mlx5.c:2501: mlx5_dev_spawn(): FCS stripping configuration is supported
net_mlx5: mlx5.c:2531: mlx5_dev_spawn(): MPS is disabled
net_mlx5: mlx5.c:2656: mlx5_dev_spawn(): port 1 MAC address is 24:8a:07:5b:14:15
net_mlx5: mlx5.c:2663: mlx5_dev_spawn(): port 1 ifname is "enp94s0f1"
net_mlx5: mlx5.c:2676: mlx5_dev_spawn(): port 1 MTU is 9216
net_mlx5: mlx5.c:2703: mlx5_dev_spawn(): port 1 forcing Ethernet interface up
net_mlx5: mlx5.c:1836: mlx5_set_min_inline(): min tx inline configured: 18
net_mlx5: mlx5_utils.c:41: mlx5_hlist_create(): Hash list with mlx5_1_flow_table size 0x1000 is created.

net_mlx5: mlx5_utils.c:41: mlx5_hlist_create(): Hash list with mlx5_1_tags size 0x2000 is created.

net_mlx5: mlx5_flow.c:550: mlx5_flow_discover_priorities(): port 1 flow maximum priority: 3
net_mlx5: mlx5.c:1887: mlx5_set_metadata_mask(): metadata mode 0
net_mlx5: mlx5.c:1888: mlx5_set_metadata_mask(): metadata MARK mask 00FFFFFF
net_mlx5: mlx5.c:1889: mlx5_set_metadata_mask(): metadata META mask FFFFFFFF
net_mlx5: mlx5.c:1890: mlx5_set_metadata_mask(): metadata reg_c0 mask FFFFFFFF
net_mlx5: mlx5.c:2771: mlx5_dev_spawn(): port 1 extensive metadata register is not supported
Auto-start selected
Set rxonly packet forwarding mode
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
net_mlx5: mlx5_ethdev.c:424: mlx5_dev_configure(): port 0 Tx queues number update: 0 -> 1
net_mlx5: mlx5_ethdev.c:435: mlx5_dev_configure(): port 0 Rx queues number update: 0 -> 1
net_mlx5: mlx5_txq.c:172: mlx5_tx_queue_pre_setup(): port 0 configuring queue 0 for 256 descriptors
net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree 0x101051c28 with table 0x1010505c0
net_mlx5: mlx5_txq.c:225: mlx5_tx_queue_setup(): port 0 adding Tx queue 0 to list
net_mlx5: mlx5_rxq.c:468: mlx5_rx_queue_pre_setup(): port 0 configuring Rx queue 0 for 256 descriptors
net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree 0x10104fa6c with table 0x10104e500
net_mlx5: mlx5_rxq.c:1921: mlx5_rxq_new(): port 0 maximum number of segments per packet: 1
net_mlx5: mlx5_rxq.c:1759: mlx5_max_lro_msg_size_adjust(): port 0 Rx Queue 0 max LRO message size adjusted to 1280 bytes
net_mlx5: mlx5_rxq.c:1968: mlx5_rxq_new(): port 0 CRC stripping is enabled, 0 bytes will be subtracted from incoming frames to hide it
net_mlx5: mlx5_rxq.c:525: mlx5_rx_queue_setup(): port 0 adding Rx queue 0 to list
net_mlx5: mlx5_trigger.c:276: mlx5_dev_start(): port 0 starting device
net_mlx5: mlx5_ethdev.c:493: mlx5_dev_configure_rss_reta(): port 0 Rx queues number update: 1 -> 1
net_mlx5: mlx5_txq.c:55: txq_alloc_elts(): port 0 Tx queue 0 allocated and configured 256 WRs
net_mlx5: mlx5_txq.c:771: mlx5_txq_obj_new(): port 0: uar_mmap_offset 0x306000
net_mlx5: mlx5_trigger.c:145: mlx5_rxq_start(): port 0 Rx queue 0 registering mp mbuf_pool_socket_0 having 1 chunks
net_mlx5: mlx5_mr.c:600: mlx5_mr_create_primary(): port 0 creating a MR using address (0x10109b4c0)
net_mlx5: mlx5_mr.c:649: mlx5_mr_create_primary(): port 0 extending 0x10109b4c0 to [0x100200000, 0x118200000), page_sz=0x200000, ms_n=192
net_mlx5: mlx5_mr.c:786: mlx5_mr_create_primary(): port 0 MR CREATED (0x10104e140) for 0x10109b4c0:
  [0x100200000, 0x118200000), lkey=0x104b0100 base_idx=0 ms_n=192, ms_bmp_n=192
net_mlx5: mlx5_mr.c:345: mr_insert_dev_cache(): device mlx5_0 inserting MR(0x10104e140) to global cache
net_mlx5: mlx5_mr.c:173: mr_btree_insert(): inserted B-tree(0x1003dcb20)[1], [0x100200000, 0x118200000) lkey=0x104b0100
net_mlx5: mlx5_mr.c:173: mr_btree_insert(): inserted B-tree(0x10104fa6c)[1], [0x100200000, 0x118200000) lkey=0x104b0100
net_mlx5: mlx5_rxq.c:257: rxq_alloc_elts_sprq(): port 0 Rx queue 0 allocated and configured 256 segments (max 256 packets)
net_mlx5: mlx5_rxq.c:1402: mlx5_rxq_obj_new(): port 0 device_attr.max_qp_wr is 32768
net_mlx5: mlx5_rxq.c:1404: mlx5_rxq_obj_new(): port 0 device_attr.max_sge is 30
net_mlx5: mlx5_rxq.c:1478: mlx5_rxq_obj_new(): port 0 rxq 0 updated with 0x7ffda24f21f8
net_mlx5: mlx5_trigger.c:322: mlx5_dev_start(): port 0 failed to set defaults flows
net_mlx5: mlx5_rxq.c:333: rxq_free_elts_sprq(): port 0 Rx queue 0 freeing WRs
Fail to start port 0
Configuring Port 1 (socket 0)
net_mlx5: mlx5_ethdev.c:424: mlx5_dev_configure(): port 1 Tx queues number update: 0 -> 1
net_mlx5: mlx5_ethdev.c:435: mlx5_dev_configure(): port 1 Rx queues number update: 0 -> 1
net_mlx5: mlx5_txq.c:172: mlx5_tx_queue_pre_setup(): port 1 configuring queue 0 for 256 descriptors
net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree 0x10104d6a8 with table 0x10104c040
net_mlx5: mlx5_txq.c:225: mlx5_tx_queue_setup(): port 1 adding Tx queue 0 to list
net_mlx5: mlx5_rxq.c:468: mlx5_rx_queue_pre_setup(): port 1 configuring Rx queue 0 for 256 descriptors
net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree 0x10104b4ec with table 0x101049f80
net_mlx5: mlx5_rxq.c:1921: mlx5_rxq_new(): port 1 maximum number of segments per packet: 1
net_mlx5: mlx5_rxq.c:1759: mlx5_max_lro_msg_size_adjust(): port 1 Rx Queue 0 max LRO message size adjusted to 1280 bytes
net_mlx5: mlx5_rxq.c:1968: mlx5_rxq_new(): port 1 CRC stripping is enabled, 0 bytes will be subtracted from incoming frames to hide it
net_mlx5: mlx5_rxq.c:525: mlx5_rx_queue_setup(): port 1 adding Rx queue 0 to list
net_mlx5: mlx5_trigger.c:276: mlx5_dev_start(): port 1 starting device
net_mlx5: mlx5_ethdev.c:493: mlx5_dev_configure_rss_reta(): port 1 Rx queues number update: 1 -> 1
net_mlx5: mlx5_txq.c:55: txq_alloc_elts(): port 1 Tx queue 0 allocated and configured 256 WRs
net_mlx5: mlx5_txq.c:771: mlx5_txq_obj_new(): port 1: uar_mmap_offset 0x306000
net_mlx5: mlx5_trigger.c:145: mlx5_rxq_start(): port 1 Rx queue 0 registering mp mbuf_pool_socket_0 having 1 chunks
net_mlx5: mlx5_mr.c:600: mlx5_mr_create_primary(): port 1 creating a MR using address (0x10109b4c0)
net_mlx5: mlx5_mr.c:649: mlx5_mr_create_primary(): port 1 extending 0x10109b4c0 to [0x100200000, 0x118200000), page_sz=0x200000, ms_n=192
net_mlx5: mlx5_mr.c:786: mlx5_mr_create_primary(): port 1 MR CREATED (0x101045dc0) for 0x10109b4c0:
  [0x100200000, 0x118200000), lkey=0xcac80f00 base_idx=0 ms_n=192, ms_bmp_n=192
net_mlx5: mlx5_mr.c:345: mr_insert_dev_cache(): device mlx5_1 inserting MR(0x101045dc0) to global cache
net_mlx5: mlx5_mr.c:173: mr_btree_insert(): inserted B-tree(0x10037b420)[1], [0x100200000, 0x118200000) lkey=0xcac80f00
net_mlx5: mlx5_mr.c:173: mr_btree_insert(): inserted B-tree(0x10104b4ec)[1], [0x100200000, 0x118200000) lkey=0xcac80f00
net_mlx5: mlx5_rxq.c:257: rxq_alloc_elts_sprq(): port 1 Rx queue 0 allocated and configured 256 segments (max 256 packets)
net_mlx5: mlx5_rxq.c:1402: mlx5_rxq_obj_new(): port 1 device_attr.max_qp_wr is 32768
net_mlx5: mlx5_rxq.c:1404: mlx5_rxq_obj_new(): port 1 device_attr.max_sge is 30
net_mlx5: mlx5_rxq.c:1478: mlx5_rxq_obj_new(): port 1 rxq 0 updated with 0x7ffda24f21f8
net_mlx5: mlx5_trigger.c:322: mlx5_dev_start(): port 1 failed to set defaults flows
net_mlx5: mlx5_rxq.c:333: rxq_free_elts_sprq(): port 1 Rx queue 0 freeing WRs
Fail to start port 1
Please stop the ports first
Done
No commandline core given, start packet forwarding
Not all ports were started
Press enter to exit

Stopping port 0...
Stopping ports...
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
net_mlx5: mlx5.c:1233: mlx5_dev_close(): port 0 closing device "mlx5_0"
net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree 0x10104fa6c with table 0x10104e500
net_mlx5: mlx5_txq.c:77: txq_free_elts(): port 0 Tx queue 0 freeing WRs
net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree 0x101051c28 with table 0x1010505c0
net_mlx5: mlx5_mr.c:1569: mlx5_mr_dump_dev(): device mlx5_0 MR[0], LKey = 0x104b0100, ms_n = 192, ms_bmp_n = 192
net_mlx5: mlx5_mr.c:1579: mlx5_mr_dump_dev():   chunk[0], [0x100200000, 0x118200000)
net_mlx5: mlx5_mr.c:1582: mlx5_mr_dump_dev(): device mlx5_0 dumping global cache
net_mlx5: mlx5_mr.c:256: mlx5_mr_btree_dump(): B-tree(0x1003dcb20)[0], [0x0, 0x0) lkey=0xffffffff
net_mlx5: mlx5_mr.c:256: mlx5_mr_btree_dump(): B-tree(0x1003dcb20)[1], [0x100200000, 0x118200000) lkey=0x104b0100
net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree 0x1003dcb20 with table 0x1003d9640
net_mlx5: mlx5_mr.c:459: mr_free(): freeing MR(0x10104e140):
Done

Shutting down port 1...
Closing ports...
net_mlx5: mlx5.c:1233: mlx5_dev_close(): port 1 closing device "mlx5_1"
net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree 0x10104b4ec with table 0x101049f80
net_mlx5: mlx5_txq.c:77: txq_free_elts(): port 1 Tx queue 0 freeing WRs
net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree 0x10104d6a8 with table 0x10104c040
net_mlx5: mlx5_mr.c:1569: mlx5_mr_dump_dev(): device mlx5_1 MR[0], LKey = 0xcac80f00, ms_n = 192, ms_bmp_n = 192
net_mlx5: mlx5_mr.c:1579: mlx5_mr_dump_dev():   chunk[0], [0x100200000, 0x118200000)
net_mlx5: mlx5_mr.c:1582: mlx5_mr_dump_dev(): device mlx5_1 dumping global cache
net_mlx5: mlx5_mr.c:256: mlx5_mr_btree_dump(): B-tree(0x10037b420)[0], [0x0, 0x0) lkey=0xffffffff
net_mlx5: mlx5_mr.c:256: mlx5_mr_btree_dump(): B-tree(0x10037b420)[1], [0x100200000, 0x118200000) lkey=0xcac80f00
net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree 0x10037b420 with table 0x100377f40
net_mlx5: mlx5_mr.c:459: mr_free(): freeing MR(0x101045dc0):
Done

Bye...

Best
ben

> From: Benoit Ganne (bganne) <bganne@cisco.com>
> Sent: Thursday, April 2, 2020 6:50:22 PM
> To: users@dpdk.org <users@dpdk.org>
> Cc: Matan Azrad <matan@mellanox.com>; Shahaf Shuler
> <shahafs@mellanox.com>; Slava Ovsiienko <viacheslavo@mellanox.com>
> Subject: RE: mlx5 pmd + rdma-core 28 init failure
> 
> Adding MLX5 PMD maintainers.
> 
> I also checked with latest rdma-core master and latest DPDK master and it
> is failing with the same issue.
> Any recommendation?
> 
> Best
> ben
> 
> > -----Original Message-----
> > From: Benoit Ganne (bganne)
> > Sent: mercredi 1 avril 2020 18:52
> > To: users@dpdk.org
> > Subject: mlx5 pmd + rdma-core 28 init failure
> >
> > Hi all,
> >
> > I have troubles making DPDK v20.02 MLX5 PMD working with rdma-core
> v28.0:
> > it looks like the flow initialization done while initializing the device
> > fails in rdma-core providers/mlx5/dr_table.c:mlx5dv_dr_table_create()
> > because of unsupported parameters.
> > The issue is the following test in rdma-core
> > providers/mlx5/dr_table.c:mlx5dv_dr_table_create():
> >     if (level && !dmn->info.supp_sw_steering) {
> >         errno = EOPNOTSUPP;
> >         goto dec_ref;
> >     }
> > Where level == 65534 and dmn->info.supp_sw_steering == false, hence the
> > test is true and the function returns EOPNOTSUPP.
> >
> > Is this expected? It does work fine with ibv instead of dv API. Any help
> > appreciated.
> >
> > Here is what I do:
> >    1) checkout & compile rdma-core v28.0
> > ~# git clone
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.c
> om%2Flinux-rdma%2Frdma-
> core&amp;data=02%7C01%7Cmatan%40mellanox.com%7C7984449b13af4519d70608d7d71
> d9000%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637214394282618427&amp;
> sdata=DSLvWg3cWwY8prSFOm8BAFf1I2bbKnqPOKBPEiy4b44%3D&amp;reserved=0
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
> com%2Flinux-rdma%2Frdma-
> core&amp;data=02%7C01%7Cmatan%40mellanox.com%7C7984449b13af4519d70608d7d71
> d9000%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637214394282618427&amp;
> sdata=DSLvWg3cWwY8prSFOm8BAFf1I2bbKnqPOKBPEiy4b44%3D&amp;reserved=0>
> > ~# cd rdma-core
> > ~# git checkout v28.0
> > ~# mkdir build
> > ~# cd build
> > ~# CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja ..
> > ~# ninja
> >
> >    2) checkout & compile dpdk v20.02
> > ~# git clone git://dpdk.org/dpdk
> > ~# cd dpdk
> > ~# make config T=x86_64-native-linuxapp-gcc
> > ~# sed -ri 's,(MLX5_PMD=).*,\1y,' build/.config
> > ~# sed -ri 's,(IBVERBS_LINK_STATIC_PMD=).*,\1y,' build/.config
> > ~# make EXTRA_CFLAGS=-I/home/bganne/src/rdma-core/build/include
> > EXTRA_LDFLAGS=-L/home/bganne/src/rdma-core/build/lib
> > PKG_CONFIG_PATH=/home/bganne/src/rdma-core/build/lib/pkgconfig
> > ~# sudo gdb --args ./build/app/testpmd -w 0000:5e:00.0 -w 0000:5e:00.1 -
> l
> > 4,11,35 -- -a --forward-mode=rxonly -i
> >
> > The backtrace looks like this:
> > #0  mlx5dv_dr_table_create (dmn=0x555556c641b0, level=65534) at
> > ../providers/mlx5/dr_table.c:183
> > #1  0x0000555555dfaeaa in flow_dv_tbl_resource_get (dev=<optimized out>,
> > table_id=65534, egress=<optimized out>, transfer=<optimized out>,
> > error=0x7fffffffdca0) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:6746
> > #2  0x0000555555e02b28 in __flow_dv_translate
> > (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>, dev_flow=0x100388300,
> > attr=<optimized out>, items=<optimized out>, actions=<optimized out>,
> > error=<optimized out>) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:7503
> > #3  0x0000555555e04954 in flow_dv_translate (dev=0x555556bbcdc0
> > <rte_eth_devices>, dev_flow=<optimized out>, attr=<optimized out>,
> > items=<optimized out>, actions=<optimized out>, error=<optimized out>)
> at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:8841
> > #4  0x0000555555df152f in flow_drv_translate (error=0x7fffffffdca0,
> > actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> > dev_flow=<optimized out>, dev=0x555556bbcdc0 <rte_eth_devices>) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:2571
> > #5  flow_create_split_inner (error=0x7fffffffdca0, external=false,
> > actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> > prefix_layers=0, sub_flow=0x0, flow=0x1003885c0, dev=0x555556bbcdc0
> > <rte_eth_devices>) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3490
> > #6  flow_create_split_metadata (error=0x7fffffffdca0, external=false,
> > actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> > prefix_layers=0, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>)
> > at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3865
> > #7  flow_create_split_meter (error=0x7fffffffdca0, external=false,
> > actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94,
> > flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4121
> > #8  flow_create_split_outer (error=0x7fffffffdca0, external=false,
> > actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94,
> > flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4178
> > #9  flow_list_create (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>,
> > list=list@entry=0x0, attr=attr@entry=0x7fffffffdc94,
> > items=items@entry=0x7fffffffdcc0, actions=actions@entry=0x7fffffffdce0,
> > external=external@entry=false, error=0x7fffffffdca0) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4306
> > #10 0x0000555555df8587 in mlx5_flow_discover_mreg_c
> > (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:5747
> > #11 0x0000555555d692a6 in mlx5_dev_spawn (config=..., spawn=0x1003e9e00,
> > dpdk_dev=0x555556dd6fe0) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:2763
> > #12 mlx5_pci_probe (pci_drv=<optimized out>, pci_dev=<optimized out>) at
> > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:3363
> > #13 0x0000555555a411c8 in pci_probe_all_drivers ()
> > #14 0x0000555555a412f8 in rte_pci_probe ()
> > #15 0x0000555555a083da in rte_bus_probe ()
> > #16 0x00005555559f204d in rte_eal_init ()
> > #17 0x00005555556a0d45 in main ()
> >
> > Best
> > ben

Re: [dpdk-users] mlx5 pmd + rdma-core 28 init failure

[Matan Azrad]

Hi Benoit

Thanks.

Please let us know if working with devargs dv_flow_en=0 solves the issue....

./build/app/testpmd --log-level=8 --log-level=pmd.common.mlx5:8 - -log-level=pmd.net.mlx5:8 -w 0000:5e:00.0.dv_flow_en=0 -w 0000:5e:00.1,dv_flow_en=0 -l 4,11,35 -- -a -- forward-mode=rxonly

Please let us know also your FW version...

Matan

> -----Original Message-----
> From: Benoit Ganne (bganne) <bganne@cisco.com>
> Sent: Thursday, April 2, 2020 8:03 PM
> To: Matan Azrad <matan@mellanox.com>; users@dpdk.org
> Cc: Shahaf Shuler <shahafs@mellanox.com>; Slava Ovsiienko
> <viacheslavo@mellanox.com>
> Subject: RE: mlx5 pmd + rdma-core 28 init failure
> 
> > Can you run with log level debug and send us the log?
> 
> Here it is:
> ~# sudo ./build/app/testpmd --log-level=8 --log-level=pmd.common.mlx5:8 -
> -log-level=pmd.net.mlx5:8 -w 0000:5e:00.0 -w 0000:5e:00.1 -l 4,11,35 -- -a --
> forward-mode=rxonly
> EAL: Detected 48 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:5e:00.0 on NUMA socket 0
> EAL:   probe driver: 15b3:1013 net_mlx5
> net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_0"
> net_mlx5: mlx5.c:3074: mlx5_pci_probe(): PCI information matches for
> device "mlx5_0"
> net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_1"
> net_mlx5: mlx5.c:3305: mlx5_pci_probe(): no E-Switch support detected
> net_mlx5: mlx5.c:2178: mlx5_dev_spawn(): naming Ethernet device
> "0000:5e:00.0"
> net_mlx5: mlx5.c:548: mlx5_alloc_shared_ibctx(): DevX is NOT supported
> net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree
> 0x1003dcb20 with table 0x1003d9640
> net_mlx5: mlx5.c:2251: mlx5_dev_spawn(): MPW isn't supported
> net_mlx5: mlx5.c:2257: mlx5_dev_spawn(): SWP support: 7
> net_mlx5: mlx5.c:2266: mlx5_dev_spawn():
> 	min_single_stride_log_num_of_bytes: 0
> net_mlx5: mlx5.c:2268: mlx5_dev_spawn():
> 	max_single_stride_log_num_of_bytes: 0
> net_mlx5: mlx5.c:2270: mlx5_dev_spawn():
> 	min_single_wqe_log_num_of_strides: 0
> net_mlx5: mlx5.c:2272: mlx5_dev_spawn():
> 	max_single_wqe_log_num_of_strides: 0
> net_mlx5: mlx5.c:2274: mlx5_dev_spawn(): 	supported_qpts: 0
> net_mlx5: mlx5.c:2275: mlx5_dev_spawn(): device supports Multi-Packet RQ
> net_mlx5: mlx5.c:2310: mlx5_dev_spawn(): tunnel offloading is supported
> net_mlx5: mlx5.c:2322: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel
> offloading is not supported
> net_mlx5: mlx5.c:2473: mlx5_dev_spawn(): checksum offloading is
> supported
> net_mlx5: mlx5.c:2493: mlx5_dev_spawn(): maximum Rx indirection table
> size is 512
> net_mlx5: mlx5.c:2497: mlx5_dev_spawn(): VLAN stripping is supported
> net_mlx5: mlx5.c:2501: mlx5_dev_spawn(): FCS stripping configuration is
> supported
> net_mlx5: mlx5.c:2531: mlx5_dev_spawn(): MPS is disabled
> net_mlx5: mlx5.c:2656: mlx5_dev_spawn(): port 0 MAC address is
> 24:8a:07:5b:14:14
> net_mlx5: mlx5.c:2663: mlx5_dev_spawn(): port 0 ifname is "enp94s0f0"
> net_mlx5: mlx5.c:2676: mlx5_dev_spawn(): port 0 MTU is 9216
> net_mlx5: mlx5.c:2703: mlx5_dev_spawn(): port 0 forcing Ethernet interface
> up
> net_mlx5: mlx5.c:1836: mlx5_set_min_inline(): min tx inline configured: 18
> net_mlx5: mlx5_utils.c:41: mlx5_hlist_create(): Hash list with
> mlx5_0_flow_table size 0x1000 is created.
> 
> net_mlx5: mlx5_utils.c:41: mlx5_hlist_create(): Hash list with mlx5_0_tags
> size 0x2000 is created.
> 
> net_mlx5: mlx5_flow.c:550: mlx5_flow_discover_priorities(): port 0 flow
> maximum priority: 3
> net_mlx5: mlx5.c:1887: mlx5_set_metadata_mask(): metadata mode 0
> net_mlx5: mlx5.c:1888: mlx5_set_metadata_mask(): metadata MARK mask
> 00FFFFFF
> net_mlx5: mlx5.c:1889: mlx5_set_metadata_mask(): metadata META mask
> FFFFFFFF
> net_mlx5: mlx5.c:1890: mlx5_set_metadata_mask(): metadata reg_c0 mask
> FFFFFFFF
> net_mlx5: mlx5.c:2771: mlx5_dev_spawn(): port 0 extensive metadata
> register is not supported
> EAL: PCI device 0000:5e:00.1 on NUMA socket 0
> EAL:   probe driver: 15b3:1013 net_mlx5
> net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_0"
> net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_1"
> net_mlx5: mlx5.c:3074: mlx5_pci_probe(): PCI information matches for
> device "mlx5_1"
> net_mlx5: mlx5.c:3305: mlx5_pci_probe(): no E-Switch support detected
> net_mlx5: mlx5.c:2178: mlx5_dev_spawn(): naming Ethernet device
> "0000:5e:00.1"
> net_mlx5: mlx5.c:548: mlx5_alloc_shared_ibctx(): DevX is NOT supported
> net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree
> 0x10037b420 with table 0x100377f40
> net_mlx5: mlx5.c:2251: mlx5_dev_spawn(): MPW isn't supported
> net_mlx5: mlx5.c:2257: mlx5_dev_spawn(): SWP support: 7
> net_mlx5: mlx5.c:2266: mlx5_dev_spawn():
> 	min_single_stride_log_num_of_bytes: 0
> net_mlx5: mlx5.c:2268: mlx5_dev_spawn():
> 	max_single_stride_log_num_of_bytes: 0
> net_mlx5: mlx5.c:2270: mlx5_dev_spawn():
> 	min_single_wqe_log_num_of_strides: 0
> net_mlx5: mlx5.c:2272: mlx5_dev_spawn():
> 	max_single_wqe_log_num_of_strides: 0
> net_mlx5: mlx5.c:2274: mlx5_dev_spawn(): 	supported_qpts: 0
> net_mlx5: mlx5.c:2275: mlx5_dev_spawn(): device supports Multi-Packet RQ
> net_mlx5: mlx5.c:2310: mlx5_dev_spawn(): tunnel offloading is supported
> net_mlx5: mlx5.c:2322: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel
> offloading is not supported
> net_mlx5: mlx5.c:2473: mlx5_dev_spawn(): checksum offloading is
> supported
> net_mlx5: mlx5.c:2493: mlx5_dev_spawn(): maximum Rx indirection table
> size is 512
> net_mlx5: mlx5.c:2497: mlx5_dev_spawn(): VLAN stripping is supported
> net_mlx5: mlx5.c:2501: mlx5_dev_spawn(): FCS stripping configuration is
> supported
> net_mlx5: mlx5.c:2531: mlx5_dev_spawn(): MPS is disabled
> net_mlx5: mlx5.c:2656: mlx5_dev_spawn(): port 1 MAC address is
> 24:8a:07:5b:14:15
> net_mlx5: mlx5.c:2663: mlx5_dev_spawn(): port 1 ifname is "enp94s0f1"
> net_mlx5: mlx5.c:2676: mlx5_dev_spawn(): port 1 MTU is 9216
> net_mlx5: mlx5.c:2703: mlx5_dev_spawn(): port 1 forcing Ethernet interface
> up
> net_mlx5: mlx5.c:1836: mlx5_set_min_inline(): min tx inline configured: 18
> net_mlx5: mlx5_utils.c:41: mlx5_hlist_create(): Hash list with
> mlx5_1_flow_table size 0x1000 is created.
> 
> net_mlx5: mlx5_utils.c:41: mlx5_hlist_create(): Hash list with mlx5_1_tags
> size 0x2000 is created.
> 
> net_mlx5: mlx5_flow.c:550: mlx5_flow_discover_priorities(): port 1 flow
> maximum priority: 3
> net_mlx5: mlx5.c:1887: mlx5_set_metadata_mask(): metadata mode 0
> net_mlx5: mlx5.c:1888: mlx5_set_metadata_mask(): metadata MARK mask
> 00FFFFFF
> net_mlx5: mlx5.c:1889: mlx5_set_metadata_mask(): metadata META mask
> FFFFFFFF
> net_mlx5: mlx5.c:1890: mlx5_set_metadata_mask(): metadata reg_c0 mask
> FFFFFFFF
> net_mlx5: mlx5.c:2771: mlx5_dev_spawn(): port 1 extensive metadata
> register is not supported Auto-start selected Set rxonly packet forwarding
> mode
> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456,
> size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0
> (socket 0)
> net_mlx5: mlx5_ethdev.c:424: mlx5_dev_configure(): port 0 Tx queues
> number update: 0 -> 1
> net_mlx5: mlx5_ethdev.c:435: mlx5_dev_configure(): port 0 Rx queues
> number update: 0 -> 1
> net_mlx5: mlx5_txq.c:172: mlx5_tx_queue_pre_setup(): port 0 configuring
> queue 0 for 256 descriptors
> net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree
> 0x101051c28 with table 0x1010505c0
> net_mlx5: mlx5_txq.c:225: mlx5_tx_queue_setup(): port 0 adding Tx queue
> 0 to list
> net_mlx5: mlx5_rxq.c:468: mlx5_rx_queue_pre_setup(): port 0 configuring
> Rx queue 0 for 256 descriptors
> net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree
> 0x10104fa6c with table 0x10104e500
> net_mlx5: mlx5_rxq.c:1921: mlx5_rxq_new(): port 0 maximum number of
> segments per packet: 1
> net_mlx5: mlx5_rxq.c:1759: mlx5_max_lro_msg_size_adjust(): port 0 Rx
> Queue 0 max LRO message size adjusted to 1280 bytes
> net_mlx5: mlx5_rxq.c:1968: mlx5_rxq_new(): port 0 CRC stripping is enabled,
> 0 bytes will be subtracted from incoming frames to hide it
> net_mlx5: mlx5_rxq.c:525: mlx5_rx_queue_setup(): port 0 adding Rx queue
> 0 to list
> net_mlx5: mlx5_trigger.c:276: mlx5_dev_start(): port 0 starting device
> net_mlx5: mlx5_ethdev.c:493: mlx5_dev_configure_rss_reta(): port 0 Rx
> queues number update: 1 -> 1
> net_mlx5: mlx5_txq.c:55: txq_alloc_elts(): port 0 Tx queue 0 allocated and
> configured 256 WRs
> net_mlx5: mlx5_txq.c:771: mlx5_txq_obj_new(): port 0: uar_mmap_offset
> 0x306000
> net_mlx5: mlx5_trigger.c:145: mlx5_rxq_start(): port 0 Rx queue 0 registering
> mp mbuf_pool_socket_0 having 1 chunks
> net_mlx5: mlx5_mr.c:600: mlx5_mr_create_primary(): port 0 creating a MR
> using address (0x10109b4c0)
> net_mlx5: mlx5_mr.c:649: mlx5_mr_create_primary(): port 0 extending
> 0x10109b4c0 to [0x100200000, 0x118200000), page_sz=0x200000, ms_n=192
> net_mlx5: mlx5_mr.c:786: mlx5_mr_create_primary(): port 0 MR CREATED
> (0x10104e140) for 0x10109b4c0:
>   [0x100200000, 0x118200000), lkey=0x104b0100 base_idx=0 ms_n=192,
> ms_bmp_n=192
> net_mlx5: mlx5_mr.c:345: mr_insert_dev_cache(): device mlx5_0 inserting
> MR(0x10104e140) to global cache
> net_mlx5: mlx5_mr.c:173: mr_btree_insert(): inserted B-
> tree(0x1003dcb20)[1], [0x100200000, 0x118200000) lkey=0x104b0100
> net_mlx5: mlx5_mr.c:173: mr_btree_insert(): inserted B-
> tree(0x10104fa6c)[1], [0x100200000, 0x118200000) lkey=0x104b0100
> net_mlx5: mlx5_rxq.c:257: rxq_alloc_elts_sprq(): port 0 Rx queue 0 allocated
> and configured 256 segments (max 256 packets)
> net_mlx5: mlx5_rxq.c:1402: mlx5_rxq_obj_new(): port 0
> device_attr.max_qp_wr is 32768
> net_mlx5: mlx5_rxq.c:1404: mlx5_rxq_obj_new(): port 0
> device_attr.max_sge is 30
> net_mlx5: mlx5_rxq.c:1478: mlx5_rxq_obj_new(): port 0 rxq 0 updated with
> 0x7ffda24f21f8
> net_mlx5: mlx5_trigger.c:322: mlx5_dev_start(): port 0 failed to set defaults
> flows
> net_mlx5: mlx5_rxq.c:333: rxq_free_elts_sprq(): port 0 Rx queue 0 freeing
> WRs Fail to start port 0 Configuring Port 1 (socket 0)
> net_mlx5: mlx5_ethdev.c:424: mlx5_dev_configure(): port 1 Tx queues
> number update: 0 -> 1
> net_mlx5: mlx5_ethdev.c:435: mlx5_dev_configure(): port 1 Rx queues
> number update: 0 -> 1
> net_mlx5: mlx5_txq.c:172: mlx5_tx_queue_pre_setup(): port 1 configuring
> queue 0 for 256 descriptors
> net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree
> 0x10104d6a8 with table 0x10104c040
> net_mlx5: mlx5_txq.c:225: mlx5_tx_queue_setup(): port 1 adding Tx queue
> 0 to list
> net_mlx5: mlx5_rxq.c:468: mlx5_rx_queue_pre_setup(): port 1 configuring
> Rx queue 0 for 256 descriptors
> net_mlx5: mlx5_mr.c:214: mlx5_mr_btree_init(): initialized B-tree
> 0x10104b4ec with table 0x101049f80
> net_mlx5: mlx5_rxq.c:1921: mlx5_rxq_new(): port 1 maximum number of
> segments per packet: 1
> net_mlx5: mlx5_rxq.c:1759: mlx5_max_lro_msg_size_adjust(): port 1 Rx
> Queue 0 max LRO message size adjusted to 1280 bytes
> net_mlx5: mlx5_rxq.c:1968: mlx5_rxq_new(): port 1 CRC stripping is enabled,
> 0 bytes will be subtracted from incoming frames to hide it
> net_mlx5: mlx5_rxq.c:525: mlx5_rx_queue_setup(): port 1 adding Rx queue
> 0 to list
> net_mlx5: mlx5_trigger.c:276: mlx5_dev_start(): port 1 starting device
> net_mlx5: mlx5_ethdev.c:493: mlx5_dev_configure_rss_reta(): port 1 Rx
> queues number update: 1 -> 1
> net_mlx5: mlx5_txq.c:55: txq_alloc_elts(): port 1 Tx queue 0 allocated and
> configured 256 WRs
> net_mlx5: mlx5_txq.c:771: mlx5_txq_obj_new(): port 1: uar_mmap_offset
> 0x306000
> net_mlx5: mlx5_trigger.c:145: mlx5_rxq_start(): port 1 Rx queue 0 registering
> mp mbuf_pool_socket_0 having 1 chunks
> net_mlx5: mlx5_mr.c:600: mlx5_mr_create_primary(): port 1 creating a MR
> using address (0x10109b4c0)
> net_mlx5: mlx5_mr.c:649: mlx5_mr_create_primary(): port 1 extending
> 0x10109b4c0 to [0x100200000, 0x118200000), page_sz=0x200000, ms_n=192
> net_mlx5: mlx5_mr.c:786: mlx5_mr_create_primary(): port 1 MR CREATED
> (0x101045dc0) for 0x10109b4c0:
>   [0x100200000, 0x118200000), lkey=0xcac80f00 base_idx=0 ms_n=192,
> ms_bmp_n=192
> net_mlx5: mlx5_mr.c:345: mr_insert_dev_cache(): device mlx5_1 inserting
> MR(0x101045dc0) to global cache
> net_mlx5: mlx5_mr.c:173: mr_btree_insert(): inserted B-
> tree(0x10037b420)[1], [0x100200000, 0x118200000) lkey=0xcac80f00
> net_mlx5: mlx5_mr.c:173: mr_btree_insert(): inserted B-
> tree(0x10104b4ec)[1], [0x100200000, 0x118200000) lkey=0xcac80f00
> net_mlx5: mlx5_rxq.c:257: rxq_alloc_elts_sprq(): port 1 Rx queue 0 allocated
> and configured 256 segments (max 256 packets)
> net_mlx5: mlx5_rxq.c:1402: mlx5_rxq_obj_new(): port 1
> device_attr.max_qp_wr is 32768
> net_mlx5: mlx5_rxq.c:1404: mlx5_rxq_obj_new(): port 1
> device_attr.max_sge is 30
> net_mlx5: mlx5_rxq.c:1478: mlx5_rxq_obj_new(): port 1 rxq 0 updated with
> 0x7ffda24f21f8
> net_mlx5: mlx5_trigger.c:322: mlx5_dev_start(): port 1 failed to set defaults
> flows
> net_mlx5: mlx5_rxq.c:333: rxq_free_elts_sprq(): port 1 Rx queue 0 freeing
> WRs Fail to start port 1 Please stop the ports first Done No commandline core
> given, start packet forwarding Not all ports were started Press enter to exit
> 
> Stopping port 0...
> Stopping ports...
> Done
> 
> Stopping port 1...
> Stopping ports...
> Done
> 
> Shutting down port 0...
> Closing ports...
> net_mlx5: mlx5.c:1233: mlx5_dev_close(): port 0 closing device "mlx5_0"
> net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree
> 0x10104fa6c with table 0x10104e500
> net_mlx5: mlx5_txq.c:77: txq_free_elts(): port 0 Tx queue 0 freeing WRs
> net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree
> 0x101051c28 with table 0x1010505c0
> net_mlx5: mlx5_mr.c:1569: mlx5_mr_dump_dev(): device mlx5_0 MR[0],
> LKey = 0x104b0100, ms_n = 192, ms_bmp_n = 192
> net_mlx5: mlx5_mr.c:1579: mlx5_mr_dump_dev():   chunk[0], [0x100200000,
> 0x118200000)
> net_mlx5: mlx5_mr.c:1582: mlx5_mr_dump_dev(): device mlx5_0 dumping
> global cache
> net_mlx5: mlx5_mr.c:256: mlx5_mr_btree_dump(): B-tree(0x1003dcb20)[0],
> [0x0, 0x0) lkey=0xffffffff
> net_mlx5: mlx5_mr.c:256: mlx5_mr_btree_dump(): B-tree(0x1003dcb20)[1],
> [0x100200000, 0x118200000) lkey=0x104b0100
> net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree
> 0x1003dcb20 with table 0x1003d9640
> net_mlx5: mlx5_mr.c:459: mr_free(): freeing MR(0x10104e140):
> Done
> 
> Shutting down port 1...
> Closing ports...
> net_mlx5: mlx5.c:1233: mlx5_dev_close(): port 1 closing device "mlx5_1"
> net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree
> 0x10104b4ec with table 0x101049f80
> net_mlx5: mlx5_txq.c:77: txq_free_elts(): port 1 Tx queue 0 freeing WRs
> net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree
> 0x10104d6a8 with table 0x10104c040
> net_mlx5: mlx5_mr.c:1569: mlx5_mr_dump_dev(): device mlx5_1 MR[0],
> LKey = 0xcac80f00, ms_n = 192, ms_bmp_n = 192
> net_mlx5: mlx5_mr.c:1579: mlx5_mr_dump_dev():   chunk[0], [0x100200000,
> 0x118200000)
> net_mlx5: mlx5_mr.c:1582: mlx5_mr_dump_dev(): device mlx5_1 dumping
> global cache
> net_mlx5: mlx5_mr.c:256: mlx5_mr_btree_dump(): B-tree(0x10037b420)[0],
> [0x0, 0x0) lkey=0xffffffff
> net_mlx5: mlx5_mr.c:256: mlx5_mr_btree_dump(): B-tree(0x10037b420)[1],
> [0x100200000, 0x118200000) lkey=0xcac80f00
> net_mlx5: mlx5_mr.c:230: mlx5_mr_btree_free(): freeing B-tree
> 0x10037b420 with table 0x100377f40
> net_mlx5: mlx5_mr.c:459: mr_free(): freeing MR(0x101045dc0):
> Done
> 
> Bye...
> 
> Best
> ben
> 
> > From: Benoit Ganne (bganne) <bganne@cisco.com>
> > Sent: Thursday, April 2, 2020 6:50:22 PM
> > To: users@dpdk.org <users@dpdk.org>
> > Cc: Matan Azrad <matan@mellanox.com>; Shahaf Shuler
> > <shahafs@mellanox.com>; Slava Ovsiienko <viacheslavo@mellanox.com>
> > Subject: RE: mlx5 pmd + rdma-core 28 init failure
> >
> > Adding MLX5 PMD maintainers.
> >
> > I also checked with latest rdma-core master and latest DPDK master and
> > it is failing with the same issue.
> > Any recommendation?
> >
> > Best
> > ben
> >
> > > -----Original Message-----
> > > From: Benoit Ganne (bganne)
> > > Sent: mercredi 1 avril 2020 18:52
> > > To: users@dpdk.org
> > > Subject: mlx5 pmd + rdma-core 28 init failure
> > >
> > > Hi all,
> > >
> > > I have troubles making DPDK v20.02 MLX5 PMD working with rdma-core
> > v28.0:
> > > it looks like the flow initialization done while initializing the
> > > device fails in rdma-core
> > > providers/mlx5/dr_table.c:mlx5dv_dr_table_create()
> > > because of unsupported parameters.
> > > The issue is the following test in rdma-core
> > > providers/mlx5/dr_table.c:mlx5dv_dr_table_create():
> > >     if (level && !dmn->info.supp_sw_steering) {
> > >         errno = EOPNOTSUPP;
> > >         goto dec_ref;
> > >     }
> > > Where level == 65534 and dmn->info.supp_sw_steering == false, hence
> > > the test is true and the function returns EOPNOTSUPP.
> > >
> > > Is this expected? It does work fine with ibv instead of dv API. Any
> > > help appreciated.
> > >
> > > Here is what I do:
> > >    1) checkout & compile rdma-core v28.0 ~# git clone
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> > ub.c
> > om%2Flinux-rdma%2Frdma-
> >
> core&amp;data=02%7C01%7Cmatan%40mellanox.com%7C7984449b13af4519
> d70608d
> > 7d71
> >
> d9000%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637214394282
> 618427&
> > amp;
> >
> sdata=DSLvWg3cWwY8prSFOm8BAFf1I2bbKnqPOKBPEiy4b44%3D&amp;rese
> rved=0
> >
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.
> > com%2Flinux-rdma%2Frdma-
> >
> core&amp;data=02%7C01%7Cmatan%40mellanox.com%7C7984449b13af4519
> d70608d
> > 7d71
> >
> d9000%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637214394282
> 618427&
> > amp;
> >
> sdata=DSLvWg3cWwY8prSFOm8BAFf1I2bbKnqPOKBPEiy4b44%3D&amp;rese
> rved=0>
> > > ~# cd rdma-core
> > > ~# git checkout v28.0
> > > ~# mkdir build
> > > ~# cd build
> > > ~# CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja ..
> > > ~# ninja
> > >
> > >    2) checkout & compile dpdk v20.02 ~# git clone
> > > git://dpdk.org/dpdk ~# cd dpdk ~# make config
> > > T=x86_64-native-linuxapp-gcc ~# sed -ri 's,(MLX5_PMD=).*,\1y,'
> > > build/.config ~# sed -ri 's,(IBVERBS_LINK_STATIC_PMD=).*,\1y,'
> > > build/.config ~# make
> > > EXTRA_CFLAGS=-I/home/bganne/src/rdma-core/build/include
> > > EXTRA_LDFLAGS=-L/home/bganne/src/rdma-core/build/lib
> > > PKG_CONFIG_PATH=/home/bganne/src/rdma-core/build/lib/pkgconfig
> > > ~# sudo gdb --args ./build/app/testpmd -w 0000:5e:00.0 -w
> > > 0000:5e:00.1 -
> > l
> > > 4,11,35 -- -a --forward-mode=rxonly -i
> > >
> > > The backtrace looks like this:
> > > #0  mlx5dv_dr_table_create (dmn=0x555556c641b0, level=65534) at
> > > ../providers/mlx5/dr_table.c:183
> > > #1  0x0000555555dfaeaa in flow_dv_tbl_resource_get (dev=<optimized
> > > out>, table_id=65534, egress=<optimized out>, transfer=<optimized
> > > out>,
> > > error=0x7fffffffdca0) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:6746
> > > #2  0x0000555555e02b28 in __flow_dv_translate
> > > (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>,
> > > dev_flow=0x100388300, attr=<optimized out>, items=<optimized out>,
> > > actions=<optimized out>, error=<optimized out>) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:7503
> > > #3  0x0000555555e04954 in flow_dv_translate (dev=0x555556bbcdc0
> > > <rte_eth_devices>, dev_flow=<optimized out>, attr=<optimized out>,
> > > items=<optimized out>, actions=<optimized out>, error=<optimized
> > > out>)
> > at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:8841
> > > #4  0x0000555555df152f in flow_drv_translate (error=0x7fffffffdca0,
> > > actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> > > dev_flow=<optimized out>, dev=0x555556bbcdc0 <rte_eth_devices>) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:2571
> > > #5  flow_create_split_inner (error=0x7fffffffdca0, external=false,
> > > actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88,
> > > prefix_layers=0, sub_flow=0x0, flow=0x1003885c0, dev=0x555556bbcdc0
> > > <rte_eth_devices>) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3490
> > > #6  flow_create_split_metadata (error=0x7fffffffdca0,
> > > external=false, actions=0x7fffffffdce0, items=0x7fffffffdcc0,
> > > attr=0x7fffffffbb88, prefix_layers=0, flow=0x1003885c0,
> > > dev=0x555556bbcdc0 <rte_eth_devices>) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3865
> > > #7  flow_create_split_meter (error=0x7fffffffdca0, external=false,
> > > actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94,
> > > flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4121
> > > #8  flow_create_split_outer (error=0x7fffffffdca0, external=false,
> > > actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94,
> > > flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4178
> > > #9  flow_list_create (dev=dev@entry=0x555556bbcdc0
> > > <rte_eth_devices>, list=list@entry=0x0,
> > > attr=attr@entry=0x7fffffffdc94, items=items@entry=0x7fffffffdcc0,
> > > actions=actions@entry=0x7fffffffdce0,
> > > external=external@entry=false, error=0x7fffffffdca0) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4306
> > > #10 0x0000555555df8587 in mlx5_flow_discover_mreg_c
> > > (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:5747
> > > #11 0x0000555555d692a6 in mlx5_dev_spawn (config=...,
> > > spawn=0x1003e9e00,
> > > dpdk_dev=0x555556dd6fe0) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:2763
> > > #12 mlx5_pci_probe (pci_drv=<optimized out>, pci_dev=<optimized
> > > out>) at
> > > /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:3363
> > > #13 0x0000555555a411c8 in pci_probe_all_drivers ()
> > > #14 0x0000555555a412f8 in rte_pci_probe ()
> > > #15 0x0000555555a083da in rte_bus_probe ()
> > > #16 0x00005555559f204d in rte_eal_init ()
> > > #17 0x00005555556a0d45 in main ()
> > >
> > > Best
> > > ben

Re: [dpdk-users] mlx5 pmd + rdma-core 28 init failure

[Benoit Ganne (bganne)]

Hi Matan,

> Please let us know if working with devargs dv_flow_en=0 solves the
> issue....
> ./build/app/testpmd --log-level=8 --log-level=pmd.common.mlx5:8 - -log-
> level=pmd.net.mlx5:8 -w 0000:5e:00.0.dv_flow_en=0 -w
> 0000:5e:00.1,dv_flow_en=0 -l 4,11,35 -- -a -- forward-mode=rxonly

Yes it does.
However it seems to let the device in an unstable state: it doesn't stop properly, and I need to restart the whole server for the interface to come back (reloading all mlx drivers modules and/or PCIe resetting is not enough). Here is what happens when I try to start testpmd again after the 1st try:
~# sudo ./build/app/testpmd --log-level=8 --log-level=pmd.common.mlx5:8 --log-level=pmd.net.mlx5:8 -w 0000:5e:00.0,dv_flow_en=0 -w 0000:5e:00.1,dv_flow_en=0 -l 4,11,35 -- -a --forward-mode=rxonly
EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:5e:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1013 net_mlx5
net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_1"
net_mlx5: mlx5.c:3084: mlx5_pci_probe(): no Verbs device matches PCI device 0000:5e:00.0, are kernel drivers loaded?
EAL: Requested device 0000:5e:00.0 cannot be used
EAL: PCI device 0000:5e:00.1 on NUMA socket 0
EAL:   probe driver: 15b3:1013 net_mlx5
net_mlx5: mlx5.c:3040: mlx5_pci_probe(): checking device "mlx5_1"
net_mlx5: mlx5.c:3074: mlx5_pci_probe(): PCI information matches for device "mlx5_1"
net_mlx5: mlx5.c:3305: mlx5_pci_probe(): no E-Switch support detected
net_mlx5: mlx5.c:2178: mlx5_dev_spawn(): naming Ethernet device "0000:5e:00.1"
net_mlx5: mlx5.c:548: mlx5_alloc_shared_ibctx(): DevX is NOT supported
PANIC in mlx5_mr_btree_init():
line 197        assert "!bt->table && !bt->size" failed
14: [./build/app/testpmd(_start+0x2a) [0x560a46a469aa]]
13: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f082ac3ab97]]
12: [./build/app/testpmd(main+0x96) [0x560a46a50da9]]
11: [./build/app/testpmd(rte_eal_init+0x99d) [0x560a46d6f305]]
10: [./build/app/testpmd(rte_bus_probe+0x4c) [0x560a46d898ae]]
9: [./build/app/testpmd(rte_pci_probe+0x96) [0x560a46dcdc0e]]
8: [./build/app/testpmd(+0x4c4b42) [0x560a46dcdb42]]
7: [./build/app/testpmd(+0x4c4952) [0x560a46dcd952]]
6: [./build/app/testpmd(+0xb33efe) [0x560a4743cefe]]
5: [./build/app/testpmd(+0xb2ff4e) [0x560a47438f4e]]
4: [./build/app/testpmd(+0xb2bd88) [0x560a47434d88]]
3: [./build/app/testpmd(mlx5_mr_btree_init+0x90) [0x560a47b7de13]]
2: [./build/app/testpmd(__rte_panic+0xed) [0x560a46d7e046]]
1: [./build/app/testpmd(rte_dump_stack+0x39) [0x560a46d7dea7]]

> Please let us know also your FW version...

~# sudo mstflint -d 5e:00.0 q
Image type:            FS3
FW Version:            12.26.4012
FW Release Date:       10.12.2019
Product Version:       6.4012
Rom Info:              type=UEFI version=14.19.17 cpu=AMD64,AARCH64
                       type=PXE version=3.5.805 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             248a0703005b1414        4
Base MAC:              0000248a075b1414        4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_2150110033
Security Attributes:   N/A

Best,
ben