[dpdk-users] mlx5 pmd + rdma-core 28 init failure

["Benoit Ganne (bganne)" <bganne@cisco.com>]

Hi all,

I have troubles making DPDK v20.02 MLX5 PMD working with rdma-core v28.0: it looks like the flow initialization done while initializing the device fails in rdma-core providers/mlx5/dr_table.c:mlx5dv_dr_table_create() because of unsupported parameters.
The issue is the following test in rdma-core providers/mlx5/dr_table.c:mlx5dv_dr_table_create():
    if (level && !dmn->info.supp_sw_steering) {
        errno = EOPNOTSUPP;
        goto dec_ref;
    }
Where level == 65534 and dmn->info.supp_sw_steering == false, hence the test is true and the function returns EOPNOTSUPP.

Is this expected? It does work fine with ibv instead of dv API. Any help appreciated.

Here is what I do:
   1) checkout & compile rdma-core v28.0
~# git clone https://github.com/linux-rdma/rdma-core
~# cd rdma-core
~# git checkout v28.0
~# mkdir build
~# cd build
~# CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja ..
~# ninja

   2) checkout & compile dpdk v20.02
~# git clone git://dpdk.org/dpdk
~# cd dpdk
~# make config T=x86_64-native-linuxapp-gcc
~# sed -ri 's,(MLX5_PMD=).*,\1y,' build/.config
~# sed -ri 's,(IBVERBS_LINK_STATIC_PMD=).*,\1y,' build/.config
~# make EXTRA_CFLAGS=-I/home/bganne/src/rdma-core/build/include EXTRA_LDFLAGS=-L/home/bganne/src/rdma-core/build/lib PKG_CONFIG_PATH=/home/bganne/src/rdma-core/build/lib/pkgconfig
~# sudo gdb --args ./build/app/testpmd -w 0000:5e:00.0 -w 0000:5e:00.1 -l 4,11,35 -- -a --forward-mode=rxonly -i

The backtrace looks like this:
#0  mlx5dv_dr_table_create (dmn=0x555556c641b0, level=65534) at ../providers/mlx5/dr_table.c:183
#1  0x0000555555dfaeaa in flow_dv_tbl_resource_get (dev=<optimized out>, table_id=65534, egress=<optimized out>, transfer=<optimized out>, error=0x7fffffffdca0) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:6746
#2  0x0000555555e02b28 in __flow_dv_translate (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>, dev_flow=0x100388300, attr=<optimized out>, items=<optimized out>, actions=<optimized out>, error=<optimized out>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:7503
#3  0x0000555555e04954 in flow_dv_translate (dev=0x555556bbcdc0 <rte_eth_devices>, dev_flow=<optimized out>, attr=<optimized out>, items=<optimized out>, actions=<optimized out>, error=<optimized out>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow_dv.c:8841
#4  0x0000555555df152f in flow_drv_translate (error=0x7fffffffdca0, actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88, dev_flow=<optimized out>, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:2571
#5  flow_create_split_inner (error=0x7fffffffdca0, external=false, actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88, prefix_layers=0, sub_flow=0x0, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3490
#6  flow_create_split_metadata (error=0x7fffffffdca0, external=false, actions=0x7fffffffdce0, items=0x7fffffffdcc0, attr=0x7fffffffbb88, prefix_layers=0, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:3865
#7  flow_create_split_meter (error=0x7fffffffdca0, external=false, actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4121
#8  flow_create_split_outer (error=0x7fffffffdca0, external=false, actions=0x7fffffffdce0, items=<optimized out>, attr=0x7fffffffdc94, flow=0x1003885c0, dev=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4178
#9  flow_list_create (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>, list=list@entry=0x0, attr=attr@entry=0x7fffffffdc94, items=items@entry=0x7fffffffdcc0, actions=actions@entry=0x7fffffffdce0, external=external@entry=false, error=0x7fffffffdca0) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:4306
#10 0x0000555555df8587 in mlx5_flow_discover_mreg_c (dev=dev@entry=0x555556bbcdc0 <rte_eth_devices>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5_flow.c:5747
#11 0x0000555555d692a6 in mlx5_dev_spawn (config=..., spawn=0x1003e9e00, dpdk_dev=0x555556dd6fe0) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:2763
#12 mlx5_pci_probe (pci_drv=<optimized out>, pci_dev=<optimized out>) at /home/bganne/src/dpdk/drivers/net/mlx5/mlx5.c:3363
#13 0x0000555555a411c8 in pci_probe_all_drivers ()
#14 0x0000555555a412f8 in rte_pci_probe ()
#15 0x0000555555a083da in rte_bus_probe ()
#16 0x00005555559f204d in rte_eal_init ()
#17 0x00005555556a0d45 in main ()

Best
ben