[dpdk-users] Issue while running dpdk-20.02 test-pmd with Huawei hinic Nic (IN200 Hi1822)

[dpdk-users] Issue while running dpdk-20.02 test-pmd with Huawei hinic Nic (IN200 Hi1822)

[梁 明超]

Hi Everyone,
I am trying to run testpmd with hinic nic card on arm64 but getting following error. Can anyone plesase help me to solve this issue.

1. NetCard Info
ethtool -i enp4s0
driver: hinic
version: 2.6.0.9
firmware-version: 2.4.1.0
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no



./usertools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:05:00.0 'Hi1822 Family (4*25GE) 1822' drv=igb_uio unused=hinic
0000:06:00.0 'Hi1822 Family (4*25GE) 1822' drv=igb_uio unused=hinic
0000:07:00.0 'Hi1822 Family (4*25GE) 1822' drv=igb_uio unused=hinic

Network devices using kernel driver
===================================
0000:04:00.0 'Hi1822 Family (4*25GE) 1822' if=enp4s0 drv=hinic unused=igb_uio
0000:7d:00.0 'HNS GE/10GE/25GE RDMA Network Controller a222' if=enp125s0f0 drv=hns3 unused=hns_roce_hw_v2,igb_uio
0000:7d:00.1 'HNS GE/10GE/25GE Network Controller a221' if=enp125s0f1 drv=hns3 unused=igb_uio *Active*
0000:7d:00.2 'HNS GE/10GE/25GE RDMA Network Controller a222' if=enp125s0f2 drv=hns3 unused=hns_roce_hw_v2,igb_uio
0000:7d:00.3 'HNS GE/10GE/25GE Network Controller a221' if=enp125s0f3 drv=hns3 unused=igb_uio


modinfo hinic
filename:       /lib/modules/4.14.0-115.5.1.el7a.06.aarch64/extra/hinic/hinic.ko
license:        GPL
version:        2.6.0.9
description:    Huawei(R) Intelligent Network Interface Card Driver
author:         Huawei Technologies CO., Ltd
rhelversion:    7.6
srcversion:     28E265C6E29E0725CD231AA
alias:          pci:v000019E5d00000206sv*sd*bc*sc*i*
alias:          pci:v000019E5d00000200sv*sd*bc*sc*i*
alias:          pci:v000019E5d00000211sv*sd*bc*sc*i*
alias:          pci:v000019E5d00000210sv*sd*bc*sc*i*
alias:          pci:v000019E5d00000205sv*sd*bc*sc*i*
alias:          pci:v000019E5d0000020Dsv*sd*bc*sc*i*
alias:          pci:v000019E5d00000204sv*sd*bc*sc*i*
alias:          pci:v000019E5d00000208sv*sd*bc*sc*i*
alias:          pci:v000019E5d0000020Bsv*sd*bc*sc*i*
alias:          pci:v000019E5d0000379Esv*sd*bc*sc*i*
alias:          pci:v000019E5d0000375Esv*sd*bc*sc*i*
alias:          pci:v000019E5d00001822sv*sd*bc*sc*i*
depends:
name:           hinic
vermagic:       4.14.0-115.5.1.el7a.06.aarch64 SMP mod_unload modversions aarch64
parm:           g_rdma_mtts_num:number of roce used mtts, use default value when pass 0 (uint)
parm:           g_rdma_qps_num:number of roce used qps, use default value when pass 0 (uint)
parm:           g_rdma_mpts_num:number of roce used mpts, use default value when pass 0 (uint)
parm:           g_vfs_num:number of used vfs, use default value when pass 0  (uint)
parm:           g_aeq_len:aeq depth, valid range is 64 - (512 * 1024) (uint)
parm:           g_ceq_len:ceq depth, valid range is 64 - (1024 * 1024) (uint)
parm:           g_num_ceqe_in_tasklet:The max number of ceqe can be processed in tasklet, default = 1024 (uint)
parm:           tx_pending_limit:TX CI coalescing parameter pending_limit (default=0) (byte)
parm:           tx_coalescing_time:TX CI coalescing parameter coalescing_time (default=0) (byte)
parm:           set_vf_link_state:Set vf link state, 0 represents link auto, 1 represents link always up, 2 represents link always down. - default is 0. (byte)
parm:           g_mtt_page_size:0:4K, 1:8K, 2:16K, default:4K (uint)
parm:           disable_vf_load:Disable virtual functions probe or not - default is false (bool)
parm:           qp_coalesc_use_drv_params_switch:QP MSI-X Interrupt coalescing parameter switch (default=0,not use drv parameter) (byte)
parm:           qp_pending_limit:QP MSI-X Interrupt coalescing parameter pending_limit (default=2) (byte)
parm:           qp_coalesc_timer_cfg:QP MSI-X Interrupt coalescing parameter coalesc_timer_cfg (default=32) (byte)
parm:           set_max_wqe_num:Set lro max wqe number, valid range is 1 - 32, default is 4(arm) / 8(x86) (byte)
parm:           set_lro_timer:Set lro timer in micro second, valid range is 1 - 1024, default is 16 (uint)
parm:           lro_en_status:lro enable status. 0 - disable, 1 - enable, other - unset. (default unset) (byte)
parm:           qp_pending_limit_low:MSI-X adaptive low coalesce pending limit, range is 0 - 255 (byte)
parm:           qp_coalesc_timer_low:MSI-X adaptive low coalesce time, range is 0 - 255 (byte)
parm:           qp_pending_limit_high:MSI-X adaptive high coalesce pending limit, range is 0 - 255 (byte)
parm:           qp_coalesc_timer_high:MSI-X adaptive high coalesce time, range is 0 - 255 (byte)
parm:           l2nic_interrupt_switch:Control whether execute l2nic io interrupt switch or not, default is true (bool)
parm:           num_qps:Number of Queue Pairs (default unset) (ushort)
parm:           ovs_num_qps:Number of Queue Pairs in ovs mode (default=16) (ushort)
parm:           poll_weight:Number packets for NAPI budget (default=64) (uint)
parm:           rx_buff:Set rx_buff size, buffer len must be 2^n. 2 - 16, default is 2KB (ushort)
parm:           set_link_status_follow:Set link status follow port status. 0 - default, 1 - follow, 2 - separate, other - unset. (default unset) (byte)
parm:           lro_replenish_thld:Number wqe for lro replenish buffer (default=256) (uint

2. Run testpmd

[root@localhost test-pmd]# ./testpmd -l 0-3 -- -i
EAL: Detected 96 lcore(s)
EAL: Detected 4 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No free hugepages reported in hugepages-2048kB
EAL: No free hugepages reported in hugepages-2048kB
EAL: No free hugepages reported in hugepages-2048kB
EAL: No free hugepages reported in hugepages-2048kB
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 19e5:1822 net_hinic
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 19e5:1822 net_hinic
net_hinic: Initializing pf hinic-0000:05:00.0 in primary process
net_hinic: Device 0000:05:00.0 hwif attribute:
net_hinic: func_idx:1, p2p_idx:1, pciintf_idx:0, vf_in_pf:0, ppf_idx:0, global_vf_id:75, func_type:0
net_hinic: num_aeqs:4, num_ceqs:4, num_irqs:32, dma_attr:2
net_hinic: API CMD poll status timeout
net_hinic: chain type: 0x7
net_hinic: chain hw cpld error: 0x1
net_hinic: chain hw check error: 0x0
net_hinic: chain hw current fsm: 0x0
net_hinic: chain hw current ci: 0x0
net_hinic: Chain hw current pi: 0x1
net_hinic: Send msg to mgmt failed
net_hinic: Failed to get board info, err: -110, status: 0x0, out size: 0x0
net_hinic: Check card workmode failed, dev_name: 0000:05:00.0
net_hinic: Create nic device failed, dev_name: 0000:05:00.0
net_hinic: Initialize 0000:05:00.0 in primary failed
EAL: Requested device 0000:05:00.0 cannot be used
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 19e5:1822 net_hinic
net_hinic: Initializing pf hinic-0000:06:00.0 in primary process
net_hinic: Device 0000:06:00.0 hwif attribute:
net_hinic: func_idx:2, p2p_idx:2, pciintf_idx:0, vf_in_pf:0, ppf_idx:0, global_vf_id:135, func_type:0
net_hinic: num_aeqs:4, num_ceqs:4, num_irqs:32, dma_attr:2
net_hinic: API CMD poll status timeout
net_hinic: chain type: 0x7
net_hinic: chain hw cpld error: 0x1
net_hinic: chain hw check error: 0x0
net_hinic: chain hw current fsm: 0x0
net_hinic: chain hw current ci: 0x0
net_hinic: Chain hw current pi: 0x1
net_hinic: Send msg to mgmt failed
net_hinic: Failed to get board info, err: -110, status: 0x0, out size: 0x0
net_hinic: Check card workmode failed, dev_name: 0000:06:00.0
net_hinic: Create nic device failed, dev_name: 0000:06:00.0
net_hinic: Initialize 0000:06:00.0 in primary failed
EAL: Requested device 0000:06:00.0 cannot be used
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL:   probe driver: 19e5:1822 net_hinic
net_hinic: Initializing pf hinic-0000:07:00.0 in primary process
net_hinic: Device 0000:07:00.0 hwif attribute:
net_hinic: func_idx:3, p2p_idx:3, pciintf_idx:0, vf_in_pf:0, ppf_idx:0, global_vf_id:195, func_type:0
net_hinic: num_aeqs:4, num_ceqs:4, num_irqs:32, dma_attr:2
net_hinic: API CMD poll status timeout
net_hinic: chain type: 0x7
net_hinic: chain hw cpld error: 0x1
net_hinic: chain hw check error: 0x0
net_hinic: chain hw current fsm: 0x0
net_hinic: chain hw current ci: 0x0
net_hinic: Chain hw current pi: 0x1
net_hinic: Send msg to mgmt failed
net_hinic: Failed to get board info, err: -110, status: 0x0, out size: 0x0
net_hinic: Check card workmode failed, dev_name: 0000:07:00.0
net_hinic: Create nic device failed, dev_name: 0000:07:00.0
net_hinic: Initialize 0000:07:00.0 in primary failed
EAL: Requested device 0000:07:00.0 cannot be used
EAL: PCI device 0000:7d:00.0 on NUMA socket 0
EAL:   probe driver: 19e5:a222 net_hns3
EAL: PCI device 0000:7d:00.1 on NUMA socket 0
EAL:   probe driver: 19e5:a221 net_hns3
EAL: PCI device 0000:7d:00.2 on NUMA socket 0
EAL:   probe driver: 19e5:a222 net_hns3
EAL: PCI device 0000:7d:00.3 on NUMA socket 0
EAL:   probe driver: 19e5:a221 net_hns3
testpmd: No probed ethernet devices
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
testpmd

3. Run testpmd with gdb

[root@localhost test-pmd]# gdb --args ./testpmd -l 0-3 -- -i
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-neokylin-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/lmc/dpdk-20.02/app/test-pmd/testpmd...done.
(gdb) b hinic_pmd_api_cmd.c:355
Breakpoint 1 at 0x7f68b4: file /home/lmc/dpdk-20.02/drivers/net/hinic/base/hinic_pmd_api_cmd.c, line 355.
(gdb) r
Starting program: /home/lmc/dpdk-20.02/app/test-pmd/./testpmd -l 0-3 -- -i
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
EAL: Detected 96 lcore(s)
EAL: Detected 4 NUMA nodes
[New Thread 0xffffbe43d910 (LWP 88171)]
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
[New Thread 0xffffbdc2d910 (LWP 88172)]
EAL: Selected IOVA mode 'PA'
EAL: No free hugepages reported in hugepages-2048kB
EAL: No free hugepages reported in hugepages-2048kB
EAL: No free hugepages reported in hugepages-2048kB
EAL: No free hugepages reported in hugepages-2048kB
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
[New Thread 0xffffbd40d910 (LWP 88173)]
[New Thread 0xffffbcbfd910 (LWP 88174)]
[New Thread 0xffffbc3ed910 (LWP 88175)]
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 19e5:1822 net_hinic
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 19e5:1822 net_hinic
net_hinic: Initializing pf hinic-0000:05:00.0 in primary process
net_hinic: Device 0000:05:00.0 hwif attribute:
net_hinic: func_idx:1, p2p_idx:1, pciintf_idx:0, vf_in_pf:0, ppf_idx:0, global_vf_id:75, func_type:0
net_hinic: num_aeqs:4, num_ceqs:4, num_irqs:32, dma_attr:2

Breakpoint 1, wait_for_api_cmd_completion (chain=0x13ff45100, ctxt=0x13ff24500, ack=0x0, ack_size=0)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/base/hinic_pmd_api_cmd.c:355
355 PMD_DRV_LOG(ERR, "API CMD poll status timeout");
Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7.ns7.01.aarch64 libgcc-4.8.5-36.el7.ns7.01.aarch64 numactl-libs-2.0.9-7.el7.aarch64
(gdb) bt
#0  wait_for_api_cmd_completion (chain=0x13ff45100, ctxt=0x13ff24500, ack=0x0, ack_size=0)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/base/hinic_pmd_api_cmd.c:355
#1  0x00000000007f6a60 in api_cmd (chain=0x13ff45100, dest=HINIC_NODE_ID_MGMT_HOST, cmd=0x13ff3cf00, cmd_size=116, ack=0x0, ack_size=0)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/base/hinic_pmd_api_cmd.c:416
#2  0x00000000007f6a9c in hinic_api_cmd_write (chain=0x13ff45100, dest=HINIC_NODE_ID_MGMT_HOST, cmd=0x13ff3cf00, size=116)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/base/hinic_pmd_api_cmd.c:431
#3  0x0000000000800bb4 in send_msg_to_mgmt_sync (pf_to_mgmt=0x13ff3fe80, mod=HINIC_MOD_COMM, cmd=82 'R', msg=0xffffffffef50, msg_len=88,
    ack_type=HINIC_MSG_ACK, direction=HINIC_MSG_DIRECT_SEND, resp_msg_id=65535)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/base/hinic_pmd_mgmt.c:323
#4  0x0000000000800df4 in hinic_pf_to_mgmt_sync (hwdev=0x13ff97000, mod=HINIC_MOD_COMM, cmd=82 'R', buf_in=0xffffffffef50, in_size=88,
    buf_out=0xffffffffef50, out_size=0xffffffffef4e, timeout=0) at /home/lmc/dpdk-20.02/drivers/net/hinic/base/hinic_pmd_mgmt.c:407
#5  0x00000000008011ec in hinic_msg_to_mgmt_sync (hwdev=0x13ff97000, mod=HINIC_MOD_COMM, cmd=82 'R', buf_in=0xffffffffef50, in_size=88,
    buf_out=0xffffffffef50, out_size=0xffffffffef4e, timeout=0) at /home/lmc/dpdk-20.02/drivers/net/hinic/base/hinic_pmd_mgmt.c:495
#6  0x00000000007fdad4 in hinic_get_board_info (hwdev=0x13ff97000, info=0xffffffffefd0)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/base/hinic_pmd_hwdev.c:997
#7  0x0000000000813ac0 in hinic_card_workmode_check (nic_dev=0x13ff9a500) at /home/lmc/dpdk-20.02/drivers/net/hinic/hinic_pmd_ethdev.c:2559
#8  0x0000000000813fa8 in hinic_nic_dev_create (eth_dev=0x1437480 <rte_eth_devices>)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/hinic_pmd_ethdev.c:2693
#9  0x0000000000814664 in hinic_func_init (eth_dev=0x1437480 <rte_eth_devices>)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/hinic_pmd_ethdev.c:3009
#10 0x0000000000814a2c in hinic_dev_init (eth_dev=0x1437480 <rte_eth_devices>)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/hinic_pmd_ethdev.c:3105
#11 0x000000000080ec98 in rte_eth_dev_pci_generic_probe (pci_dev=0x15c8fe0, private_data_size=1320, dev_init=0x814948 <hinic_dev_init>)
    at /home/lmc/dpdk-20.02/arm64-armv8a-linux-gcc/include/rte_ethdev_pci.h:166
#12 0x0000000000814af8 in hinic_pci_probe (pci_drv=0xf347e0 <rte_hinic_pmd>, pci_dev=0x15c8fe0)
    at /home/lmc/dpdk-20.02/drivers/net/hinic/hinic_pmd_ethdev.c:3147
#13 0x0000000000663888 in rte_pci_probe_one_driver (dr=0xf347e0 <rte_hinic_pmd>, dev=0x15c8fe0)
    at /home/lmc/dpdk-20.02/drivers/bus/pci/pci_common.c:199
#14 0x0000000000663a80 in pci_probe_all_drivers (dev=0x15c8fe0) at /home/lmc/dpdk-20.02/drivers/bus/pci/pci_common.c:274
#15 0x0000000000663b4c in rte_pci_probe () at /home/lmc/dpdk-20.02/drivers/bus/pci/pci_common.c:309
#16 0x00000000006309a8 in rte_bus_probe () at /home/lmc/dpdk-20.02/lib/librte_eal/common/eal_common_bus.c:72
#17 0x0000000000615de8 in rte_eal_init (argc=5, argv=0xfffffffff578) at /home/lmc/dpdk-20.02/lib/librte_eal/linux/eal/eal.c:1259
#18 0x000000000048f210 in main (argc=5, argv=0xfffffffff578) at testpmd.c:3474



Regards

Mingchao