Re: [dpdk-dev] Status of bnx2x pmd?

["Mussar, Gary" <gmussar@ciena.com>]

> -----Original Message-----
> From: Harish Patil [mailto:harish.patil@qlogic.com]
> Sent: Monday, November 02, 2015 01:51
> To: Mussar, Gary; dev@dpdk.org
> Subject: Re: [dpdk-dev] Status of bnx2x pmd?
> 
> >
> >I have been attempting to build testpmd with the bnx2x pmd enabled to try
> >and talk through a bcm57810 10G NIC card. I have tried using SR-IOV VFs
> >and I have tried using the PFs. Neither seem to be able to be properly
> >initialized in testpmd and/or testpmd aborts.
> >
> >Is this PMD actually usable or is it trash?
> >
> >Gary
> >
> 
> Could you please explain the SR-IOV configuration that you are trying out?
> Please provide all the details.
> 
> Thanks,


I ultimately want to run dpdk in a VM with PCI pass through on SR-IOV ports. I have had no luck there so I thought I would start with DPDK on bare metal using the PFs.

I am using Ubuntu 14.04.3 with the Ubuntu 3.19.0-31 kernel and dpdk from the dpdk.org master branch (d739ba4c6abf00eeeb680ab4e9540899182ea74d).

# ./tools/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:04:00.0 'NetXtreme II BCM57810 10 Gigabit Ethernet' drv=igb_uio unused=
0000:04:00.1 'NetXtreme II BCM57810 10 Gigabit Ethernet' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:05:00.0 'I350 Gigabit Network Connection' if=eth0 drv=igb unused=igb_uio *Active*
0000:05:00.1 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices
=====================
<none>

Running testpmd:

./app/testpmd -c 7 -n 4 --socket-mem 900 -- -i --burst=64 --txqflags=0xf00 --nb-cores=2 --nb-ports=2 --portmask=0x3
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 6 on socket 0
EAL: Detected lcore 7 as core 7 on socket 0
EAL: Detected lcore 8 as core 0 on socket 0
EAL: Detected lcore 9 as core 1 on socket 0
EAL: Detected lcore 10 as core 2 on socket 0
EAL: Detected lcore 11 as core 3 on socket 0
EAL: Detected lcore 12 as core 4 on socket 0
EAL: Detected lcore 13 as core 5 on socket 0
EAL: Detected lcore 14 as core 6 on socket 0
EAL: Detected lcore 15 as core 7 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ffff4200000 (size = 0x200000)
EAL: Ask a virtual area of 0x14a00000 bytes
EAL: Virtual area found at 0x7fffdf600000 (size = 0x14a00000)
EAL: Ask a virtual area of 0x56800000 bytes
EAL: Virtual area found at 0x7fff88c00000 (size = 0x56800000)
EAL: Ask a virtual area of 0x4000000 bytes
EAL: Virtual area found at 0x7fff84a00000 (size = 0x4000000)
EAL: Ask a virtual area of 0x54200000 bytes
EAL: Virtual area found at 0x7fff30600000 (size = 0x54200000)
EAL: Ask a virtual area of 0x14c00000 bytes
EAL: Virtual area found at 0x7fff1b800000 (size = 0x14c00000)
EAL: Ask a virtual area of 0x125400000 bytes
EAL: Virtual area found at 0x7ffdf6200000 (size = 0x125400000)
EAL: Ask a virtual area of 0x1b000000 bytes
EAL: Virtual area found at 0x7ffddb000000 (size = 0x1b000000)
EAL: Ask a virtual area of 0x18600000 bytes
EAL: Virtual area found at 0x7ffdc2800000 (size = 0x18600000)
EAL: Ask a virtual area of 0xf8600000 bytes
EAL: Virtual area found at 0x7ffcca000000 (size = 0xf8600000)
EAL: Ask a virtual area of 0x39400000 bytes
EAL: Virtual area found at 0x7ffc90a00000 (size = 0x39400000)
EAL: Ask a virtual area of 0x1fc00000 bytes
EAL: Virtual area found at 0x7ffc70c00000 (size = 0x1fc00000)
EAL: Ask a virtual area of 0x74c00000 bytes
EAL: Virtual area found at 0x7ffbfbe00000 (size = 0x74c00000)
EAL: Ask a virtual area of 0x9000000 bytes
EAL: Virtual area found at 0x7ffbf2c00000 (size = 0x9000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ffbf2800000 (size = 0x200000)
EAL: Requesting 450 pages of size 2MB from socket 0
EAL: TSC frequency is ~1995380 KHz
EAL: Master lcore 0 is ready (tid=f7fe4940;cpuset=[0])
PMD: rte_bnx2x_pmd_init():  >>
PMD: rte_bnx2xvf_pmd_init():  >>
EAL: lcore 2 is ready (tid=f5ec9700;cpuset=[2])
EAL: lcore 1 is ready (tid=f66ca700;cpuset=[1])
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
EAL:   PCI memory mapped at 0x7ffff4400000
EAL:   PCI memory mapped at 0x7ffff4c00000
EAL:   PCI memory mapped at 0x7ffff5400000
PMD: eth_bnx2x_dev_init():  >>
PMD: bnx2x_common_dev_init():  >>
PMD: bnx2x_probe_pci_caps():  >>
PMD: bnx2x_get_shmem_info():  >>
PMD: bnx2x_acquire_hw_lock():  >>
PMD: bnx2x_release_hw_lock():  >>
PMD: bnx2x_get_phy_info():  >>
PMD: bnx2x_reset_common():  >>
EAL: PCI device 0000:04:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
EAL:   PCI memory mapped at 0x7fffdee00000
EAL:   PCI memory mapped at 0x7fffde600000
EAL:   PCI memory mapped at 0x7ffff7fa1000
PMD: eth_bnx2x_dev_init():  >>
PMD: bnx2x_common_dev_init():  >>
PMD: bnx2x_probe_pci_caps():  >>
PMD: bnx2x_get_shmem_info():  >>
PMD: bnx2x_acquire_hw_lock():  >>
PMD: bnx2x_release_hw_lock():  >>
PMD: bnx2x_get_phy_info():  >>
PMD: bnx2x_reset_common():  >>
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
Interactive-mode selected
Configuring Port 0 (socket 0)
PMD: bnx2x_dev_configure():  >>
PMD: bnx2x_dev_tx_queue_setup(): fp[00] req_bd=512, thresh=512, usable_bd=1020, total_bd=1024, tx_pages=4
PMD: bnx2x_dev_rx_queue_setup(): fp[00] req_bd=128, thresh=0, usable_bd=510, total_bd=512, rx_pages=1, cq_pages=8
PMD: bnx2x_dev_start():  >>
PMD: bnx2x_nic_load():  >>
PMD: bnx2x_ilt_set_info():  >>
PMD: bnx2x_acquire_hw_lock():  >>
PMD: bnx2x_set_pf_load():  >>
PMD: bnx2x_release_hw_lock():  >>
PMD: bnx2x_nic_load_request():  >>
PMD: bnx2x_nic_load_pmf():  >>
PMD: bnx2x_init_func_obj():  >>
PMD: bnx2x_init_hw():  >>
PMD: bnx2x_acquire_hw_lock():  >>
PMD: bnx2x_reset_common():  >>
PMD: bnx2x_release_hw_lock():  >>
PANIC in bnx2x_write_dmae():
DMAE failed (-1)15: [./dpdk-2.1.0/app/testpmd() [0x438303]]
14: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7ffff6eedec5]]
13: [./dpdk-2.1.0/app/testpmd(main+0x2c1) [0x436a81]]
12: [./dpdk-2.1.0/app/testpmd(start_port+0x318) [0x43a8b8]]
11: [./dpdk-2.1.0/app/testpmd(rte_eth_dev_start+0xad) [0x573d2d]]
10: [./dpdk-2.1.0/app/testpmd() [0x466a24]]
9: [./dpdk-2.1.0/app/testpmd(bnx2x_init+0x21b) [0x476ffb]]
8: [./dpdk-2.1.0/app/testpmd() [0x471eb4]]
7: [./dpdk-2.1.0/app/testpmd(ecore_func_state_change+0x102) [0x4c4aa2]]
6: [./dpdk-2.1.0/app/testpmd() [0x4c1f62]]
5: [./dpdk-2.1.0/app/testpmd() [0x46ce47]]
4: [./dpdk-2.1.0/app/testpmd() [0x46c1e7]]
3: [./dpdk-2.1.0/app/testpmd() [0x468867]]
2: [./dpdk-2.1.0/app/testpmd(__rte_panic+0xc9) [0x430267]]
1: [./dpdk-2.1.0/app/testpmd(rte_dump_stack+0x1a) [0x4b3c3a]]
./startTestPmd: line 14: 19899 Aborted                 (core dumped) "$@"

If I turn on more debugging, I can see:

PMD: bnx2x_issue_dmae_with_comp(): DMAE timeout!
PANIC in bnx2x_write_dmae():
DMAE failed (-1)16: [./dpdk-2.1.0/app/testpmd() [0x439043]]


If running dpdk with the PFs on bare metal fails, I am not surprised that running with SR-IOV fails.

# ./tools/dpdk_nic_bind.py --bind igb_uio 0000:04:01.0 0000:04:09.0
root@gary-desk:/home/skyfall# ifconfig eth4 up^C
root@gary-desk:/home/skyfall# ./tools/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:04:01.0 'NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function' drv=igb_uio unused=
0000:04:09.0 'NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme II BCM57810 10 Gigabit Ethernet' if=eth4 drv=bnx2x unused=igb_uio
0000:04:00.1 'NetXtreme II BCM57810 10 Gigabit Ethernet' if=eth5 drv=bnx2x unused=igb_uio
0000:05:00.0 'I350 Gigabit Network Connection' if=eth0 drv=igb unused=igb_uio *Active*
0000:05:00.1 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices
=====================
<none>


Running testpmd with SR-IOV on bare metal:
# ./app/testpmd -c 7 -n 4 --socket-mem 900 -- -i --burst=64 --txqflags=0xf00 --nb-cores=2 --nb-ports=2 --portmask=0x3
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 6 on socket 0
EAL: Detected lcore 7 as core 7 on socket 0
EAL: Detected lcore 8 as core 0 on socket 0
EAL: Detected lcore 9 as core 1 on socket 0
EAL: Detected lcore 10 as core 2 on socket 0
EAL: Detected lcore 11 as core 3 on socket 0
EAL: Detected lcore 12 as core 4 on socket 0
EAL: Detected lcore 13 as core 5 on socket 0
EAL: Detected lcore 14 as core 6 on socket 0
EAL: Detected lcore 15 as core 7 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ffff4200000 (size = 0x200000)
EAL: Ask a virtual area of 0x14a00000 bytes
EAL: Virtual area found at 0x7fffdf600000 (size = 0x14a00000)
EAL: Ask a virtual area of 0x56800000 bytes
EAL: Virtual area found at 0x7fff88c00000 (size = 0x56800000)
EAL: Ask a virtual area of 0x4000000 bytes
EAL: Virtual area found at 0x7fff84a00000 (size = 0x4000000)
EAL: Ask a virtual area of 0x54200000 bytes
EAL: Virtual area found at 0x7fff30600000 (size = 0x54200000)
EAL: Ask a virtual area of 0x14c00000 bytes
EAL: Virtual area found at 0x7fff1b800000 (size = 0x14c00000)
EAL: Ask a virtual area of 0x125400000 bytes
EAL: Virtual area found at 0x7ffdf6200000 (size = 0x125400000)
EAL: Ask a virtual area of 0x1b000000 bytes
EAL: Virtual area found at 0x7ffddb000000 (size = 0x1b000000)
EAL: Ask a virtual area of 0x18600000 bytes
EAL: Virtual area found at 0x7ffdc2800000 (size = 0x18600000)
EAL: Ask a virtual area of 0xf8600000 bytes
EAL: Virtual area found at 0x7ffcca000000 (size = 0xf8600000)
EAL: Ask a virtual area of 0x39400000 bytes
EAL: Virtual area found at 0x7ffc90a00000 (size = 0x39400000)
EAL: Ask a virtual area of 0x1fc00000 bytes
EAL: Virtual area found at 0x7ffc70c00000 (size = 0x1fc00000)
EAL: Ask a virtual area of 0x74c00000 bytes
EAL: Virtual area found at 0x7ffbfbe00000 (size = 0x74c00000)
EAL: Ask a virtual area of 0x9000000 bytes
EAL: Virtual area found at 0x7ffbf2c00000 (size = 0x9000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ffbf2800000 (size = 0x200000)
EAL: Requesting 450 pages of size 2MB from socket 0
EAL: TSC frequency is ~1995380 KHz
EAL: Master lcore 0 is ready (tid=f7fe4940;cpuset=[0])
PMD: rte_bnx2x_pmd_init():  >>
PMD: rte_bnx2xvf_pmd_init():  >>
EAL: lcore 2 is ready (tid=f5ec9700;cpuset=[2])
EAL: lcore 1 is ready (tid=f66ca700;cpuset=[1])
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:04:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:04:01.0 on NUMA socket 0
EAL:   probe driver: 14e4:16af rte_bnx2xvf_pmd
EAL:   PCI memory mapped at 0x7ffff4400000
EAL:   PCI memory mapped at 0x7ffff4408000
PMD: eth_bnx2xvf_dev_init():  >>
PMD: bnx2x_common_dev_init():  >>
PMD: bnx2x_probe_pci_caps():  >>
EAL: PCI device 0000:04:09.0 on NUMA socket 0
EAL:   probe driver: 14e4:16af rte_bnx2xvf_pmd
EAL:   PCI memory mapped at 0x7ffff440a000
EAL:   PCI memory mapped at 0x7ffff4412000
PMD: eth_bnx2xvf_dev_init():  >>
PMD: bnx2x_common_dev_init():  >>
PMD: bnx2x_probe_pci_caps():  >>
Interactive-mode selected
Configuring Port 0 (socket 0)
PMD: bnx2x_dev_configure():  >>
PMD: bnx2x_dev_tx_queue_setup(): fp[00] req_bd=512, thresh=512, usable_bd=1020, total_bd=1024, tx_pages=4
PMD: bnx2x_dev_rx_queue_setup(): fp[00] req_bd=128, thresh=0, usable_bd=510, total_bd=512, rx_pages=1, cq_pages=8
PMD: bnx2x_dev_start():  >>
PMD: bnx2x_nic_load():  >>
Fail to start port 0
Configuring Port 1 (socket 0)
PMD: bnx2x_dev_configure():  >>
PMD: bnx2x_dev_tx_queue_setup(): fp[00] req_bd=512, thresh=512, usable_bd=1020, total_bd=1024, tx_pages=4
PMD: bnx2x_dev_rx_queue_setup(): fp[00] req_bd=128, thresh=0, usable_bd=510, total_bd=512, rx_pages=1, cq_pages=8
PMD: bnx2x_dev_start():  >>
PMD: bnx2x_nic_load():  >>
Fail to start port 1
Please stop the ports first
Done
PMD: bnx2x_promisc_enable():  >>
PMD: bnx2x_promisc_enable():  >>
testpmd>

Gary

> Harish
> 
> 
> ________________________________
> 
> This message and any attached documents contain information from the
> sending company or its parent company(s), subsidiaries, divisions or
> branch offices that may be confidential. If you are not the intended
> recipient, you may not read, copy, distribute, or use this information. If
> you have received this transmission in error, please notify the sender
> immediately by reply e-mail and then delete this message.