[dpdk-dev] Status of bnx2x pmd?

[dpdk-dev] Status of bnx2x pmd?

[Mussar, Gary]

I have been attempting to build testpmd with the bnx2x pmd enabled to try and talk through a bcm57810 10G NIC card. I have tried using SR-IOV VFs and I have tried using the PFs. Neither seem to be able to be properly initialized in testpmd and/or testpmd aborts.

Is this PMD actually usable or is it trash?

Gary

Re: [dpdk-dev] Status of bnx2x pmd?

[Charles (Chas) Williams]

On Fri, 2015-10-30 at 18:10 -0400, Mussar, Gary wrote:
> I have been attempting to build testpmd with the bnx2x pmd enabled to try and talk through a bcm57810 10G NIC card. I have tried using SR-IOV VFs and I have tried using the PFs. Neither seem to be able to be properly initialized in testpmd and/or testpmd aborts.
> 
> Is this PMD actually usable or is it trash?
> 
> Gary

See the two patches I have posted to this list recently w/respect to the
bnx2x.  At this point, I have a VF on a 57810 working in the TX direction.
I am unable to figure out what is wrong with the RX direction though.

The rx packet counters increment but I don't seem to get packet delivery.

There is probably a minor issue with bnx2x_fill_accept_flags() being
used in both the PF and VF drivers when setting the rx filter but fixing
that didn't seem to help.

At this point it is a long slog comparing the linux driver to the
PMD driver.

Re: [dpdk-dev] Status of bnx2x pmd?

[Mussar, Gary]

> -----Original Message-----
> From: Harish Patil [mailto:harish.patil@qlogic.com]
> Sent: Monday, November 02, 2015 01:51
> To: Mussar, Gary; dev@dpdk.org
> Subject: Re: [dpdk-dev] Status of bnx2x pmd?
> 
> >
> >I have been attempting to build testpmd with the bnx2x pmd enabled to try
> >and talk through a bcm57810 10G NIC card. I have tried using SR-IOV VFs
> >and I have tried using the PFs. Neither seem to be able to be properly
> >initialized in testpmd and/or testpmd aborts.
> >
> >Is this PMD actually usable or is it trash?
> >
> >Gary
> >
> 
> Could you please explain the SR-IOV configuration that you are trying out?
> Please provide all the details.
> 
> Thanks,


I ultimately want to run dpdk in a VM with PCI pass through on SR-IOV ports. I have had no luck there so I thought I would start with DPDK on bare metal using the PFs.

I am using Ubuntu 14.04.3 with the Ubuntu 3.19.0-31 kernel and dpdk from the dpdk.org master branch (d739ba4c6abf00eeeb680ab4e9540899182ea74d).

# ./tools/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:04:00.0 'NetXtreme II BCM57810 10 Gigabit Ethernet' drv=igb_uio unused=
0000:04:00.1 'NetXtreme II BCM57810 10 Gigabit Ethernet' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:05:00.0 'I350 Gigabit Network Connection' if=eth0 drv=igb unused=igb_uio *Active*
0000:05:00.1 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices
=====================
<none>

Running testpmd:

./app/testpmd -c 7 -n 4 --socket-mem 900 -- -i --burst=64 --txqflags=0xf00 --nb-cores=2 --nb-ports=2 --portmask=0x3
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 6 on socket 0
EAL: Detected lcore 7 as core 7 on socket 0
EAL: Detected lcore 8 as core 0 on socket 0
EAL: Detected lcore 9 as core 1 on socket 0
EAL: Detected lcore 10 as core 2 on socket 0
EAL: Detected lcore 11 as core 3 on socket 0
EAL: Detected lcore 12 as core 4 on socket 0
EAL: Detected lcore 13 as core 5 on socket 0
EAL: Detected lcore 14 as core 6 on socket 0
EAL: Detected lcore 15 as core 7 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ffff4200000 (size = 0x200000)
EAL: Ask a virtual area of 0x14a00000 bytes
EAL: Virtual area found at 0x7fffdf600000 (size = 0x14a00000)
EAL: Ask a virtual area of 0x56800000 bytes
EAL: Virtual area found at 0x7fff88c00000 (size = 0x56800000)
EAL: Ask a virtual area of 0x4000000 bytes
EAL: Virtual area found at 0x7fff84a00000 (size = 0x4000000)
EAL: Ask a virtual area of 0x54200000 bytes
EAL: Virtual area found at 0x7fff30600000 (size = 0x54200000)
EAL: Ask a virtual area of 0x14c00000 bytes
EAL: Virtual area found at 0x7fff1b800000 (size = 0x14c00000)
EAL: Ask a virtual area of 0x125400000 bytes
EAL: Virtual area found at 0x7ffdf6200000 (size = 0x125400000)
EAL: Ask a virtual area of 0x1b000000 bytes
EAL: Virtual area found at 0x7ffddb000000 (size = 0x1b000000)
EAL: Ask a virtual area of 0x18600000 bytes
EAL: Virtual area found at 0x7ffdc2800000 (size = 0x18600000)
EAL: Ask a virtual area of 0xf8600000 bytes
EAL: Virtual area found at 0x7ffcca000000 (size = 0xf8600000)
EAL: Ask a virtual area of 0x39400000 bytes
EAL: Virtual area found at 0x7ffc90a00000 (size = 0x39400000)
EAL: Ask a virtual area of 0x1fc00000 bytes
EAL: Virtual area found at 0x7ffc70c00000 (size = 0x1fc00000)
EAL: Ask a virtual area of 0x74c00000 bytes
EAL: Virtual area found at 0x7ffbfbe00000 (size = 0x74c00000)
EAL: Ask a virtual area of 0x9000000 bytes
EAL: Virtual area found at 0x7ffbf2c00000 (size = 0x9000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ffbf2800000 (size = 0x200000)
EAL: Requesting 450 pages of size 2MB from socket 0
EAL: TSC frequency is ~1995380 KHz
EAL: Master lcore 0 is ready (tid=f7fe4940;cpuset=[0])
PMD: rte_bnx2x_pmd_init():  >>
PMD: rte_bnx2xvf_pmd_init():  >>
EAL: lcore 2 is ready (tid=f5ec9700;cpuset=[2])
EAL: lcore 1 is ready (tid=f66ca700;cpuset=[1])
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
EAL:   PCI memory mapped at 0x7ffff4400000
EAL:   PCI memory mapped at 0x7ffff4c00000
EAL:   PCI memory mapped at 0x7ffff5400000
PMD: eth_bnx2x_dev_init():  >>
PMD: bnx2x_common_dev_init():  >>
PMD: bnx2x_probe_pci_caps():  >>
PMD: bnx2x_get_shmem_info():  >>
PMD: bnx2x_acquire_hw_lock():  >>
PMD: bnx2x_release_hw_lock():  >>
PMD: bnx2x_get_phy_info():  >>
PMD: bnx2x_reset_common():  >>
EAL: PCI device 0000:04:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
EAL:   PCI memory mapped at 0x7fffdee00000
EAL:   PCI memory mapped at 0x7fffde600000
EAL:   PCI memory mapped at 0x7ffff7fa1000
PMD: eth_bnx2x_dev_init():  >>
PMD: bnx2x_common_dev_init():  >>
PMD: bnx2x_probe_pci_caps():  >>
PMD: bnx2x_get_shmem_info():  >>
PMD: bnx2x_acquire_hw_lock():  >>
PMD: bnx2x_release_hw_lock():  >>
PMD: bnx2x_get_phy_info():  >>
PMD: bnx2x_reset_common():  >>
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
Interactive-mode selected
Configuring Port 0 (socket 0)
PMD: bnx2x_dev_configure():  >>
PMD: bnx2x_dev_tx_queue_setup(): fp[00] req_bd=512, thresh=512, usable_bd=1020, total_bd=1024, tx_pages=4
PMD: bnx2x_dev_rx_queue_setup(): fp[00] req_bd=128, thresh=0, usable_bd=510, total_bd=512, rx_pages=1, cq_pages=8
PMD: bnx2x_dev_start():  >>
PMD: bnx2x_nic_load():  >>
PMD: bnx2x_ilt_set_info():  >>
PMD: bnx2x_acquire_hw_lock():  >>
PMD: bnx2x_set_pf_load():  >>
PMD: bnx2x_release_hw_lock():  >>
PMD: bnx2x_nic_load_request():  >>
PMD: bnx2x_nic_load_pmf():  >>
PMD: bnx2x_init_func_obj():  >>
PMD: bnx2x_init_hw():  >>
PMD: bnx2x_acquire_hw_lock():  >>
PMD: bnx2x_reset_common():  >>
PMD: bnx2x_release_hw_lock():  >>
PANIC in bnx2x_write_dmae():
DMAE failed (-1)15: [./dpdk-2.1.0/app/testpmd() [0x438303]]
14: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7ffff6eedec5]]
13: [./dpdk-2.1.0/app/testpmd(main+0x2c1) [0x436a81]]
12: [./dpdk-2.1.0/app/testpmd(start_port+0x318) [0x43a8b8]]
11: [./dpdk-2.1.0/app/testpmd(rte_eth_dev_start+0xad) [0x573d2d]]
10: [./dpdk-2.1.0/app/testpmd() [0x466a24]]
9: [./dpdk-2.1.0/app/testpmd(bnx2x_init+0x21b) [0x476ffb]]
8: [./dpdk-2.1.0/app/testpmd() [0x471eb4]]
7: [./dpdk-2.1.0/app/testpmd(ecore_func_state_change+0x102) [0x4c4aa2]]
6: [./dpdk-2.1.0/app/testpmd() [0x4c1f62]]
5: [./dpdk-2.1.0/app/testpmd() [0x46ce47]]
4: [./dpdk-2.1.0/app/testpmd() [0x46c1e7]]
3: [./dpdk-2.1.0/app/testpmd() [0x468867]]
2: [./dpdk-2.1.0/app/testpmd(__rte_panic+0xc9) [0x430267]]
1: [./dpdk-2.1.0/app/testpmd(rte_dump_stack+0x1a) [0x4b3c3a]]
./startTestPmd: line 14: 19899 Aborted                 (core dumped) "$@"

If I turn on more debugging, I can see:

PMD: bnx2x_issue_dmae_with_comp(): DMAE timeout!
PANIC in bnx2x_write_dmae():
DMAE failed (-1)16: [./dpdk-2.1.0/app/testpmd() [0x439043]]


If running dpdk with the PFs on bare metal fails, I am not surprised that running with SR-IOV fails.

# ./tools/dpdk_nic_bind.py --bind igb_uio 0000:04:01.0 0000:04:09.0
root@gary-desk:/home/skyfall# ifconfig eth4 up^C
root@gary-desk:/home/skyfall# ./tools/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:04:01.0 'NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function' drv=igb_uio unused=
0000:04:09.0 'NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme II BCM57810 10 Gigabit Ethernet' if=eth4 drv=bnx2x unused=igb_uio
0000:04:00.1 'NetXtreme II BCM57810 10 Gigabit Ethernet' if=eth5 drv=bnx2x unused=igb_uio
0000:05:00.0 'I350 Gigabit Network Connection' if=eth0 drv=igb unused=igb_uio *Active*
0000:05:00.1 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio

Other network devices
=====================
<none>


Running testpmd with SR-IOV on bare metal:
# ./app/testpmd -c 7 -n 4 --socket-mem 900 -- -i --burst=64 --txqflags=0xf00 --nb-cores=2 --nb-ports=2 --portmask=0x3
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 6 on socket 0
EAL: Detected lcore 7 as core 7 on socket 0
EAL: Detected lcore 8 as core 0 on socket 0
EAL: Detected lcore 9 as core 1 on socket 0
EAL: Detected lcore 10 as core 2 on socket 0
EAL: Detected lcore 11 as core 3 on socket 0
EAL: Detected lcore 12 as core 4 on socket 0
EAL: Detected lcore 13 as core 5 on socket 0
EAL: Detected lcore 14 as core 6 on socket 0
EAL: Detected lcore 15 as core 7 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ffff4200000 (size = 0x200000)
EAL: Ask a virtual area of 0x14a00000 bytes
EAL: Virtual area found at 0x7fffdf600000 (size = 0x14a00000)
EAL: Ask a virtual area of 0x56800000 bytes
EAL: Virtual area found at 0x7fff88c00000 (size = 0x56800000)
EAL: Ask a virtual area of 0x4000000 bytes
EAL: Virtual area found at 0x7fff84a00000 (size = 0x4000000)
EAL: Ask a virtual area of 0x54200000 bytes
EAL: Virtual area found at 0x7fff30600000 (size = 0x54200000)
EAL: Ask a virtual area of 0x14c00000 bytes
EAL: Virtual area found at 0x7fff1b800000 (size = 0x14c00000)
EAL: Ask a virtual area of 0x125400000 bytes
EAL: Virtual area found at 0x7ffdf6200000 (size = 0x125400000)
EAL: Ask a virtual area of 0x1b000000 bytes
EAL: Virtual area found at 0x7ffddb000000 (size = 0x1b000000)
EAL: Ask a virtual area of 0x18600000 bytes
EAL: Virtual area found at 0x7ffdc2800000 (size = 0x18600000)
EAL: Ask a virtual area of 0xf8600000 bytes
EAL: Virtual area found at 0x7ffcca000000 (size = 0xf8600000)
EAL: Ask a virtual area of 0x39400000 bytes
EAL: Virtual area found at 0x7ffc90a00000 (size = 0x39400000)
EAL: Ask a virtual area of 0x1fc00000 bytes
EAL: Virtual area found at 0x7ffc70c00000 (size = 0x1fc00000)
EAL: Ask a virtual area of 0x74c00000 bytes
EAL: Virtual area found at 0x7ffbfbe00000 (size = 0x74c00000)
EAL: Ask a virtual area of 0x9000000 bytes
EAL: Virtual area found at 0x7ffbf2c00000 (size = 0x9000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7ffbf2800000 (size = 0x200000)
EAL: Requesting 450 pages of size 2MB from socket 0
EAL: TSC frequency is ~1995380 KHz
EAL: Master lcore 0 is ready (tid=f7fe4940;cpuset=[0])
PMD: rte_bnx2x_pmd_init():  >>
PMD: rte_bnx2xvf_pmd_init():  >>
EAL: lcore 2 is ready (tid=f5ec9700;cpuset=[2])
EAL: lcore 1 is ready (tid=f66ca700;cpuset=[1])
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:04:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:168e rte_bnx2x_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:04:01.0 on NUMA socket 0
EAL:   probe driver: 14e4:16af rte_bnx2xvf_pmd
EAL:   PCI memory mapped at 0x7ffff4400000
EAL:   PCI memory mapped at 0x7ffff4408000
PMD: eth_bnx2xvf_dev_init():  >>
PMD: bnx2x_common_dev_init():  >>
PMD: bnx2x_probe_pci_caps():  >>
EAL: PCI device 0000:04:09.0 on NUMA socket 0
EAL:   probe driver: 14e4:16af rte_bnx2xvf_pmd
EAL:   PCI memory mapped at 0x7ffff440a000
EAL:   PCI memory mapped at 0x7ffff4412000
PMD: eth_bnx2xvf_dev_init():  >>
PMD: bnx2x_common_dev_init():  >>
PMD: bnx2x_probe_pci_caps():  >>
Interactive-mode selected
Configuring Port 0 (socket 0)
PMD: bnx2x_dev_configure():  >>
PMD: bnx2x_dev_tx_queue_setup(): fp[00] req_bd=512, thresh=512, usable_bd=1020, total_bd=1024, tx_pages=4
PMD: bnx2x_dev_rx_queue_setup(): fp[00] req_bd=128, thresh=0, usable_bd=510, total_bd=512, rx_pages=1, cq_pages=8
PMD: bnx2x_dev_start():  >>
PMD: bnx2x_nic_load():  >>
Fail to start port 0
Configuring Port 1 (socket 0)
PMD: bnx2x_dev_configure():  >>
PMD: bnx2x_dev_tx_queue_setup(): fp[00] req_bd=512, thresh=512, usable_bd=1020, total_bd=1024, tx_pages=4
PMD: bnx2x_dev_rx_queue_setup(): fp[00] req_bd=128, thresh=0, usable_bd=510, total_bd=512, rx_pages=1, cq_pages=8
PMD: bnx2x_dev_start():  >>
PMD: bnx2x_nic_load():  >>
Fail to start port 1
Please stop the ports first
Done
PMD: bnx2x_promisc_enable():  >>
PMD: bnx2x_promisc_enable():  >>
testpmd>

Gary

> Harish
> 
> 
> ________________________________
> 
> This message and any attached documents contain information from the
> sending company or its parent company(s), subsidiaries, divisions or
> branch offices that may be confidential. If you are not the intended
> recipient, you may not read, copy, distribute, or use this information. If
> you have received this transmission in error, please notify the sender
> immediately by reply e-mail and then delete this message.

Re: [dpdk-dev] Status of bnx2x pmd?

[Harish Patil]

>
>I have been attempting to build testpmd with the bnx2x pmd enabled to try
>and talk through a bcm57810 10G NIC card. I have tried using SR-IOV VFs
>and I have tried using the PFs. Neither seem to be able to be properly
>initialized in testpmd and/or testpmd aborts.
>
>Is this PMD actually usable or is it trash?
>
>Gary
>

Could you please explain the SR-IOV configuration that you are trying out?
Please provide all the details.

Thanks,
Harish


________________________________

This message and any attached documents contain information from the sending company or its parent company(s), subsidiaries, divisions or branch offices that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.