intel X550T dpdk support issue

intel X550T dpdk support issue

[Satish Patel]

Folks,

I am new to this nic series and just found a strange issue, i am sure
people are using this nic in production with dpdk.

# ovs-vsctl add-port br-vlan dpdk-1 -- set Interface dpdk-1 type=dpdk
options:dpdk-devargs=0000:3b:00.1
ovs-vsctl: Error detected while setting up 'dpdk-1': Error attaching
device '0000:3b:00.1' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".


2021-10-04T13:20:05.751Z|00272|dpdk|ERR|EAL: Driver cannot attach the
device (0000:3b:00.1)
2021-10-04T13:20:05.751Z|00273|dpdk|ERR|EAL: Failed to attach device
on primary process
2021-10-04T13:20:05.751Z|00274|netdev_dpdk|WARN|Error attaching device
'0000:3b:00.1' to DPDK
2021-10-04T13:20:05.751Z|00275|netdev|WARN|dpdk-1: could not set
configuration (Invalid argument)
2021-10-04T13:20:05.751Z|00276|dpdk|ERR|Invalid port_id=32


# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:3b:00.0 'Ethernet Controller 10G X550T 1563' drv=vfio-pci unused=ixgbe
0000:3b:00.1 'Ethernet Controller 10G X550T 1563' drv=vfio-pci unused=ixgbe

Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f'
if=eno1 drv=tg3 unused=vfio-pci
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f'
if=eno2 drv=tg3 unused=vfio-pci
0000:af:00.0 'Ethernet Controller 10G X550T 1563' if=int1 drv=ixgbe
unused=vfio-pci
0000:af:00.1 'Ethernet Controller 10G X550T 1563' if=ext1 drv=ixgbe
unused=vfio-pci


# ethtool -i ext1
driver: ixgbe
version: 5.9.4
firmware-version: 0x800013d1, 20.0.16
expansion-rom-version:
bus-info: 0000:af:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ovs-vswitchd -V
ovs-vswitchd (Open vSwitch) 2.13.3
DPDK 19.11.7

# uname  -a
Linux ostack-ams-comp-dpdk-1.example.com 5.4.0-86-generic #97-Ubuntu
SMP Fri Sep 17 19:19:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Re: intel X550T dpdk support issue

[Satish Patel]

Is this my issue ?

# dpdk-proc-info
EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_4651_a967f74152a
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Cannot initialize tailq: RTE_EVENT_RING
Tailq 0: qname:<RTE_RING>, tqh_first:(nil), tqh_last:0x100004488
Tailq 1: qname:<RTE_HASH>, tqh_first:(nil), tqh_last:0x1000044b8
Tailq 2: qname:<RTE_FBK_HASH>, tqh_first:(nil), tqh_last:0x1000044e8
Tailq 3: qname:<RTE_MEMPOOL>, tqh_first:(nil), tqh_last:0x100004518
Tailq 4: qname:<RTE_MBUF_DYNFIELD>, tqh_first:(nil), tqh_last:0x100004548
Tailq 5: qname:<RTE_MBUF_DYNFLAG>, tqh_first:(nil), tqh_last:0x100004578
Tailq 6: qname:<UIO_RESOURCE_LIST>, tqh_first:(nil), tqh_last:0x1000045a8
Tailq 7: qname:<VFIO_RESOURCE_LIST>, tqh_first:(nil), tqh_last:0x1000045d8
Tailq 8: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 9: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 10: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 11: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 12: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 13: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 14: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 15: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 16: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 17: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 18: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 19: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 20: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 21: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 22: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 23: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 24: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 25: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 26: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 27: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 28: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 29: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 30: qname:<>, tqh_first:(nil), tqh_last:(nil)
Tailq 31: qname:<>, tqh_first:(nil), tqh_last:(nil)
EAL: FATAL: Cannot init tail queues for objects
EAL: Cannot init tail queues for objects
PANIC in main():
Cannot init EAL
5: [dpdk-proc-info(+0x3e2e) [0x561372f34e2e]]
4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f7f2a8560b3]]
3: [dpdk-proc-info(+0x293c) [0x561372f3393c]]
2: [/lib/x86_64-linux-gnu/librte_eal.so.20.0(__rte_panic+0xc5) [0x7f7f2ab7a3cf]]
1: [/lib/x86_64-linux-gnu/librte_eal.so.20.0(rte_dump_stack+0x32)
[0x7f7f2ab91eb2]]
Aborted




On Mon, Oct 4, 2021 at 2:52 PM Satish Patel <satish.txt@gmail.com> wrote:
>
> Folks,
>
> I am new to this nic series and just found a strange issue, i am sure
> people are using this nic in production with dpdk.
>
> # ovs-vsctl add-port br-vlan dpdk-1 -- set Interface dpdk-1 type=dpdk
> options:dpdk-devargs=0000:3b:00.1
> ovs-vsctl: Error detected while setting up 'dpdk-1': Error attaching
> device '0000:3b:00.1' to DPDK.  See ovs-vswitchd log for details.
> ovs-vsctl: The default log directory is "/var/log/openvswitch".
>
>
> 2021-10-04T13:20:05.751Z|00272|dpdk|ERR|EAL: Driver cannot attach the
> device (0000:3b:00.1)
> 2021-10-04T13:20:05.751Z|00273|dpdk|ERR|EAL: Failed to attach device
> on primary process
> 2021-10-04T13:20:05.751Z|00274|netdev_dpdk|WARN|Error attaching device
> '0000:3b:00.1' to DPDK
> 2021-10-04T13:20:05.751Z|00275|netdev|WARN|dpdk-1: could not set
> configuration (Invalid argument)
> 2021-10-04T13:20:05.751Z|00276|dpdk|ERR|Invalid port_id=32
>
>
> # dpdk-devbind.py -s
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:3b:00.0 'Ethernet Controller 10G X550T 1563' drv=vfio-pci unused=ixgbe
> 0000:3b:00.1 'Ethernet Controller 10G X550T 1563' drv=vfio-pci unused=ixgbe
>
> Network devices using kernel driver
> ===================================
> 0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f'
> if=eno1 drv=tg3 unused=vfio-pci
> 0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f'
> if=eno2 drv=tg3 unused=vfio-pci
> 0000:af:00.0 'Ethernet Controller 10G X550T 1563' if=int1 drv=ixgbe
> unused=vfio-pci
> 0000:af:00.1 'Ethernet Controller 10G X550T 1563' if=ext1 drv=ixgbe
> unused=vfio-pci
>
>
> # ethtool -i ext1
> driver: ixgbe
> version: 5.9.4
> firmware-version: 0x800013d1, 20.0.16
> expansion-rom-version:
> bus-info: 0000:af:00.1
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> # ovs-vswitchd -V
> ovs-vswitchd (Open vSwitch) 2.13.3
> DPDK 19.11.7
>
> # uname  -a
> Linux ostack-ams-comp-dpdk-1.example.com 5.4.0-86-generic #97-Ubuntu
> SMP Fri Sep 17 19:19:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Re: intel X550T dpdk support issue

[David Marchand]

On Mon, Oct 4, 2021 at 11:20 PM Satish Patel <satish.txt@gmail.com> wrote:
>
> Is this my issue ?

No.

For this specific trace below, you are probably trying to attach a
secondary process to a primary process (which was OVS iiuc) which does
not have the same enabled DPDK libraries/drivers.
Having the same dpdk components (from the same version) in both
binaries is the only way for multiprocess to work: I'd recommend
linking both your binaries against a built once and for all set of
dpdk libraries.

Besides, please note that the OVS does not support multiprocess.
Issues had been reported in the past (like pdump secondary process
crashing OVS).
There is work atm in OVS that will make multiprocess disabled.


>
> # dpdk-proc-info
> EAL: Detected 48 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_4651_a967f74152a
> EAL: Selected IOVA mode 'VA'
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: Cannot initialize tailq: RTE_EVENT_RING
> Tailq 0: qname:<RTE_RING>, tqh_first:(nil), tqh_last:0x100004488
> Tailq 1: qname:<RTE_HASH>, tqh_first:(nil), tqh_last:0x1000044b8
> Tailq 2: qname:<RTE_FBK_HASH>, tqh_first:(nil), tqh_last:0x1000044e8
> Tailq 3: qname:<RTE_MEMPOOL>, tqh_first:(nil), tqh_last:0x100004518
> Tailq 4: qname:<RTE_MBUF_DYNFIELD>, tqh_first:(nil), tqh_last:0x100004548
> Tailq 5: qname:<RTE_MBUF_DYNFLAG>, tqh_first:(nil), tqh_last:0x100004578
> Tailq 6: qname:<UIO_RESOURCE_LIST>, tqh_first:(nil), tqh_last:0x1000045a8
> Tailq 7: qname:<VFIO_RESOURCE_LIST>, tqh_first:(nil), tqh_last:0x1000045d8
> Tailq 8: qname:<>, tqh_first:(nil), tqh_last:(nil)
> Tailq 9: qname:<>, tqh_first:(nil), tqh_last:(nil)
> Tailq 10: qname:<>, tqh_first:(nil), tqh_last:(nil)
> Tailq 11: qname:<>, tqh_first:(nil), tqh_last:(nil)


-- 
David Marchand

Re: intel X550T dpdk support issue

[David Marchand]

On Mon, Oct 4, 2021 at 8:53 PM Satish Patel <satish.txt@gmail.com> wrote:
>
> Folks,
>
> I am new to this nic series and just found a strange issue, i am sure
> people are using this nic in production with dpdk.
>
> # ovs-vsctl add-port br-vlan dpdk-1 -- set Interface dpdk-1 type=dpdk
> options:dpdk-devargs=0000:3b:00.1
> ovs-vsctl: Error detected while setting up 'dpdk-1': Error attaching
> device '0000:3b:00.1' to DPDK.  See ovs-vswitchd log for details.
> ovs-vsctl: The default log directory is "/var/log/openvswitch".
>
>
> 2021-10-04T13:20:05.751Z|00272|dpdk|ERR|EAL: Driver cannot attach the
> device (0000:3b:00.1)
> 2021-10-04T13:20:05.751Z|00273|dpdk|ERR|EAL: Failed to attach device
> on primary process
> 2021-10-04T13:20:05.751Z|00274|netdev_dpdk|WARN|Error attaching device
> '0000:3b:00.1' to DPDK
> 2021-10-04T13:20:05.751Z|00275|netdev|WARN|dpdk-1: could not set
> configuration (Invalid argument)
> 2021-10-04T13:20:05.751Z|00276|dpdk|ERR|Invalid port_id=32

Cc: ixgbe maintainer.

Let's first check if this is a OVS issue.
Can you run testpmd with this nic?



-- 
David Marchand

Re: intel X550T dpdk support issue

[Satish Patel]

Update: my issue has been resolved after installing the libdpdk-dev
package. look like ubuntu split drivers/libs in separate packages.

On Tue, Oct 5, 2021 at 2:59 AM David Marchand <david.marchand@redhat.com> wrote:
>
> On Mon, Oct 4, 2021 at 8:53 PM Satish Patel <satish.txt@gmail.com> wrote:
> >
> > Folks,
> >
> > I am new to this nic series and just found a strange issue, i am sure
> > people are using this nic in production with dpdk.
> >
> > # ovs-vsctl add-port br-vlan dpdk-1 -- set Interface dpdk-1 type=dpdk
> > options:dpdk-devargs=0000:3b:00.1
> > ovs-vsctl: Error detected while setting up 'dpdk-1': Error attaching
> > device '0000:3b:00.1' to DPDK.  See ovs-vswitchd log for details.
> > ovs-vsctl: The default log directory is "/var/log/openvswitch".
> >
> >
> > 2021-10-04T13:20:05.751Z|00272|dpdk|ERR|EAL: Driver cannot attach the
> > device (0000:3b:00.1)
> > 2021-10-04T13:20:05.751Z|00273|dpdk|ERR|EAL: Failed to attach device
> > on primary process
> > 2021-10-04T13:20:05.751Z|00274|netdev_dpdk|WARN|Error attaching device
> > '0000:3b:00.1' to DPDK
> > 2021-10-04T13:20:05.751Z|00275|netdev|WARN|dpdk-1: could not set
> > configuration (Invalid argument)
> > 2021-10-04T13:20:05.751Z|00276|dpdk|ERR|Invalid port_id=32
>
> Cc: ixgbe maintainer.
>
> Let's first check if this is a OVS issue.
> Can you run testpmd with this nic?
>
>
>
> --
> David Marchand
>