[dpdk-dev] Problem with Intel i40e XL710 dpdk driver

[dpdk-dev] Problem with Intel i40e XL710 dpdk driver

[Karthick, A.R.]

Hi,
 I am seeing a "Failed to init adminq: -54" or admin queue timeouts
 while initializing the admin queue for i40e xl710 intel nic.
 (Intel server is a E5-2670)

 First things first.
 I am running the latest firmware.
 The kernel module is not loaded and yes, it works with the i40e kernel
driver. (latest or otherwise)
And this problem comes even with dpdk 2.0/2.1 or the latest stable. So
there's that.

 I have done a bunch of debugging and here are my findings.
 With the card configured in 2x40g or 4x10g mode, it _ALWAYS_ works with
 successfully initializing pci function 0 or port 0.
 It always fails to subsequently initialize the rest.
 Even if unbind the igb uio for port 0 and bind only port 1 or port 2,3,4
in 4x10g mode,
 it fails.

 Since it works with the kernel driver, I tried to see if there were
differences in the way registers are setup for i40e driver in kernel and
dpdk.
 They look mostly to be the same but obviously there were subtle
differences.
 From what I could fathom, I couldn't see much and whatever little was
caught, I tried to keep the dpdk code in sync and it still failed.

 While stepping through gdb all the way from eal pci to pci uio map to
eth_i40e_dev_init,
 to the failure in obtaining the firmware revision for port1 during
i40e_init_adminq,
 I did confirm that the memory map was right for the pci.

 So the hw->hw_addr looks correct for port 1 correlating it to the uio1 map
or the physical address from lspci or kernel driver when using the kernel
driver which works.

 However the admin queue seems to be not processing any request for port 1.
 Note that port 0 always works and its the same code for others with a
different eal dev/hw instance.

But for other ports like port1, after correctly setting up the adminq
registers and memory map,
 it always fails to obtain the firmware revision since the i40e_asq_done is
returning 0 for the
 head register at 0x80300 and doesn't match the next_in_use when starting
at 1.
 So it always returns pending or false in i40e_asq_done which is retried a
certain times after resetting the aq by i40e_init_adminq but ultimately
gives up.

 Thoughts and wondering if you guys have seen this and have a fix or patch
that is not in upstream yet.

 Failure enclosed below as mentioned above in detail: (with a 4x10g mode
for the card but same failure with 2x40g mode as well. No difference. Port
0 always succeeds but subsequent ports fail.
 And same result even with port 0 not bound and starting with the
initialization of port 2,3,4 which always fails.

EAL: lcore 1 is ready (tid=6bd30700;cpuset=[1])
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:83:00.0 on NUMA socket 1
EAL:   probe driver: 8086:1583 rte_i40e_pmd
EAL:   PCI memory mapped at 0x7f2f80000000
EAL:   PCI memory mapped at 0x7f2f80800000
PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001dca
PMD: i40e_pf_parameter_init(): Max supported VSIs:34
PMD: i40e_pf_parameter_init(): PF queue pairs:64
PMD: i40e_pf_parameter_init(): Max VMDQ VSI num:34
PMD: i40e_pf_parameter_init(): VMDQ queue pairs:4
EAL: PCI device 0000:83:00.1 on NUMA socket 1
EAL:   probe driver: 8086:1583 rte_i40e_pmd
EAL:   PCI memory mapped at 0x7f2f80808000
EAL:   PCI memory mapped at 0x7f2f81008000
PMD: eth_i40e_dev_init(): Failed to init adminq: -54
EAL: Error - exiting with code: 1
 Cause: Requested device 0000:83:00.1 cannot be used

Regards,
-Karthick

Re: [dpdk-dev] Problem with Intel i40e XL710 dpdk driver

[Karthick, A.R.]

I found DMAR errors while bringing up other ports except port 0.
So rebooting the kernel with intel_iommu=off fixes it and dpdk i40e
initializes fine for all ports.

For some reason, the intel qcu64e mode change utility doesn't work if I
want to change back the mode from 4x10 to 2x40.
Tried several times but never reverts back to 2x40 from 4x10 though that is
a different issue.

Regards,
-Karthick

On Thu, Jan 14, 2016 at 6:26 PM, Karthick, A.R. <kramanar@ciena.com> wrote:

> Hi,
>  I am seeing a "Failed to init adminq: -54" or admin queue timeouts
>  while initializing the admin queue for i40e xl710 intel nic.
>  (Intel server is a E5-2670)
>
>  First things first.
>  I am running the latest firmware.
>  The kernel module is not loaded and yes, it works with the i40e kernel
> driver. (latest or otherwise)
> And this problem comes even with dpdk 2.0/2.1 or the latest stable. So
> there's that.
>
>  I have done a bunch of debugging and here are my findings.
>  With the card configured in 2x40g or 4x10g mode, it _ALWAYS_ works with
>  successfully initializing pci function 0 or port 0.
>  It always fails to subsequently initialize the rest.
>  Even if unbind the igb uio for port 0 and bind only port 1 or port 2,3,4
> in 4x10g mode,
>  it fails.
>
>  Since it works with the kernel driver, I tried to see if there were
> differences in the way registers are setup for i40e driver in kernel and
> dpdk.
>  They look mostly to be the same but obviously there were subtle
> differences.
>  From what I could fathom, I couldn't see much and whatever little was
> caught, I tried to keep the dpdk code in sync and it still failed.
>
>  While stepping through gdb all the way from eal pci to pci uio map to
> eth_i40e_dev_init,
>  to the failure in obtaining the firmware revision for port1 during
> i40e_init_adminq,
>  I did confirm that the memory map was right for the pci.
>
>  So the hw->hw_addr looks correct for port 1 correlating it to the uio1
> map or the physical address from lspci or kernel driver when using the
> kernel driver which works.
>
>  However the admin queue seems to be not processing any request for port 1.
>  Note that port 0 always works and its the same code for others with a
> different eal dev/hw instance.
>
> But for other ports like port1, after correctly setting up the adminq
> registers and memory map,
>  it always fails to obtain the firmware revision since the i40e_asq_done
> is returning 0 for the
>  head register at 0x80300 and doesn't match the next_in_use when starting
> at 1.
>  So it always returns pending or false in i40e_asq_done which is retried a
> certain times after resetting the aq by i40e_init_adminq but ultimately
> gives up.
>
>  Thoughts and wondering if you guys have seen this and have a fix or patch
> that is not in upstream yet.
>
>  Failure enclosed below as mentioned above in detail: (with a 4x10g mode
> for the card but same failure with 2x40g mode as well. No difference. Port
> 0 always succeeds but subsequent ports fail.
>  And same result even with port 0 not bound and starting with the
> initialization of port 2,3,4 which always fails.
>
> EAL: lcore 1 is ready (tid=6bd30700;cpuset=[1])
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 rte_igb_pmd
> EAL:   Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:01:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:1521 rte_igb_pmd
> EAL:   Not managed by a supported kernel driver, skipped
> EAL: PCI device 0000:83:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:1583 rte_i40e_pmd
> EAL:   PCI memory mapped at 0x7f2f80000000
> EAL:   PCI memory mapped at 0x7f2f80800000
> PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001dca
> PMD: i40e_pf_parameter_init(): Max supported VSIs:34
> PMD: i40e_pf_parameter_init(): PF queue pairs:64
> PMD: i40e_pf_parameter_init(): Max VMDQ VSI num:34
> PMD: i40e_pf_parameter_init(): VMDQ queue pairs:4
> EAL: PCI device 0000:83:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:1583 rte_i40e_pmd
> EAL:   PCI memory mapped at 0x7f2f80808000
> EAL:   PCI memory mapped at 0x7f2f81008000
> PMD: eth_i40e_dev_init(): Failed to init adminq: -54
> EAL: Error - exiting with code: 1
>  Cause: Requested device 0000:83:00.1 cannot be used
>
> Regards,
> -Karthick
>
>
>
>