[dpdk-users] "eth_i40e_dev_init(): Failed to init adminq: -66" problem

[dpdk-users] "eth_i40e_dev_init(): Failed to init adminq: -66" problem

[Sunghyuk Kay]

Hi all,
I'm setting DPDK on our hardware to do some experiment, but suffering from .

I was trying to use DPDK on Ubuntu 18.04 with Intel XL710 Ethernet Adapter
with igb_uio driver. I've compiled DPDK from source, and bound the driver
to NIC.
But when I run `testpmd` application, it returns an error like this.

```
$ sudo ./build/app/testpmd
EAL: Detected 128 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:81:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1583 net_i40e
eth_i40e_dev_init(): Failed to init adminq: -66
EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.0 cannot
be used
EAL: PCI device 0000:81:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1583 net_i40e
eth_i40e_dev_init(): Failed to init adminq: -66
EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.1 cannot
be used
EAL: PCI device 0000:c4:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1533 net_e1000_igb
EAL: PCI device 0000:c5:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1533 net_e1000_igb
testpmd: No probed ethernet devices
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=1163456, size=2176,
socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support
enabled, MP allocation mode: native

  io packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=0
Press enter to exit

Telling cores to stop...
Waiting for lcores to finish...

  +++++++++++++++ Accumulated forward statistics for all
ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Bye...
```
I tried to find what return code -66 from adminq means, but I couldn't find
it.
Some say this might have to do with Intel IOMMU pass-through, but I'm not
sure that's the case.

Has anyone had this issue before? Maybe some tip on this issue would be
very helpful.

Thank you,
Sunghyuk

Re: [dpdk-users] "eth_i40e_dev_init(): Failed to init adminq: -66" problem

[Sunghyuk Kay]

Hi,
Thanks for the solution.

Actually, I was trying with igb_uio driver, rather than vfio-pci.
Would that make a difference?

Thank you,
Sunghyuk

2020년 4월 10일 (금) 오후 9:54, Dheeraj Dang <dheerajdang138@gmail.com>님이 작성:

> Hi,
>
> I think you are using virtual functions while running your test. If yes,
> please try the following commands
>
> Make changes in GRUB setting, update grub and reboot the machine.
> GRUB_CMDLINE_LINUX="intel_iommu=on amd_iommu=on"
> When machine is up, check that iommu is enabled in the output of cat
> /proc/cmdline. If iommu is enabled, then run the following commands.
> modprobe vfio enable_unsafe_noiommu_mode=Y modprobe vfio-pci
>
> Bind your dpdk interfaces to vfio-pci (*dpdk-devbind.py -b vfio-pci
> <PCI_ID>*) and run the test.
>
> Regards
> Dheeraj Dang
>
>
> On Fri, Apr 10, 2020 at 6:16 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr>
> wrote:
>
>> Hi all,
>> I'm setting DPDK on our hardware to do some experiment, but suffering
>> from .
>>
>> I was trying to use DPDK on Ubuntu 18.04 with Intel XL710 Ethernet Adapter
>> with igb_uio driver. I've compiled DPDK from source, and bound the driver
>> to NIC.
>> But when I run `testpmd` application, it returns an error like this.
>>
>> ```
>> $ sudo ./build/app/testpmd
>> EAL: Detected 128 lcore(s)
>> EAL: Detected 1 NUMA nodes
>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> EAL: Selected IOVA mode 'PA'
>> EAL: No available hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> EAL: PCI device 0000:81:00.0 on NUMA socket 0
>> EAL:   probe driver: 8086:1583 net_i40e
>> eth_i40e_dev_init(): Failed to init adminq: -66
>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.0 cannot
>> be used
>> EAL: PCI device 0000:81:00.1 on NUMA socket 0
>> EAL:   probe driver: 8086:1583 net_i40e
>> eth_i40e_dev_init(): Failed to init adminq: -66
>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.1 cannot
>> be used
>> EAL: PCI device 0000:c4:00.0 on NUMA socket 0
>> EAL:   probe driver: 8086:1533 net_e1000_igb
>> EAL: PCI device 0000:c5:00.0 on NUMA socket 0
>> EAL:   probe driver: 8086:1533 net_e1000_igb
>> testpmd: No probed ethernet devices
>> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=1163456,
>> size=2176,
>> socket=0
>> testpmd: preferred mempool ops selected: ring_mp_mc
>> Done
>> No commandline core given, start packet forwarding
>> io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support
>> enabled, MP allocation mode: native
>>
>>   io packet forwarding packets/burst=32
>>   nb forwarding cores=1 - nb forwarding ports=0
>> Press enter to exit
>>
>> Telling cores to stop...
>> Waiting for lcores to finish...
>>
>>   +++++++++++++++ Accumulated forward statistics for all
>> ports+++++++++++++++
>>   RX-packets: 0              RX-dropped: 0             RX-total: 0
>>   TX-packets: 0              TX-dropped: 0             TX-total: 0
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Done.
>>
>> Bye...
>> ```
>> I tried to find what return code -66 from adminq means, but I couldn't
>> find
>> it.
>> Some say this might have to do with Intel IOMMU pass-through, but I'm not
>> sure that's the case.
>>
>> Has anyone had this issue before? Maybe some tip on this issue would be
>> very helpful.
>>
>> Thank you,
>> Sunghyuk
>>
> --
감사합니다
계성혁 드림
=========
계성혁 (Kay, Sung Hyuk)
KAIST Concurrency and Parallelism Laboratory
Masters Student @ Software Graduate Program
Mobile: 010-2794-6906

Re: [dpdk-users] "eth_i40e_dev_init(): Failed to init adminq: -66" problem

[Dheeraj Dang]

Hi,

If you want to use igb_uio instead of vfio-pci, then you have to make the
following changes in the GRUB
GRUB_CMDLINE_LINUX="iommu=pt, intel_iommu=on"

update grub settings and reboot the machine.

When we are running DPDK on virtual functions, then it is recommended to
use vfio-pci instead of igb_uio.
You can refer this link
http://mails.dpdk.org/archives/dev/2014-August/004610.html for more details.

Regards
Dheeraj Dang


On Fri, Apr 10, 2020 at 7:01 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr>
wrote:

> Hi,
> Thanks for the solution.
>
> Actually, I was trying with igb_uio driver, rather than vfio-pci.
> Would that make a difference?
>
> Thank you,
> Sunghyuk
>
> 2020년 4월 10일 (금) 오후 9:54, Dheeraj Dang <dheerajdang138@gmail.com>님이 작성:
>
>> Hi,
>>
>> I think you are using virtual functions while running your test. If yes,
>> please try the following commands
>>
>> Make changes in GRUB setting, update grub and reboot the machine.
>> GRUB_CMDLINE_LINUX="intel_iommu=on amd_iommu=on"
>> When machine is up, check that iommu is enabled in the output of cat
>> /proc/cmdline. If iommu is enabled, then run the following commands.
>> modprobe vfio enable_unsafe_noiommu_mode=Y modprobe vfio-pci
>>
>> Bind your dpdk interfaces to vfio-pci (*dpdk-devbind.py -b vfio-pci
>> <PCI_ID>*) and run the test.
>>
>> Regards
>> Dheeraj Dang
>>
>>
>> On Fri, Apr 10, 2020 at 6:16 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr>
>> wrote:
>>
>>> Hi all,
>>> I'm setting DPDK on our hardware to do some experiment, but suffering
>>> from .
>>>
>>> I was trying to use DPDK on Ubuntu 18.04 with Intel XL710 Ethernet
>>> Adapter
>>> with igb_uio driver. I've compiled DPDK from source, and bound the driver
>>> to NIC.
>>> But when I run `testpmd` application, it returns an error like this.
>>>
>>> ```
>>> $ sudo ./build/app/testpmd
>>> EAL: Detected 128 lcore(s)
>>> EAL: Detected 1 NUMA nodes
>>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>>> EAL: Selected IOVA mode 'PA'
>>> EAL: No available hugepages reported in hugepages-1048576kB
>>> EAL: Probing VFIO support...
>>> EAL: PCI device 0000:81:00.0 on NUMA socket 0
>>> EAL:   probe driver: 8086:1583 net_i40e
>>> eth_i40e_dev_init(): Failed to init adminq: -66
>>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.0
>>> cannot
>>> be used
>>> EAL: PCI device 0000:81:00.1 on NUMA socket 0
>>> EAL:   probe driver: 8086:1583 net_i40e
>>> eth_i40e_dev_init(): Failed to init adminq: -66
>>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.1
>>> cannot
>>> be used
>>> EAL: PCI device 0000:c4:00.0 on NUMA socket 0
>>> EAL:   probe driver: 8086:1533 net_e1000_igb
>>> EAL: PCI device 0000:c5:00.0 on NUMA socket 0
>>> EAL:   probe driver: 8086:1533 net_e1000_igb
>>> testpmd: No probed ethernet devices
>>> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=1163456,
>>> size=2176,
>>> socket=0
>>> testpmd: preferred mempool ops selected: ring_mp_mc
>>> Done
>>> No commandline core given, start packet forwarding
>>> io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support
>>> enabled, MP allocation mode: native
>>>
>>>   io packet forwarding packets/burst=32
>>>   nb forwarding cores=1 - nb forwarding ports=0
>>> Press enter to exit
>>>
>>> Telling cores to stop...
>>> Waiting for lcores to finish...
>>>
>>>   +++++++++++++++ Accumulated forward statistics for all
>>> ports+++++++++++++++
>>>   RX-packets: 0              RX-dropped: 0             RX-total: 0
>>>   TX-packets: 0              TX-dropped: 0             TX-total: 0
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>> Done.
>>>
>>> Bye...
>>> ```
>>> I tried to find what return code -66 from adminq means, but I couldn't
>>> find
>>> it.
>>> Some say this might have to do with Intel IOMMU pass-through, but I'm not
>>> sure that's the case.
>>>
>>> Has anyone had this issue before? Maybe some tip on this issue would be
>>> very helpful.
>>>
>>> Thank you,
>>> Sunghyuk
>>>
>> --
> 감사합니다
> 계성혁 드림
> =========
> 계성혁 (Kay, Sung Hyuk)
> KAIST Concurrency and Parallelism Laboratory
> Masters Student @ Software Graduate Program
> Mobile: 010-2794-6906
>

Re: [dpdk-users] "eth_i40e_dev_init(): Failed to init adminq: -66" problem

[Sunghyuk Kay]

Hi, sorry for the late reply.

I had done more digging on the message, and I guess the issue was because
of NIC's low firmware version.
(I found IO_PAGE_FAULT on dmesg)

Since I didn't want to screw up with the current environment (including
firmware), I managed to run DPDK on other hardware with your advice on GRUB
settings, so I might be able to see what my real problem was later in the
future :)

Thanks for the kind response.

Regards,
Sunghyuk


On Fri, Apr 10, 2020 at 11:30 PM Dheeraj Dang <dheerajdang138@gmail.com>
wrote:

> Hi,
>
> If you want to use igb_uio instead of vfio-pci, then you have to make the
> following changes in the GRUB
> GRUB_CMDLINE_LINUX="iommu=pt, intel_iommu=on"
>
> update grub settings and reboot the machine.
>
> When we are running DPDK on virtual functions, then it is recommended to
> use vfio-pci instead of igb_uio.
> You can refer this link
> http://mails.dpdk.org/archives/dev/2014-August/004610.html for more
> details.
>
> Regards
> Dheeraj Dang
>
>
> On Fri, Apr 10, 2020 at 7:01 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr>
> wrote:
>
>> Hi,
>> Thanks for the solution.
>>
>> Actually, I was trying with igb_uio driver, rather than vfio-pci.
>> Would that make a difference?
>>
>> Thank you,
>> Sunghyuk
>>
>> 2020년 4월 10일 (금) 오후 9:54, Dheeraj Dang <dheerajdang138@gmail.com>님이 작성:
>>
>>> Hi,
>>>
>>> I think you are using virtual functions while running your test. If yes,
>>> please try the following commands
>>>
>>> Make changes in GRUB setting, update grub and reboot the machine.
>>> GRUB_CMDLINE_LINUX="intel_iommu=on amd_iommu=on"
>>> When machine is up, check that iommu is enabled in the output of cat
>>> /proc/cmdline. If iommu is enabled, then run the following commands.
>>> modprobe vfio enable_unsafe_noiommu_mode=Y modprobe vfio-pci
>>>
>>> Bind your dpdk interfaces to vfio-pci (*dpdk-devbind.py -b vfio-pci
>>> <PCI_ID>*) and run the test.
>>>
>>> Regards
>>> Dheeraj Dang
>>>
>>>
>>> On Fri, Apr 10, 2020 at 6:16 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr>
>>> wrote:
>>>
>>>> Hi all,
>>>> I'm setting DPDK on our hardware to do some experiment, but suffering
>>>> from .
>>>>
>>>> I was trying to use DPDK on Ubuntu 18.04 with Intel XL710 Ethernet
>>>> Adapter
>>>> with igb_uio driver. I've compiled DPDK from source, and bound the
>>>> driver
>>>> to NIC.
>>>> But when I run `testpmd` application, it returns an error like this.
>>>>
>>>> ```
>>>> $ sudo ./build/app/testpmd
>>>> EAL: Detected 128 lcore(s)
>>>> EAL: Detected 1 NUMA nodes
>>>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>>>> EAL: Selected IOVA mode 'PA'
>>>> EAL: No available hugepages reported in hugepages-1048576kB
>>>> EAL: Probing VFIO support...
>>>> EAL: PCI device 0000:81:00.0 on NUMA socket 0
>>>> EAL:   probe driver: 8086:1583 net_i40e
>>>> eth_i40e_dev_init(): Failed to init adminq: -66
>>>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.0
>>>> cannot
>>>> be used
>>>> EAL: PCI device 0000:81:00.1 on NUMA socket 0
>>>> EAL:   probe driver: 8086:1583 net_i40e
>>>> eth_i40e_dev_init(): Failed to init adminq: -66
>>>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.1
>>>> cannot
>>>> be used
>>>> EAL: PCI device 0000:c4:00.0 on NUMA socket 0
>>>> EAL:   probe driver: 8086:1533 net_e1000_igb
>>>> EAL: PCI device 0000:c5:00.0 on NUMA socket 0
>>>> EAL:   probe driver: 8086:1533 net_e1000_igb
>>>> testpmd: No probed ethernet devices
>>>> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=1163456,
>>>> size=2176,
>>>> socket=0
>>>> testpmd: preferred mempool ops selected: ring_mp_mc
>>>> Done
>>>> No commandline core given, start packet forwarding
>>>> io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support
>>>> enabled, MP allocation mode: native
>>>>
>>>>   io packet forwarding packets/burst=32
>>>>   nb forwarding cores=1 - nb forwarding ports=0
>>>> Press enter to exit
>>>>
>>>> Telling cores to stop...
>>>> Waiting for lcores to finish...
>>>>
>>>>   +++++++++++++++ Accumulated forward statistics for all
>>>> ports+++++++++++++++
>>>>   RX-packets: 0              RX-dropped: 0             RX-total: 0
>>>>   TX-packets: 0              TX-dropped: 0             TX-total: 0
>>>>
>>>>
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>> Done.
>>>>
>>>> Bye...
>>>> ```
>>>> I tried to find what return code -66 from adminq means, but I couldn't
>>>> find
>>>> it.
>>>> Some say this might have to do with Intel IOMMU pass-through, but I'm
>>>> not
>>>> sure that's the case.
>>>>
>>>> Has anyone had this issue before? Maybe some tip on this issue would be
>>>> very helpful.
>>>>
>>>> Thank you,
>>>> Sunghyuk
>>>>
>>> --
>> 감사합니다
>> 계성혁 드림
>> =========
>> 계성혁 (Kay, Sung Hyuk)
>> KAIST Concurrency and Parallelism Laboratory
>> Masters Student @ Software Graduate Program
>> Mobile: 010-2794-6906
>>
>