Hi all, I'm setting DPDK on our hardware to do some experiment, but suffering from . I was trying to use DPDK on Ubuntu 18.04 with Intel XL710 Ethernet Adapter with igb_uio driver. I've compiled DPDK from source, and bound the driver to NIC. But when I run `testpmd` application, it returns an error like this. ``` $ sudo ./build/app/testpmd EAL: Detected 128 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: PCI device 0000:81:00.0 on NUMA socket 0 EAL: probe driver: 8086:1583 net_i40e eth_i40e_dev_init(): Failed to init adminq: -66 EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.0 cannot be used EAL: PCI device 0000:81:00.1 on NUMA socket 0 EAL: probe driver: 8086:1583 net_i40e eth_i40e_dev_init(): Failed to init adminq: -66 EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.1 cannot be used EAL: PCI device 0000:c4:00.0 on NUMA socket 0 EAL: probe driver: 8086:1533 net_e1000_igb EAL: PCI device 0000:c5:00.0 on NUMA socket 0 EAL: probe driver: 8086:1533 net_e1000_igb testpmd: No probed ethernet devices testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=1163456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Done No commandline core given, start packet forwarding io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support enabled, MP allocation mode: native io packet forwarding packets/burst=32 nb forwarding cores=1 - nb forwarding ports=0 Press enter to exit Telling cores to stop... Waiting for lcores to finish... +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 0 TX-dropped: 0 TX-total: 0 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Done. Bye... ``` I tried to find what return code -66 from adminq means, but I couldn't find it. Some say this might have to do with Intel IOMMU pass-through, but I'm not sure that's the case. Has anyone had this issue before? Maybe some tip on this issue would be very helpful. Thank you, Sunghyuk
Hi,
Thanks for the solution.
Actually, I was trying with igb_uio driver, rather than vfio-pci.
Would that make a difference?
Thank you,
Sunghyuk
2020년 4월 10일 (금) 오후 9:54, Dheeraj Dang <dheerajdang138@gmail.com>님이 작성:
> Hi,
>
> I think you are using virtual functions while running your test. If yes,
> please try the following commands
>
> Make changes in GRUB setting, update grub and reboot the machine.
> GRUB_CMDLINE_LINUX="intel_iommu=on amd_iommu=on"
> When machine is up, check that iommu is enabled in the output of cat
> /proc/cmdline. If iommu is enabled, then run the following commands.
> modprobe vfio enable_unsafe_noiommu_mode=Y modprobe vfio-pci
>
> Bind your dpdk interfaces to vfio-pci (*dpdk-devbind.py -b vfio-pci
> <PCI_ID>*) and run the test.
>
> Regards
> Dheeraj Dang
>
>
> On Fri, Apr 10, 2020 at 6:16 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr>
> wrote:
>
>> Hi all,
>> I'm setting DPDK on our hardware to do some experiment, but suffering
>> from .
>>
>> I was trying to use DPDK on Ubuntu 18.04 with Intel XL710 Ethernet Adapter
>> with igb_uio driver. I've compiled DPDK from source, and bound the driver
>> to NIC.
>> But when I run `testpmd` application, it returns an error like this.
>>
>> ```
>> $ sudo ./build/app/testpmd
>> EAL: Detected 128 lcore(s)
>> EAL: Detected 1 NUMA nodes
>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> EAL: Selected IOVA mode 'PA'
>> EAL: No available hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> EAL: PCI device 0000:81:00.0 on NUMA socket 0
>> EAL: probe driver: 8086:1583 net_i40e
>> eth_i40e_dev_init(): Failed to init adminq: -66
>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.0 cannot
>> be used
>> EAL: PCI device 0000:81:00.1 on NUMA socket 0
>> EAL: probe driver: 8086:1583 net_i40e
>> eth_i40e_dev_init(): Failed to init adminq: -66
>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.1 cannot
>> be used
>> EAL: PCI device 0000:c4:00.0 on NUMA socket 0
>> EAL: probe driver: 8086:1533 net_e1000_igb
>> EAL: PCI device 0000:c5:00.0 on NUMA socket 0
>> EAL: probe driver: 8086:1533 net_e1000_igb
>> testpmd: No probed ethernet devices
>> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=1163456,
>> size=2176,
>> socket=0
>> testpmd: preferred mempool ops selected: ring_mp_mc
>> Done
>> No commandline core given, start packet forwarding
>> io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support
>> enabled, MP allocation mode: native
>>
>> io packet forwarding packets/burst=32
>> nb forwarding cores=1 - nb forwarding ports=0
>> Press enter to exit
>>
>> Telling cores to stop...
>> Waiting for lcores to finish...
>>
>> +++++++++++++++ Accumulated forward statistics for all
>> ports+++++++++++++++
>> RX-packets: 0 RX-dropped: 0 RX-total: 0
>> TX-packets: 0 TX-dropped: 0 TX-total: 0
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Done.
>>
>> Bye...
>> ```
>> I tried to find what return code -66 from adminq means, but I couldn't
>> find
>> it.
>> Some say this might have to do with Intel IOMMU pass-through, but I'm not
>> sure that's the case.
>>
>> Has anyone had this issue before? Maybe some tip on this issue would be
>> very helpful.
>>
>> Thank you,
>> Sunghyuk
>>
> --
감사합니다
계성혁 드림
=========
계성혁 (Kay, Sung Hyuk)
KAIST Concurrency and Parallelism Laboratory
Masters Student @ Software Graduate Program
Mobile: 010-2794-6906
Hi, If you want to use igb_uio instead of vfio-pci, then you have to make the following changes in the GRUB GRUB_CMDLINE_LINUX="iommu=pt, intel_iommu=on" update grub settings and reboot the machine. When we are running DPDK on virtual functions, then it is recommended to use vfio-pci instead of igb_uio. You can refer this link http://mails.dpdk.org/archives/dev/2014-August/004610.html for more details. Regards Dheeraj Dang On Fri, Apr 10, 2020 at 7:01 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr> wrote: > Hi, > Thanks for the solution. > > Actually, I was trying with igb_uio driver, rather than vfio-pci. > Would that make a difference? > > Thank you, > Sunghyuk > > 2020년 4월 10일 (금) 오후 9:54, Dheeraj Dang <dheerajdang138@gmail.com>님이 작성: > >> Hi, >> >> I think you are using virtual functions while running your test. If yes, >> please try the following commands >> >> Make changes in GRUB setting, update grub and reboot the machine. >> GRUB_CMDLINE_LINUX="intel_iommu=on amd_iommu=on" >> When machine is up, check that iommu is enabled in the output of cat >> /proc/cmdline. If iommu is enabled, then run the following commands. >> modprobe vfio enable_unsafe_noiommu_mode=Y modprobe vfio-pci >> >> Bind your dpdk interfaces to vfio-pci (*dpdk-devbind.py -b vfio-pci >> <PCI_ID>*) and run the test. >> >> Regards >> Dheeraj Dang >> >> >> On Fri, Apr 10, 2020 at 6:16 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr> >> wrote: >> >>> Hi all, >>> I'm setting DPDK on our hardware to do some experiment, but suffering >>> from . >>> >>> I was trying to use DPDK on Ubuntu 18.04 with Intel XL710 Ethernet >>> Adapter >>> with igb_uio driver. I've compiled DPDK from source, and bound the driver >>> to NIC. >>> But when I run `testpmd` application, it returns an error like this. >>> >>> ``` >>> $ sudo ./build/app/testpmd >>> EAL: Detected 128 lcore(s) >>> EAL: Detected 1 NUMA nodes >>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket >>> EAL: Selected IOVA mode 'PA' >>> EAL: No available hugepages reported in hugepages-1048576kB >>> EAL: Probing VFIO support... >>> EAL: PCI device 0000:81:00.0 on NUMA socket 0 >>> EAL: probe driver: 8086:1583 net_i40e >>> eth_i40e_dev_init(): Failed to init adminq: -66 >>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.0 >>> cannot >>> be used >>> EAL: PCI device 0000:81:00.1 on NUMA socket 0 >>> EAL: probe driver: 8086:1583 net_i40e >>> eth_i40e_dev_init(): Failed to init adminq: -66 >>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.1 >>> cannot >>> be used >>> EAL: PCI device 0000:c4:00.0 on NUMA socket 0 >>> EAL: probe driver: 8086:1533 net_e1000_igb >>> EAL: PCI device 0000:c5:00.0 on NUMA socket 0 >>> EAL: probe driver: 8086:1533 net_e1000_igb >>> testpmd: No probed ethernet devices >>> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=1163456, >>> size=2176, >>> socket=0 >>> testpmd: preferred mempool ops selected: ring_mp_mc >>> Done >>> No commandline core given, start packet forwarding >>> io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support >>> enabled, MP allocation mode: native >>> >>> io packet forwarding packets/burst=32 >>> nb forwarding cores=1 - nb forwarding ports=0 >>> Press enter to exit >>> >>> Telling cores to stop... >>> Waiting for lcores to finish... >>> >>> +++++++++++++++ Accumulated forward statistics for all >>> ports+++++++++++++++ >>> RX-packets: 0 RX-dropped: 0 RX-total: 0 >>> TX-packets: 0 TX-dropped: 0 TX-total: 0 >>> >>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>> Done. >>> >>> Bye... >>> ``` >>> I tried to find what return code -66 from adminq means, but I couldn't >>> find >>> it. >>> Some say this might have to do with Intel IOMMU pass-through, but I'm not >>> sure that's the case. >>> >>> Has anyone had this issue before? Maybe some tip on this issue would be >>> very helpful. >>> >>> Thank you, >>> Sunghyuk >>> >> -- > 감사합니다 > 계성혁 드림 > ========= > 계성혁 (Kay, Sung Hyuk) > KAIST Concurrency and Parallelism Laboratory > Masters Student @ Software Graduate Program > Mobile: 010-2794-6906 >
Hi, sorry for the late reply.
I had done more digging on the message, and I guess the issue was because
of NIC's low firmware version.
(I found IO_PAGE_FAULT on dmesg)
Since I didn't want to screw up with the current environment (including
firmware), I managed to run DPDK on other hardware with your advice on GRUB
settings, so I might be able to see what my real problem was later in the
future :)
Thanks for the kind response.
Regards,
Sunghyuk
On Fri, Apr 10, 2020 at 11:30 PM Dheeraj Dang <dheerajdang138@gmail.com>
wrote:
> Hi,
>
> If you want to use igb_uio instead of vfio-pci, then you have to make the
> following changes in the GRUB
> GRUB_CMDLINE_LINUX="iommu=pt, intel_iommu=on"
>
> update grub settings and reboot the machine.
>
> When we are running DPDK on virtual functions, then it is recommended to
> use vfio-pci instead of igb_uio.
> You can refer this link
> http://mails.dpdk.org/archives/dev/2014-August/004610.html for more
> details.
>
> Regards
> Dheeraj Dang
>
>
> On Fri, Apr 10, 2020 at 7:01 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr>
> wrote:
>
>> Hi,
>> Thanks for the solution.
>>
>> Actually, I was trying with igb_uio driver, rather than vfio-pci.
>> Would that make a difference?
>>
>> Thank you,
>> Sunghyuk
>>
>> 2020년 4월 10일 (금) 오후 9:54, Dheeraj Dang <dheerajdang138@gmail.com>님이 작성:
>>
>>> Hi,
>>>
>>> I think you are using virtual functions while running your test. If yes,
>>> please try the following commands
>>>
>>> Make changes in GRUB setting, update grub and reboot the machine.
>>> GRUB_CMDLINE_LINUX="intel_iommu=on amd_iommu=on"
>>> When machine is up, check that iommu is enabled in the output of cat
>>> /proc/cmdline. If iommu is enabled, then run the following commands.
>>> modprobe vfio enable_unsafe_noiommu_mode=Y modprobe vfio-pci
>>>
>>> Bind your dpdk interfaces to vfio-pci (*dpdk-devbind.py -b vfio-pci
>>> <PCI_ID>*) and run the test.
>>>
>>> Regards
>>> Dheeraj Dang
>>>
>>>
>>> On Fri, Apr 10, 2020 at 6:16 PM Sunghyuk Kay <sunghyuk.kay@kaist.ac.kr>
>>> wrote:
>>>
>>>> Hi all,
>>>> I'm setting DPDK on our hardware to do some experiment, but suffering
>>>> from .
>>>>
>>>> I was trying to use DPDK on Ubuntu 18.04 with Intel XL710 Ethernet
>>>> Adapter
>>>> with igb_uio driver. I've compiled DPDK from source, and bound the
>>>> driver
>>>> to NIC.
>>>> But when I run `testpmd` application, it returns an error like this.
>>>>
>>>> ```
>>>> $ sudo ./build/app/testpmd
>>>> EAL: Detected 128 lcore(s)
>>>> EAL: Detected 1 NUMA nodes
>>>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>>>> EAL: Selected IOVA mode 'PA'
>>>> EAL: No available hugepages reported in hugepages-1048576kB
>>>> EAL: Probing VFIO support...
>>>> EAL: PCI device 0000:81:00.0 on NUMA socket 0
>>>> EAL: probe driver: 8086:1583 net_i40e
>>>> eth_i40e_dev_init(): Failed to init adminq: -66
>>>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.0
>>>> cannot
>>>> be used
>>>> EAL: PCI device 0000:81:00.1 on NUMA socket 0
>>>> EAL: probe driver: 8086:1583 net_i40e
>>>> eth_i40e_dev_init(): Failed to init adminq: -66
>>>> EAL: ethdev initialisation failedEAL: Requested device 0000:81:00.1
>>>> cannot
>>>> be used
>>>> EAL: PCI device 0000:c4:00.0 on NUMA socket 0
>>>> EAL: probe driver: 8086:1533 net_e1000_igb
>>>> EAL: PCI device 0000:c5:00.0 on NUMA socket 0
>>>> EAL: probe driver: 8086:1533 net_e1000_igb
>>>> testpmd: No probed ethernet devices
>>>> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=1163456,
>>>> size=2176,
>>>> socket=0
>>>> testpmd: preferred mempool ops selected: ring_mp_mc
>>>> Done
>>>> No commandline core given, start packet forwarding
>>>> io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support
>>>> enabled, MP allocation mode: native
>>>>
>>>> io packet forwarding packets/burst=32
>>>> nb forwarding cores=1 - nb forwarding ports=0
>>>> Press enter to exit
>>>>
>>>> Telling cores to stop...
>>>> Waiting for lcores to finish...
>>>>
>>>> +++++++++++++++ Accumulated forward statistics for all
>>>> ports+++++++++++++++
>>>> RX-packets: 0 RX-dropped: 0 RX-total: 0
>>>> TX-packets: 0 TX-dropped: 0 TX-total: 0
>>>>
>>>>
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>> Done.
>>>>
>>>> Bye...
>>>> ```
>>>> I tried to find what return code -66 from adminq means, but I couldn't
>>>> find
>>>> it.
>>>> Some say this might have to do with Intel IOMMU pass-through, but I'm
>>>> not
>>>> sure that's the case.
>>>>
>>>> Has anyone had this issue before? Maybe some tip on this issue would be
>>>> very helpful.
>>>>
>>>> Thank you,
>>>> Sunghyuk
>>>>
>>> --
>> 감사합니다
>> 계성혁 드림
>> =========
>> 계성혁 (Kay, Sung Hyuk)
>> KAIST Concurrency and Parallelism Laboratory
>> Masters Student @ Software Graduate Program
>> Mobile: 010-2794-6906
>>
>