[dpdk-dev] seeing a problem dpdk startup

[dpdk-dev] seeing a problem dpdk startup

[Venumadhav Josyula]

We are seeing following error, no device is detected....
==========
EAL: Detected 8 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1528 net_ixgbe
EAL:   0000:04:00.0 failed to select IOMMU type
EAL: Requested device 0000:04:00.0 cannot be used
EAL: PCI device 0000:04:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1528 net_ixgbe
EAL:   0000:04:00.1 failed to select IOMMU type
EAL: Requested device 0000:04:00.1 cannot be used

==================
dmesg

[55455.259493] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain
attach due to platform RMRR requirement.  Contact your platform vendor.
[55455.260947] vfio-pci 0000:04:00.1: Device is ineligible for IOMMU domain
attach due to platform RMRR requirement.  Contact your platform vendor.

======
BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.el7.pw.1.x86_64 root=/dev/sda3
intel_iommu=on


Can anybody suggest how to work around the issue ?

Thanks,
Regards,
Venu

Re: [dpdk-dev] seeing a problem dpdk startup

[Gavin Hu (Arm Technology China)]

Hi Venumadhav,

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Venumadhav Josyula
> Sent: Thursday, November 28, 2019 3:01 PM
> To: dev@dpdk.org; users@dpdk.org
> Cc: Venumadhav Josyula <vjosyula@parallelwireless.com>
> Subject: [dpdk-dev] seeing a problem dpdk startup
> 
> We are seeing following error, no device is detected....
> ==========
> EAL: Detected 8 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:04:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1528 net_ixgbe
> EAL:   0000:04:00.0 failed to select IOMMU type
> EAL: Requested device 0000:04:00.0 cannot be used
> EAL: PCI device 0000:04:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:1528 net_ixgbe
> EAL:   0000:04:00.1 failed to select IOMMU type
> EAL: Requested device 0000:04:00.1 cannot be used
> 
> ==================
> dmesg
> 
> [55455.259493] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain
> attach due to platform RMRR requirement.  Contact your platform vendor.
> [55455.260947] vfio-pci 0000:04:00.1: Device is ineligible for IOMMU domain
> attach due to platform RMRR requirement.  Contact your platform vendor.
> 
> ======
> BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.el7.pw.1.x86_64 root=/dev/sda3
> intel_iommu=on
> 
> 
> Can anybody suggest how to work around the issue ?
Did you check if the device is bound to DPDK? 
Here is an example on my host:
gavin@net-arm-thunderx2-01:~/dpdk$ ./usertools/dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:0f:00.0 'Ethernet Controller XL710 for 40GbE QSFP+ 1583' drv=vfio-pci unused=i40e
0000:8a:00.0 'Ethernet Controller XXV710 for 25GbE SFP28 158b' drv=vfio-pci unused=i40e
0000:8a:00.1 'Ethernet Controller XXV710 for 25GbE SFP28 158b' drv=vfio-pci unused=i40e
> 
> Thanks,
> Regards,
> Venu

Re: [dpdk-dev] seeing a problem dpdk startup

[Venumadhav Josyula]

Hi Gavin,

It is
[root@localhost scripts]# ./dpdk-devbind.py --status-dev net

Network devices using DPDK-compatible driver
============================================
0000:04:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci
unused=ixgbe
0000:04:00.1 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci
unused=ixgbe

Network devices using kernel driver
===================================
0000:02:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=eno1 drv=tg3
unused=vfio-pci *Active*
0000:02:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=eno2 drv=tg3
unused=vfio-pci
0000:02:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=eno3 drv=tg3
unused=vfio-pci
0000:02:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=eno4 drv=tg3
unused=vfio-pci
[root@localhost scripts]#

I understand from the that IOMMU type selection is failing, is it possible
to somehow default it to either igb_uio ?

Thanks,
Regards,
Venu

On Thu, 28 Nov 2019 at 13:09, Gavin Hu (Arm Technology China) <
Gavin.Hu@arm.com> wrote:

> Hi Venumadhav,
>
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Venumadhav Josyula
> > Sent: Thursday, November 28, 2019 3:01 PM
> > To: dev@dpdk.org; users@dpdk.org
> > Cc: Venumadhav Josyula <vjosyula@parallelwireless.com>
> > Subject: [dpdk-dev] seeing a problem dpdk startup
> >
> > We are seeing following error, no device is detected....
> > ==========
> > EAL: Detected 8 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: No free hugepages reported in hugepages-1048576kB
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:04:00.0 on NUMA socket 0
> > EAL:   probe driver: 8086:1528 net_ixgbe
> > EAL:   0000:04:00.0 failed to select IOMMU type
> > EAL: Requested device 0000:04:00.0 cannot be used
> > EAL: PCI device 0000:04:00.1 on NUMA socket 0
> > EAL:   probe driver: 8086:1528 net_ixgbe
> > EAL:   0000:04:00.1 failed to select IOMMU type
> > EAL: Requested device 0000:04:00.1 cannot be used
> >
> > ==================
> > dmesg
> >
> > [55455.259493] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU
> domain
> > attach due to platform RMRR requirement.  Contact your platform vendor.
> > [55455.260947] vfio-pci 0000:04:00.1: Device is ineligible for IOMMU
> domain
> > attach due to platform RMRR requirement.  Contact your platform vendor.
> >
> > ======
> > BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.el7.pw.1.x86_64 root=/dev/sda3
> > intel_iommu=on
> >
> >
> > Can anybody suggest how to work around the issue ?
> Did you check if the device is bound to DPDK?
> Here is an example on my host:
> gavin@net-arm-thunderx2-01:~/dpdk$ ./usertools/dpdk-devbind.py -s
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:0f:00.0 'Ethernet Controller XL710 for 40GbE QSFP+ 1583' drv=vfio-pci
> unused=i40e
> 0000:8a:00.0 'Ethernet Controller XXV710 for 25GbE SFP28 158b'
> drv=vfio-pci unused=i40e
> 0000:8a:00.1 'Ethernet Controller XXV710 for 25GbE SFP28 158b'
> drv=vfio-pci unused=i40e
> >
> > Thanks,
> > Regards,
> > Venu
>

Re: [dpdk-dev] seeing a problem dpdk startup

[Hui Wei]

>We are seeing following error, no device is detected....
>==========
>EAL: Detected 8 lcore(s)
>EAL: Detected 2 NUMA nodes
>EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>EAL: No free hugepages reported in hugepages-1048576kB
>EAL: Probing VFIO support...
>EAL: VFIO support initialized
>EAL: PCI device 0000:04:00.0 on NUMA socket 0
>EAL:   probe driver: 8086:1528 net_ixgbe
>EAL:   0000:04:00.0 failed to select IOMMU type
>EAL: Requested device 0000:04:00.0 cannot be used
>EAL: PCI device 0000:04:00.1 on NUMA socket 0
>EAL:   probe driver: 8086:1528 net_ixgbe
>EAL:   0000:04:00.1 failed to select IOMMU type
>EAL: Requested device 0000:04:00.1 cannot be used
>
>==================
>dmesg
>
>[55455.259493] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain
>attach due to platform RMRR requirement.  Contact your platform vendor.
>[55455.260947] vfio-pci 0000:04:00.1: Device is ineligible for IOMMU domain
>attach due to platform RMRR requirement.  Contact your platform vendor. 

Intel Vt-d document addresses RMRR, but not that clearly. Check BIOS setup, 
no link share, for example, a nic detect by linux kernel, it was shared by BIOS for
IPMI. If change BIOS setup don't solve your problem, contact platform vendor update 
nic firmware, a server made by HP, check HP's website for firmware.

>
>======
>BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.el7.pw.1.x86_64 root=/dev/sda3
>intel_iommu=on
>
>
>Can anybody suggest how to work around the issue ?
>
>Thanks,
>Regards,
>Venu

Re: [dpdk-dev] seeing a problem dpdk startup

[Venumadhav Josyula]

Hi Hui,

I understand that and we are exploring that.

But is there no work around there where in it defaults to igb_uio driver
something like that..... Basic idea is we should not be situation of no
port detected at all under this situation ?

Thanks,
Regards,
Venu

On Thu, 28 Nov 2019 at 13:15, Hui Wei <huiweics@hotmail.com> wrote:

>
> >We are seeing following error, no device is detected....
> >==========
> >EAL: Detected 8 lcore(s)
> >EAL: Detected 2 NUMA nodes
> >EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >EAL: No free hugepages reported in hugepages-1048576kB
> >EAL: Probing VFIO support...
> >EAL: VFIO support initialized
> >EAL: PCI device 0000:04:00.0 on NUMA socket 0
> >EAL:   probe driver: 8086:1528 net_ixgbe
> >EAL:   0000:04:00.0 failed to select IOMMU type
> >EAL: Requested device 0000:04:00.0 cannot be used
> >EAL: PCI device 0000:04:00.1 on NUMA socket 0
> >EAL:   probe driver: 8086:1528 net_ixgbe
> >EAL:   0000:04:00.1 failed to select IOMMU type
> >EAL: Requested device 0000:04:00.1 cannot be used
> >
> >==================
> >dmesg
> >
> >[55455.259493] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU
> domain
> >attach due to platform RMRR requirement.  Contact your platform vendor.
> >[55455.260947] vfio-pci 0000:04:00.1: Device is ineligible for IOMMU
> domain
> >attach due to platform RMRR requirement.  Contact your platform vendor.
>
> Intel Vt-d document addresses RMRR, but not that clearly. Check BIOS
> setup,
> no link share, for example, a nic detect by linux kernel, it was shared by
> BIOS for
> IPMI. If change BIOS setup don't solve your problem, contact platform
> vendor update
> nic firmware, a server made by HP, check HP's website for firmware.
>
> >
> >======
> >BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.el7.pw.1.x86_64 root=/dev/sda3
> >intel_iommu=on
> >
> >
> >Can anybody suggest how to work around the issue ?
> >
> >Thanks,
> >Regards,
> >Venu

Re: [dpdk-dev] seeing a problem dpdk startup

[Hui Wei]

I think  BIOS turn off vt-d, use igb_uio can work around.

>Hi Hui,
>
>I understand that and we are exploring that.
>
>But is there no work around there where in it defaults to igb_uio driver
>something like that..... Basic idea is we should not be situation of no
>port detected at all under this situation ?
>
>Thanks,
>Regards,
>Venu
>
>On Thu, 28 Nov 2019 at 13:15, Hui Wei <huiweics@hotmail.com> wrote:
>
>>
>> >We are seeing following error, no device is detected....
>> >==========
>> >EAL: Detected 8 lcore(s)
>> >EAL: Detected 2 NUMA nodes
>> >EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> >EAL: No free hugepages reported in hugepages-1048576kB
>> >EAL: Probing VFIO support...
>> >EAL: VFIO support initialized
>> >EAL: PCI device 0000:04:00.0 on NUMA socket 0
>> >EAL:   probe driver: 8086:1528 net_ixgbe
>> >EAL:   0000:04:00.0 failed to select IOMMU type
>> >EAL: Requested device 0000:04:00.0 cannot be used
>> >EAL: PCI device 0000:04:00.1 on NUMA socket 0
>> >EAL:   probe driver: 8086:1528 net_ixgbe
>> >EAL:   0000:04:00.1 failed to select IOMMU type
>> >EAL: Requested device 0000:04:00.1 cannot be used
>> >
>> >==================
>> >dmesg
>> >
>> >[55455.259493] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU
>> domain
>> >attach due to platform RMRR requirement.  Contact your platform vendor.
>> >[55455.260947] vfio-pci 0000:04:00.1: Device is ineligible for IOMMU
>> domain
>> >attach due to platform RMRR requirement.  Contact your platform vendor.
>>
>> Intel Vt-d document addresses RMRR, but not that clearly. Check BIOS
>> setup,
>> no link share, for example, a nic detect by linux kernel, it was shared by
>> BIOS for
>> IPMI. If change BIOS setup don't solve your problem, contact platform
>> vendor update
>> nic firmware, a server made by HP, check HP's website for firmware.
>>
>> >
>> >======
>> >BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.el7.pw.1.x86_64 root=/dev/sda3
>> >intel_iommu=on
>> >
>> >
>> >Can anybody suggest how to work around the issue ?
>> >
>> >Thanks,
>> >Regards,
>> >Venu

Re: [dpdk-dev] seeing a problem dpdk startup

[Roberts, Lee A.]

Please see the HPE customer advisory at https://support.hpe.com/hpsc/doc/public/display?sp4ts.oid=7271241&docId=emr_na-c04781229&docLocale=en_US.
You will need to disable the shared memory features for the NIC card.

                                 - Lee

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hui Wei
Sent: Thursday, November 28, 2019 1:01 AM
To: Venumadhav Josyula <vjosyula@gmail.com>
Cc: dev <dev@dpdk.org>; users <users@dpdk.org>; Venumadhav Josyula <vjosyula@parallelwireless.com>
Subject: Re: [dpdk-dev] seeing a problem dpdk startup

I think  BIOS turn off vt-d, use igb_uio can work around.

>Hi Hui,
>
>I understand that and we are exploring that.
>
>But is there no work around there where in it defaults to igb_uio driver
>something like that..... Basic idea is we should not be situation of no
>port detected at all under this situation ?
>
>Thanks,
>Regards,
>Venu
>
>On Thu, 28 Nov 2019 at 13:15, Hui Wei <huiweics@hotmail.com> wrote:
>
>>
>> >We are seeing following error, no device is detected....
>> >==========
>> >EAL: Detected 8 lcore(s)
>> >EAL: Detected 2 NUMA nodes
>> >EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> >EAL: No free hugepages reported in hugepages-1048576kB
>> >EAL: Probing VFIO support...
>> >EAL: VFIO support initialized
>> >EAL: PCI device 0000:04:00.0 on NUMA socket 0
>> >EAL:   probe driver: 8086:1528 net_ixgbe
>> >EAL:   0000:04:00.0 failed to select IOMMU type
>> >EAL: Requested device 0000:04:00.0 cannot be used
>> >EAL: PCI device 0000:04:00.1 on NUMA socket 0
>> >EAL:   probe driver: 8086:1528 net_ixgbe
>> >EAL:   0000:04:00.1 failed to select IOMMU type
>> >EAL: Requested device 0000:04:00.1 cannot be used
>> >
>> >==================
>> >dmesg
>> >
>> >[55455.259493] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU
>> domain
>> >attach due to platform RMRR requirement.  Contact your platform vendor.
>> >[55455.260947] vfio-pci 0000:04:00.1: Device is ineligible for IOMMU
>> domain
>> >attach due to platform RMRR requirement.  Contact your platform vendor.
>>
>> Intel Vt-d document addresses RMRR, but not that clearly. Check BIOS
>> setup,
>> no link share, for example, a nic detect by linux kernel, it was shared by
>> BIOS for
>> IPMI. If change BIOS setup don't solve your problem, contact platform
>> vendor update
>> nic firmware, a server made by HP, check HP's website for firmware.
>>
>> >
>> >======
>> >BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.el7.pw.1.x86_64 root=/dev/sda3
>> >intel_iommu=on
>> >
>> >
>> >Can anybody suggest how to work around the issue ?
>> >
>> >Thanks,
>> >Regards,
>> >Venu

Re: [dpdk-dev] seeing a problem dpdk startup

[Venumadhav Josyula]

Hi Roberts,

Thanks for your reply. Basically, our BIOS is old we do not have that
feature in our BIOS , we need to upgrade it.

But after enabling will it work, can you confirm that based on your
experience or somebody informing you success story.

Thanks,
Regards,
Venu

On Thu, 28 Nov 2019 at 19:45, Roberts, Lee A. <lee.roberts@hpe.com> wrote:

> Please see the HPE customer advisory at
> https://support.hpe.com/hpsc/doc/public/display?sp4ts.oid=7271241&docId=emr_na-c04781229&docLocale=en_US
> .
> You will need to disable the shared memory features for the NIC card.
>
>                                  - Lee
>
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hui Wei
> Sent: Thursday, November 28, 2019 1:01 AM
> To: Venumadhav Josyula <vjosyula@gmail.com>
> Cc: dev <dev@dpdk.org>; users <users@dpdk.org>; Venumadhav Josyula <
> vjosyula@parallelwireless.com>
> Subject: Re: [dpdk-dev] seeing a problem dpdk startup
>
> I think  BIOS turn off vt-d, use igb_uio can work around.
>
> >Hi Hui,
> >
> >I understand that and we are exploring that.
> >
> >But is there no work around there where in it defaults to igb_uio driver
> >something like that..... Basic idea is we should not be situation of no
> >port detected at all under this situation ?
> >
> >Thanks,
> >Regards,
> >Venu
> >
> >On Thu, 28 Nov 2019 at 13:15, Hui Wei <huiweics@hotmail.com> wrote:
> >
> >>
> >> >We are seeing following error, no device is detected....
> >> >==========
> >> >EAL: Detected 8 lcore(s)
> >> >EAL: Detected 2 NUMA nodes
> >> >EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> >> >EAL: No free hugepages reported in hugepages-1048576kB
> >> >EAL: Probing VFIO support...
> >> >EAL: VFIO support initialized
> >> >EAL: PCI device 0000:04:00.0 on NUMA socket 0
> >> >EAL:   probe driver: 8086:1528 net_ixgbe
> >> >EAL:   0000:04:00.0 failed to select IOMMU type
> >> >EAL: Requested device 0000:04:00.0 cannot be used
> >> >EAL: PCI device 0000:04:00.1 on NUMA socket 0
> >> >EAL:   probe driver: 8086:1528 net_ixgbe
> >> >EAL:   0000:04:00.1 failed to select IOMMU type
> >> >EAL: Requested device 0000:04:00.1 cannot be used
> >> >
> >> >==================
> >> >dmesg
> >> >
> >> >[55455.259493] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU
> >> domain
> >> >attach due to platform RMRR requirement.  Contact your platform vendor.
> >> >[55455.260947] vfio-pci 0000:04:00.1: Device is ineligible for IOMMU
> >> domain
> >> >attach due to platform RMRR requirement.  Contact your platform vendor.
> >>
> >> Intel Vt-d document addresses RMRR, but not that clearly. Check BIOS
> >> setup,
> >> no link share, for example, a nic detect by linux kernel, it was shared
> by
> >> BIOS for
> >> IPMI. If change BIOS setup don't solve your problem, contact platform
> >> vendor update
> >> nic firmware, a server made by HP, check HP's website for firmware.
> >>
> >> >
> >> >======
> >> >BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.el7.pw.1.x86_64 root=/dev/sda3
> >> >intel_iommu=on
> >> >
> >> >
> >> >Can anybody suggest how to work around the issue ?
> >> >
> >> >Thanks,
> >> >Regards,
> >> >Venu
>

Re: [dpdk-dev] seeing a problem dpdk startup

[Stephen Hemminger]

On Thu, 28 Nov 2019 19:50:41 +0530
Venumadhav Josyula <vjosyula@gmail.com> wrote:

> Hi Roberts,
> 
> Thanks for your reply. Basically, our BIOS is old we do not have that
> feature in our BIOS , we need to upgrade it.
> 
> But after enabling will it work, can you confirm that based on your
> experience or somebody informing you success story.
> 
> Thanks,
> Regards,
> Venu

This may help explain the issues.
https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf