[dpdk-users] Unable to bind intel NIC 82599ES to vfio-pci, uio_pci_generic

[dpdk-users] Unable to bind intel NIC 82599ES to vfio-pci, uio_pci_generic

[Shyam Shrivastav]

Hi All

I am not able to bind 82599 to either uio_pci_generic or vfio-pci
successfully. Any help greatly appreciated, I am completely stuck at this
initial step.

*1) uio_pci_generic* : tools/dpdk-devbind.py script reports success but it
is not detected by EAL on initialisation, still ixgbe driver is detected.
here is relevant console output

[root@unassigned dpdk-16.11]# tools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
drv=uio_pci_generic unused=ixgbe,vfio-pci
0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
drv=uio_pci_generic unused=ixgbe,vfio-pci

[root@unassigned dpdk-16.11]# examples/l2fwd/build/l2fwd  -c f -n 2 p ffff
EAL: Detected 8 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1533 net_e1000_igb
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1533 net_e1000_igb
MAC updating enabled
Skipping disabled port 0
Skipping disabled port 1
EAL: Error - exiting with code: 1
  Cause: All available ports are disabled. Please set portmask.



*2) vfio-pci :* Configured vfio permissions using setup.sh, bind script
reports error in this case as under

[root@unassigned dpdk-16.11]# tools/dpdk-devbind.py --force -b vfio-pci
01:00.0 01:00.1
Error: bind failed for 0000:01:00.0 - Cannot bind to driver vfio-pci
Error: bind failed for 0000:01:00.1 - Cannot bind to driver vfio-pci

Here is relevant configuration for this to work

[root@unassigned dpdk-16.11]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.6.1.el7.x86_64
root=UUID=814b781b-1955-4b34-81d6-5c5360325c2e ro crashkernel=auto rhgb
quiet intel_iommu=on iommu=pt
[root@unassigned dpdk-16.11]# dmesg|grep IOMMU
[    0.000000] DMAR: IOMMU enabled
[    0.027925] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
[root@unassigned dpdk-16.11]# ulimit -l
unlimited
[root@unassigned dpdk-16.11]# lsmod|grep pci
vfio_pci               37039  0
vfio                   26136  2 vfio_iommu_type1,vfio_pci
irqbypass              13503  2 kvm,vfio_pci
uio_pci_generic        12588  0
uio                    19259  1 uio_pci_generic


Thanks
Shyam

Re: [dpdk-users] Unable to bind intel NIC 82599ES to vfio-pci, uio_pci_generic

[Ferruh Yigit]

On 2/8/2017 7:19 AM, Shyam Shrivastav wrote:
> Hi All
> 
> I am not able to bind 82599 to either uio_pci_generic or vfio-pci
> successfully. Any help greatly appreciated, I am completely stuck at this
> initial step.
> 
> *1) uio_pci_generic* : tools/dpdk-devbind.py script reports success but it
> is not detected by EAL on initialisation, still ixgbe driver is detected.
> here is relevant console output

It is detected, but according following log ports are not enabled by
proper application (not eal) command line argument:
"   Cause: All available ports are disabled. Please set portmask."

./examples/l2fwd/build/l2fwd [EAL options] -- -p PORTMASK [-q NQ]
  -p PORTMASK: hexadecimal bitmask of ports to configure
  -q NQ: number of queue (=ports) per lcore (default is 1)
  -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to
disable, 10 default, 86400 maximum)
  --[no-]mac-updating: Enable or disable MAC addresses updating (enabled
by default)
      When enabled:
       - The source MAC address is replaced by the TX port MAC address
       - The destination MAC address is replaced by
02:00:00:00:00:TX_PORT_ID

<...>

> 
> 
> 
> *2) vfio-pci :* Configured vfio permissions using setup.sh, bind script
> reports error in this case as under

Not sure about this one, below looks correct. Perhaps dmesg can tell more.

<...>

> 
> Thanks
> Shyam
> 

Re: [dpdk-users] Unable to bind intel NIC 82599ES to vfio-pci, uio_pci_generic

[Shyam Shrivastav]

Thanks Ferruh, I figured my mistake immediately afterwards. I am able to
work with uio_pci_generic and proceeding with my project pilot for now,
have not spent much time on debugging vfio-pci issue.

Here is one more try to bind vfio-pci, includes dmesg


[root@unassigned dpdk-16.11]# dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
drv=uio_pci_generic unused=ixgbe,igb_uio,vfio-pci

Network devices using kernel driver
===================================
0000:02:00.0 'I210 Gigabit Network Connection' if=eno1 drv=igb
unused=igb_uio,vfio-pci,uio_pci_generic
0000:03:00.0 'I210 Gigabit Network Connection' if=eno2 drv=igb
unused=igb_uio,vfio-pci,uio_pci_generic

Other network devices
=====================
0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
unused=ixgbe,igb_uio,vfio-pci,uio_pci_generic

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>

[root@unassigned dpdk-16.11]# dpdk-devbind -b vfio-pci 01:00.1
Error: bind failed for 0000:01:00.1 - Cannot bind to driver vfio-pci

[root@unassigned dpdk-16.11]# dmesg | tail -n 10
[ 1989.451194] VFIO - User Level meta-driver version: 0.3
[16700.417269] VFIO - User Level meta-driver version: 0.3
[16708.620870] rte_kni: loading out-of-tree module taints kernel.
[16708.621016] rte_kni: module verification failed: signature and/or
required key missing - tainting kernel
[16762.831711] igb_uio: Use MSIX interrupt by default
[378754.897617] Intel(R) 10GbE PCI Express Linux Network Driver - version
4.5.4
[378754.897623] Copyright(c) 1999 - 2016 Intel Corporation.
[491762.064211] Intel(R) 10GbE PCI Express Linux Network Driver - version
4.5.4
[491762.064213] Copyright(c) 1999 - 2016 Intel Corporation.
[515594.211309] VFIO - User Level meta-driver version: 0.3

[root@unassigned dpdk-16.11]# ulimit -l
unlimited


dmesg for initialisation check

[root@unassigned dpdk-16.11]# dmesg | grep DMAR
[    0.000000] ACPI: DMAR 000000007d705e98 00070 (v01 INTEL      SKL
00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.027917] DMAR: Host address width 39
[    0.027918] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.027923] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
d2008c40660462 ecap f050da
[    0.027923] DMAR: RMRR base: 0x0000007d190000 end: 0x0000007d1affff
[    0.027925] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
[    0.027926] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.027927] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out
bit.
[    0.027927] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the
BIOS setting.
[    0.028288] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.511886] DMAR: No ATSR found
[    0.511942] DMAR: dmar0: Using Queued invalidation
[    0.511954] DMAR: Hardware identity mapping for device 0000:00:00.0
[    0.511955] DMAR: Hardware identity mapping for device 0000:00:01.0
[    0.511956] DMAR: Hardware identity mapping for device 0000:00:14.0
[    0.511957] DMAR: Hardware identity mapping for device 0000:00:14.2
[    0.511958] DMAR: Hardware identity mapping for device 0000:00:16.0
[    0.511959] DMAR: Hardware identity mapping for device 0000:00:17.0
[    0.511959] DMAR: Hardware identity mapping for device 0000:00:1d.0
[    0.511960] DMAR: Hardware identity mapping for device 0000:00:1d.1
[    0.511961] DMAR: Hardware identity mapping for device 0000:00:1d.2
[    0.511962] DMAR: Hardware identity mapping for device 0000:00:1f.0
[    0.511963] DMAR: Hardware identity mapping for device 0000:00:1f.2
[    0.511964] DMAR: Hardware identity mapping for device 0000:00:1f.4
[    0.511965] DMAR: Hardware identity mapping for device 0000:01:00.0
[    0.511966] DMAR: Hardware identity mapping for device 0000:01:00.1
[    0.511969] DMAR: Hardware identity mapping for device 0000:02:00.0
[    0.511971] DMAR: Hardware identity mapping for device 0000:03:00.0
[    0.511972] DMAR: Setting RMRR:
[    0.511973] DMAR: Ignoring identity map for HW passthrough device
0000:00:14.0 [0x7d190000 - 0x7d1affff]
[    0.511974] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.511975] DMAR: Ignoring identity map for HW passthrough device
0000:00:1f.0 [0x0 - 0xffffff]
[    0.511976] DMAR: Intel(R) Virtualization Technology for Directed I/O
[root@unassigned dpdk-16.11]#





On Mon, Feb 13, 2017 at 10:31 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 2/8/2017 7:19 AM, Shyam Shrivastav wrote:
> > Hi All
> >
> > I am not able to bind 82599 to either uio_pci_generic or vfio-pci
> > successfully. Any help greatly appreciated, I am completely stuck at this
> > initial step.
> >
> > *1) uio_pci_generic* : tools/dpdk-devbind.py script reports success but
> it
> > is not detected by EAL on initialisation, still ixgbe driver is detected.
> > here is relevant console output
>
> It is detected, but according following log ports are not enabled by
> proper application (not eal) command line argument:
> "   Cause: All available ports are disabled. Please set portmask."
>
> ./examples/l2fwd/build/l2fwd [EAL options] -- -p PORTMASK [-q NQ]
>   -p PORTMASK: hexadecimal bitmask of ports to configure
>   -q NQ: number of queue (=ports) per lcore (default is 1)
>   -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to
> disable, 10 default, 86400 maximum)
>   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled
> by default)
>       When enabled:
>        - The source MAC address is replaced by the TX port MAC address
>        - The destination MAC address is replaced by
> 02:00:00:00:00:TX_PORT_ID
>
> <...>
>
> >
> >
> >
> > *2) vfio-pci :* Configured vfio permissions using setup.sh, bind script
> > reports error in this case as under
>
> Not sure about this one, below looks correct. Perhaps dmesg can tell more.
>
> <...>
>
> >
> > Thanks
> > Shyam
> >
>
>

Re: [dpdk-users] Unable to bind intel NIC 82599ES to vfio-pci, uio_pci_generic

[Shyam Shrivastav]

Please ignore my previous messages. I was using command line options
wrongly, being my first day with dpdk, missed the "--" separating EAL and
app options, caught it only going through code !
Still I am unable to make vfio-pci work bind to ports, uio_pci_generic as
well as igb_uio are working.

Thanks and rgds
Shyam