[dpdk-dev] vfio detection

[dpdk-dev] vfio detection

[Richardson, Bruce]

Hi all,

Now that vfio support has merged into the dpdk.org mainline, I'm seeing a number of issues with running applications, and I'm hoping someone can explain the observed behaviour.

I have a number of NIC ports which were working correctly yesterday and are bound correctly to the igb_uio driver - and I want to keep using them through the igb_uio driver for now, not vfio. However, whenever I run a dpdk application today, I find that the vfio kernel module is getting loaded each time - even after I manually remove it, and verify that it has been removed by checking lsmod. Is this expected? If so, why are we loading the vfio driver when I just want to continue using igb_uio which works fine?
Secondly, then, when testpmd or any other app loads, it automatically tries to map the NIC using vfio and then aborts on the very first NIC port when it fails to do so. This a) prevents the port from being mapped using igb_uio, and b) for ports which are meant to stay under linux control, forces me to start enumerating ports using blacklist or whitelisting, rather than having things "just work" on a properly configured system as before, i.e. if a port is bound to igb_uio or vfio it is used, if not bound, it is ignored. Again, is this by design and expected, because it seems a major regression in usability?

Regards,
/Bruce

Re: [dpdk-dev] vfio detection

[Burakov, Anatoly]

Hi Bruce,

> I have a number of NIC ports which were working correctly yesterday and are
> bound correctly to the igb_uio driver - and I want to keep using them
> through the igb_uio driver for now, not vfio. However, whenever I run a
> dpdk application today, I find that the vfio kernel module is getting loaded
> each time - even after I manually remove it, and verify that it has been
> removed by checking lsmod. Is this expected? If so, why are we loading the
> vfio driver when I just want to continue using igb_uio which works fine?

Can you elaborate a bit on what do you mean by "loading vfio driver"? Do you mean the vfio-pci kernel gets loaded by DPDK? I certainly didn't put in any code that would automatically load that driver, and certainly not binding devices to it.

> Secondly, then, when testpmd or any other app loads, it automatically tries
> to map the NIC using vfio and then aborts on the very first NIC port when it
> fails to do so.

This shouldn't happen, unless you have a device bound to VFIO and have another device in the same IOMMU group that is bound to something else. Can you provide a log of what you are seeing?

> This a) prevents the port from being mapped using igb_uio, and
> b) for ports which are meant to stay under linux control, forces me to start
> enumerating ports using blacklist or whitelisting, rather than having things
> "just work" on a properly configured system as before, i.e. if a port is bound
> to igb_uio or vfio it is used, if not bound, it is ignored. Again, is this by design
> and expected, because it seems a major regression in usability?

I think automatic port unbinding and binding was removed, so this again shouldn't happen at all.

It would be useful to have logs for all of these described situations, because we certainly didn't encounter any of that during the validation cycle.

Best regards,
Anatoly Burakov
DPDK SW Engineer


 

Re: [dpdk-dev] vfio detection

[Richardson, Bruce]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, June 17, 2014 1:40 AM
> To: Richardson, Bruce; dev@dpdk.org
> Subject: RE: vfio detection
> 
> Hi Bruce,
> 
> > I have a number of NIC ports which were working correctly yesterday and are
> > bound correctly to the igb_uio driver - and I want to keep using them
> > through the igb_uio driver for now, not vfio. However, whenever I run a
> > dpdk application today, I find that the vfio kernel module is getting loaded
> > each time - even after I manually remove it, and verify that it has been
> > removed by checking lsmod. Is this expected? If so, why are we loading the
> > vfio driver when I just want to continue using igb_uio which works fine?
> 
> Can you elaborate a bit on what do you mean by "loading vfio driver"? Do you
> mean the vfio-pci kernel gets loaded by DPDK? I certainly didn't put in any code
> that would automatically load that driver, and certainly not binding devices to it.

The kernel module called just "vfio" is constantly getting reloaded, and there is always a "/dev/vfio" directory, which triggers the vfio code handling every time I run dpdk.

> 
> > Secondly, then, when testpmd or any other app loads, it automatically tries
> > to map the NIC using vfio and then aborts on the very first NIC port when it
> > fails to do so.
> 
> This shouldn't happen, unless you have a device bound to VFIO and have another
> device in the same IOMMU group that is bound to something else. Can you
> provide a log of what you are seeing?

Log of testpmd run attached.

> 
> > This a) prevents the port from being mapped using igb_uio, and
> > b) for ports which are meant to stay under linux control, forces me to start
> > enumerating ports using blacklist or whitelisting, rather than having things
> > "just work" on a properly configured system as before, i.e. if a port is bound
> > to igb_uio or vfio it is used, if not bound, it is ignored. Again, is this by design
> > and expected, because it seems a major regression in usability?
> 
> I think automatic port unbinding and binding was removed, so this again
> shouldn't happen at all.
> 
> It would be useful to have logs for all of these described situations, because we
> certainly didn't encounter any of that during the validation cycle.
> 
Log of testpmd run attached. If you need any more debugging info, let me know.
I'll also test out the patch you just posted to the list - see if it makes any difference, and I'll send on a log from it.

/Bruce

Re: [dpdk-dev] vfio detection

[Thomas Monjalon]

Bruce, your log files have been automatically removed on the mailing list.
It's simpler to put logs in the email body.

-- 
Thomas

Re: [dpdk-dev] vfio detection

[Richardson, Bruce]

Yes, so I see. Resending with log in body.

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Tuesday, June 17, 2014 9:36 AM
> To: Richardson, Bruce
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] vfio detection
> 
> Bruce, your log files have been automatically removed on the mailing list.
> It's simpler to put logs in the email body.
> 
> --
> Thomas

Re: [dpdk-dev] vfio detection

[Richardson, Bruce]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Richardson, Bruce
> Sent: Tuesday, June 17, 2014 9:29 AM
> To: Burakov, Anatoly; dev@dpdk.org
> Subject: Re: [dpdk-dev] vfio detection
> 
> > -----Original Message-----
> > From: Burakov, Anatoly
> > Sent: Tuesday, June 17, 2014 1:40 AM
> > To: Richardson, Bruce; dev@dpdk.org
> > Subject: RE: vfio detection
> >
> > Hi Bruce,
> >
> > > I have a number of NIC ports which were working correctly yesterday and
> are
> > > bound correctly to the igb_uio driver - and I want to keep using them
> > > through the igb_uio driver for now, not vfio. However, whenever I run a
> > > dpdk application today, I find that the vfio kernel module is getting loaded
> > > each time - even after I manually remove it, and verify that it has been
> > > removed by checking lsmod. Is this expected? If so, why are we loading the
> > > vfio driver when I just want to continue using igb_uio which works fine?
> >
> > Can you elaborate a bit on what do you mean by "loading vfio driver"? Do you
> > mean the vfio-pci kernel gets loaded by DPDK? I certainly didn't put in any code
> > that would automatically load that driver, and certainly not binding devices to
> it.
> 
> The kernel module called just "vfio" is constantly getting reloaded, and there is
> always a "/dev/vfio" directory, which triggers the vfio code handling every time I
> run dpdk.
> 
> >
> > > Secondly, then, when testpmd or any other app loads, it automatically tries
> > > to map the NIC using vfio and then aborts on the very first NIC port when it
> > > fails to do so.
> >
> > This shouldn't happen, unless you have a device bound to VFIO and have
> another
> > device in the same IOMMU group that is bound to something else. Can you
> > provide a log of what you are seeing?
> 
> Log of testpmd run attached.

Log got stripped from mail, including below instead.

Script started on Tue 17 Jun 2014 17:23:54 IST

bruce@silpixa00372841:dpdk.org$ ./tools/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:84:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
0000:87:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
0000:8b:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
0000:8e:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe

Network devices using kernel driver
===================================
0000:04:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio *Active*
0000:04:00.1 'I350 Gigabit Network Connection' if=ens2f1 drv=igb unused=igb_uio 
0000:04:00.2 'I350 Gigabit Network Connection' if=ens2f2 drv=igb unused=igb_uio 
0000:04:00.3 'I350 Gigabit Network Connection' if=ens2f3 drv=igb unused=igb_uio 

Other network devices
=====================
0000:0a:00.1 'DH8900CC Null Device' unused=igb_uio
0000:0b:00.1 'DH8900CC Null Device' unused=igb_uio
0000:0c:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=ixgbe,igb_uio
0000:0c:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=ixgbe,igb_uio
0000:84:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
0000:87:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
0000:8b:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
0000:8e:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio


bruce@silpixa00372841:dpdk.org$ sudo ./x86_64-native-linuxapp-gcc/app/testpmd -c F00 -n 4 -- -i
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 6 on socket 0
EAL: Detected lcore 7 as core 7 on socket 0
EAL: Detected lcore 8 as core 0 on socket 1
EAL: Detected lcore 9 as core 1 on socket 1
EAL: Detected lcore 10 as core 2 on socket 1
EAL: Detected lcore 11 as core 3 on socket 1
EAL: Detected lcore 12 as core 4 on socket 1
EAL: Detected lcore 13 as core 5 on socket 1
EAL: Detected lcore 14 as core 6 on socket 1
EAL: Detected lcore 15 as core 7 on socket 1
EAL: Detected lcore 16 as core 0 on socket 0
EAL: Detected lcore 17 as core 1 on socket 0
EAL: Detected lcore 18 as core 2 on socket 0
EAL: Detected lcore 19 as core 3 on socket 0
EAL: Detected lcore 20 as core 4 on socket 0
EAL: Detected lcore 21 as core 5 on socket 0
EAL: Detected lcore 22 as core 6 on socket 0
EAL: Detected lcore 23 as core 7 on socket 0
EAL: Detected lcore 24 as core 0 on socket 1
EAL: Detected lcore 25 as core 1 on socket 1
EAL: Detected lcore 26 as core 2 on socket 1
EAL: Detected lcore 27 as core 3 on socket 1
EAL: Detected lcore 28 as core 4 on socket 1
EAL: Detected lcore 29 as core 5 on socket 1
EAL: Detected lcore 30 as core 6 on socket 1
EAL: Detected lcore 31 as core 7 on socket 1
EAL: Support maximum 64 logical core(s) by configuration.
EAL: Detected 32 lcore(s)
EAL: 1024 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
EAL: Setting up memory...
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Virtual area found at 0x7fe5c0000000 (size = 0x80000000)
EAL: Ask a virtual area of 0x80000000 bytes
EAL: Virtual area found at 0x7fe500000000 (size = 0x80000000)
EAL: Requesting 2 pages of size 1024MB from socket 0
EAL: Requesting 2 pages of size 1024MB from socket 1
EAL: TSC frequency is ~2693503 KHz
EAL: Master core 8 is ready (tid=e877c880)
EAL: Core 11 is ready (tid=e6741700)
EAL: Core 10 is ready (tid=e6f42700)
EAL: Core 9 is ready (tid=e7743700)
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   unknown IOMMU driver!
EAL:   0000:04:00.0 cannot open VFIO container!
EAL: Error - exiting with code: 1
  Cause: Requested device 0000:04:00.0 cannot be used


bruce@silpixa00372841:dpdk.org$ exit
exit

Script done on Tue 17 Jun 2014 17:25:29 IST

Re: [dpdk-dev] vfio detection

[Neil Horman]

On Tue, Jun 17, 2014 at 04:38:38PM +0000, Richardson, Bruce wrote:
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Richardson, Bruce
> > Sent: Tuesday, June 17, 2014 9:29 AM
> > To: Burakov, Anatoly; dev@dpdk.org
> > Subject: Re: [dpdk-dev] vfio detection
> > 
> > > -----Original Message-----
> > > From: Burakov, Anatoly
> > > Sent: Tuesday, June 17, 2014 1:40 AM
> > > To: Richardson, Bruce; dev@dpdk.org
> > > Subject: RE: vfio detection
> > >
> > > Hi Bruce,
> > >
> > > > I have a number of NIC ports which were working correctly yesterday and
> > are
> > > > bound correctly to the igb_uio driver - and I want to keep using them
> > > > through the igb_uio driver for now, not vfio. However, whenever I run a
> > > > dpdk application today, I find that the vfio kernel module is getting loaded
> > > > each time - even after I manually remove it, and verify that it has been
> > > > removed by checking lsmod. Is this expected? If so, why are we loading the
> > > > vfio driver when I just want to continue using igb_uio which works fine?
> > >
> > > Can you elaborate a bit on what do you mean by "loading vfio driver"? Do you
> > > mean the vfio-pci kernel gets loaded by DPDK? I certainly didn't put in any code
> > > that would automatically load that driver, and certainly not binding devices to
> > it.
> > 
> > The kernel module called just "vfio" is constantly getting reloaded, and there is
> > always a "/dev/vfio" directory, which triggers the vfio code handling every time I
> > run dpdk.
> > 
> > >
> > > > Secondly, then, when testpmd or any other app loads, it automatically tries
> > > > to map the NIC using vfio and then aborts on the very first NIC port when it
> > > > fails to do so.
> > >
> > > This shouldn't happen, unless you have a device bound to VFIO and have
> > another
> > > device in the same IOMMU group that is bound to something else. Can you
> > > provide a log of what you are seeing?
> > 
> > Log of testpmd run attached.
> 
> Log got stripped from mail, including below instead.
> 
> Script started on Tue 17 Jun 2014 17:23:54 IST
> 
> bruce@silpixa00372841:dpdk.org$ ./tools/dpdk_nic_bind.py --status
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:84:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> 0000:87:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> 0000:8b:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> 0000:8e:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> 
> Network devices using kernel driver
> ===================================
> 0000:04:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio *Active*
> 0000:04:00.1 'I350 Gigabit Network Connection' if=ens2f1 drv=igb unused=igb_uio 
> 0000:04:00.2 'I350 Gigabit Network Connection' if=ens2f2 drv=igb unused=igb_uio 
> 0000:04:00.3 'I350 Gigabit Network Connection' if=ens2f3 drv=igb unused=igb_uio 
> 
> Other network devices
> =====================
> 0000:0a:00.1 'DH8900CC Null Device' unused=igb_uio
> 0000:0b:00.1 'DH8900CC Null Device' unused=igb_uio
> 0000:0c:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=ixgbe,igb_uio
> 0000:0c:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=ixgbe,igb_uio
> 0000:84:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> 0000:87:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> 0000:8b:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> 0000:8e:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> 
> 
> bruce@silpixa00372841:dpdk.org$ sudo ./x86_64-native-linuxapp-gcc/app/testpmd -c F00 -n 4 -- -i
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 2 on socket 0
> EAL: Detected lcore 3 as core 3 on socket 0
> EAL: Detected lcore 4 as core 4 on socket 0
> EAL: Detected lcore 5 as core 5 on socket 0
> EAL: Detected lcore 6 as core 6 on socket 0
> EAL: Detected lcore 7 as core 7 on socket 0
> EAL: Detected lcore 8 as core 0 on socket 1
> EAL: Detected lcore 9 as core 1 on socket 1
> EAL: Detected lcore 10 as core 2 on socket 1
> EAL: Detected lcore 11 as core 3 on socket 1
> EAL: Detected lcore 12 as core 4 on socket 1
> EAL: Detected lcore 13 as core 5 on socket 1
> EAL: Detected lcore 14 as core 6 on socket 1
> EAL: Detected lcore 15 as core 7 on socket 1
> EAL: Detected lcore 16 as core 0 on socket 0
> EAL: Detected lcore 17 as core 1 on socket 0
> EAL: Detected lcore 18 as core 2 on socket 0
> EAL: Detected lcore 19 as core 3 on socket 0
> EAL: Detected lcore 20 as core 4 on socket 0
> EAL: Detected lcore 21 as core 5 on socket 0
> EAL: Detected lcore 22 as core 6 on socket 0
> EAL: Detected lcore 23 as core 7 on socket 0
> EAL: Detected lcore 24 as core 0 on socket 1
> EAL: Detected lcore 25 as core 1 on socket 1
> EAL: Detected lcore 26 as core 2 on socket 1
> EAL: Detected lcore 27 as core 3 on socket 1
> EAL: Detected lcore 28 as core 4 on socket 1
> EAL: Detected lcore 29 as core 5 on socket 1
> EAL: Detected lcore 30 as core 6 on socket 1
> EAL: Detected lcore 31 as core 7 on socket 1
> EAL: Support maximum 64 logical core(s) by configuration.
> EAL: Detected 32 lcore(s)
> EAL: 1024 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
> EAL: Setting up memory...
> EAL: Ask a virtual area of 0x80000000 bytes
> EAL: Virtual area found at 0x7fe5c0000000 (size = 0x80000000)
> EAL: Ask a virtual area of 0x80000000 bytes
> EAL: Virtual area found at 0x7fe500000000 (size = 0x80000000)
> EAL: Requesting 2 pages of size 1024MB from socket 0
> EAL: Requesting 2 pages of size 1024MB from socket 1
> EAL: TSC frequency is ~2693503 KHz
> EAL: Master core 8 is ready (tid=e877c880)
> EAL: Core 11 is ready (tid=e6741700)
> EAL: Core 10 is ready (tid=e6f42700)
> EAL: Core 9 is ready (tid=e7743700)
> EAL: PCI device 0000:04:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 rte_igb_pmd
> EAL:   unknown IOMMU driver!
> EAL:   0000:04:00.0 cannot open VFIO container!
> EAL: Error - exiting with code: 1
>   Cause: Requested device 0000:04:00.0 cannot be used
> 
> 
> bruce@silpixa00372841:dpdk.org$ exit
> exit
> 
> Script done on Tue 17 Jun 2014 17:25:29 IST
> 
> 
Can you attach the lspci output for this system?
Neil

Re: [dpdk-dev] vfio detection

[Richardson, Bruce]

> -----Original Message-----
> From: Neil Horman [mailto:nhorman@tuxdriver.com]
> Sent: Tuesday, June 17, 2014 10:42 AM
> To: Richardson, Bruce
> Cc: Burakov, Anatoly; dev@dpdk.org
> Subject: Re: [dpdk-dev] vfio detection
> 
> On Tue, Jun 17, 2014 at 04:38:38PM +0000, Richardson, Bruce wrote:
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Richardson, Bruce
> > > Sent: Tuesday, June 17, 2014 9:29 AM
> > > To: Burakov, Anatoly; dev@dpdk.org
> > > Subject: Re: [dpdk-dev] vfio detection
> > >
> > > > -----Original Message-----
> > > > From: Burakov, Anatoly
> > > > Sent: Tuesday, June 17, 2014 1:40 AM
> > > > To: Richardson, Bruce; dev@dpdk.org
> > > > Subject: RE: vfio detection
> > > >
> > > > Hi Bruce,
> > > >
> > > > > I have a number of NIC ports which were working correctly yesterday and
> > > are
> > > > > bound correctly to the igb_uio driver - and I want to keep using them
> > > > > through the igb_uio driver for now, not vfio. However, whenever I run a
> > > > > dpdk application today, I find that the vfio kernel module is getting
> loaded
> > > > > each time - even after I manually remove it, and verify that it has been
> > > > > removed by checking lsmod. Is this expected? If so, why are we loading
> the
> > > > > vfio driver when I just want to continue using igb_uio which works fine?
> > > >
> > > > Can you elaborate a bit on what do you mean by "loading vfio driver"? Do
> you
> > > > mean the vfio-pci kernel gets loaded by DPDK? I certainly didn't put in any
> code
> > > > that would automatically load that driver, and certainly not binding devices
> to
> > > it.
> > >
> > > The kernel module called just "vfio" is constantly getting reloaded, and there
> is
> > > always a "/dev/vfio" directory, which triggers the vfio code handling every
> time I
> > > run dpdk.
> > >
> > > >
> > > > > Secondly, then, when testpmd or any other app loads, it automatically
> tries
> > > > > to map the NIC using vfio and then aborts on the very first NIC port when
> it
> > > > > fails to do so.
> > > >
> > > > This shouldn't happen, unless you have a device bound to VFIO and have
> > > another
> > > > device in the same IOMMU group that is bound to something else. Can you
> > > > provide a log of what you are seeing?
> > >
> > > Log of testpmd run attached.
> >
> > Log got stripped from mail, including below instead.
> >
> > Script started on Tue 17 Jun 2014 17:23:54 IST
> >
> > bruce@silpixa00372841:dpdk.org$ ./tools/dpdk_nic_bind.py --status
> >
> > Network devices using DPDK-compatible driver
> > ============================================
> > 0000:84:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> > 0000:87:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> > 0000:8b:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> > 0000:8e:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> >
> > Network devices using kernel driver
> > ===================================
> > 0000:04:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb
> unused=igb_uio *Active*
> > 0000:04:00.1 'I350 Gigabit Network Connection' if=ens2f1 drv=igb
> unused=igb_uio
> > 0000:04:00.2 'I350 Gigabit Network Connection' if=ens2f2 drv=igb
> unused=igb_uio
> > 0000:04:00.3 'I350 Gigabit Network Connection' if=ens2f3 drv=igb
> unused=igb_uio
> >
> > Other network devices
> > =====================
> > 0000:0a:00.1 'DH8900CC Null Device' unused=igb_uio
> > 0000:0b:00.1 'DH8900CC Null Device' unused=igb_uio
> > 0000:0c:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
> unused=ixgbe,igb_uio
> > 0000:0c:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
> unused=ixgbe,igb_uio
> > 0000:84:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> > 0000:87:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> > 0000:8b:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> > 0000:8e:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> >
> >
> > bruce@silpixa00372841:dpdk.org$ sudo ./x86_64-native-linuxapp-
> gcc/app/testpmd -c F00 -n 4 -- -i
> > EAL: Detected lcore 0 as core 0 on socket 0
> > EAL: Detected lcore 1 as core 1 on socket 0
> > EAL: Detected lcore 2 as core 2 on socket 0
> > EAL: Detected lcore 3 as core 3 on socket 0
> > EAL: Detected lcore 4 as core 4 on socket 0
> > EAL: Detected lcore 5 as core 5 on socket 0
> > EAL: Detected lcore 6 as core 6 on socket 0
> > EAL: Detected lcore 7 as core 7 on socket 0
> > EAL: Detected lcore 8 as core 0 on socket 1
> > EAL: Detected lcore 9 as core 1 on socket 1
> > EAL: Detected lcore 10 as core 2 on socket 1
> > EAL: Detected lcore 11 as core 3 on socket 1
> > EAL: Detected lcore 12 as core 4 on socket 1
> > EAL: Detected lcore 13 as core 5 on socket 1
> > EAL: Detected lcore 14 as core 6 on socket 1
> > EAL: Detected lcore 15 as core 7 on socket 1
> > EAL: Detected lcore 16 as core 0 on socket 0
> > EAL: Detected lcore 17 as core 1 on socket 0
> > EAL: Detected lcore 18 as core 2 on socket 0
> > EAL: Detected lcore 19 as core 3 on socket 0
> > EAL: Detected lcore 20 as core 4 on socket 0
> > EAL: Detected lcore 21 as core 5 on socket 0
> > EAL: Detected lcore 22 as core 6 on socket 0
> > EAL: Detected lcore 23 as core 7 on socket 0
> > EAL: Detected lcore 24 as core 0 on socket 1
> > EAL: Detected lcore 25 as core 1 on socket 1
> > EAL: Detected lcore 26 as core 2 on socket 1
> > EAL: Detected lcore 27 as core 3 on socket 1
> > EAL: Detected lcore 28 as core 4 on socket 1
> > EAL: Detected lcore 29 as core 5 on socket 1
> > EAL: Detected lcore 30 as core 6 on socket 1
> > EAL: Detected lcore 31 as core 7 on socket 1
> > EAL: Support maximum 64 logical core(s) by configuration.
> > EAL: Detected 32 lcore(s)
> > EAL: 1024 hugepages of size 2097152 reserved, but no mounted hugetlbfs
> found for that size
> > EAL: Setting up memory...
> > EAL: Ask a virtual area of 0x80000000 bytes
> > EAL: Virtual area found at 0x7fe5c0000000 (size = 0x80000000)
> > EAL: Ask a virtual area of 0x80000000 bytes
> > EAL: Virtual area found at 0x7fe500000000 (size = 0x80000000)
> > EAL: Requesting 2 pages of size 1024MB from socket 0
> > EAL: Requesting 2 pages of size 1024MB from socket 1
> > EAL: TSC frequency is ~2693503 KHz
> > EAL: Master core 8 is ready (tid=e877c880)
> > EAL: Core 11 is ready (tid=e6741700)
> > EAL: Core 10 is ready (tid=e6f42700)
> > EAL: Core 9 is ready (tid=e7743700)
> > EAL: PCI device 0000:04:00.0 on NUMA socket 0
> > EAL:   probe driver: 8086:1521 rte_igb_pmd
> > EAL:   unknown IOMMU driver!
> > EAL:   0000:04:00.0 cannot open VFIO container!
> > EAL: Error - exiting with code: 1
> >   Cause: Requested device 0000:04:00.0 cannot be used
> >
> >
> > bruce@silpixa00372841:dpdk.org$ exit
> > exit
> >
> > Script done on Tue 17 Jun 2014 17:25:29 IST
> >
> >
> Can you attach the lspci output for this system?
> Neil

See below. 
However, Anatoly's patch to do further one-time checks for vfio at startup fixes the issue nicely for me. It properly detects that vfio is not enabled and to skip it for all devices.

00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07)
00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07)
00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
00:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 07)
00:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
00:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
00:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
00:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
00:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
00:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
00:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
00:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07)
00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b6)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
01:00.0 PCI bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Upstream Port (rev 06)
02:08.0 PCI bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Virtual Switch Port (rev 06)
03:00.0 Serial Attached SCSI controller: Intel Corporation C600/X79 series chipset Dual 4-Port SATA Storage Control Unit (rev 06)
03:00.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Controller 0 (rev 06)
03:00.4 SMBus: Intel Corporation C608/C606/X79 series chipset SMBus Controller 1 (rev 06)
04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
09:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
09:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0a:00.0 Co-processor: Intel Corporation Device 0434 (rev 21)
0a:00.1 Ethernet controller: Intel Corporation DH8900CC Null Device (rev 21)
0b:00.0 Co-processor: Intel Corporation Device 0434 (rev 21)
0b:00.1 Ethernet controller: Intel Corporation DH8900CC Null Device (rev 21)
0c:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
0c:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
0f:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 07)
7f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
7f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
7f:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07)
7f:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07)
7f:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07)
7f:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07)
7f:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07)
7f:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07)
7f:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07)
7f:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07)
7f:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07)
7f:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07)
7f:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07)
7f:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07)
7f:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07)
7f:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07)
7f:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07)
7f:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07)
7f:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07)
7f:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07)
7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
7f:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
7f:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07)
7f:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07)
7f:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07)
7f:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
7f:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07)
7f:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07)
7f:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07)
7f:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07)
7f:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07)
7f:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07)
7f:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
7f:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)
80:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
80:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07)
80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
80:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 07)
80:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
80:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
80:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
80:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
80:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
80:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
80:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
80:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
80:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
80:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
80:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07)
82:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca)
83:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca)
83:08.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca)
83:09.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca)
84:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
84:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
87:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
87:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
89:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba)
8a:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba)
8a:08.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba)
8a:09.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba)
8b:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
8b:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
8e:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
8e:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
ff:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07)
ff:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07)
ff:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07)
ff:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07)
ff:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07)
ff:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07)
ff:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07)
ff:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07)
ff:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07)
ff:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07)
ff:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07)
ff:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07)
ff:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07)
ff:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07)
ff:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07)
ff:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07)
ff:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07)
ff:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07)
ff:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
ff:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
ff:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07)
ff:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07)
ff:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07)
ff:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
ff:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07)
ff:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07)
ff:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07)
ff:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07)
ff:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07)
ff:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07)
ff:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
ff:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)

Re: [dpdk-dev] vfio detection

[Neil Horman]

On Tue, Jun 17, 2014 at 05:45:55PM +0000, Richardson, Bruce wrote:
> 
> 
> > -----Original Message-----
> > From: Neil Horman [mailto:nhorman@tuxdriver.com]
> > Sent: Tuesday, June 17, 2014 10:42 AM
> > To: Richardson, Bruce
> > Cc: Burakov, Anatoly; dev@dpdk.org
> > Subject: Re: [dpdk-dev] vfio detection
> > 
> > On Tue, Jun 17, 2014 at 04:38:38PM +0000, Richardson, Bruce wrote:
> > > > -----Original Message-----
> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Richardson, Bruce
> > > > Sent: Tuesday, June 17, 2014 9:29 AM
> > > > To: Burakov, Anatoly; dev@dpdk.org
> > > > Subject: Re: [dpdk-dev] vfio detection
> > > >
> > > > > -----Original Message-----
> > > > > From: Burakov, Anatoly
> > > > > Sent: Tuesday, June 17, 2014 1:40 AM
> > > > > To: Richardson, Bruce; dev@dpdk.org
> > > > > Subject: RE: vfio detection
> > > > >
> > > > > Hi Bruce,
> > > > >
> > > > > > I have a number of NIC ports which were working correctly yesterday and
> > > > are
> > > > > > bound correctly to the igb_uio driver - and I want to keep using them
> > > > > > through the igb_uio driver for now, not vfio. However, whenever I run a
> > > > > > dpdk application today, I find that the vfio kernel module is getting
> > loaded
> > > > > > each time - even after I manually remove it, and verify that it has been
> > > > > > removed by checking lsmod. Is this expected? If so, why are we loading
> > the
> > > > > > vfio driver when I just want to continue using igb_uio which works fine?
> > > > >
> > > > > Can you elaborate a bit on what do you mean by "loading vfio driver"? Do
> > you
> > > > > mean the vfio-pci kernel gets loaded by DPDK? I certainly didn't put in any
> > code
> > > > > that would automatically load that driver, and certainly not binding devices
> > to
> > > > it.
> > > >
> > > > The kernel module called just "vfio" is constantly getting reloaded, and there
> > is
> > > > always a "/dev/vfio" directory, which triggers the vfio code handling every
> > time I
> > > > run dpdk.
> > > >
> > > > >
> > > > > > Secondly, then, when testpmd or any other app loads, it automatically
> > tries
> > > > > > to map the NIC using vfio and then aborts on the very first NIC port when
> > it
> > > > > > fails to do so.
> > > > >
> > > > > This shouldn't happen, unless you have a device bound to VFIO and have
> > > > another
> > > > > device in the same IOMMU group that is bound to something else. Can you
> > > > > provide a log of what you are seeing?
> > > >
> > > > Log of testpmd run attached.
> > >
> > > Log got stripped from mail, including below instead.
> > >
> > > Script started on Tue 17 Jun 2014 17:23:54 IST
> > >
> > > bruce@silpixa00372841:dpdk.org$ ./tools/dpdk_nic_bind.py --status
> > >
> > > Network devices using DPDK-compatible driver
> > > ============================================
> > > 0000:84:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> > > 0000:87:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> > > 0000:8b:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> > > 0000:8e:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe
> > >
> > > Network devices using kernel driver
> > > ===================================
> > > 0000:04:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb
> > unused=igb_uio *Active*
> > > 0000:04:00.1 'I350 Gigabit Network Connection' if=ens2f1 drv=igb
> > unused=igb_uio
> > > 0000:04:00.2 'I350 Gigabit Network Connection' if=ens2f2 drv=igb
> > unused=igb_uio
> > > 0000:04:00.3 'I350 Gigabit Network Connection' if=ens2f3 drv=igb
> > unused=igb_uio
> > >
> > > Other network devices
> > > =====================
> > > 0000:0a:00.1 'DH8900CC Null Device' unused=igb_uio
> > > 0000:0b:00.1 'DH8900CC Null Device' unused=igb_uio
> > > 0000:0c:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
> > unused=ixgbe,igb_uio
> > > 0000:0c:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
> > unused=ixgbe,igb_uio
> > > 0000:84:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> > > 0000:87:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> > > 0000:8b:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> > > 0000:8e:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio
> > >
> > >
> > > bruce@silpixa00372841:dpdk.org$ sudo ./x86_64-native-linuxapp-
> > gcc/app/testpmd -c F00 -n 4 -- -i
> > > EAL: Detected lcore 0 as core 0 on socket 0
> > > EAL: Detected lcore 1 as core 1 on socket 0
> > > EAL: Detected lcore 2 as core 2 on socket 0
> > > EAL: Detected lcore 3 as core 3 on socket 0
> > > EAL: Detected lcore 4 as core 4 on socket 0
> > > EAL: Detected lcore 5 as core 5 on socket 0
> > > EAL: Detected lcore 6 as core 6 on socket 0
> > > EAL: Detected lcore 7 as core 7 on socket 0
> > > EAL: Detected lcore 8 as core 0 on socket 1
> > > EAL: Detected lcore 9 as core 1 on socket 1
> > > EAL: Detected lcore 10 as core 2 on socket 1
> > > EAL: Detected lcore 11 as core 3 on socket 1
> > > EAL: Detected lcore 12 as core 4 on socket 1
> > > EAL: Detected lcore 13 as core 5 on socket 1
> > > EAL: Detected lcore 14 as core 6 on socket 1
> > > EAL: Detected lcore 15 as core 7 on socket 1
> > > EAL: Detected lcore 16 as core 0 on socket 0
> > > EAL: Detected lcore 17 as core 1 on socket 0
> > > EAL: Detected lcore 18 as core 2 on socket 0
> > > EAL: Detected lcore 19 as core 3 on socket 0
> > > EAL: Detected lcore 20 as core 4 on socket 0
> > > EAL: Detected lcore 21 as core 5 on socket 0
> > > EAL: Detected lcore 22 as core 6 on socket 0
> > > EAL: Detected lcore 23 as core 7 on socket 0
> > > EAL: Detected lcore 24 as core 0 on socket 1
> > > EAL: Detected lcore 25 as core 1 on socket 1
> > > EAL: Detected lcore 26 as core 2 on socket 1
> > > EAL: Detected lcore 27 as core 3 on socket 1
> > > EAL: Detected lcore 28 as core 4 on socket 1
> > > EAL: Detected lcore 29 as core 5 on socket 1
> > > EAL: Detected lcore 30 as core 6 on socket 1
> > > EAL: Detected lcore 31 as core 7 on socket 1
> > > EAL: Support maximum 64 logical core(s) by configuration.
> > > EAL: Detected 32 lcore(s)
> > > EAL: 1024 hugepages of size 2097152 reserved, but no mounted hugetlbfs
> > found for that size
> > > EAL: Setting up memory...
> > > EAL: Ask a virtual area of 0x80000000 bytes
> > > EAL: Virtual area found at 0x7fe5c0000000 (size = 0x80000000)
> > > EAL: Ask a virtual area of 0x80000000 bytes
> > > EAL: Virtual area found at 0x7fe500000000 (size = 0x80000000)
> > > EAL: Requesting 2 pages of size 1024MB from socket 0
> > > EAL: Requesting 2 pages of size 1024MB from socket 1
> > > EAL: TSC frequency is ~2693503 KHz
> > > EAL: Master core 8 is ready (tid=e877c880)
> > > EAL: Core 11 is ready (tid=e6741700)
> > > EAL: Core 10 is ready (tid=e6f42700)
> > > EAL: Core 9 is ready (tid=e7743700)
> > > EAL: PCI device 0000:04:00.0 on NUMA socket 0
> > > EAL:   probe driver: 8086:1521 rte_igb_pmd
> > > EAL:   unknown IOMMU driver!
> > > EAL:   0000:04:00.0 cannot open VFIO container!
> > > EAL: Error - exiting with code: 1
> > >   Cause: Requested device 0000:04:00.0 cannot be used
> > >
> > >
> > > bruce@silpixa00372841:dpdk.org$ exit
> > > exit
> > >
> > > Script done on Tue 17 Jun 2014 17:25:29 IST
> > >
> > >
> > Can you attach the lspci output for this system?
> > Neil
> 
> See below. 
> However, Anatoly's patch to do further one-time checks for vfio at startup fixes the issue nicely for me. It properly detects that vfio is not enabled and to skip it for all devices.
> 
Sure, and thats fine.  It just struck me as odd that it found that you had an
iommu, but that it wasn't the right type.  Made me wonder what type of iommu you
have present.  It seems that you may just have gotten caught on bad error
checking in the ioctl return code
Neil


> 00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)
> 00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
> 00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07)
> 00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
> 00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07)
> 00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
> 00:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 07)
> 00:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
> 00:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
> 00:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
> 00:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
> 00:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
> 00:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
> 00:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
> 00:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
> 00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
> 00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
> 00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07)
> 00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
> 00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)
> 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
> 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
> 00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b6)
> 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
> 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
> 00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)
> 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
> 01:00.0 PCI bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Upstream Port (rev 06)
> 02:08.0 PCI bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Virtual Switch Port (rev 06)
> 03:00.0 Serial Attached SCSI controller: Intel Corporation C600/X79 series chipset Dual 4-Port SATA Storage Control Unit (rev 06)
> 03:00.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Controller 0 (rev 06)
> 03:00.4 SMBus: Intel Corporation C608/C606/X79 series chipset SMBus Controller 1 (rev 06)
> 04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
> 04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
> 04:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
> 04:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
> 08:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
> 09:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
> 09:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
> 0a:00.0 Co-processor: Intel Corporation Device 0434 (rev 21)
> 0a:00.1 Ethernet controller: Intel Corporation DH8900CC Null Device (rev 21)
> 0b:00.0 Co-processor: Intel Corporation Device 0434 (rev 21)
> 0b:00.1 Ethernet controller: Intel Corporation DH8900CC Null Device (rev 21)
> 0c:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 0c:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 0f:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 07)
> 7f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
> 7f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
> 7f:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07)
> 7f:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07)
> 7f:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07)
> 7f:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07)
> 7f:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07)
> 7f:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07)
> 7f:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> 7f:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> 7f:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> 7f:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> 7f:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07)
> 7f:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07)
> 7f:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> 7f:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> 7f:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> 7f:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> 7f:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07)
> 7f:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07)
> 7f:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07)
> 7f:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07)
> 7f:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07)
> 7f:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07)
> 7f:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07)
> 7f:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07)
> 7f:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07)
> 7f:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07)
> 7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
> 7f:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
> 7f:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07)
> 7f:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07)
> 7f:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07)
> 7f:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
> 7f:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07)
> 7f:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07)
> 7f:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07)
> 7f:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07)
> 7f:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07)
> 7f:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07)
> 7f:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
> 7f:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)
> 80:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
> 80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
> 80:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07)
> 80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
> 80:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 07)
> 80:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
> 80:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
> 80:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
> 80:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
> 80:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
> 80:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
> 80:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
> 80:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
> 80:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
> 80:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
> 80:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07)
> 82:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca)
> 83:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca)
> 83:08.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca)
> 83:09.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca)
> 84:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
> 84:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
> 87:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
> 87:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
> 89:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba)
> 8a:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba)
> 8a:08.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba)
> 8a:09.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba)
> 8b:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
> 8b:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
> 8e:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
> 8e:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01)
> ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
> ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
> ff:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07)
> ff:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07)
> ff:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07)
> ff:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07)
> ff:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07)
> ff:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07)
> ff:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> ff:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> ff:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> ff:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> ff:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07)
> ff:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07)
> ff:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> ff:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> ff:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> ff:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
> ff:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07)
> ff:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07)
> ff:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07)
> ff:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07)
> ff:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07)
> ff:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07)
> ff:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07)
> ff:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07)
> ff:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07)
> ff:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07)
> ff:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
> ff:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
> ff:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07)
> ff:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07)
> ff:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07)
> ff:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
> ff:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07)
> ff:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07)
> ff:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07)
> ff:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07)
> ff:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07)
> ff:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07)
> ff:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
> ff:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)
> 
> 

Re: [dpdk-dev] vfio detection

[Burakov, Anatoly]

Hi Bruce,

> > > I have a number of NIC ports which were working correctly yesterday
> > > and are bound correctly to the igb_uio driver - and I want to keep
> > > using them through the igb_uio driver for now, not vfio. However,
> > > whenever I run a dpdk application today, I find that the vfio kernel
> > > module is getting loaded each time - even after I manually remove
> > > it, and verify that it has been removed by checking lsmod. Is this
> > > expected? If so, why are we loading the vfio driver when I just want to
> continue using igb_uio which works fine?
> >
> > Can you elaborate a bit on what do you mean by "loading vfio driver"?
> > Do you mean the vfio-pci kernel gets loaded by DPDK? I certainly
> > didn't put in any code that would automatically load that driver, and
> certainly not binding devices to it.
> 
> The kernel module called just "vfio" is constantly getting reloaded, and there
> is always a "/dev/vfio" directory, which triggers the vfio code handling every
> time I run dpdk.

I can't reproduce this.

Please note that VFIO actually consists of three drivers (on an x86 system, that is) - vfio (the core VFIO infrastructure such as containers), vfio_iommu_type1 (support for x86-style IOMMU) and vfio-pci (the generic PCI driver). I have unloaded all three and ran dpdk_nic_bind and testpmd - it worked fine and no VFIO kernel drivers were loaded as a result.

Best regards,
Anatoly Burakov
DPDK SW Engineer

Re: [dpdk-dev] vfio detection

[Richardson, Bruce]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Wednesday, June 18, 2014 4:01 AM
> To: Richardson, Bruce; dev@dpdk.org
> Subject: RE: vfio detection
> 
> Hi Bruce,
> 
> > > > I have a number of NIC ports which were working correctly yesterday
> > > > and are bound correctly to the igb_uio driver - and I want to keep
> > > > using them through the igb_uio driver for now, not vfio. However,
> > > > whenever I run a dpdk application today, I find that the vfio kernel
> > > > module is getting loaded each time - even after I manually remove
> > > > it, and verify that it has been removed by checking lsmod. Is this
> > > > expected? If so, why are we loading the vfio driver when I just want to
> > continue using igb_uio which works fine?
> > >
> > > Can you elaborate a bit on what do you mean by "loading vfio driver"?
> > > Do you mean the vfio-pci kernel gets loaded by DPDK? I certainly
> > > didn't put in any code that would automatically load that driver, and
> > certainly not binding devices to it.
> >
> > The kernel module called just "vfio" is constantly getting reloaded, and there
> > is always a "/dev/vfio" directory, which triggers the vfio code handling every
> > time I run dpdk.
> 
> I can't reproduce this.
> 
> Please note that VFIO actually consists of three drivers (on an x86 system, that
> is) - vfio (the core VFIO infrastructure such as containers), vfio_iommu_type1
> (support for x86-style IOMMU) and vfio-pci (the generic PCI driver). I have
> unloaded all three and ran dpdk_nic_bind and testpmd - it worked fine and no
> VFIO kernel drivers were loaded as a result.
> 
Ok, maybe a one-off. The proposed patches to dpdk to improve the detection of vfio fix the real issue for me anyway, so I'm no longer concerned.