From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.tuxdriver.com (charlotte.tuxdriver.com [70.61.120.58]) by dpdk.org (Postfix) with ESMTP id D15E3AFD1 for ; Tue, 17 Jun 2014 20:43:44 +0200 (CEST) Received: from hmsreliant.think-freely.org ([2001:470:8:a08:7aac:c0ff:fec2:933b] helo=localhost) by smtp.tuxdriver.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.63) (envelope-from ) id 1WwyMJ-0004To-3M; Tue, 17 Jun 2014 14:43:59 -0400 Date: Tue, 17 Jun 2014 14:43:50 -0400 From: Neil Horman To: "Richardson, Bruce" Message-ID: <20140617184350.GG8539@hmsreliant.think-freely.org> References: <59AF69C657FD0841A61C55336867B5B01AA36B6E@IRSMSX103.ger.corp.intel.com> <59AF69C657FD0841A61C55336867B5B01AA370A7@IRSMSX103.ger.corp.intel.com> <59AF69C657FD0841A61C55336867B5B01AA37143@IRSMSX103.ger.corp.intel.com> <20140617174155.GF8539@hmsreliant.think-freely.org> <59AF69C657FD0841A61C55336867B5B01AA37285@IRSMSX103.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <59AF69C657FD0841A61C55336867B5B01AA37285@IRSMSX103.ger.corp.intel.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Score: -2.9 (--) X-Spam-Status: No Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] vfio detection X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2014 18:43:45 -0000 On Tue, Jun 17, 2014 at 05:45:55PM +0000, Richardson, Bruce wrote: > > > > -----Original Message----- > > From: Neil Horman [mailto:nhorman@tuxdriver.com] > > Sent: Tuesday, June 17, 2014 10:42 AM > > To: Richardson, Bruce > > Cc: Burakov, Anatoly; dev@dpdk.org > > Subject: Re: [dpdk-dev] vfio detection > > > > On Tue, Jun 17, 2014 at 04:38:38PM +0000, Richardson, Bruce wrote: > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Richardson, Bruce > > > > Sent: Tuesday, June 17, 2014 9:29 AM > > > > To: Burakov, Anatoly; dev@dpdk.org > > > > Subject: Re: [dpdk-dev] vfio detection > > > > > > > > > -----Original Message----- > > > > > From: Burakov, Anatoly > > > > > Sent: Tuesday, June 17, 2014 1:40 AM > > > > > To: Richardson, Bruce; dev@dpdk.org > > > > > Subject: RE: vfio detection > > > > > > > > > > Hi Bruce, > > > > > > > > > > > I have a number of NIC ports which were working correctly yesterday and > > > > are > > > > > > bound correctly to the igb_uio driver - and I want to keep using them > > > > > > through the igb_uio driver for now, not vfio. However, whenever I run a > > > > > > dpdk application today, I find that the vfio kernel module is getting > > loaded > > > > > > each time - even after I manually remove it, and verify that it has been > > > > > > removed by checking lsmod. Is this expected? If so, why are we loading > > the > > > > > > vfio driver when I just want to continue using igb_uio which works fine? > > > > > > > > > > Can you elaborate a bit on what do you mean by "loading vfio driver"? Do > > you > > > > > mean the vfio-pci kernel gets loaded by DPDK? I certainly didn't put in any > > code > > > > > that would automatically load that driver, and certainly not binding devices > > to > > > > it. > > > > > > > > The kernel module called just "vfio" is constantly getting reloaded, and there > > is > > > > always a "/dev/vfio" directory, which triggers the vfio code handling every > > time I > > > > run dpdk. > > > > > > > > > > > > > > > Secondly, then, when testpmd or any other app loads, it automatically > > tries > > > > > > to map the NIC using vfio and then aborts on the very first NIC port when > > it > > > > > > fails to do so. > > > > > > > > > > This shouldn't happen, unless you have a device bound to VFIO and have > > > > another > > > > > device in the same IOMMU group that is bound to something else. Can you > > > > > provide a log of what you are seeing? > > > > > > > > Log of testpmd run attached. > > > > > > Log got stripped from mail, including below instead. > > > > > > Script started on Tue 17 Jun 2014 17:23:54 IST > > > > > > bruce@silpixa00372841:dpdk.org$ ./tools/dpdk_nic_bind.py --status > > > > > > Network devices using DPDK-compatible driver > > > ============================================ > > > 0000:84:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe > > > 0000:87:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe > > > 0000:8b:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe > > > 0000:8e:00.0 'Ethernet Server Adapter X520-4' drv=igb_uio unused=ixgbe > > > > > > Network devices using kernel driver > > > =================================== > > > 0000:04:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb > > unused=igb_uio *Active* > > > 0000:04:00.1 'I350 Gigabit Network Connection' if=ens2f1 drv=igb > > unused=igb_uio > > > 0000:04:00.2 'I350 Gigabit Network Connection' if=ens2f2 drv=igb > > unused=igb_uio > > > 0000:04:00.3 'I350 Gigabit Network Connection' if=ens2f3 drv=igb > > unused=igb_uio > > > > > > Other network devices > > > ===================== > > > 0000:0a:00.1 'DH8900CC Null Device' unused=igb_uio > > > 0000:0b:00.1 'DH8900CC Null Device' unused=igb_uio > > > 0000:0c:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' > > unused=ixgbe,igb_uio > > > 0000:0c:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' > > unused=ixgbe,igb_uio > > > 0000:84:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio > > > 0000:87:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio > > > 0000:8b:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio > > > 0000:8e:00.1 'Ethernet Server Adapter X520-4' unused=ixgbe,igb_uio > > > > > > > > > bruce@silpixa00372841:dpdk.org$ sudo ./x86_64-native-linuxapp- > > gcc/app/testpmd -c F00 -n 4 -- -i > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > EAL: Detected lcore 1 as core 1 on socket 0 > > > EAL: Detected lcore 2 as core 2 on socket 0 > > > EAL: Detected lcore 3 as core 3 on socket 0 > > > EAL: Detected lcore 4 as core 4 on socket 0 > > > EAL: Detected lcore 5 as core 5 on socket 0 > > > EAL: Detected lcore 6 as core 6 on socket 0 > > > EAL: Detected lcore 7 as core 7 on socket 0 > > > EAL: Detected lcore 8 as core 0 on socket 1 > > > EAL: Detected lcore 9 as core 1 on socket 1 > > > EAL: Detected lcore 10 as core 2 on socket 1 > > > EAL: Detected lcore 11 as core 3 on socket 1 > > > EAL: Detected lcore 12 as core 4 on socket 1 > > > EAL: Detected lcore 13 as core 5 on socket 1 > > > EAL: Detected lcore 14 as core 6 on socket 1 > > > EAL: Detected lcore 15 as core 7 on socket 1 > > > EAL: Detected lcore 16 as core 0 on socket 0 > > > EAL: Detected lcore 17 as core 1 on socket 0 > > > EAL: Detected lcore 18 as core 2 on socket 0 > > > EAL: Detected lcore 19 as core 3 on socket 0 > > > EAL: Detected lcore 20 as core 4 on socket 0 > > > EAL: Detected lcore 21 as core 5 on socket 0 > > > EAL: Detected lcore 22 as core 6 on socket 0 > > > EAL: Detected lcore 23 as core 7 on socket 0 > > > EAL: Detected lcore 24 as core 0 on socket 1 > > > EAL: Detected lcore 25 as core 1 on socket 1 > > > EAL: Detected lcore 26 as core 2 on socket 1 > > > EAL: Detected lcore 27 as core 3 on socket 1 > > > EAL: Detected lcore 28 as core 4 on socket 1 > > > EAL: Detected lcore 29 as core 5 on socket 1 > > > EAL: Detected lcore 30 as core 6 on socket 1 > > > EAL: Detected lcore 31 as core 7 on socket 1 > > > EAL: Support maximum 64 logical core(s) by configuration. > > > EAL: Detected 32 lcore(s) > > > EAL: 1024 hugepages of size 2097152 reserved, but no mounted hugetlbfs > > found for that size > > > EAL: Setting up memory... > > > EAL: Ask a virtual area of 0x80000000 bytes > > > EAL: Virtual area found at 0x7fe5c0000000 (size = 0x80000000) > > > EAL: Ask a virtual area of 0x80000000 bytes > > > EAL: Virtual area found at 0x7fe500000000 (size = 0x80000000) > > > EAL: Requesting 2 pages of size 1024MB from socket 0 > > > EAL: Requesting 2 pages of size 1024MB from socket 1 > > > EAL: TSC frequency is ~2693503 KHz > > > EAL: Master core 8 is ready (tid=e877c880) > > > EAL: Core 11 is ready (tid=e6741700) > > > EAL: Core 10 is ready (tid=e6f42700) > > > EAL: Core 9 is ready (tid=e7743700) > > > EAL: PCI device 0000:04:00.0 on NUMA socket 0 > > > EAL: probe driver: 8086:1521 rte_igb_pmd > > > EAL: unknown IOMMU driver! > > > EAL: 0000:04:00.0 cannot open VFIO container! > > > EAL: Error - exiting with code: 1 > > > Cause: Requested device 0000:04:00.0 cannot be used > > > > > > > > > bruce@silpixa00372841:dpdk.org$ exit > > > exit > > > > > > Script done on Tue 17 Jun 2014 17:25:29 IST > > > > > > > > Can you attach the lspci output for this system? > > Neil > > See below. > However, Anatoly's patch to do further one-time checks for vfio at startup fixes the issue nicely for me. It properly detects that vfio is not enabled and to skip it for all devices. > Sure, and thats fine. It just struck me as odd that it found that you had an iommu, but that it wasn't the right type. Made me wonder what type of iommu you have present. It seems that you may just have gotten caught on bad error checking in the ioctl return code Neil > 00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07) > 00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07) > 00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07) > 00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07) > 00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07) > 00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07) > 00:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 07) > 00:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07) > 00:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07) > 00:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07) > 00:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07) > 00:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07) > 00:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07) > 00:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07) > 00:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07) > 00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07) > 00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07) > 00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07) > 00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05) > 00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05) > 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06) > 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6) > 00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b6) > 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06) > 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6) > 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06) > 00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06) > 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06) > 01:00.0 PCI bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Upstream Port (rev 06) > 02:08.0 PCI bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Virtual Switch Port (rev 06) > 03:00.0 Serial Attached SCSI controller: Intel Corporation C600/X79 series chipset Dual 4-Port SATA Storage Control Unit (rev 06) > 03:00.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Controller 0 (rev 06) > 03:00.4 SMBus: Intel Corporation C608/C606/X79 series chipset SMBus Controller 1 (rev 06) > 04:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) > 04:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) > 04:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) > 04:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) > 08:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) > 09:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) > 09:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) > 0a:00.0 Co-processor: Intel Corporation Device 0434 (rev 21) > 0a:00.1 Ethernet controller: Intel Corporation DH8900CC Null Device (rev 21) > 0b:00.0 Co-processor: Intel Corporation Device 0434 (rev 21) > 0b:00.1 Ethernet controller: Intel Corporation DH8900CC Null Device (rev 21) > 0c:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) > 0c:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) > 0f:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 07) > 7f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07) > 7f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07) > 7f:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07) > 7f:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07) > 7f:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07) > 7f:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07) > 7f:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07) > 7f:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07) > 7f:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > 7f:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > 7f:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > 7f:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > 7f:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07) > 7f:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07) > 7f:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > 7f:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > 7f:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > 7f:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > 7f:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07) > 7f:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07) > 7f:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07) > 7f:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07) > 7f:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07) > 7f:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07) > 7f:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07) > 7f:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07) > 7f:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07) > 7f:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07) > 7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07) > 7f:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07) > 7f:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07) > 7f:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07) > 7f:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07) > 7f:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07) > 7f:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07) > 7f:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07) > 7f:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07) > 7f:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07) > 7f:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07) > 7f:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07) > 7f:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07) > 7f:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07) > 80:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07) > 80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07) > 80:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07) > 80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07) > 80:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 07) > 80:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07) > 80:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07) > 80:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07) > 80:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07) > 80:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07) > 80:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07) > 80:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07) > 80:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07) > 80:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07) > 80:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07) > 80:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07) > 82:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca) > 83:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca) > 83:08.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca) > 83:09.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ca) > 84:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01) > 84:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01) > 87:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01) > 87:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01) > 89:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba) > 8a:00.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba) > 8a:08.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba) > 8a:09.0 PCI bridge: PLX Technology, Inc. Device 8724 (rev ba) > 8b:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01) > 8b:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01) > 8e:00.0 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01) > 8e:00.1 Ethernet controller: Intel Corporation Ethernet Server Adapter X520-4 (rev 01) > ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07) > ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07) > ff:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07) > ff:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07) > ff:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07) > ff:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07) > ff:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07) > ff:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07) > ff:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > ff:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > ff:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > ff:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > ff:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07) > ff:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07) > ff:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > ff:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > ff:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > ff:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) > ff:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07) > ff:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07) > ff:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07) > ff:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07) > ff:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07) > ff:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07) > ff:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07) > ff:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07) > ff:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07) > ff:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07) > ff:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07) > ff:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07) > ff:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07) > ff:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07) > ff:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07) > ff:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07) > ff:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07) > ff:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07) > ff:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07) > ff:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07) > ff:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07) > ff:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07) > ff:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07) > ff:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07) > >