From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from BLU004-OMC4S14.hotmail.com (blu004-omc4s14.hotmail.com [65.55.111.153]) by dpdk.org (Postfix) with ESMTP id 482C658CB for ; Thu, 3 Sep 2015 00:18:28 +0200 (CEST) Received: from BLU179-W12 ([65.55.111.135]) by BLU004-OMC4S14.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Wed, 2 Sep 2015 15:18:27 -0700 X-TMN: [lyUr4VFVP6dbr3oxyHnOEC+LJadkG8fF] X-Originating-Email: [mansooraa@hotmail.com] Message-ID: From: Ale Mansoor To: "dev@dpdk.org" Date: Wed, 2 Sep 2015 22:18:27 +0000 Importance: Normal MIME-Version: 1.0 X-OriginalArrivalTime: 02 Sep 2015 22:18:27.0995 (UTC) FILETIME=[4AD00EB0:01D0E5CD] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] Poor SRIOV performance with ESXi Linux guest X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Sep 2015 22:18:28 -0000 Getting less than 100 packets per second throughput between VF's under my = Fedora FC20 VM running under ESXi 6.0 with DPDK l2fwd (Used as ./l2fwd -c 0= xf -n 4 -- -p 0x3 -T 1) =20 Questions: --------------- =20 Q1) Is DPDK + SRIOV under ESXi supposed to use the igb_uio driver or the vf= io-pci driver inside Linux guest os ? Q2) What is the expected l2fwd performance when running DPDK under the Linu= x guest OS under ESXI with SRIOV ? Q3) Any idea what may be preventing vfio-pci driver from binding to the VF'= s inside the guest instance ? Q4) Why is igb_uio performing so poorly ? =20 =20 =20 My hardware setup: --------------------------- HP DL380-G8 host with dual Xeon-2670 CPU 64 GB RAM Intel 82599 NIC connected to 10G switch HP onboard bnx2 NIC's. =20 VMware Hypervisor ESXi 6.0: --------------------------------------- On above hardware running ESXi 6.0 hypervisor and Intel ixgbe VMware (drive= r version: 3.21.4.3iov).=20 =20 ESXi side SRIOV is setup as follows: "ixgbe_enabled =3D 1 options =3D max_v= fs=3D0=2C8=2C8=2C0" (Total 16 VF's configured) =20 VF's are pass-through to the two guest instances.=20 =20 First 4 VF's to one vm instance and next 2 VF's to second vm instance=2C re= maining VF's are unused=2C have also tried with single guest instances havi= ng 4 VF's and remaining unused=2C but it did not make any difference =20 =20 My VMware guest instances: ---------------------------------------- =20 Have tried two different guests: =20 Linux guest setup each with 8 cores and 8GB memory (memory is setup as lock= ed down to vm to help with iommu). =20 =20 1) Fedora FC20 OS (Linux kernel 3.19.8-100.fc20)=20 2) Ubuntu 15.04 Server distro (Linux kernel 3.19.0-15)=2C =20 On both these guests systems my Linux kernel boot line is set with "intel_i= ommu=3Don and iommu=3Dpt" =20 My kernel command info for Fedora guest is as follows: =20 =20 $cat /proc/cmdline --------------------------- BOOT_IMAGE=3D/vmlinuz-3.19.8-100.fc20.x86_64 root=3DUUID=3Df0cde7fc-d835-4d= 90-b086-82bf88f58f88 ro console=3Dtty0 console=3DttyS0=2C9600 rhgb quiet ne= t.ifnames=3D0 biosdevname=3D0 intel_iommu=3Don iommu=3Dpt =20 My dmesg has message: =20 "Intel-IOMMU: enabled" =20 Kernel IOMMU config: ------------------------------ =20 cat /boot/config-3.19.8-100.fc20.x86_64 |grep -i iommu # CONFIG_GART_IOMMU is not set # CONFIG_CALGARY_IOMMU is not set CONFIG_IOMMU_HELPER=3Dy CONFIG_VFIO_IOMMU_TYPE1=3Dm CONFIG_IOMMU_API=3Dy CONFIG_IOMMU_SUPPORT=3Dy CONFIG_AMD_IOMMU=3Dy CONFIG_AMD_IOMMU_STATS=3Dy CONFIG_AMD_IOMMU_V2=3Dm CONFIG_INTEL_IOMMU=3Dy # CONFIG_INTEL_IOMMU_DEFAULT_ON is not set CONFIG_INTEL_IOMMU_FLOPPY_WA=3Dy # CONFIG_IOMMU_STRESS is not set =20 cat /boot/config-3.19.8-100.fc20.x86_64 |grep -i VIRTUAL CONFIG_FB_VIRTUAL=3Dm # CONFIG_DEBUG_VIRTUAL is not set CONFIG_VIRTUALIZATION=3Dy =20 ./dpdk_nic_bind.py --status ---------------------------------------- Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:03:00.0 '82599 Ethernet Controller Virtual Function' drv=3Digb_uio unu= sed=3Dixgbevf=2Cvfio-pci 0000:0b:00.0 '82599 Ethernet Controller Virtual Function' drv=3Digb_uio unu= sed=3Dixgbevf=2Cvfio-pci 0000:13:00.0 '82599 Ethernet Controller Virtual Function' drv=3Digb_uio unu= sed=3Dixgbevf=2Cvfio-pci 0000:1b:00.0 '82599 Ethernet Controller Virtual Function' drv=3Digb_uio unu= sed=3Dixgbevf=2Cvfio-pci Network devices using kernel driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=3Deth4 drv= =3De1000 unused=3Digb_uio=2Cvfio-pci *Active* Other network devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 The l2fw program is run with: ./l2fwd -c 0xf -n 4 -- -p 0x3 -T 1 =20 [Using the default igb_uio driver shipped with Fedora Fc20] =20 Getting around 70 packets per second with these settings: =20 When I tried using the vfio-pci driver (again using the default vfio-pci dr= iver from Fedora FC20)=2C get the following errors: =20 ./dpdk_nic_bind.py --bind vfio-pci 0000:03:00.0 Error: bind failed for 0000:03:00.0 - Cannot bind to driver vfio-pci =20 dmesg shows: [ 882.685134] vfio-pci: probe of 0000:03:00.0 failed with error -22 =20 My guest side lspci shows:=20 ----------------------------------- =20 03:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Vi= rtual Function (rev 01) 0b:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Vi= rtual Function (rev 01) 13:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Vi= rtual Function (rev 01) 1b:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Vi= rtual Function (rev 01) =20 My ESXi side VMware lscpi shows: ---------------------------------------------- =20 0000:09:00.0 Network controller: Intel Corporation 82599 10 Gigabit Dual Po= rt Network Connection [vmnic4] 0000:09:00.1 Network controller: Intel Corporation 82599 10 Gigabit Dual Po= rt Network Connection [vmnic5] 0000:09:10.1 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.9.1_VF_0] 0000:09:10.3 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.9.1_VF_1] 0000:09:10.5 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.9.1_VF_2] 0000:09:10.7 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.9.1_VF_3] 0000:09:11.1 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.9.1_VF_4] 0000:09:11.3 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.9.1_VF_5] 0000:09:11.5 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.9.1_VF_6] 0000:09:11.7 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.9.1_VF_7] 0000:0c:00.0 Network controller: Intel Corporation 82599 10 Gigabit Dual Po= rt Network Connection [vmnic6] 0000:0c:00.1 Network controller: Intel Corporation 82599 10 Gigabit Dual Po= rt Network Connection [vmnic7] 0000:0c:10.0 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.12.0_VF_0] 0000:0c:10.2 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.12.0_VF_1] 0000:0c:10.4 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.12.0_VF_2] 0000:0c:10.6 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.12.0_VF_3] 0000:0c:11.0 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.12.0_VF_4] 0000:0c:11.2 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.12.0_VF_5] 0000:0c:11.4 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.12.0_VF_6] 0000:0c:11.6 Network controller: Intel Corporation 82599 Ethernet Controlle= r Virtual Function [PF_0.12.0_VF_7] =20 --- Regards =20 Ale =20 =20 =20 =20 =