From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-00191d01.pphosted.com (mx0b-00191d01.pphosted.com [67.231.157.136]) by dpdk.org (Postfix) with ESMTP id 2A79C5A17 for ; Mon, 23 May 2016 15:39:37 +0200 (CEST) Received: from pps.filterd (m0049458.ppops.net [127.0.0.1]) by m0049458.ppops.net-00191d01. (8.16.0.11/8.16.0.11) with SMTP id u4NDcdPY010048 for ; Mon, 23 May 2016 09:39:36 -0400 Received: from alpi155.enaf.aldc.att.com (sbcsmtp7.sbc.com [144.160.229.24]) by m0049458.ppops.net-00191d01. with ESMTP id 23428bhuf8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 23 May 2016 09:39:36 -0400 Received: from enaf.aldc.att.com (localhost [127.0.0.1]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id u4NDdZP1001973 for ; Mon, 23 May 2016 09:39:36 -0400 Received: from mlpi409.sfdc.sbc.com (mlpi409.sfdc.sbc.com [130.9.128.241]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id u4NDdUf2001901 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 23 May 2016 09:39:31 -0400 Received: from MISOUT7MSGHUBAA.ITServices.sbc.com (MISOUT7MSGHUBAA.itservices.sbc.com [130.9.129.145]) by mlpi409.sfdc.sbc.com (RSA Interceptor) for ; Mon, 23 May 2016 13:39:21 GMT Received: from MISOUT7MSGUSRCB.ITServices.sbc.com ([169.254.2.178]) by MISOUT7MSGHUBAA.ITServices.sbc.com ([130.9.129.145]) with mapi id 14.03.0294.000; Mon, 23 May 2016 09:39:20 -0400 From: "DING, TAO" To: "dev@dpdk.org" Thread-Topic: unable to bind to vfio_pci inside RHEL virtual machine. Thread-Index: AdG0+IHv3eOweEDTR0WRQEoXHURIrg== Date: Mon, 23 May 2016 13:39:20 +0000 Message-ID: <7BC361771D33CB489E50B2EC34311AA001913829@MISOUT7MSGUSRCB.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.44.219.95] MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2016-05-23_05:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1604210000 definitions=main-1605230164 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] unable to bind to vfio_pci inside RHEL virtual machine. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 May 2016 13:39:37 -0000 Hello dpdk dev, Do you know if the vfio_pci can be bind to network interface from within Re= dHat virtual machine ? I read the doc that igb_uio should not be used ; it= is not stable. (http://people.redhat.com/~pmatilai/dpdk-guide/index.html) = however I cannot use vfio_pci driver from inside VM. Currently I am working on a project that migrating a network package captur= e application into virtual machines so that it can be hosted on cloud. My = intent is using SR-IOR to ensure the data sending from physical NIC to vNIC= in line speed; using DPDK inside VM to read data from vNIC to get good pe= rformance because the libpcap does not perform well inside VM. Following dpdk instruction, I was able to set up the SR-IOV and bind the vf= io_pci to virtual Function on the host. Once the VM starts, the Virtual Fun= ctions bind to vfio-pci automatically on the host. . The following is the o= utput from host. Option: 22 Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:04:10.4 'X540 Ethernet Controller Virtual Function' drv=3Dvfio-pci unu= sed=3D 0000:04:10.6 'X540 Ethernet Controller Virtual Function' drv=3Dvfio-pci unu= sed=3D 0000:04:11.4 'X540 Ethernet Controller Virtual Function' drv=3Dvfio-pci unu= sed=3D 0000:04:11.6 'X540 Ethernet Controller Virtual Function' drv=3Dvfio-pci unu= sed=3D Network devices using kernel driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:01:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=3Dem1 drv=3Dtg3 u= nused=3Dvfio-pci 0000:01:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=3Dem2 drv=3Dtg3 u= nused=3Dvfio-pci 0000:02:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=3Dem3 drv=3Dtg3 u= nused=3Dvfio-pci 0000:02:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=3Dem4 drv=3Dtg3 u= nused=3Dvfio-pci 0000:04:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=3Dp3p1 drv=3Dixgb= e unused=3Dvfio-pci *Active* 0000:04:00.1 'Ethernet Controller 10-Gigabit X540-AT2' if=3Dp3p2 drv=3Dixgb= e unused=3Dvfio-pci 0000:04:10.0 'X540 Ethernet Controller Virtual Function' if=3Dp3p1_0 drv=3D= ixgbevf unused=3Dvfio-pci 0000:04:10.2 'X540 Ethernet Controller Virtual Function' if=3Dp3p1_1 drv=3D= ixgbevf unused=3Dvfio-pci 0000:04:11.0 'X540 Ethernet Controller Virtual Function' if=3Dp3p1_4 drv=3D= ixgbevf unused=3Dvfio-pci 0000:04:11.2 'X540 Ethernet Controller Virtual Function' if=3Dp3p1_5 drv=3D= ixgbevf unused=3Dvfio-pci I repeated the same set up within the VM which has 4 Virtual Functions assi= gned to it, I could not successfully bind any of network devices to vfio-p= ci. I followed different suggestion from the web , but no luck. (however = I was able to bind UIO driver to the network devices inside the VM) One difference I noticed between VM and host is the outcome of IOMMU settin= g. On the host, the /sys/kernel/iommu_groups/ is NOT empty , but on the VM= , it is empty. I rebooted VM several time. There's not luck. The following is the output from inside the VM. The driver vfio_pci is vis= ible to lsmod , but not visible to driverctl. [root@hn14vm3 tools]# driverctl -v list-devices | grep -i net 0000:00:03.0 ixgbevf (X540 Ethernet Controller Virtual Function) 0000:00:08.0 ixgbevf (X540 Ethernet Controller Virtual Function) 0000:00:09.0 ixgbevf (X540 Ethernet Controller Virtual Function) 0000:00:0b.0 ixgbevf (X540 Ethernet Controller Virtual Function) 0000:00:0c.0 e1000 (82540EM Gigabit Ethernet Controller (QEMU Virtual Machi= ne)) 0000:00:0d.0 e1000 (82540EM Gigabit Ethernet Controller (QEMU Virtual Machi= ne)) [root@hn14vm3 tools]# lsmod |grep uio igb_uio 13224 0 uio 19259 1 igb_uio [root@hn14vm3 tools]# lsmod |grep vfio vfio_pci 36735 0 vfio_iommu_type1 17632 0 vfio 25291 2 vfio_iommu_type1,vfio_pci [root@hn14vm3 tools]# driverctl set-override 0000:00:03.0 vfio_pci driverctl: failed to bind device 0000:00:03.0 to driver vfio_pci [root@hn14vm3 tools]# driverctl set-override 0000:00:03.0 igb_uio [root@hn14vm3 tools]# driverctl -v list-devices | grep -i net 0000:00:03.0 igb_uio [*] (X540 Ethernet Controller Virtual Function) 0000:00:08.0 ixgbevf (X540 Ethernet Controller Virtual Function) 0000:00:09.0 ixgbevf (X540 Ethernet Controller Virtual Function) 0000:00:0b.0 ixgbevf (X540 Ethernet Controller Virtual Function) 0000:00:0c.0 e1000 (82540EM Gigabit Ethernet Controller (QEMU Virtual Machi= ne)) 0000:00:0d.0 e1000 (82540EM Gigabit Ethernet Controller (QEMU Virtual Machi= ne)) Here are some info about host and vm. OS, -- both host and VM , redhat enterprise 7.2. Dell R630 with intel 10GB= , (and one 1GB interface). Dpdk version -- dpdk-16.04. VM has 4 cores and 10G RAM. VM created with virt-manager , used the default setting except the "IDE Di= sk 1/Advanced options/Performance options/Cache mode chose writeback" Any pointers would be appreciated Thanks a lot for your time. Tao Ding