From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <michael.qiu@intel.com> Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 577889A8F for <dev@dpdk.org>; Wed, 20 May 2015 11:19:35 +0200 (CEST) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga102.jf.intel.com with ESMTP; 20 May 2015 02:19:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.13,464,1427785200"; d="scan'208";a="495965940" Received: from kmsmsx151.gar.corp.intel.com ([172.21.73.86]) by FMSMGA003.fm.intel.com with ESMTP; 20 May 2015 02:19:21 -0700 Received: from shsmsx104.ccr.corp.intel.com (10.239.4.70) by KMSMSX151.gar.corp.intel.com (172.21.73.86) with Microsoft SMTP Server (TLS) id 14.3.224.2; Wed, 20 May 2015 17:19:20 +0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.120]) by SHSMSX104.ccr.corp.intel.com ([169.254.5.94]) with mapi id 14.03.0224.002; Wed, 20 May 2015 17:19:18 +0800 From: "Qiu, Michael" <michael.qiu@intel.com> To: "Assaad, Sami (Sami)" <sami.assaad@alcatel-lucent.com>, "Richardson, Bruce" <bruce.richardson@intel.com> Thread-Topic: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)? Thread-Index: AQHQj0W3XBeRDnTDh0KVnJLaxShRaw== Date: Wed, 20 May 2015 09:19:19 +0000 Message-ID: <533710CFB86FA344BFBF2D6802E602860467F9C0@SHSMSX101.ccr.corp.intel.com> References: <9478F0FB69DAA249AF0A9BDA1E6ED95218817AB9@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150514164719.7b68b0ef@urahara> <20150515092719.GA1520@bricha3-MOBL3> <9478F0FB69DAA249AF0A9BDA1E6ED95218818129@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150515130804.GB5884@bricha3-MOBL3> <9478F0FB69DAA249AF0A9BDA1E6ED9521881848E@US70TWXCHMBA07.zam.alcatel-lucent.com> <533710CFB86FA344BFBF2D6802E602860467EA7E@SHSMSX101.ccr.corp.intel.com> <9478F0FB69DAA249AF0A9BDA1E6ED9521881BDE6@US70TWXCHMBA07.zam.alcatel-lucent.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" <dev@dpdk.org> Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Wed, 20 May 2015 09:19:37 -0000 =0A= =0A= 25 <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/Documenta= tion/Intel-IOMMU.txt#L25> What is RMRR?=0A= 26 <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/Documenta= tion/Intel-IOMMU.txt#L26> -------------=0A= 27 <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/Documenta= tion/Intel-IOMMU.txt#L27> =0A= 28 <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/Documenta= tion/Intel-IOMMU.txt#L28> There are some devices the BIOS controls, for e.g= USB devices to perform=0A= 29 <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/Documenta= tion/Intel-IOMMU.txt#L29> PS2 emulation. The regions of memory used for the= se devices are marked=0A= 30 <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/Documenta= tion/Intel-IOMMU.txt#L30> reserved in the e820 map. When we turn on DMA tra= nslation, DMA to those=0A= 31 <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/Documenta= tion/Intel-IOMMU.txt#L31> regions will fail. Hence BIOS uses RMRR to specif= y these regions along with=0A= 32 <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/Documenta= tion/Intel-IOMMU.txt#L32> devices that need to access these regions. OS is = expected to setup=0A= 33 <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/Documenta= tion/Intel-IOMMU.txt#L33> unity mappings for these regions for these device= s to access these regions.=0A= =0A= =0A= So what type of your NIC? Is on-board device or a plug in device?=0A= =0A= Thanks,=0A= Michael=0A= =0A= On 5/20/2015 3:24 AM, Assaad, Sami (Sami) wrote:=0A= > Hello Michael,=0A= >=0A= > I've updated the kernel and QEMU. Here are the packages I'm using:=0A= >=0A= > --> CentOS 7 - 3.10.0-229.4.2.el7.x86_64=0A= > - qemu-kvm-1.5.3-86.el7_1.2.x86_64=0A= > - libvirt-1.2.8-16.el7_1.3.x86_64=0A= > - virt-manager-1.1.0-12.el7.noarch=0A= > - virt-what-1.13-5.el7.x86_64=0A= > - libvirt-glib-0.1.7-3.el7.x86_64=0A= >=0A= > I've modified the virtual machine XML file to include the following:=0A= >=0A= > <hostdev mode=3D'subsystem' type=3D'pci' managed=3D'yes'>=0A= > <driver name=3D'vfio'/>=0A= > <source>=0A= > <address domain=3D'0x0000' bus=3D'0x04' slot=3D'0x10' function=3D'0= x0'/>=0A= > </source>=0A= > </hostdev>=0A= > <hostdev mode=3D'subsystem' type=3D'pci' managed=3D'yes'>=0A= > <driver name=3D'vfio'/>=0A= > <source>=0A= > <address domain=3D'0x0000' bus=3D'0x04' slot=3D'0x10' function=3D'0= x1'/>=0A= > </source>=0A= > </hostdev>=0A= >=0A= >=0A= > The syslog error I'm obtaining relating to the iommu is the following:=0A= > #dmesg | grep -e DMAR -e IOMMU=0A= >=0A= > [ 3362.370564] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU doma= in attach due to platform RMRR requirement. Contact your platform vendor.= =0A= >=0A= >=0A= > From the /var/log/messages file, the complete VM log is the following:=0A= >=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): carri= er is OFF=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): new T= un device (driver: 'unknown' ifindex: 30)=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): expor= ted as /org/freedesktop/NetworkManager/Devices/29=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (virbr0): brid= ge port vnet0 was attached=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): ensla= ved to virbr0=0A= > May 19 15:10:12 ni-nfvhost01 kernel: device vnet0 entered promiscuous mod= e=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): link = connected=0A= > May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered listen= ing state=0A= > May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered listen= ing state=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): devic= e state change: unmanaged -> unavailable (reason 'connection-assumed') [10 = 20 41]=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): devic= e state change: unavailable -> disconnected (reason 'connection-assumed') [= 20 30 41]=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: starting connection 'vnet0'=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 1 of 5 (Device Prepare) scheduled...=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 1 of 5 (Device Prepare) started...=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): devic= e state change: disconnected -> prepare (reason 'none') [30 40 0]=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 2 of 5 (Device Configure) scheduled...=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 1 of 5 (Device Prepare) complete.=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 2 of 5 (Device Configure) starting...=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): devic= e state change: prepare -> config (reason 'none') [40 50 0]=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 2 of 5 (Device Configure) successful.=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 3 of 5 (IP Configure Start) scheduled.=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 2 of 5 (Device Configure) complete.=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 3 of 5 (IP Configure Start) started...=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): devic= e state change: config -> ip-config (reason 'none') [50 70 0]=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: Stage 3 of 5 (IP Configure Start) complete.=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): devic= e state change: ip-config -> secondaries (reason 'none') [70 90 0]=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): devic= e state change: secondaries -> activated (reason 'none') [90 100 0]=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): Activ= ation: successful, device activated.=0A= > May 19 15:10:12 ni-nfvhost01 dbus-daemon: dbus[1295]: [system] Activating= via systemd: service name=3D'org.freedesktop.nm_dispatcher' unit=3D'dbus-o= rg.freedesktop.nm-dispatcher.service'=0A= > May 19 15:10:12 ni-nfvhost01 dbus[1295]: [system] Activating via systemd:= service name=3D'org.freedesktop.nm_dispatcher' unit=3D'dbus-org.freedeskto= p.nm-dispatcher.service'=0A= > May 19 15:10:12 ni-nfvhost01 systemd: Starting Network Manager Script Dis= patcher Service...=0A= > May 19 15:10:12 ni-nfvhost01 systemd: Starting Virtual Machine qemu-vNIDS= -VM1.=0A= > May 19 15:10:12 ni-nfvhost01 systemd-machined: New machine qemu-vNIDS-VM1= .=0A= > May 19 15:10:12 ni-nfvhost01 systemd: Started Virtual Machine qemu-vNIDS-= VM1.=0A= > May 19 15:10:12 ni-nfvhost01 dbus-daemon: dbus[1295]: [system] Successful= ly activated service 'org.freedesktop.nm_dispatcher'=0A= > May 19 15:10:12 ni-nfvhost01 dbus[1295]: [system] Successfully activated = service 'org.freedesktop.nm_dispatcher'=0A= > May 19 15:10:12 ni-nfvhost01 systemd: Started Network Manager Script Disp= atcher Service.=0A= > May 19 15:10:12 ni-nfvhost01 nm-dispatcher: Dispatching action 'up' for v= net0=0A= > May 19 15:10:12 ni-nfvhost01 kvm: 1 guest now active=0A= > May 19 15:10:12 ni-nfvhost01 systemd: Unit iscsi.service cannot be reload= ed because it is inactive.=0A= > May 19 15:10:12 ni-nfvhost01 kernel: vfio-pci 0000:04:00.0: Device is ine= ligible for IOMMU domain attach due to platform RMRR requirement. Contact = your platform vendor.=0A= > May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered disabl= ed state=0A= > May 19 15:10:12 ni-nfvhost01 kernel: device vnet0 left promiscuous mode= =0A= > May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered disabl= ed state=0A= > May 19 15:10:12 ni-nfvhost01 avahi-daemon[1280]: Withdrawing workstation = service for vnet0.=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): devic= e state change: activated -> unmanaged (reason 'removed') [100 10 36]=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <info> (vnet0): deact= ivating device (reason 'removed') [36]=0A= > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: <warn> (virbr0): fail= ed to detach bridge port vnet0=0A= > May 19 15:10:12 ni-nfvhost01 nm-dispatcher: Dispatching action 'down' for= vnet0=0A= > May 19 15:10:12 ni-nfvhost01 journal: Unable to read from monitor: Connec= tion reset by peer=0A= > May 19 15:10:12 ni-nfvhost01 journal: internal error: early end of file f= rom monitor: possible problem:=0A= > 2015-05-19T19:10:12.674077Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id= =3Dhostdev0,bus=3Dpci.0,addr=3D0x9: vfio: failed to set iommu for container= : Operation not permitted=0A= > 2015-05-19T19:10:12.674118Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id= =3Dhostdev0,bus=3Dpci.0,addr=3D0x9: vfio: failed to setup container for gro= up 19=0A= > 2015-05-19T19:10:12.674128Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id= =3Dhostdev0,bus=3Dpci.0,addr=3D0x9: vfio: failed to get group 19=0A= > 2015-05-19T19:10:12.674141Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id= =3Dhostdev0,bus=3Dpci.0,addr=3D0x9: Device initialization failed.=0A= > 2015-05-19T19:10:12.674155Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id= =3Dhostdev0,bus=3Dpci.0,addr=3D0x9: Device 'vfio-pci' could not be initiali= zed=0A= >=0A= > May 19 15:10:12 ni-nfvhost01 kvm: 0 guests now active=0A= > May 19 15:10:12 ni-nfvhost01 systemd-machined: Machine qemu-vNIDS-VM1 ter= minated.=0A= > May 19 15:11:01 ni-nfvhost01 systemd: Created slice user-0.slice.=0A= > May 19 15:11:01 ni-nfvhost01 systemd: Starting Session 329 of user root.= =0A= >=0A= >=0A= > Overall Hypothesis: The issue seems to be related to the Ethernet Control= ler's interfaces which I'm trying to bring into the VM. My Ethernet Control= ler is : Intel 10G x540-AT2 (rev 01).=0A= > The problem is associated to RMRR. =0A= > Can this issue be attributed to my BIOS? My Bios is t= he following: ProLiant System BIOS P89 V1.21 11/03/2014.=0A= >=0A= > Thanks in advance.=0A= >=0A= > Best Regards,=0A= > Sami.=0A= >=0A= > -----Original Message-----=0A= > From: Qiu, Michael [mailto:michael.qiu@intel.com] =0A= > Sent: Monday, May 18, 2015 6:01 AM=0A= > To: Assaad, Sami (Sami); Richardson, Bruce=0A= > Cc: dev@dpdk.org=0A= > Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using P= CI Pass-Through (SR-IOV)?=0A= >=0A= > Hi, Sami=0A= >=0A= > Could you mind to supply the syslog? Especially iommu related parts.=0A= >=0A= > Also you could update the qemu or kernel to see if this issue still exist= s.=0A= >=0A= >=0A= > Thanks,=0A= > Michael=0A= >=0A= > On 5/16/2015 3:31 AM, Assaad, Sami (Sami) wrote:=0A= >> On Fri, May 15, 2015 at 12:54:19PM +0000, Assaad, Sami (Sami) wrote:=0A= >>> Thanks Bruce for your reply.=0A= >>>=0A= >>> Yes, your idea of bringing the PF into the VM looks like an option. How= ever, how do you configure the physical interfaces within the VM supporting= SRIOV?=0A= >>> I always believed that the VM needed to be associated with a virtual/em= ulated interface card. With your suggestion, I would actually configure the= physical interface card/non-emulated within the VM.=0A= >>>=0A= >>> If you could provide me some example configuration commands, it would b= e really appreciated. =0A= >>>=0A= >> You'd pass in the PF in the same way as the VF, just skip all the steps = creating the VF on the host. To the system and hypervisor, both are just PC= I devices!=0A= >>=0A= >> As for configuration, the setup and configuration of the PF in the guest= is exactly the same as on the host - it's the same hardware with the same = PCI bars.=0A= >> It's the IOMMU on your platform that takes care of memory isolation and = address translation and that should work with either PF or VF.=0A= >>=0A= >> Regards,=0A= >> /Bruce=0A= >>=0A= >>> Thanks in advance.=0A= >>>=0A= >>> Best Regards,=0A= >>> Sami.=0A= >>>=0A= >>> -----Original Message-----=0A= >>> From: Bruce Richardson [mailto:bruce.richardson@intel.com]=0A= >>> Sent: Friday, May 15, 2015 5:27 AM=0A= >>> To: Stephen Hemminger=0A= >>> Cc: Assaad, Sami (Sami); dev@dpdk.org=0A= >>> Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using= PCI Pass-Through (SR-IOV)?=0A= >>>=0A= >>> On Thu, May 14, 2015 at 04:47:19PM -0700, Stephen Hemminger wrote:=0A= >>>> On Thu, 14 May 2015 21:38:24 +0000=0A= >>>> "Assaad, Sami (Sami)" <sami.assaad@alcatel-lucent.com> wrote:=0A= >>>>=0A= >>>>> Hello,=0A= >>>>>=0A= >>>>> My Hardware consists of the following:=0A= >>>>> - DL380 Gen 9 Server supporting two Haswell Processors (Xeon CPU E5= -2680 v3 @ 2.50GHz)=0A= >>>>> - An x540 Ethernet Controller Card supporting 2x10G ports.=0A= >>>>>=0A= >>>>> Software:=0A= >>>>> - CentOS 7 (3.10.0-229.1.2.el7.x86_64)=0A= >>>>> - DPDK 1.8=0A= >>>>>=0A= >>>>> I want all the network traffic received on the two 10G ports to be tr= ansmitted to my VM. The issue is that the Virtual Function / Physical Funct= ions have setup the internal virtual switch to only route Ethernet packets = with destination MAC address matching the VM virtual interface MAC. How can= I configure my virtual environment to provide all network traffic to the V= M...i.e. set the virtual functions for both PCI devices in Promiscuous mode= ?=0A= >>>>>=0A= >>>>> [ If a l2fwd-vf example exists, this would actually solve this =0A= >>>>> problem ... Is there a DPDK l2fwd-vf example available? ]=0A= >>>>>=0A= >>>>>=0A= >>>>> Thanks in advance.=0A= >>>>>=0A= >>>>> Best Regards,=0A= >>>>> Sami Assaad.=0A= >>>> This is a host side (not DPDK) issue.=0A= >>>>=0A= >>>> Intel PF driver will not allow guest (VF) to go into promiscious =0A= >>>> mode since it would allow traffic stealing which is a security violati= on.=0A= >>> Could you maybe try passing the PF directly into the VM, rather than a = VF based off it? Since you seem to want all traffic to go to the one VM, th= ere seems little point in creating a VF on the device, and should let the V= M control the whole NIC directly.=0A= >>>=0A= >>> Regards,=0A= >>> /Bruce=0A= >> Hi Bruce,=0A= >>=0A= >> I was provided two options:=0A= >> 1. Pass the PF directly into the VM=0A= >> 2. Use ixgbe VF mirroring=0A= >>=0A= >> I decided to first try your proposal of passing the PF directly into the= VM. However, I ran into some issues. =0A= >> But prior to providing the problem details, the following is my server = environment:=0A= >> I'm using CentOS 7 KVM/QEMU=0A= >> [root@ni-nfvhost01 qemu]# uname -a=0A= >> Linux ni-nfvhost01 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 =0A= >> 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux=0A= >>=0A= >> [root@ni-nfvhost01 qemu]# lspci -n -s 04:00.0=0A= >> 04:00.0 0200: 8086:1528 (rev 01)=0A= >>=0A= >> [root@ni-nfvhost01 qemu]# lspci | grep -i eth=0A= >> 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 =0A= >> Gigabit Ethernet PCIe (rev 01)=0A= >> 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 =0A= >> Gigabit Ethernet PCIe (rev 01)=0A= >> 02:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 =0A= >> Gigabit Ethernet PCIe (rev 01)=0A= >> 02:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 =0A= >> Gigabit Ethernet PCIe (rev 01)=0A= >> 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller =0A= >> 10-Gigabit X540-AT2 (rev 01)=0A= >> 04:00.1 Ethernet controller: Intel Corporation Ethernet Controller =0A= >> 10-Gigabit X540-AT2 (rev 01)=0A= >>=0A= >> - The following is my grub execution:=0A= >> [root@ni-nfvhost01 qemu]# cat /proc/cmdline=0A= >> BOOT_IMAGE=3D/vmlinuz-3.10.0-229.1.2.el7.x86_64 =0A= >> root=3D/dev/mapper/centos-root ro rd.lvm.lv=3Dcentos/swap =0A= >> vconsole.font=3Dlatarcyrheb-sun17 rd.lvm.lv=3Dcentos/root crashkernel=3D= auto =0A= >> vconsole.keymap=3Dus rhgb quiet iommu=3Dpt intel_iommu=3Don hugepages=3D= 8192=0A= >>=0A= >>=0A= >> This is the error I'm obtaining when the VM has one of the PCI devices a= ssociated to the Ethernet Controller card:=0A= >> [root@ni-nfvhost01 qemu]# qemu-system-x86_64 -m 2048 -vga std -vnc :0 = =0A= >> -net none -enable-kvm -device vfio-pci,host=3D04:00.0,id=3Dnet0=0A= >> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: =0A= >> failed to set iommu for container: Operation not permitted=0A= >> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: =0A= >> failed to setup container for group 19=0A= >> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: =0A= >> failed to get group 19=0A= >> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device in= itialization failed.=0A= >> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device = =0A= >> 'vfio-pci' could not be initialized=0A= >>=0A= >> Hence, I tried the following, but again with no success :-( Decided to = =0A= >> bind the PCI device associated to the Ethernet Controller to vfio (To = =0A= >> enable the VM PCI device access and have the IOMMU operate properly) Her= e are the commands I used to configure the PCI pass-through for the Etherne= t device:=0A= >>=0A= >> # modprobe vfio-pci=0A= >>=0A= >> 1) Device I want to assign as passthrough:=0A= >> 04:00.0=0A= >>=0A= >> 2) Find the vfio group of this device=0A= >>=0A= >> # readlink /sys/bus/pci/devices/0000:04:00.0/iommu_group=0A= >> ../../../../kernel/iommu_groups/19=0A= >> =0A= >> ( IOMMU Group =3D 19 )=0A= >>=0A= >> 3) Check the devices in the group:=0A= >> # ls /sys/bus/pci/devices/0000:04:00.0/iommu_group/devices/=0A= >> 0000:04:00.0=0A= >> =0A= >> (so this group has only 1 device)=0A= >> =0A= >> 4) Unbind from device driver=0A= >> # echo 0000:04:00.0 >/sys/bus/pci/devices/0000:04:00.0/driver/unbind=0A= >> =0A= >> 5) Find vendor & device ID=0A= >> $ lspci -n -s 04:00.0=0A= >>> 04:00.0 0200: 8086:1528 (rev 01)=0A= >> =0A= >> 6) Bind to vfio-pci=0A= >> $ echo 8086 1528 > /sys/bus/pci/drivers/vfio-pci/new_id=0A= >> =0A= >> (this results in a new device node "/dev/vfio/19", which is what qemu = =0A= >> will use to setup the device for passthrough)=0A= >> =0A= >> 7) chown the device node so it is accessible by qemu user:=0A= >> # chown qemu /dev/vfio/19; chgrp qemu /dev/vfio/19=0A= >>=0A= >> Now, on the VM side, using virt-manager, I removed the initial PCI devic= e and re-added it.=0A= >> After re-booting the VM, I obtained the same issue.=0A= >>=0A= >> What am I doing wrong?=0A= >>=0A= >> Thanks a million!=0A= >>=0A= >> Best Regards,=0A= >> Sami.=0A= >>=0A= >>=0A= >=0A= =0A=