From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-fr.alcatel-lucent.com (fr-hpida-esg-02.alcatel-lucent.com [135.245.210.21]) by dpdk.org (Postfix) with ESMTP id CE752592A for ; Tue, 19 May 2015 21:24:10 +0200 (CEST) Received: from us70uusmtp4.zam.alcatel-lucent.com (unknown [135.5.2.66]) by Websense Email Security Gateway with ESMTPS id 5CF073D2FFCA7; Tue, 19 May 2015 19:24:06 +0000 (GMT) Received: from US70UWXCHHUB02.zam.alcatel-lucent.com (us70uwxchhub02.zam.alcatel-lucent.com [135.5.2.49]) by us70uusmtp4.zam.alcatel-lucent.com (GMO) with ESMTP id t4JJNs1c005736 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Tue, 19 May 2015 15:24:03 -0400 Received: from US70TWXCHMBA07.zam.alcatel-lucent.com ([169.254.1.180]) by US70UWXCHHUB02.zam.alcatel-lucent.com ([135.5.2.49]) with mapi id 14.03.0195.001; Tue, 19 May 2015 15:23:42 -0400 From: "Assaad, Sami (Sami)" To: "Qiu, Michael" , "Richardson, Bruce" Thread-Topic: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)? Thread-Index: AQHQj0Wp9qgMWyC0x0GEwx0p0WM3Qp2DqjPQ Date: Tue, 19 May 2015 19:23:41 +0000 Message-ID: <9478F0FB69DAA249AF0A9BDA1E6ED9521881BDE6@US70TWXCHMBA07.zam.alcatel-lucent.com> References: <9478F0FB69DAA249AF0A9BDA1E6ED95218817AB9@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150514164719.7b68b0ef@urahara> <20150515092719.GA1520@bricha3-MOBL3> <9478F0FB69DAA249AF0A9BDA1E6ED95218818129@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150515130804.GB5884@bricha3-MOBL3> <9478F0FB69DAA249AF0A9BDA1E6ED9521881848E@US70TWXCHMBA07.zam.alcatel-lucent.com> <533710CFB86FA344BFBF2D6802E602860467EA7E@SHSMSX101.ccr.corp.intel.com> In-Reply-To: <533710CFB86FA344BFBF2D6802E602860467EA7E@SHSMSX101.ccr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.5.27.17] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2015 19:24:11 -0000 Hello Michael, I've updated the kernel and QEMU. Here are the packages I'm using: --> CentOS 7 - 3.10.0-229.4.2.el7.x86_64 - qemu-kvm-1.5.3-86.el7_1.2.x86_64 - libvirt-1.2.8-16.el7_1.3.x86_64 - virt-manager-1.1.0-12.el7.noarch - virt-what-1.13-5.el7.x86_64 - libvirt-glib-0.1.7-3.el7.x86_64 I've modified the virtual machine XML file to include the following:
The syslog error I'm obtaining relating to the iommu is the following: #dmesg | grep -e DMAR -e IOMMU [ 3362.370564] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain= attach due to platform RMRR requirement. Contact your platform vendor. >>From the /var/log/messages file, the complete VM log is the following: May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): carrier= is OFF May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): new Tun= device (driver: 'unknown' ifindex: 30) May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): exporte= d as /org/freedesktop/NetworkManager/Devices/29 May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (virbr0): bridge= port vnet0 was attached May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): enslave= d to virbr0 May 19 15:10:12 ni-nfvhost01 kernel: device vnet0 entered promiscuous mode May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): link co= nnected May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered listenin= g state May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered listenin= g state May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device = state change: unmanaged -> unavailable (reason 'connection-assumed') [10 20= 41] May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device = state change: unavailable -> disconnected (reason 'connection-assumed') [20= 30 41] May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: starting connection 'vnet0' May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 1 of 5 (Device Prepare) scheduled... May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 1 of 5 (Device Prepare) started... May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device = state change: disconnected -> prepare (reason 'none') [30 40 0] May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 2 of 5 (Device Configure) scheduled... May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 1 of 5 (Device Prepare) complete. May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 2 of 5 (Device Configure) starting... May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device = state change: prepare -> config (reason 'none') [40 50 0] May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 2 of 5 (Device Configure) successful. May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 3 of 5 (IP Configure Start) scheduled. May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 2 of 5 (Device Configure) complete. May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 3 of 5 (IP Configure Start) started... May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device = state change: config -> ip-config (reason 'none') [50 70 0] May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: Stage 3 of 5 (IP Configure Start) complete. May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device = state change: ip-config -> secondaries (reason 'none') [70 90 0] May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device = state change: secondaries -> activated (reason 'none') [90 100 0] May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activat= ion: successful, device activated. May 19 15:10:12 ni-nfvhost01 dbus-daemon: dbus[1295]: [system] Activating v= ia systemd: service name=3D'org.freedesktop.nm_dispatcher' unit=3D'dbus-org= .freedesktop.nm-dispatcher.service' May 19 15:10:12 ni-nfvhost01 dbus[1295]: [system] Activating via systemd: s= ervice name=3D'org.freedesktop.nm_dispatcher' unit=3D'dbus-org.freedesktop.= nm-dispatcher.service' May 19 15:10:12 ni-nfvhost01 systemd: Starting Network Manager Script Dispa= tcher Service... May 19 15:10:12 ni-nfvhost01 systemd: Starting Virtual Machine qemu-vNIDS-V= M1. May 19 15:10:12 ni-nfvhost01 systemd-machined: New machine qemu-vNIDS-VM1. May 19 15:10:12 ni-nfvhost01 systemd: Started Virtual Machine qemu-vNIDS-VM= 1. May 19 15:10:12 ni-nfvhost01 dbus-daemon: dbus[1295]: [system] Successfully= activated service 'org.freedesktop.nm_dispatcher' May 19 15:10:12 ni-nfvhost01 dbus[1295]: [system] Successfully activated se= rvice 'org.freedesktop.nm_dispatcher' May 19 15:10:12 ni-nfvhost01 systemd: Started Network Manager Script Dispat= cher Service. May 19 15:10:12 ni-nfvhost01 nm-dispatcher: Dispatching action 'up' for vne= t0 May 19 15:10:12 ni-nfvhost01 kvm: 1 guest now active May 19 15:10:12 ni-nfvhost01 systemd: Unit iscsi.service cannot be reloaded= because it is inactive. May 19 15:10:12 ni-nfvhost01 kernel: vfio-pci 0000:04:00.0: Device is ineli= gible for IOMMU domain attach due to platform RMRR requirement. Contact yo= ur platform vendor. May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered disabled= state May 19 15:10:12 ni-nfvhost01 kernel: device vnet0 left promiscuous mode May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered disabled= state May 19 15:10:12 ni-nfvhost01 avahi-daemon[1280]: Withdrawing workstation se= rvice for vnet0. May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device = state change: activated -> unmanaged (reason 'removed') [100 10 36] May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): deactiv= ating device (reason 'removed') [36] May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (virbr0): failed= to detach bridge port vnet0 May 19 15:10:12 ni-nfvhost01 nm-dispatcher: Dispatching action 'down' for v= net0 May 19 15:10:12 ni-nfvhost01 journal: Unable to read from monitor: Connecti= on reset by peer May 19 15:10:12 ni-nfvhost01 journal: internal error: early end of file fro= m monitor: possible problem: 2015-05-19T19:10:12.674077Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id=3D= hostdev0,bus=3Dpci.0,addr=3D0x9: vfio: failed to set iommu for container: O= peration not permitted 2015-05-19T19:10:12.674118Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id=3D= hostdev0,bus=3Dpci.0,addr=3D0x9: vfio: failed to setup container for group = 19 2015-05-19T19:10:12.674128Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id=3D= hostdev0,bus=3Dpci.0,addr=3D0x9: vfio: failed to get group 19 2015-05-19T19:10:12.674141Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id=3D= hostdev0,bus=3Dpci.0,addr=3D0x9: Device initialization failed. 2015-05-19T19:10:12.674155Z qemu-kvm: -device vfio-pci,host=3D04:00.0,id=3D= hostdev0,bus=3Dpci.0,addr=3D0x9: Device 'vfio-pci' could not be initialized May 19 15:10:12 ni-nfvhost01 kvm: 0 guests now active May 19 15:10:12 ni-nfvhost01 systemd-machined: Machine qemu-vNIDS-VM1 termi= nated. May 19 15:11:01 ni-nfvhost01 systemd: Created slice user-0.slice. May 19 15:11:01 ni-nfvhost01 systemd: Starting Session 329 of user root. Overall Hypothesis: The issue seems to be related to the Ethernet Controlle= r's interfaces which I'm trying to bring into the VM. My Ethernet Controlle= r is : Intel 10G x540-AT2 (rev 01). The problem is associated to RMRR.=20 Can this issue be attributed to my BIOS? My Bios is the= following: ProLiant System BIOS P89 V1.21 11/03/2014. Thanks in advance. Best Regards, Sami. -----Original Message----- From: Qiu, Michael [mailto:michael.qiu@intel.com]=20 Sent: Monday, May 18, 2015 6:01 AM To: Assaad, Sami (Sami); Richardson, Bruce Cc: dev@dpdk.org Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI= Pass-Through (SR-IOV)? Hi, Sami Could you mind to supply the syslog? Especially iommu related parts. Also you could update the qemu or kernel to see if this issue still exists. Thanks, Michael On 5/16/2015 3:31 AM, Assaad, Sami (Sami) wrote: > On Fri, May 15, 2015 at 12:54:19PM +0000, Assaad, Sami (Sami) wrote: >> Thanks Bruce for your reply. >> >> Yes, your idea of bringing the PF into the VM looks like an option. Howe= ver, how do you configure the physical interfaces within the VM supporting = SRIOV? >> I always believed that the VM needed to be associated with a virtual/emu= lated interface card. With your suggestion, I would actually configure the = physical interface card/non-emulated within the VM. >> >> If you could provide me some example configuration commands, it would be= really appreciated.=20 >> > You'd pass in the PF in the same way as the VF, just skip all the steps c= reating the VF on the host. To the system and hypervisor, both are just PCI= devices! > > As for configuration, the setup and configuration of the PF in the guest = is exactly the same as on the host - it's the same hardware with the same P= CI bars. > It's the IOMMU on your platform that takes care of memory isolation and a= ddress translation and that should work with either PF or VF. > > Regards, > /Bruce > >> Thanks in advance. >> >> Best Regards, >> Sami. >> >> -----Original Message----- >> From: Bruce Richardson [mailto:bruce.richardson@intel.com] >> Sent: Friday, May 15, 2015 5:27 AM >> To: Stephen Hemminger >> Cc: Assaad, Sami (Sami); dev@dpdk.org >> Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using = PCI Pass-Through (SR-IOV)? >> >> On Thu, May 14, 2015 at 04:47:19PM -0700, Stephen Hemminger wrote: >>> On Thu, 14 May 2015 21:38:24 +0000 >>> "Assaad, Sami (Sami)" wrote: >>> >>>> Hello, >>>> >>>> My Hardware consists of the following: >>>> - DL380 Gen 9 Server supporting two Haswell Processors (Xeon CPU E5-= 2680 v3 @ 2.50GHz) >>>> - An x540 Ethernet Controller Card supporting 2x10G ports. >>>> >>>> Software: >>>> - CentOS 7 (3.10.0-229.1.2.el7.x86_64) >>>> - DPDK 1.8 >>>> >>>> I want all the network traffic received on the two 10G ports to be tra= nsmitted to my VM. The issue is that the Virtual Function / Physical Functi= ons have setup the internal virtual switch to only route Ethernet packets w= ith destination MAC address matching the VM virtual interface MAC. How can = I configure my virtual environment to provide all network traffic to the VM= ...i.e. set the virtual functions for both PCI devices in Promiscuous mode? >>>> >>>> [ If a l2fwd-vf example exists, this would actually solve this=20 >>>> problem ... Is there a DPDK l2fwd-vf example available? ] >>>> >>>> >>>> Thanks in advance. >>>> >>>> Best Regards, >>>> Sami Assaad. >>> This is a host side (not DPDK) issue. >>> >>> Intel PF driver will not allow guest (VF) to go into promiscious=20 >>> mode since it would allow traffic stealing which is a security violatio= n. >> Could you maybe try passing the PF directly into the VM, rather than a V= F based off it? Since you seem to want all traffic to go to the one VM, the= re seems little point in creating a VF on the device, and should let the VM= control the whole NIC directly. >> >> Regards, >> /Bruce > > Hi Bruce, > > I was provided two options: > 1. Pass the PF directly into the VM > 2. Use ixgbe VF mirroring > > I decided to first try your proposal of passing the PF directly into the = VM. However, I ran into some issues.=20 > But prior to providing the problem details, the following is my server e= nvironment: > I'm using CentOS 7 KVM/QEMU > [root@ni-nfvhost01 qemu]# uname -a > Linux ni-nfvhost01 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27=20 > 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux > > [root@ni-nfvhost01 qemu]# lspci -n -s 04:00.0 > 04:00.0 0200: 8086:1528 (rev 01) > > [root@ni-nfvhost01 qemu]# lspci | grep -i eth > 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719=20 > Gigabit Ethernet PCIe (rev 01) > 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719=20 > Gigabit Ethernet PCIe (rev 01) > 02:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719=20 > Gigabit Ethernet PCIe (rev 01) > 02:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719=20 > Gigabit Ethernet PCIe (rev 01) > 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller=20 > 10-Gigabit X540-AT2 (rev 01) > 04:00.1 Ethernet controller: Intel Corporation Ethernet Controller=20 > 10-Gigabit X540-AT2 (rev 01) > > - The following is my grub execution: > [root@ni-nfvhost01 qemu]# cat /proc/cmdline > BOOT_IMAGE=3D/vmlinuz-3.10.0-229.1.2.el7.x86_64=20 > root=3D/dev/mapper/centos-root ro rd.lvm.lv=3Dcentos/swap=20 > vconsole.font=3Dlatarcyrheb-sun17 rd.lvm.lv=3Dcentos/root crashkernel=3Da= uto=20 > vconsole.keymap=3Dus rhgb quiet iommu=3Dpt intel_iommu=3Don hugepages=3D8= 192 > > > This is the error I'm obtaining when the VM has one of the PCI devices as= sociated to the Ethernet Controller card: > [root@ni-nfvhost01 qemu]# qemu-system-x86_64 -m 2048 -vga std -vnc :0=20 > -net none -enable-kvm -device vfio-pci,host=3D04:00.0,id=3Dnet0 > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio:=20 > failed to set iommu for container: Operation not permitted > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio:=20 > failed to setup container for group 19 > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio:=20 > failed to get group 19 > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device ini= tialization failed. > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device=20 > 'vfio-pci' could not be initialized > > Hence, I tried the following, but again with no success :-( Decided to=20 > bind the PCI device associated to the Ethernet Controller to vfio (To=20 > enable the VM PCI device access and have the IOMMU operate properly) Here= are the commands I used to configure the PCI pass-through for the Ethernet= device: > > # modprobe vfio-pci > > 1) Device I want to assign as passthrough: > 04:00.0 > > 2) Find the vfio group of this device > > # readlink /sys/bus/pci/devices/0000:04:00.0/iommu_group > ../../../../kernel/iommu_groups/19 > =20 > ( IOMMU Group =3D 19 ) > > 3) Check the devices in the group: > # ls /sys/bus/pci/devices/0000:04:00.0/iommu_group/devices/ > 0000:04:00.0 > =20 > (so this group has only 1 device) > =20 > 4) Unbind from device driver > # echo 0000:04:00.0 >/sys/bus/pci/devices/0000:04:00.0/driver/unbind > =20 > 5) Find vendor & device ID > $ lspci -n -s 04:00.0 >> 04:00.0 0200: 8086:1528 (rev 01) > =20 > 6) Bind to vfio-pci > $ echo 8086 1528 > /sys/bus/pci/drivers/vfio-pci/new_id > =20 > (this results in a new device node "/dev/vfio/19", which is what qemu=20 > will use to setup the device for passthrough) > =20 > 7) chown the device node so it is accessible by qemu user: > # chown qemu /dev/vfio/19; chgrp qemu /dev/vfio/19 > > Now, on the VM side, using virt-manager, I removed the initial PCI device= and re-added it. > After re-booting the VM, I obtained the same issue. > > What am I doing wrong? > > Thanks a million! > > Best Regards, > Sami. > >