From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 6DD229A8E for ; Mon, 18 May 2015 12:06:15 +0200 (CEST) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP; 18 May 2015 03:06:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.13,452,1427785200"; d="scan'208";a="727672355" Received: from pgsmsx106.gar.corp.intel.com ([10.221.44.98]) by fmsmga002.fm.intel.com with ESMTP; 18 May 2015 03:06:10 -0700 Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by PGSMSX106.gar.corp.intel.com (10.221.44.98) with Microsoft SMTP Server (TLS) id 14.3.224.2; Mon, 18 May 2015 18:01:37 +0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.120]) by SHSMSX152.ccr.corp.intel.com ([169.254.6.50]) with mapi id 14.03.0224.002; Mon, 18 May 2015 18:01:14 +0800 From: "Qiu, Michael" To: "Assaad, Sami (Sami)" , "Richardson, Bruce" Thread-Topic: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)? Thread-Index: AQHQj0W3XBeRDnTDh0KVnJLaxShRaw== Date: Mon, 18 May 2015 10:01:14 +0000 Message-ID: <533710CFB86FA344BFBF2D6802E602860467EA7E@SHSMSX101.ccr.corp.intel.com> References: <9478F0FB69DAA249AF0A9BDA1E6ED95218817AB9@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150514164719.7b68b0ef@urahara> <20150515092719.GA1520@bricha3-MOBL3> <9478F0FB69DAA249AF0A9BDA1E6ED95218818129@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150515130804.GB5884@bricha3-MOBL3> <9478F0FB69DAA249AF0A9BDA1E6ED9521881848E@US70TWXCHMBA07.zam.alcatel-lucent.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 May 2015 10:06:16 -0000 Hi, Sami=0A= =0A= Could you mind to supply the syslog? Especially iommu related parts.=0A= =0A= Also you could update the qemu or kernel to see if this issue still exists.= =0A= =0A= =0A= Thanks,=0A= Michael=0A= =0A= On 5/16/2015 3:31 AM, Assaad, Sami (Sami) wrote:=0A= > On Fri, May 15, 2015 at 12:54:19PM +0000, Assaad, Sami (Sami) wrote:=0A= >> Thanks Bruce for your reply.=0A= >>=0A= >> Yes, your idea of bringing the PF into the VM looks like an option. Howe= ver, how do you configure the physical interfaces within the VM supporting = SRIOV?=0A= >> I always believed that the VM needed to be associated with a virtual/emu= lated interface card. With your suggestion, I would actually configure the = physical interface card/non-emulated within the VM.=0A= >>=0A= >> If you could provide me some example configuration commands, it would be= really appreciated. =0A= >>=0A= > You'd pass in the PF in the same way as the VF, just skip all the steps c= reating the VF on the host. To the system and hypervisor, both are just PCI= devices!=0A= >=0A= > As for configuration, the setup and configuration of the PF in the guest = is exactly the same as on the host - it's the same hardware with the same P= CI bars.=0A= > It's the IOMMU on your platform that takes care of memory isolation and a= ddress translation and that should work with either PF or VF.=0A= >=0A= > Regards,=0A= > /Bruce=0A= >=0A= >> Thanks in advance.=0A= >>=0A= >> Best Regards,=0A= >> Sami.=0A= >>=0A= >> -----Original Message-----=0A= >> From: Bruce Richardson [mailto:bruce.richardson@intel.com]=0A= >> Sent: Friday, May 15, 2015 5:27 AM=0A= >> To: Stephen Hemminger=0A= >> Cc: Assaad, Sami (Sami); dev@dpdk.org=0A= >> Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using = PCI Pass-Through (SR-IOV)?=0A= >>=0A= >> On Thu, May 14, 2015 at 04:47:19PM -0700, Stephen Hemminger wrote:=0A= >>> On Thu, 14 May 2015 21:38:24 +0000=0A= >>> "Assaad, Sami (Sami)" wrote:=0A= >>>=0A= >>>> Hello,=0A= >>>>=0A= >>>> My Hardware consists of the following:=0A= >>>> - DL380 Gen 9 Server supporting two Haswell Processors (Xeon CPU E5-= 2680 v3 @ 2.50GHz)=0A= >>>> - An x540 Ethernet Controller Card supporting 2x10G ports.=0A= >>>>=0A= >>>> Software:=0A= >>>> - CentOS 7 (3.10.0-229.1.2.el7.x86_64)=0A= >>>> - DPDK 1.8=0A= >>>>=0A= >>>> I want all the network traffic received on the two 10G ports to be tra= nsmitted to my VM. The issue is that the Virtual Function / Physical Functi= ons have setup the internal virtual switch to only route Ethernet packets w= ith destination MAC address matching the VM virtual interface MAC. How can = I configure my virtual environment to provide all network traffic to the VM= ...i.e. set the virtual functions for both PCI devices in Promiscuous mode?= =0A= >>>>=0A= >>>> [ If a l2fwd-vf example exists, this would actually solve this =0A= >>>> problem ... Is there a DPDK l2fwd-vf example available? ]=0A= >>>>=0A= >>>>=0A= >>>> Thanks in advance.=0A= >>>>=0A= >>>> Best Regards,=0A= >>>> Sami Assaad.=0A= >>> This is a host side (not DPDK) issue.=0A= >>>=0A= >>> Intel PF driver will not allow guest (VF) to go into promiscious =0A= >>> mode since it would allow traffic stealing which is a security violatio= n.=0A= >> Could you maybe try passing the PF directly into the VM, rather than a V= F based off it? Since you seem to want all traffic to go to the one VM, the= re seems little point in creating a VF on the device, and should let the VM= control the whole NIC directly.=0A= >>=0A= >> Regards,=0A= >> /Bruce=0A= >=0A= > Hi Bruce, =0A= >=0A= > I was provided two options:=0A= > 1. Pass the PF directly into the VM=0A= > 2. Use ixgbe VF mirroring=0A= >=0A= > I decided to first try your proposal of passing the PF directly into the = VM. However, I ran into some issues. =0A= > But prior to providing the problem details, the following is my server e= nvironment:=0A= > I'm using CentOS 7 KVM/QEMU=0A= > [root@ni-nfvhost01 qemu]# uname -a=0A= > Linux ni-nfvhost01 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 U= TC 2015 x86_64 x86_64 x86_64 GNU/Linux=0A= >=0A= > [root@ni-nfvhost01 qemu]# lspci -n -s 04:00.0=0A= > 04:00.0 0200: 8086:1528 (rev 01)=0A= >=0A= > [root@ni-nfvhost01 qemu]# lspci | grep -i eth=0A= > 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigab= it Ethernet PCIe (rev 01)=0A= > 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigab= it Ethernet PCIe (rev 01)=0A= > 02:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigab= it Ethernet PCIe (rev 01)=0A= > 02:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigab= it Ethernet PCIe (rev 01)=0A= > 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gig= abit X540-AT2 (rev 01)=0A= > 04:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gig= abit X540-AT2 (rev 01)=0A= >=0A= > - The following is my grub execution:=0A= > [root@ni-nfvhost01 qemu]# cat /proc/cmdline =0A= > BOOT_IMAGE=3D/vmlinuz-3.10.0-229.1.2.el7.x86_64 root=3D/dev/mapper/centos= -root ro rd.lvm.lv=3Dcentos/swap vconsole.font=3Dlatarcyrheb-sun17 rd.lvm.l= v=3Dcentos/root crashkernel=3Dauto vconsole.keymap=3Dus rhgb quiet iommu=3D= pt intel_iommu=3Don hugepages=3D8192=0A= >=0A= >=0A= > This is the error I'm obtaining when the VM has one of the PCI devices as= sociated to the Ethernet Controller card:=0A= > [root@ni-nfvhost01 qemu]# qemu-system-x86_64 -m 2048 -vga std -vnc :0 -ne= t none -enable-kvm -device vfio-pci,host=3D04:00.0,id=3Dnet0=0A= > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: fail= ed to set iommu for container: Operation not permitted=0A= > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: fail= ed to setup container for group 19=0A= > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: fail= ed to get group 19=0A= > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device ini= tialization failed.=0A= > qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device 'vf= io-pci' could not be initialized=0A= >=0A= > Hence, I tried the following, but again with no success :-( =0A= > Decided to bind the PCI device associated to the Ethernet Controller to = vfio (To enable the VM PCI device access and have the IOMMU operate properl= y)=0A= > Here are the commands I used to configure the PCI pass-through for the Et= hernet device:=0A= >=0A= > # modprobe vfio-pci=0A= >=0A= > 1) Device I want to assign as passthrough:=0A= > 04:00.0 =0A= >=0A= > 2) Find the vfio group of this device=0A= >=0A= > # readlink /sys/bus/pci/devices/0000:04:00.0/iommu_group=0A= > ../../../../kernel/iommu_groups/19=0A= > =0A= > ( IOMMU Group =3D 19 )=0A= >=0A= > 3) Check the devices in the group:=0A= > # ls /sys/bus/pci/devices/0000:04:00.0/iommu_group/devices/=0A= > 0000:04:00.0=0A= > =0A= > (so this group has only 1 device)=0A= > =0A= > 4) Unbind from device driver=0A= > # echo 0000:04:00.0 >/sys/bus/pci/devices/0000:04:00.0/driver/unbind=0A= > =0A= > 5) Find vendor & device ID=0A= > $ lspci -n -s 04:00.0=0A= >> 04:00.0 0200: 8086:1528 (rev 01)=0A= > =0A= > 6) Bind to vfio-pci=0A= > $ echo 8086 1528 > /sys/bus/pci/drivers/vfio-pci/new_id=0A= > =0A= > (this results in a new device node "/dev/vfio/19", which is what qemu wi= ll use to setup the device for passthrough)=0A= > =0A= > 7) chown the device node so it is accessible by qemu user:=0A= > # chown qemu /dev/vfio/19; chgrp qemu /dev/vfio/19=0A= >=0A= > Now, on the VM side, using virt-manager, I removed the initial PCI device= and re-added it.=0A= > After re-booting the VM, I obtained the same issue.=0A= >=0A= > What am I doing wrong?=0A= >=0A= > Thanks a million!=0A= >=0A= > Best Regards,=0A= > Sami.=0A= >=0A= >=0A= =0A=