From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-fr.alcatel-lucent.com (fr-hpida-esg-02.alcatel-lucent.com [135.245.210.21]) by dpdk.org (Postfix) with ESMTP id 064895A38 for ; Fri, 15 May 2015 21:31:00 +0200 (CEST) Received: from us70uusmtp3.zam.alcatel-lucent.com (unknown [135.5.2.65]) by Websense Email Security Gateway with ESMTPS id 20D49E74F837A; Fri, 15 May 2015 19:30:54 +0000 (GMT) Received: from US70TWXCHHUB04.zam.alcatel-lucent.com (us70twxchhub04.zam.alcatel-lucent.com [135.5.2.36]) by us70uusmtp3.zam.alcatel-lucent.com (GMO) with ESMTP id t4FJUuir009691 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Fri, 15 May 2015 15:30:56 -0400 Received: from US70TWXCHMBA07.zam.alcatel-lucent.com ([169.254.1.180]) by US70TWXCHHUB04.zam.alcatel-lucent.com ([135.5.2.36]) with mapi id 14.03.0195.001; Fri, 15 May 2015 15:30:56 -0400 From: "Assaad, Sami (Sami)" To: Bruce Richardson Thread-Topic: How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)? Thread-Index: AQHQj0Wp9qgMWyC0x0GEwx0p0WM3Qg== Date: Fri, 15 May 2015 19:30:56 +0000 Message-ID: <9478F0FB69DAA249AF0A9BDA1E6ED9521881848E@US70TWXCHMBA07.zam.alcatel-lucent.com> References: <9478F0FB69DAA249AF0A9BDA1E6ED95218817AB9@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150514164719.7b68b0ef@urahara> <20150515092719.GA1520@bricha3-MOBL3> <9478F0FB69DAA249AF0A9BDA1E6ED95218818129@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150515130804.GB5884@bricha3-MOBL3> In-Reply-To: <20150515130804.GB5884@bricha3-MOBL3> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.5.27.17] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 May 2015 19:31:00 -0000 On Fri, May 15, 2015 at 12:54:19PM +0000, Assaad, Sami (Sami) wrote: > Thanks Bruce for your reply. >=20 > Yes, your idea of bringing the PF into the VM looks like an option. Howev= er, how do you configure the physical interfaces within the VM supporting S= RIOV? > I always believed that the VM needed to be associated with a virtual/emul= ated interface card. With your suggestion, I would actually configure the p= hysical interface card/non-emulated within the VM. >=20 > If you could provide me some example configuration commands, it would be = really appreciated.=20 >=20 You'd pass in the PF in the same way as the VF, just skip all the steps cre= ating the VF on the host. To the system and hypervisor, both are just PCI d= evices! As for configuration, the setup and configuration of the PF in the guest is= exactly the same as on the host - it's the same hardware with the same PCI= bars. It's the IOMMU on your platform that takes care of memory isolation and add= ress translation and that should work with either PF or VF. Regards, /Bruce > Thanks in advance. >=20 > Best Regards, > Sami. >=20 > -----Original Message----- > From: Bruce Richardson [mailto:bruce.richardson@intel.com] > Sent: Friday, May 15, 2015 5:27 AM > To: Stephen Hemminger > Cc: Assaad, Sami (Sami); dev@dpdk.org > Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using P= CI Pass-Through (SR-IOV)? >=20 > On Thu, May 14, 2015 at 04:47:19PM -0700, Stephen Hemminger wrote: > > On Thu, 14 May 2015 21:38:24 +0000 > > "Assaad, Sami (Sami)" wrote: > >=20 > > > Hello, > > >=20 > > > My Hardware consists of the following: > > > - DL380 Gen 9 Server supporting two Haswell Processors (Xeon CPU E5= -2680 v3 @ 2.50GHz) > > > - An x540 Ethernet Controller Card supporting 2x10G ports. > > >=20 > > > Software: > > > - CentOS 7 (3.10.0-229.1.2.el7.x86_64) > > > - DPDK 1.8 > > >=20 > > > I want all the network traffic received on the two 10G ports to be tr= ansmitted to my VM. The issue is that the Virtual Function / Physical Funct= ions have setup the internal virtual switch to only route Ethernet packets = with destination MAC address matching the VM virtual interface MAC. How can= I configure my virtual environment to provide all network traffic to the V= M...i.e. set the virtual functions for both PCI devices in Promiscuous mode= ? > > >=20 > > > [ If a l2fwd-vf example exists, this would actually solve this=20 > > > problem ... Is there a DPDK l2fwd-vf example available? ] > > >=20 > > >=20 > > > Thanks in advance. > > >=20 > > > Best Regards, > > > Sami Assaad. > >=20 > > This is a host side (not DPDK) issue. > >=20 > > Intel PF driver will not allow guest (VF) to go into promiscious=20 > > mode since it would allow traffic stealing which is a security violatio= n. >=20 > Could you maybe try passing the PF directly into the VM, rather than a VF= based off it? Since you seem to want all traffic to go to the one VM, ther= e seems little point in creating a VF on the device, and should let the VM = control the whole NIC directly. >=20 > Regards, > /Bruce Hi Bruce,=20 I was provided two options: 1. Pass the PF directly into the VM 2. Use ixgbe VF mirroring I decided to first try your proposal of passing the PF directly into the VM= . However, I ran into some issues.=20 But prior to providing the problem details, the following is my server env= ironment: I'm using CentOS 7 KVM/QEMU [root@ni-nfvhost01 qemu]# uname -a Linux ni-nfvhost01 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC= 2015 x86_64 x86_64 x86_64 GNU/Linux [root@ni-nfvhost01 qemu]# lspci -n -s 04:00.0 04:00.0 0200: 8086:1528 (rev 01) [root@ni-nfvhost01 qemu]# lspci | grep -i eth 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit= Ethernet PCIe (rev 01) 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit= Ethernet PCIe (rev 01) 02:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit= Ethernet PCIe (rev 01) 02:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit= Ethernet PCIe (rev 01) 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigab= it X540-AT2 (rev 01) 04:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigab= it X540-AT2 (rev 01) - The following is my grub execution: [root@ni-nfvhost01 qemu]# cat /proc/cmdline=20 BOOT_IMAGE=3D/vmlinuz-3.10.0-229.1.2.el7.x86_64 root=3D/dev/mapper/centos-r= oot ro rd.lvm.lv=3Dcentos/swap vconsole.font=3Dlatarcyrheb-sun17 rd.lvm.lv= =3Dcentos/root crashkernel=3Dauto vconsole.keymap=3Dus rhgb quiet iommu=3Dp= t intel_iommu=3Don hugepages=3D8192 This is the error I'm obtaining when the VM has one of the PCI devices asso= ciated to the Ethernet Controller card: [root@ni-nfvhost01 qemu]# qemu-system-x86_64 -m 2048 -vga std -vnc :0 -net = none -enable-kvm -device vfio-pci,host=3D04:00.0,id=3Dnet0 qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: failed= to set iommu for container: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: failed= to setup container for group 19 qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: failed= to get group 19 qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device initi= alization failed. qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device 'vfio= -pci' could not be initialized Hence, I tried the following, but again with no success :-(=20 Decided to bind the PCI device associated to the Ethernet Controller to vf= io (To enable the VM PCI device access and have the IOMMU operate properly) Here are the commands I used to configure the PCI pass-through for the Ethe= rnet device: # modprobe vfio-pci 1) Device I want to assign as passthrough: 04:00.0=20 2) Find the vfio group of this device # readlink /sys/bus/pci/devices/0000:04:00.0/iommu_group ../../../../kernel/iommu_groups/19 =20 ( IOMMU Group =3D 19 ) 3) Check the devices in the group: # ls /sys/bus/pci/devices/0000:04:00.0/iommu_group/devices/ 0000:04:00.0 =20 (so this group has only 1 device) =20 4) Unbind from device driver # echo 0000:04:00.0 >/sys/bus/pci/devices/0000:04:00.0/driver/unbind =20 5) Find vendor & device ID $ lspci -n -s 04:00.0 > 04:00.0 0200: 8086:1528 (rev 01) =20 6) Bind to vfio-pci $ echo 8086 1528 > /sys/bus/pci/drivers/vfio-pci/new_id =20 (this results in a new device node "/dev/vfio/19", which is what qemu will= use to setup the device for passthrough) =20 7) chown the device node so it is accessible by qemu user: # chown qemu /dev/vfio/19; chgrp qemu /dev/vfio/19 Now, on the VM side, using virt-manager, I removed the initial PCI device a= nd re-added it. After re-booting the VM, I obtained the same issue. What am I doing wrong? Thanks a million! Best Regards, Sami.