From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <michael.qiu@intel.com>
Received: from mga02.intel.com (mga02.intel.com [134.134.136.20])
 by dpdk.org (Postfix) with ESMTP id 6DD229A8E
 for <dev@dpdk.org>; Mon, 18 May 2015 12:06:15 +0200 (CEST)
Received: from fmsmga002.fm.intel.com ([10.253.24.26])
 by orsmga101.jf.intel.com with ESMTP; 18 May 2015 03:06:13 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.13,452,1427785200"; d="scan'208";a="727672355"
Received: from pgsmsx106.gar.corp.intel.com ([10.221.44.98])
 by fmsmga002.fm.intel.com with ESMTP; 18 May 2015 03:06:10 -0700
Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by
 PGSMSX106.gar.corp.intel.com (10.221.44.98) with Microsoft SMTP Server (TLS)
 id 14.3.224.2; Mon, 18 May 2015 18:01:37 +0800
Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.120]) by
 SHSMSX152.ccr.corp.intel.com ([169.254.6.50]) with mapi id 14.03.0224.002;
 Mon, 18 May 2015 18:01:14 +0800
From: "Qiu, Michael" <michael.qiu@intel.com>
To: "Assaad, Sami (Sami)" <sami.assaad@alcatel-lucent.com>, "Richardson,
 Bruce" <bruce.richardson@intel.com>
Thread-Topic: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI
 Pass-Through (SR-IOV)?
Thread-Index: AQHQj0W3XBeRDnTDh0KVnJLaxShRaw==
Date: Mon, 18 May 2015 10:01:14 +0000
Message-ID: <533710CFB86FA344BFBF2D6802E602860467EA7E@SHSMSX101.ccr.corp.intel.com>
References: <9478F0FB69DAA249AF0A9BDA1E6ED95218817AB9@US70TWXCHMBA07.zam.alcatel-lucent.com>
 <20150514164719.7b68b0ef@urahara> <20150515092719.GA1520@bricha3-MOBL3>
 <9478F0FB69DAA249AF0A9BDA1E6ED95218818129@US70TWXCHMBA07.zam.alcatel-lucent.com>
 <20150515130804.GB5884@bricha3-MOBL3>
 <9478F0FB69DAA249AF0A9BDA1E6ED9521881848E@US70TWXCHMBA07.zam.alcatel-lucent.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.239.127.40]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI
 Pass-Through (SR-IOV)?
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 18 May 2015 10:06:16 -0000

Hi, Sami=0A=
=0A=
Could you mind to supply the syslog? Especially iommu related parts.=0A=
=0A=
Also you could update the qemu or kernel to see if this issue still exists.=
=0A=
=0A=
=0A=
Thanks,=0A=
Michael=0A=
=0A=
On 5/16/2015 3:31 AM, Assaad, Sami (Sami) wrote:=0A=
> On Fri, May 15, 2015 at 12:54:19PM +0000, Assaad, Sami (Sami) wrote:=0A=
>> Thanks Bruce for your reply.=0A=
>>=0A=
>> Yes, your idea of bringing the PF into the VM looks like an option. Howe=
ver, how do you configure the physical interfaces within the VM supporting =
SRIOV?=0A=
>> I always believed that the VM needed to be associated with a virtual/emu=
lated interface card. With your suggestion, I would actually configure the =
physical interface card/non-emulated within the VM.=0A=
>>=0A=
>> If you could provide me some example configuration commands, it would be=
 really appreciated. =0A=
>>=0A=
> You'd pass in the PF in the same way as the VF, just skip all the steps c=
reating the VF on the host. To the system and hypervisor, both are just PCI=
 devices!=0A=
>=0A=
> As for configuration, the setup and configuration of the PF in the guest =
is exactly the same as on the host - it's the same hardware with the same P=
CI bars.=0A=
> It's the IOMMU on your platform that takes care of memory isolation and a=
ddress translation and that should work with either PF or VF.=0A=
>=0A=
> Regards,=0A=
> /Bruce=0A=
>=0A=
>> Thanks in advance.=0A=
>>=0A=
>> Best Regards,=0A=
>> Sami.=0A=
>>=0A=
>> -----Original Message-----=0A=
>> From: Bruce Richardson [mailto:bruce.richardson@intel.com]=0A=
>> Sent: Friday, May 15, 2015 5:27 AM=0A=
>> To: Stephen Hemminger=0A=
>> Cc: Assaad, Sami (Sami); dev@dpdk.org=0A=
>> Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using =
PCI Pass-Through (SR-IOV)?=0A=
>>=0A=
>> On Thu, May 14, 2015 at 04:47:19PM -0700, Stephen Hemminger wrote:=0A=
>>> On Thu, 14 May 2015 21:38:24 +0000=0A=
>>> "Assaad, Sami (Sami)" <sami.assaad@alcatel-lucent.com> wrote:=0A=
>>>=0A=
>>>> Hello,=0A=
>>>>=0A=
>>>> My Hardware consists of the following:=0A=
>>>>   - DL380 Gen 9 Server supporting two Haswell Processors (Xeon CPU E5-=
2680 v3 @ 2.50GHz)=0A=
>>>>   - An x540 Ethernet Controller Card supporting 2x10G ports.=0A=
>>>>=0A=
>>>> Software:=0A=
>>>>   - CentOS 7 (3.10.0-229.1.2.el7.x86_64)=0A=
>>>>   - DPDK 1.8=0A=
>>>>=0A=
>>>> I want all the network traffic received on the two 10G ports to be tra=
nsmitted to my VM. The issue is that the Virtual Function / Physical Functi=
ons have setup the internal virtual switch to only route Ethernet packets w=
ith destination MAC address matching the VM virtual interface MAC. How can =
I configure my virtual environment to provide all network traffic to the VM=
...i.e. set the virtual functions for both PCI devices in Promiscuous mode?=
=0A=
>>>>=0A=
>>>> [ If a l2fwd-vf example exists, this would actually solve this =0A=
>>>> problem ... Is there a DPDK l2fwd-vf example available? ]=0A=
>>>>=0A=
>>>>=0A=
>>>> Thanks in advance.=0A=
>>>>=0A=
>>>> Best Regards,=0A=
>>>> Sami Assaad.=0A=
>>> This is a host side (not DPDK) issue.=0A=
>>>=0A=
>>> Intel PF driver will not allow guest (VF) to go into promiscious =0A=
>>> mode since it would allow traffic stealing which is a security violatio=
n.=0A=
>> Could you maybe try passing the PF directly into the VM, rather than a V=
F based off it? Since you seem to want all traffic to go to the one VM, the=
re seems little point in creating a VF on the device, and should let the VM=
 control the whole NIC directly.=0A=
>>=0A=
>> Regards,=0A=
>> /Bruce=0A=
>=0A=
> Hi Bruce, =0A=
>=0A=
> I was provided two options:=0A=
> 1. Pass the PF directly into the VM=0A=
> 2. Use ixgbe VF mirroring=0A=
>=0A=
> I decided to first try your proposal of passing the PF directly into the =
VM. However, I ran into some issues. =0A=
> But prior to providing the problem details, the following is my  server e=
nvironment:=0A=
> I'm using CentOS 7 KVM/QEMU=0A=
> [root@ni-nfvhost01 qemu]# uname -a=0A=
> Linux ni-nfvhost01 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 U=
TC 2015 x86_64 x86_64 x86_64 GNU/Linux=0A=
>=0A=
> [root@ni-nfvhost01 qemu]# lspci -n -s 04:00.0=0A=
> 04:00.0 0200: 8086:1528 (rev 01)=0A=
>=0A=
> [root@ni-nfvhost01 qemu]# lspci | grep -i eth=0A=
> 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigab=
it Ethernet PCIe (rev 01)=0A=
> 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigab=
it Ethernet PCIe (rev 01)=0A=
> 02:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigab=
it Ethernet PCIe (rev 01)=0A=
> 02:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigab=
it Ethernet PCIe (rev 01)=0A=
> 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gig=
abit X540-AT2 (rev 01)=0A=
> 04:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gig=
abit X540-AT2 (rev 01)=0A=
>=0A=
> - The following is my grub execution:=0A=
> [root@ni-nfvhost01 qemu]# cat  /proc/cmdline =0A=
> BOOT_IMAGE=3D/vmlinuz-3.10.0-229.1.2.el7.x86_64 root=3D/dev/mapper/centos=
-root ro rd.lvm.lv=3Dcentos/swap vconsole.font=3Dlatarcyrheb-sun17 rd.lvm.l=
v=3Dcentos/root crashkernel=3Dauto vconsole.keymap=3Dus rhgb quiet iommu=3D=
pt intel_iommu=3Don hugepages=3D8192=0A=
>=0A=
>=0A=
> This is the error I'm obtaining when the VM has one of the PCI devices as=
sociated to the Ethernet Controller card:=0A=
> [root@ni-nfvhost01 qemu]# qemu-system-x86_64 -m 2048 -vga std -vnc :0 -ne=
t none -enable-kvm -device vfio-pci,host=3D04:00.0,id=3Dnet0=0A=
> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: fail=
ed to set iommu for container: Operation not permitted=0A=
> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: fail=
ed to setup container for group 19=0A=
> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: vfio: fail=
ed to get group 19=0A=
> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device ini=
tialization failed.=0A=
> qemu-system-x86_64: -device vfio-pci,host=3D04:00.0,id=3Dnet0: Device 'vf=
io-pci' could not be initialized=0A=
>=0A=
> Hence, I tried the following, but again with no success :-( =0A=
> Decided to bind the  PCI device associated to the Ethernet Controller to =
vfio (To enable the VM PCI device access and have the IOMMU operate properl=
y)=0A=
> Here are the commands I used to configure the PCI pass-through for the Et=
hernet device:=0A=
>=0A=
> # modprobe vfio-pci=0A=
>=0A=
> 1) Device I want to assign as passthrough:=0A=
> 04:00.0 =0A=
>=0A=
> 2) Find the vfio group of this device=0A=
>=0A=
> # readlink /sys/bus/pci/devices/0000:04:00.0/iommu_group=0A=
> ../../../../kernel/iommu_groups/19=0A=
>  =0A=
> ( IOMMU Group =3D 19 )=0A=
>=0A=
> 3) Check the devices in the group:=0A=
> # ls /sys/bus/pci/devices/0000:04:00.0/iommu_group/devices/=0A=
> 0000:04:00.0=0A=
>  =0A=
> (so this group has only 1 device)=0A=
>  =0A=
> 4) Unbind from device driver=0A=
> # echo 0000:04:00.0 >/sys/bus/pci/devices/0000:04:00.0/driver/unbind=0A=
>  =0A=
> 5) Find vendor & device ID=0A=
> $ lspci -n -s 04:00.0=0A=
>> 04:00.0 0200: 8086:1528 (rev 01)=0A=
>  =0A=
> 6) Bind to vfio-pci=0A=
> $ echo 8086 1528 > /sys/bus/pci/drivers/vfio-pci/new_id=0A=
>  =0A=
> (this results in a new device node "/dev/vfio/19",  which is what qemu wi=
ll use to setup the device for passthrough)=0A=
>  =0A=
> 7) chown the device node so it is accessible by qemu user:=0A=
> # chown qemu /dev/vfio/19; chgrp qemu /dev/vfio/19=0A=
>=0A=
> Now, on the VM side, using virt-manager, I removed the initial PCI device=
 and re-added it.=0A=
> After re-booting the VM, I obtained the same issue.=0A=
>=0A=
> What am I doing wrong?=0A=
>=0A=
> Thanks a million!=0A=
>=0A=
> Best Regards,=0A=
> Sami.=0A=
>=0A=
>=0A=
=0A=