Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)?

["Qiu, Michael" <michael.qiu@intel.com>]

Hi, Sami

Could you mind to supply the syslog? Especially iommu related parts.

Also you could update the qemu or kernel to see if this issue still exists.


Thanks,
Michael

On 5/16/2015 3:31 AM, Assaad, Sami (Sami) wrote:
> On Fri, May 15, 2015 at 12:54:19PM +0000, Assaad, Sami (Sami) wrote:
>> Thanks Bruce for your reply.
>>
>> Yes, your idea of bringing the PF into the VM looks like an option. However, how do you configure the physical interfaces within the VM supporting SRIOV?
>> I always believed that the VM needed to be associated with a virtual/emulated interface card. With your suggestion, I would actually configure the physical interface card/non-emulated within the VM.
>>
>> If you could provide me some example configuration commands, it would be really appreciated. 
>>
> You'd pass in the PF in the same way as the VF, just skip all the steps creating the VF on the host. To the system and hypervisor, both are just PCI devices!
>
> As for configuration, the setup and configuration of the PF in the guest is exactly the same as on the host - it's the same hardware with the same PCI bars.
> It's the IOMMU on your platform that takes care of memory isolation and address translation and that should work with either PF or VF.
>
> Regards,
> /Bruce
>
>> Thanks in advance.
>>
>> Best Regards,
>> Sami.
>>
>> -----Original Message-----
>> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
>> Sent: Friday, May 15, 2015 5:27 AM
>> To: Stephen Hemminger
>> Cc: Assaad, Sami (Sami); dev@dpdk.org
>> Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)?
>>
>> On Thu, May 14, 2015 at 04:47:19PM -0700, Stephen Hemminger wrote:
>>> On Thu, 14 May 2015 21:38:24 +0000
>>> "Assaad, Sami (Sami)" <sami.assaad@alcatel-lucent.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> My Hardware consists of the following:
>>>>   - DL380 Gen 9 Server supporting two Haswell Processors (Xeon CPU E5-2680 v3 @ 2.50GHz)
>>>>   - An x540 Ethernet Controller Card supporting 2x10G ports.
>>>>
>>>> Software:
>>>>   - CentOS 7 (3.10.0-229.1.2.el7.x86_64)
>>>>   - DPDK 1.8
>>>>
>>>> I want all the network traffic received on the two 10G ports to be transmitted to my VM. The issue is that the Virtual Function / Physical Functions have setup the internal virtual switch to only route Ethernet packets with destination MAC address matching the VM virtual interface MAC. How can I configure my virtual environment to provide all network traffic to the VM...i.e. set the virtual functions for both PCI devices in Promiscuous mode?
>>>>
>>>> [ If a l2fwd-vf example exists, this would actually solve this 
>>>> problem ... Is there a DPDK l2fwd-vf example available? ]
>>>>
>>>>
>>>> Thanks in advance.
>>>>
>>>> Best Regards,
>>>> Sami Assaad.
>>> This is a host side (not DPDK) issue.
>>>
>>> Intel PF driver will not allow guest (VF) to go into promiscious 
>>> mode since it would allow traffic stealing which is a security violation.
>> Could you maybe try passing the PF directly into the VM, rather than a VF based off it? Since you seem to want all traffic to go to the one VM, there seems little point in creating a VF on the device, and should let the VM control the whole NIC directly.
>>
>> Regards,
>> /Bruce
>
> Hi Bruce, 
>
> I was provided two options:
> 1. Pass the PF directly into the VM
> 2. Use ixgbe VF mirroring
>
> I decided to first try your proposal of passing the PF directly into the VM. However, I ran into some issues. 
> But prior to providing the problem details, the following is my  server environment:
> I'm using CentOS 7 KVM/QEMU
> [root@ni-nfvhost01 qemu]# uname -a
> Linux ni-nfvhost01 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> [root@ni-nfvhost01 qemu]# lspci -n -s 04:00.0
> 04:00.0 0200: 8086:1528 (rev 01)
>
> [root@ni-nfvhost01 qemu]# lspci | grep -i eth
> 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
> 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
> 02:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
> 02:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
> 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
> 04:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
>
> - The following is my grub execution:
> [root@ni-nfvhost01 qemu]# cat  /proc/cmdline 
> BOOT_IMAGE=/vmlinuz-3.10.0-229.1.2.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun17 rd.lvm.lv=centos/root crashkernel=auto vconsole.keymap=us rhgb quiet iommu=pt intel_iommu=on hugepages=8192
>
>
> This is the error I'm obtaining when the VM has one of the PCI devices associated to the Ethernet Controller card:
> [root@ni-nfvhost01 qemu]# qemu-system-x86_64 -m 2048 -vga std -vnc :0 -net none -enable-kvm -device vfio-pci,host=04:00.0,id=net0
> qemu-system-x86_64: -device vfio-pci,host=04:00.0,id=net0: vfio: failed to set iommu for container: Operation not permitted
> qemu-system-x86_64: -device vfio-pci,host=04:00.0,id=net0: vfio: failed to setup container for group 19
> qemu-system-x86_64: -device vfio-pci,host=04:00.0,id=net0: vfio: failed to get group 19
> qemu-system-x86_64: -device vfio-pci,host=04:00.0,id=net0: Device initialization failed.
> qemu-system-x86_64: -device vfio-pci,host=04:00.0,id=net0: Device 'vfio-pci' could not be initialized
>
> Hence, I tried the following, but again with no success :-( 
> Decided to bind the  PCI device associated to the Ethernet Controller to vfio (To enable the VM PCI device access and have the IOMMU operate properly)
> Here are the commands I used to configure the PCI pass-through for the Ethernet device:
>
> # modprobe vfio-pci
>
> 1) Device I want to assign as passthrough:
> 04:00.0 
>
> 2) Find the vfio group of this device
>
> # readlink /sys/bus/pci/devices/0000:04:00.0/iommu_group
> ../../../../kernel/iommu_groups/19
>  
> ( IOMMU Group = 19 )
>
> 3) Check the devices in the group:
> # ls /sys/bus/pci/devices/0000:04:00.0/iommu_group/devices/
> 0000:04:00.0
>  
> (so this group has only 1 device)
>  
> 4) Unbind from device driver
> # echo 0000:04:00.0 >/sys/bus/pci/devices/0000:04:00.0/driver/unbind
>  
> 5) Find vendor & device ID
> $ lspci -n -s 04:00.0
>> 04:00.0 0200: 8086:1528 (rev 01)
>  
> 6) Bind to vfio-pci
> $ echo 8086 1528 > /sys/bus/pci/drivers/vfio-pci/new_id
>  
> (this results in a new device node "/dev/vfio/19",  which is what qemu will use to setup the device for passthrough)
>  
> 7) chown the device node so it is accessible by qemu user:
> # chown qemu /dev/vfio/19; chgrp qemu /dev/vfio/19
>
> Now, on the VM side, using virt-manager, I removed the initial PCI device and re-added it.
> After re-booting the VM, I obtained the same issue.
>
> What am I doing wrong?
>
> Thanks a million!
>
> Best Regards,
> Sami.
>
>