From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 943D05A37 for ; Wed, 20 May 2015 12:56:48 +0200 (CEST) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga103.jf.intel.com with ESMTP; 20 May 2015 03:56:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.13,464,1427785200"; d="scan'208";a="496001163" Received: from smonroyx-mobl.ger.corp.intel.com (HELO [10.237.220.34]) ([10.237.220.34]) by FMSMGA003.fm.intel.com with ESMTP; 20 May 2015 03:56:45 -0700 Message-ID: <555C686C.3020300@intel.com> Date: Wed, 20 May 2015 11:56:44 +0100 From: "Gonzalez Monroy, Sergio" User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: "Assaad, Sami (Sami)" , "Qiu, Michael" , "Richardson, Bruce" References: <9478F0FB69DAA249AF0A9BDA1E6ED95218817AB9@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150514164719.7b68b0ef@urahara> <20150515092719.GA1520@bricha3-MOBL3> <9478F0FB69DAA249AF0A9BDA1E6ED95218818129@US70TWXCHMBA07.zam.alcatel-lucent.com> <20150515130804.GB5884@bricha3-MOBL3> <9478F0FB69DAA249AF0A9BDA1E6ED9521881848E@US70TWXCHMBA07.zam.alcatel-lucent.com> <533710CFB86FA344BFBF2D6802E602860467EA7E@SHSMSX101.ccr.corp.intel.com> <9478F0FB69DAA249AF0A9BDA1E6ED9521881BDE6@US70TWXCHMBA07.zam.alcatel-lucent.com> In-Reply-To: <9478F0FB69DAA249AF0A9BDA1E6ED9521881BDE6@US70TWXCHMBA07.zam.alcatel-lucent.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] How do you setup a VM in Promiscuous Mode using PCI Pass-Through (SR-IOV)? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 10:56:49 -0000 On 19/05/2015 20:23, Assaad, Sami (Sami) wrote: > Hello Michael, > > I've updated the kernel and QEMU. Here are the packages I'm using: > > --> CentOS 7 - 3.10.0-229.4.2.el7.x86_64 > - qemu-kvm-1.5.3-86.el7_1.2.x86_64 > - libvirt-1.2.8-16.el7_1.3.x86_64 > - virt-manager-1.1.0-12.el7.noarch > - virt-what-1.13-5.el7.x86_64 > - libvirt-glib-0.1.7-3.el7.x86_64 > > I've modified the virtual machine XML file to include the following: > > > > >
> > > > > >
> > > > > The syslog error I'm obtaining relating to the iommu is the following: > #dmesg | grep -e DMAR -e IOMMU > > [ 3362.370564] vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. > > > From the /var/log/messages file, the complete VM log is the following: > > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): carrier is OFF > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): new Tun device (driver: 'unknown' ifindex: 30) > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): exported as /org/freedesktop/NetworkManager/Devices/29 > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (virbr0): bridge port vnet0 was attached > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): enslaved to virbr0 > May 19 15:10:12 ni-nfvhost01 kernel: device vnet0 entered promiscuous mode > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): link connected > May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered listening state > May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered listening state > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device state change: unmanaged -> unavailable (reason 'connection-assumed') [10 20 41] > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device state change: unavailable -> disconnected (reason 'connection-assumed') [20 30 41] > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: starting connection 'vnet0' > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 1 of 5 (Device Prepare) scheduled... > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 1 of 5 (Device Prepare) started... > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device state change: disconnected -> prepare (reason 'none') [30 40 0] > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 2 of 5 (Device Configure) scheduled... > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 1 of 5 (Device Prepare) complete. > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 2 of 5 (Device Configure) starting... > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device state change: prepare -> config (reason 'none') [40 50 0] > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 2 of 5 (Device Configure) successful. > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 3 of 5 (IP Configure Start) scheduled. > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 2 of 5 (Device Configure) complete. > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 3 of 5 (IP Configure Start) started... > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device state change: config -> ip-config (reason 'none') [50 70 0] > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: Stage 3 of 5 (IP Configure Start) complete. > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device state change: ip-config -> secondaries (reason 'none') [70 90 0] > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device state change: secondaries -> activated (reason 'none') [90 100 0] > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): Activation: successful, device activated. > May 19 15:10:12 ni-nfvhost01 dbus-daemon: dbus[1295]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' > May 19 15:10:12 ni-nfvhost01 dbus[1295]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' > May 19 15:10:12 ni-nfvhost01 systemd: Starting Network Manager Script Dispatcher Service... > May 19 15:10:12 ni-nfvhost01 systemd: Starting Virtual Machine qemu-vNIDS-VM1. > May 19 15:10:12 ni-nfvhost01 systemd-machined: New machine qemu-vNIDS-VM1. > May 19 15:10:12 ni-nfvhost01 systemd: Started Virtual Machine qemu-vNIDS-VM1. > May 19 15:10:12 ni-nfvhost01 dbus-daemon: dbus[1295]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' > May 19 15:10:12 ni-nfvhost01 dbus[1295]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' > May 19 15:10:12 ni-nfvhost01 systemd: Started Network Manager Script Dispatcher Service. > May 19 15:10:12 ni-nfvhost01 nm-dispatcher: Dispatching action 'up' for vnet0 > May 19 15:10:12 ni-nfvhost01 kvm: 1 guest now active > May 19 15:10:12 ni-nfvhost01 systemd: Unit iscsi.service cannot be reloaded because it is inactive. > May 19 15:10:12 ni-nfvhost01 kernel: vfio-pci 0000:04:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. > May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered disabled state > May 19 15:10:12 ni-nfvhost01 kernel: device vnet0 left promiscuous mode > May 19 15:10:12 ni-nfvhost01 kernel: virbr0: port 2(vnet0) entered disabled state > May 19 15:10:12 ni-nfvhost01 avahi-daemon[1280]: Withdrawing workstation service for vnet0. > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): device state change: activated -> unmanaged (reason 'removed') [100 10 36] > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (vnet0): deactivating device (reason 'removed') [36] > May 19 15:10:12 ni-nfvhost01 NetworkManager[1371]: (virbr0): failed to detach bridge port vnet0 > May 19 15:10:12 ni-nfvhost01 nm-dispatcher: Dispatching action 'down' for vnet0 > May 19 15:10:12 ni-nfvhost01 journal: Unable to read from monitor: Connection reset by peer > May 19 15:10:12 ni-nfvhost01 journal: internal error: early end of file from monitor: possible problem: > 2015-05-19T19:10:12.674077Z qemu-kvm: -device vfio-pci,host=04:00.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to set iommu for container: Operation not permitted > 2015-05-19T19:10:12.674118Z qemu-kvm: -device vfio-pci,host=04:00.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to setup container for group 19 > 2015-05-19T19:10:12.674128Z qemu-kvm: -device vfio-pci,host=04:00.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to get group 19 > 2015-05-19T19:10:12.674141Z qemu-kvm: -device vfio-pci,host=04:00.0,id=hostdev0,bus=pci.0,addr=0x9: Device initialization failed. > 2015-05-19T19:10:12.674155Z qemu-kvm: -device vfio-pci,host=04:00.0,id=hostdev0,bus=pci.0,addr=0x9: Device 'vfio-pci' could not be initialized > > May 19 15:10:12 ni-nfvhost01 kvm: 0 guests now active > May 19 15:10:12 ni-nfvhost01 systemd-machined: Machine qemu-vNIDS-VM1 terminated. > May 19 15:11:01 ni-nfvhost01 systemd: Created slice user-0.slice. > May 19 15:11:01 ni-nfvhost01 systemd: Starting Session 329 of user root. > > > Overall Hypothesis: The issue seems to be related to the Ethernet Controller's interfaces which I'm trying to bring into the VM. My Ethernet Controller is : Intel 10G x540-AT2 (rev 01). > The problem is associated to RMRR. > Can this issue be attributed to my BIOS? My Bios is the following: ProLiant System BIOS P89 V1.21 11/03/2014. Hi Sami, I think the following answer from Lee Roberts could be helpful: http://dpdk.org/ml/archives/dev/2015-March/015504.html AFAIK, new Linux kernels (v3.8+) will not attach RMRR-affected devices to an IOMMU domain. commit ea2447f700cab264019b52e2b417d689e052dcfd Author: Tom Mingarelli Date: Tue Nov 20 19:43:17 2012 +0000 intel-iommu: Prevent devices with RMRRs from being placed into SI Domain This patch is to prevent non-USB devices that have RMRRs associated with the being placed into the SI Domain during init. This fixes the issue where the for devices being placed in and out of the SI Domain gets lost. Signed-off-by: Thomas Mingarelli Tested-by: Shuah Khan Reviewed-by: Donald Dutile Reviewed-by: Alex Williamson Signed-off-by: Joerg Roedel Sergio > Thanks in advance. > > Best Regards, > Sami. >