From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id 2BA6729D6 for ; Fri, 2 Feb 2018 11:28:38 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 02 Feb 2018 02:28:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,448,1511856000"; d="scan'208";a="200806953" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.237.220.145]) ([10.237.220.145]) by fmsmga006.fm.intel.com with ESMTP; 02 Feb 2018 02:28:37 -0800 To: Ravi Kerur Cc: dev@dpdk.org References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <10911b54-57ee-370b-a4f7-f34accf4811e@intel.com> <3464b900-8648-c128-e959-dc60a8883a2d@intel.com> From: "Burakov, Anatoly" Message-ID: <0120c68f-cf42-5d8b-5600-0514f76209b0@intel.com> Date: Fri, 2 Feb 2018 10:28:36 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Feb 2018 10:28:39 -0000 On 01-Feb-18 7:26 PM, Ravi Kerur wrote: > > > On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly > > wrote: > > On 31-Jan-18 9:51 PM, Ravi Kerur wrote: > > > Hi Anatoly, > > Thanks. I am following wiki link below which uses vIOMMU with > DPDK as a use-case and instantiate VM as specified with Q35 > chipset in Qemu. > > https://wiki.qemu.org/Features/VT-d > > > Qemu-version is 2.11 > Host kernel 4.9 > Guest kernel 4.4 > > I can only guess that guest kernel needs an upgrade in my setup > to work correctly, if versions on my setup rings a bell on not > having support kindly let me know. > > When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on > guest I get following error > ... > vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored > ... > > in guest. > > Thanks. > > > AFAIK kernel 4.4 should have noiommu mode - it was introduced in > 3.1x days. However, in order for that to work, kernel also has to be > built with this mode enabled. My guess is, whoever is the supplier > of your kernel, did not do that. You should double-check the kernel > configuration of your distribution. > > However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode > - "regular" vfio should work fine. noiommu mode should only be > needed if you know you don't have IOMMU enabled in your kernel, and > even if you can't enable it, you can still use igb_uio. > > Hi Anatoly, > > Do you suggest I take this discussion to kvm/qemu mailing list as I am > not sure which component has the issue? I check dmesg for BIOS physical > memory map and address reported as fault by DMAR is reported by BIOS as > usable on both host and vm. > > [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr > *33a128000 *[fault reason 06] PTE Read access is not set > > dmesg | grep BIOS > [    0.000000] e820: BIOS-provided physical RAM map: > [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] usable > [    0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] > reserved > [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] > reserved > [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] usable > [    0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] > reserved > [    0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] > ACPI data > [    0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] > ACPI NVS > [    0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] > reserved > [    0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] usable > [    0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] > reserved > [    0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] usable > [    0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] > reserved > [    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] > reserved > [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] > reserved > [*    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] > usable* > * > * > Kindly let me know your inputs. > > Thanks. > > > -- > Thanks, > Anatoly > > The "PTE Read not set" error usually indicates that you are trying to use a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio when IOMMU is on). That, to me, indicates that you do have IOMMU emulation enabled. I would go about it this way. First, i'd ensure that your VM has IOMMU emulation enabled and working. You have mentioned that your QEMU version should have IOMMU emulation, so let's assume that's the case. I am not sure of the exact command-line needed to activate the vIOMMU emulation, but assuming your VM emulates an Intel processor, your kernel command-line should have "iommu=on intel_iommu=on" in it. Check /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are not in there, add the above changes, do "update-grub" and reboot your VM. If it already did have the necessary kernel configuration, do "dmesg | grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU is enabled and working in the kernel. After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it should be working. Please bear in mind that all of that is how i would've gone about it if i had similar problems on baremetal, but i'm hoping all of it is applicable to VM's. So, either disable IOMMU and use igb_uio, or enable IOMMU and use VFIO. Both will work. -- Thanks, Anatoly