From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id D68331B2D4 for ; Mon, 5 Feb 2018 11:01:45 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Feb 2018 02:01:44 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,464,1511856000"; d="scan'208";a="28848573" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.237.220.145]) ([10.237.220.145]) by orsmga001.jf.intel.com with ESMTP; 05 Feb 2018 02:01:42 -0800 To: Ravi Kerur Cc: dev@dpdk.org References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <10911b54-57ee-370b-a4f7-f34accf4811e@intel.com> <3464b900-8648-c128-e959-dc60a8883a2d@intel.com> <0120c68f-cf42-5d8b-5600-0514f76209b0@intel.com> From: "Burakov, Anatoly" Message-ID: <4e5ef551-b7b3-e12d-6254-b882bb952bbb@intel.com> Date: Mon, 5 Feb 2018 10:01:41 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Feb 2018 10:01:46 -0000 On 02-Feb-18 8:51 PM, Ravi Kerur wrote: > > > On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur > wrote: > > > > On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly > > wrote: > > On 01-Feb-18 7:26 PM, Ravi Kerur wrote: > > > > On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly > > >> wrote: > >     On 31-Jan-18 9:51 PM, Ravi Kerur wrote: > > >         Hi Anatoly, > >         Thanks. I am following wiki link below which uses > vIOMMU with >         DPDK as a use-case and instantiate VM as specified > with Q35 >         chipset in Qemu. > > https://wiki.qemu.org/Features/VT-d > >         > > >         Qemu-version is 2.11 >         Host kernel 4.9 >         Guest kernel 4.4 > >         I can only guess that guest kernel needs an upgrade > in my setup >         to work correctly, if versions on my setup rings a > bell on not >         having support kindly let me know. > >         When 'modprobe vfio enable_unsafe_noiommu_node=Y' > is executed on >         guest I get following error >         ... >         vfio: unknown parameter > 'enable_unsafe_noiommu_node' ignored >         ... > >         in guest. > >         Thanks. > > >     AFAIK kernel 4.4 should have noiommu mode - it was > introduced in >     3.1x days. However, in order for that to work, kernel > also has to be >     built with this mode enabled. My guess is, whoever is > the supplier >     of your kernel, did not do that. You should > double-check the kernel >     configuration of your distribution. > >     However, if you have vIOMMU in QEMU, you shouldn't need > noiommu mode >     - "regular" vfio should work fine. noiommu mode should > only be >     needed if you know you don't have IOMMU enabled in your > kernel, and >     even if you can't enable it, you can still use igb_uio. > > Hi Anatoly, > > Do you suggest I take this discussion to kvm/qemu mailing > list as I am not sure which component has the issue? I check > dmesg for BIOS physical memory map and address reported as > fault by DMAR is reported by BIOS as usable on both host and vm. > > [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] > fault addr *33a128000 *[fault reason 06] PTE Read access is > not set > > dmesg | grep BIOS > [    0.000000] e820: BIOS-provided physical RAM map: > [    0.000000] BIOS-e820: [mem > 0x0000000000000000-0x000000000009afff] usable > [    0.000000] BIOS-e820: [mem > 0x000000000009b000-0x000000000009ffff] reserved > [    0.000000] BIOS-e820: [mem > 0x00000000000e0000-0x00000000000fffff] reserved > [    0.000000] BIOS-e820: [mem > 0x0000000000100000-0x000000007938afff] usable > [    0.000000] BIOS-e820: [mem > 0x000000007938b000-0x000000007994bfff] reserved > [    0.000000] BIOS-e820: [mem > 0x000000007994c000-0x000000007999cfff] ACPI data > [    0.000000] BIOS-e820: [mem > 0x000000007999d000-0x0000000079f7dfff] ACPI NVS > [    0.000000] BIOS-e820: [mem > 0x0000000079f7e000-0x000000007bd37fff] reserved > [    0.000000] BIOS-e820: [mem > 0x000000007bd38000-0x000000007bd38fff] usable > [    0.000000] BIOS-e820: [mem > 0x000000007bd39000-0x000000007bdbefff] reserved > [    0.000000] BIOS-e820: [mem > 0x000000007bdbf000-0x000000007bffffff] usable > [    0.000000] BIOS-e820: [mem > 0x000000007c000000-0x000000008fffffff] reserved > [    0.000000] BIOS-e820: [mem > 0x00000000fed1c000-0x00000000fed44fff] reserved > [    0.000000] BIOS-e820: [mem > 0x00000000ff000000-0x00000000ffffffff] reserved > [*    0.000000] BIOS-e820: [mem > 0x0000000100000000-0x000000407fffffff] usable* > * > * > Kindly let me know your inputs. > > Thanks. > > >     --     Thanks, >     Anatoly > > > > The "PTE Read not set" error usually indicates that you are > trying to use a non-IOMMU method when you have IOMMU enabled > (i.e. trying to use igb_uio when IOMMU is on). That, to me, > indicates that you do have IOMMU emulation enabled. > > I would go about it this way. > > First, i'd ensure that your VM has IOMMU emulation enabled and > working. You have mentioned that your QEMU version should have > IOMMU emulation, so let's assume that's the case. > > I am not sure of the exact command-line needed to activate the > vIOMMU emulation, but assuming your VM emulates an Intel > processor, your kernel command-line should have "iommu=on > intel_iommu=on" in it. Check /etc/default/grub for > GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are > not in there, add the above changes, do "update-grub" and reboot > your VM. > > If it already did have the necessary kernel configuration, do > "dmesg | grep IOMMU" and look for "IOMMU Enabled". That should > tell you that IOMMU is enabled and working in the kernel. > > After that, you can modprobe vfio and vfio-pci, bind NICs to it, > and it should be working. Please bear in mind that all of that > is how i would've gone about it if i had similar problems on > baremetal, but i'm hoping all of it is applicable to VM's. So, > either disable IOMMU and use igb_uio, or enable IOMMU and use > VFIO. Both will work. > > > Thanks for your help so far on this. I have everything enabled as > you have mentioned in your email. Below is the error from EAL when > DPDK is run on a VM. I am referring EAL code to check why it is failing. > > ************VM DPDK errors*********** > EAL: Detected 14 lcore(s) > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > unreliable clock cycles ! > EAL: PCI device 0000:00:07.0 on NUMA socket -1 > EAL:   Invalid NUMA socket, default to 0 > EAL:   probe driver: 8086:10ed net_ixgbe_vf > EAL:   using IOMMU type 1 (Type 1) > *EAL:   RTE_IOVA_VA 140051367329792 > EAL:   cannot set up DMA remapping, error 14 (Bad address) > EAL:   0000:00:07.0 DMA remapping failed, error 14 (Bad address)* > EAL: Requested device 0000:00:07.0 cannot be used > EAL: PCI device 0000:03:00.0 on NUMA socket -1 > EAL:   Invalid NUMA socket, default to 0 > EAL:   probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:04:00.0 on NUMA socket -1 > EAL:   Invalid NUMA socket, default to 0 > EAL:   probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:05:00.0 on NUMA socket -1 > EAL:   Invalid NUMA socket, default to 0 > EAL:   probe driver: 1af4:1041 net_virtio > EAL: No probed ethernet devices > Interactive-mode selected > USER1: create a new mbuf pool : n=171456, > size=2176, socket=0 > > > ********************VM************ > root@sriov-ravi-01:~# lsmod | grep vfio > *vfio_pci               40960  1 > vfio_virqfd            16384  1 vfio_pci > vfio_iommu_type1       20480  1 > vfio                   28672  5 vfio_iommu_type1,vfio_pci > irqbypass              16384  3 kvm,vfio_pci* > root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU > [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS > BXPCDMAR 00000001 BXPC 00000001) > *[    0.000000] DMAR: IOMMU enabled* > [    9.423839] DMAR: Host address width 39 > [    9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 > [    9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap > 12008c22260286 ecap f00f5e > [    9.427314] DMAR: ATSR flags: 0x1 > [    9.428047] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU 0 > [    9.429255] DMAR-IR: Queued invalidation will be enabled to > support x2apic and Intr-remapping. > [    9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode > [   11.167334] DMAR: No RMRR found > [   11.168855] DMAR: dmar0: Using Queued invalidation > [   12.119060] DMAR: Setting RMRR: > [   12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC > [   12.124787] DMAR: Setting identity map for device 0000:00:1f.0 > [0x0 - 0xffffff] > [   12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O > root@sriov-ravi-01:~# dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed' > drv=vfio-pci unused=* > > Network devices using kernel driver > =================================== > 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci > *Active* > 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci > 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci > > Other Network devices > ===================== > > > Crypto devices using DPDK-compatible driver > =========================================== > > > Crypto devices using kernel driver > ================================== > > > Other Crypto devices > ==================== > > > Eventdev devices using DPDK-compatible driver > ============================================= > > > Eventdev devices using kernel driver > ==================================== > > > Other Eventdev devices > ====================== > > > Mempool devices using DPDK-compatible driver > ============================================ > > > Mempool devices using kernel driver > =================================== > > > Other Mempool devices > ===================== > > root@sriov-ravi-01:~# > > root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | > grep -i vfio > [   77.676711] VFIO - User Level meta-driver version: 0.3 > [  149.013806] VFIO - User Level meta-driver version: 0.3 > [  275.876440] VFIO - User Level meta-driver version: 0.3 > > Thanks, > Ravi > > > > Additonal information. On VM following message is seen > > [  298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient > for the mapped address (7f7300000000) > [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient > for the mapped address (7fa580000000) > > ON VM, DMAR reports address width = 39 > root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep > -e IOMMU -e DMAR > [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR > 00000001 BXPC 00000001) > [    0.000000] DMAR: IOMMU enabled > [    9.209145] DMAR: Host address width 39 > > On Host, DMAR reports address width = 46 > # dmesg | grep -e DMAR -e IOMMU > [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I > 00000001 INTL 20091013) > [    0.000000] DMAR: IOMMU enabled > [    0.519368] DMAR: Host address width 46 > [    0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > > Searching for IOMMU errors found following link from RedHat which points > to DPDK 17.11 issue. > > https://bugzilla.redhat.com/show_bug.cgi?id=1530957 > > Thanks. > > -- > Thanks, > Anatoly > > > Ah, yes, that. We have recently applied a patch dealing with exactly this: http://dpdk.org/dev/patchwork/patch/33650/ Unfortunately, as far as i know, other than refusing to run (like the patch does), there's no way to fix this issue, so you'll have to disable IOMMU in your VM, and use either igb_uio or vfio in noiommu mode. -- Thanks, Anatoly