From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f41.google.com (mail-lf0-f41.google.com [209.85.215.41]) by dpdk.org (Postfix) with ESMTP id D6801532C for ; Fri, 2 Feb 2018 21:51:59 +0100 (CET) Received: by mail-lf0-f41.google.com with SMTP id k19so33411757lfj.1 for ; Fri, 02 Feb 2018 12:51:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=NZppbMnqWl2GBBy7efqXcDbzuWftyluBSE5mNXYUL3o=; b=nb2HoXVXvqf39zUJgI9pk79tpEQ5x89ScZEdLjraJvLBIkngPmfzByDPGHB8n+EeYA AobAEwP9BCiz4QSKzDsnwXV1pOXVcVZFUC3Q3p2mTo/5ZB+gtAhbxEQZUX4qM4Wo078v TfD69Pnw5UFYfcg0ubtF35USK+tmmHXVlfQ+mVs1DLlxOCd9BiSBlNV+aLUuNTOaUH3n TAXyJYaf3bARQ2gYxwO5zbxSesvWIlEuuj1DpGXGyRkPGYu6Y1MxtYy78J/bLmCaC+/f CzxstN8/ZfSx7PZUTUjpeaOPU9r2ukup2jIyvKr2D6g0pKc8CNzoZ9Gw8N8VNVXdiC0p Mxmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=NZppbMnqWl2GBBy7efqXcDbzuWftyluBSE5mNXYUL3o=; b=fl+E4Pnwls+w8bRlmTCKYaQD90WMUg6cXwPCe5lR+K4b/6pQFpc1R7T5crCsKYY7eL q8sN/GHYlRgKSDtehdN5lmypwPzv7D+coUIG2aQllznwiWIbGhX5CxHcy9HgfF2HGT/K i91eeL3SRmGuFoGNTEYikfW8774Z7M+WKcLb5nWhooM9/1yb+/DlD9eK6lK+d7ICBUAs bWNEj5BdPTTxyRUMHzQLL4JAjCfJwFTCiwG1hbkWkn9P1Px+QZ5FhSPUbuvCEFJjlO+n FMcA90yewJRkpgwQOYFhDnFbKaP6rFq5Xqz+jzBX7FdoeFTkQmvagZ4kRjgriAEC46qH vTXg== X-Gm-Message-State: AKwxytcL8fQrglw96yTEfP7hRcLYmETgPOIQeZp/xupOwVsqle2TTRTJ PEjd6NmJ102y9paaftxxTYonrqhaElve7uI+6oA= X-Google-Smtp-Source: AH8x2274mpPCcQSigDe7qLdU4kxXJtmQSoQaJKeeuKPuyPBBWpgisiXd4Sfp/a0KqVzTd+krusNuQlT4GN2QcE2BWyc= X-Received: by 10.46.0.136 with SMTP id e8mr5631455lji.77.1517604719309; Fri, 02 Feb 2018 12:51:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.196.17 with HTTP; Fri, 2 Feb 2018 12:51:58 -0800 (PST) In-Reply-To: References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <10911b54-57ee-370b-a4f7-f34accf4811e@intel.com> <3464b900-8648-c128-e959-dc60a8883a2d@intel.com> <0120c68f-cf42-5d8b-5600-0514f76209b0@intel.com> From: Ravi Kerur Date: Fri, 2 Feb 2018 12:51:58 -0800 Message-ID: To: "Burakov, Anatoly" Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Feb 2018 20:52:00 -0000 On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur wrote: > > > On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly < > anatoly.burakov@intel.com> wrote: > >> On 01-Feb-18 7:26 PM, Ravi Kerur wrote: >> >>> >>> >>> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly < >>> anatoly.burakov@intel.com > wrote: >>> >>> On 31-Jan-18 9:51 PM, Ravi Kerur wrote: >>> >>> >>> Hi Anatoly, >>> >>> Thanks. I am following wiki link below which uses vIOMMU with >>> DPDK as a use-case and instantiate VM as specified with Q35 >>> chipset in Qemu. >>> >>> https://wiki.qemu.org/Features/VT-d >>> >>> >>> Qemu-version is 2.11 >>> Host kernel 4.9 >>> Guest kernel 4.4 >>> >>> I can only guess that guest kernel needs an upgrade in my setup >>> to work correctly, if versions on my setup rings a bell on not >>> having support kindly let me know. >>> >>> When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on >>> guest I get following error >>> ... >>> vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored >>> ... >>> >>> in guest. >>> >>> Thanks. >>> >>> >>> AFAIK kernel 4.4 should have noiommu mode - it was introduced in >>> 3.1x days. However, in order for that to work, kernel also has to be >>> built with this mode enabled. My guess is, whoever is the supplier >>> of your kernel, did not do that. You should double-check the kernel >>> configuration of your distribution. >>> >>> However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode >>> - "regular" vfio should work fine. noiommu mode should only be >>> needed if you know you don't have IOMMU enabled in your kernel, and >>> even if you can't enable it, you can still use igb_uio. >>> >>> Hi Anatoly, >>> >>> Do you suggest I take this discussion to kvm/qemu mailing list as I am >>> not sure which component has the issue? I check dmesg for BIOS physical >>> memory map and address reported as fault by DMAR is reported by BIOS as >>> usable on both host and vm. >>> >>> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr >>> *33a128000 *[fault reason 06] PTE Read access is not set >>> >>> dmesg | grep BIOS >>> [ 0.000000] e820: BIOS-provided physical RAM map: >>> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] >>> ACPI data >>> [ 0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] >>> ACPI NVS >>> [ 0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] >>> reserved >>> [* 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] >>> usable* >>> * >>> * >>> Kindly let me know your inputs. >>> >>> Thanks. >>> >>> >>> -- Thanks, >>> Anatoly >>> >>> >>> >> The "PTE Read not set" error usually indicates that you are trying to use >> a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio >> when IOMMU is on). That, to me, indicates that you do have IOMMU emulation >> enabled. >> >> I would go about it this way. >> >> First, i'd ensure that your VM has IOMMU emulation enabled and working. >> You have mentioned that your QEMU version should have IOMMU emulation, so >> let's assume that's the case. >> >> I am not sure of the exact command-line needed to activate the vIOMMU >> emulation, but assuming your VM emulates an Intel processor, your kernel >> command-line should have "iommu=on intel_iommu=on" in it. Check >> /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above >> values are not in there, add the above changes, do "update-grub" and reboot >> your VM. >> >> If it already did have the necessary kernel configuration, do "dmesg | >> grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU >> is enabled and working in the kernel. >> >> After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it >> should be working. Please bear in mind that all of that is how i would've >> gone about it if i had similar problems on baremetal, but i'm hoping all of >> it is applicable to VM's. So, either disable IOMMU and use igb_uio, or >> enable IOMMU and use VFIO. Both will work. >> >> > Thanks for your help so far on this. I have everything enabled as you have > mentioned in your email. Below is the error from EAL when DPDK is run on a > VM. I am referring EAL code to check why it is failing. > > ************VM DPDK errors*********** > EAL: Detected 14 lcore(s) > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > unreliable clock cycles ! > EAL: PCI device 0000:00:07.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 8086:10ed net_ixgbe_vf > EAL: using IOMMU type 1 (Type 1) > > > *EAL: RTE_IOVA_VA 140051367329792EAL: cannot set up DMA remapping, > error 14 (Bad address)EAL: 0000:00:07.0 DMA remapping failed, error 14 > (Bad address)* > EAL: Requested device 0000:00:07.0 cannot be used > EAL: PCI device 0000:03:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:04:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:05:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: No probed ethernet devices > Interactive-mode selected > USER1: create a new mbuf pool : n=171456, size=2176, > socket=0 > > > ********************VM************ > root@sriov-ravi-01:~# lsmod | grep vfio > > > > > *vfio_pci 40960 1vfio_virqfd 16384 1 > vfio_pcivfio_iommu_type1 20480 1vfio 28672 5 > vfio_iommu_type1,vfio_pciirqbypass 16384 3 kvm,vfio_pci* > root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU > [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR > 00000001 BXPC 00000001) > *[ 0.000000] DMAR: IOMMU enabled* > [ 9.423839] DMAR: Host address width 39 > [ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 > [ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap > 12008c22260286 ecap f00f5e > [ 9.427314] DMAR: ATSR flags: 0x1 > [ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 > [ 9.429255] DMAR-IR: Queued invalidation will be enabled to support > x2apic and Intr-remapping. > [ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 11.167334] DMAR: No RMRR found > [ 11.168855] DMAR: dmar0: Using Queued invalidation > [ 12.119060] DMAR: Setting RMRR: > [ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - > 0xffffff] > [ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O > root@sriov-ravi-01:~# dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed' > drv=vfio-pci unused=* > > Network devices using kernel driver > =================================== > 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* > 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci > 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci > > Other Network devices > ===================== > > > Crypto devices using DPDK-compatible driver > =========================================== > > > Crypto devices using kernel driver > ================================== > > > Other Crypto devices > ==================== > > > Eventdev devices using DPDK-compatible driver > ============================================= > > > Eventdev devices using kernel driver > ==================================== > > > Other Eventdev devices > ====================== > > > Mempool devices using DPDK-compatible driver > ============================================ > > > Mempool devices using kernel driver > =================================== > > > Other Mempool devices > ===================== > > root@sriov-ravi-01:~# > > root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i > vfio > [ 77.676711] VFIO - User Level meta-driver version: 0.3 > [ 149.013806] VFIO - User Level meta-driver version: 0.3 > [ 275.876440] VFIO - User Level meta-driver version: 0.3 > > Thanks, > Ravi > > > Additonal information. On VM following message is seen [ 298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7f7300000000) [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7fa580000000) ON VM, DMAR reports address width = 39 root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -e IOMMU -e DMAR [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR 00000001 BXPC 00000001) [ 0.000000] DMAR: IOMMU enabled [ 9.209145] DMAR: Host address width 39 On Host, DMAR reports address width = 46 # dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I 00000001 INTL 20091013) [ 0.000000] DMAR: IOMMU enabled [ 0.519368] DMAR: Host address width 46 [ 0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 Searching for IOMMU errors found following link from RedHat which points to DPDK 17.11 issue. https://bugzilla.redhat.com/show_bug.cgi?id=1530957 Thanks. > -- >> Thanks, >> Anatoly >> > >