From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f53.google.com (mail-lf0-f53.google.com [209.85.215.53]) by dpdk.org (Postfix) with ESMTP id 489A81E34 for ; Fri, 2 Feb 2018 21:21:44 +0100 (CET) Received: by mail-lf0-f53.google.com with SMTP id a204so33263611lfa.2 for ; Fri, 02 Feb 2018 12:21:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=nim+V92QwcC4DIGM9gY7Hv5i7xblDhsc6Okxh41ELyc=; b=P+mfNCqrZ2BDqXL9FfqORDND4GdRS3eRQLz87lli6h2a/xdBlgrOmeb8hJSkXee5CI iJOqKZ7up/0W9AKpod5ekHOJl9XEf6G6Lhq9w1HyJnBP4ajeI5yZKCJyprGc7TxA6sug fuFaeu1PIeqFKKk70sMgFUxiS2NGaz18QsU94g9ffdueeSq3RbooI/EZAtMt2G1MtRxs dDPJkeS6SSIugGQRUBU282RYHKE2fzAPEMfp9nyn41/6cRHkI3I2UzRZK7GikkbWwOvj 2WP2JVdAaAI54wQE2z/5+hYZvyOzBqBtjHbjtvCcL3SG02xqcumFcue57uIhLyygCigG LOKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=nim+V92QwcC4DIGM9gY7Hv5i7xblDhsc6Okxh41ELyc=; b=qunfaiKyP5wo1m/6Sm7rQHhcWX89IMWbwpNn1Q4D61ytDadWx2yOfRbdovPjkLwJ8y 2VEX/+5OHMUBkBVh8wq0gwojogTUd5/QYrWucNEeBSjm9mzq6JoYNobmqR+CGZphG7EL KLLWHmha0Pg/et0EZUsyS3xTJ3i87lqmS+rYYlZffbWXzDSWe93OjTHJWkXBvqBRlAHn srfgbl2+rSqa2PwNASumHVCrhDoiPfcoQ/115+WrVFm98J/fODYJOyQo6qQggj5aJB8l fI5SctZ+d9NjhiJO+tvoHfzyAgesZDtelnn0J8wbfUOIySzOEIM18+MbqqSVHiz5eX8F 1bdg== X-Gm-Message-State: AKwxyteBQHcdcH3R4cdALx6dpeV5UtaEonemugCetabG3E3nHbNU5cF/ /mTaj65j3N1FDPfWOkuEirZyI29A3wVSjUj0uVEXyg== X-Google-Smtp-Source: AH8x225UK1edOTTgujmshlU/p4OkZ11tztwlXwnqixBT+ysdSqJuRbLagCixngejPakfq+7ePFLlzikV3Jx1DTSSEY0= X-Received: by 10.25.80.92 with SMTP id z28mr23921497lfj.34.1517602903671; Fri, 02 Feb 2018 12:21:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.196.17 with HTTP; Fri, 2 Feb 2018 12:21:42 -0800 (PST) In-Reply-To: <0120c68f-cf42-5d8b-5600-0514f76209b0@intel.com> References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <10911b54-57ee-370b-a4f7-f34accf4811e@intel.com> <3464b900-8648-c128-e959-dc60a8883a2d@intel.com> <0120c68f-cf42-5d8b-5600-0514f76209b0@intel.com> From: Ravi Kerur Date: Fri, 2 Feb 2018 12:21:42 -0800 Message-ID: To: "Burakov, Anatoly" Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Feb 2018 20:21:44 -0000 On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly wrote: > On 01-Feb-18 7:26 PM, Ravi Kerur wrote: > >> >> >> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly < >> anatoly.burakov@intel.com > wrote: >> >> On 31-Jan-18 9:51 PM, Ravi Kerur wrote: >> >> >> Hi Anatoly, >> >> Thanks. I am following wiki link below which uses vIOMMU with >> DPDK as a use-case and instantiate VM as specified with Q35 >> chipset in Qemu. >> >> https://wiki.qemu.org/Features/VT-d >> >> >> Qemu-version is 2.11 >> Host kernel 4.9 >> Guest kernel 4.4 >> >> I can only guess that guest kernel needs an upgrade in my setup >> to work correctly, if versions on my setup rings a bell on not >> having support kindly let me know. >> >> When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on >> guest I get following error >> ... >> vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored >> ... >> >> in guest. >> >> Thanks. >> >> >> AFAIK kernel 4.4 should have noiommu mode - it was introduced in >> 3.1x days. However, in order for that to work, kernel also has to be >> built with this mode enabled. My guess is, whoever is the supplier >> of your kernel, did not do that. You should double-check the kernel >> configuration of your distribution. >> >> However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode >> - "regular" vfio should work fine. noiommu mode should only be >> needed if you know you don't have IOMMU enabled in your kernel, and >> even if you can't enable it, you can still use igb_uio. >> >> Hi Anatoly, >> >> Do you suggest I take this discussion to kvm/qemu mailing list as I am >> not sure which component has the issue? I check dmesg for BIOS physical >> memory map and address reported as fault by DMAR is reported by BIOS as >> usable on both host and vm. >> >> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr >> *33a128000 *[fault reason 06] PTE Read access is not set >> >> dmesg | grep BIOS >> [ 0.000000] e820: BIOS-provided physical RAM map: >> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] >> ACPI data >> [ 0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] >> ACPI NVS >> [ 0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] >> reserved >> [* 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] >> usable* >> * >> * >> Kindly let me know your inputs. >> >> Thanks. >> >> >> -- Thanks, >> Anatoly >> >> >> > The "PTE Read not set" error usually indicates that you are trying to use > a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio > when IOMMU is on). That, to me, indicates that you do have IOMMU emulation > enabled. > > I would go about it this way. > > First, i'd ensure that your VM has IOMMU emulation enabled and working. > You have mentioned that your QEMU version should have IOMMU emulation, so > let's assume that's the case. > > I am not sure of the exact command-line needed to activate the vIOMMU > emulation, but assuming your VM emulates an Intel processor, your kernel > command-line should have "iommu=on intel_iommu=on" in it. Check > /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above > values are not in there, add the above changes, do "update-grub" and reboot > your VM. > > If it already did have the necessary kernel configuration, do "dmesg | > grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU > is enabled and working in the kernel. > > After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it > should be working. Please bear in mind that all of that is how i would've > gone about it if i had similar problems on baremetal, but i'm hoping all of > it is applicable to VM's. So, either disable IOMMU and use igb_uio, or > enable IOMMU and use VFIO. Both will work. > > Thanks for your help so far on this. I have everything enabled as you have mentioned in your email. Below is the error from EAL when DPDK is run on a VM. I am referring EAL code to check why it is failing. ************VM DPDK errors*********** EAL: Detected 14 lcore(s) EAL: Probing VFIO support... EAL: VFIO support initialized EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: PCI device 0000:00:07.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 8086:10ed net_ixgbe_vf EAL: using IOMMU type 1 (Type 1) *EAL: RTE_IOVA_VA 140051367329792EAL: cannot set up DMA remapping, error 14 (Bad address)EAL: 0000:00:07.0 DMA remapping failed, error 14 (Bad address)* EAL: Requested device 0000:00:07.0 cannot be used EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: PCI device 0000:04:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: PCI device 0000:05:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: No probed ethernet devices Interactive-mode selected USER1: create a new mbuf pool : n=171456, size=2176, socket=0 ********************VM************ root@sriov-ravi-01:~# lsmod | grep vfio *vfio_pci 40960 1vfio_virqfd 16384 1 vfio_pcivfio_iommu_type1 20480 1vfio 28672 5 vfio_iommu_type1,vfio_pciirqbypass 16384 3 kvm,vfio_pci* root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR 00000001 BXPC 00000001) *[ 0.000000] DMAR: IOMMU enabled* [ 9.423839] DMAR: Host address width 39 [ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 [ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 12008c22260286 ecap f00f5e [ 9.427314] DMAR: ATSR flags: 0x1 [ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 [ 9.429255] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 11.167334] DMAR: No RMRR found [ 11.168855] DMAR: dmar0: Using Queued invalidation [ 12.119060] DMAR: Setting RMRR: [ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC [ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O root@sriov-ravi-01:~# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=* Network devices using kernel driver =================================== 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci Other Network devices ===================== Crypto devices using DPDK-compatible driver =========================================== Crypto devices using kernel driver ================================== Other Crypto devices ==================== Eventdev devices using DPDK-compatible driver ============================================= Eventdev devices using kernel driver ==================================== Other Eventdev devices ====================== Mempool devices using DPDK-compatible driver ============================================ Mempool devices using kernel driver =================================== Other Mempool devices ===================== root@sriov-ravi-01:~# root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i vfio [ 77.676711] VFIO - User Level meta-driver version: 0.3 [ 149.013806] VFIO - User Level meta-driver version: 0.3 [ 275.876440] VFIO - User Level meta-driver version: 0.3 Thanks, Ravi -- > Thanks, > Anatoly >