From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f52.google.com (mail-lf0-f52.google.com [209.85.215.52]) by dpdk.org (Postfix) with ESMTP id A96031B1A5 for ; Wed, 24 Jan 2018 20:13:21 +0100 (CET) Received: by mail-lf0-f52.google.com with SMTP id v188so6616927lfa.11 for ; Wed, 24 Jan 2018 11:13:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=M+8qy0WYu99n8QkJl0Yr9JZemW2tqGGCWK3hhSX0Lks=; b=FxtgWHRKaDv3iZ4UdNbF3SDdoAyrX11j5YWiqTvqECcb2KcnnDTDYYZssu8LsnoQdn o1Ga/F9grphzRax/yJGNtshz2YTg+YcWhmNDDa5wpGaY0Aq57zrJ8tTrmYtArI3HUdsP +dEgsOujkvUxbvFo7XUNG1Wfs86G7qjhCMeKRIr58IpaMsx1bqgHC7SwwrZ31kMlNZCp XnmRQScSr2sxVqkq0hp0Qw6nvp2cF/RKAPpYRsa0zEGnjJKGiLVoccKE+MBWFltV3TLC RzpjF69126POWFsP6mSquMl/9XBfR2EjxMP0k40Vzl3EXQV2xA7UWWvUx7cVlzYcL5l+ GY3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=M+8qy0WYu99n8QkJl0Yr9JZemW2tqGGCWK3hhSX0Lks=; b=rrPL9sXp06EnKzfm3z1P11gMgVarsFSJBiIsRfaYLcHmkNcDt/QTfPTGGGcqbNsEO6 vnepoMTvKLqclP/HNnP+5uIy8JoZTXIw3f5Yn5Dro7Ot6fjyjXPqNAKWn3uGTF4rQ45u XZHROEzqeMxizO5VBP3VfUutHExNvkx6h6hQczePyuh8TBJTDLGk/WljzB+C6IJX1XRY DHRe6ma1h23G2T7XIEmHYDugzZZpl/W9bMYirP2Vxx7XmQy0zTlLDAiF6Fq5UGxPoMcq wdl4GgMFXNTKXWGN7L6Poqk/7rQU/omyp6njBJ1SzT7I6gfauMUquQQr8BJgI1ERetED ZaWA== X-Gm-Message-State: AKwxytfIwfIHzsrdVugBtugb953fSqeXpEMjz5FmeraCewg6OpUSz5uC 4RgnyQAaRfqnWgzElboeNKq3nShPCl2S+7wIpUw= X-Google-Smtp-Source: AH8x2257eL/OeMEBFJQzQYVGQ7YeycTbcOK0tyhL4aduRQXD5FsGV49T9Eu/LPPo365TC+wv3rRu9XQRDYkIsnf2fxc= X-Received: by 10.46.68.11 with SMTP id r11mr4042495lja.13.1516821200438; Wed, 24 Jan 2018 11:13:20 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.196.17 with HTTP; Wed, 24 Jan 2018 11:13:19 -0800 (PST) In-Reply-To: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> From: Ravi Kerur Date: Wed, 24 Jan 2018 11:13:19 -0800 Message-ID: To: "Burakov, Anatoly" Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jan 2018 19:13:21 -0000 Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it will work for both PF and VF interfaces since I am using both in my setup? Thanks. On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly wrote: > On 23-Jan-18 5:25 PM, Ravi Kerur wrote: > >> Hi, >> >> I am running into an issue when DPDK is started with iommu on via GRUB >> command. Problem is not seen with regular kernel driver, error messages >> show when DPDK is started and happens for both PF and VF interfaces. >> >> I am using DPDK 17.05 so the patch proposed in the following link is >> available >> http://dpdk.org/ml/archives/dev/2017-February/057048.html >> >> Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I >> checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any >> details >> on it. >> >> Kindly let me know how to resolve this issue. >> >> Following are the details >> >> (1) Linux kernel 4.9 >> (2) DPDK 17.05 >> >> (3) IXGBE details >> ethtool -i enp4s0f0 (PF driver) >> driver: ixgbe >> version: 5.3.3 >> firmware-version: 0x800007b8, 1.1018.0 >> bus-info: 0000:04:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: yes >> >> ethtool -i enp4s16f2 (VF driver) >> driver: ixgbevf >> version: 4.3.2 >> firmware-version: >> bus-info: 0000:04:10.2 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: no >> supports-register-dump: yes >> supports-priv-flags: no >> >> Bus info Device Class Description >> ========================================================= >> pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit SFI/SFP+ >> Network Connection >> pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit SFI/SFP+ >> Network Connection >> pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit SFI/SFP+ >> Network Connection >> pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit SFI/SFP+ >> Network Connection >> pci@0000:04:10.0 enp4s16 network Illegal Vendor ID >> pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID >> >> (4) DPDK bind interfaces >> >> # dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> drv=igb_uio unused=vfio-pci >> 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio >> unused=vfio-pci >> >> Network devices using kernel driver >> =================================== >> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16 >> drv=ixgbevf unused=igb_uio,vfio-pci >> 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb >> unused=igb_uio,vfio-pci *Active* >> >> Other Network devices >> ===================== >> >> >> ... >> >> (5) Kernel dmesg >> >> # dmesg | grep -e DMAR >> [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I >> 00000001 INTL 20091013) >> [ 0.000000] DMAR: IOMMU enabled >> [ 0.518747] DMAR: Host address width 46 >> [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 >> [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 >> [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff >> [ 0.593344] DMAR: ATSR flags: 0x0 >> [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0 >> [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1 >> [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 IOMMU 0 >> [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 IOMMU 1 >> [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 IOMMU 1 >> [ 0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 >> [ 0.675326] DMAR-IR: Queued invalidation will be enabled to support >> x2apic and Intr-remapping. >> [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode >> [ 9.395170] DMAR: dmar1: Using Queued invalidation >> [ 9.405011] DMAR: Setting RMRR: >> [ 9.412006] DMAR: Setting identity map for device 0000:00:1d.0 >> [0x7bbc6000 - 0x7bbd4fff] >> [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC >> [ 9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - >> 0xffffff] >> [ 9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O >> [ 287.023068] DMAR: DRHD: handling fault status reg 2 >> [ 287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 18a260a000 [fault reason 06] PTE Read access is not set >> [ 287.023180] DMAR: DRHD: handling fault status reg 102 >> [ 287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 287.038250] DMAR: DRHD: handling fault status reg 202 >> [ 287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 288.170165] DMAR: DRHD: handling fault status reg 302 >> [ 288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 288.694496] DMAR: DRHD: handling fault status reg 402 >> [ 288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr >> 189069c000 [fault reason 06] PTE Read access is not set >> [ 289.927113] DMAR: DRHD: handling fault status reg 502 >> [ 289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 290.174275] DMAR: DRHD: handling fault status reg 602 >> [ 290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 292.174247] DMAR: DRHD: handling fault status reg 702 >> [ 292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 294.174227] DMAR: DRHD: handling fault status reg 2 >> [ 294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 296.174216] DMAR: DRHD: handling fault status reg 102 >> [ 296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [root@infradev-comp006.naw02.infradev.viasat.io ~] >> # >> >> Thanks. >> >> > Hi Ravi, > > The "iommu=pt" workaround applies only when you want to use igb_uio > driver. VFIO driver is able to fully utilize IOMMU without the need for > pass-through mode. From your log i can see that some devices are bound to > igb_uio while others are bound to vfio-pci. Just bind all of the devices > you want to use with DPDK to vfio-pci and these errors should go away. > > -- > Thanks, > Anatoly >