From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 8119A2B96 for ; Thu, 25 Jan 2018 11:49:29 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 Jan 2018 02:49:28 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,411,1511856000"; d="scan'208";a="26241475" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.237.220.145]) ([10.237.220.145]) by orsmga001.jf.intel.com with ESMTP; 25 Jan 2018 02:49:27 -0800 To: Ravi Kerur Cc: dev@dpdk.org References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> From: "Burakov, Anatoly" Message-ID: Date: Thu, 25 Jan 2018 10:49:27 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jan 2018 10:49:30 -0000 On 24-Jan-18 7:13 PM, Ravi Kerur wrote: > Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it > will work for both PF and VF interfaces since I am using both in my setup? > > Thanks. Yes, it should work for both PF and VF devices. > > On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly > > wrote: > > On 23-Jan-18 5:25 PM, Ravi Kerur wrote: > > Hi, > > I am running into an issue when DPDK is started with iommu on > via GRUB > command. Problem is not seen with regular kernel driver, error > messages > show when DPDK is started and happens for both PF and VF interfaces. > > I am using DPDK 17.05 so the patch proposed in the following link is > available > http://dpdk.org/ml/archives/dev/2017-February/057048.html > > > Workaround is to use "iommu=pt" but I want iommu enabled in my > setup. I > checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get > any details > on it. > > Kindly let me know how to resolve this issue. > > Following are the details > > (1) Linux kernel 4.9 > (2) DPDK 17.05 > > (3) IXGBE details > ethtool -i enp4s0f0  (PF driver) > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > ethtool -i enp4s16f2 (VF driver) > driver: ixgbevf > version: 4.3.2 > firmware-version: > bus-info: 0000:04:10.2 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: yes > supports-priv-flags: no > > Bus info          Device       Class          Description > ========================================================= > pci@0000:01:00.0  ens11f0      network        82599ES 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:01:00.1  ens11f1      network        82599ES 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:04:00.0  enp4s0f0     network        82599ES 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:04:00.1  enp4s0f1     network        82599ES 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:04:10.0  enp4s16      network        Illegal Vendor ID > pci@0000:04:10.2  enp4s16f2    network        Illegal Vendor ID > > (4) DPDK bind interfaces > > # dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > drv=igb_uio unused=vfio-pci > 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' > drv=igb_uio > unused=vfio-pci > > Network devices using kernel driver > =================================== > 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' > if=enp4s16 > drv=ixgbevf unused=igb_uio,vfio-pci > 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb > unused=igb_uio,vfio-pci *Active* > > Other Network devices > ===================== > > > ... > > (5) Kernel dmesg > > # dmesg | grep -e DMAR > [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA > A M I > 00000001 INTL 20091013) > [    0.000000] DMAR: IOMMU enabled > [    0.518747] DMAR: Host address width 46 > [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 > [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: > 0x0000007bbd4fff > [    0.593344] DMAR: ATSR flags: 0x0 > [    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity > domain: 0x0 > [    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity > domain: 0x1 > [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 > IOMMU 0 > [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 > IOMMU 1 > [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 > IOMMU 1 > [    0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 > [    0.675326] DMAR-IR: Queued invalidation will be enabled to > support > x2apic and Intr-remapping. > [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode > [    9.395170] DMAR: dmar1: Using Queued invalidation > [    9.405011] DMAR: Setting RMRR: > [    9.412006] DMAR: Setting identity map for device 0000:00:1d.0 > [0x7bbc6000 - 0x7bbd4fff] > [    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC > [    9.439712] DMAR: Setting identity map for device > 0000:00:1f.0 [0x0 - > 0xffffff] > [    9.454684] DMAR: Intel(R) Virtualization Technology for > Directed I/O > [  287.023068] DMAR: DRHD: handling fault status reg 2 > [  287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr > 18a260a000 [fault reason 06] PTE Read access is not set > [  287.023180] DMAR: DRHD: handling fault status reg 102 > [  287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr > 18a3010000 [fault reason 06] PTE Read access is not set > [  287.038250] DMAR: DRHD: handling fault status reg 202 > [  287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr > 18a3010000 [fault reason 06] PTE Read access is not set > [  288.170165] DMAR: DRHD: handling fault status reg 302 > [  288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [  288.694496] DMAR: DRHD: handling fault status reg 402 > [  288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr > 189069c000 [fault reason 06] PTE Read access is not set > [  289.927113] DMAR: DRHD: handling fault status reg 502 > [  289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [  290.174275] DMAR: DRHD: handling fault status reg 602 > [  290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [  292.174247] DMAR: DRHD: handling fault status reg 702 > [  292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [  294.174227] DMAR: DRHD: handling fault status reg 2 > [  294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [  296.174216] DMAR: DRHD: handling fault status reg 102 > [  296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [root@infradev-comp006.naw02.infradev.viasat.io > ~] > # > > Thanks. > > > Hi Ravi, > > The "iommu=pt" workaround applies only when you want to use igb_uio > driver. VFIO driver is able to fully utilize IOMMU without the need > for pass-through mode. From your log i can see that some devices are > bound to igb_uio while others are bound to vfio-pci. Just bind all > of the devices you want to use with DPDK to vfio-pci and these > errors should go away. > > -- > Thanks, > Anatoly > > -- Thanks, Anatoly