From: Ravi Kerur <rkerur@gmail.com>
To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
Date: Wed, 24 Jan 2018 11:13:19 -0800 [thread overview]
Message-ID: <CAFb4SLC5X8nubBW6s4LBGb-kcEp-7hiRzzo2qjNyuiwrPRaiqQ@mail.gmail.com> (raw)
In-Reply-To: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com>
Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it
will work for both PF and VF interfaces since I am using both in my setup?
Thanks.
On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:
> On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
>
>> Hi,
>>
>> I am running into an issue when DPDK is started with iommu on via GRUB
>> command. Problem is not seen with regular kernel driver, error messages
>> show when DPDK is started and happens for both PF and VF interfaces.
>>
>> I am using DPDK 17.05 so the patch proposed in the following link is
>> available
>> http://dpdk.org/ml/archives/dev/2017-February/057048.html
>>
>> Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I
>> checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any
>> details
>> on it.
>>
>> Kindly let me know how to resolve this issue.
>>
>> Following are the details
>>
>> (1) Linux kernel 4.9
>> (2) DPDK 17.05
>>
>> (3) IXGBE details
>> ethtool -i enp4s0f0 (PF driver)
>> driver: ixgbe
>> version: 5.3.3
>> firmware-version: 0x800007b8, 1.1018.0
>> bus-info: 0000:04:00.0
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: yes
>>
>> ethtool -i enp4s16f2 (VF driver)
>> driver: ixgbevf
>> version: 4.3.2
>> firmware-version:
>> bus-info: 0000:04:10.2
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: no
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>> Bus info Device Class Description
>> =========================================================
>> pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit SFI/SFP+
>> Network Connection
>> pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit SFI/SFP+
>> Network Connection
>> pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit SFI/SFP+
>> Network Connection
>> pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit SFI/SFP+
>> Network Connection
>> pci@0000:04:10.0 enp4s16 network Illegal Vendor ID
>> pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID
>>
>> (4) DPDK bind interfaces
>>
>> # dpdk-devbind -s
>>
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> drv=igb_uio unused=vfio-pci
>> 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio
>> unused=vfio-pci
>>
>> Network devices using kernel driver
>> ===================================
>> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
>> 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
>> 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
>> 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16
>> drv=ixgbevf unused=igb_uio,vfio-pci
>> 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb
>> unused=igb_uio,vfio-pci *Active*
>>
>> Other Network devices
>> =====================
>> <none>
>>
>> ...
>>
>> (5) Kernel dmesg
>>
>> # dmesg | grep -e DMAR
>> [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
>> 00000001 INTL 20091013)
>> [ 0.000000] DMAR: IOMMU enabled
>> [ 0.518747] DMAR: Host address width 46
>> [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>> [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
>> d2078c106f0466 ecap f020df
>> [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
>> [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
>> d2078c106f0466 ecap f020df
>> [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff
>> [ 0.593344] DMAR: ATSR flags: 0x0
>> [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0
>> [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1
>> [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 IOMMU 0
>> [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 IOMMU 1
>> [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 IOMMU 1
>> [ 0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
>> [ 0.675326] DMAR-IR: Queued invalidation will be enabled to support
>> x2apic and Intr-remapping.
>> [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode
>> [ 9.395170] DMAR: dmar1: Using Queued invalidation
>> [ 9.405011] DMAR: Setting RMRR:
>> [ 9.412006] DMAR: Setting identity map for device 0000:00:1d.0
>> [0x7bbc6000 - 0x7bbd4fff]
>> [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
>> [ 9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
>> 0xffffff]
>> [ 9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O
>> [ 287.023068] DMAR: DRHD: handling fault status reg 2
>> [ 287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 18a260a000 [fault reason 06] PTE Read access is not set
>> [ 287.023180] DMAR: DRHD: handling fault status reg 102
>> [ 287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 18a3010000 [fault reason 06] PTE Read access is not set
>> [ 287.038250] DMAR: DRHD: handling fault status reg 202
>> [ 287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 18a3010000 [fault reason 06] PTE Read access is not set
>> [ 288.170165] DMAR: DRHD: handling fault status reg 302
>> [ 288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [ 288.694496] DMAR: DRHD: handling fault status reg 402
>> [ 288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr
>> 189069c000 [fault reason 06] PTE Read access is not set
>> [ 289.927113] DMAR: DRHD: handling fault status reg 502
>> [ 289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [ 290.174275] DMAR: DRHD: handling fault status reg 602
>> [ 290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [ 292.174247] DMAR: DRHD: handling fault status reg 702
>> [ 292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [ 294.174227] DMAR: DRHD: handling fault status reg 2
>> [ 294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [ 296.174216] DMAR: DRHD: handling fault status reg 102
>> [ 296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [root@infradev-comp006.naw02.infradev.viasat.io ~]
>> #
>>
>> Thanks.
>>
>>
> Hi Ravi,
>
> The "iommu=pt" workaround applies only when you want to use igb_uio
> driver. VFIO driver is able to fully utilize IOMMU without the need for
> pass-through mode. From your log i can see that some devices are bound to
> igb_uio while others are bound to vfio-pci. Just bind all of the devices
> you want to use with DPDK to vfio-pci and these errors should go away.
>
> --
> Thanks,
> Anatoly
>
next prev parent reply other threads:[~2018-01-24 19:13 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-23 17:25 Ravi Kerur
2018-01-24 10:31 ` Burakov, Anatoly
2018-01-24 19:13 ` Ravi Kerur [this message]
2018-01-25 10:49 ` Burakov, Anatoly
2018-01-29 22:35 ` Ravi Kerur
2018-01-31 9:59 ` Burakov, Anatoly
2018-01-31 21:51 ` Ravi Kerur
2018-02-01 10:10 ` Burakov, Anatoly
2018-02-01 19:26 ` Ravi Kerur
2018-02-02 10:28 ` Burakov, Anatoly
2018-02-02 20:21 ` Ravi Kerur
2018-02-02 20:51 ` Ravi Kerur
2018-02-05 10:01 ` Burakov, Anatoly
2018-02-06 17:55 ` Ravi Kerur
2018-02-08 11:20 ` Burakov, Anatoly
2018-02-09 17:41 ` Ravi Kerur
2018-02-10 10:11 ` Burakov, Anatoly
2018-02-10 10:58 ` Burakov, Anatoly
2018-02-10 17:53 ` Ravi Kerur
2018-02-12 10:13 ` Burakov, Anatoly
2018-02-12 22:00 ` Ravi Kerur
2018-02-13 14:31 ` Burakov, Anatoly
2018-02-14 20:00 ` Ravi Kerur
2018-02-15 10:28 ` Burakov, Anatoly
2018-02-15 18:27 ` Ravi Kerur
2018-02-15 20:53 ` Ravi Kerur
2018-02-16 9:41 ` Burakov, Anatoly
2019-01-15 7:07 ` Hu, Xuekun
2019-01-15 11:22 ` Burakov, Anatoly
2019-01-15 13:07 ` Hu, Xuekun
2019-01-21 13:18 ` Hu, Xuekun
2019-01-21 13:39 ` Burakov, Anatoly
2019-01-21 14:44 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAFb4SLC5X8nubBW6s4LBGb-kcEp-7hiRzzo2qjNyuiwrPRaiqQ@mail.gmail.com \
--to=rkerur@gmail.com \
--cc=anatoly.burakov@intel.com \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).