* [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue @ 2018-01-23 17:25 Ravi Kerur 2018-01-24 10:31 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-01-23 17:25 UTC (permalink / raw) To: dev, users Hi, I am running into an issue when DPDK is started with iommu on via GRUB command. Problem is not seen with regular kernel driver, error messages show when DPDK is started and happens for both PF and VF interfaces. I am using DPDK 17.05 so the patch proposed in the following link is available http://dpdk.org/ml/archives/dev/2017-February/057048.html Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any details on it. Kindly let me know how to resolve this issue. Following are the details (1) Linux kernel 4.9 (2) DPDK 17.05 (3) IXGBE details ethtool -i enp4s0f0 (PF driver) driver: ixgbe version: 5.3.3 firmware-version: 0x800007b8, 1.1018.0 bus-info: 0000:04:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes ethtool -i enp4s16f2 (VF driver) driver: ixgbevf version: 4.3.2 firmware-version: bus-info: 0000:04:10.2 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: yes supports-priv-flags: no Bus info Device Class Description ========================================================= pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit SFI/SFP+ Network Connection pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit SFI/SFP+ Network Connection pci@0000:04:10.0 enp4s16 network Illegal Vendor ID pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID (4) DPDK bind interfaces # dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=vfio-pci 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio unused=vfio-pci Network devices using kernel driver =================================== 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16 drv=ixgbevf unused=igb_uio,vfio-pci 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb unused=igb_uio,vfio-pci *Active* Other Network devices ===================== <none> ... (5) Kernel dmesg # dmesg | grep -e DMAR [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I 00000001 INTL 20091013) [ 0.000000] DMAR: IOMMU enabled [ 0.518747] DMAR: Host address width 46 [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff [ 0.593344] DMAR: ATSR flags: 0x0 [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0 [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1 [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 IOMMU 0 [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 IOMMU 1 [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 IOMMU 1 [ 0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 [ 0.675326] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 9.395170] DMAR: dmar1: Using Queued invalidation [ 9.405011] DMAR: Setting RMRR: [ 9.412006] DMAR: Setting identity map for device 0000:00:1d.0 [0x7bbc6000 - 0x7bbd4fff] [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC [ 9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 287.023068] DMAR: DRHD: handling fault status reg 2 [ 287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr 18a260a000 [fault reason 06] PTE Read access is not set [ 287.023180] DMAR: DRHD: handling fault status reg 102 [ 287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr 18a3010000 [fault reason 06] PTE Read access is not set [ 287.038250] DMAR: DRHD: handling fault status reg 202 [ 287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr 18a3010000 [fault reason 06] PTE Read access is not set [ 288.170165] DMAR: DRHD: handling fault status reg 302 [ 288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 288.694496] DMAR: DRHD: handling fault status reg 402 [ 288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr 189069c000 [fault reason 06] PTE Read access is not set [ 289.927113] DMAR: DRHD: handling fault status reg 502 [ 289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 290.174275] DMAR: DRHD: handling fault status reg 602 [ 290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 292.174247] DMAR: DRHD: handling fault status reg 702 [ 292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 294.174227] DMAR: DRHD: handling fault status reg 2 [ 294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [ 296.174216] DMAR: DRHD: handling fault status reg 102 [ 296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr 1890754000 [fault reason 06] PTE Read access is not set [root@infradev-comp006.naw02.infradev.viasat.io ~] # Thanks. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-01-23 17:25 [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue Ravi Kerur @ 2018-01-24 10:31 ` Burakov, Anatoly 2018-01-24 19:13 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-01-24 10:31 UTC (permalink / raw) To: dev On 23-Jan-18 5:25 PM, Ravi Kerur wrote: > Hi, > > I am running into an issue when DPDK is started with iommu on via GRUB > command. Problem is not seen with regular kernel driver, error messages > show when DPDK is started and happens for both PF and VF interfaces. > > I am using DPDK 17.05 so the patch proposed in the following link is > available > http://dpdk.org/ml/archives/dev/2017-February/057048.html > > Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I > checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any details > on it. > > Kindly let me know how to resolve this issue. > > Following are the details > > (1) Linux kernel 4.9 > (2) DPDK 17.05 > > (3) IXGBE details > ethtool -i enp4s0f0 (PF driver) > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > ethtool -i enp4s16f2 (VF driver) > driver: ixgbevf > version: 4.3.2 > firmware-version: > bus-info: 0000:04:10.2 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: yes > supports-priv-flags: no > > Bus info Device Class Description > ========================================================= > pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit SFI/SFP+ > Network Connection > pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit SFI/SFP+ > Network Connection > pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit SFI/SFP+ > Network Connection > pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit SFI/SFP+ > Network Connection > pci@0000:04:10.0 enp4s16 network Illegal Vendor ID > pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID > > (4) DPDK bind interfaces > > # dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > drv=igb_uio unused=vfio-pci > 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio > unused=vfio-pci > > Network devices using kernel driver > =================================== > 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16 > drv=ixgbevf unused=igb_uio,vfio-pci > 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb > unused=igb_uio,vfio-pci *Active* > > Other Network devices > ===================== > <none> > > ... > > (5) Kernel dmesg > > # dmesg | grep -e DMAR > [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I > 00000001 INTL 20091013) > [ 0.000000] DMAR: IOMMU enabled > [ 0.518747] DMAR: Host address width 46 > [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 > [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff > [ 0.593344] DMAR: ATSR flags: 0x0 > [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0 > [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1 > [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 IOMMU 0 > [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 IOMMU 1 > [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 IOMMU 1 > [ 0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 > [ 0.675326] DMAR-IR: Queued invalidation will be enabled to support > x2apic and Intr-remapping. > [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 9.395170] DMAR: dmar1: Using Queued invalidation > [ 9.405011] DMAR: Setting RMRR: > [ 9.412006] DMAR: Setting identity map for device 0000:00:1d.0 > [0x7bbc6000 - 0x7bbd4fff] > [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - > 0xffffff] > [ 9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O > [ 287.023068] DMAR: DRHD: handling fault status reg 2 > [ 287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr > 18a260a000 [fault reason 06] PTE Read access is not set > [ 287.023180] DMAR: DRHD: handling fault status reg 102 > [ 287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr > 18a3010000 [fault reason 06] PTE Read access is not set > [ 287.038250] DMAR: DRHD: handling fault status reg 202 > [ 287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr > 18a3010000 [fault reason 06] PTE Read access is not set > [ 288.170165] DMAR: DRHD: handling fault status reg 302 > [ 288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 288.694496] DMAR: DRHD: handling fault status reg 402 > [ 288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr > 189069c000 [fault reason 06] PTE Read access is not set > [ 289.927113] DMAR: DRHD: handling fault status reg 502 > [ 289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 290.174275] DMAR: DRHD: handling fault status reg 602 > [ 290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 292.174247] DMAR: DRHD: handling fault status reg 702 > [ 292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 294.174227] DMAR: DRHD: handling fault status reg 2 > [ 294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 296.174216] DMAR: DRHD: handling fault status reg 102 > [ 296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [root@infradev-comp006.naw02.infradev.viasat.io ~] > # > > Thanks. > Hi Ravi, The "iommu=pt" workaround applies only when you want to use igb_uio driver. VFIO driver is able to fully utilize IOMMU without the need for pass-through mode. From your log i can see that some devices are bound to igb_uio while others are bound to vfio-pci. Just bind all of the devices you want to use with DPDK to vfio-pci and these errors should go away. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-01-24 10:31 ` Burakov, Anatoly @ 2018-01-24 19:13 ` Ravi Kerur 2018-01-25 10:49 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-01-24 19:13 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it will work for both PF and VF interfaces since I am using both in my setup? Thanks. On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly <anatoly.burakov@intel.com > wrote: > On 23-Jan-18 5:25 PM, Ravi Kerur wrote: > >> Hi, >> >> I am running into an issue when DPDK is started with iommu on via GRUB >> command. Problem is not seen with regular kernel driver, error messages >> show when DPDK is started and happens for both PF and VF interfaces. >> >> I am using DPDK 17.05 so the patch proposed in the following link is >> available >> http://dpdk.org/ml/archives/dev/2017-February/057048.html >> >> Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I >> checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any >> details >> on it. >> >> Kindly let me know how to resolve this issue. >> >> Following are the details >> >> (1) Linux kernel 4.9 >> (2) DPDK 17.05 >> >> (3) IXGBE details >> ethtool -i enp4s0f0 (PF driver) >> driver: ixgbe >> version: 5.3.3 >> firmware-version: 0x800007b8, 1.1018.0 >> bus-info: 0000:04:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: yes >> >> ethtool -i enp4s16f2 (VF driver) >> driver: ixgbevf >> version: 4.3.2 >> firmware-version: >> bus-info: 0000:04:10.2 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: no >> supports-register-dump: yes >> supports-priv-flags: no >> >> Bus info Device Class Description >> ========================================================= >> pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit SFI/SFP+ >> Network Connection >> pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit SFI/SFP+ >> Network Connection >> pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit SFI/SFP+ >> Network Connection >> pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit SFI/SFP+ >> Network Connection >> pci@0000:04:10.0 enp4s16 network Illegal Vendor ID >> pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID >> >> (4) DPDK bind interfaces >> >> # dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> drv=igb_uio unused=vfio-pci >> 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio >> unused=vfio-pci >> >> Network devices using kernel driver >> =================================== >> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16 >> drv=ixgbevf unused=igb_uio,vfio-pci >> 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb >> unused=igb_uio,vfio-pci *Active* >> >> Other Network devices >> ===================== >> <none> >> >> ... >> >> (5) Kernel dmesg >> >> # dmesg | grep -e DMAR >> [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I >> 00000001 INTL 20091013) >> [ 0.000000] DMAR: IOMMU enabled >> [ 0.518747] DMAR: Host address width 46 >> [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 >> [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 >> [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff >> [ 0.593344] DMAR: ATSR flags: 0x0 >> [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0 >> [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1 >> [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 IOMMU 0 >> [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 IOMMU 1 >> [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 IOMMU 1 >> [ 0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 >> [ 0.675326] DMAR-IR: Queued invalidation will be enabled to support >> x2apic and Intr-remapping. >> [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode >> [ 9.395170] DMAR: dmar1: Using Queued invalidation >> [ 9.405011] DMAR: Setting RMRR: >> [ 9.412006] DMAR: Setting identity map for device 0000:00:1d.0 >> [0x7bbc6000 - 0x7bbd4fff] >> [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC >> [ 9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - >> 0xffffff] >> [ 9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O >> [ 287.023068] DMAR: DRHD: handling fault status reg 2 >> [ 287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 18a260a000 [fault reason 06] PTE Read access is not set >> [ 287.023180] DMAR: DRHD: handling fault status reg 102 >> [ 287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 287.038250] DMAR: DRHD: handling fault status reg 202 >> [ 287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 288.170165] DMAR: DRHD: handling fault status reg 302 >> [ 288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 288.694496] DMAR: DRHD: handling fault status reg 402 >> [ 288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr >> 189069c000 [fault reason 06] PTE Read access is not set >> [ 289.927113] DMAR: DRHD: handling fault status reg 502 >> [ 289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 290.174275] DMAR: DRHD: handling fault status reg 602 >> [ 290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 292.174247] DMAR: DRHD: handling fault status reg 702 >> [ 292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 294.174227] DMAR: DRHD: handling fault status reg 2 >> [ 294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 296.174216] DMAR: DRHD: handling fault status reg 102 >> [ 296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [root@infradev-comp006.naw02.infradev.viasat.io ~] >> # >> >> Thanks. >> >> > Hi Ravi, > > The "iommu=pt" workaround applies only when you want to use igb_uio > driver. VFIO driver is able to fully utilize IOMMU without the need for > pass-through mode. From your log i can see that some devices are bound to > igb_uio while others are bound to vfio-pci. Just bind all of the devices > you want to use with DPDK to vfio-pci and these errors should go away. > > -- > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-01-24 19:13 ` Ravi Kerur @ 2018-01-25 10:49 ` Burakov, Anatoly 2018-01-29 22:35 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-01-25 10:49 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 24-Jan-18 7:13 PM, Ravi Kerur wrote: > Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it > will work for both PF and VF interfaces since I am using both in my setup? > > Thanks. Yes, it should work for both PF and VF devices. > > On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: > > On 23-Jan-18 5:25 PM, Ravi Kerur wrote: > > Hi, > > I am running into an issue when DPDK is started with iommu on > via GRUB > command. Problem is not seen with regular kernel driver, error > messages > show when DPDK is started and happens for both PF and VF interfaces. > > I am using DPDK 17.05 so the patch proposed in the following link is > available > http://dpdk.org/ml/archives/dev/2017-February/057048.html > <http://dpdk.org/ml/archives/dev/2017-February/057048.html> > > Workaround is to use "iommu=pt" but I want iommu enabled in my > setup. I > checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get > any details > on it. > > Kindly let me know how to resolve this issue. > > Following are the details > > (1) Linux kernel 4.9 > (2) DPDK 17.05 > > (3) IXGBE details > ethtool -i enp4s0f0 (PF driver) > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > ethtool -i enp4s16f2 (VF driver) > driver: ixgbevf > version: 4.3.2 > firmware-version: > bus-info: 0000:04:10.2 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: yes > supports-priv-flags: no > > Bus info Device Class Description > ========================================================= > pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:04:10.0 enp4s16 network Illegal Vendor ID > pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID > > (4) DPDK bind interfaces > > # dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > drv=igb_uio unused=vfio-pci > 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' > drv=igb_uio > unused=vfio-pci > > Network devices using kernel driver > =================================== > 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' > if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' > if=enp4s16 > drv=ixgbevf unused=igb_uio,vfio-pci > 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb > unused=igb_uio,vfio-pci *Active* > > Other Network devices > ===================== > <none> > > ... > > (5) Kernel dmesg > > # dmesg | grep -e DMAR > [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA > A M I > 00000001 INTL 20091013) > [ 0.000000] DMAR: IOMMU enabled > [ 0.518747] DMAR: Host address width 46 > [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 > [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: > 0x0000007bbd4fff > [ 0.593344] DMAR: ATSR flags: 0x0 > [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity > domain: 0x0 > [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity > domain: 0x1 > [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 > IOMMU 0 > [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 > IOMMU 1 > [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 > IOMMU 1 > [ 0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 > [ 0.675326] DMAR-IR: Queued invalidation will be enabled to > support > x2apic and Intr-remapping. > [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 9.395170] DMAR: dmar1: Using Queued invalidation > [ 9.405011] DMAR: Setting RMRR: > [ 9.412006] DMAR: Setting identity map for device 0000:00:1d.0 > [0x7bbc6000 - 0x7bbd4fff] > [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 9.439712] DMAR: Setting identity map for device > 0000:00:1f.0 [0x0 - > 0xffffff] > [ 9.454684] DMAR: Intel(R) Virtualization Technology for > Directed I/O > [ 287.023068] DMAR: DRHD: handling fault status reg 2 > [ 287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr > 18a260a000 [fault reason 06] PTE Read access is not set > [ 287.023180] DMAR: DRHD: handling fault status reg 102 > [ 287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr > 18a3010000 [fault reason 06] PTE Read access is not set > [ 287.038250] DMAR: DRHD: handling fault status reg 202 > [ 287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr > 18a3010000 [fault reason 06] PTE Read access is not set > [ 288.170165] DMAR: DRHD: handling fault status reg 302 > [ 288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 288.694496] DMAR: DRHD: handling fault status reg 402 > [ 288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr > 189069c000 [fault reason 06] PTE Read access is not set > [ 289.927113] DMAR: DRHD: handling fault status reg 502 > [ 289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 290.174275] DMAR: DRHD: handling fault status reg 602 > [ 290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 292.174247] DMAR: DRHD: handling fault status reg 702 > [ 292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 294.174227] DMAR: DRHD: handling fault status reg 2 > [ 294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 296.174216] DMAR: DRHD: handling fault status reg 102 > [ 296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [root@infradev-comp006.naw02.infradev.viasat.io > <mailto:root@infradev-comp006.naw02.infradev.viasat.io> ~] > # > > Thanks. > > > Hi Ravi, > > The "iommu=pt" workaround applies only when you want to use igb_uio > driver. VFIO driver is able to fully utilize IOMMU without the need > for pass-through mode. From your log i can see that some devices are > bound to igb_uio while others are bound to vfio-pci. Just bind all > of the devices you want to use with DPDK to vfio-pci and these > errors should go away. > > -- > Thanks, > Anatoly > > -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-01-25 10:49 ` Burakov, Anatoly @ 2018-01-29 22:35 ` Ravi Kerur 2018-01-31 9:59 ` Burakov, Anatoly 2018-02-10 10:58 ` Burakov, Anatoly 0 siblings, 2 replies; 33+ messages in thread From: Ravi Kerur @ 2018-01-29 22:35 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev Hi Burakov, When using vfio-pci on host both VF and PF interfaces works fine with dpdk i.e. I don't see DMAR fault messages anymore. However, when I attach a VF interface to a VM and start DPDK with vfio-pci inside VM I still see DMAR fault messages on host. Both host and VM are booted with 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work (I think it's expected because of DMAR faults), however, when VF interface uses ixgbevf driver ping works. Following are some details /*****************On VM***************/ dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci unused=ixgbevf Network devices using kernel driver =================================== 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci Other network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other crypto devices ==================== <none> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) Subsystem: Intel Corporation 82599 Ethernet Controller Virtual Function Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K] Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K] Capabilities: [70] MSI-X: Enable+ Count=3 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v1) Root Complex Integrated Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- RBE- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: vfio-pci Kernel modules: ixgbevf /***************on Host*************/ dmesg | grep DMAR ... [ 978.268143] DMAR: DRHD: handling fault status reg 2 [ 978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr 33a128000 [fault reason 06] PTE Read access is not set [ 1286.677726] DMAR: DRHD: handling fault status reg 102 [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr fb663000 [fault reason 06] PTE Read access is not set [ 1676.436145] DMAR: DRHD: handling fault status reg 202 [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr 33a128000 [fault reason 06] PTE Read access is not set [ 1734.433649] DMAR: DRHD: handling fault status reg 302 [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr 33a128000 [fault reason 06] PTE Read access is not set [ 2324.428938] DMAR: DRHD: handling fault status reg 402 [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr 7770c000 [fault reason 06] PTE Read access is not set [ 2388.553640] DMAR: DRHD: handling fault status reg 502 [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr 33a128000 [fault reason 06] PTE Read access is not set VM is started with qemu-system-x86_64 -enable-kvm -M q35,accel=kvm,kernel-irqchip=split -object iothread,id=iothread0 -device intel-iommu,intremap=on,device-iotlb=on,caching-mode=on -cpu host -daemonize -m 16G -smp 14 -uuid 0fc91c66-f0b1-11e7-acf4-525400123456 -name 212748-sriov-ravi-smac-alpha-SMAC10 -device ioh3420,id=root.1,chassis=1 -device ioh3420,id=root.2,chassis=2 -netdev tap,vhost=on,queues=2,ifname=vn-vn2_1_,downscript=no,id=vn-vn2_1_,script=no -device ioh3420,id=root.3,chassis=3 -device virtio-net-pci,netdev=vn-vn2_1_,bus=root.3,ats=on,mq=on,vectors=6,mac=DE:AD:02:88:10:37,id=vn-vn2_1__dev -netdev tap,vhost=on,queues=2,ifname=vn-vn92_1_,downscript=no,id=vn-vn92_1_,script=no -device ioh3420,id=root.4,chassis=4 -device virtio-net-pci,mac=DE:AD:02:88:10:38,netdev=vn-vn92_1_,bus=root.4,ats=on,mq=on,vectors=6,id=vn-vn92_1__dev -netdev tap,vhost=on,queues=2,ifname=vn-vn93_1_,downscript=no,id=vn-vn93_1_,script=no -device ioh3420,id=root.5,chassis=5 -device virtio-net-pci,mac=DE:AD:02:88:10:39,netdev=vn-vn93_1_,bus=root.5,ats=on,mq=on,vectors=6,id=vn-vn93_1__dev -vnc :16,websocket=15916 -qmp tcp:127.0.0.1:12001,server,nowait -chardev socket,id=charmonitor,path=/tmp/mon.12001,server,nowait -mon chardev=charmonitor,id=monitor -cdrom /var/venom/cloud_init/0fc91c66-f0b1-11e7-acf4-525400123456.iso -*device vfio-pci,host=04:10.0* -drive file=/var/venom/instance_repo/test.img,if=none,id=drive-virtio-disk0,format=raw,aio=native,cache=none -balloon none -device virtio-blk-pci,scsi=off,iothread=iothread0,drive=drive-virtio-disk0,id=virtio-disk0,bus=root.1,ats=on,bootindex=1 Thanks. On Thu, Jan 25, 2018 at 2:49 AM, Burakov, Anatoly <anatoly.burakov@intel.com > wrote: > On 24-Jan-18 7:13 PM, Ravi Kerur wrote: > >> Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it >> will work for both PF and VF interfaces since I am using both in my setup? >> >> Thanks. >> > > Yes, it should work for both PF and VF devices. > > >> On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly < >> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: >> >> On 23-Jan-18 5:25 PM, Ravi Kerur wrote: >> >> Hi, >> >> I am running into an issue when DPDK is started with iommu on >> via GRUB >> command. Problem is not seen with regular kernel driver, error >> messages >> show when DPDK is started and happens for both PF and VF >> interfaces. >> >> I am using DPDK 17.05 so the patch proposed in the following link >> is >> available >> http://dpdk.org/ml/archives/dev/2017-February/057048.html >> <http://dpdk.org/ml/archives/dev/2017-February/057048.html> >> >> Workaround is to use "iommu=pt" but I want iommu enabled in my >> setup. I >> checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get >> any details >> on it. >> >> Kindly let me know how to resolve this issue. >> >> Following are the details >> >> (1) Linux kernel 4.9 >> (2) DPDK 17.05 >> >> (3) IXGBE details >> ethtool -i enp4s0f0 (PF driver) >> driver: ixgbe >> version: 5.3.3 >> firmware-version: 0x800007b8, 1.1018.0 >> bus-info: 0000:04:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: yes >> >> ethtool -i enp4s16f2 (VF driver) >> driver: ixgbevf >> version: 4.3.2 >> firmware-version: >> bus-info: 0000:04:10.2 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: no >> supports-register-dump: yes >> supports-priv-flags: no >> >> Bus info Device Class Description >> ========================================================= >> pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:04:10.0 enp4s16 network Illegal Vendor ID >> pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID >> >> (4) DPDK bind interfaces >> >> # dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> drv=igb_uio unused=vfio-pci >> 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' >> drv=igb_uio >> unused=vfio-pci >> >> Network devices using kernel driver >> =================================== >> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' >> if=enp4s16 >> drv=ixgbevf unused=igb_uio,vfio-pci >> 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 >> drv=igb >> unused=igb_uio,vfio-pci *Active* >> >> Other Network devices >> ===================== >> <none> >> >> ... >> >> (5) Kernel dmesg >> >> # dmesg | grep -e DMAR >> [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA >> A M I >> 00000001 INTL 20091013) >> [ 0.000000] DMAR: IOMMU enabled >> [ 0.518747] DMAR: Host address width 46 >> [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 >> [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 >> [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: >> 0x0000007bbd4fff >> [ 0.593344] DMAR: ATSR flags: 0x0 >> [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity >> domain: 0x0 >> [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity >> domain: 0x1 >> [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 >> IOMMU 0 >> [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 >> IOMMU 1 >> [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 >> IOMMU 1 >> [ 0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 >> [ 0.675326] DMAR-IR: Queued invalidation will be enabled to >> support >> x2apic and Intr-remapping. >> [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode >> [ 9.395170] DMAR: dmar1: Using Queued invalidation >> [ 9.405011] DMAR: Setting RMRR: >> [ 9.412006] DMAR: Setting identity map for device 0000:00:1d.0 >> [0x7bbc6000 - 0x7bbd4fff] >> [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC >> [ 9.439712] DMAR: Setting identity map for device >> 0000:00:1f.0 [0x0 - >> 0xffffff] >> [ 9.454684] DMAR: Intel(R) Virtualization Technology for >> Directed I/O >> [ 287.023068] DMAR: DRHD: handling fault status reg 2 >> [ 287.023073] DMAR: [DMA Read] Request device [01:00.0] fault >> addr >> 18a260a000 [fault reason 06] PTE Read access is not set >> [ 287.023180] DMAR: DRHD: handling fault status reg 102 >> [ 287.023183] DMAR: [DMA Read] Request device [01:00.0] fault >> addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 287.038250] DMAR: DRHD: handling fault status reg 202 >> [ 287.038252] DMAR: [DMA Read] Request device [01:00.0] fault >> addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 288.170165] DMAR: DRHD: handling fault status reg 302 >> [ 288.170170] DMAR: [DMA Read] Request device [01:00.0] fault >> addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 288.694496] DMAR: DRHD: handling fault status reg 402 >> [ 288.694499] DMAR: [DMA Read] Request device [04:10.2] fault >> addr >> 189069c000 [fault reason 06] PTE Read access is not set >> [ 289.927113] DMAR: DRHD: handling fault status reg 502 >> [ 289.927116] DMAR: [DMA Read] Request device [01:00.0] fault >> addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 290.174275] DMAR: DRHD: handling fault status reg 602 >> [ 290.174279] DMAR: [DMA Read] Request device [01:00.0] fault >> addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 292.174247] DMAR: DRHD: handling fault status reg 702 >> [ 292.174251] DMAR: [DMA Read] Request device [01:00.0] fault >> addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 294.174227] DMAR: DRHD: handling fault status reg 2 >> [ 294.174230] DMAR: [DMA Read] Request device [01:00.0] fault >> addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 296.174216] DMAR: DRHD: handling fault status reg 102 >> [ 296.174219] DMAR: [DMA Read] Request device [01:00.0] fault >> addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [root@infradev-comp006.naw02.infradev.viasat.io >> <mailto:root@infradev-comp006.naw02.infradev.viasat.io> ~] >> # >> >> Thanks. >> >> >> Hi Ravi, >> >> The "iommu=pt" workaround applies only when you want to use igb_uio >> driver. VFIO driver is able to fully utilize IOMMU without the need >> for pass-through mode. From your log i can see that some devices are >> bound to igb_uio while others are bound to vfio-pci. Just bind all >> of the devices you want to use with DPDK to vfio-pci and these >> errors should go away. >> >> -- Thanks, >> Anatoly >> >> >> > > -- > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-01-29 22:35 ` Ravi Kerur @ 2018-01-31 9:59 ` Burakov, Anatoly 2018-01-31 21:51 ` Ravi Kerur 2018-02-10 10:58 ` Burakov, Anatoly 1 sibling, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-01-31 9:59 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 29-Jan-18 10:35 PM, Ravi Kerur wrote: > Hi Burakov, > > When using vfio-pci on host both VF and PF interfaces works fine with > dpdk i.e. I don't see DMAR fault messages anymore. However, when I > attach a VF interface to a VM and start DPDK with vfio-pci inside VM I > still see DMAR fault messages on host. Both host and VM are booted with > 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work > (I think it's expected because of DMAR faults), however, when VF > interface uses ixgbevf driver ping works. > > Following are some details > > /*****************On VM***************/ > dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci > unused=ixgbevf > > Network devices using kernel driver > =================================== > 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* > 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci > 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci > > Other network devices > ===================== > <none> > > Crypto devices using DPDK-compatible driver > =========================================== > <none> > > Crypto devices using kernel driver > ================================== > <none> > > Other crypto devices > ==================== > <none> > > > 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller > Virtual Function (rev 01) > Subsystem: Intel Corporation 82599 Ethernet Controller Virtual > Function > Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- > ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K] > Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K] > Capabilities: [70] MSI-X: Enable+ Count=3 Masked- > Vector table: BAR=3 offset=00000000 > PBA: BAR=3 offset=00002000 > Capabilities: [a0] Express (v1) Root Complex Integrated > Endpoint, MSI 00 > DevCap: MaxPayload 128 bytes, PhantFunc 0 > ExtTag- RBE- > DevCtl: Report errors: Correctable- Non-Fatal- Fatal- > Unsupported- > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- > MaxPayload 128 bytes, MaxReadReq 128 bytes > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- > AuxPwr- TransPend- > Capabilities: [100 v1] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr- > AERCap: First Error Pointer: 00, GenCap- CGenEn- > ChkCap- ChkEn- > Kernel driver in use: vfio-pci > Kernel modules: ixgbevf > > /***************on Host*************/ > dmesg | grep DMAR > ... > [ 978.268143] DMAR: DRHD: handling fault status reg 2 > [ 978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr > 33a128000 [fault reason 06] PTE Read access is not set > [ 1286.677726] DMAR: DRHD: handling fault status reg 102 > [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr > fb663000 [fault reason 06] PTE Read access is not set > [ 1676.436145] DMAR: DRHD: handling fault status reg 202 > [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set > [ 1734.433649] DMAR: DRHD: handling fault status reg 302 > [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set > [ 2324.428938] DMAR: DRHD: handling fault status reg 402 > [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr > 7770c000 [fault reason 06] PTE Read access is not set > [ 2388.553640] DMAR: DRHD: handling fault status reg 502 > [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr > 33a128000 [fault reason 06] PTE Read access is not set > > > VM is started with > > qemu-system-x86_64 -enable-kvm -M q35,accel=kvm,kernel-irqchip=split > -object iothread,id=iothread0 -device > intel-iommu,intremap=on,device-iotlb=on,caching-mode=on -cpu host > -daemonize -m 16G -smp 14 -uuid 0fc91c66-f0b1-11e7-acf4-525400123456 > -name 212748-sriov-ravi-smac-alpha-SMAC10 -device > ioh3420,id=root.1,chassis=1 -device ioh3420,id=root.2,chassis=2 -netdev > tap,vhost=on,queues=2,ifname=vn-vn2_1_,downscript=no,id=vn-vn2_1_,script=no > -device ioh3420,id=root.3,chassis=3 -device > virtio-net-pci,netdev=vn-vn2_1_,bus=root.3,ats=on,mq=on,vectors=6,mac=DE:AD:02:88:10:37,id=vn-vn2_1__dev > -netdev > tap,vhost=on,queues=2,ifname=vn-vn92_1_,downscript=no,id=vn-vn92_1_,script=no > -device ioh3420,id=root.4,chassis=4 -device > virtio-net-pci,mac=DE:AD:02:88:10:38,netdev=vn-vn92_1_,bus=root.4,ats=on,mq=on,vectors=6,id=vn-vn92_1__dev > -netdev > tap,vhost=on,queues=2,ifname=vn-vn93_1_,downscript=no,id=vn-vn93_1_,script=no > -device ioh3420,id=root.5,chassis=5 -device > virtio-net-pci,mac=DE:AD:02:88:10:39,netdev=vn-vn93_1_,bus=root.5,ats=on,mq=on,vectors=6,id=vn-vn93_1__dev > -vnc :16,websocket=15916 -qmp tcp:127.0.0.1:12001 > <http://127.0.0.1:12001>,server,nowait -chardev > socket,id=charmonitor,path=/tmp/mon.12001,server,nowait -mon > chardev=charmonitor,id=monitor -cdrom > /var/venom/cloud_init/0fc91c66-f0b1-11e7-acf4-525400123456.iso -*device > vfio-pci,host=04:10.0* -drive > file=/var/venom/instance_repo/test.img,if=none,id=drive-virtio-disk0,format=raw,aio=native,cache=none > -balloon none -device > virtio-blk-pci,scsi=off,iothread=iothread0,drive=drive-virtio-disk0,id=virtio-disk0,bus=root.1,ats=on,bootindex=1 > > Thanks. > > > On Thu, Jan 25, 2018 at 2:49 AM, Burakov, Anatoly > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: > > On 24-Jan-18 7:13 PM, Ravi Kerur wrote: > > Hi Burakov, Thank you. I will try with vfio-pci driver. I am > assuming it will work for both PF and VF interfaces since I am > using both in my setup? > > Thanks. > > > Yes, it should work for both PF and VF devices. > > > On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> > <mailto:anatoly.burakov@intel.com > <mailto:anatoly.burakov@intel.com>>> wrote: > > On 23-Jan-18 5:25 PM, Ravi Kerur wrote: > > Hi, > > I am running into an issue when DPDK is started with > iommu on > via GRUB > command. Problem is not seen with regular kernel > driver, error > messages > show when DPDK is started and happens for both PF and > VF interfaces. > > I am using DPDK 17.05 so the patch proposed in the > following link is > available > http://dpdk.org/ml/archives/dev/2017-February/057048.html > <http://dpdk.org/ml/archives/dev/2017-February/057048.html> > > <http://dpdk.org/ml/archives/dev/2017-February/057048.html > <http://dpdk.org/ml/archives/dev/2017-February/057048.html>> > > Workaround is to use "iommu=pt" but I want iommu > enabled in my > setup. I > checked BIOS for reserved memory(DMA RMRR for IXGBE) > didn't get > any details > on it. > > Kindly let me know how to resolve this issue. > > Following are the details > > (1) Linux kernel 4.9 > (2) DPDK 17.05 > > (3) IXGBE details > ethtool -i enp4s0f0 (PF driver) > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > ethtool -i enp4s16f2 (VF driver) > driver: ixgbevf > version: 4.3.2 > firmware-version: > bus-info: 0000:04:10.2 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: yes > supports-priv-flags: no > > Bus info Device Class Description > ========================================================= > pci@0000:01:00.0 ens11f0 network 82599ES > 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:01:00.1 ens11f1 network 82599ES > 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:04:00.0 enp4s0f0 network 82599ES > 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:04:00.1 enp4s0f1 network 82599ES > 10-Gigabit > SFI/SFP+ > Network Connection > pci@0000:04:10.0 enp4s16 network Illegal > Vendor ID > pci@0000:04:10.2 enp4s16f2 network Illegal > Vendor ID > > (4) DPDK bind interfaces > > # dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network > Connection 10fb' > drv=igb_uio unused=vfio-pci > 0000:04:10.2 '82599 Ethernet Controller Virtual > Function 10ed' > drv=igb_uio > unused=vfio-pci > > Network devices using kernel driver > =================================== > 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network > Connection 10fb' > if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network > Connection 10fb' > if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network > Connection 10fb' > if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci > 0000:04:10.0 '82599 Ethernet Controller Virtual > Function 10ed' > if=enp4s16 > drv=ixgbevf unused=igb_uio,vfio-pci > 0000:06:00.0 'I210 Gigabit Network Connection 1533' > if=eno1 drv=igb > unused=igb_uio,vfio-pci *Active* > > Other Network devices > ===================== > <none> > > ... > > (5) Kernel dmesg > > # dmesg | grep -e DMAR > [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 > (v01 ALASKA > A M I > 00000001 INTL 20091013) > [ 0.000000] DMAR: IOMMU enabled > [ 0.518747] DMAR: Host address width 46 > [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver > 1:0 cap > d2078c106f0466 ecap f020df > [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 > [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver > 1:0 cap > d2078c106f0466 ecap f020df > [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: > 0x0000007bbd4fff > [ 0.593344] DMAR: ATSR flags: 0x0 > [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity > domain: 0x0 > [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity > domain: 0x1 > [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base > 0xfbffc000 > IOMMU 0 > [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base > 0xc7ffc000 > IOMMU 1 > [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base > 0xc7ffc000 > IOMMU 1 > [ 0.664324] DMAR-IR: HPET id 0 under DRHD base > 0xc7ffc000 > [ 0.675326] DMAR-IR: Queued invalidation will be > enabled to > support > x2apic and Intr-remapping. > [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic > mode > [ 9.395170] DMAR: dmar1: Using Queued invalidation > [ 9.405011] DMAR: Setting RMRR: > [ 9.412006] DMAR: Setting identity map for device > 0000:00:1d.0 > [0x7bbc6000 - 0x7bbd4fff] > [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 9.439712] DMAR: Setting identity map for device > 0000:00:1f.0 [0x0 - > 0xffffff] > [ 9.454684] DMAR: Intel(R) Virtualization Technology for > Directed I/O > [ 287.023068] DMAR: DRHD: handling fault status reg 2 > [ 287.023073] DMAR: [DMA Read] Request device > [01:00.0] fault addr > 18a260a000 [fault reason 06] PTE Read access is not set > [ 287.023180] DMAR: DRHD: handling fault status reg 102 > [ 287.023183] DMAR: [DMA Read] Request device > [01:00.0] fault addr > 18a3010000 [fault reason 06] PTE Read access is not set > [ 287.038250] DMAR: DRHD: handling fault status reg 202 > [ 287.038252] DMAR: [DMA Read] Request device > [01:00.0] fault addr > 18a3010000 [fault reason 06] PTE Read access is not set > [ 288.170165] DMAR: DRHD: handling fault status reg 302 > [ 288.170170] DMAR: [DMA Read] Request device > [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 288.694496] DMAR: DRHD: handling fault status reg 402 > [ 288.694499] DMAR: [DMA Read] Request device > [04:10.2] fault addr > 189069c000 [fault reason 06] PTE Read access is not set > [ 289.927113] DMAR: DRHD: handling fault status reg 502 > [ 289.927116] DMAR: [DMA Read] Request device > [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 290.174275] DMAR: DRHD: handling fault status reg 602 > [ 290.174279] DMAR: [DMA Read] Request device > [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 292.174247] DMAR: DRHD: handling fault status reg 702 > [ 292.174251] DMAR: [DMA Read] Request device > [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 294.174227] DMAR: DRHD: handling fault status reg 2 > [ 294.174230] DMAR: [DMA Read] Request device > [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [ 296.174216] DMAR: DRHD: handling fault status reg 102 > [ 296.174219] DMAR: [DMA Read] Request device > [01:00.0] fault addr > 1890754000 [fault reason 06] PTE Read access is not set > [root@infradev-comp006.naw02.infradev.viasat.io > <mailto:root@infradev-comp006.naw02.infradev.viasat.io> > <mailto:root@infradev-comp006.naw02.infradev.viasat.io > <mailto:root@infradev-comp006.naw02.infradev.viasat.io>> ~] > # > > Thanks. > > > Hi Ravi, > > The "iommu=pt" workaround applies only when you want to use > igb_uio > driver. VFIO driver is able to fully utilize IOMMU without > the need > for pass-through mode. From your log i can see that some > devices are > bound to igb_uio while others are bound to vfio-pci. Just > bind all > of the devices you want to use with DPDK to vfio-pci and these > errors should go away. > > -- Thanks, > Anatoly > > > > > -- > Thanks, > Anatoly > > Hi Ravi, Using vfio-pci in IOMMU mode will only work if your VM provides IOMMU emulation (it's a fairly recent development, so your QEMU must be of an appropriate version - can't recall which one off the top of my head). Otherwise you'd have to treat your VM as if it was a machine without IOMMU, i.e. use noiommu mode for VFIO, or igb_uio driver. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-01-31 9:59 ` Burakov, Anatoly @ 2018-01-31 21:51 ` Ravi Kerur 2018-02-01 10:10 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-01-31 21:51 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Wed, Jan 31, 2018 at 1:59 AM, Burakov, Anatoly <anatoly.burakov@intel.com > wrote: > On 29-Jan-18 10:35 PM, Ravi Kerur wrote: > >> Hi Burakov, >> >> When using vfio-pci on host both VF and PF interfaces works fine with >> dpdk i.e. I don't see DMAR fault messages anymore. However, when I attach a >> VF interface to a VM and start DPDK with vfio-pci inside VM I still see >> DMAR fault messages on host. Both host and VM are booted with >> 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work (I >> think it's expected because of DMAR faults), however, when VF interface >> uses ixgbevf driver ping works. >> >> Following are some details >> >> /*****************On VM***************/ >> dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci >> unused=ixgbevf >> >> Network devices using kernel driver >> =================================== >> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* >> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci >> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci >> >> Other network devices >> ===================== >> <none> >> >> Crypto devices using DPDK-compatible driver >> =========================================== >> <none> >> >> Crypto devices using kernel driver >> ================================== >> <none> >> >> Other crypto devices >> ==================== >> <none> >> >> >> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller >> Virtual Function (rev 01) >> Subsystem: Intel Corporation 82599 Ethernet Controller Virtual >> Function >> Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- >> ParErr- Stepping- SERR- FastB2B- DisINTx+ >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- >> <TAbort- <MAbort- >SERR- <PERR- INTx- >> Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K] >> Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K] >> Capabilities: [70] MSI-X: Enable+ Count=3 Masked- >> Vector table: BAR=3 offset=00000000 >> PBA: BAR=3 offset=00002000 >> Capabilities: [a0] Express (v1) Root Complex Integrated >> Endpoint, MSI 00 >> DevCap: MaxPayload 128 bytes, PhantFunc 0 >> ExtTag- RBE- >> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- >> Unsupported- >> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- >> MaxPayload 128 bytes, MaxReadReq 128 bytes >> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- >> TransPend- >> Capabilities: [100 v1] Advanced Error Reporting >> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- >> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- >> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- >> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >> NonFatalErr- >> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >> NonFatalErr- >> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- >> ChkEn- >> Kernel driver in use: vfio-pci >> Kernel modules: ixgbevf >> >> /***************on Host*************/ >> dmesg | grep DMAR >> ... >> [ 978.268143] DMAR: DRHD: handling fault status reg 2 >> [ 978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> [ 1286.677726] DMAR: DRHD: handling fault status reg 102 >> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr >> fb663000 [fault reason 06] PTE Read access is not set >> [ 1676.436145] DMAR: DRHD: handling fault status reg 202 >> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> [ 1734.433649] DMAR: DRHD: handling fault status reg 302 >> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> [ 2324.428938] DMAR: DRHD: handling fault status reg 402 >> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 7770c000 [fault reason 06] PTE Read access is not set >> [ 2388.553640] DMAR: DRHD: handling fault status reg 502 >> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> >> >> VM is started with >> >> qemu-system-x86_64 -enable-kvm -M q35,accel=kvm,kernel-irqchip=split >> -object iothread,id=iothread0 -device intel-iommu,intremap=on,device-iotlb=on,caching-mode=on >> -cpu host -daemonize -m 16G -smp 14 -uuid 0fc91c66-f0b1-11e7-acf4-525400123456 >> -name 212748-sriov-ravi-smac-alpha-SMAC10 -device >> ioh3420,id=root.1,chassis=1 -device ioh3420,id=root.2,chassis=2 -netdev >> tap,vhost=on,queues=2,ifname=vn-vn2_1_,downscript=no,id=vn-vn2_1_,script=no >> -device ioh3420,id=root.3,chassis=3 -device virtio-net-pci,netdev=vn-vn2_1 >> _,bus=root.3,ats=on,mq=on,vectors=6,mac=DE:AD:02:88:10:37,id=vn-vn2_1__dev >> -netdev tap,vhost=on,queues=2,ifname=vn-vn92_1_,downscript=no,id=vn-vn92_1_,script=no >> -device ioh3420,id=root.4,chassis=4 -device virtio-net-pci,mac=DE:AD:02:88 >> :10:38,netdev=vn-vn92_1_,bus=root.4,ats=on,mq=on,vectors=6,id=vn-vn92_1__dev >> -netdev tap,vhost=on,queues=2,ifname=vn-vn93_1_,downscript=no,id=vn-vn93_1_,script=no >> -device ioh3420,id=root.5,chassis=5 -device virtio-net-pci,mac=DE:AD:02:88 >> :10:39,netdev=vn-vn93_1_,bus=root.5,ats=on,mq=on,vectors=6,id=vn-vn93_1__dev >> -vnc :16,websocket=15916 -qmp tcp:127.0.0.1:12001 <http://127.0.0.1:12001 >> >,server,nowait -chardev socket,id=charmonitor,path=/tmp/mon.12001,server,nowait >> -mon chardev=charmonitor,id=monitor -cdrom /var/venom/cloud_init/0fc91c66 >> -f0b1-11e7-acf4-525400123456.iso -*device vfio-pci,host=04:10.0* -drive >> file=/var/venom/instance_repo/test.img,if=none,id=drive-virt >> io-disk0,format=raw,aio=native,cache=none -balloon none -device >> virtio-blk-pci,scsi=off,iothread=iothread0,drive=drive- >> virtio-disk0,id=virtio-disk0,bus=root.1,ats=on,bootindex=1 >> >> Thanks. >> >> >> On Thu, Jan 25, 2018 at 2:49 AM, Burakov, Anatoly < >> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: >> >> On 24-Jan-18 7:13 PM, Ravi Kerur wrote: >> >> Hi Burakov, Thank you. I will try with vfio-pci driver. I am >> assuming it will work for both PF and VF interfaces since I am >> using both in my setup? >> >> Thanks. >> >> >> Yes, it should work for both PF and VF devices. >> >> >> On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly >> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> >> <mailto:anatoly.burakov@intel.com >> >> <mailto:anatoly.burakov@intel.com>>> wrote: >> >> On 23-Jan-18 5:25 PM, Ravi Kerur wrote: >> >> Hi, >> >> I am running into an issue when DPDK is started with >> iommu on >> via GRUB >> command. Problem is not seen with regular kernel >> driver, error >> messages >> show when DPDK is started and happens for both PF and >> VF interfaces. >> >> I am using DPDK 17.05 so the patch proposed in the >> following link is >> available >> http://dpdk.org/ml/archives/dev/2017-February/057048.html >> <http://dpdk.org/ml/archives/dev/2017-February/057048.html> >> <http://dpdk.org/ml/archives/d >> ev/2017-February/057048.html >> <http://dpdk.org/ml/archives/dev/2017-February/057048.html>> >> >> Workaround is to use "iommu=pt" but I want iommu >> enabled in my >> setup. I >> checked BIOS for reserved memory(DMA RMRR for IXGBE) >> didn't get >> any details >> on it. >> >> Kindly let me know how to resolve this issue. >> >> Following are the details >> >> (1) Linux kernel 4.9 >> (2) DPDK 17.05 >> >> (3) IXGBE details >> ethtool -i enp4s0f0 (PF driver) >> driver: ixgbe >> version: 5.3.3 >> firmware-version: 0x800007b8, 1.1018.0 >> bus-info: 0000:04:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: yes >> >> ethtool -i enp4s16f2 (VF driver) >> driver: ixgbevf >> version: 4.3.2 >> firmware-version: >> bus-info: 0000:04:10.2 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: no >> supports-register-dump: yes >> supports-priv-flags: no >> >> Bus info Device Class Description >> ============================== >> =========================== >> pci@0000:01:00.0 ens11f0 network 82599ES >> 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:01:00.1 ens11f1 network 82599ES >> 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:04:00.0 enp4s0f0 network 82599ES >> 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:04:00.1 enp4s0f1 network 82599ES >> 10-Gigabit >> SFI/SFP+ >> Network Connection >> pci@0000:04:10.0 enp4s16 network Illegal >> Vendor ID >> pci@0000:04:10.2 enp4s16f2 network Illegal >> Vendor ID >> >> (4) DPDK bind interfaces >> >> # dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network >> Connection 10fb' >> drv=igb_uio unused=vfio-pci >> 0000:04:10.2 '82599 Ethernet Controller Virtual >> Function 10ed' >> drv=igb_uio >> unused=vfio-pci >> >> Network devices using kernel driver >> =================================== >> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network >> Connection 10fb' >> if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network >> Connection 10fb' >> if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network >> Connection 10fb' >> if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci >> 0000:04:10.0 '82599 Ethernet Controller Virtual >> Function 10ed' >> if=enp4s16 >> drv=ixgbevf unused=igb_uio,vfio-pci >> 0000:06:00.0 'I210 Gigabit Network Connection 1533' >> if=eno1 drv=igb >> unused=igb_uio,vfio-pci *Active* >> >> Other Network devices >> ===================== >> <none> >> >> ... >> >> (5) Kernel dmesg >> >> # dmesg | grep -e DMAR >> [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 >> (v01 ALASKA >> A M I >> 00000001 INTL 20091013) >> [ 0.000000] DMAR: IOMMU enabled >> [ 0.518747] DMAR: Host address width 46 >> [ 0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: >> 0x0 >> [ 0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver >> 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: >> 0x1 >> [ 0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver >> 1:0 cap >> d2078c106f0466 ecap f020df >> [ 0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: >> 0x0000007bbd4fff >> [ 0.593344] DMAR: ATSR flags: 0x0 >> [ 0.600178] DMAR: RHSA base: 0x000000c7ffc000 >> proximity >> domain: 0x0 >> [ 0.612905] DMAR: RHSA base: 0x000000fbffc000 >> proximity >> domain: 0x1 >> [ 0.625632] DMAR-IR: IOAPIC id 3 under DRHD base >> 0xfbffc000 >> IOMMU 0 >> [ 0.638522] DMAR-IR: IOAPIC id 1 under DRHD base >> 0xc7ffc000 >> IOMMU 1 >> [ 0.651426] DMAR-IR: IOAPIC id 2 under DRHD base >> 0xc7ffc000 >> IOMMU 1 >> [ 0.664324] DMAR-IR: HPET id 0 under DRHD base >> 0xc7ffc000 >> [ 0.675326] DMAR-IR: Queued invalidation will be >> enabled to >> support >> x2apic and Intr-remapping. >> [ 0.693805] DMAR-IR: Enabled IRQ remapping in x2apic >> mode >> [ 9.395170] DMAR: dmar1: Using Queued invalidation >> [ 9.405011] DMAR: Setting RMRR: >> [ 9.412006] DMAR: Setting identity map for device >> 0000:00:1d.0 >> [0x7bbc6000 - 0x7bbd4fff] >> [ 9.428569] DMAR: Prepare 0-16MiB unity mapping for >> LPC >> [ 9.439712] DMAR: Setting identity map for device >> 0000:00:1f.0 [0x0 - >> 0xffffff] >> [ 9.454684] DMAR: Intel(R) Virtualization Technology >> for >> Directed I/O >> [ 287.023068] DMAR: DRHD: handling fault status reg 2 >> [ 287.023073] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 18a260a000 [fault reason 06] PTE Read access is not set >> [ 287.023180] DMAR: DRHD: handling fault status reg 102 >> [ 287.023183] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 287.038250] DMAR: DRHD: handling fault status reg 202 >> [ 287.038252] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 18a3010000 [fault reason 06] PTE Read access is not set >> [ 288.170165] DMAR: DRHD: handling fault status reg 302 >> [ 288.170170] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 288.694496] DMAR: DRHD: handling fault status reg 402 >> [ 288.694499] DMAR: [DMA Read] Request device >> [04:10.2] fault addr >> 189069c000 [fault reason 06] PTE Read access is not set >> [ 289.927113] DMAR: DRHD: handling fault status reg 502 >> [ 289.927116] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 290.174275] DMAR: DRHD: handling fault status reg 602 >> [ 290.174279] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 292.174247] DMAR: DRHD: handling fault status reg 702 >> [ 292.174251] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 294.174227] DMAR: DRHD: handling fault status reg 2 >> [ 294.174230] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [ 296.174216] DMAR: DRHD: handling fault status reg 102 >> [ 296.174219] DMAR: [DMA Read] Request device >> [01:00.0] fault addr >> 1890754000 [fault reason 06] PTE Read access is not set >> [root@infradev-comp006.naw02.infradev.viasat.io >> <mailto:root@infradev-comp006.naw02.infradev.viasat.io> >> <mailto:root@infradev-comp006.naw02.infradev.viasat.io >> <mailto:root@infradev-comp006.naw02.infradev.viasat.io>> ~] >> # >> >> Thanks. >> >> >> Hi Ravi, >> >> The "iommu=pt" workaround applies only when you want to use >> igb_uio >> driver. VFIO driver is able to fully utilize IOMMU without >> the need >> for pass-through mode. From your log i can see that some >> devices are >> bound to igb_uio while others are bound to vfio-pci. Just >> bind all >> of the devices you want to use with DPDK to vfio-pci and >> these >> errors should go away. >> >> -- Thanks, >> Anatoly >> >> >> >> >> -- Thanks, >> Anatoly >> >> >> > Hi Ravi, > > Using vfio-pci in IOMMU mode will only work if your VM provides IOMMU > emulation (it's a fairly recent development, so your QEMU must be of an > appropriate version - can't recall which one off the top of my head). > Otherwise you'd have to treat your VM as if it was a machine without IOMMU, > i.e. use noiommu mode for VFIO, or igb_uio driver. > > -- > Thanks, > Anatoly > Hi Anatoly, Thanks. I am following wiki link below which uses vIOMMU with DPDK as a use-case and instantiate VM as specified with Q35 chipset in Qemu. https://wiki.qemu.org/Features/VT-d Qemu-version is 2.11 Host kernel 4.9 Guest kernel 4.4 I can only guess that guest kernel needs an upgrade in my setup to work correctly, if versions on my setup rings a bell on not having support kindly let me know. When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on guest I get following error ... vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored ... in guest. Thanks. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-01-31 21:51 ` Ravi Kerur @ 2018-02-01 10:10 ` Burakov, Anatoly 2018-02-01 19:26 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-01 10:10 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 31-Jan-18 9:51 PM, Ravi Kerur wrote: > > Hi Anatoly, > > Thanks. I am following wiki link below which uses vIOMMU with DPDK as a > use-case and instantiate VM as specified with Q35 chipset in Qemu. > > https://wiki.qemu.org/Features/VT-d > > Qemu-version is 2.11 > Host kernel 4.9 > Guest kernel 4.4 > > I can only guess that guest kernel needs an upgrade in my setup to work > correctly, if versions on my setup rings a bell on not having support > kindly let me know. > > When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on guest I > get following error > ... > vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored > ... > > in guest. > > Thanks. AFAIK kernel 4.4 should have noiommu mode - it was introduced in 3.1x days. However, in order for that to work, kernel also has to be built with this mode enabled. My guess is, whoever is the supplier of your kernel, did not do that. You should double-check the kernel configuration of your distribution. However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode - "regular" vfio should work fine. noiommu mode should only be needed if you know you don't have IOMMU enabled in your kernel, and even if you can't enable it, you can still use igb_uio. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-01 10:10 ` Burakov, Anatoly @ 2018-02-01 19:26 ` Ravi Kerur 2018-02-02 10:28 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-01 19:26 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly <anatoly.burakov@intel.com> wrote: > On 31-Jan-18 9:51 PM, Ravi Kerur wrote: > >> >> Hi Anatoly, >> >> Thanks. I am following wiki link below which uses vIOMMU with DPDK as a >> use-case and instantiate VM as specified with Q35 chipset in Qemu. >> >> https://wiki.qemu.org/Features/VT-d >> >> Qemu-version is 2.11 >> Host kernel 4.9 >> Guest kernel 4.4 >> >> I can only guess that guest kernel needs an upgrade in my setup to work >> correctly, if versions on my setup rings a bell on not having support >> kindly let me know. >> >> When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on guest I >> get following error >> ... >> vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored >> ... >> >> in guest. >> >> Thanks. >> > > AFAIK kernel 4.4 should have noiommu mode - it was introduced in 3.1x days. > However, in order for that to work, kernel also has to be built with this > mode enabled. My guess is, whoever is the supplier of your kernel, did not > do that. You should double-check the kernel configuration of your > distribution. > > However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode - > "regular" vfio should work fine. noiommu mode should only be needed if you > know you don't have IOMMU enabled in your kernel, and even if you can't > enable it, you can still use igb_uio. > > Hi Anatoly, Do you suggest I take this discussion to kvm/qemu mailing list as I am not sure which component has the issue? I check dmesg for BIOS physical memory map and address reported as fault by DMAR is reported by BIOS as usable on both host and vm. [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr *33a128000 *[fault reason 06] PTE Read access is not set dmesg | grep BIOS [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] usable [ 0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] reserved [ 0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] usable [ 0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] reserved [ 0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] usable [ 0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved [* 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] usable* Kindly let me know your inputs. Thanks. -- > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-01 19:26 ` Ravi Kerur @ 2018-02-02 10:28 ` Burakov, Anatoly 2018-02-02 20:21 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-02 10:28 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 01-Feb-18 7:26 PM, Ravi Kerur wrote: > > > On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: > > On 31-Jan-18 9:51 PM, Ravi Kerur wrote: > > > Hi Anatoly, > > Thanks. I am following wiki link below which uses vIOMMU with > DPDK as a use-case and instantiate VM as specified with Q35 > chipset in Qemu. > > https://wiki.qemu.org/Features/VT-d > <https://wiki.qemu.org/Features/VT-d> > > Qemu-version is 2.11 > Host kernel 4.9 > Guest kernel 4.4 > > I can only guess that guest kernel needs an upgrade in my setup > to work correctly, if versions on my setup rings a bell on not > having support kindly let me know. > > When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on > guest I get following error > ... > vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored > ... > > in guest. > > Thanks. > > > AFAIK kernel 4.4 should have noiommu mode - it was introduced in > 3.1x days. However, in order for that to work, kernel also has to be > built with this mode enabled. My guess is, whoever is the supplier > of your kernel, did not do that. You should double-check the kernel > configuration of your distribution. > > However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode > - "regular" vfio should work fine. noiommu mode should only be > needed if you know you don't have IOMMU enabled in your kernel, and > even if you can't enable it, you can still use igb_uio. > > Hi Anatoly, > > Do you suggest I take this discussion to kvm/qemu mailing list as I am > not sure which component has the issue? I check dmesg for BIOS physical > memory map and address reported as fault by DMAR is reported by BIOS as > usable on both host and vm. > > [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr > *33a128000 *[fault reason 06] PTE Read access is not set > > dmesg | grep BIOS > [ 0.000000] e820: BIOS-provided physical RAM map: > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] usable > [ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] > reserved > [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] > reserved > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] usable > [ 0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] > reserved > [ 0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] > ACPI data > [ 0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] > ACPI NVS > [ 0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] > reserved > [ 0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] usable > [ 0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] > reserved > [ 0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] usable > [ 0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] > reserved > [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] > reserved > [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] > reserved > [* 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] > usable* > * > * > Kindly let me know your inputs. > > Thanks. > > > -- > Thanks, > Anatoly > > The "PTE Read not set" error usually indicates that you are trying to use a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio when IOMMU is on). That, to me, indicates that you do have IOMMU emulation enabled. I would go about it this way. First, i'd ensure that your VM has IOMMU emulation enabled and working. You have mentioned that your QEMU version should have IOMMU emulation, so let's assume that's the case. I am not sure of the exact command-line needed to activate the vIOMMU emulation, but assuming your VM emulates an Intel processor, your kernel command-line should have "iommu=on intel_iommu=on" in it. Check /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are not in there, add the above changes, do "update-grub" and reboot your VM. If it already did have the necessary kernel configuration, do "dmesg | grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU is enabled and working in the kernel. After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it should be working. Please bear in mind that all of that is how i would've gone about it if i had similar problems on baremetal, but i'm hoping all of it is applicable to VM's. So, either disable IOMMU and use igb_uio, or enable IOMMU and use VFIO. Both will work. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-02 10:28 ` Burakov, Anatoly @ 2018-02-02 20:21 ` Ravi Kerur 2018-02-02 20:51 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-02 20:21 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly <anatoly.burakov@intel.com> wrote: > On 01-Feb-18 7:26 PM, Ravi Kerur wrote: > >> >> >> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly < >> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: >> >> On 31-Jan-18 9:51 PM, Ravi Kerur wrote: >> >> >> Hi Anatoly, >> >> Thanks. I am following wiki link below which uses vIOMMU with >> DPDK as a use-case and instantiate VM as specified with Q35 >> chipset in Qemu. >> >> https://wiki.qemu.org/Features/VT-d >> <https://wiki.qemu.org/Features/VT-d> >> >> Qemu-version is 2.11 >> Host kernel 4.9 >> Guest kernel 4.4 >> >> I can only guess that guest kernel needs an upgrade in my setup >> to work correctly, if versions on my setup rings a bell on not >> having support kindly let me know. >> >> When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on >> guest I get following error >> ... >> vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored >> ... >> >> in guest. >> >> Thanks. >> >> >> AFAIK kernel 4.4 should have noiommu mode - it was introduced in >> 3.1x days. However, in order for that to work, kernel also has to be >> built with this mode enabled. My guess is, whoever is the supplier >> of your kernel, did not do that. You should double-check the kernel >> configuration of your distribution. >> >> However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode >> - "regular" vfio should work fine. noiommu mode should only be >> needed if you know you don't have IOMMU enabled in your kernel, and >> even if you can't enable it, you can still use igb_uio. >> >> Hi Anatoly, >> >> Do you suggest I take this discussion to kvm/qemu mailing list as I am >> not sure which component has the issue? I check dmesg for BIOS physical >> memory map and address reported as fault by DMAR is reported by BIOS as >> usable on both host and vm. >> >> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr >> *33a128000 *[fault reason 06] PTE Read access is not set >> >> dmesg | grep BIOS >> [ 0.000000] e820: BIOS-provided physical RAM map: >> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] >> ACPI data >> [ 0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] >> ACPI NVS >> [ 0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] >> usable >> [ 0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] >> reserved >> [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] >> reserved >> [* 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] >> usable* >> * >> * >> Kindly let me know your inputs. >> >> Thanks. >> >> >> -- Thanks, >> Anatoly >> >> >> > The "PTE Read not set" error usually indicates that you are trying to use > a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio > when IOMMU is on). That, to me, indicates that you do have IOMMU emulation > enabled. > > I would go about it this way. > > First, i'd ensure that your VM has IOMMU emulation enabled and working. > You have mentioned that your QEMU version should have IOMMU emulation, so > let's assume that's the case. > > I am not sure of the exact command-line needed to activate the vIOMMU > emulation, but assuming your VM emulates an Intel processor, your kernel > command-line should have "iommu=on intel_iommu=on" in it. Check > /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above > values are not in there, add the above changes, do "update-grub" and reboot > your VM. > > If it already did have the necessary kernel configuration, do "dmesg | > grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU > is enabled and working in the kernel. > > After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it > should be working. Please bear in mind that all of that is how i would've > gone about it if i had similar problems on baremetal, but i'm hoping all of > it is applicable to VM's. So, either disable IOMMU and use igb_uio, or > enable IOMMU and use VFIO. Both will work. > > Thanks for your help so far on this. I have everything enabled as you have mentioned in your email. Below is the error from EAL when DPDK is run on a VM. I am referring EAL code to check why it is failing. ************VM DPDK errors*********** EAL: Detected 14 lcore(s) EAL: Probing VFIO support... EAL: VFIO support initialized EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: PCI device 0000:00:07.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 8086:10ed net_ixgbe_vf EAL: using IOMMU type 1 (Type 1) *EAL: RTE_IOVA_VA 140051367329792EAL: cannot set up DMA remapping, error 14 (Bad address)EAL: 0000:00:07.0 DMA remapping failed, error 14 (Bad address)* EAL: Requested device 0000:00:07.0 cannot be used EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: PCI device 0000:04:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: PCI device 0000:05:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: No probed ethernet devices Interactive-mode selected USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 ********************VM************ root@sriov-ravi-01:~# lsmod | grep vfio *vfio_pci 40960 1vfio_virqfd 16384 1 vfio_pcivfio_iommu_type1 20480 1vfio 28672 5 vfio_iommu_type1,vfio_pciirqbypass 16384 3 kvm,vfio_pci* root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR 00000001 BXPC 00000001) *[ 0.000000] DMAR: IOMMU enabled* [ 9.423839] DMAR: Host address width 39 [ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 [ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 12008c22260286 ecap f00f5e [ 9.427314] DMAR: ATSR flags: 0x1 [ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 [ 9.429255] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 11.167334] DMAR: No RMRR found [ 11.168855] DMAR: dmar0: Using Queued invalidation [ 12.119060] DMAR: Setting RMRR: [ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC [ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O root@sriov-ravi-01:~# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=* Network devices using kernel driver =================================== 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> root@sriov-ravi-01:~# root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i vfio [ 77.676711] VFIO - User Level meta-driver version: 0.3 [ 149.013806] VFIO - User Level meta-driver version: 0.3 [ 275.876440] VFIO - User Level meta-driver version: 0.3 Thanks, Ravi -- > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-02 20:21 ` Ravi Kerur @ 2018-02-02 20:51 ` Ravi Kerur 2018-02-05 10:01 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-02 20:51 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rkerur@gmail.com> wrote: > > > On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly < > anatoly.burakov@intel.com> wrote: > >> On 01-Feb-18 7:26 PM, Ravi Kerur wrote: >> >>> >>> >>> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly < >>> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: >>> >>> On 31-Jan-18 9:51 PM, Ravi Kerur wrote: >>> >>> >>> Hi Anatoly, >>> >>> Thanks. I am following wiki link below which uses vIOMMU with >>> DPDK as a use-case and instantiate VM as specified with Q35 >>> chipset in Qemu. >>> >>> https://wiki.qemu.org/Features/VT-d >>> <https://wiki.qemu.org/Features/VT-d> >>> >>> Qemu-version is 2.11 >>> Host kernel 4.9 >>> Guest kernel 4.4 >>> >>> I can only guess that guest kernel needs an upgrade in my setup >>> to work correctly, if versions on my setup rings a bell on not >>> having support kindly let me know. >>> >>> When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on >>> guest I get following error >>> ... >>> vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored >>> ... >>> >>> in guest. >>> >>> Thanks. >>> >>> >>> AFAIK kernel 4.4 should have noiommu mode - it was introduced in >>> 3.1x days. However, in order for that to work, kernel also has to be >>> built with this mode enabled. My guess is, whoever is the supplier >>> of your kernel, did not do that. You should double-check the kernel >>> configuration of your distribution. >>> >>> However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode >>> - "regular" vfio should work fine. noiommu mode should only be >>> needed if you know you don't have IOMMU enabled in your kernel, and >>> even if you can't enable it, you can still use igb_uio. >>> >>> Hi Anatoly, >>> >>> Do you suggest I take this discussion to kvm/qemu mailing list as I am >>> not sure which component has the issue? I check dmesg for BIOS physical >>> memory map and address reported as fault by DMAR is reported by BIOS as >>> usable on both host and vm. >>> >>> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr >>> *33a128000 *[fault reason 06] PTE Read access is not set >>> >>> dmesg | grep BIOS >>> [ 0.000000] e820: BIOS-provided physical RAM map: >>> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] >>> ACPI data >>> [ 0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] >>> ACPI NVS >>> [ 0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] >>> usable >>> [ 0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] >>> reserved >>> [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] >>> reserved >>> [* 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] >>> usable* >>> * >>> * >>> Kindly let me know your inputs. >>> >>> Thanks. >>> >>> >>> -- Thanks, >>> Anatoly >>> >>> >>> >> The "PTE Read not set" error usually indicates that you are trying to use >> a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio >> when IOMMU is on). That, to me, indicates that you do have IOMMU emulation >> enabled. >> >> I would go about it this way. >> >> First, i'd ensure that your VM has IOMMU emulation enabled and working. >> You have mentioned that your QEMU version should have IOMMU emulation, so >> let's assume that's the case. >> >> I am not sure of the exact command-line needed to activate the vIOMMU >> emulation, but assuming your VM emulates an Intel processor, your kernel >> command-line should have "iommu=on intel_iommu=on" in it. Check >> /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above >> values are not in there, add the above changes, do "update-grub" and reboot >> your VM. >> >> If it already did have the necessary kernel configuration, do "dmesg | >> grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU >> is enabled and working in the kernel. >> >> After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it >> should be working. Please bear in mind that all of that is how i would've >> gone about it if i had similar problems on baremetal, but i'm hoping all of >> it is applicable to VM's. So, either disable IOMMU and use igb_uio, or >> enable IOMMU and use VFIO. Both will work. >> >> > Thanks for your help so far on this. I have everything enabled as you have > mentioned in your email. Below is the error from EAL when DPDK is run on a > VM. I am referring EAL code to check why it is failing. > > ************VM DPDK errors*********** > EAL: Detected 14 lcore(s) > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > unreliable clock cycles ! > EAL: PCI device 0000:00:07.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 8086:10ed net_ixgbe_vf > EAL: using IOMMU type 1 (Type 1) > > > *EAL: RTE_IOVA_VA 140051367329792EAL: cannot set up DMA remapping, > error 14 (Bad address)EAL: 0000:00:07.0 DMA remapping failed, error 14 > (Bad address)* > EAL: Requested device 0000:00:07.0 cannot be used > EAL: PCI device 0000:03:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:04:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:05:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: No probed ethernet devices > Interactive-mode selected > USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, > socket=0 > > > ********************VM************ > root@sriov-ravi-01:~# lsmod | grep vfio > > > > > *vfio_pci 40960 1vfio_virqfd 16384 1 > vfio_pcivfio_iommu_type1 20480 1vfio 28672 5 > vfio_iommu_type1,vfio_pciirqbypass 16384 3 kvm,vfio_pci* > root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU > [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR > 00000001 BXPC 00000001) > *[ 0.000000] DMAR: IOMMU enabled* > [ 9.423839] DMAR: Host address width 39 > [ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 > [ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap > 12008c22260286 ecap f00f5e > [ 9.427314] DMAR: ATSR flags: 0x1 > [ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 > [ 9.429255] DMAR-IR: Queued invalidation will be enabled to support > x2apic and Intr-remapping. > [ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 11.167334] DMAR: No RMRR found > [ 11.168855] DMAR: dmar0: Using Queued invalidation > [ 12.119060] DMAR: Setting RMRR: > [ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - > 0xffffff] > [ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O > root@sriov-ravi-01:~# dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed' > drv=vfio-pci unused=* > > Network devices using kernel driver > =================================== > 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* > 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci > 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci > > Other Network devices > ===================== > <none> > > Crypto devices using DPDK-compatible driver > =========================================== > <none> > > Crypto devices using kernel driver > ================================== > <none> > > Other Crypto devices > ==================== > <none> > > Eventdev devices using DPDK-compatible driver > ============================================= > <none> > > Eventdev devices using kernel driver > ==================================== > <none> > > Other Eventdev devices > ====================== > <none> > > Mempool devices using DPDK-compatible driver > ============================================ > <none> > > Mempool devices using kernel driver > =================================== > <none> > > Other Mempool devices > ===================== > <none> > root@sriov-ravi-01:~# > > root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i > vfio > [ 77.676711] VFIO - User Level meta-driver version: 0.3 > [ 149.013806] VFIO - User Level meta-driver version: 0.3 > [ 275.876440] VFIO - User Level meta-driver version: 0.3 > > Thanks, > Ravi > > > Additonal information. On VM following message is seen [ 298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7f7300000000) [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient for the mapped address (7fa580000000) ON VM, DMAR reports address width = 39 root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -e IOMMU -e DMAR [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR 00000001 BXPC 00000001) [ 0.000000] DMAR: IOMMU enabled [ 9.209145] DMAR: Host address width 39 On Host, DMAR reports address width = 46 # dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I 00000001 INTL 20091013) [ 0.000000] DMAR: IOMMU enabled [ 0.519368] DMAR: Host address width 46 [ 0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 Searching for IOMMU errors found following link from RedHat which points to DPDK 17.11 issue. https://bugzilla.redhat.com/show_bug.cgi?id=1530957 Thanks. > -- >> Thanks, >> Anatoly >> > > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-02 20:51 ` Ravi Kerur @ 2018-02-05 10:01 ` Burakov, Anatoly 2018-02-06 17:55 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-05 10:01 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 02-Feb-18 8:51 PM, Ravi Kerur wrote: > > > On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rkerur@gmail.com > <mailto:rkerur@gmail.com>> wrote: > > > > On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: > > On 01-Feb-18 7:26 PM, Ravi Kerur wrote: > > > > On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly > <anatoly.burakov@intel.com > <mailto:anatoly.burakov@intel.com> > <mailto:anatoly.burakov@intel.com > <mailto:anatoly.burakov@intel.com>>> wrote: > > On 31-Jan-18 9:51 PM, Ravi Kerur wrote: > > > Hi Anatoly, > > Thanks. I am following wiki link below which uses > vIOMMU with > DPDK as a use-case and instantiate VM as specified > with Q35 > chipset in Qemu. > > https://wiki.qemu.org/Features/VT-d > <https://wiki.qemu.org/Features/VT-d> > <https://wiki.qemu.org/Features/VT-d > <https://wiki.qemu.org/Features/VT-d>> > > Qemu-version is 2.11 > Host kernel 4.9 > Guest kernel 4.4 > > I can only guess that guest kernel needs an upgrade > in my setup > to work correctly, if versions on my setup rings a > bell on not > having support kindly let me know. > > When 'modprobe vfio enable_unsafe_noiommu_node=Y' > is executed on > guest I get following error > ... > vfio: unknown parameter > 'enable_unsafe_noiommu_node' ignored > ... > > in guest. > > Thanks. > > > AFAIK kernel 4.4 should have noiommu mode - it was > introduced in > 3.1x days. However, in order for that to work, kernel > also has to be > built with this mode enabled. My guess is, whoever is > the supplier > of your kernel, did not do that. You should > double-check the kernel > configuration of your distribution. > > However, if you have vIOMMU in QEMU, you shouldn't need > noiommu mode > - "regular" vfio should work fine. noiommu mode should > only be > needed if you know you don't have IOMMU enabled in your > kernel, and > even if you can't enable it, you can still use igb_uio. > > Hi Anatoly, > > Do you suggest I take this discussion to kvm/qemu mailing > list as I am not sure which component has the issue? I check > dmesg for BIOS physical memory map and address reported as > fault by DMAR is reported by BIOS as usable on both host and vm. > > [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] > fault addr *33a128000 *[fault reason 06] PTE Read access is > not set > > dmesg | grep BIOS > [ 0.000000] e820: BIOS-provided physical RAM map: > [ 0.000000] BIOS-e820: [mem > 0x0000000000000000-0x000000000009afff] usable > [ 0.000000] BIOS-e820: [mem > 0x000000000009b000-0x000000000009ffff] reserved > [ 0.000000] BIOS-e820: [mem > 0x00000000000e0000-0x00000000000fffff] reserved > [ 0.000000] BIOS-e820: [mem > 0x0000000000100000-0x000000007938afff] usable > [ 0.000000] BIOS-e820: [mem > 0x000000007938b000-0x000000007994bfff] reserved > [ 0.000000] BIOS-e820: [mem > 0x000000007994c000-0x000000007999cfff] ACPI data > [ 0.000000] BIOS-e820: [mem > 0x000000007999d000-0x0000000079f7dfff] ACPI NVS > [ 0.000000] BIOS-e820: [mem > 0x0000000079f7e000-0x000000007bd37fff] reserved > [ 0.000000] BIOS-e820: [mem > 0x000000007bd38000-0x000000007bd38fff] usable > [ 0.000000] BIOS-e820: [mem > 0x000000007bd39000-0x000000007bdbefff] reserved > [ 0.000000] BIOS-e820: [mem > 0x000000007bdbf000-0x000000007bffffff] usable > [ 0.000000] BIOS-e820: [mem > 0x000000007c000000-0x000000008fffffff] reserved > [ 0.000000] BIOS-e820: [mem > 0x00000000fed1c000-0x00000000fed44fff] reserved > [ 0.000000] BIOS-e820: [mem > 0x00000000ff000000-0x00000000ffffffff] reserved > [* 0.000000] BIOS-e820: [mem > 0x0000000100000000-0x000000407fffffff] usable* > * > * > Kindly let me know your inputs. > > Thanks. > > > -- Thanks, > Anatoly > > > > The "PTE Read not set" error usually indicates that you are > trying to use a non-IOMMU method when you have IOMMU enabled > (i.e. trying to use igb_uio when IOMMU is on). That, to me, > indicates that you do have IOMMU emulation enabled. > > I would go about it this way. > > First, i'd ensure that your VM has IOMMU emulation enabled and > working. You have mentioned that your QEMU version should have > IOMMU emulation, so let's assume that's the case. > > I am not sure of the exact command-line needed to activate the > vIOMMU emulation, but assuming your VM emulates an Intel > processor, your kernel command-line should have "iommu=on > intel_iommu=on" in it. Check /etc/default/grub for > GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are > not in there, add the above changes, do "update-grub" and reboot > your VM. > > If it already did have the necessary kernel configuration, do > "dmesg | grep IOMMU" and look for "IOMMU Enabled". That should > tell you that IOMMU is enabled and working in the kernel. > > After that, you can modprobe vfio and vfio-pci, bind NICs to it, > and it should be working. Please bear in mind that all of that > is how i would've gone about it if i had similar problems on > baremetal, but i'm hoping all of it is applicable to VM's. So, > either disable IOMMU and use igb_uio, or enable IOMMU and use > VFIO. Both will work. > > > Thanks for your help so far on this. I have everything enabled as > you have mentioned in your email. Below is the error from EAL when > DPDK is run on a VM. I am referring EAL code to check why it is failing. > > ************VM DPDK errors*********** > EAL: Detected 14 lcore(s) > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > unreliable clock cycles ! > EAL: PCI device 0000:00:07.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 8086:10ed net_ixgbe_vf > EAL: using IOMMU type 1 (Type 1) > *EAL: RTE_IOVA_VA 140051367329792 > EAL: cannot set up DMA remapping, error 14 (Bad address) > EAL: 0000:00:07.0 DMA remapping failed, error 14 (Bad address)* > EAL: Requested device 0000:00:07.0 cannot be used > EAL: PCI device 0000:03:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:04:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: PCI device 0000:05:00.0 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 1af4:1041 net_virtio > EAL: No probed ethernet devices > Interactive-mode selected > USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, > size=2176, socket=0 > > > ********************VM************ > root@sriov-ravi-01:~# lsmod | grep vfio > *vfio_pci 40960 1 > vfio_virqfd 16384 1 vfio_pci > vfio_iommu_type1 20480 1 > vfio 28672 5 vfio_iommu_type1,vfio_pci > irqbypass 16384 3 kvm,vfio_pci* > root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU > [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS > BXPCDMAR 00000001 BXPC 00000001) > *[ 0.000000] DMAR: IOMMU enabled* > [ 9.423839] DMAR: Host address width 39 > [ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 > [ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap > 12008c22260286 ecap f00f5e > [ 9.427314] DMAR: ATSR flags: 0x1 > [ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 > [ 9.429255] DMAR-IR: Queued invalidation will be enabled to > support x2apic and Intr-remapping. > [ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 11.167334] DMAR: No RMRR found > [ 11.168855] DMAR: dmar0: Using Queued invalidation > [ 12.119060] DMAR: Setting RMRR: > [ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0 > [0x0 - 0xffffff] > [ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O > root@sriov-ravi-01:~# dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed' > drv=vfio-pci unused=* > > Network devices using kernel driver > =================================== > 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci > *Active* > 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci > 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci > > Other Network devices > ===================== > <none> > > Crypto devices using DPDK-compatible driver > =========================================== > <none> > > Crypto devices using kernel driver > ================================== > <none> > > Other Crypto devices > ==================== > <none> > > Eventdev devices using DPDK-compatible driver > ============================================= > <none> > > Eventdev devices using kernel driver > ==================================== > <none> > > Other Eventdev devices > ====================== > <none> > > Mempool devices using DPDK-compatible driver > ============================================ > <none> > > Mempool devices using kernel driver > =================================== > <none> > > Other Mempool devices > ===================== > <none> > root@sriov-ravi-01:~# > > root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | > grep -i vfio > [ 77.676711] VFIO - User Level meta-driver version: 0.3 > [ 149.013806] VFIO - User Level meta-driver version: 0.3 > [ 275.876440] VFIO - User Level meta-driver version: 0.3 > > Thanks, > Ravi > > > > Additonal information. On VM following message is seen > > [ 298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient > for the mapped address (7f7300000000) > [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient > for the mapped address (7fa580000000) > > ON VM, DMAR reports address width = 39 > root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep > -e IOMMU -e DMAR > [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR > 00000001 BXPC 00000001) > [ 0.000000] DMAR: IOMMU enabled > [ 9.209145] DMAR: Host address width 39 > > On Host, DMAR reports address width = 46 > # dmesg | grep -e DMAR -e IOMMU > [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I > 00000001 INTL 20091013) > [ 0.000000] DMAR: IOMMU enabled > [ 0.519368] DMAR: Host address width 46 > [ 0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > > Searching for IOMMU errors found following link from RedHat which points > to DPDK 17.11 issue. > > https://bugzilla.redhat.com/show_bug.cgi?id=1530957 > > Thanks. > > -- > Thanks, > Anatoly > > > Ah, yes, that. We have recently applied a patch dealing with exactly this: http://dpdk.org/dev/patchwork/patch/33650/ Unfortunately, as far as i know, other than refusing to run (like the patch does), there's no way to fix this issue, so you'll have to disable IOMMU in your VM, and use either igb_uio or vfio in noiommu mode. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-05 10:01 ` Burakov, Anatoly @ 2018-02-06 17:55 ` Ravi Kerur 2018-02-08 11:20 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-06 17:55 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Mon, Feb 5, 2018 at 2:01 AM, Burakov, Anatoly <anatoly.burakov@intel.com> wrote: > On 02-Feb-18 8:51 PM, Ravi Kerur wrote: > >> >> >> On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rkerur@gmail.com <mailto: >> rkerur@gmail.com>> wrote: >> >> >> >> On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly >> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: >> >> On 01-Feb-18 7:26 PM, Ravi Kerur wrote: >> >> >> >> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly >> <anatoly.burakov@intel.com >> <mailto:anatoly.burakov@intel.com> >> <mailto:anatoly.burakov@intel.com >> >> <mailto:anatoly.burakov@intel.com>>> wrote: >> >> On 31-Jan-18 9:51 PM, Ravi Kerur wrote: >> >> >> Hi Anatoly, >> >> Thanks. I am following wiki link below which uses >> vIOMMU with >> DPDK as a use-case and instantiate VM as specified >> with Q35 >> chipset in Qemu. >> >> https://wiki.qemu.org/Features/VT-d >> <https://wiki.qemu.org/Features/VT-d> >> <https://wiki.qemu.org/Features/VT-d >> <https://wiki.qemu.org/Features/VT-d>> >> >> Qemu-version is 2.11 >> Host kernel 4.9 >> Guest kernel 4.4 >> >> I can only guess that guest kernel needs an upgrade >> in my setup >> to work correctly, if versions on my setup rings a >> bell on not >> having support kindly let me know. >> >> When 'modprobe vfio enable_unsafe_noiommu_node=Y' >> is executed on >> guest I get following error >> ... >> vfio: unknown parameter >> 'enable_unsafe_noiommu_node' ignored >> ... >> >> in guest. >> >> Thanks. >> >> >> AFAIK kernel 4.4 should have noiommu mode - it was >> introduced in >> 3.1x days. However, in order for that to work, kernel >> also has to be >> built with this mode enabled. My guess is, whoever is >> the supplier >> of your kernel, did not do that. You should >> double-check the kernel >> configuration of your distribution. >> >> However, if you have vIOMMU in QEMU, you shouldn't need >> noiommu mode >> - "regular" vfio should work fine. noiommu mode should >> only be >> needed if you know you don't have IOMMU enabled in your >> kernel, and >> even if you can't enable it, you can still use igb_uio. >> >> Hi Anatoly, >> >> Do you suggest I take this discussion to kvm/qemu mailing >> list as I am not sure which component has the issue? I check >> dmesg for BIOS physical memory map and address reported as >> fault by DMAR is reported by BIOS as usable on both host and >> vm. >> >> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] >> fault addr *33a128000 *[fault reason 06] PTE Read access is >> not set >> >> dmesg | grep BIOS >> [ 0.000000] e820: BIOS-provided physical RAM map: >> [ 0.000000] BIOS-e820: [mem >> 0x0000000000000000-0x000000000009afff] usable >> [ 0.000000] BIOS-e820: [mem >> 0x000000000009b000-0x000000000009ffff] reserved >> [ 0.000000] BIOS-e820: [mem >> 0x00000000000e0000-0x00000000000fffff] reserved >> [ 0.000000] BIOS-e820: [mem >> 0x0000000000100000-0x000000007938afff] usable >> [ 0.000000] BIOS-e820: [mem >> 0x000000007938b000-0x000000007994bfff] reserved >> [ 0.000000] BIOS-e820: [mem >> 0x000000007994c000-0x000000007999cfff] ACPI data >> [ 0.000000] BIOS-e820: [mem >> 0x000000007999d000-0x0000000079f7dfff] ACPI NVS >> [ 0.000000] BIOS-e820: [mem >> 0x0000000079f7e000-0x000000007bd37fff] reserved >> [ 0.000000] BIOS-e820: [mem >> 0x000000007bd38000-0x000000007bd38fff] usable >> [ 0.000000] BIOS-e820: [mem >> 0x000000007bd39000-0x000000007bdbefff] reserved >> [ 0.000000] BIOS-e820: [mem >> 0x000000007bdbf000-0x000000007bffffff] usable >> [ 0.000000] BIOS-e820: [mem >> 0x000000007c000000-0x000000008fffffff] reserved >> [ 0.000000] BIOS-e820: [mem >> 0x00000000fed1c000-0x00000000fed44fff] reserved >> [ 0.000000] BIOS-e820: [mem >> 0x00000000ff000000-0x00000000ffffffff] reserved >> [* 0.000000] BIOS-e820: [mem >> 0x0000000100000000-0x000000407fffffff] usable* >> * >> * >> Kindly let me know your inputs. >> >> Thanks. >> >> >> -- Thanks, >> Anatoly >> >> >> >> The "PTE Read not set" error usually indicates that you are >> trying to use a non-IOMMU method when you have IOMMU enabled >> (i.e. trying to use igb_uio when IOMMU is on). That, to me, >> indicates that you do have IOMMU emulation enabled. >> >> I would go about it this way. >> >> First, i'd ensure that your VM has IOMMU emulation enabled and >> working. You have mentioned that your QEMU version should have >> IOMMU emulation, so let's assume that's the case. >> >> I am not sure of the exact command-line needed to activate the >> vIOMMU emulation, but assuming your VM emulates an Intel >> processor, your kernel command-line should have "iommu=on >> intel_iommu=on" in it. Check /etc/default/grub for >> GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are >> not in there, add the above changes, do "update-grub" and reboot >> your VM. >> >> If it already did have the necessary kernel configuration, do >> "dmesg | grep IOMMU" and look for "IOMMU Enabled". That should >> tell you that IOMMU is enabled and working in the kernel. >> >> After that, you can modprobe vfio and vfio-pci, bind NICs to it, >> and it should be working. Please bear in mind that all of that >> is how i would've gone about it if i had similar problems on >> baremetal, but i'm hoping all of it is applicable to VM's. So, >> either disable IOMMU and use igb_uio, or enable IOMMU and use >> VFIO. Both will work. >> >> >> Thanks for your help so far on this. I have everything enabled as >> you have mentioned in your email. Below is the error from EAL when >> DPDK is run on a VM. I am referring EAL code to check why it is >> failing. >> >> ************VM DPDK errors*********** >> EAL: Detected 14 lcore(s) >> EAL: Probing VFIO support... >> EAL: VFIO support initialized >> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using >> unreliable clock cycles ! >> EAL: PCI device 0000:00:07.0 on NUMA socket -1 >> EAL: Invalid NUMA socket, default to 0 >> EAL: probe driver: 8086:10ed net_ixgbe_vf >> EAL: using IOMMU type 1 (Type 1) >> *EAL: RTE_IOVA_VA 140051367329792 >> EAL: cannot set up DMA remapping, error 14 (Bad address) >> EAL: 0000:00:07.0 DMA remapping failed, error 14 (Bad address)* >> EAL: Requested device 0000:00:07.0 cannot be used >> EAL: PCI device 0000:03:00.0 on NUMA socket -1 >> EAL: Invalid NUMA socket, default to 0 >> EAL: probe driver: 1af4:1041 net_virtio >> EAL: PCI device 0000:04:00.0 on NUMA socket -1 >> EAL: Invalid NUMA socket, default to 0 >> EAL: probe driver: 1af4:1041 net_virtio >> EAL: PCI device 0000:05:00.0 on NUMA socket -1 >> EAL: Invalid NUMA socket, default to 0 >> EAL: probe driver: 1af4:1041 net_virtio >> EAL: No probed ethernet devices >> Interactive-mode selected >> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, >> size=2176, socket=0 >> >> >> ********************VM************ >> root@sriov-ravi-01:~# lsmod | grep vfio >> *vfio_pci 40960 1 >> vfio_virqfd 16384 1 vfio_pci >> vfio_iommu_type1 20480 1 >> vfio 28672 5 vfio_iommu_type1,vfio_pci >> irqbypass 16384 3 kvm,vfio_pci* >> root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU >> [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS >> BXPCDMAR 00000001 BXPC 00000001) >> *[ 0.000000] DMAR: IOMMU enabled* >> [ 9.423839] DMAR: Host address width 39 >> [ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 >> [ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap >> 12008c22260286 ecap f00f5e >> [ 9.427314] DMAR: ATSR flags: 0x1 >> [ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU >> 0 >> [ 9.429255] DMAR-IR: Queued invalidation will be enabled to >> support x2apic and Intr-remapping. >> [ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode >> [ 11.167334] DMAR: No RMRR found >> [ 11.168855] DMAR: dmar0: Using Queued invalidation >> [ 12.119060] DMAR: Setting RMRR: >> [ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC >> [ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0 >> [0x0 - 0xffffff] >> [ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed >> I/O >> root@sriov-ravi-01:~# dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed' >> drv=vfio-pci unused=* >> >> >> Network devices using kernel driver >> =================================== >> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci >> *Active* >> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci >> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci >> >> Other Network devices >> ===================== >> <none> >> >> Crypto devices using DPDK-compatible driver >> =========================================== >> <none> >> >> Crypto devices using kernel driver >> ================================== >> <none> >> >> Other Crypto devices >> ==================== >> <none> >> >> Eventdev devices using DPDK-compatible driver >> ============================================= >> <none> >> >> Eventdev devices using kernel driver >> ==================================== >> <none> >> >> Other Eventdev devices >> ====================== >> <none> >> >> Mempool devices using DPDK-compatible driver >> ============================================ >> <none> >> >> Mempool devices using kernel driver >> =================================== >> <none> >> >> Other Mempool devices >> ===================== >> <none> >> root@sriov-ravi-01:~# >> >> root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | >> grep -i vfio >> [ 77.676711] VFIO - User Level meta-driver version: 0.3 >> [ 149.013806] VFIO - User Level meta-driver version: 0.3 >> [ 275.876440] VFIO - User Level meta-driver version: 0.3 >> >> Thanks, >> Ravi >> >> >> >> Additonal information. On VM following message is seen >> >> [ 298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient >> for the mapped address (7f7300000000) >> [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient >> for the mapped address (7fa580000000) >> >> ON VM, DMAR reports address width = 39 >> root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep >> -e IOMMU -e DMAR >> [ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR >> 00000001 BXPC 00000001) >> [ 0.000000] DMAR: IOMMU enabled >> [ 9.209145] DMAR: Host address width 39 >> >> On Host, DMAR reports address width = 46 >> # dmesg | grep -e DMAR -e IOMMU >> [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I >> 00000001 INTL 20091013) >> [ 0.000000] DMAR: IOMMU enabled >> [ 0.519368] DMAR: Host address width 46 >> [ 0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 >> >> Searching for IOMMU errors found following link from RedHat which points >> to DPDK 17.11 issue. >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1530957 >> >> Thanks. >> >> -- Thanks, >> Anatoly >> >> >> >> > Ah, yes, that. We have recently applied a patch dealing with exactly this: > > http://dpdk.org/dev/patchwork/patch/33650/ > > Unfortunately, as far as i know, other than refusing to run (like the > patch does), there's no way to fix this issue, so you'll have to disable > IOMMU in your VM, and use either igb_uio or vfio in noiommu mode. > > Hi Anatoly, I am actually confused with the state of vIOMMU + DPDK. Can you please help me clarify? I tested following DPDK versions (1) DPDK 17.11, exhibits the issue (IOMMU width as reported by RedHat and solution is to prevent using the patch) (2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to bind a device in VM with VFIO driver and no DMAR error message on host (3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to bind a device in VM with VFIO driver and no DMAR error message on host Clearly issue seen in 17.11 without the patch you mentioned is a regression or the issue was masked in earlier DPDK version? I did not test traffic with any DPDK version because I wanted to first get DMAR errors on host gone. Our application 'v1' is integrated with DPDK 16.11 and 'v2' is integrated with DPDK 17.05.01. In both 'v1' and 'v2' cases I don't see IOMMU width error messages on VM, however, DMAR error messages are seen host. I am not able to relate what causes DMAR error messages on host? Thanks. > -- > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-06 17:55 ` Ravi Kerur @ 2018-02-08 11:20 ` Burakov, Anatoly 2018-02-09 17:41 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-08 11:20 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 06-Feb-18 5:55 PM, Ravi Kerur wrote: > > Hi Anatoly, > > I am actually confused with the state of vIOMMU + DPDK. Can you please > help me clarify? > > I tested following DPDK versions > > (1) DPDK 17.11, exhibits the issue (IOMMU width as reported by RedHat > and solution is to prevent using the patch) > (2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to bind a > device in VM with VFIO driver and no DMAR error message on host > (3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to bind a > device in VM with VFIO driver and no DMAR error message on host > > Clearly issue seen in 17.11 without the patch you mentioned is a > regression or the issue was masked in earlier DPDK version? I did not > test traffic with any DPDK version because I wanted to first get DMAR > errors on host gone. > > Our application 'v1' is integrated with DPDK 16.11 and 'v2' is > integrated with DPDK 17.05.01. In both 'v1' and 'v2' cases I don't see > IOMMU width error messages on VM, however, DMAR error messages are seen > host. I am not able to relate what causes DMAR error messages on host? > > > Thanks. > Hi Ravi, vIOMMU support is out of our hands, really - we can only make use of hardware (or emulation of it) that is available. 39-bit wide address *can* work, you just have to be lucky and get PA addresses that would fit into 39 bits (so under 512G limit), because we set up IOVA addresses to be 1:1 to physical addresses. We could, in principle, set up IOVA addresses to go from zero instead of them being a 1:1 mapping to physical addresses, but that would introduce need to translate addresses between IOVA and physical in some cases (e.g. KNI). I'm not aware of any changes between 16.11 and 17.11 (and indeed 18.02) that would make or break support for 39-bit wide PA addresses for IOMMU. It is possible that VF/PF drivers do something differently which results in DMAR errors showing up sooner rather than later, but as far as VFIO support itself is concerned, there were no major changes in those releases. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-08 11:20 ` Burakov, Anatoly @ 2018-02-09 17:41 ` Ravi Kerur 2018-02-10 10:11 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-09 17:41 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Thu, Feb 8, 2018 at 3:20 AM, Burakov, Anatoly <anatoly.burakov@intel.com> wrote: > On 06-Feb-18 5:55 PM, Ravi Kerur wrote: > >> >> Hi Anatoly, >> >> I am actually confused with the state of vIOMMU + DPDK. Can you please >> help me clarify? >> >> I tested following DPDK versions >> >> (1) DPDK 17.11, exhibits the issue (IOMMU width as reported by RedHat and >> solution is to prevent using the patch) >> (2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to bind a >> device in VM with VFIO driver and no DMAR error message on host >> (3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to bind a >> device in VM with VFIO driver and no DMAR error message on host >> >> Clearly issue seen in 17.11 without the patch you mentioned is a >> regression or the issue was masked in earlier DPDK version? I did not test >> traffic with any DPDK version because I wanted to first get DMAR errors on >> host gone. >> >> Our application 'v1' is integrated with DPDK 16.11 and 'v2' is integrated >> with DPDK 17.05.01. In both 'v1' and 'v2' cases I don't see IOMMU width >> error messages on VM, however, DMAR error messages are seen host. I am not >> able to relate what causes DMAR error messages on host? >> >> >> Thanks. >> >> > Hi Ravi, > > vIOMMU support is out of our hands, really - we can only make use of > hardware (or emulation of it) that is available. 39-bit wide address *can* > work, you just have to be lucky and get PA addresses that would fit into 39 > bits (so under 512G limit), because we set up IOVA addresses to be 1:1 to > physical addresses. We could, in principle, set up IOVA addresses to go > from zero instead of them being a 1:1 mapping to physical addresses, but > that would introduce need to translate addresses between IOVA and physical > in some cases (e.g. KNI). > > I'm not aware of any changes between 16.11 and 17.11 (and indeed 18.02) > that would make or break support for 39-bit wide PA addresses for IOMMU. It > is possible that VF/PF drivers do something differently which results in > DMAR errors showing up sooner rather than later, but as far as VFIO support > itself is concerned, there were no major changes in those releases. > > Hi Anatoly, Thank you for your explanation. I would like to ask one more thing as I need to get v-iommu+ dpdk working in VM. Can you please tell me what determines 'Host Address Width", I know my question has nothing to do with dpdk and this is a dpdk list, but if you have any information please share it? I googled and found couple of ways to influence 'Host Address Width = 46' in guest as well (since dpdk + iommu works fine on host and DMAR on host reports address width as 46). (1) Qemu has CPU param 'host-phys-bits' boolean, when set to true copies it from host (2) Qemu has 'phys-bits' integer, when set to '46' should influence guest Using above options when instantiating a VM doesn't help, Guest VM still ends up with 'Host address width = 39'. (3) There is another Qemu option 'x-aw-bits' which is for VT-d which can be set to '39' or '48'. This doesn't help either. Thanks. -- > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-09 17:41 ` Ravi Kerur @ 2018-02-10 10:11 ` Burakov, Anatoly 0 siblings, 0 replies; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-10 10:11 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 09-Feb-18 5:41 PM, Ravi Kerur wrote: > > > On Thu, Feb 8, 2018 at 3:20 AM, Burakov, Anatoly > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: > > On 06-Feb-18 5:55 PM, Ravi Kerur wrote: > > > Hi Anatoly, > > I am actually confused with the state of vIOMMU + DPDK. Can you > please help me clarify? > > I tested following DPDK versions > > (1) DPDK 17.11, exhibits the issue (IOMMU width as reported by > RedHat and solution is to prevent using the patch) > (2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to > bind a device in VM with VFIO driver and no DMAR error message > on host > (3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to > bind a device in VM with VFIO driver and no DMAR error message > on host > > Clearly issue seen in 17.11 without the patch you mentioned is a > regression or the issue was masked in earlier DPDK version? I > did not test traffic with any DPDK version because I wanted to > first get DMAR errors on host gone. > > Our application 'v1' is integrated with DPDK 16.11 and 'v2' is > integrated with DPDK 17.05.01. In both 'v1' and 'v2' cases I > don't see IOMMU width error messages on VM, however, DMAR error > messages are seen host. I am not able to relate what causes DMAR > error messages on host? > > > Thanks. > > > Hi Ravi, > > vIOMMU support is out of our hands, really - we can only make use of > hardware (or emulation of it) that is available. 39-bit wide address > *can* work, you just have to be lucky and get PA addresses that > would fit into 39 bits (so under 512G limit), because we set up IOVA > addresses to be 1:1 to physical addresses. We could, in principle, > set up IOVA addresses to go from zero instead of them being a 1:1 > mapping to physical addresses, but that would introduce need to > translate addresses between IOVA and physical in some cases (e.g. KNI). > > I'm not aware of any changes between 16.11 and 17.11 (and indeed > 18.02) that would make or break support for 39-bit wide PA addresses > for IOMMU. It is possible that VF/PF drivers do something > differently which results in DMAR errors showing up sooner rather > than later, but as far as VFIO support itself is concerned, there > were no major changes in those releases. > > > Hi Anatoly, Hi Ravi, > > Thank you for your explanation. I would like to ask one more thing as I > need to get v-iommu+ dpdk working in VM. Can you please tell me what > determines 'Host Address Width", I know my question has nothing to do > with dpdk and this is a dpdk list, but if you have any information > please share it? I googled and found couple of ways to influence 'Host > Address Width = 46' in guest as well (since dpdk + iommu works fine on > host and DMAR on host reports address width as 46). As far as i'm aware, address width supported by IOMMU is taken from hardware (real or emulated). If e.g. QEMU's vIOMMU emulates an IOMMU that only supports 39-bit addresses, that's what we'll get. Not sure what "host address width" means exactly, but if i had to guess, usually in VM-speak, "host" refers to, well, host - that is, the physical machine the VM is running on. So, under that assumption, "host address width" would be telling QEMU maximum width of IOMMU addresses on the host (i.e. what you have in hardware, on the host - for example, you might be running on a platform that only supports a 39-bit wide physical address). Naturally, changing it around wouldn't change much - it would presumably affect only the way vIOMMU remaps requests from guest to host, but not the emulated hardware itself. However, i'm not knowledgeable enough in that area to answer this question definitively. > > (1) Qemu has CPU param 'host-phys-bits' boolean, when set to true copies > it from host > (2) Qemu has 'phys-bits' integer, when set to '46' should influence guest > > Using above options when instantiating a VM doesn't help, Guest VM still > ends up with 'Host address width = 39'. > > (3) There is another Qemu option 'x-aw-bits' which is for VT-d which can > be set to '39' or '48'. This doesn't help either. > > Thanks. > > -- > Thanks, > Anatoly > > -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-01-29 22:35 ` Ravi Kerur 2018-01-31 9:59 ` Burakov, Anatoly @ 2018-02-10 10:58 ` Burakov, Anatoly 2018-02-10 17:53 ` Ravi Kerur 1 sibling, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-10 10:58 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 29-Jan-18 10:35 PM, Ravi Kerur wrote: > Hi Burakov, > > When using vfio-pci on host both VF and PF interfaces works fine with > dpdk i.e. I don't see DMAR fault messages anymore. However, when I > attach a VF interface to a VM and start DPDK with vfio-pci inside VM I > still see DMAR fault messages on host. Both host and VM are booted with > 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work > (I think it's expected because of DMAR faults), however, when VF > interface uses ixgbevf driver ping works. > > Following are some details > > /*****************On VM***************/ > dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci > unused=ixgbevf > > Network devices using kernel driver > =================================== > 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* > 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci > 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci > > Other network devices > ===================== > <none> > > Crypto devices using DPDK-compatible driver > =========================================== > <none> > > Crypto devices using kernel driver > ================================== > <none> > > Other crypto devices > ==================== > <none> > > > 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller > Virtual Function (rev 01) > Subsystem: Intel Corporation 82599 Ethernet Controller Virtual > Function > Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- > ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K] > Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K] > Capabilities: [70] MSI-X: Enable+ Count=3 Masked- > Vector table: BAR=3 offset=00000000 > PBA: BAR=3 offset=00002000 > Capabilities: [a0] Express (v1) Root Complex Integrated > Endpoint, MSI 00 > DevCap: MaxPayload 128 bytes, PhantFunc 0 > ExtTag- RBE- > DevCtl: Report errors: Correctable- Non-Fatal- Fatal- > Unsupported- > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- > MaxPayload 128 bytes, MaxReadReq 128 bytes > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- > AuxPwr- TransPend- > Capabilities: [100 v1] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr- > AERCap: First Error Pointer: 00, GenCap- CGenEn- > ChkCap- ChkEn- > Kernel driver in use: vfio-pci > Kernel modules: ixgbevf > > /***************on Host*************/ > dmesg | grep DMAR > ... > [ 978.268143] DMAR: DRHD: handling fault status reg 2 > [ 978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr > 33a128000 [fault reason 06] PTE Read access is not set > [ 1286.677726] DMAR: DRHD: handling fault status reg 102 > [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr > fb663000 [fault reason 06] PTE Read access is not set > [ 1676.436145] DMAR: DRHD: handling fault status reg 202 > [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set > [ 1734.433649] DMAR: DRHD: handling fault status reg 302 > [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set > [ 2324.428938] DMAR: DRHD: handling fault status reg 402 > [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr > 7770c000 [fault reason 06] PTE Read access is not set > [ 2388.553640] DMAR: DRHD: handling fault status reg 502 > [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr > 33a128000 [fault reason 06] PTE Read access is not set > > Going back to this, i would like to suggest run a few tests to ensure that we have all information that we can gather. First of all, i'm assuming that you're using native ixgbe Linux driver on the host, and that you're only passing through the VF device to the VM using VFIO. Is my understanding correct here? Now, let's forget about the iommu=pt and igb_uio for a moment. Boot both your host and your VM with iommu=on and intel_iommu=on (or whatever command-line enables full IOMMU support on both host and guest) and do the same tests you've done before. Do you still see your issues? It would also be very useful to also try native Linux kernel driver on the guest *with traffic forwarding* and see how it works in your VM. Therefore i would suggest you to compile DPDK with PCAP support, bind your (VM) interface to native Linux driver, and use the interface via our pcap driver (creating a vdev should do the trick - please refer to PCAP PMD documentation [1]). Simple forwarding test should be enough - just make sure to pass traffic to and from DPDK in both cases, and that it doesn't give you any DMAR errors. We can go from there. [1] http://dpdk.org/doc/guides/nics/pcap_ring.html -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-10 10:58 ` Burakov, Anatoly @ 2018-02-10 17:53 ` Ravi Kerur 2018-02-12 10:13 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-10 17:53 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Sat, Feb 10, 2018 at 2:58 AM, Burakov, Anatoly <anatoly.burakov@intel.com > wrote: > On 29-Jan-18 10:35 PM, Ravi Kerur wrote: > >> Hi Burakov, >> >> When using vfio-pci on host both VF and PF interfaces works fine with >> dpdk i.e. I don't see DMAR fault messages anymore. However, when I attach a >> VF interface to a VM and start DPDK with vfio-pci inside VM I still see >> DMAR fault messages on host. Both host and VM are booted with >> 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work (I >> think it's expected because of DMAR faults), however, when VF interface >> uses ixgbevf driver ping works. >> >> Following are some details >> >> /*****************On VM***************/ >> dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci >> unused=ixgbevf >> >> Network devices using kernel driver >> =================================== >> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active* >> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci >> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci >> >> Other network devices >> ===================== >> <none> >> >> Crypto devices using DPDK-compatible driver >> =========================================== >> <none> >> >> Crypto devices using kernel driver >> ================================== >> <none> >> >> Other crypto devices >> ==================== >> <none> >> >> >> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller >> Virtual Function (rev 01) >> Subsystem: Intel Corporation 82599 Ethernet Controller Virtual >> Function >> Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- >> ParErr- Stepping- SERR- FastB2B- DisINTx+ >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- >> <TAbort- <MAbort- >SERR- <PERR- INTx- >> Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K] >> Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K] >> Capabilities: [70] MSI-X: Enable+ Count=3 Masked- >> Vector table: BAR=3 offset=00000000 >> PBA: BAR=3 offset=00002000 >> Capabilities: [a0] Express (v1) Root Complex Integrated >> Endpoint, MSI 00 >> DevCap: MaxPayload 128 bytes, PhantFunc 0 >> ExtTag- RBE- >> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- >> Unsupported- >> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- >> MaxPayload 128 bytes, MaxReadReq 128 bytes >> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- >> TransPend- >> Capabilities: [100 v1] Advanced Error Reporting >> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- >> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- >> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- >> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >> NonFatalErr- >> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- >> NonFatalErr- >> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- >> ChkEn- >> Kernel driver in use: vfio-pci >> Kernel modules: ixgbevf >> >> /***************on Host*************/ >> dmesg | grep DMAR >> ... >> [ 978.268143] DMAR: DRHD: handling fault status reg 2 >> [ 978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> [ 1286.677726] DMAR: DRHD: handling fault status reg 102 >> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr >> fb663000 [fault reason 06] PTE Read access is not set >> [ 1676.436145] DMAR: DRHD: handling fault status reg 202 >> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> [ 1734.433649] DMAR: DRHD: handling fault status reg 302 >> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> [ 2324.428938] DMAR: DRHD: handling fault status reg 402 >> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 7770c000 [fault reason 06] PTE Read access is not set >> [ 2388.553640] DMAR: DRHD: handling fault status reg 502 >> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr >> 33a128000 [fault reason 06] PTE Read access is not set >> >> >> > Going back to this, i would like to suggest run a few tests to ensure that > we have all information that we can gather. > > First of all, i'm assuming that you're using native ixgbe Linux driver on > the host, and that you're only passing through the VF device to the VM > using VFIO. Is my understanding correct here? > > Now, let's forget about the iommu=pt and igb_uio for a moment. Boot both > your host and your VM with iommu=on and intel_iommu=on (or whatever > command-line enables full IOMMU support on both host and guest) and do the > same tests you've done before. Do you still see your issues? > > It would also be very useful to also try native Linux kernel driver on the > guest *with traffic forwarding* and see how it works in your VM. Therefore > i would suggest you to compile DPDK with PCAP support, bind your (VM) > interface to native Linux driver, and use the interface via our pcap driver > (creating a vdev should do the trick - please refer to PCAP PMD > documentation [1]). Simple forwarding test should be enough - just make > sure to pass traffic to and from DPDK in both cases, and that it doesn't > give you any DMAR errors. > > We can go from there. > > Let me just give you what has been tested and working/nonworking scenarios. Some of your questions might get answered as well. Test bed is very simple with 2 VF's created under IXGBE PF on host with one VF interface added to ovs-bridge on host and another VF interface given to guest. Test connectivity between VF's via ping. Host and guest -- Kernel 4.9 Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the git (2.11.50)) DPDK -- 17.05.1 on host and guest Host and guest -- booted with GRUB intel_iommu=on (which enables IOMMU). Have tried with "iommu=on and intel_iommu=on" as well, but iommu=on is not needed when intel_iommu=on is set. Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver ping works Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf driver ping works Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK vfio-pci driver, DMAR errors seen on host, ping doesn't work DPDK works fine on host with vfio-pci, however, has issues when used inside the guest. Please let me know if more information is needed. Thanks, Ravi [1] http://dpdk.org/doc/guides/nics/pcap_ring.html > > -- > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-10 17:53 ` Ravi Kerur @ 2018-02-12 10:13 ` Burakov, Anatoly 2018-02-12 22:00 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-12 10:13 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 10-Feb-18 5:53 PM, Ravi Kerur wrote: > > > On Sat, Feb 10, 2018 at 2:58 AM, Burakov, Anatoly > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: > > On 29-Jan-18 10:35 PM, Ravi Kerur wrote: > > Hi Burakov, > > When using vfio-pci on host both VF and PF interfaces works fine > with dpdk i.e. I don't see DMAR fault messages anymore. However, > when I attach a VF interface to a VM and start DPDK with > vfio-pci inside VM I still see DMAR fault messages on host. Both > host and VM are booted with 'intel-iommu=on' on GRUB. Ping from > VM with DPDK/vfio-pci doesn't work (I think it's expected > because of DMAR faults), however, when VF interface uses ixgbevf > driver ping works. > > Following are some details > > /*****************On VM***************/ > dpdk-devbind -s > > Network devices using DPDK-compatible driver > ============================================ > 0000:00:07.0 '82599 Ethernet Controller Virtual Function' > drv=vfio-pci unused=ixgbevf > > Network devices using kernel driver > =================================== > 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci > unused=vfio-pci *Active* > 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci > 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci > > Other network devices > ===================== > <none> > > Crypto devices using DPDK-compatible driver > =========================================== > <none> > > Crypto devices using kernel driver > ================================== > <none> > > Other crypto devices > ==================== > <none> > > > 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet > Controller Virtual Function (rev 01) > Subsystem: Intel Corporation 82599 Ethernet Controller > Virtual Function > Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- > VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Region 0: Memory at fda00000 (64-bit, prefetchable) > [size=16K] > Region 3: Memory at fda04000 (64-bit, prefetchable) > [size=16K] > Capabilities: [70] MSI-X: Enable+ Count=3 Masked- > Vector table: BAR=3 offset=00000000 > PBA: BAR=3 offset=00002000 > Capabilities: [a0] Express (v1) Root Complex > Integrated Endpoint, MSI 00 > DevCap: MaxPayload 128 bytes, PhantFunc 0 > ExtTag- RBE- > DevCtl: Report errors: Correctable- Non-Fatal- > Fatal- Unsupported- > RlxdOrd- ExtTag- PhantFunc- AuxPwr- > NoSnoop- > MaxPayload 128 bytes, MaxReadReq 128 bytes > DevSta: CorrErr- UncorrErr- FatalErr- > UnsuppReq- AuxPwr- TransPend- > Capabilities: [100 v1] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- > CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- > CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP- SDES- TLP- FCP- CmpltTO- > CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- > Timeout- NonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- > Timeout- NonFatalErr- > AERCap: First Error Pointer: 00, GenCap- > CGenEn- ChkCap- ChkEn- > Kernel driver in use: vfio-pci > Kernel modules: ixgbevf > > /***************on Host*************/ > dmesg | grep DMAR > ... > [ 978.268143] DMAR: DRHD: handling fault status reg 2 > [ 978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault > addr 33a128000 [fault reason 06] PTE Read access is not set > [ 1286.677726] DMAR: DRHD: handling fault status reg 102 > [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault > addr fb663000 [fault reason 06] PTE Read access is not set > [ 1676.436145] DMAR: DRHD: handling fault status reg 202 > [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault > addr 33a128000 [fault reason 06] PTE Read access is not set > [ 1734.433649] DMAR: DRHD: handling fault status reg 302 > [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault > addr 33a128000 [fault reason 06] PTE Read access is not set > [ 2324.428938] DMAR: DRHD: handling fault status reg 402 > [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault > addr 7770c000 [fault reason 06] PTE Read access is not set > [ 2388.553640] DMAR: DRHD: handling fault status reg 502 > [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault > addr 33a128000 [fault reason 06] PTE Read access is not set > > > > Going back to this, i would like to suggest run a few tests to > ensure that we have all information that we can gather. > > First of all, i'm assuming that you're using native ixgbe Linux > driver on the host, and that you're only passing through the VF > device to the VM using VFIO. Is my understanding correct here? > > Now, let's forget about the iommu=pt and igb_uio for a moment. Boot > both your host and your VM with iommu=on and intel_iommu=on (or > whatever command-line enables full IOMMU support on both host and > guest) and do the same tests you've done before. Do you still see > your issues? > > It would also be very useful to also try native Linux kernel driver > on the guest *with traffic forwarding* and see how it works in your > VM. Therefore i would suggest you to compile DPDK with PCAP support, > bind your (VM) interface to native Linux driver, and use the > interface via our pcap driver (creating a vdev should do the trick - > please refer to PCAP PMD documentation [1]). Simple forwarding test > should be enough - just make sure to pass traffic to and from DPDK > in both cases, and that it doesn't give you any DMAR errors. > > We can go from there. > > > Let me just give you what has been tested and working/nonworking > scenarios. Some of your questions might get answered as well. Test bed > is very simple with 2 VF's created under IXGBE PF on host with one VF > interface added to ovs-bridge on host and another VF interface given to > guest. Test connectivity between VF's via ping. > > Host and guest -- Kernel 4.9 > Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the git (2.11.50)) > DPDK -- 17.05.1 on host and guest > Host and guest -- booted with GRUB intel_iommu=on (which enables IOMMU). > Have tried with "iommu=on and intel_iommu=on" as well, but iommu=on is > not needed when intel_iommu=on is set. > > Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver ping works > Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf driver > ping works > Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK vfio-pci > driver, DMAR errors seen on host, ping doesn't work OK, that makes it clearer, thanks. Does the third scenario work in other DPDK versions? > > DPDK works fine on host with vfio-pci, however, has issues when used > inside the guest. Please let me know if more information is needed. > > Thanks, > Ravi > > [1] http://dpdk.org/doc/guides/nics/pcap_ring.html > <http://dpdk.org/doc/guides/nics/pcap_ring.html> > > -- > Thanks, > Anatoly > > -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-12 10:13 ` Burakov, Anatoly @ 2018-02-12 22:00 ` Ravi Kerur 2018-02-13 14:31 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-12 22:00 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Mon, Feb 12, 2018 at 2:13 AM, Burakov, Anatoly <anatoly.burakov@intel.com > wrote: > On 10-Feb-18 5:53 PM, Ravi Kerur wrote: > > >> >> On Sat, Feb 10, 2018 at 2:58 AM, Burakov, Anatoly < >> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: >> >> On 29-Jan-18 10:35 PM, Ravi Kerur wrote: >> >> Hi Burakov, >> >> When using vfio-pci on host both VF and PF interfaces works fine >> with dpdk i.e. I don't see DMAR fault messages anymore. However, >> when I attach a VF interface to a VM and start DPDK with >> vfio-pci inside VM I still see DMAR fault messages on host. Both >> host and VM are booted with 'intel-iommu=on' on GRUB. Ping from >> VM with DPDK/vfio-pci doesn't work (I think it's expected >> because of DMAR faults), however, when VF interface uses ixgbevf >> driver ping works. >> >> Following are some details >> >> /*****************On VM***************/ >> dpdk-devbind -s >> >> Network devices using DPDK-compatible driver >> ============================================ >> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' >> drv=vfio-pci unused=ixgbevf >> >> Network devices using kernel driver >> =================================== >> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci >> unused=vfio-pci *Active* >> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci >> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci >> >> Other network devices >> ===================== >> <none> >> >> Crypto devices using DPDK-compatible driver >> =========================================== >> <none> >> >> Crypto devices using kernel driver >> ================================== >> <none> >> >> Other crypto devices >> ==================== >> <none> >> >> >> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet >> Controller Virtual Function (rev 01) >> Subsystem: Intel Corporation 82599 Ethernet Controller >> Virtual Function >> Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- >> VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- >> Region 0: Memory at fda00000 (64-bit, prefetchable) >> [size=16K] >> Region 3: Memory at fda04000 (64-bit, prefetchable) >> [size=16K] >> Capabilities: [70] MSI-X: Enable+ Count=3 Masked- >> Vector table: BAR=3 offset=00000000 >> PBA: BAR=3 offset=00002000 >> Capabilities: [a0] Express (v1) Root Complex >> Integrated Endpoint, MSI 00 >> DevCap: MaxPayload 128 bytes, PhantFunc 0 >> ExtTag- RBE- >> DevCtl: Report errors: Correctable- Non-Fatal- >> Fatal- Unsupported- >> RlxdOrd- ExtTag- PhantFunc- AuxPwr- >> NoSnoop- >> MaxPayload 128 bytes, MaxReadReq 128 >> bytes >> DevSta: CorrErr- UncorrErr- FatalErr- >> UnsuppReq- AuxPwr- TransPend- >> Capabilities: [100 v1] Advanced Error Reporting >> UESta: DLP- SDES- TLP- FCP- CmpltTO- >> CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- >> CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> UESvrt: DLP- SDES- TLP- FCP- CmpltTO- >> CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >> CESta: RxErr- BadTLP- BadDLLP- Rollover- >> Timeout- NonFatalErr- >> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- >> Timeout- NonFatalErr- >> AERCap: First Error Pointer: 00, GenCap- >> CGenEn- ChkCap- ChkEn- >> Kernel driver in use: vfio-pci >> Kernel modules: ixgbevf >> >> /***************on Host*************/ >> dmesg | grep DMAR >> ... >> [ 978.268143] DMAR: DRHD: handling fault status reg 2 >> [ 978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault >> addr 33a128000 [fault reason 06] PTE Read access is not set >> [ 1286.677726] DMAR: DRHD: handling fault status reg 102 >> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault >> addr fb663000 [fault reason 06] PTE Read access is not set >> [ 1676.436145] DMAR: DRHD: handling fault status reg 202 >> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault >> addr 33a128000 [fault reason 06] PTE Read access is not set >> [ 1734.433649] DMAR: DRHD: handling fault status reg 302 >> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault >> addr 33a128000 [fault reason 06] PTE Read access is not set >> [ 2324.428938] DMAR: DRHD: handling fault status reg 402 >> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault >> addr 7770c000 [fault reason 06] PTE Read access is not set >> [ 2388.553640] DMAR: DRHD: handling fault status reg 502 >> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault >> addr 33a128000 [fault reason 06] PTE Read access is not set >> >> >> >> Going back to this, i would like to suggest run a few tests to >> ensure that we have all information that we can gather. >> >> First of all, i'm assuming that you're using native ixgbe Linux >> driver on the host, and that you're only passing through the VF >> device to the VM using VFIO. Is my understanding correct here? >> >> Now, let's forget about the iommu=pt and igb_uio for a moment. Boot >> both your host and your VM with iommu=on and intel_iommu=on (or >> whatever command-line enables full IOMMU support on both host and >> guest) and do the same tests you've done before. Do you still see >> your issues? >> >> It would also be very useful to also try native Linux kernel driver >> on the guest *with traffic forwarding* and see how it works in your >> VM. Therefore i would suggest you to compile DPDK with PCAP support, >> bind your (VM) interface to native Linux driver, and use the >> interface via our pcap driver (creating a vdev should do the trick - >> please refer to PCAP PMD documentation [1]). Simple forwarding test >> should be enough - just make sure to pass traffic to and from DPDK >> in both cases, and that it doesn't give you any DMAR errors. >> >> We can go from there. >> >> >> Let me just give you what has been tested and working/nonworking >> scenarios. Some of your questions might get answered as well. Test bed is >> very simple with 2 VF's created under IXGBE PF on host with one VF >> interface added to ovs-bridge on host and another VF interface given to >> guest. Test connectivity between VF's via ping. >> >> Host and guest -- Kernel 4.9 >> Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the git >> (2.11.50)) >> DPDK -- 17.05.1 on host and guest >> Host and guest -- booted with GRUB intel_iommu=on (which enables IOMMU). >> Have tried with "iommu=on and intel_iommu=on" as well, but iommu=on is not >> needed when intel_iommu=on is set. >> >> Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver ping works >> Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf driver ping >> works >> Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK vfio-pci >> driver, DMAR errors seen on host, ping doesn't work >> > > OK, that makes it clearer, thanks. Does the third scenario work in other > DPDK versions? No. Tried 16.11 same issue on guest and works fine on host. > > > >> DPDK works fine on host with vfio-pci, however, has issues when used >> inside the guest. Please let me know if more information is needed. >> >> Thanks, >> Ravi >> >> [1] http://dpdk.org/doc/guides/nics/pcap_ring.html >> <http://dpdk.org/doc/guides/nics/pcap_ring.html> >> >> -- Thanks, >> Anatoly >> >> >> > > -- > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-12 22:00 ` Ravi Kerur @ 2018-02-13 14:31 ` Burakov, Anatoly 2018-02-14 20:00 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-13 14:31 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 12-Feb-18 10:00 PM, Ravi Kerur wrote: > > Let me just give you what has been tested and working/nonworking > scenarios. Some of your questions might get answered as well. > Test bed is very simple with 2 VF's created under IXGBE PF on > host with one VF interface added to ovs-bridge on host and > another VF interface given to guest. Test connectivity between > VF's via ping. > > Host and guest -- Kernel 4.9 > Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the > git (2.11.50)) > DPDK -- 17.05.1 on host and guest > Host and guest -- booted with GRUB intel_iommu=on (which enables > IOMMU). Have tried with "iommu=on and intel_iommu=on" as well, > but iommu=on is not needed when intel_iommu=on is set. > > Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver > ping works > Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf > driver ping works > Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK > vfio-pci driver, DMAR errors seen on host, ping doesn't work > > > OK, that makes it clearer, thanks. Does the third scenario work in > other DPDK versions? > > > No. Tried 16.11 same issue on guest and works fine on host. > > So now we've moved from "this worked on 16.11" to "this never worked". It would be good to see output of rte_dump_physmem_layout() on both host and guest, and check which address triggers the DMAR error (i.e. if the physical address is present in mappings for either DPDK process). -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-13 14:31 ` Burakov, Anatoly @ 2018-02-14 20:00 ` Ravi Kerur 2018-02-15 10:28 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-14 20:00 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Tue, Feb 13, 2018 at 6:31 AM, Burakov, Anatoly <anatoly.burakov@intel.com > wrote: > On 12-Feb-18 10:00 PM, Ravi Kerur wrote: > >> >> Let me just give you what has been tested and working/nonworking >> scenarios. Some of your questions might get answered as well. >> Test bed is very simple with 2 VF's created under IXGBE PF on >> host with one VF interface added to ovs-bridge on host and >> another VF interface given to guest. Test connectivity between >> VF's via ping. >> >> Host and guest -- Kernel 4.9 >> Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the >> git (2.11.50)) >> DPDK -- 17.05.1 on host and guest >> Host and guest -- booted with GRUB intel_iommu=on (which enables >> IOMMU). Have tried with "iommu=on and intel_iommu=on" as well, >> but iommu=on is not needed when intel_iommu=on is set. >> >> Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver >> ping works >> Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf >> driver ping works >> Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK >> vfio-pci driver, DMAR errors seen on host, ping doesn't work >> >> >> OK, that makes it clearer, thanks. Does the third scenario work in >> other DPDK versions? >> >> >> No. Tried 16.11 same issue on guest and works fine on host. >> >> >> So now we've moved from "this worked on 16.11" to "this never worked". > > It would be good to see output of rte_dump_physmem_layout() on both host > and guest, and check which address triggers the DMAR error (i.e. if the > physical address is present in mappings for either DPDK process). > > -- > Earlier I was focusing only on DMAR errors and I might have said 'it worked' when I didn't notice them on host when dpdk was started on guest. When trying to send packets out of that interface from guest I did see DMAR errors. I am attaching information you requested. I have enabled log-level=8 and files contain dpdk EAL/PMD logs as well. Snippets below on host, DMAR fault address from dmesg [351576.998109] DMAR: DRHD: handling fault status reg 702 [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr 257617000 [fault reason 06] PTE Read access is not set on guest (dump phys_mem_layout) Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 ... PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600 PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80 ... Thanks, Ravi > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-14 20:00 ` Ravi Kerur @ 2018-02-15 10:28 ` Burakov, Anatoly 2018-02-15 18:27 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-15 10:28 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev On 14-Feb-18 8:00 PM, Ravi Kerur wrote: > > Earlier I was focusing only on DMAR errors and I might have said 'it > worked' when I didn't notice them on host when dpdk was started on > guest. When trying to send packets out of that interface from guest I > did see DMAR errors. I am attaching information you requested. I have > enabled log-level=8 and files contain dpdk EAL/PMD logs as well. Great, now we're on the same page. > > Snippets below > > on host, DMAR fault address from dmesg > > [351576.998109] DMAR: DRHD: handling fault status reg 702 > [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr > 257617000 [fault reason 06] PTE Read access is not set > > on guest (dump phys_mem_layout) > > Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000, > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 > ... > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 > sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600 > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 > sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80 > ... > To me this looks like host (i.e. either QEMU or the PF driver) is trying to do DMA using guest-physical (and not host-physical). I'm not too well-versed in how QEMU works, but i'm pretty sure that's not supposed to happen. Is PF also bound to DPDK, or are you using native Linux ixgbe driver? -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-15 10:28 ` Burakov, Anatoly @ 2018-02-15 18:27 ` Ravi Kerur 2018-02-15 20:53 ` Ravi Kerur 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-15 18:27 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly <anatoly.burakov@intel.com > wrote: > On 14-Feb-18 8:00 PM, Ravi Kerur wrote: > >> >> Earlier I was focusing only on DMAR errors and I might have said 'it >> worked' when I didn't notice them on host when dpdk was started on guest. >> When trying to send packets out of that interface from guest I did see DMAR >> errors. I am attaching information you requested. I have enabled >> log-level=8 and files contain dpdk EAL/PMD logs as well. >> > > Great, now we're on the same page. > > >> Snippets below >> >> on host, DMAR fault address from dmesg >> >> [351576.998109] DMAR: DRHD: handling fault status reg 702 >> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 257617000 [fault reason 06] PTE Read access is not set >> >> on guest (dump phys_mem_layout) >> >> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000, >> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 >> ... >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 >> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600 >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 >> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80 >> ... >> >> > To me this looks like host (i.e. either QEMU or the PF driver) is trying > to do DMA using guest-physical (and not host-physical). I'm not too > well-versed in how QEMU works, but i'm pretty sure that's not supposed to > happen. > > Is PF also bound to DPDK, or are you using native Linux ixgbe driver? > Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF interfaces disappear after it is bound to DPDK. If there is a way to use PF and VF with DPDK let me know I can try it out. I am not sure how to move forward on this, Is CPU/IXGBE PF driver playing a role? Following are the versions I have lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 56 On-line CPU(s) list: 0-27 Off-line CPU(s) list: 28-55 Thread(s) per core: 1 Core(s) per socket: 14 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz Stepping: 2 CPU MHz: 2500.610 CPU max MHz: 3000.0000 CPU min MHz: 1200.0000 BogoMIPS: 4000.74 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 35840K NUMA node0 CPU(s): 0-13 NUMA node1 CPU(s): 14-27 # ethtool -i enp4s0f0 driver: ixgbe version: 5.3.3 firmware-version: 0x800007b8, 1.1018.0 bus-info: 0000:04:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes Thanks, Ravi > > -- > Thanks, > Anatoly > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-15 18:27 ` Ravi Kerur @ 2018-02-15 20:53 ` Ravi Kerur 2018-02-16 9:41 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Ravi Kerur @ 2018-02-15 20:53 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: dev On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com> wrote: > > > On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly < > anatoly.burakov@intel.com> wrote: > >> On 14-Feb-18 8:00 PM, Ravi Kerur wrote: >> >>> >>> Earlier I was focusing only on DMAR errors and I might have said 'it >>> worked' when I didn't notice them on host when dpdk was started on guest. >>> When trying to send packets out of that interface from guest I did see DMAR >>> errors. I am attaching information you requested. I have enabled >>> log-level=8 and files contain dpdk EAL/PMD logs as well. >>> >> >> Great, now we're on the same page. >> >> >>> Snippets below >>> >>> on host, DMAR fault address from dmesg >>> >>> [351576.998109] DMAR: DRHD: handling fault status reg 702 >>> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr >>> 257617000 [fault reason 06] PTE Read access is not set >>> >>> on guest (dump phys_mem_layout) >>> >>> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000, >>> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 >>> ... >>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 >>> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600 >>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 >>> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80 >>> ... >>> >>> >> To me this looks like host (i.e. either QEMU or the PF driver) is trying >> to do DMA using guest-physical (and not host-physical). I'm not too >> well-versed in how QEMU works, but i'm pretty sure that's not supposed to >> happen. >> >> Is PF also bound to DPDK, or are you using native Linux ixgbe driver? >> > > Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF interfaces > disappear after it is bound to DPDK. If there is a way to use PF and VF > with DPDK let me know I can try it out. I am not sure how to move forward > on this, Is CPU/IXGBE PF driver playing a role? Following are the versions > I have > > lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 56 > On-line CPU(s) list: 0-27 > Off-line CPU(s) list: 28-55 > Thread(s) per core: 1 > Core(s) per socket: 14 > Socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 63 > Model name: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz > Stepping: 2 > CPU MHz: 2500.610 > CPU max MHz: 3000.0000 > CPU min MHz: 1200.0000 > BogoMIPS: 4000.74 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 35840K > NUMA node0 CPU(s): 0-13 > NUMA node1 CPU(s): 14-27 > > # ethtool -i enp4s0f0 > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > Thanks, > Ravi > > > Debugging this I could co-relate doing interface link-up associated with the dpdk inside the guest causes DMAR errors on host and an additional vflr message. [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [ 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074] ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [ 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0 Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it possible that dpdk pmd vf driver is missing some fixes/porting from ixgbevf driver since this issue is not seen when ixgbevf kernel driver is used? Thanks, Ravi >> -- >> Thanks, >> Anatoly >> > > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-15 20:53 ` Ravi Kerur @ 2018-02-16 9:41 ` Burakov, Anatoly 2019-01-15 7:07 ` Hu, Xuekun 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2018-02-16 9:41 UTC (permalink / raw) To: Ravi Kerur; +Cc: dev, Ananyev, Konstantin, wenzhuo.lu On 15-Feb-18 8:53 PM, Ravi Kerur wrote: > > > On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com > <mailto:rkerur@gmail.com>> wrote: > > > > On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: > > On 14-Feb-18 8:00 PM, Ravi Kerur wrote: > > > Earlier I was focusing only on DMAR errors and I might have > said 'it worked' when I didn't notice them on host when dpdk > was started on guest. When trying to send packets out of > that interface from guest I did see DMAR errors. I am > attaching information you requested. I have enabled > log-level=8 and files contain dpdk EAL/PMD logs as well. > > > Great, now we're on the same page. > > > Snippets below > > on host, DMAR fault address from dmesg > > [351576.998109] DMAR: DRHD: handling fault status reg 702 > [351576.998113] DMAR: [DMA Read] Request device [04:10.0] > fault addr 257617000 [fault reason 06] PTE Read access is > not set > > on guest (dump phys_mem_layout) > > Segment 235: phys:0x257600000, len:2097152, > virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152, > nchannel:0, nrank:0 > ... > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 > sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 > dma_addr=0x257617600 > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 > sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 > dma_addr=0x25406fe80 > ... > > > To me this looks like host (i.e. either QEMU or the PF driver) > is trying to do DMA using guest-physical (and not > host-physical). I'm not too well-versed in how QEMU works, but > i'm pretty sure that's not supposed to happen. > > Is PF also bound to DPDK, or are you using native Linux ixgbe > driver? > > > Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF > interfaces disappear after it is bound to DPDK. If there is a way to > use PF and VF with DPDK let me know I can try it out. I am not sure > how to move forward on this, Is CPU/IXGBE PF driver playing a role? > Following are the versions I have > > lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 56 > On-line CPU(s) list: 0-27 > Off-line CPU(s) list: 28-55 > Thread(s) per core: 1 > Core(s) per socket: 14 > Socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 63 > Model name: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz > Stepping: 2 > CPU MHz: 2500.610 > CPU max MHz: 3000.0000 > CPU min MHz: 1200.0000 > BogoMIPS: 4000.74 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 35840K > NUMA node0 CPU(s): 0-13 > NUMA node1 CPU(s): 14-27 > > # ethtool -i enp4s0f0 > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > Thanks, > Ravi > > > > Debugging this I could co-relate doing interface link-up associated with > the dpdk inside the guest causes DMAR errors on host and an additional > vflr message. > > [ 8135.861622] DMAR: DRHD: handling fault status reg 402 > [ 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr > 1b648a000 [fault reason 06] PTE Read access is not set > [ 8136.588074] ixgbe 0000:04:00.0: Issuing VFLR with pending transactions > [ 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0 > > Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from > 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it > possible that dpdk pmd vf driver is missing some fixes/porting from > ixgbevf driver since this issue is not seen when ixgbevf kernel driver > is used? > Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2018-02-16 9:41 ` Burakov, Anatoly @ 2019-01-15 7:07 ` Hu, Xuekun 2019-01-15 11:22 ` Burakov, Anatoly 0 siblings, 1 reply; 33+ messages in thread From: Hu, Xuekun @ 2019-01-15 7:07 UTC (permalink / raw) To: Burakov, Anatoly, Ravi Kerur; +Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo Hi, Ravi Did you resolve this issue that VF used in guest with vIOMMU enabled? I googled, but still can't get the answer that it is driver or qemu vt-d emulation issue. Currently I met the same issue again that host reported DMAR error: [59939.130110] DMAR: DRHD: handling fault status reg 2 [59939.130116] DMAR: [DMA Read] Request device [83:10.0] fault addr 15f03d000 [fault reason 06] PTE Read access is not set [59940.180859] ixgbe 0000:83:00.0: Issuing VFLR with pending transactions [59940.180863] ixgbe 0000:83:00.0: Issuing VFLR for VF 0000:83:10.0 [59989.344683] ixgbe 0000:83:00.0 ens817f0: VF Reset msg received from vf 0 I'm using DPDK 18.11 in guest, and ixgbe.ko in host. Thx, Xuekun -----Original Message----- From: dev <dev-bounces@dpdk.org> On Behalf Of Burakov, Anatoly Sent: Friday, February 16, 2018 5:42 PM To: Ravi Kerur <rkerur@gmail.com> Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com> Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue On 15-Feb-18 8:53 PM, Ravi Kerur wrote: > > > On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com > <mailto:rkerur@gmail.com>> wrote: > > > > On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: > > On 14-Feb-18 8:00 PM, Ravi Kerur wrote: > > > Earlier I was focusing only on DMAR errors and I might have > said 'it worked' when I didn't notice them on host when dpdk > was started on guest. When trying to send packets out of > that interface from guest I did see DMAR errors. I am > attaching information you requested. I have enabled > log-level=8 and files contain dpdk EAL/PMD logs as well. > > > Great, now we're on the same page. > > > Snippets below > > on host, DMAR fault address from dmesg > > [351576.998109] DMAR: DRHD: handling fault status reg 702 > [351576.998113] DMAR: [DMA Read] Request device [04:10.0] > fault addr 257617000 [fault reason 06] PTE Read access is > not set > > on guest (dump phys_mem_layout) > > Segment 235: phys:0x257600000, len:2097152, > virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152, > nchannel:0, nrank:0 > ... > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 > sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 > dma_addr=0x257617600 > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 > sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 > dma_addr=0x25406fe80 > ... > > > To me this looks like host (i.e. either QEMU or the PF driver) > is trying to do DMA using guest-physical (and not > host-physical). I'm not too well-versed in how QEMU works, but > i'm pretty sure that's not supposed to happen. > > Is PF also bound to DPDK, or are you using native Linux ixgbe > driver? > > > Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF > interfaces disappear after it is bound to DPDK. If there is a way to > use PF and VF with DPDK let me know I can try it out. I am not sure > how to move forward on this, Is CPU/IXGBE PF driver playing a role? > Following are the versions I have > > lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 56 > On-line CPU(s) list: 0-27 > Off-line CPU(s) list: 28-55 > Thread(s) per core: 1 > Core(s) per socket: 14 > Socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 63 > Model name: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz > Stepping: 2 > CPU MHz: 2500.610 > CPU max MHz: 3000.0000 > CPU min MHz: 1200.0000 > BogoMIPS: 4000.74 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 35840K > NUMA node0 CPU(s): 0-13 > NUMA node1 CPU(s): 14-27 > > # ethtool -i enp4s0f0 > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > Thanks, > Ravi > > > > Debugging this I could co-relate doing interface link-up associated > with the dpdk inside the guest causes DMAR errors on host and an > additional vflr message. > > [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [ > 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr > 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074] > ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [ > 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0 > > Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from > 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it > possible that dpdk pmd vf driver is missing some fixes/porting from > ixgbevf driver since this issue is not seen when ixgbevf kernel driver > is used? > Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2019-01-15 7:07 ` Hu, Xuekun @ 2019-01-15 11:22 ` Burakov, Anatoly 2019-01-15 13:07 ` Hu, Xuekun 2019-01-21 13:18 ` Hu, Xuekun 0 siblings, 2 replies; 33+ messages in thread From: Burakov, Anatoly @ 2019-01-15 11:22 UTC (permalink / raw) To: Hu, Xuekun, Ravi Kerur; +Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo On 15-Jan-19 7:07 AM, Hu, Xuekun wrote: > Hi, Ravi > > Did you resolve this issue that VF used in guest with vIOMMU enabled? I googled, but still can't get the answer that it is driver or qemu vt-d emulation issue. > > Currently I met the same issue again that host reported DMAR error: > [59939.130110] DMAR: DRHD: handling fault status reg 2 > [59939.130116] DMAR: [DMA Read] Request device [83:10.0] fault addr 15f03d000 [fault reason 06] PTE Read access is not set > [59940.180859] ixgbe 0000:83:00.0: Issuing VFLR with pending transactions > [59940.180863] ixgbe 0000:83:00.0: Issuing VFLR for VF 0000:83:10.0 > [59989.344683] ixgbe 0000:83:00.0 ens817f0: VF Reset msg received from vf 0 > > I'm using DPDK 18.11 in guest, and ixgbe.ko in host. > > Thx, Xuekun > > -----Original Message----- > From: dev <dev-bounces@dpdk.org> On Behalf Of Burakov, Anatoly > Sent: Friday, February 16, 2018 5:42 PM > To: Ravi Kerur <rkerur@gmail.com> > Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com> > Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue > > On 15-Feb-18 8:53 PM, Ravi Kerur wrote: >> >> >> On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com >> <mailto:rkerur@gmail.com>> wrote: >> >> >> >> On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly >> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: >> >> On 14-Feb-18 8:00 PM, Ravi Kerur wrote: >> >> >> Earlier I was focusing only on DMAR errors and I might have >> said 'it worked' when I didn't notice them on host when dpdk >> was started on guest. When trying to send packets out of >> that interface from guest I did see DMAR errors. I am >> attaching information you requested. I have enabled >> log-level=8 and files contain dpdk EAL/PMD logs as well. >> >> >> Great, now we're on the same page. >> >> >> Snippets below >> >> on host, DMAR fault address from dmesg >> >> [351576.998109] DMAR: DRHD: handling fault status reg 702 >> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] >> fault addr 257617000 [fault reason 06] PTE Read access is >> not set >> >> on guest (dump phys_mem_layout) >> >> Segment 235: phys:0x257600000, len:2097152, >> virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152, >> nchannel:0, nrank:0 >> ... >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 >> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 >> dma_addr=0x257617600 >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 >> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 >> dma_addr=0x25406fe80 >> ... >> >> >> To me this looks like host (i.e. either QEMU or the PF driver) >> is trying to do DMA using guest-physical (and not >> host-physical). I'm not too well-versed in how QEMU works, but >> i'm pretty sure that's not supposed to happen. >> >> Is PF also bound to DPDK, or are you using native Linux ixgbe >> driver? >> >> >> Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF >> interfaces disappear after it is bound to DPDK. If there is a way to >> use PF and VF with DPDK let me know I can try it out. I am not sure >> how to move forward on this, Is CPU/IXGBE PF driver playing a role? >> Following are the versions I have >> >> lscpu >> Architecture: x86_64 >> CPU op-mode(s): 32-bit, 64-bit >> Byte Order: Little Endian >> CPU(s): 56 >> On-line CPU(s) list: 0-27 >> Off-line CPU(s) list: 28-55 >> Thread(s) per core: 1 >> Core(s) per socket: 14 >> Socket(s): 2 >> NUMA node(s): 2 >> Vendor ID: GenuineIntel >> CPU family: 6 >> Model: 63 >> Model name: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz >> Stepping: 2 >> CPU MHz: 2500.610 >> CPU max MHz: 3000.0000 >> CPU min MHz: 1200.0000 >> BogoMIPS: 4000.74 >> Virtualization: VT-x >> L1d cache: 32K >> L1i cache: 32K >> L2 cache: 256K >> L3 cache: 35840K >> NUMA node0 CPU(s): 0-13 >> NUMA node1 CPU(s): 14-27 >> >> # ethtool -i enp4s0f0 >> driver: ixgbe >> version: 5.3.3 >> firmware-version: 0x800007b8, 1.1018.0 >> bus-info: 0000:04:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: yes >> >> Thanks, >> Ravi >> >> >> >> Debugging this I could co-relate doing interface link-up associated >> with the dpdk inside the guest causes DMAR errors on host and an >> additional vflr message. >> >> [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [ >> 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074] >> ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [ >> 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0 >> >> Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from >> 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it >> possible that dpdk pmd vf driver is missing some fixes/porting from >> ixgbevf driver since this issue is not seen when ixgbevf kernel driver >> is used? >> > > Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue. > > -- > Thanks, > Anatoly > You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76 There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2019-01-15 11:22 ` Burakov, Anatoly @ 2019-01-15 13:07 ` Hu, Xuekun 2019-01-21 13:18 ` Hu, Xuekun 1 sibling, 0 replies; 33+ messages in thread From: Hu, Xuekun @ 2019-01-15 13:07 UTC (permalink / raw) To: Burakov, Anatoly, Ravi Kerur; +Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo Thanks, I'm just using CentOS7.5 3.10! I will try a newer kernel. Stay tuned. -----Original Message----- From: Burakov, Anatoly Sent: Tuesday, January 15, 2019 7:22 PM To: Hu, Xuekun <xuekun.hu@intel.com>; Ravi Kerur <rkerur@gmail.com> Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com> Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue On 15-Jan-19 7:07 AM, Hu, Xuekun wrote: > Hi, Ravi > > Did you resolve this issue that VF used in guest with vIOMMU enabled? I googled, but still can't get the answer that it is driver or qemu vt-d emulation issue. > > Currently I met the same issue again that host reported DMAR error: > [59939.130110] DMAR: DRHD: handling fault status reg 2 [59939.130116] > DMAR: [DMA Read] Request device [83:10.0] fault addr 15f03d000 [fault > reason 06] PTE Read access is not set [59940.180859] ixgbe > 0000:83:00.0: Issuing VFLR with pending transactions [59940.180863] > ixgbe 0000:83:00.0: Issuing VFLR for VF 0000:83:10.0 [59989.344683] > ixgbe 0000:83:00.0 ens817f0: VF Reset msg received from vf 0 > > I'm using DPDK 18.11 in guest, and ixgbe.ko in host. > > Thx, Xuekun > > -----Original Message----- > From: dev <dev-bounces@dpdk.org> On Behalf Of Burakov, Anatoly > Sent: Friday, February 16, 2018 5:42 PM > To: Ravi Kerur <rkerur@gmail.com> > Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; > Lu, Wenzhuo <wenzhuo.lu@intel.com> > Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue > > On 15-Feb-18 8:53 PM, Ravi Kerur wrote: >> >> >> On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com >> <mailto:rkerur@gmail.com>> wrote: >> >> >> >> On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly >> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote: >> >> On 14-Feb-18 8:00 PM, Ravi Kerur wrote: >> >> >> Earlier I was focusing only on DMAR errors and I might have >> said 'it worked' when I didn't notice them on host when dpdk >> was started on guest. When trying to send packets out of >> that interface from guest I did see DMAR errors. I am >> attaching information you requested. I have enabled >> log-level=8 and files contain dpdk EAL/PMD logs as well. >> >> >> Great, now we're on the same page. >> >> >> Snippets below >> >> on host, DMAR fault address from dmesg >> >> [351576.998109] DMAR: DRHD: handling fault status reg 702 >> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] >> fault addr 257617000 [fault reason 06] PTE Read access is >> not set >> >> on guest (dump phys_mem_layout) >> >> Segment 235: phys:0x257600000, len:2097152, >> virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152, >> nchannel:0, nrank:0 >> ... >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 >> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 >> dma_addr=0x257617600 >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 >> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 >> dma_addr=0x25406fe80 >> ... >> >> >> To me this looks like host (i.e. either QEMU or the PF driver) >> is trying to do DMA using guest-physical (and not >> host-physical). I'm not too well-versed in how QEMU works, but >> i'm pretty sure that's not supposed to happen. >> >> Is PF also bound to DPDK, or are you using native Linux ixgbe >> driver? >> >> >> Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF >> interfaces disappear after it is bound to DPDK. If there is a way to >> use PF and VF with DPDK let me know I can try it out. I am not sure >> how to move forward on this, Is CPU/IXGBE PF driver playing a role? >> Following are the versions I have >> >> lscpu >> Architecture: x86_64 >> CPU op-mode(s): 32-bit, 64-bit >> Byte Order: Little Endian >> CPU(s): 56 >> On-line CPU(s) list: 0-27 >> Off-line CPU(s) list: 28-55 >> Thread(s) per core: 1 >> Core(s) per socket: 14 >> Socket(s): 2 >> NUMA node(s): 2 >> Vendor ID: GenuineIntel >> CPU family: 6 >> Model: 63 >> Model name: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz >> Stepping: 2 >> CPU MHz: 2500.610 >> CPU max MHz: 3000.0000 >> CPU min MHz: 1200.0000 >> BogoMIPS: 4000.74 >> Virtualization: VT-x >> L1d cache: 32K >> L1i cache: 32K >> L2 cache: 256K >> L3 cache: 35840K >> NUMA node0 CPU(s): 0-13 >> NUMA node1 CPU(s): 14-27 >> >> # ethtool -i enp4s0f0 >> driver: ixgbe >> version: 5.3.3 >> firmware-version: 0x800007b8, 1.1018.0 >> bus-info: 0000:04:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: yes >> >> Thanks, >> Ravi >> >> >> >> Debugging this I could co-relate doing interface link-up associated >> with the dpdk inside the guest causes DMAR errors on host and an >> additional vflr message. >> >> [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [ >> 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074] >> ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [ >> 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0 >> >> Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from >> 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is >> it possible that dpdk pmd vf driver is missing some fixes/porting >> from ixgbevf driver since this issue is not seen when ixgbevf kernel >> driver is used? >> > > Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue. > > -- > Thanks, > Anatoly > You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76 There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2019-01-15 11:22 ` Burakov, Anatoly 2019-01-15 13:07 ` Hu, Xuekun @ 2019-01-21 13:18 ` Hu, Xuekun 2019-01-21 13:39 ` Burakov, Anatoly 1 sibling, 1 reply; 33+ messages in thread From: Hu, Xuekun @ 2019-01-21 13:18 UTC (permalink / raw) To: Burakov, Anatoly, Ravi Kerur; +Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo > You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76 > There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM. Thanks, Anatoly. By updating host kernel to 4.18, the issue is gone. 😊 ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2019-01-21 13:18 ` Hu, Xuekun @ 2019-01-21 13:39 ` Burakov, Anatoly 2019-01-21 14:44 ` Thomas Monjalon 0 siblings, 1 reply; 33+ messages in thread From: Burakov, Anatoly @ 2019-01-21 13:39 UTC (permalink / raw) To: Hu, Xuekun, Ravi Kerur, Thomas Monjalon Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo On 21-Jan-19 1:18 PM, Hu, Xuekun wrote: > >> You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76 > >> There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM. > > Thanks, Anatoly. By updating host kernel to 4.18, the issue is gone. 😊 > @Thomas, does this look like something we should add to known issues? This question has been asked a few times already. -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue 2019-01-21 13:39 ` Burakov, Anatoly @ 2019-01-21 14:44 ` Thomas Monjalon 0 siblings, 0 replies; 33+ messages in thread From: Thomas Monjalon @ 2019-01-21 14:44 UTC (permalink / raw) To: Burakov, Anatoly Cc: Hu, Xuekun, Ravi Kerur, dev, Ananyev, Konstantin, Lu, Wenzhuo 21/01/2019 14:39, Burakov, Anatoly: > On 21-Jan-19 1:18 PM, Hu, Xuekun wrote: > > > >> You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76 > > > >> There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM. > > > > Thanks, Anatoly. By updating host kernel to 4.18, the issue is gone. 😊 > > > > @Thomas, does this look like something we should add to known issues? > This question has been asked a few times already. Yes sure ^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2019-01-21 14:45 UTC | newest] Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-01-23 17:25 [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue Ravi Kerur 2018-01-24 10:31 ` Burakov, Anatoly 2018-01-24 19:13 ` Ravi Kerur 2018-01-25 10:49 ` Burakov, Anatoly 2018-01-29 22:35 ` Ravi Kerur 2018-01-31 9:59 ` Burakov, Anatoly 2018-01-31 21:51 ` Ravi Kerur 2018-02-01 10:10 ` Burakov, Anatoly 2018-02-01 19:26 ` Ravi Kerur 2018-02-02 10:28 ` Burakov, Anatoly 2018-02-02 20:21 ` Ravi Kerur 2018-02-02 20:51 ` Ravi Kerur 2018-02-05 10:01 ` Burakov, Anatoly 2018-02-06 17:55 ` Ravi Kerur 2018-02-08 11:20 ` Burakov, Anatoly 2018-02-09 17:41 ` Ravi Kerur 2018-02-10 10:11 ` Burakov, Anatoly 2018-02-10 10:58 ` Burakov, Anatoly 2018-02-10 17:53 ` Ravi Kerur 2018-02-12 10:13 ` Burakov, Anatoly 2018-02-12 22:00 ` Ravi Kerur 2018-02-13 14:31 ` Burakov, Anatoly 2018-02-14 20:00 ` Ravi Kerur 2018-02-15 10:28 ` Burakov, Anatoly 2018-02-15 18:27 ` Ravi Kerur 2018-02-15 20:53 ` Ravi Kerur 2018-02-16 9:41 ` Burakov, Anatoly 2019-01-15 7:07 ` Hu, Xuekun 2019-01-15 11:22 ` Burakov, Anatoly 2019-01-15 13:07 ` Hu, Xuekun 2019-01-21 13:18 ` Hu, Xuekun 2019-01-21 13:39 ` Burakov, Anatoly 2019-01-21 14:44 ` Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).