DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
@ 2018-01-23 17:25 Ravi Kerur
  2018-01-24 10:31 ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-01-23 17:25 UTC (permalink / raw)
  To: dev, users

Hi,

I am running into an issue when DPDK is started with iommu on via GRUB
command. Problem is not seen with regular kernel driver, error messages
show when DPDK is started and happens for both PF and VF interfaces.

I am using DPDK 17.05 so the patch proposed in the following link is
available
http://dpdk.org/ml/archives/dev/2017-February/057048.html

Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I
checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any details
on it.

Kindly let me know how to resolve this issue.

Following are the details

(1) Linux kernel 4.9
(2) DPDK 17.05

(3) IXGBE details
ethtool -i enp4s0f0  (PF driver)
driver: ixgbe
version: 5.3.3
firmware-version: 0x800007b8, 1.1018.0
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

ethtool -i enp4s16f2 (VF driver)
driver: ixgbevf
version: 4.3.2
firmware-version:
bus-info: 0000:04:10.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

Bus info          Device       Class          Description
=========================================================
pci@0000:01:00.0  ens11f0      network        82599ES 10-Gigabit SFI/SFP+
Network Connection
pci@0000:01:00.1  ens11f1      network        82599ES 10-Gigabit SFI/SFP+
Network Connection
pci@0000:04:00.0  enp4s0f0     network        82599ES 10-Gigabit SFI/SFP+
Network Connection
pci@0000:04:00.1  enp4s0f1     network        82599ES 10-Gigabit SFI/SFP+
Network Connection
pci@0000:04:10.0  enp4s16      network        Illegal Vendor ID
pci@0000:04:10.2  enp4s16f2    network        Illegal Vendor ID

(4) DPDK bind interfaces

# dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
drv=igb_uio unused=vfio-pci
0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio
unused=vfio-pci

Network devices using kernel driver
===================================
0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16
drv=ixgbevf unused=igb_uio,vfio-pci
0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb
unused=igb_uio,vfio-pci *Active*

Other Network devices
=====================
<none>

...

(5) Kernel dmesg

# dmesg | grep -e DMAR
[    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
00000001 INTL 20091013)
[    0.000000] DMAR: IOMMU enabled
[    0.518747] DMAR: Host address width 46
[    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
d2078c106f0466 ecap f020df
[    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
d2078c106f0466 ecap f020df
[    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff
[    0.593344] DMAR: ATSR flags: 0x0
[    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0
[    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1
[    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
[    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 IOMMU 1
[    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 IOMMU 1
[    0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    0.675326] DMAR-IR: Queued invalidation will be enabled to support
x2apic and Intr-remapping.
[    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    9.395170] DMAR: dmar1: Using Queued invalidation
[    9.405011] DMAR: Setting RMRR:
[    9.412006] DMAR: Setting identity map for device 0000:00:1d.0
[0x7bbc6000 - 0x7bbd4fff]
[    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
[    9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
0xffffff]
[    9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O
[  287.023068] DMAR: DRHD: handling fault status reg 2
[  287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr
18a260a000 [fault reason 06] PTE Read access is not set
[  287.023180] DMAR: DRHD: handling fault status reg 102
[  287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr
18a3010000 [fault reason 06] PTE Read access is not set
[  287.038250] DMAR: DRHD: handling fault status reg 202
[  287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr
18a3010000 [fault reason 06] PTE Read access is not set
[  288.170165] DMAR: DRHD: handling fault status reg 302
[  288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr
1890754000 [fault reason 06] PTE Read access is not set
[  288.694496] DMAR: DRHD: handling fault status reg 402
[  288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr
189069c000 [fault reason 06] PTE Read access is not set
[  289.927113] DMAR: DRHD: handling fault status reg 502
[  289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr
1890754000 [fault reason 06] PTE Read access is not set
[  290.174275] DMAR: DRHD: handling fault status reg 602
[  290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr
1890754000 [fault reason 06] PTE Read access is not set
[  292.174247] DMAR: DRHD: handling fault status reg 702
[  292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr
1890754000 [fault reason 06] PTE Read access is not set
[  294.174227] DMAR: DRHD: handling fault status reg 2
[  294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr
1890754000 [fault reason 06] PTE Read access is not set
[  296.174216] DMAR: DRHD: handling fault status reg 102
[  296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr
1890754000 [fault reason 06] PTE Read access is not set
[root@infradev-comp006.naw02.infradev.viasat.io ~]
#

Thanks.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-01-23 17:25 [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue Ravi Kerur
@ 2018-01-24 10:31 ` Burakov, Anatoly
  2018-01-24 19:13   ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-01-24 10:31 UTC (permalink / raw)
  To: dev

On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
> Hi,
> 
> I am running into an issue when DPDK is started with iommu on via GRUB
> command. Problem is not seen with regular kernel driver, error messages
> show when DPDK is started and happens for both PF and VF interfaces.
> 
> I am using DPDK 17.05 so the patch proposed in the following link is
> available
> http://dpdk.org/ml/archives/dev/2017-February/057048.html
> 
> Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I
> checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any details
> on it.
> 
> Kindly let me know how to resolve this issue.
> 
> Following are the details
> 
> (1) Linux kernel 4.9
> (2) DPDK 17.05
> 
> (3) IXGBE details
> ethtool -i enp4s0f0  (PF driver)
> driver: ixgbe
> version: 5.3.3
> firmware-version: 0x800007b8, 1.1018.0
> bus-info: 0000:04:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
> 
> ethtool -i enp4s16f2 (VF driver)
> driver: ixgbevf
> version: 4.3.2
> firmware-version:
> bus-info: 0000:04:10.2
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: no
> supports-register-dump: yes
> supports-priv-flags: no
> 
> Bus info          Device       Class          Description
> =========================================================
> pci@0000:01:00.0  ens11f0      network        82599ES 10-Gigabit SFI/SFP+
> Network Connection
> pci@0000:01:00.1  ens11f1      network        82599ES 10-Gigabit SFI/SFP+
> Network Connection
> pci@0000:04:00.0  enp4s0f0     network        82599ES 10-Gigabit SFI/SFP+
> Network Connection
> pci@0000:04:00.1  enp4s0f1     network        82599ES 10-Gigabit SFI/SFP+
> Network Connection
> pci@0000:04:10.0  enp4s16      network        Illegal Vendor ID
> pci@0000:04:10.2  enp4s16f2    network        Illegal Vendor ID
> 
> (4) DPDK bind interfaces
> 
> # dpdk-devbind -s
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> drv=igb_uio unused=vfio-pci
> 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio
> unused=vfio-pci
> 
> Network devices using kernel driver
> ===================================
> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
> 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
> 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
> 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16
> drv=ixgbevf unused=igb_uio,vfio-pci
> 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb
> unused=igb_uio,vfio-pci *Active*
> 
> Other Network devices
> =====================
> <none>
> 
> ...
> 
> (5) Kernel dmesg
> 
> # dmesg | grep -e DMAR
> [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
> 00000001 INTL 20091013)
> [    0.000000] DMAR: IOMMU enabled
> [    0.518747] DMAR: Host address width 46
> [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
> [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
> d2078c106f0466 ecap f020df
> [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
> [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
> d2078c106f0466 ecap f020df
> [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff
> [    0.593344] DMAR: ATSR flags: 0x0
> [    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0
> [    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1
> [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
> [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 IOMMU 1
> [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 IOMMU 1
> [    0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
> [    0.675326] DMAR-IR: Queued invalidation will be enabled to support
> x2apic and Intr-remapping.
> [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [    9.395170] DMAR: dmar1: Using Queued invalidation
> [    9.405011] DMAR: Setting RMRR:
> [    9.412006] DMAR: Setting identity map for device 0000:00:1d.0
> [0x7bbc6000 - 0x7bbd4fff]
> [    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
> [    9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
> 0xffffff]
> [    9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O
> [  287.023068] DMAR: DRHD: handling fault status reg 2
> [  287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 18a260a000 [fault reason 06] PTE Read access is not set
> [  287.023180] DMAR: DRHD: handling fault status reg 102
> [  287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 18a3010000 [fault reason 06] PTE Read access is not set
> [  287.038250] DMAR: DRHD: handling fault status reg 202
> [  287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 18a3010000 [fault reason 06] PTE Read access is not set
> [  288.170165] DMAR: DRHD: handling fault status reg 302
> [  288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  288.694496] DMAR: DRHD: handling fault status reg 402
> [  288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr
> 189069c000 [fault reason 06] PTE Read access is not set
> [  289.927113] DMAR: DRHD: handling fault status reg 502
> [  289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  290.174275] DMAR: DRHD: handling fault status reg 602
> [  290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  292.174247] DMAR: DRHD: handling fault status reg 702
> [  292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  294.174227] DMAR: DRHD: handling fault status reg 2
> [  294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  296.174216] DMAR: DRHD: handling fault status reg 102
> [  296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [root@infradev-comp006.naw02.infradev.viasat.io ~]
> #
> 
> Thanks.
> 

Hi Ravi,

The "iommu=pt" workaround applies only when you want to use igb_uio 
driver. VFIO driver is able to fully utilize IOMMU without the need for 
pass-through mode. From your log i can see that some devices are bound 
to igb_uio while others are bound to vfio-pci. Just bind all of the 
devices you want to use with DPDK to vfio-pci and these errors should go 
away.

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-01-24 10:31 ` Burakov, Anatoly
@ 2018-01-24 19:13   ` Ravi Kerur
  2018-01-25 10:49     ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-01-24 19:13 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it
will work for both PF and VF interfaces since I am using both in my setup?

Thanks.

On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
>
>> Hi,
>>
>> I am running into an issue when DPDK is started with iommu on via GRUB
>> command. Problem is not seen with regular kernel driver, error messages
>> show when DPDK is started and happens for both PF and VF interfaces.
>>
>> I am using DPDK 17.05 so the patch proposed in the following link is
>> available
>> http://dpdk.org/ml/archives/dev/2017-February/057048.html
>>
>> Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I
>> checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any
>> details
>> on it.
>>
>> Kindly let me know how to resolve this issue.
>>
>> Following are the details
>>
>> (1) Linux kernel 4.9
>> (2) DPDK 17.05
>>
>> (3) IXGBE details
>> ethtool -i enp4s0f0  (PF driver)
>> driver: ixgbe
>> version: 5.3.3
>> firmware-version: 0x800007b8, 1.1018.0
>> bus-info: 0000:04:00.0
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: yes
>>
>> ethtool -i enp4s16f2 (VF driver)
>> driver: ixgbevf
>> version: 4.3.2
>> firmware-version:
>> bus-info: 0000:04:10.2
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: no
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>> Bus info          Device       Class          Description
>> =========================================================
>> pci@0000:01:00.0  ens11f0      network        82599ES 10-Gigabit SFI/SFP+
>> Network Connection
>> pci@0000:01:00.1  ens11f1      network        82599ES 10-Gigabit SFI/SFP+
>> Network Connection
>> pci@0000:04:00.0  enp4s0f0     network        82599ES 10-Gigabit SFI/SFP+
>> Network Connection
>> pci@0000:04:00.1  enp4s0f1     network        82599ES 10-Gigabit SFI/SFP+
>> Network Connection
>> pci@0000:04:10.0  enp4s16      network        Illegal Vendor ID
>> pci@0000:04:10.2  enp4s16f2    network        Illegal Vendor ID
>>
>> (4) DPDK bind interfaces
>>
>> # dpdk-devbind -s
>>
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> drv=igb_uio unused=vfio-pci
>> 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio
>> unused=vfio-pci
>>
>> Network devices using kernel driver
>> ===================================
>> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
>> 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
>> 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>> if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
>> 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16
>> drv=ixgbevf unused=igb_uio,vfio-pci
>> 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb
>> unused=igb_uio,vfio-pci *Active*
>>
>> Other Network devices
>> =====================
>> <none>
>>
>> ...
>>
>> (5) Kernel dmesg
>>
>> # dmesg | grep -e DMAR
>> [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
>> 00000001 INTL 20091013)
>> [    0.000000] DMAR: IOMMU enabled
>> [    0.518747] DMAR: Host address width 46
>> [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>> [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
>> d2078c106f0466 ecap f020df
>> [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
>> [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
>> d2078c106f0466 ecap f020df
>> [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff
>> [    0.593344] DMAR: ATSR flags: 0x0
>> [    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0
>> [    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1
>> [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
>> [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 IOMMU 1
>> [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 IOMMU 1
>> [    0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
>> [    0.675326] DMAR-IR: Queued invalidation will be enabled to support
>> x2apic and Intr-remapping.
>> [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode
>> [    9.395170] DMAR: dmar1: Using Queued invalidation
>> [    9.405011] DMAR: Setting RMRR:
>> [    9.412006] DMAR: Setting identity map for device 0000:00:1d.0
>> [0x7bbc6000 - 0x7bbd4fff]
>> [    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
>> [    9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
>> 0xffffff]
>> [    9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O
>> [  287.023068] DMAR: DRHD: handling fault status reg 2
>> [  287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 18a260a000 [fault reason 06] PTE Read access is not set
>> [  287.023180] DMAR: DRHD: handling fault status reg 102
>> [  287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 18a3010000 [fault reason 06] PTE Read access is not set
>> [  287.038250] DMAR: DRHD: handling fault status reg 202
>> [  287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 18a3010000 [fault reason 06] PTE Read access is not set
>> [  288.170165] DMAR: DRHD: handling fault status reg 302
>> [  288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [  288.694496] DMAR: DRHD: handling fault status reg 402
>> [  288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr
>> 189069c000 [fault reason 06] PTE Read access is not set
>> [  289.927113] DMAR: DRHD: handling fault status reg 502
>> [  289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [  290.174275] DMAR: DRHD: handling fault status reg 602
>> [  290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [  292.174247] DMAR: DRHD: handling fault status reg 702
>> [  292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [  294.174227] DMAR: DRHD: handling fault status reg 2
>> [  294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [  296.174216] DMAR: DRHD: handling fault status reg 102
>> [  296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr
>> 1890754000 [fault reason 06] PTE Read access is not set
>> [root@infradev-comp006.naw02.infradev.viasat.io ~]
>> #
>>
>> Thanks.
>>
>>
> Hi Ravi,
>
> The "iommu=pt" workaround applies only when you want to use igb_uio
> driver. VFIO driver is able to fully utilize IOMMU without the need for
> pass-through mode. From your log i can see that some devices are bound to
> igb_uio while others are bound to vfio-pci. Just bind all of the devices
> you want to use with DPDK to vfio-pci and these errors should go away.
>
> --
> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-01-24 19:13   ` Ravi Kerur
@ 2018-01-25 10:49     ` Burakov, Anatoly
  2018-01-29 22:35       ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-01-25 10:49 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 24-Jan-18 7:13 PM, Ravi Kerur wrote:
> Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it 
> will work for both PF and VF interfaces since I am using both in my setup?
> 
> Thanks.

Yes, it should work for both PF and VF devices.

> 
> On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
> 
>         Hi,
> 
>         I am running into an issue when DPDK is started with iommu on
>         via GRUB
>         command. Problem is not seen with regular kernel driver, error
>         messages
>         show when DPDK is started and happens for both PF and VF interfaces.
> 
>         I am using DPDK 17.05 so the patch proposed in the following link is
>         available
>         http://dpdk.org/ml/archives/dev/2017-February/057048.html
>         <http://dpdk.org/ml/archives/dev/2017-February/057048.html>
> 
>         Workaround is to use "iommu=pt" but I want iommu enabled in my
>         setup. I
>         checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get
>         any details
>         on it.
> 
>         Kindly let me know how to resolve this issue.
> 
>         Following are the details
> 
>         (1) Linux kernel 4.9
>         (2) DPDK 17.05
> 
>         (3) IXGBE details
>         ethtool -i enp4s0f0  (PF driver)
>         driver: ixgbe
>         version: 5.3.3
>         firmware-version: 0x800007b8, 1.1018.0
>         bus-info: 0000:04:00.0
>         supports-statistics: yes
>         supports-test: yes
>         supports-eeprom-access: yes
>         supports-register-dump: yes
>         supports-priv-flags: yes
> 
>         ethtool -i enp4s16f2 (VF driver)
>         driver: ixgbevf
>         version: 4.3.2
>         firmware-version:
>         bus-info: 0000:04:10.2
>         supports-statistics: yes
>         supports-test: yes
>         supports-eeprom-access: no
>         supports-register-dump: yes
>         supports-priv-flags: no
> 
>         Bus info          Device       Class          Description
>         =========================================================
>         pci@0000:01:00.0  ens11f0      network        82599ES 10-Gigabit
>         SFI/SFP+
>         Network Connection
>         pci@0000:01:00.1  ens11f1      network        82599ES 10-Gigabit
>         SFI/SFP+
>         Network Connection
>         pci@0000:04:00.0  enp4s0f0     network        82599ES 10-Gigabit
>         SFI/SFP+
>         Network Connection
>         pci@0000:04:00.1  enp4s0f1     network        82599ES 10-Gigabit
>         SFI/SFP+
>         Network Connection
>         pci@0000:04:10.0  enp4s16      network        Illegal Vendor ID
>         pci@0000:04:10.2  enp4s16f2    network        Illegal Vendor ID
> 
>         (4) DPDK bind interfaces
> 
>         # dpdk-devbind -s
> 
>         Network devices using DPDK-compatible driver
>         ============================================
>         0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>         drv=igb_uio unused=vfio-pci
>         0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed'
>         drv=igb_uio
>         unused=vfio-pci
> 
>         Network devices using kernel driver
>         ===================================
>         0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>         if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
>         0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>         if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
>         0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>         if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
>         0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed'
>         if=enp4s16
>         drv=ixgbevf unused=igb_uio,vfio-pci
>         0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb
>         unused=igb_uio,vfio-pci *Active*
> 
>         Other Network devices
>         =====================
>         <none>
> 
>         ...
> 
>         (5) Kernel dmesg
> 
>         # dmesg | grep -e DMAR
>         [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA
>         A M I
>         00000001 INTL 20091013)
>         [    0.000000] DMAR: IOMMU enabled
>         [    0.518747] DMAR: Host address width 46
>         [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>         [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
>         d2078c106f0466 ecap f020df
>         [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
>         [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
>         d2078c106f0466 ecap f020df
>         [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end:
>         0x0000007bbd4fff
>         [    0.593344] DMAR: ATSR flags: 0x0
>         [    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity
>         domain: 0x0
>         [    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity
>         domain: 0x1
>         [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000
>         IOMMU 0
>         [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000
>         IOMMU 1
>         [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000
>         IOMMU 1
>         [    0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
>         [    0.675326] DMAR-IR: Queued invalidation will be enabled to
>         support
>         x2apic and Intr-remapping.
>         [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode
>         [    9.395170] DMAR: dmar1: Using Queued invalidation
>         [    9.405011] DMAR: Setting RMRR:
>         [    9.412006] DMAR: Setting identity map for device 0000:00:1d.0
>         [0x7bbc6000 - 0x7bbd4fff]
>         [    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
>         [    9.439712] DMAR: Setting identity map for device
>         0000:00:1f.0 [0x0 -
>         0xffffff]
>         [    9.454684] DMAR: Intel(R) Virtualization Technology for
>         Directed I/O
>         [  287.023068] DMAR: DRHD: handling fault status reg 2
>         [  287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         18a260a000 [fault reason 06] PTE Read access is not set
>         [  287.023180] DMAR: DRHD: handling fault status reg 102
>         [  287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         18a3010000 [fault reason 06] PTE Read access is not set
>         [  287.038250] DMAR: DRHD: handling fault status reg 202
>         [  287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         18a3010000 [fault reason 06] PTE Read access is not set
>         [  288.170165] DMAR: DRHD: handling fault status reg 302
>         [  288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  288.694496] DMAR: DRHD: handling fault status reg 402
>         [  288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr
>         189069c000 [fault reason 06] PTE Read access is not set
>         [  289.927113] DMAR: DRHD: handling fault status reg 502
>         [  289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  290.174275] DMAR: DRHD: handling fault status reg 602
>         [  290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  292.174247] DMAR: DRHD: handling fault status reg 702
>         [  292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  294.174227] DMAR: DRHD: handling fault status reg 2
>         [  294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [  296.174216] DMAR: DRHD: handling fault status reg 102
>         [  296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr
>         1890754000 [fault reason 06] PTE Read access is not set
>         [root@infradev-comp006.naw02.infradev.viasat.io
>         <mailto:root@infradev-comp006.naw02.infradev.viasat.io> ~]
>         #
> 
>         Thanks.
> 
> 
>     Hi Ravi,
> 
>     The "iommu=pt" workaround applies only when you want to use igb_uio
>     driver. VFIO driver is able to fully utilize IOMMU without the need
>     for pass-through mode. From your log i can see that some devices are
>     bound to igb_uio while others are bound to vfio-pci. Just bind all
>     of the devices you want to use with DPDK to vfio-pci and these
>     errors should go away.
> 
>     -- 
>     Thanks,
>     Anatoly
> 
> 


-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-01-25 10:49     ` Burakov, Anatoly
@ 2018-01-29 22:35       ` Ravi Kerur
  2018-01-31  9:59         ` Burakov, Anatoly
  2018-02-10 10:58         ` Burakov, Anatoly
  0 siblings, 2 replies; 33+ messages in thread
From: Ravi Kerur @ 2018-01-29 22:35 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

Hi Burakov,

When using vfio-pci on host both VF and PF interfaces works fine with dpdk
i.e. I don't see DMAR fault messages anymore. However, when I attach a VF
interface to a VM and start DPDK with vfio-pci inside VM I still see DMAR
fault messages on host. Both host and VM are booted with 'intel-iommu=on'
on GRUB. Ping from VM with DPDK/vfio-pci doesn't work (I think it's
expected because of DMAR faults), however, when VF interface uses ixgbevf
driver ping works.

Following are some details

/*****************On VM***************/
dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci
unused=ixgbevf

Network devices using kernel driver
===================================
0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>


00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
Virtual Function (rev 01)
        Subsystem: Intel Corporation 82599 Ethernet Controller Virtual
Function
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K]
        Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K]
        Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Capabilities: [a0] Express (v1) Root Complex Integrated Endpoint,
MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap-
ChkEn-
        Kernel driver in use: vfio-pci
        Kernel modules: ixgbevf

/***************on Host*************/
dmesg | grep DMAR
...
[  978.268143] DMAR: DRHD: handling fault status reg 2
[  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
33a128000 [fault reason 06] PTE Read access is not set
[ 1286.677726] DMAR: DRHD: handling fault status reg 102
[ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr
fb663000 [fault reason 06] PTE Read access is not set
[ 1676.436145] DMAR: DRHD: handling fault status reg 202
[ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr
33a128000 [fault reason 06] PTE Read access is not set
[ 1734.433649] DMAR: DRHD: handling fault status reg 302
[ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr
33a128000 [fault reason 06] PTE Read access is not set
[ 2324.428938] DMAR: DRHD: handling fault status reg 402
[ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr
7770c000 [fault reason 06] PTE Read access is not set
[ 2388.553640] DMAR: DRHD: handling fault status reg 502
[ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
33a128000 [fault reason 06] PTE Read access is not set


VM is started with

qemu-system-x86_64 -enable-kvm -M q35,accel=kvm,kernel-irqchip=split
-object iothread,id=iothread0 -device
intel-iommu,intremap=on,device-iotlb=on,caching-mode=on -cpu host
-daemonize -m 16G -smp 14 -uuid 0fc91c66-f0b1-11e7-acf4-525400123456 -name
212748-sriov-ravi-smac-alpha-SMAC10 -device ioh3420,id=root.1,chassis=1
-device ioh3420,id=root.2,chassis=2 -netdev
tap,vhost=on,queues=2,ifname=vn-vn2_1_,downscript=no,id=vn-vn2_1_,script=no
-device ioh3420,id=root.3,chassis=3 -device
virtio-net-pci,netdev=vn-vn2_1_,bus=root.3,ats=on,mq=on,vectors=6,mac=DE:AD:02:88:10:37,id=vn-vn2_1__dev
-netdev
tap,vhost=on,queues=2,ifname=vn-vn92_1_,downscript=no,id=vn-vn92_1_,script=no
-device ioh3420,id=root.4,chassis=4 -device
virtio-net-pci,mac=DE:AD:02:88:10:38,netdev=vn-vn92_1_,bus=root.4,ats=on,mq=on,vectors=6,id=vn-vn92_1__dev
-netdev
tap,vhost=on,queues=2,ifname=vn-vn93_1_,downscript=no,id=vn-vn93_1_,script=no
-device ioh3420,id=root.5,chassis=5 -device
virtio-net-pci,mac=DE:AD:02:88:10:39,netdev=vn-vn93_1_,bus=root.5,ats=on,mq=on,vectors=6,id=vn-vn93_1__dev
-vnc :16,websocket=15916 -qmp tcp:127.0.0.1:12001,server,nowait -chardev
socket,id=charmonitor,path=/tmp/mon.12001,server,nowait -mon
chardev=charmonitor,id=monitor -cdrom
/var/venom/cloud_init/0fc91c66-f0b1-11e7-acf4-525400123456.iso -*device
vfio-pci,host=04:10.0* -drive
file=/var/venom/instance_repo/test.img,if=none,id=drive-virtio-disk0,format=raw,aio=native,cache=none
-balloon none -device
virtio-blk-pci,scsi=off,iothread=iothread0,drive=drive-virtio-disk0,id=virtio-disk0,bus=root.1,ats=on,bootindex=1

Thanks.


On Thu, Jan 25, 2018 at 2:49 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 24-Jan-18 7:13 PM, Ravi Kerur wrote:
>
>> Hi Burakov, Thank you. I will try with vfio-pci driver. I am assuming it
>> will work for both PF and VF interfaces since I am using both in my setup?
>>
>> Thanks.
>>
>
> Yes, it should work for both PF and VF devices.
>
>
>> On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly <
>> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>     On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
>>
>>         Hi,
>>
>>         I am running into an issue when DPDK is started with iommu on
>>         via GRUB
>>         command. Problem is not seen with regular kernel driver, error
>>         messages
>>         show when DPDK is started and happens for both PF and VF
>> interfaces.
>>
>>         I am using DPDK 17.05 so the patch proposed in the following link
>> is
>>         available
>>         http://dpdk.org/ml/archives/dev/2017-February/057048.html
>>         <http://dpdk.org/ml/archives/dev/2017-February/057048.html>
>>
>>         Workaround is to use "iommu=pt" but I want iommu enabled in my
>>         setup. I
>>         checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get
>>         any details
>>         on it.
>>
>>         Kindly let me know how to resolve this issue.
>>
>>         Following are the details
>>
>>         (1) Linux kernel 4.9
>>         (2) DPDK 17.05
>>
>>         (3) IXGBE details
>>         ethtool -i enp4s0f0  (PF driver)
>>         driver: ixgbe
>>         version: 5.3.3
>>         firmware-version: 0x800007b8, 1.1018.0
>>         bus-info: 0000:04:00.0
>>         supports-statistics: yes
>>         supports-test: yes
>>         supports-eeprom-access: yes
>>         supports-register-dump: yes
>>         supports-priv-flags: yes
>>
>>         ethtool -i enp4s16f2 (VF driver)
>>         driver: ixgbevf
>>         version: 4.3.2
>>         firmware-version:
>>         bus-info: 0000:04:10.2
>>         supports-statistics: yes
>>         supports-test: yes
>>         supports-eeprom-access: no
>>         supports-register-dump: yes
>>         supports-priv-flags: no
>>
>>         Bus info          Device       Class          Description
>>         =========================================================
>>         pci@0000:01:00.0  ens11f0      network        82599ES 10-Gigabit
>>         SFI/SFP+
>>         Network Connection
>>         pci@0000:01:00.1  ens11f1      network        82599ES 10-Gigabit
>>         SFI/SFP+
>>         Network Connection
>>         pci@0000:04:00.0  enp4s0f0     network        82599ES 10-Gigabit
>>         SFI/SFP+
>>         Network Connection
>>         pci@0000:04:00.1  enp4s0f1     network        82599ES 10-Gigabit
>>         SFI/SFP+
>>         Network Connection
>>         pci@0000:04:10.0  enp4s16      network        Illegal Vendor ID
>>         pci@0000:04:10.2  enp4s16f2    network        Illegal Vendor ID
>>
>>         (4) DPDK bind interfaces
>>
>>         # dpdk-devbind -s
>>
>>         Network devices using DPDK-compatible driver
>>         ============================================
>>         0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>>         drv=igb_uio unused=vfio-pci
>>         0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed'
>>         drv=igb_uio
>>         unused=vfio-pci
>>
>>         Network devices using kernel driver
>>         ===================================
>>         0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>>         if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
>>         0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>>         if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
>>         0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
>>         if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
>>         0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed'
>>         if=enp4s16
>>         drv=ixgbevf unused=igb_uio,vfio-pci
>>         0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1
>> drv=igb
>>         unused=igb_uio,vfio-pci *Active*
>>
>>         Other Network devices
>>         =====================
>>         <none>
>>
>>         ...
>>
>>         (5) Kernel dmesg
>>
>>         # dmesg | grep -e DMAR
>>         [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA
>>         A M I
>>         00000001 INTL 20091013)
>>         [    0.000000] DMAR: IOMMU enabled
>>         [    0.518747] DMAR: Host address width 46
>>         [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>>         [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
>>         d2078c106f0466 ecap f020df
>>         [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
>>         [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
>>         d2078c106f0466 ecap f020df
>>         [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end:
>>         0x0000007bbd4fff
>>         [    0.593344] DMAR: ATSR flags: 0x0
>>         [    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity
>>         domain: 0x0
>>         [    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity
>>         domain: 0x1
>>         [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000
>>         IOMMU 0
>>         [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000
>>         IOMMU 1
>>         [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000
>>         IOMMU 1
>>         [    0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
>>         [    0.675326] DMAR-IR: Queued invalidation will be enabled to
>>         support
>>         x2apic and Intr-remapping.
>>         [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode
>>         [    9.395170] DMAR: dmar1: Using Queued invalidation
>>         [    9.405011] DMAR: Setting RMRR:
>>         [    9.412006] DMAR: Setting identity map for device 0000:00:1d.0
>>         [0x7bbc6000 - 0x7bbd4fff]
>>         [    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
>>         [    9.439712] DMAR: Setting identity map for device
>>         0000:00:1f.0 [0x0 -
>>         0xffffff]
>>         [    9.454684] DMAR: Intel(R) Virtualization Technology for
>>         Directed I/O
>>         [  287.023068] DMAR: DRHD: handling fault status reg 2
>>         [  287.023073] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         18a260a000 [fault reason 06] PTE Read access is not set
>>         [  287.023180] DMAR: DRHD: handling fault status reg 102
>>         [  287.023183] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         18a3010000 [fault reason 06] PTE Read access is not set
>>         [  287.038250] DMAR: DRHD: handling fault status reg 202
>>         [  287.038252] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         18a3010000 [fault reason 06] PTE Read access is not set
>>         [  288.170165] DMAR: DRHD: handling fault status reg 302
>>         [  288.170170] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  288.694496] DMAR: DRHD: handling fault status reg 402
>>         [  288.694499] DMAR: [DMA Read] Request device [04:10.2] fault
>> addr
>>         189069c000 [fault reason 06] PTE Read access is not set
>>         [  289.927113] DMAR: DRHD: handling fault status reg 502
>>         [  289.927116] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  290.174275] DMAR: DRHD: handling fault status reg 602
>>         [  290.174279] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  292.174247] DMAR: DRHD: handling fault status reg 702
>>         [  292.174251] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  294.174227] DMAR: DRHD: handling fault status reg 2
>>         [  294.174230] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [  296.174216] DMAR: DRHD: handling fault status reg 102
>>         [  296.174219] DMAR: [DMA Read] Request device [01:00.0] fault
>> addr
>>         1890754000 [fault reason 06] PTE Read access is not set
>>         [root@infradev-comp006.naw02.infradev.viasat.io
>>         <mailto:root@infradev-comp006.naw02.infradev.viasat.io> ~]
>>         #
>>
>>         Thanks.
>>
>>
>>     Hi Ravi,
>>
>>     The "iommu=pt" workaround applies only when you want to use igb_uio
>>     driver. VFIO driver is able to fully utilize IOMMU without the need
>>     for pass-through mode. From your log i can see that some devices are
>>     bound to igb_uio while others are bound to vfio-pci. Just bind all
>>     of the devices you want to use with DPDK to vfio-pci and these
>>     errors should go away.
>>
>>     --     Thanks,
>>     Anatoly
>>
>>
>>
>
> --
> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-01-29 22:35       ` Ravi Kerur
@ 2018-01-31  9:59         ` Burakov, Anatoly
  2018-01-31 21:51           ` Ravi Kerur
  2018-02-10 10:58         ` Burakov, Anatoly
  1 sibling, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-01-31  9:59 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 29-Jan-18 10:35 PM, Ravi Kerur wrote:
> Hi Burakov,
> 
> When using vfio-pci on host both VF and PF interfaces works fine with 
> dpdk i.e. I don't see DMAR fault messages anymore. However, when I 
> attach a VF interface to a VM and start DPDK with vfio-pci inside VM I 
> still see DMAR fault messages on host. Both host and VM are booted with 
> 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work 
> (I think it's expected because of DMAR faults), however, when VF 
> interface uses ixgbevf driver ping works.
> 
> Following are some details
> 
> /*****************On VM***************/
> dpdk-devbind -s
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci 
> unused=ixgbevf
> 
> Network devices using kernel driver
> ===================================
> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
> 
> Other network devices
> =====================
> <none>
> 
> Crypto devices using DPDK-compatible driver
> ===========================================
> <none>
> 
> Crypto devices using kernel driver
> ==================================
> <none>
> 
> Other crypto devices
> ====================
> <none>
> 
> 
> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller 
> Virtual Function (rev 01)
>          Subsystem: Intel Corporation 82599 Ethernet Controller Virtual 
> Function
>          Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- 
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>          Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K]
>          Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K]
>          Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>                  Vector table: BAR=3 offset=00000000
>                  PBA: BAR=3 offset=00002000
>          Capabilities: [a0] Express (v1) Root Complex Integrated 
> Endpoint, MSI 00
>                  DevCap: MaxPayload 128 bytes, PhantFunc 0
>                          ExtTag- RBE-
>                  DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
> Unsupported-
>                          RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                          MaxPayload 128 bytes, MaxReadReq 128 bytes
>                  DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- 
> AuxPwr- TransPend-
>          Capabilities: [100 v1] Advanced Error Reporting
>                  UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                  UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                  UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                  CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
> NonFatalErr-
>                  CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
> NonFatalErr-
>                  AERCap: First Error Pointer: 00, GenCap- CGenEn- 
> ChkCap- ChkEn-
>          Kernel driver in use: vfio-pci
>          Kernel modules: ixgbevf
> 
> /***************on Host*************/
> dmesg | grep DMAR
> ...
> [  978.268143] DMAR: DRHD: handling fault status reg 2
> [  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> [ 1286.677726] DMAR: DRHD: handling fault status reg 102
> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> fb663000 [fault reason 06] PTE Read access is not set
> [ 1676.436145] DMAR: DRHD: handling fault status reg 202
> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> [ 1734.433649] DMAR: DRHD: handling fault status reg 302
> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> [ 2324.428938] DMAR: DRHD: handling fault status reg 402
> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 7770c000 [fault reason 06] PTE Read access is not set
> [ 2388.553640] DMAR: DRHD: handling fault status reg 502
> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> 
> 
> VM is started with
> 
> qemu-system-x86_64 -enable-kvm -M q35,accel=kvm,kernel-irqchip=split 
> -object iothread,id=iothread0 -device 
> intel-iommu,intremap=on,device-iotlb=on,caching-mode=on -cpu host 
> -daemonize -m 16G -smp 14 -uuid 0fc91c66-f0b1-11e7-acf4-525400123456 
> -name 212748-sriov-ravi-smac-alpha-SMAC10 -device 
> ioh3420,id=root.1,chassis=1 -device ioh3420,id=root.2,chassis=2 -netdev 
> tap,vhost=on,queues=2,ifname=vn-vn2_1_,downscript=no,id=vn-vn2_1_,script=no 
> -device ioh3420,id=root.3,chassis=3 -device 
> virtio-net-pci,netdev=vn-vn2_1_,bus=root.3,ats=on,mq=on,vectors=6,mac=DE:AD:02:88:10:37,id=vn-vn2_1__dev 
> -netdev 
> tap,vhost=on,queues=2,ifname=vn-vn92_1_,downscript=no,id=vn-vn92_1_,script=no 
> -device ioh3420,id=root.4,chassis=4 -device 
> virtio-net-pci,mac=DE:AD:02:88:10:38,netdev=vn-vn92_1_,bus=root.4,ats=on,mq=on,vectors=6,id=vn-vn92_1__dev 
> -netdev 
> tap,vhost=on,queues=2,ifname=vn-vn93_1_,downscript=no,id=vn-vn93_1_,script=no 
> -device ioh3420,id=root.5,chassis=5 -device 
> virtio-net-pci,mac=DE:AD:02:88:10:39,netdev=vn-vn93_1_,bus=root.5,ats=on,mq=on,vectors=6,id=vn-vn93_1__dev 
> -vnc :16,websocket=15916 -qmp tcp:127.0.0.1:12001 
> <http://127.0.0.1:12001>,server,nowait -chardev 
> socket,id=charmonitor,path=/tmp/mon.12001,server,nowait -mon 
> chardev=charmonitor,id=monitor -cdrom 
> /var/venom/cloud_init/0fc91c66-f0b1-11e7-acf4-525400123456.iso -*device 
> vfio-pci,host=04:10.0* -drive 
> file=/var/venom/instance_repo/test.img,if=none,id=drive-virtio-disk0,format=raw,aio=native,cache=none 
> -balloon none -device 
> virtio-blk-pci,scsi=off,iothread=iothread0,drive=drive-virtio-disk0,id=virtio-disk0,bus=root.1,ats=on,bootindex=1
> 
> Thanks.
> 
> 
> On Thu, Jan 25, 2018 at 2:49 AM, Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     On 24-Jan-18 7:13 PM, Ravi Kerur wrote:
> 
>         Hi Burakov, Thank you. I will try with vfio-pci driver. I am
>         assuming it will work for both PF and VF interfaces since I am
>         using both in my setup?
> 
>         Thanks.
> 
> 
>     Yes, it should work for both PF and VF devices.
> 
> 
>         On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly
>         <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
>         <mailto:anatoly.burakov@intel.com
>         <mailto:anatoly.burakov@intel.com>>> wrote:
> 
>              On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
> 
>                  Hi,
> 
>                  I am running into an issue when DPDK is started with
>         iommu on
>                  via GRUB
>                  command. Problem is not seen with regular kernel
>         driver, error
>                  messages
>                  show when DPDK is started and happens for both PF and
>         VF interfaces.
> 
>                  I am using DPDK 17.05 so the patch proposed in the
>         following link is
>                  available
>         http://dpdk.org/ml/archives/dev/2017-February/057048.html
>         <http://dpdk.org/ml/archives/dev/2017-February/057048.html>
>                 
>         <http://dpdk.org/ml/archives/dev/2017-February/057048.html
>         <http://dpdk.org/ml/archives/dev/2017-February/057048.html>>
> 
>                  Workaround is to use "iommu=pt" but I want iommu
>         enabled in my
>                  setup. I
>                  checked BIOS for reserved memory(DMA RMRR for IXGBE)
>         didn't get
>                  any details
>                  on it.
> 
>                  Kindly let me know how to resolve this issue.
> 
>                  Following are the details
> 
>                  (1) Linux kernel 4.9
>                  (2) DPDK 17.05
> 
>                  (3) IXGBE details
>                  ethtool -i enp4s0f0  (PF driver)
>                  driver: ixgbe
>                  version: 5.3.3
>                  firmware-version: 0x800007b8, 1.1018.0
>                  bus-info: 0000:04:00.0
>                  supports-statistics: yes
>                  supports-test: yes
>                  supports-eeprom-access: yes
>                  supports-register-dump: yes
>                  supports-priv-flags: yes
> 
>                  ethtool -i enp4s16f2 (VF driver)
>                  driver: ixgbevf
>                  version: 4.3.2
>                  firmware-version:
>                  bus-info: 0000:04:10.2
>                  supports-statistics: yes
>                  supports-test: yes
>                  supports-eeprom-access: no
>                  supports-register-dump: yes
>                  supports-priv-flags: no
> 
>                  Bus info          Device       Class          Description
>                  =========================================================
>                  pci@0000:01:00.0  ens11f0      network        82599ES
>         10-Gigabit
>                  SFI/SFP+
>                  Network Connection
>                  pci@0000:01:00.1  ens11f1      network        82599ES
>         10-Gigabit
>                  SFI/SFP+
>                  Network Connection
>                  pci@0000:04:00.0  enp4s0f0     network        82599ES
>         10-Gigabit
>                  SFI/SFP+
>                  Network Connection
>                  pci@0000:04:00.1  enp4s0f1     network        82599ES
>         10-Gigabit
>                  SFI/SFP+
>                  Network Connection
>                  pci@0000:04:10.0  enp4s16      network        Illegal
>         Vendor ID
>                  pci@0000:04:10.2  enp4s16f2    network        Illegal
>         Vendor ID
> 
>                  (4) DPDK bind interfaces
> 
>                  # dpdk-devbind -s
> 
>                  Network devices using DPDK-compatible driver
>                  ============================================
>                  0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network
>         Connection 10fb'
>                  drv=igb_uio unused=vfio-pci
>                  0000:04:10.2 '82599 Ethernet Controller Virtual
>         Function 10ed'
>                  drv=igb_uio
>                  unused=vfio-pci
> 
>                  Network devices using kernel driver
>                  ===================================
>                  0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network
>         Connection 10fb'
>                  if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
>                  0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network
>         Connection 10fb'
>                  if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
>                  0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network
>         Connection 10fb'
>                  if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
>                  0000:04:10.0 '82599 Ethernet Controller Virtual
>         Function 10ed'
>                  if=enp4s16
>                  drv=ixgbevf unused=igb_uio,vfio-pci
>                  0000:06:00.0 'I210 Gigabit Network Connection 1533'
>         if=eno1 drv=igb
>                  unused=igb_uio,vfio-pci *Active*
> 
>                  Other Network devices
>                  =====================
>                  <none>
> 
>                  ...
> 
>                  (5) Kernel dmesg
> 
>                  # dmesg | grep -e DMAR
>                  [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0
>         (v01 ALASKA
>                  A M I
>                  00000001 INTL 20091013)
>                  [    0.000000] DMAR: IOMMU enabled
>                  [    0.518747] DMAR: Host address width 46
>                  [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>                  [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver
>         1:0 cap
>                  d2078c106f0466 ecap f020df
>                  [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
>                  [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver
>         1:0 cap
>                  d2078c106f0466 ecap f020df
>                  [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end:
>                  0x0000007bbd4fff
>                  [    0.593344] DMAR: ATSR flags: 0x0
>                  [    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity
>                  domain: 0x0
>                  [    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity
>                  domain: 0x1
>                  [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base 
>         0xfbffc000
>                  IOMMU 0
>                  [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base 
>         0xc7ffc000
>                  IOMMU 1
>                  [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base 
>         0xc7ffc000
>                  IOMMU 1
>                  [    0.664324] DMAR-IR: HPET id 0 under DRHD base
>         0xc7ffc000
>                  [    0.675326] DMAR-IR: Queued invalidation will be
>         enabled to
>                  support
>                  x2apic and Intr-remapping.
>                  [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic
>         mode
>                  [    9.395170] DMAR: dmar1: Using Queued invalidation
>                  [    9.405011] DMAR: Setting RMRR:
>                  [    9.412006] DMAR: Setting identity map for device
>         0000:00:1d.0
>                  [0x7bbc6000 - 0x7bbd4fff]
>                  [    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
>                  [    9.439712] DMAR: Setting identity map for device
>                  0000:00:1f.0 [0x0 -
>                  0xffffff]
>                  [    9.454684] DMAR: Intel(R) Virtualization Technology for
>                  Directed I/O
>                  [  287.023068] DMAR: DRHD: handling fault status reg 2
>                  [  287.023073] DMAR: [DMA Read] Request device
>         [01:00.0] fault addr
>                  18a260a000 [fault reason 06] PTE Read access is not set
>                  [  287.023180] DMAR: DRHD: handling fault status reg 102
>                  [  287.023183] DMAR: [DMA Read] Request device
>         [01:00.0] fault addr
>                  18a3010000 [fault reason 06] PTE Read access is not set
>                  [  287.038250] DMAR: DRHD: handling fault status reg 202
>                  [  287.038252] DMAR: [DMA Read] Request device
>         [01:00.0] fault addr
>                  18a3010000 [fault reason 06] PTE Read access is not set
>                  [  288.170165] DMAR: DRHD: handling fault status reg 302
>                  [  288.170170] DMAR: [DMA Read] Request device
>         [01:00.0] fault addr
>                  1890754000 [fault reason 06] PTE Read access is not set
>                  [  288.694496] DMAR: DRHD: handling fault status reg 402
>                  [  288.694499] DMAR: [DMA Read] Request device
>         [04:10.2] fault addr
>                  189069c000 [fault reason 06] PTE Read access is not set
>                  [  289.927113] DMAR: DRHD: handling fault status reg 502
>                  [  289.927116] DMAR: [DMA Read] Request device
>         [01:00.0] fault addr
>                  1890754000 [fault reason 06] PTE Read access is not set
>                  [  290.174275] DMAR: DRHD: handling fault status reg 602
>                  [  290.174279] DMAR: [DMA Read] Request device
>         [01:00.0] fault addr
>                  1890754000 [fault reason 06] PTE Read access is not set
>                  [  292.174247] DMAR: DRHD: handling fault status reg 702
>                  [  292.174251] DMAR: [DMA Read] Request device
>         [01:00.0] fault addr
>                  1890754000 [fault reason 06] PTE Read access is not set
>                  [  294.174227] DMAR: DRHD: handling fault status reg 2
>                  [  294.174230] DMAR: [DMA Read] Request device
>         [01:00.0] fault addr
>                  1890754000 [fault reason 06] PTE Read access is not set
>                  [  296.174216] DMAR: DRHD: handling fault status reg 102
>                  [  296.174219] DMAR: [DMA Read] Request device
>         [01:00.0] fault addr
>                  1890754000 [fault reason 06] PTE Read access is not set
>                  [root@infradev-comp006.naw02.infradev.viasat.io
>         <mailto:root@infradev-comp006.naw02.infradev.viasat.io>
>                  <mailto:root@infradev-comp006.naw02.infradev.viasat.io
>         <mailto:root@infradev-comp006.naw02.infradev.viasat.io>> ~]
>                  #
> 
>                  Thanks.
> 
> 
>              Hi Ravi,
> 
>              The "iommu=pt" workaround applies only when you want to use
>         igb_uio
>              driver. VFIO driver is able to fully utilize IOMMU without
>         the need
>              for pass-through mode. From your log i can see that some
>         devices are
>              bound to igb_uio while others are bound to vfio-pci. Just
>         bind all
>              of the devices you want to use with DPDK to vfio-pci and these
>              errors should go away.
> 
>              --     Thanks,
>              Anatoly
> 
> 
> 
> 
>     -- 
>     Thanks,
>     Anatoly
> 
> 

Hi Ravi,

Using vfio-pci in IOMMU mode will only work if your VM provides IOMMU 
emulation (it's a fairly recent development, so your QEMU must be of an 
appropriate version - can't recall which one off the top of my head). 
Otherwise you'd have to treat your VM as if it was a machine without 
IOMMU, i.e. use noiommu mode for VFIO, or igb_uio driver.

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-01-31  9:59         ` Burakov, Anatoly
@ 2018-01-31 21:51           ` Ravi Kerur
  2018-02-01 10:10             ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-01-31 21:51 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Wed, Jan 31, 2018 at 1:59 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 29-Jan-18 10:35 PM, Ravi Kerur wrote:
>
>> Hi Burakov,
>>
>> When using vfio-pci on host both VF and PF interfaces works fine with
>> dpdk i.e. I don't see DMAR fault messages anymore. However, when I attach a
>> VF interface to a VM and start DPDK with vfio-pci inside VM I still see
>> DMAR fault messages on host. Both host and VM are booted with
>> 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work (I
>> think it's expected because of DMAR faults), however, when VF interface
>> uses ixgbevf driver ping works.
>>
>> Following are some details
>>
>> /*****************On VM***************/
>> dpdk-devbind -s
>>
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci
>> unused=ixgbevf
>>
>> Network devices using kernel driver
>> ===================================
>> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
>> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
>> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
>>
>> Other network devices
>> =====================
>> <none>
>>
>> Crypto devices using DPDK-compatible driver
>> ===========================================
>> <none>
>>
>> Crypto devices using kernel driver
>> ==================================
>> <none>
>>
>> Other crypto devices
>> ====================
>> <none>
>>
>>
>> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
>> Virtual Function (rev 01)
>>          Subsystem: Intel Corporation 82599 Ethernet Controller Virtual
>> Function
>>          Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
>> ParErr- Stepping- SERR- FastB2B- DisINTx+
>>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>          Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K]
>>          Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K]
>>          Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>>                  Vector table: BAR=3 offset=00000000
>>                  PBA: BAR=3 offset=00002000
>>          Capabilities: [a0] Express (v1) Root Complex Integrated
>> Endpoint, MSI 00
>>                  DevCap: MaxPayload 128 bytes, PhantFunc 0
>>                          ExtTag- RBE-
>>                  DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                          RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>>                          MaxPayload 128 bytes, MaxReadReq 128 bytes
>>                  DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>> TransPend-
>>          Capabilities: [100 v1] Advanced Error Reporting
>>                  UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
>> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                  UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
>> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                  UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
>> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                  CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>> NonFatalErr-
>>                  CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>> NonFatalErr-
>>                  AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap-
>> ChkEn-
>>          Kernel driver in use: vfio-pci
>>          Kernel modules: ixgbevf
>>
>> /***************on Host*************/
>> dmesg | grep DMAR
>> ...
>> [  978.268143] DMAR: DRHD: handling fault status reg 2
>> [  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>> [ 1286.677726] DMAR: DRHD: handling fault status reg 102
>> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> fb663000 [fault reason 06] PTE Read access is not set
>> [ 1676.436145] DMAR: DRHD: handling fault status reg 202
>> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>> [ 1734.433649] DMAR: DRHD: handling fault status reg 302
>> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>> [ 2324.428938] DMAR: DRHD: handling fault status reg 402
>> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 7770c000 [fault reason 06] PTE Read access is not set
>> [ 2388.553640] DMAR: DRHD: handling fault status reg 502
>> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>>
>>
>> VM is started with
>>
>> qemu-system-x86_64 -enable-kvm -M q35,accel=kvm,kernel-irqchip=split
>> -object iothread,id=iothread0 -device intel-iommu,intremap=on,device-iotlb=on,caching-mode=on
>> -cpu host -daemonize -m 16G -smp 14 -uuid 0fc91c66-f0b1-11e7-acf4-525400123456
>> -name 212748-sriov-ravi-smac-alpha-SMAC10 -device
>> ioh3420,id=root.1,chassis=1 -device ioh3420,id=root.2,chassis=2 -netdev
>> tap,vhost=on,queues=2,ifname=vn-vn2_1_,downscript=no,id=vn-vn2_1_,script=no
>> -device ioh3420,id=root.3,chassis=3 -device virtio-net-pci,netdev=vn-vn2_1
>> _,bus=root.3,ats=on,mq=on,vectors=6,mac=DE:AD:02:88:10:37,id=vn-vn2_1__dev
>> -netdev tap,vhost=on,queues=2,ifname=vn-vn92_1_,downscript=no,id=vn-vn92_1_,script=no
>> -device ioh3420,id=root.4,chassis=4 -device virtio-net-pci,mac=DE:AD:02:88
>> :10:38,netdev=vn-vn92_1_,bus=root.4,ats=on,mq=on,vectors=6,id=vn-vn92_1__dev
>> -netdev tap,vhost=on,queues=2,ifname=vn-vn93_1_,downscript=no,id=vn-vn93_1_,script=no
>> -device ioh3420,id=root.5,chassis=5 -device virtio-net-pci,mac=DE:AD:02:88
>> :10:39,netdev=vn-vn93_1_,bus=root.5,ats=on,mq=on,vectors=6,id=vn-vn93_1__dev
>> -vnc :16,websocket=15916 -qmp tcp:127.0.0.1:12001 <http://127.0.0.1:12001
>> >,server,nowait -chardev socket,id=charmonitor,path=/tmp/mon.12001,server,nowait
>> -mon chardev=charmonitor,id=monitor -cdrom /var/venom/cloud_init/0fc91c66
>> -f0b1-11e7-acf4-525400123456.iso -*device vfio-pci,host=04:10.0* -drive
>> file=/var/venom/instance_repo/test.img,if=none,id=drive-virt
>> io-disk0,format=raw,aio=native,cache=none -balloon none -device
>> virtio-blk-pci,scsi=off,iothread=iothread0,drive=drive-
>> virtio-disk0,id=virtio-disk0,bus=root.1,ats=on,bootindex=1
>>
>> Thanks.
>>
>>
>> On Thu, Jan 25, 2018 at 2:49 AM, Burakov, Anatoly <
>> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>     On 24-Jan-18 7:13 PM, Ravi Kerur wrote:
>>
>>         Hi Burakov, Thank you. I will try with vfio-pci driver. I am
>>         assuming it will work for both PF and VF interfaces since I am
>>         using both in my setup?
>>
>>         Thanks.
>>
>>
>>     Yes, it should work for both PF and VF devices.
>>
>>
>>         On Wed, Jan 24, 2018 at 2:31 AM, Burakov, Anatoly
>>         <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
>>         <mailto:anatoly.burakov@intel.com
>>
>>         <mailto:anatoly.burakov@intel.com>>> wrote:
>>
>>              On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
>>
>>                  Hi,
>>
>>                  I am running into an issue when DPDK is started with
>>         iommu on
>>                  via GRUB
>>                  command. Problem is not seen with regular kernel
>>         driver, error
>>                  messages
>>                  show when DPDK is started and happens for both PF and
>>         VF interfaces.
>>
>>                  I am using DPDK 17.05 so the patch proposed in the
>>         following link is
>>                  available
>>         http://dpdk.org/ml/archives/dev/2017-February/057048.html
>>         <http://dpdk.org/ml/archives/dev/2017-February/057048.html>
>>                         <http://dpdk.org/ml/archives/d
>> ev/2017-February/057048.html
>>         <http://dpdk.org/ml/archives/dev/2017-February/057048.html>>
>>
>>                  Workaround is to use "iommu=pt" but I want iommu
>>         enabled in my
>>                  setup. I
>>                  checked BIOS for reserved memory(DMA RMRR for IXGBE)
>>         didn't get
>>                  any details
>>                  on it.
>>
>>                  Kindly let me know how to resolve this issue.
>>
>>                  Following are the details
>>
>>                  (1) Linux kernel 4.9
>>                  (2) DPDK 17.05
>>
>>                  (3) IXGBE details
>>                  ethtool -i enp4s0f0  (PF driver)
>>                  driver: ixgbe
>>                  version: 5.3.3
>>                  firmware-version: 0x800007b8, 1.1018.0
>>                  bus-info: 0000:04:00.0
>>                  supports-statistics: yes
>>                  supports-test: yes
>>                  supports-eeprom-access: yes
>>                  supports-register-dump: yes
>>                  supports-priv-flags: yes
>>
>>                  ethtool -i enp4s16f2 (VF driver)
>>                  driver: ixgbevf
>>                  version: 4.3.2
>>                  firmware-version:
>>                  bus-info: 0000:04:10.2
>>                  supports-statistics: yes
>>                  supports-test: yes
>>                  supports-eeprom-access: no
>>                  supports-register-dump: yes
>>                  supports-priv-flags: no
>>
>>                  Bus info          Device       Class          Description
>>                  ==============================
>> ===========================
>>                  pci@0000:01:00.0  ens11f0      network        82599ES
>>         10-Gigabit
>>                  SFI/SFP+
>>                  Network Connection
>>                  pci@0000:01:00.1  ens11f1      network        82599ES
>>         10-Gigabit
>>                  SFI/SFP+
>>                  Network Connection
>>                  pci@0000:04:00.0  enp4s0f0     network        82599ES
>>         10-Gigabit
>>                  SFI/SFP+
>>                  Network Connection
>>                  pci@0000:04:00.1  enp4s0f1     network        82599ES
>>         10-Gigabit
>>                  SFI/SFP+
>>                  Network Connection
>>                  pci@0000:04:10.0  enp4s16      network        Illegal
>>         Vendor ID
>>                  pci@0000:04:10.2  enp4s16f2    network        Illegal
>>         Vendor ID
>>
>>                  (4) DPDK bind interfaces
>>
>>                  # dpdk-devbind -s
>>
>>                  Network devices using DPDK-compatible driver
>>                  ============================================
>>                  0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network
>>         Connection 10fb'
>>                  drv=igb_uio unused=vfio-pci
>>                  0000:04:10.2 '82599 Ethernet Controller Virtual
>>         Function 10ed'
>>                  drv=igb_uio
>>                  unused=vfio-pci
>>
>>                  Network devices using kernel driver
>>                  ===================================
>>                  0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network
>>         Connection 10fb'
>>                  if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
>>                  0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network
>>         Connection 10fb'
>>                  if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
>>                  0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network
>>         Connection 10fb'
>>                  if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
>>                  0000:04:10.0 '82599 Ethernet Controller Virtual
>>         Function 10ed'
>>                  if=enp4s16
>>                  drv=ixgbevf unused=igb_uio,vfio-pci
>>                  0000:06:00.0 'I210 Gigabit Network Connection 1533'
>>         if=eno1 drv=igb
>>                  unused=igb_uio,vfio-pci *Active*
>>
>>                  Other Network devices
>>                  =====================
>>                  <none>
>>
>>                  ...
>>
>>                  (5) Kernel dmesg
>>
>>                  # dmesg | grep -e DMAR
>>                  [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0
>>         (v01 ALASKA
>>                  A M I
>>                  00000001 INTL 20091013)
>>                  [    0.000000] DMAR: IOMMU enabled
>>                  [    0.518747] DMAR: Host address width 46
>>                  [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags:
>> 0x0
>>                  [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver
>>         1:0 cap
>>                  d2078c106f0466 ecap f020df
>>                  [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags:
>> 0x1
>>                  [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver
>>         1:0 cap
>>                  d2078c106f0466 ecap f020df
>>                  [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end:
>>                  0x0000007bbd4fff
>>                  [    0.593344] DMAR: ATSR flags: 0x0
>>                  [    0.600178] DMAR: RHSA base: 0x000000c7ffc000
>> proximity
>>                  domain: 0x0
>>                  [    0.612905] DMAR: RHSA base: 0x000000fbffc000
>> proximity
>>                  domain: 0x1
>>                  [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base
>>    0xfbffc000
>>                  IOMMU 0
>>                  [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base
>>    0xc7ffc000
>>                  IOMMU 1
>>                  [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base
>>    0xc7ffc000
>>                  IOMMU 1
>>                  [    0.664324] DMAR-IR: HPET id 0 under DRHD base
>>         0xc7ffc000
>>                  [    0.675326] DMAR-IR: Queued invalidation will be
>>         enabled to
>>                  support
>>                  x2apic and Intr-remapping.
>>                  [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic
>>         mode
>>                  [    9.395170] DMAR: dmar1: Using Queued invalidation
>>                  [    9.405011] DMAR: Setting RMRR:
>>                  [    9.412006] DMAR: Setting identity map for device
>>         0000:00:1d.0
>>                  [0x7bbc6000 - 0x7bbd4fff]
>>                  [    9.428569] DMAR: Prepare 0-16MiB unity mapping for
>> LPC
>>                  [    9.439712] DMAR: Setting identity map for device
>>                  0000:00:1f.0 [0x0 -
>>                  0xffffff]
>>                  [    9.454684] DMAR: Intel(R) Virtualization Technology
>> for
>>                  Directed I/O
>>                  [  287.023068] DMAR: DRHD: handling fault status reg 2
>>                  [  287.023073] DMAR: [DMA Read] Request device
>>         [01:00.0] fault addr
>>                  18a260a000 [fault reason 06] PTE Read access is not set
>>                  [  287.023180] DMAR: DRHD: handling fault status reg 102
>>                  [  287.023183] DMAR: [DMA Read] Request device
>>         [01:00.0] fault addr
>>                  18a3010000 [fault reason 06] PTE Read access is not set
>>                  [  287.038250] DMAR: DRHD: handling fault status reg 202
>>                  [  287.038252] DMAR: [DMA Read] Request device
>>         [01:00.0] fault addr
>>                  18a3010000 [fault reason 06] PTE Read access is not set
>>                  [  288.170165] DMAR: DRHD: handling fault status reg 302
>>                  [  288.170170] DMAR: [DMA Read] Request device
>>         [01:00.0] fault addr
>>                  1890754000 [fault reason 06] PTE Read access is not set
>>                  [  288.694496] DMAR: DRHD: handling fault status reg 402
>>                  [  288.694499] DMAR: [DMA Read] Request device
>>         [04:10.2] fault addr
>>                  189069c000 [fault reason 06] PTE Read access is not set
>>                  [  289.927113] DMAR: DRHD: handling fault status reg 502
>>                  [  289.927116] DMAR: [DMA Read] Request device
>>         [01:00.0] fault addr
>>                  1890754000 [fault reason 06] PTE Read access is not set
>>                  [  290.174275] DMAR: DRHD: handling fault status reg 602
>>                  [  290.174279] DMAR: [DMA Read] Request device
>>         [01:00.0] fault addr
>>                  1890754000 [fault reason 06] PTE Read access is not set
>>                  [  292.174247] DMAR: DRHD: handling fault status reg 702
>>                  [  292.174251] DMAR: [DMA Read] Request device
>>         [01:00.0] fault addr
>>                  1890754000 [fault reason 06] PTE Read access is not set
>>                  [  294.174227] DMAR: DRHD: handling fault status reg 2
>>                  [  294.174230] DMAR: [DMA Read] Request device
>>         [01:00.0] fault addr
>>                  1890754000 [fault reason 06] PTE Read access is not set
>>                  [  296.174216] DMAR: DRHD: handling fault status reg 102
>>                  [  296.174219] DMAR: [DMA Read] Request device
>>         [01:00.0] fault addr
>>                  1890754000 [fault reason 06] PTE Read access is not set
>>                  [root@infradev-comp006.naw02.infradev.viasat.io
>>         <mailto:root@infradev-comp006.naw02.infradev.viasat.io>
>>                  <mailto:root@infradev-comp006.naw02.infradev.viasat.io
>>         <mailto:root@infradev-comp006.naw02.infradev.viasat.io>> ~]
>>                  #
>>
>>                  Thanks.
>>
>>
>>              Hi Ravi,
>>
>>              The "iommu=pt" workaround applies only when you want to use
>>         igb_uio
>>              driver. VFIO driver is able to fully utilize IOMMU without
>>         the need
>>              for pass-through mode. From your log i can see that some
>>         devices are
>>              bound to igb_uio while others are bound to vfio-pci. Just
>>         bind all
>>              of the devices you want to use with DPDK to vfio-pci and
>> these
>>              errors should go away.
>>
>>              --     Thanks,
>>              Anatoly
>>
>>
>>
>>
>>     --     Thanks,
>>     Anatoly
>>
>>
>>
> Hi Ravi,
>
> Using vfio-pci in IOMMU mode will only work if your VM provides IOMMU
> emulation (it's a fairly recent development, so your QEMU must be of an
> appropriate version - can't recall which one off the top of my head).
> Otherwise you'd have to treat your VM as if it was a machine without IOMMU,
> i.e. use noiommu mode for VFIO, or igb_uio driver.
>
> --
> Thanks,
> Anatoly
>

Hi Anatoly,

Thanks. I am following wiki link below which uses vIOMMU with DPDK as a
use-case and instantiate VM as specified with Q35 chipset in Qemu.

https://wiki.qemu.org/Features/VT-d

Qemu-version is 2.11
Host kernel 4.9
Guest kernel 4.4

I can only guess that guest kernel needs an upgrade in my setup to work
correctly, if versions on my setup rings a bell on not having support
kindly let me know.

When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on guest I
get following error
...
vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored
...

in guest.

Thanks.

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-01-31 21:51           ` Ravi Kerur
@ 2018-02-01 10:10             ` Burakov, Anatoly
  2018-02-01 19:26               ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-01 10:10 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
> 
> Hi Anatoly,
> 
> Thanks. I am following wiki link below which uses vIOMMU with DPDK as a 
> use-case and instantiate VM as specified with Q35 chipset in Qemu.
> 
> https://wiki.qemu.org/Features/VT-d
> 
> Qemu-version is 2.11
> Host kernel 4.9
> Guest kernel 4.4
> 
> I can only guess that guest kernel needs an upgrade in my setup to work 
> correctly, if versions on my setup rings a bell on not having support 
> kindly let me know.
> 
> When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on guest I 
> get following error
> ...
> vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored
> ...
> 
> in guest.
> 
> Thanks.

AFAIK kernel 4.4 should have noiommu mode - it was introduced in 3.1x 
days. However, in order for that to work, kernel also has to be built 
with this mode enabled. My guess is, whoever is the supplier of your 
kernel, did not do that. You should double-check the kernel 
configuration of your distribution.

However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode - 
"regular" vfio should work fine. noiommu mode should only be needed if 
you know you don't have IOMMU enabled in your kernel, and even if you 
can't enable it, you can still use igb_uio.

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-01 10:10             ` Burakov, Anatoly
@ 2018-02-01 19:26               ` Ravi Kerur
  2018-02-02 10:28                 ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-01 19:26 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
>
>>
>> Hi Anatoly,
>>
>> Thanks. I am following wiki link below which uses vIOMMU with DPDK as a
>> use-case and instantiate VM as specified with Q35 chipset in Qemu.
>>
>> https://wiki.qemu.org/Features/VT-d
>>
>> Qemu-version is 2.11
>> Host kernel 4.9
>> Guest kernel 4.4
>>
>> I can only guess that guest kernel needs an upgrade in my setup to work
>> correctly, if versions on my setup rings a bell on not having support
>> kindly let me know.
>>
>> When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on guest I
>> get following error
>> ...
>> vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored
>> ...
>>
>> in guest.
>>
>> Thanks.
>>
>
>
AFAIK kernel 4.4 should have noiommu mode - it was introduced in 3.1x days.
> However, in order for that to work, kernel also has to be built with this
> mode enabled. My guess is, whoever is the supplier of your kernel, did not
> do that. You should double-check the kernel configuration of your
> distribution.
>
> However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode -
> "regular" vfio should work fine. noiommu mode should only be needed if you
> know you don't have IOMMU enabled in your kernel, and even if you can't
> enable it, you can still use igb_uio.
>
> Hi Anatoly,

Do you suggest I take this discussion to kvm/qemu mailing list as I am not
sure which component has the issue? I check dmesg for BIOS physical memory
map and address reported as fault by DMAR is reported by BIOS as usable on
both host and vm.

[ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr *33a128000
*[fault reason 06] PTE Read access is not set

dmesg | grep BIOS
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff]
reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff]
reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] usable
[    0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff]
reserved
[    0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] ACPI
data
[    0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] ACPI
NVS
[    0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff]
reserved
[    0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff]
reserved
[    0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] usable
[    0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff]
reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff]
reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff]
reserved
[*    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff]
usable*

Kindly let me know your inputs.

Thanks.


-- 
> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-01 19:26               ` Ravi Kerur
@ 2018-02-02 10:28                 ` Burakov, Anatoly
  2018-02-02 20:21                   ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-02 10:28 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
> 
> 
> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
> 
> 
>         Hi Anatoly,
> 
>         Thanks. I am following wiki link below which uses vIOMMU with
>         DPDK as a use-case and instantiate VM as specified with Q35
>         chipset in Qemu.
> 
>         https://wiki.qemu.org/Features/VT-d
>         <https://wiki.qemu.org/Features/VT-d>
> 
>         Qemu-version is 2.11
>         Host kernel 4.9
>         Guest kernel 4.4
> 
>         I can only guess that guest kernel needs an upgrade in my setup
>         to work correctly, if versions on my setup rings a bell on not
>         having support kindly let me know.
> 
>         When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on
>         guest I get following error
>         ...
>         vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored
>         ...
> 
>         in guest.
> 
>         Thanks.
> 
> 
>     AFAIK kernel 4.4 should have noiommu mode - it was introduced in
>     3.1x days. However, in order for that to work, kernel also has to be
>     built with this mode enabled. My guess is, whoever is the supplier
>     of your kernel, did not do that. You should double-check the kernel
>     configuration of your distribution.
> 
>     However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode
>     - "regular" vfio should work fine. noiommu mode should only be
>     needed if you know you don't have IOMMU enabled in your kernel, and
>     even if you can't enable it, you can still use igb_uio.
> 
> Hi Anatoly,
> 
> Do you suggest I take this discussion to kvm/qemu mailing list as I am 
> not sure which component has the issue? I check dmesg for BIOS physical 
> memory map and address reported as fault by DMAR is reported by BIOS as 
> usable on both host and vm.
> 
> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> *33a128000 *[fault reason 06] PTE Read access is not set
> 
> dmesg | grep BIOS
> [    0.000000] e820: BIOS-provided physical RAM map:
> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff] usable
> [    0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff] usable
> [    0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff] 
> ACPI data
> [    0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff] 
> ACPI NVS
> [    0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff] usable
> [    0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff] usable
> [    0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] 
> reserved
> [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] 
> reserved
> [*    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff] 
> usable*
> *
> *
> Kindly let me know your inputs.
> 
> Thanks.
> 
> 
>     -- 
>     Thanks,
>     Anatoly
> 
> 

The "PTE Read not set" error usually indicates that you are trying to 
use a non-IOMMU method when you have IOMMU enabled (i.e. trying to use 
igb_uio when IOMMU is on). That, to me, indicates that you do have IOMMU 
emulation enabled.

I would go about it this way.

First, i'd ensure that your VM has IOMMU emulation enabled and working. 
You have mentioned that your QEMU version should have IOMMU emulation, 
so let's assume that's the case.

I am not sure of the exact command-line needed to activate the vIOMMU 
emulation, but assuming your VM emulates an Intel processor, your kernel 
command-line should have "iommu=on intel_iommu=on" in it. Check 
/etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above 
values are not in there, add the above changes, do "update-grub" and 
reboot your VM.

If it already did have the necessary kernel configuration, do "dmesg | 
grep IOMMU" and look for "IOMMU Enabled". That should tell you that 
IOMMU is enabled and working in the kernel.

After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it 
should be working. Please bear in mind that all of that is how i 
would've gone about it if i had similar problems on baremetal, but i'm 
hoping all of it is applicable to VM's. So, either disable IOMMU and use 
igb_uio, or enable IOMMU and use VFIO. Both will work.

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-02 10:28                 ` Burakov, Anatoly
@ 2018-02-02 20:21                   ` Ravi Kerur
  2018-02-02 20:51                     ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-02 20:21 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
>
>>
>>
>> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly <
>> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>     On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
>>
>>
>>         Hi Anatoly,
>>
>>         Thanks. I am following wiki link below which uses vIOMMU with
>>         DPDK as a use-case and instantiate VM as specified with Q35
>>         chipset in Qemu.
>>
>>         https://wiki.qemu.org/Features/VT-d
>>         <https://wiki.qemu.org/Features/VT-d>
>>
>>         Qemu-version is 2.11
>>         Host kernel 4.9
>>         Guest kernel 4.4
>>
>>         I can only guess that guest kernel needs an upgrade in my setup
>>         to work correctly, if versions on my setup rings a bell on not
>>         having support kindly let me know.
>>
>>         When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on
>>         guest I get following error
>>         ...
>>         vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored
>>         ...
>>
>>         in guest.
>>
>>         Thanks.
>>
>>
>>     AFAIK kernel 4.4 should have noiommu mode - it was introduced in
>>     3.1x days. However, in order for that to work, kernel also has to be
>>     built with this mode enabled. My guess is, whoever is the supplier
>>     of your kernel, did not do that. You should double-check the kernel
>>     configuration of your distribution.
>>
>>     However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode
>>     - "regular" vfio should work fine. noiommu mode should only be
>>     needed if you know you don't have IOMMU enabled in your kernel, and
>>     even if you can't enable it, you can still use igb_uio.
>>
>> Hi Anatoly,
>>
>> Do you suggest I take this discussion to kvm/qemu mailing list as I am
>> not sure which component has the issue? I check dmesg for BIOS physical
>> memory map and address reported as fault by DMAR is reported by BIOS as
>> usable on both host and vm.
>>
>> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> *33a128000 *[fault reason 06] PTE Read access is not set
>>
>> dmesg | grep BIOS
>> [    0.000000] e820: BIOS-provided physical RAM map:
>> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff]
>> usable
>> [    0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff]
>> usable
>> [    0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff]
>> ACPI data
>> [    0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff]
>> ACPI NVS
>> [    0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff]
>> usable
>> [    0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff]
>> usable
>> [    0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff]
>> reserved
>> [*    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff]
>> usable*
>> *
>> *
>> Kindly let me know your inputs.
>>
>> Thanks.
>>
>>
>>     --     Thanks,
>>     Anatoly
>>
>>
>>
> The "PTE Read not set" error usually indicates that you are trying to use
> a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio
> when IOMMU is on). That, to me, indicates that you do have IOMMU emulation
> enabled.
>
> I would go about it this way.
>
> First, i'd ensure that your VM has IOMMU emulation enabled and working.
> You have mentioned that your QEMU version should have IOMMU emulation, so
> let's assume that's the case.
>
> I am not sure of the exact command-line needed to activate the vIOMMU
> emulation, but assuming your VM emulates an Intel processor, your kernel
> command-line should have "iommu=on intel_iommu=on" in it. Check
> /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above
> values are not in there, add the above changes, do "update-grub" and reboot
> your VM.
>
> If it already did have the necessary kernel configuration, do "dmesg |
> grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU
> is enabled and working in the kernel.
>
> After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it
> should be working. Please bear in mind that all of that is how i would've
> gone about it if i had similar problems on baremetal, but i'm hoping all of
> it is applicable to VM's. So, either disable IOMMU and use igb_uio, or
> enable IOMMU and use VFIO. Both will work.
>
>
Thanks for your help so far on this. I have everything enabled as you have
mentioned in your email. Below is the error from EAL when DPDK is run on a
VM. I am referring EAL code to check why it is failing.

************VM DPDK errors***********
EAL: Detected 14 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable
clock cycles !
EAL: PCI device 0000:00:07.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10ed net_ixgbe_vf
EAL:   using IOMMU type 1 (Type 1)


*EAL:   RTE_IOVA_VA 140051367329792EAL:   cannot set up DMA remapping,
error 14 (Bad address)EAL:   0000:00:07.0 DMA remapping failed, error 14
(Bad address)*
EAL: Requested device 0000:00:07.0 cannot be used
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
EAL: No probed ethernet devices
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176,
socket=0


********************VM************
root@sriov-ravi-01:~# lsmod | grep vfio




*vfio_pci               40960  1vfio_virqfd            16384  1
vfio_pcivfio_iommu_type1       20480  1vfio                   28672  5
vfio_iommu_type1,vfio_pciirqbypass              16384  3 kvm,vfio_pci*
root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR
00000001 BXPC 00000001)
*[    0.000000] DMAR: IOMMU enabled*
[    9.423839] DMAR: Host address width 39
[    9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
12008c22260286 ecap f00f5e
[    9.427314] DMAR: ATSR flags: 0x1
[    9.428047] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU 0
[    9.429255] DMAR-IR: Queued invalidation will be enabled to support
x2apic and Intr-remapping.
[    9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode
[   11.167334] DMAR: No RMRR found
[   11.168855] DMAR: dmar0: Using Queued invalidation
[   12.119060] DMAR: Setting RMRR:
[   12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC
[   12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
0xffffff]
[   12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O
root@sriov-ravi-01:~# dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
*0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed'
drv=vfio-pci unused=*

Network devices using kernel driver
===================================
0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci

Other Network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other Crypto devices
====================
<none>

Eventdev devices using DPDK-compatible driver
=============================================
<none>

Eventdev devices using kernel driver
====================================
<none>

Other Eventdev devices
======================
<none>

Mempool devices using DPDK-compatible driver
============================================
<none>

Mempool devices using kernel driver
===================================
<none>

Other Mempool devices
=====================
<none>
root@sriov-ravi-01:~#

root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i
vfio
[   77.676711] VFIO - User Level meta-driver version: 0.3
[  149.013806] VFIO - User Level meta-driver version: 0.3
[  275.876440] VFIO - User Level meta-driver version: 0.3

Thanks,
Ravi


-- 
> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-02 20:21                   ` Ravi Kerur
@ 2018-02-02 20:51                     ` Ravi Kerur
  2018-02-05 10:01                       ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-02 20:51 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rkerur@gmail.com> wrote:

>
>
> On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly <
> anatoly.burakov@intel.com> wrote:
>
>> On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
>>
>>>
>>>
>>> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly <
>>> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>>
>>>     On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
>>>
>>>
>>>         Hi Anatoly,
>>>
>>>         Thanks. I am following wiki link below which uses vIOMMU with
>>>         DPDK as a use-case and instantiate VM as specified with Q35
>>>         chipset in Qemu.
>>>
>>>         https://wiki.qemu.org/Features/VT-d
>>>         <https://wiki.qemu.org/Features/VT-d>
>>>
>>>         Qemu-version is 2.11
>>>         Host kernel 4.9
>>>         Guest kernel 4.4
>>>
>>>         I can only guess that guest kernel needs an upgrade in my setup
>>>         to work correctly, if versions on my setup rings a bell on not
>>>         having support kindly let me know.
>>>
>>>         When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on
>>>         guest I get following error
>>>         ...
>>>         vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored
>>>         ...
>>>
>>>         in guest.
>>>
>>>         Thanks.
>>>
>>>
>>>     AFAIK kernel 4.4 should have noiommu mode - it was introduced in
>>>     3.1x days. However, in order for that to work, kernel also has to be
>>>     built with this mode enabled. My guess is, whoever is the supplier
>>>     of your kernel, did not do that. You should double-check the kernel
>>>     configuration of your distribution.
>>>
>>>     However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode
>>>     - "regular" vfio should work fine. noiommu mode should only be
>>>     needed if you know you don't have IOMMU enabled in your kernel, and
>>>     even if you can't enable it, you can still use igb_uio.
>>>
>>> Hi Anatoly,
>>>
>>> Do you suggest I take this discussion to kvm/qemu mailing list as I am
>>> not sure which component has the issue? I check dmesg for BIOS physical
>>> memory map and address reported as fault by DMAR is reported by BIOS as
>>> usable on both host and vm.
>>>
>>> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr
>>> *33a128000 *[fault reason 06] PTE Read access is not set
>>>
>>> dmesg | grep BIOS
>>> [    0.000000] e820: BIOS-provided physical RAM map:
>>> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff]
>>> usable
>>> [    0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff]
>>> usable
>>> [    0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff]
>>> ACPI data
>>> [    0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff]
>>> ACPI NVS
>>> [    0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff]
>>> usable
>>> [    0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff]
>>> usable
>>> [    0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff]
>>> reserved
>>> [*    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff]
>>> usable*
>>> *
>>> *
>>> Kindly let me know your inputs.
>>>
>>> Thanks.
>>>
>>>
>>>     --     Thanks,
>>>     Anatoly
>>>
>>>
>>>
>> The "PTE Read not set" error usually indicates that you are trying to use
>> a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio
>> when IOMMU is on). That, to me, indicates that you do have IOMMU emulation
>> enabled.
>>
>> I would go about it this way.
>>
>> First, i'd ensure that your VM has IOMMU emulation enabled and working.
>> You have mentioned that your QEMU version should have IOMMU emulation, so
>> let's assume that's the case.
>>
>> I am not sure of the exact command-line needed to activate the vIOMMU
>> emulation, but assuming your VM emulates an Intel processor, your kernel
>> command-line should have "iommu=on intel_iommu=on" in it. Check
>> /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above
>> values are not in there, add the above changes, do "update-grub" and reboot
>> your VM.
>>
>> If it already did have the necessary kernel configuration, do "dmesg |
>> grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU
>> is enabled and working in the kernel.
>>
>> After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it
>> should be working. Please bear in mind that all of that is how i would've
>> gone about it if i had similar problems on baremetal, but i'm hoping all of
>> it is applicable to VM's. So, either disable IOMMU and use igb_uio, or
>> enable IOMMU and use VFIO. Both will work.
>>
>>
> Thanks for your help so far on this. I have everything enabled as you have
> mentioned in your email. Below is the error from EAL when DPDK is run on a
> VM. I am referring EAL code to check why it is failing.
>
> ************VM DPDK errors***********
> EAL: Detected 14 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> unreliable clock cycles !
> EAL: PCI device 0000:00:07.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 8086:10ed net_ixgbe_vf
> EAL:   using IOMMU type 1 (Type 1)
>
>
> *EAL:   RTE_IOVA_VA 140051367329792EAL:   cannot set up DMA remapping,
> error 14 (Bad address)EAL:   0000:00:07.0 DMA remapping failed, error 14
> (Bad address)*
> EAL: Requested device 0000:00:07.0 cannot be used
> EAL: PCI device 0000:03:00.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 1af4:1041 net_virtio
> EAL: PCI device 0000:04:00.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 1af4:1041 net_virtio
> EAL: PCI device 0000:05:00.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 1af4:1041 net_virtio
> EAL: No probed ethernet devices
> Interactive-mode selected
> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176,
> socket=0
>
>
> ********************VM************
> root@sriov-ravi-01:~# lsmod | grep vfio
>
>
>
>
> *vfio_pci               40960  1vfio_virqfd            16384  1
> vfio_pcivfio_iommu_type1       20480  1vfio                   28672  5
> vfio_iommu_type1,vfio_pciirqbypass              16384  3 kvm,vfio_pci*
> root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU
> [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR
> 00000001 BXPC 00000001)
> *[    0.000000] DMAR: IOMMU enabled*
> [    9.423839] DMAR: Host address width 39
> [    9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
> [    9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
> 12008c22260286 ecap f00f5e
> [    9.427314] DMAR: ATSR flags: 0x1
> [    9.428047] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU 0
> [    9.429255] DMAR-IR: Queued invalidation will be enabled to support
> x2apic and Intr-remapping.
> [    9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [   11.167334] DMAR: No RMRR found
> [   11.168855] DMAR: dmar0: Using Queued invalidation
> [   12.119060] DMAR: Setting RMRR:
> [   12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC
> [   12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
> 0xffffff]
> [   12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O
> root@sriov-ravi-01:~# dpdk-devbind -s
>
> Network devices using DPDK-compatible driver
> ============================================
> *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed'
> drv=vfio-pci unused=*
>
> Network devices using kernel driver
> ===================================
> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
>
> Other Network devices
> =====================
> <none>
>
> Crypto devices using DPDK-compatible driver
> ===========================================
> <none>
>
> Crypto devices using kernel driver
> ==================================
> <none>
>
> Other Crypto devices
> ====================
> <none>
>
> Eventdev devices using DPDK-compatible driver
> =============================================
> <none>
>
> Eventdev devices using kernel driver
> ====================================
> <none>
>
> Other Eventdev devices
> ======================
> <none>
>
> Mempool devices using DPDK-compatible driver
> ============================================
> <none>
>
> Mempool devices using kernel driver
> ===================================
> <none>
>
> Other Mempool devices
> =====================
> <none>
> root@sriov-ravi-01:~#
>
> root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i
> vfio
> [   77.676711] VFIO - User Level meta-driver version: 0.3
> [  149.013806] VFIO - User Level meta-driver version: 0.3
> [  275.876440] VFIO - User Level meta-driver version: 0.3
>
> Thanks,
> Ravi
>
>
>
Additonal information. On VM following message is seen

[  298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient
for the mapped address (7f7300000000)
[ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient
for the mapped address (7fa580000000)

ON VM, DMAR reports address width = 39
root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -e
IOMMU -e DMAR
[    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR
00000001 BXPC 00000001)
[    0.000000] DMAR: IOMMU enabled
[    9.209145] DMAR: Host address width 39

On Host, DMAR reports address width = 46
# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
00000001 INTL 20091013)
[    0.000000] DMAR: IOMMU enabled
[    0.519368] DMAR: Host address width 46
[    0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0

Searching for IOMMU errors found following link from RedHat which points to
DPDK 17.11 issue.

https://bugzilla.redhat.com/show_bug.cgi?id=1530957

Thanks.

> --
>> Thanks,
>> Anatoly
>>
>
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-02 20:51                     ` Ravi Kerur
@ 2018-02-05 10:01                       ` Burakov, Anatoly
  2018-02-06 17:55                         ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-05 10:01 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 02-Feb-18 8:51 PM, Ravi Kerur wrote:
> 
> 
> On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rkerur@gmail.com 
> <mailto:rkerur@gmail.com>> wrote:
> 
> 
> 
>     On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly
>     <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>         On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
> 
> 
> 
>             On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly
>             <anatoly.burakov@intel.com
>             <mailto:anatoly.burakov@intel.com>
>             <mailto:anatoly.burakov@intel.com
>             <mailto:anatoly.burakov@intel.com>>> wrote:
> 
>                  On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
> 
> 
>                      Hi Anatoly,
> 
>                      Thanks. I am following wiki link below which uses
>             vIOMMU with
>                      DPDK as a use-case and instantiate VM as specified
>             with Q35
>                      chipset in Qemu.
> 
>             https://wiki.qemu.org/Features/VT-d
>             <https://wiki.qemu.org/Features/VT-d>
>                      <https://wiki.qemu.org/Features/VT-d
>             <https://wiki.qemu.org/Features/VT-d>>
> 
>                      Qemu-version is 2.11
>                      Host kernel 4.9
>                      Guest kernel 4.4
> 
>                      I can only guess that guest kernel needs an upgrade
>             in my setup
>                      to work correctly, if versions on my setup rings a
>             bell on not
>                      having support kindly let me know.
> 
>                      When 'modprobe vfio enable_unsafe_noiommu_node=Y'
>             is executed on
>                      guest I get following error
>                      ...
>                      vfio: unknown parameter
>             'enable_unsafe_noiommu_node' ignored
>                      ...
> 
>                      in guest.
> 
>                      Thanks.
> 
> 
>                  AFAIK kernel 4.4 should have noiommu mode - it was
>             introduced in
>                  3.1x days. However, in order for that to work, kernel
>             also has to be
>                  built with this mode enabled. My guess is, whoever is
>             the supplier
>                  of your kernel, did not do that. You should
>             double-check the kernel
>                  configuration of your distribution.
> 
>                  However, if you have vIOMMU in QEMU, you shouldn't need
>             noiommu mode
>                  - "regular" vfio should work fine. noiommu mode should
>             only be
>                  needed if you know you don't have IOMMU enabled in your
>             kernel, and
>                  even if you can't enable it, you can still use igb_uio.
> 
>             Hi Anatoly,
> 
>             Do you suggest I take this discussion to kvm/qemu mailing
>             list as I am not sure which component has the issue? I check
>             dmesg for BIOS physical memory map and address reported as
>             fault by DMAR is reported by BIOS as usable on both host and vm.
> 
>             [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0]
>             fault addr *33a128000 *[fault reason 06] PTE Read access is
>             not set
> 
>             dmesg | grep BIOS
>             [    0.000000] e820: BIOS-provided physical RAM map:
>             [    0.000000] BIOS-e820: [mem
>             0x0000000000000000-0x000000000009afff] usable
>             [    0.000000] BIOS-e820: [mem
>             0x000000000009b000-0x000000000009ffff] reserved
>             [    0.000000] BIOS-e820: [mem
>             0x00000000000e0000-0x00000000000fffff] reserved
>             [    0.000000] BIOS-e820: [mem
>             0x0000000000100000-0x000000007938afff] usable
>             [    0.000000] BIOS-e820: [mem
>             0x000000007938b000-0x000000007994bfff] reserved
>             [    0.000000] BIOS-e820: [mem
>             0x000000007994c000-0x000000007999cfff] ACPI data
>             [    0.000000] BIOS-e820: [mem
>             0x000000007999d000-0x0000000079f7dfff] ACPI NVS
>             [    0.000000] BIOS-e820: [mem
>             0x0000000079f7e000-0x000000007bd37fff] reserved
>             [    0.000000] BIOS-e820: [mem
>             0x000000007bd38000-0x000000007bd38fff] usable
>             [    0.000000] BIOS-e820: [mem
>             0x000000007bd39000-0x000000007bdbefff] reserved
>             [    0.000000] BIOS-e820: [mem
>             0x000000007bdbf000-0x000000007bffffff] usable
>             [    0.000000] BIOS-e820: [mem
>             0x000000007c000000-0x000000008fffffff] reserved
>             [    0.000000] BIOS-e820: [mem
>             0x00000000fed1c000-0x00000000fed44fff] reserved
>             [    0.000000] BIOS-e820: [mem
>             0x00000000ff000000-0x00000000ffffffff] reserved
>             [*    0.000000] BIOS-e820: [mem
>             0x0000000100000000-0x000000407fffffff] usable*
>             *
>             *
>             Kindly let me know your inputs.
> 
>             Thanks.
> 
> 
>                  --     Thanks,
>                  Anatoly
> 
> 
> 
>         The "PTE Read not set" error usually indicates that you are
>         trying to use a non-IOMMU method when you have IOMMU enabled
>         (i.e. trying to use igb_uio when IOMMU is on). That, to me,
>         indicates that you do have IOMMU emulation enabled.
> 
>         I would go about it this way.
> 
>         First, i'd ensure that your VM has IOMMU emulation enabled and
>         working. You have mentioned that your QEMU version should have
>         IOMMU emulation, so let's assume that's the case.
> 
>         I am not sure of the exact command-line needed to activate the
>         vIOMMU emulation, but assuming your VM emulates an Intel
>         processor, your kernel command-line should have "iommu=on
>         intel_iommu=on" in it. Check /etc/default/grub for
>         GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are
>         not in there, add the above changes, do "update-grub" and reboot
>         your VM.
> 
>         If it already did have the necessary kernel configuration, do
>         "dmesg | grep IOMMU" and look for "IOMMU Enabled". That should
>         tell you that IOMMU is enabled and working in the kernel.
> 
>         After that, you can modprobe vfio and vfio-pci, bind NICs to it,
>         and it should be working. Please bear in mind that all of that
>         is how i would've gone about it if i had similar problems on
>         baremetal, but i'm hoping all of it is applicable to VM's. So,
>         either disable IOMMU and use igb_uio, or enable IOMMU and use
>         VFIO. Both will work.
> 
> 
>     Thanks for your help so far on this. I have everything enabled as
>     you have mentioned in your email. Below is the error from EAL when
>     DPDK is run on a VM. I am referring EAL code to check why it is failing.
> 
>     ************VM DPDK errors***********
>     EAL: Detected 14 lcore(s)
>     EAL: Probing VFIO support...
>     EAL: VFIO support initialized
>     EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
>     unreliable clock cycles !
>     EAL: PCI device 0000:00:07.0 on NUMA socket -1
>     EAL:   Invalid NUMA socket, default to 0
>     EAL:   probe driver: 8086:10ed net_ixgbe_vf
>     EAL:   using IOMMU type 1 (Type 1)
>     *EAL:   RTE_IOVA_VA 140051367329792
>     EAL:   cannot set up DMA remapping, error 14 (Bad address)
>     EAL:   0000:00:07.0 DMA remapping failed, error 14 (Bad address)*
>     EAL: Requested device 0000:00:07.0 cannot be used
>     EAL: PCI device 0000:03:00.0 on NUMA socket -1
>     EAL:   Invalid NUMA socket, default to 0
>     EAL:   probe driver: 1af4:1041 net_virtio
>     EAL: PCI device 0000:04:00.0 on NUMA socket -1
>     EAL:   Invalid NUMA socket, default to 0
>     EAL:   probe driver: 1af4:1041 net_virtio
>     EAL: PCI device 0000:05:00.0 on NUMA socket -1
>     EAL:   Invalid NUMA socket, default to 0
>     EAL:   probe driver: 1af4:1041 net_virtio
>     EAL: No probed ethernet devices
>     Interactive-mode selected
>     USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
>     size=2176, socket=0
> 
> 
>     ********************VM************
>     root@sriov-ravi-01:~# lsmod | grep vfio
>     *vfio_pci               40960  1
>     vfio_virqfd            16384  1 vfio_pci
>     vfio_iommu_type1       20480  1
>     vfio                   28672  5 vfio_iommu_type1,vfio_pci
>     irqbypass              16384  3 kvm,vfio_pci*
>     root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU
>     [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS 
>     BXPCDMAR 00000001 BXPC 00000001)
>     *[    0.000000] DMAR: IOMMU enabled*
>     [    9.423839] DMAR: Host address width 39
>     [    9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
>     [    9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
>     12008c22260286 ecap f00f5e
>     [    9.427314] DMAR: ATSR flags: 0x1
>     [    9.428047] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU 0
>     [    9.429255] DMAR-IR: Queued invalidation will be enabled to
>     support x2apic and Intr-remapping.
>     [    9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode
>     [   11.167334] DMAR: No RMRR found
>     [   11.168855] DMAR: dmar0: Using Queued invalidation
>     [   12.119060] DMAR: Setting RMRR:
>     [   12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC
>     [   12.124787] DMAR: Setting identity map for device 0000:00:1f.0
>     [0x0 - 0xffffff]
>     [   12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O
>     root@sriov-ravi-01:~# dpdk-devbind -s
> 
>     Network devices using DPDK-compatible driver
>     ============================================
>     *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed'
>     drv=vfio-pci unused=*
> 
>     Network devices using kernel driver
>     ===================================
>     0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci
>     *Active*
>     0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
>     0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
> 
>     Other Network devices
>     =====================
>     <none>
> 
>     Crypto devices using DPDK-compatible driver
>     ===========================================
>     <none>
> 
>     Crypto devices using kernel driver
>     ==================================
>     <none>
> 
>     Other Crypto devices
>     ====================
>     <none>
> 
>     Eventdev devices using DPDK-compatible driver
>     =============================================
>     <none>
> 
>     Eventdev devices using kernel driver
>     ====================================
>     <none>
> 
>     Other Eventdev devices
>     ======================
>     <none>
> 
>     Mempool devices using DPDK-compatible driver
>     ============================================
>     <none>
> 
>     Mempool devices using kernel driver
>     ===================================
>     <none>
> 
>     Other Mempool devices
>     =====================
>     <none>
>     root@sriov-ravi-01:~#
> 
>     root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg |
>     grep -i vfio
>     [   77.676711] VFIO - User Level meta-driver version: 0.3
>     [  149.013806] VFIO - User Level meta-driver version: 0.3
>     [  275.876440] VFIO - User Level meta-driver version: 0.3
> 
>     Thanks,
>     Ravi
> 
> 
> 
> Additonal information. On VM following message is seen
> 
> [  298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient 
> for the mapped address (7f7300000000)
> [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient 
> for the mapped address (7fa580000000)
> 
> ON VM, DMAR reports address width = 39
> root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep 
> -e IOMMU -e DMAR
> [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR 
> 00000001 BXPC 00000001)
> [    0.000000] DMAR: IOMMU enabled
> [    9.209145] DMAR: Host address width 39
> 
> On Host, DMAR reports address width = 46
> # dmesg | grep -e DMAR -e IOMMU
> [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I    
> 00000001 INTL 20091013)
> [    0.000000] DMAR: IOMMU enabled
> [    0.519368] DMAR: Host address width 46
> [    0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
> 
> Searching for IOMMU errors found following link from RedHat which points 
> to DPDK 17.11 issue.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1530957
> 
> Thanks.
> 
>         -- 
>         Thanks,
>         Anatoly
> 
> 
> 

Ah, yes, that. We have recently applied a patch dealing with exactly this:

http://dpdk.org/dev/patchwork/patch/33650/

Unfortunately, as far as i know, other than refusing to run (like the 
patch does), there's no way to fix this issue, so you'll have to disable 
IOMMU in your VM, and use either igb_uio or vfio in noiommu mode.

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-05 10:01                       ` Burakov, Anatoly
@ 2018-02-06 17:55                         ` Ravi Kerur
  2018-02-08 11:20                           ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-06 17:55 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Mon, Feb 5, 2018 at 2:01 AM, Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 02-Feb-18 8:51 PM, Ravi Kerur wrote:
>
>>
>>
>> On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rkerur@gmail.com <mailto:
>> rkerur@gmail.com>> wrote:
>>
>>
>>
>>     On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly
>>     <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>         On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
>>
>>
>>
>>             On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly
>>             <anatoly.burakov@intel.com
>>             <mailto:anatoly.burakov@intel.com>
>>             <mailto:anatoly.burakov@intel.com
>>
>>             <mailto:anatoly.burakov@intel.com>>> wrote:
>>
>>                  On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
>>
>>
>>                      Hi Anatoly,
>>
>>                      Thanks. I am following wiki link below which uses
>>             vIOMMU with
>>                      DPDK as a use-case and instantiate VM as specified
>>             with Q35
>>                      chipset in Qemu.
>>
>>             https://wiki.qemu.org/Features/VT-d
>>             <https://wiki.qemu.org/Features/VT-d>
>>                      <https://wiki.qemu.org/Features/VT-d
>>             <https://wiki.qemu.org/Features/VT-d>>
>>
>>                      Qemu-version is 2.11
>>                      Host kernel 4.9
>>                      Guest kernel 4.4
>>
>>                      I can only guess that guest kernel needs an upgrade
>>             in my setup
>>                      to work correctly, if versions on my setup rings a
>>             bell on not
>>                      having support kindly let me know.
>>
>>                      When 'modprobe vfio enable_unsafe_noiommu_node=Y'
>>             is executed on
>>                      guest I get following error
>>                      ...
>>                      vfio: unknown parameter
>>             'enable_unsafe_noiommu_node' ignored
>>                      ...
>>
>>                      in guest.
>>
>>                      Thanks.
>>
>>
>>                  AFAIK kernel 4.4 should have noiommu mode - it was
>>             introduced in
>>                  3.1x days. However, in order for that to work, kernel
>>             also has to be
>>                  built with this mode enabled. My guess is, whoever is
>>             the supplier
>>                  of your kernel, did not do that. You should
>>             double-check the kernel
>>                  configuration of your distribution.
>>
>>                  However, if you have vIOMMU in QEMU, you shouldn't need
>>             noiommu mode
>>                  - "regular" vfio should work fine. noiommu mode should
>>             only be
>>                  needed if you know you don't have IOMMU enabled in your
>>             kernel, and
>>                  even if you can't enable it, you can still use igb_uio.
>>
>>             Hi Anatoly,
>>
>>             Do you suggest I take this discussion to kvm/qemu mailing
>>             list as I am not sure which component has the issue? I check
>>             dmesg for BIOS physical memory map and address reported as
>>             fault by DMAR is reported by BIOS as usable on both host and
>> vm.
>>
>>             [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0]
>>             fault addr *33a128000 *[fault reason 06] PTE Read access is
>>             not set
>>
>>             dmesg | grep BIOS
>>             [    0.000000] e820: BIOS-provided physical RAM map:
>>             [    0.000000] BIOS-e820: [mem
>>             0x0000000000000000-0x000000000009afff] usable
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000000009b000-0x000000000009ffff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x00000000000e0000-0x00000000000fffff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x0000000000100000-0x000000007938afff] usable
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007938b000-0x000000007994bfff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007994c000-0x000000007999cfff] ACPI data
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007999d000-0x0000000079f7dfff] ACPI NVS
>>             [    0.000000] BIOS-e820: [mem
>>             0x0000000079f7e000-0x000000007bd37fff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007bd38000-0x000000007bd38fff] usable
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007bd39000-0x000000007bdbefff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007bdbf000-0x000000007bffffff] usable
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007c000000-0x000000008fffffff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x00000000fed1c000-0x00000000fed44fff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x00000000ff000000-0x00000000ffffffff] reserved
>>             [*    0.000000] BIOS-e820: [mem
>>             0x0000000100000000-0x000000407fffffff] usable*
>>             *
>>             *
>>             Kindly let me know your inputs.
>>
>>             Thanks.
>>
>>
>>                  --     Thanks,
>>                  Anatoly
>>
>>
>>
>>         The "PTE Read not set" error usually indicates that you are
>>         trying to use a non-IOMMU method when you have IOMMU enabled
>>         (i.e. trying to use igb_uio when IOMMU is on). That, to me,
>>         indicates that you do have IOMMU emulation enabled.
>>
>>         I would go about it this way.
>>
>>         First, i'd ensure that your VM has IOMMU emulation enabled and
>>         working. You have mentioned that your QEMU version should have
>>         IOMMU emulation, so let's assume that's the case.
>>
>>         I am not sure of the exact command-line needed to activate the
>>         vIOMMU emulation, but assuming your VM emulates an Intel
>>         processor, your kernel command-line should have "iommu=on
>>         intel_iommu=on" in it. Check /etc/default/grub for
>>         GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are
>>         not in there, add the above changes, do "update-grub" and reboot
>>         your VM.
>>
>>         If it already did have the necessary kernel configuration, do
>>         "dmesg | grep IOMMU" and look for "IOMMU Enabled". That should
>>         tell you that IOMMU is enabled and working in the kernel.
>>
>>         After that, you can modprobe vfio and vfio-pci, bind NICs to it,
>>         and it should be working. Please bear in mind that all of that
>>         is how i would've gone about it if i had similar problems on
>>         baremetal, but i'm hoping all of it is applicable to VM's. So,
>>         either disable IOMMU and use igb_uio, or enable IOMMU and use
>>         VFIO. Both will work.
>>
>>
>>     Thanks for your help so far on this. I have everything enabled as
>>     you have mentioned in your email. Below is the error from EAL when
>>     DPDK is run on a VM. I am referring EAL code to check why it is
>> failing.
>>
>>     ************VM DPDK errors***********
>>     EAL: Detected 14 lcore(s)
>>     EAL: Probing VFIO support...
>>     EAL: VFIO support initialized
>>     EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
>>     unreliable clock cycles !
>>     EAL: PCI device 0000:00:07.0 on NUMA socket -1
>>     EAL:   Invalid NUMA socket, default to 0
>>     EAL:   probe driver: 8086:10ed net_ixgbe_vf
>>     EAL:   using IOMMU type 1 (Type 1)
>>     *EAL:   RTE_IOVA_VA 140051367329792
>>     EAL:   cannot set up DMA remapping, error 14 (Bad address)
>>     EAL:   0000:00:07.0 DMA remapping failed, error 14 (Bad address)*
>>     EAL: Requested device 0000:00:07.0 cannot be used
>>     EAL: PCI device 0000:03:00.0 on NUMA socket -1
>>     EAL:   Invalid NUMA socket, default to 0
>>     EAL:   probe driver: 1af4:1041 net_virtio
>>     EAL: PCI device 0000:04:00.0 on NUMA socket -1
>>     EAL:   Invalid NUMA socket, default to 0
>>     EAL:   probe driver: 1af4:1041 net_virtio
>>     EAL: PCI device 0000:05:00.0 on NUMA socket -1
>>     EAL:   Invalid NUMA socket, default to 0
>>     EAL:   probe driver: 1af4:1041 net_virtio
>>     EAL: No probed ethernet devices
>>     Interactive-mode selected
>>     USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
>>     size=2176, socket=0
>>
>>
>>     ********************VM************
>>     root@sriov-ravi-01:~# lsmod | grep vfio
>>     *vfio_pci               40960  1
>>     vfio_virqfd            16384  1 vfio_pci
>>     vfio_iommu_type1       20480  1
>>     vfio                   28672  5 vfio_iommu_type1,vfio_pci
>>     irqbypass              16384  3 kvm,vfio_pci*
>>     root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU
>>     [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS
>>  BXPCDMAR 00000001 BXPC 00000001)
>>     *[    0.000000] DMAR: IOMMU enabled*
>>     [    9.423839] DMAR: Host address width 39
>>     [    9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
>>     [    9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
>>     12008c22260286 ecap f00f5e
>>     [    9.427314] DMAR: ATSR flags: 0x1
>>     [    9.428047] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU
>> 0
>>     [    9.429255] DMAR-IR: Queued invalidation will be enabled to
>>     support x2apic and Intr-remapping.
>>     [    9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode
>>     [   11.167334] DMAR: No RMRR found
>>     [   11.168855] DMAR: dmar0: Using Queued invalidation
>>     [   12.119060] DMAR: Setting RMRR:
>>     [   12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC
>>     [   12.124787] DMAR: Setting identity map for device 0000:00:1f.0
>>     [0x0 - 0xffffff]
>>     [   12.392826] DMAR: Intel(R) Virtualization Technology for Directed
>> I/O
>>     root@sriov-ravi-01:~# dpdk-devbind -s
>>
>>     Network devices using DPDK-compatible driver
>>     ============================================
>>     *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed'
>>     drv=vfio-pci unused=*
>>
>>
>>     Network devices using kernel driver
>>     ===================================
>>     0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci
>>     *Active*
>>     0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
>>     0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
>>
>>     Other Network devices
>>     =====================
>>     <none>
>>
>>     Crypto devices using DPDK-compatible driver
>>     ===========================================
>>     <none>
>>
>>     Crypto devices using kernel driver
>>     ==================================
>>     <none>
>>
>>     Other Crypto devices
>>     ====================
>>     <none>
>>
>>     Eventdev devices using DPDK-compatible driver
>>     =============================================
>>     <none>
>>
>>     Eventdev devices using kernel driver
>>     ====================================
>>     <none>
>>
>>     Other Eventdev devices
>>     ======================
>>     <none>
>>
>>     Mempool devices using DPDK-compatible driver
>>     ============================================
>>     <none>
>>
>>     Mempool devices using kernel driver
>>     ===================================
>>     <none>
>>
>>     Other Mempool devices
>>     =====================
>>     <none>
>>     root@sriov-ravi-01:~#
>>
>>     root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg |
>>     grep -i vfio
>>     [   77.676711] VFIO - User Level meta-driver version: 0.3
>>     [  149.013806] VFIO - User Level meta-driver version: 0.3
>>     [  275.876440] VFIO - User Level meta-driver version: 0.3
>>
>>     Thanks,
>>     Ravi
>>
>>
>>
>> Additonal information. On VM following message is seen
>>
>> [  298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient
>> for the mapped address (7f7300000000)
>> [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient
>> for the mapped address (7fa580000000)
>>
>> ON VM, DMAR reports address width = 39
>> root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep
>> -e IOMMU -e DMAR
>> [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR
>> 00000001 BXPC 00000001)
>> [    0.000000] DMAR: IOMMU enabled
>> [    9.209145] DMAR: Host address width 39
>>
>> On Host, DMAR reports address width = 46
>> # dmesg | grep -e DMAR -e IOMMU
>> [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
>> 00000001 INTL 20091013)
>> [    0.000000] DMAR: IOMMU enabled
>> [    0.519368] DMAR: Host address width 46
>> [    0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>>
>> Searching for IOMMU errors found following link from RedHat which points
>> to DPDK 17.11 issue.
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1530957
>>
>> Thanks.
>>
>>         --         Thanks,
>>         Anatoly
>>
>>
>>
>>
> Ah, yes, that. We have recently applied a patch dealing with exactly this:
>
> http://dpdk.org/dev/patchwork/patch/33650/
>
> Unfortunately, as far as i know, other than refusing to run (like the
> patch does), there's no way to fix this issue, so you'll have to disable
> IOMMU in your VM, and use either igb_uio or vfio in noiommu mode.
>
>
Hi Anatoly,

I am actually confused with the state of vIOMMU + DPDK. Can you please help
me clarify?

I tested following DPDK versions

(1) DPDK 17.11, exhibits the issue (IOMMU width as reported by RedHat and
solution is to prevent using the patch)
(2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to bind a
device in VM with VFIO driver and no DMAR error message on host
(3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to bind a
device in VM with VFIO driver and no DMAR error message on host

Clearly issue seen in 17.11 without the patch you mentioned is a regression
or the issue was masked in earlier DPDK version?  I did not test traffic
with any DPDK version because I wanted to first get DMAR errors on host
gone.

Our application 'v1' is integrated with DPDK 16.11 and 'v2' is integrated
with DPDK 17.05.01. In both 'v1' and 'v2' cases I don't see IOMMU width
error messages on VM, however, DMAR error messages are seen host. I am not
able to relate what causes DMAR error messages on host?


Thanks.

> --
> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-06 17:55                         ` Ravi Kerur
@ 2018-02-08 11:20                           ` Burakov, Anatoly
  2018-02-09 17:41                             ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-08 11:20 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 06-Feb-18 5:55 PM, Ravi Kerur wrote:
> 
> Hi Anatoly,
> 
> I am actually confused with the state of vIOMMU + DPDK. Can you please 
> help me clarify?
> 
> I tested following DPDK versions
> 
> (1) DPDK 17.11, exhibits the issue (IOMMU width as reported by RedHat 
> and solution is to prevent using the patch)
> (2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to bind a 
> device in VM with VFIO driver and no DMAR error message on host
> (3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to bind a 
> device in VM with VFIO driver and no DMAR error message on host
> 
> Clearly issue seen in 17.11 without the patch you mentioned is a 
> regression or the issue was masked in earlier DPDK version?  I did not 
> test traffic with any DPDK version because I wanted to first get DMAR 
> errors on host gone.
> 
> Our application 'v1' is integrated with DPDK 16.11 and 'v2' is 
> integrated with DPDK 17.05.01. In both 'v1' and 'v2' cases I don't see 
> IOMMU width error messages on VM, however, DMAR error messages are seen 
> host. I am not able to relate what causes DMAR error messages on host?
> 
> 
> Thanks.
> 

Hi Ravi,

vIOMMU support is out of our hands, really - we can only make use of 
hardware (or emulation of it) that is available. 39-bit wide address 
*can* work, you just have to be lucky and get PA addresses that would 
fit into 39 bits (so under 512G limit), because we set up IOVA addresses 
to be 1:1 to physical addresses. We could, in principle, set up IOVA 
addresses to go from zero instead of them being a 1:1 mapping to 
physical addresses, but that would introduce need to translate addresses 
between IOVA and physical in some cases (e.g. KNI).

I'm not aware of any changes between 16.11 and 17.11 (and indeed 18.02) 
that would make or break support for 39-bit wide PA addresses for IOMMU. 
It is possible that VF/PF drivers do something differently which results 
in DMAR errors showing up sooner rather than later, but as far as VFIO 
support itself is concerned, there were no major changes in those releases.

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-08 11:20                           ` Burakov, Anatoly
@ 2018-02-09 17:41                             ` Ravi Kerur
  2018-02-10 10:11                               ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-09 17:41 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Thu, Feb 8, 2018 at 3:20 AM, Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 06-Feb-18 5:55 PM, Ravi Kerur wrote:
>
>>
>> Hi Anatoly,
>>
>> I am actually confused with the state of vIOMMU + DPDK. Can you please
>> help me clarify?
>>
>> I tested following DPDK versions
>>
>> (1) DPDK 17.11, exhibits the issue (IOMMU width as reported by RedHat and
>> solution is to prevent using the patch)
>> (2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to bind a
>> device in VM with VFIO driver and no DMAR error message on host
>> (3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to bind a
>> device in VM with VFIO driver and no DMAR error message on host
>>
>> Clearly issue seen in 17.11 without the patch you mentioned is a
>> regression or the issue was masked in earlier DPDK version?  I did not test
>> traffic with any DPDK version because I wanted to first get DMAR errors on
>> host gone.
>>
>> Our application 'v1' is integrated with DPDK 16.11 and 'v2' is integrated
>> with DPDK 17.05.01. In both 'v1' and 'v2' cases I don't see IOMMU width
>> error messages on VM, however, DMAR error messages are seen host. I am not
>> able to relate what causes DMAR error messages on host?
>>
>>
>> Thanks.
>>
>>
> Hi Ravi,
>
> vIOMMU support is out of our hands, really - we can only make use of
> hardware (or emulation of it) that is available. 39-bit wide address *can*
> work, you just have to be lucky and get PA addresses that would fit into 39
> bits (so under 512G limit), because we set up IOVA addresses to be 1:1 to
> physical addresses. We could, in principle, set up IOVA addresses to go
> from zero instead of them being a 1:1 mapping to physical addresses, but
> that would introduce need to translate addresses between IOVA and physical
> in some cases (e.g. KNI).
>
> I'm not aware of any changes between 16.11 and 17.11 (and indeed 18.02)
> that would make or break support for 39-bit wide PA addresses for IOMMU. It
> is possible that VF/PF drivers do something differently which results in
> DMAR errors showing up sooner rather than later, but as far as VFIO support
> itself is concerned, there were no major changes in those releases.
>
>
Hi Anatoly,

Thank you for your explanation. I would like to ask one more thing as I
need to get v-iommu+ dpdk working in VM. Can you please tell me what
determines 'Host Address Width", I know my question has nothing to do with
dpdk and this is a dpdk list, but if you have any information please share
it? I googled and found couple of ways to influence 'Host Address Width =
46' in guest as well (since dpdk + iommu works fine on host and DMAR on
host reports address width as 46).

(1) Qemu has CPU param 'host-phys-bits' boolean, when set to true copies it
from host
(2) Qemu has 'phys-bits'  integer, when set to '46' should influence guest

Using above options when instantiating a VM doesn't help, Guest VM still
ends up with 'Host address width = 39'.

(3) There is another Qemu option 'x-aw-bits' which is for VT-d which can be
set to '39' or '48'. This doesn't help either.

Thanks.

-- 
> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-09 17:41                             ` Ravi Kerur
@ 2018-02-10 10:11                               ` Burakov, Anatoly
  0 siblings, 0 replies; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-10 10:11 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 09-Feb-18 5:41 PM, Ravi Kerur wrote:
> 
> 
> On Thu, Feb 8, 2018 at 3:20 AM, Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     On 06-Feb-18 5:55 PM, Ravi Kerur wrote:
> 
> 
>         Hi Anatoly,
> 
>         I am actually confused with the state of vIOMMU + DPDK. Can you
>         please help me clarify?
> 
>         I tested following DPDK versions
> 
>         (1) DPDK 17.11, exhibits the issue (IOMMU width as reported by
>         RedHat and solution is to prevent using the patch)
>         (2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to
>         bind a device in VM with VFIO driver and no DMAR error message
>         on host
>         (3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to
>         bind a device in VM with VFIO driver and no DMAR error message
>         on host
> 
>         Clearly issue seen in 17.11 without the patch you mentioned is a
>         regression or the issue was masked in earlier DPDK version?  I
>         did not test traffic with any DPDK version because I wanted to
>         first get DMAR errors on host gone.
> 
>         Our application 'v1' is integrated with DPDK 16.11 and 'v2' is
>         integrated with DPDK 17.05.01. In both 'v1' and 'v2' cases I
>         don't see IOMMU width error messages on VM, however, DMAR error
>         messages are seen host. I am not able to relate what causes DMAR
>         error messages on host?
> 
> 
>         Thanks.
> 
> 
>     Hi Ravi,
> 
>     vIOMMU support is out of our hands, really - we can only make use of
>     hardware (or emulation of it) that is available. 39-bit wide address
>     *can* work, you just have to be lucky and get PA addresses that
>     would fit into 39 bits (so under 512G limit), because we set up IOVA
>     addresses to be 1:1 to physical addresses. We could, in principle,
>     set up IOVA addresses to go from zero instead of them being a 1:1
>     mapping to physical addresses, but that would introduce need to
>     translate addresses between IOVA and physical in some cases (e.g. KNI).
> 
>     I'm not aware of any changes between 16.11 and 17.11 (and indeed
>     18.02) that would make or break support for 39-bit wide PA addresses
>     for IOMMU. It is possible that VF/PF drivers do something
>     differently which results in DMAR errors showing up sooner rather
>     than later, but as far as VFIO support itself is concerned, there
>     were no major changes in those releases.
> 
> 
> Hi Anatoly,

Hi Ravi,

> 
> Thank you for your explanation. I would like to ask one more thing as I 
> need to get v-iommu+ dpdk working in VM. Can you please tell me what 
> determines 'Host Address Width", I know my question has nothing to do 
> with dpdk and this is a dpdk list, but if you have any information 
> please share it? I googled and found couple of ways to influence 'Host 
> Address Width = 46' in guest as well (since dpdk + iommu works fine on 
> host and DMAR on host reports address width as 46).

As far as i'm aware, address width supported by IOMMU is taken from 
hardware (real or emulated). If e.g. QEMU's vIOMMU emulates an IOMMU 
that only supports 39-bit addresses, that's what we'll get.

Not sure what "host address width" means exactly, but if i had to guess, 
usually in VM-speak, "host" refers to, well, host - that is, the 
physical machine the VM is running on. So, under that assumption, "host 
address width" would be telling QEMU maximum width of IOMMU addresses on 
the host (i.e. what you have in hardware, on the host - for example, you 
might be running on a platform that only supports a 39-bit wide physical 
address). Naturally, changing it around wouldn't change much - it would 
presumably affect only the way vIOMMU remaps requests from guest to 
host, but not the emulated hardware itself. However, i'm not 
knowledgeable enough in that area to answer this question definitively.

> 
> (1) Qemu has CPU param 'host-phys-bits' boolean, when set to true copies 
> it from host
> (2) Qemu has 'phys-bits'  integer, when set to '46' should influence guest
> 
> Using above options when instantiating a VM doesn't help, Guest VM still 
> ends up with 'Host address width = 39'.
> 
> (3) There is another Qemu option 'x-aw-bits' which is for VT-d which can 
> be set to '39' or '48'. This doesn't help either.
> 
> Thanks.
> 
>     -- 
>     Thanks,
>     Anatoly
> 
>
-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-01-29 22:35       ` Ravi Kerur
  2018-01-31  9:59         ` Burakov, Anatoly
@ 2018-02-10 10:58         ` Burakov, Anatoly
  2018-02-10 17:53           ` Ravi Kerur
  1 sibling, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-10 10:58 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 29-Jan-18 10:35 PM, Ravi Kerur wrote:
> Hi Burakov,
> 
> When using vfio-pci on host both VF and PF interfaces works fine with 
> dpdk i.e. I don't see DMAR fault messages anymore. However, when I 
> attach a VF interface to a VM and start DPDK with vfio-pci inside VM I 
> still see DMAR fault messages on host. Both host and VM are booted with 
> 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work 
> (I think it's expected because of DMAR faults), however, when VF 
> interface uses ixgbevf driver ping works.
> 
> Following are some details
> 
> /*****************On VM***************/
> dpdk-devbind -s
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci 
> unused=ixgbevf
> 
> Network devices using kernel driver
> ===================================
> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
> 
> Other network devices
> =====================
> <none>
> 
> Crypto devices using DPDK-compatible driver
> ===========================================
> <none>
> 
> Crypto devices using kernel driver
> ==================================
> <none>
> 
> Other crypto devices
> ====================
> <none>
> 
> 
> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller 
> Virtual Function (rev 01)
>          Subsystem: Intel Corporation 82599 Ethernet Controller Virtual 
> Function
>          Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- 
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>          Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K]
>          Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K]
>          Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>                  Vector table: BAR=3 offset=00000000
>                  PBA: BAR=3 offset=00002000
>          Capabilities: [a0] Express (v1) Root Complex Integrated 
> Endpoint, MSI 00
>                  DevCap: MaxPayload 128 bytes, PhantFunc 0
>                          ExtTag- RBE-
>                  DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
> Unsupported-
>                          RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                          MaxPayload 128 bytes, MaxReadReq 128 bytes
>                  DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- 
> AuxPwr- TransPend-
>          Capabilities: [100 v1] Advanced Error Reporting
>                  UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                  UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                  UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                  CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
> NonFatalErr-
>                  CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
> NonFatalErr-
>                  AERCap: First Error Pointer: 00, GenCap- CGenEn- 
> ChkCap- ChkEn-
>          Kernel driver in use: vfio-pci
>          Kernel modules: ixgbevf
> 
> /***************on Host*************/
> dmesg | grep DMAR
> ...
> [  978.268143] DMAR: DRHD: handling fault status reg 2
> [  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> [ 1286.677726] DMAR: DRHD: handling fault status reg 102
> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> fb663000 [fault reason 06] PTE Read access is not set
> [ 1676.436145] DMAR: DRHD: handling fault status reg 202
> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> [ 1734.433649] DMAR: DRHD: handling fault status reg 302
> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> [ 2324.428938] DMAR: DRHD: handling fault status reg 402
> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 7770c000 [fault reason 06] PTE Read access is not set
> [ 2388.553640] DMAR: DRHD: handling fault status reg 502
> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr 
> 33a128000 [fault reason 06] PTE Read access is not set
> 
> 

Going back to this, i would like to suggest run a few tests to ensure 
that we have all information that we can gather.

First of all, i'm assuming that you're using native ixgbe Linux driver 
on the host, and that you're only passing through the VF device to the 
VM using VFIO. Is my understanding correct here?

Now, let's forget about the iommu=pt and igb_uio for a moment. Boot both 
your host and your VM with iommu=on and intel_iommu=on (or whatever 
command-line enables full IOMMU support on both host and guest) and do 
the same tests you've done before. Do you still see your issues?

It would also be very useful to also try native Linux kernel driver on 
the guest *with traffic forwarding* and see how it works in your VM. 
Therefore i would suggest you to compile DPDK with PCAP support, bind 
your (VM) interface to native Linux driver, and use the interface via 
our pcap driver (creating a vdev should do the trick - please refer to 
PCAP PMD documentation [1]). Simple forwarding test should be enough - 
just make sure to pass traffic to and from DPDK in both cases, and that 
it doesn't give you any DMAR errors.

We can go from there.

[1] http://dpdk.org/doc/guides/nics/pcap_ring.html

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-10 10:58         ` Burakov, Anatoly
@ 2018-02-10 17:53           ` Ravi Kerur
  2018-02-12 10:13             ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-10 17:53 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Sat, Feb 10, 2018 at 2:58 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 29-Jan-18 10:35 PM, Ravi Kerur wrote:
>
>> Hi Burakov,
>>
>> When using vfio-pci on host both VF and PF interfaces works fine with
>> dpdk i.e. I don't see DMAR fault messages anymore. However, when I attach a
>> VF interface to a VM and start DPDK with vfio-pci inside VM I still see
>> DMAR fault messages on host. Both host and VM are booted with
>> 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work (I
>> think it's expected because of DMAR faults), however, when VF interface
>> uses ixgbevf driver ping works.
>>
>> Following are some details
>>
>> /*****************On VM***************/
>> dpdk-devbind -s
>>
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci
>> unused=ixgbevf
>>
>> Network devices using kernel driver
>> ===================================
>> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
>> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
>> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
>>
>> Other network devices
>> =====================
>> <none>
>>
>> Crypto devices using DPDK-compatible driver
>> ===========================================
>> <none>
>>
>> Crypto devices using kernel driver
>> ==================================
>> <none>
>>
>> Other crypto devices
>> ====================
>> <none>
>>
>>
>> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
>> Virtual Function (rev 01)
>>          Subsystem: Intel Corporation 82599 Ethernet Controller Virtual
>> Function
>>          Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
>> ParErr- Stepping- SERR- FastB2B- DisINTx+
>>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>          Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K]
>>          Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K]
>>          Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>>                  Vector table: BAR=3 offset=00000000
>>                  PBA: BAR=3 offset=00002000
>>          Capabilities: [a0] Express (v1) Root Complex Integrated
>> Endpoint, MSI 00
>>                  DevCap: MaxPayload 128 bytes, PhantFunc 0
>>                          ExtTag- RBE-
>>                  DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                          RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>>                          MaxPayload 128 bytes, MaxReadReq 128 bytes
>>                  DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>> TransPend-
>>          Capabilities: [100 v1] Advanced Error Reporting
>>                  UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
>> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                  UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
>> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                  UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
>> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                  CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>> NonFatalErr-
>>                  CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>> NonFatalErr-
>>                  AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap-
>> ChkEn-
>>          Kernel driver in use: vfio-pci
>>          Kernel modules: ixgbevf
>>
>> /***************on Host*************/
>> dmesg | grep DMAR
>> ...
>> [  978.268143] DMAR: DRHD: handling fault status reg 2
>> [  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>> [ 1286.677726] DMAR: DRHD: handling fault status reg 102
>> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> fb663000 [fault reason 06] PTE Read access is not set
>> [ 1676.436145] DMAR: DRHD: handling fault status reg 202
>> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>> [ 1734.433649] DMAR: DRHD: handling fault status reg 302
>> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>> [ 2324.428938] DMAR: DRHD: handling fault status reg 402
>> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 7770c000 [fault reason 06] PTE Read access is not set
>> [ 2388.553640] DMAR: DRHD: handling fault status reg 502
>> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>>
>>
>>
> Going back to this, i would like to suggest run a few tests to ensure that
> we have all information that we can gather.
>
> First of all, i'm assuming that you're using native ixgbe Linux driver on
> the host, and that you're only passing through the VF device to the VM
> using VFIO. Is my understanding correct here?
>
> Now, let's forget about the iommu=pt and igb_uio for a moment. Boot both
> your host and your VM with iommu=on and intel_iommu=on (or whatever
> command-line enables full IOMMU support on both host and guest) and do the
> same tests you've done before. Do you still see your issues?
>
> It would also be very useful to also try native Linux kernel driver on the
> guest *with traffic forwarding* and see how it works in your VM. Therefore
> i would suggest you to compile DPDK with PCAP support, bind your (VM)
> interface to native Linux driver, and use the interface via our pcap driver
> (creating a vdev should do the trick - please refer to PCAP PMD
> documentation [1]). Simple forwarding test should be enough - just make
> sure to pass traffic to and from DPDK in both cases, and that it doesn't
> give you any DMAR errors.
>
> We can go from there.
>
>
Let me just give you what has been tested and working/nonworking scenarios.
Some of your questions might get answered as well. Test bed is very simple
with 2 VF's created under IXGBE PF on host with one VF interface added to
ovs-bridge on host and another VF interface given to guest. Test
connectivity between VF's via ping.

Host and guest -- Kernel 4.9
Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the git (2.11.50))
DPDK -- 17.05.1 on host and guest
Host and guest -- booted with GRUB intel_iommu=on (which enables IOMMU).
Have tried with "iommu=on and intel_iommu=on" as well, but iommu=on is not
needed when intel_iommu=on is set.

Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver ping works
Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf driver ping
works
Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK vfio-pci driver,
DMAR errors seen on host, ping doesn't work

DPDK works fine on host with vfio-pci, however, has issues when used inside
the guest. Please let me know if more information is needed.

Thanks,
Ravi

[1] http://dpdk.org/doc/guides/nics/pcap_ring.html
>
> --
> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-10 17:53           ` Ravi Kerur
@ 2018-02-12 10:13             ` Burakov, Anatoly
  2018-02-12 22:00               ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-12 10:13 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 10-Feb-18 5:53 PM, Ravi Kerur wrote:
> 
> 
> On Sat, Feb 10, 2018 at 2:58 AM, Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     On 29-Jan-18 10:35 PM, Ravi Kerur wrote:
> 
>         Hi Burakov,
> 
>         When using vfio-pci on host both VF and PF interfaces works fine
>         with dpdk i.e. I don't see DMAR fault messages anymore. However,
>         when I attach a VF interface to a VM and start DPDK with
>         vfio-pci inside VM I still see DMAR fault messages on host. Both
>         host and VM are booted with 'intel-iommu=on' on GRUB. Ping from
>         VM with DPDK/vfio-pci doesn't work (I think it's expected
>         because of DMAR faults), however, when VF interface uses ixgbevf
>         driver ping works.
> 
>         Following are some details
> 
>         /*****************On VM***************/
>         dpdk-devbind -s
> 
>         Network devices using DPDK-compatible driver
>         ============================================
>         0000:00:07.0 '82599 Ethernet Controller Virtual Function'
>         drv=vfio-pci unused=ixgbevf
> 
>         Network devices using kernel driver
>         ===================================
>         0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci
>         unused=vfio-pci *Active*
>         0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
>         0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
> 
>         Other network devices
>         =====================
>         <none>
> 
>         Crypto devices using DPDK-compatible driver
>         ===========================================
>         <none>
> 
>         Crypto devices using kernel driver
>         ==================================
>         <none>
> 
>         Other crypto devices
>         ====================
>         <none>
> 
> 
>         00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet
>         Controller Virtual Function (rev 01)
>                   Subsystem: Intel Corporation 82599 Ethernet Controller
>         Virtual Function
>                   Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV-
>         VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>                   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
>          >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>                   Region 0: Memory at fda00000 (64-bit, prefetchable)
>         [size=16K]
>                   Region 3: Memory at fda04000 (64-bit, prefetchable)
>         [size=16K]
>                   Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>                           Vector table: BAR=3 offset=00000000
>                           PBA: BAR=3 offset=00002000
>                   Capabilities: [a0] Express (v1) Root Complex
>         Integrated Endpoint, MSI 00
>                           DevCap: MaxPayload 128 bytes, PhantFunc 0
>                                   ExtTag- RBE-
>                           DevCtl: Report errors: Correctable- Non-Fatal-
>         Fatal- Unsupported-
>                                   RlxdOrd- ExtTag- PhantFunc- AuxPwr-
>         NoSnoop-
>                                   MaxPayload 128 bytes, MaxReadReq 128 bytes
>                           DevSta: CorrErr- UncorrErr- FatalErr-
>         UnsuppReq- AuxPwr- TransPend-
>                   Capabilities: [100 v1] Advanced Error Reporting
>                           UESta:  DLP- SDES- TLP- FCP- CmpltTO-
>         CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                           UEMsk:  DLP- SDES- TLP- FCP- CmpltTO-
>         CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                           UESvrt: DLP- SDES- TLP- FCP- CmpltTO-
>         CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                           CESta:  RxErr- BadTLP- BadDLLP- Rollover-
>         Timeout- NonFatalErr-
>                           CEMsk:  RxErr- BadTLP- BadDLLP- Rollover-
>         Timeout- NonFatalErr-
>                           AERCap: First Error Pointer: 00, GenCap-
>         CGenEn- ChkCap- ChkEn-
>                   Kernel driver in use: vfio-pci
>                   Kernel modules: ixgbevf
> 
>         /***************on Host*************/
>         dmesg | grep DMAR
>         ...
>         [  978.268143] DMAR: DRHD: handling fault status reg 2
>         [  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault
>         addr 33a128000 [fault reason 06] PTE Read access is not set
>         [ 1286.677726] DMAR: DRHD: handling fault status reg 102
>         [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault
>         addr fb663000 [fault reason 06] PTE Read access is not set
>         [ 1676.436145] DMAR: DRHD: handling fault status reg 202
>         [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault
>         addr 33a128000 [fault reason 06] PTE Read access is not set
>         [ 1734.433649] DMAR: DRHD: handling fault status reg 302
>         [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault
>         addr 33a128000 [fault reason 06] PTE Read access is not set
>         [ 2324.428938] DMAR: DRHD: handling fault status reg 402
>         [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault
>         addr 7770c000 [fault reason 06] PTE Read access is not set
>         [ 2388.553640] DMAR: DRHD: handling fault status reg 502
>         [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault
>         addr 33a128000 [fault reason 06] PTE Read access is not set
> 
> 
> 
>     Going back to this, i would like to suggest run a few tests to
>     ensure that we have all information that we can gather.
> 
>     First of all, i'm assuming that you're using native ixgbe Linux
>     driver on the host, and that you're only passing through the VF
>     device to the VM using VFIO. Is my understanding correct here?
> 
>     Now, let's forget about the iommu=pt and igb_uio for a moment. Boot
>     both your host and your VM with iommu=on and intel_iommu=on (or
>     whatever command-line enables full IOMMU support on both host and
>     guest) and do the same tests you've done before. Do you still see
>     your issues?
> 
>     It would also be very useful to also try native Linux kernel driver
>     on the guest *with traffic forwarding* and see how it works in your
>     VM. Therefore i would suggest you to compile DPDK with PCAP support,
>     bind your (VM) interface to native Linux driver, and use the
>     interface via our pcap driver (creating a vdev should do the trick -
>     please refer to PCAP PMD documentation [1]). Simple forwarding test
>     should be enough - just make sure to pass traffic to and from DPDK
>     in both cases, and that it doesn't give you any DMAR errors.
> 
>     We can go from there.
> 
> 
> Let me just give you what has been tested and working/nonworking 
> scenarios. Some of your questions might get answered as well. Test bed 
> is very simple with 2 VF's created under IXGBE PF on host with one VF 
> interface added to ovs-bridge on host and another VF interface given to 
> guest. Test connectivity between VF's via ping.
> 
> Host and guest -- Kernel 4.9
> Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the git (2.11.50))
> DPDK -- 17.05.1 on host and guest
> Host and guest -- booted with GRUB intel_iommu=on (which enables IOMMU). 
> Have tried with "iommu=on and intel_iommu=on" as well, but iommu=on is 
> not needed when intel_iommu=on is set.
> 
> Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver ping works
> Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf driver 
> ping works
> Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK vfio-pci 
> driver, DMAR errors seen on host, ping doesn't work

OK, that makes it clearer, thanks. Does the third scenario work in other 
DPDK versions?

> 
> DPDK works fine on host with vfio-pci, however, has issues when used 
> inside the guest. Please let me know if more information is needed.
> 
> Thanks,
> Ravi
> 
>     [1] http://dpdk.org/doc/guides/nics/pcap_ring.html
>     <http://dpdk.org/doc/guides/nics/pcap_ring.html>
> 
>     -- 
>     Thanks,
>     Anatoly
> 
> 


-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-12 10:13             ` Burakov, Anatoly
@ 2018-02-12 22:00               ` Ravi Kerur
  2018-02-13 14:31                 ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-12 22:00 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Mon, Feb 12, 2018 at 2:13 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 10-Feb-18 5:53 PM, Ravi Kerur wrote:
>
>
>>
>> On Sat, Feb 10, 2018 at 2:58 AM, Burakov, Anatoly <
>> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>     On 29-Jan-18 10:35 PM, Ravi Kerur wrote:
>>
>>         Hi Burakov,
>>
>>         When using vfio-pci on host both VF and PF interfaces works fine
>>         with dpdk i.e. I don't see DMAR fault messages anymore. However,
>>         when I attach a VF interface to a VM and start DPDK with
>>         vfio-pci inside VM I still see DMAR fault messages on host. Both
>>         host and VM are booted with 'intel-iommu=on' on GRUB. Ping from
>>         VM with DPDK/vfio-pci doesn't work (I think it's expected
>>         because of DMAR faults), however, when VF interface uses ixgbevf
>>         driver ping works.
>>
>>         Following are some details
>>
>>         /*****************On VM***************/
>>         dpdk-devbind -s
>>
>>         Network devices using DPDK-compatible driver
>>         ============================================
>>         0000:00:07.0 '82599 Ethernet Controller Virtual Function'
>>         drv=vfio-pci unused=ixgbevf
>>
>>         Network devices using kernel driver
>>         ===================================
>>         0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci
>>         unused=vfio-pci *Active*
>>         0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
>>         0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
>>
>>         Other network devices
>>         =====================
>>         <none>
>>
>>         Crypto devices using DPDK-compatible driver
>>         ===========================================
>>         <none>
>>
>>         Crypto devices using kernel driver
>>         ==================================
>>         <none>
>>
>>         Other crypto devices
>>         ====================
>>         <none>
>>
>>
>>         00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet
>>         Controller Virtual Function (rev 01)
>>                   Subsystem: Intel Corporation 82599 Ethernet Controller
>>         Virtual Function
>>                   Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV-
>>         VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>>                   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
>>          >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>>                   Region 0: Memory at fda00000 (64-bit, prefetchable)
>>         [size=16K]
>>                   Region 3: Memory at fda04000 (64-bit, prefetchable)
>>         [size=16K]
>>                   Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>>                           Vector table: BAR=3 offset=00000000
>>                           PBA: BAR=3 offset=00002000
>>                   Capabilities: [a0] Express (v1) Root Complex
>>         Integrated Endpoint, MSI 00
>>                           DevCap: MaxPayload 128 bytes, PhantFunc 0
>>                                   ExtTag- RBE-
>>                           DevCtl: Report errors: Correctable- Non-Fatal-
>>         Fatal- Unsupported-
>>                                   RlxdOrd- ExtTag- PhantFunc- AuxPwr-
>>         NoSnoop-
>>                                   MaxPayload 128 bytes, MaxReadReq 128
>> bytes
>>                           DevSta: CorrErr- UncorrErr- FatalErr-
>>         UnsuppReq- AuxPwr- TransPend-
>>                   Capabilities: [100 v1] Advanced Error Reporting
>>                           UESta:  DLP- SDES- TLP- FCP- CmpltTO-
>>         CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                           UEMsk:  DLP- SDES- TLP- FCP- CmpltTO-
>>         CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                           UESvrt: DLP- SDES- TLP- FCP- CmpltTO-
>>         CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                           CESta:  RxErr- BadTLP- BadDLLP- Rollover-
>>         Timeout- NonFatalErr-
>>                           CEMsk:  RxErr- BadTLP- BadDLLP- Rollover-
>>         Timeout- NonFatalErr-
>>                           AERCap: First Error Pointer: 00, GenCap-
>>         CGenEn- ChkCap- ChkEn-
>>                   Kernel driver in use: vfio-pci
>>                   Kernel modules: ixgbevf
>>
>>         /***************on Host*************/
>>         dmesg | grep DMAR
>>         ...
>>         [  978.268143] DMAR: DRHD: handling fault status reg 2
>>         [  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault
>>         addr 33a128000 [fault reason 06] PTE Read access is not set
>>         [ 1286.677726] DMAR: DRHD: handling fault status reg 102
>>         [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault
>>         addr fb663000 [fault reason 06] PTE Read access is not set
>>         [ 1676.436145] DMAR: DRHD: handling fault status reg 202
>>         [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault
>>         addr 33a128000 [fault reason 06] PTE Read access is not set
>>         [ 1734.433649] DMAR: DRHD: handling fault status reg 302
>>         [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault
>>         addr 33a128000 [fault reason 06] PTE Read access is not set
>>         [ 2324.428938] DMAR: DRHD: handling fault status reg 402
>>         [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault
>>         addr 7770c000 [fault reason 06] PTE Read access is not set
>>         [ 2388.553640] DMAR: DRHD: handling fault status reg 502
>>         [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault
>>         addr 33a128000 [fault reason 06] PTE Read access is not set
>>
>>
>>
>>     Going back to this, i would like to suggest run a few tests to
>>     ensure that we have all information that we can gather.
>>
>>     First of all, i'm assuming that you're using native ixgbe Linux
>>     driver on the host, and that you're only passing through the VF
>>     device to the VM using VFIO. Is my understanding correct here?
>>
>>     Now, let's forget about the iommu=pt and igb_uio for a moment. Boot
>>     both your host and your VM with iommu=on and intel_iommu=on (or
>>     whatever command-line enables full IOMMU support on both host and
>>     guest) and do the same tests you've done before. Do you still see
>>     your issues?
>>
>>     It would also be very useful to also try native Linux kernel driver
>>     on the guest *with traffic forwarding* and see how it works in your
>>     VM. Therefore i would suggest you to compile DPDK with PCAP support,
>>     bind your (VM) interface to native Linux driver, and use the
>>     interface via our pcap driver (creating a vdev should do the trick -
>>     please refer to PCAP PMD documentation [1]). Simple forwarding test
>>     should be enough - just make sure to pass traffic to and from DPDK
>>     in both cases, and that it doesn't give you any DMAR errors.
>>
>>     We can go from there.
>>
>>
>> Let me just give you what has been tested and working/nonworking
>> scenarios. Some of your questions might get answered as well. Test bed is
>> very simple with 2 VF's created under IXGBE PF on host with one VF
>> interface added to ovs-bridge on host and another VF interface given to
>> guest. Test connectivity between VF's via ping.
>>
>> Host and guest -- Kernel 4.9
>> Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the git
>> (2.11.50))
>> DPDK -- 17.05.1 on host and guest
>> Host and guest -- booted with GRUB intel_iommu=on (which enables IOMMU).
>> Have tried with "iommu=on and intel_iommu=on" as well, but iommu=on is not
>> needed when intel_iommu=on is set.
>>
>> Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver ping works
>> Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf driver ping
>> works
>> Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK vfio-pci
>> driver, DMAR errors seen on host, ping doesn't work
>>
>
> OK, that makes it clearer, thanks. Does the third scenario work in other
> DPDK versions?


No. Tried 16.11 same issue on guest and works fine on host.

>
>
>
>> DPDK works fine on host with vfio-pci, however, has issues when used
>> inside the guest. Please let me know if more information is needed.
>>
>> Thanks,
>> Ravi
>>
>>     [1] http://dpdk.org/doc/guides/nics/pcap_ring.html
>>     <http://dpdk.org/doc/guides/nics/pcap_ring.html>
>>
>>     --     Thanks,
>>     Anatoly
>>
>>
>>
>
> --
> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-12 22:00               ` Ravi Kerur
@ 2018-02-13 14:31                 ` Burakov, Anatoly
  2018-02-14 20:00                   ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-13 14:31 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 12-Feb-18 10:00 PM, Ravi Kerur wrote:
> 
>         Let me just give you what has been tested and working/nonworking
>         scenarios. Some of your questions might get answered as well.
>         Test bed is very simple with 2 VF's created under IXGBE PF on
>         host with one VF interface added to ovs-bridge on host and
>         another VF interface given to guest. Test connectivity between
>         VF's via ping.
> 
>         Host and guest -- Kernel 4.9
>         Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the
>         git (2.11.50))
>         DPDK -- 17.05.1 on host and guest
>         Host and guest -- booted with GRUB intel_iommu=on (which enables
>         IOMMU). Have tried with "iommu=on and intel_iommu=on" as well,
>         but iommu=on is not needed when intel_iommu=on is set.
> 
>         Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver
>         ping works
>         Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf
>         driver ping works
>         Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK
>         vfio-pci driver, DMAR errors seen on host, ping doesn't work
> 
> 
>     OK, that makes it clearer, thanks. Does the third scenario work in
>     other DPDK versions?
> 
> 
> No. Tried 16.11 same issue on guest and works fine on host.
> 
> 
So now we've moved from "this worked on 16.11" to "this never worked".

It would be good to see output of rte_dump_physmem_layout() on both host 
and guest, and check which address triggers the DMAR error (i.e. if the 
physical address is present in mappings for either DPDK process).

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-13 14:31                 ` Burakov, Anatoly
@ 2018-02-14 20:00                   ` Ravi Kerur
  2018-02-15 10:28                     ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-14 20:00 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Tue, Feb 13, 2018 at 6:31 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 12-Feb-18 10:00 PM, Ravi Kerur wrote:
>
>>
>>         Let me just give you what has been tested and working/nonworking
>>         scenarios. Some of your questions might get answered as well.
>>         Test bed is very simple with 2 VF's created under IXGBE PF on
>>         host with one VF interface added to ovs-bridge on host and
>>         another VF interface given to guest. Test connectivity between
>>         VF's via ping.
>>
>>         Host and guest -- Kernel 4.9
>>         Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the
>>         git (2.11.50))
>>         DPDK -- 17.05.1 on host and guest
>>         Host and guest -- booted with GRUB intel_iommu=on (which enables
>>         IOMMU). Have tried with "iommu=on and intel_iommu=on" as well,
>>         but iommu=on is not needed when intel_iommu=on is set.
>>
>>         Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver
>>         ping works
>>         Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf
>>         driver ping works
>>         Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK
>>         vfio-pci driver, DMAR errors seen on host, ping doesn't work
>>
>>
>>     OK, that makes it clearer, thanks. Does the third scenario work in
>>     other DPDK versions?
>>
>>
>> No. Tried 16.11 same issue on guest and works fine on host.
>>
>>
>> So now we've moved from "this worked on 16.11" to "this never worked".
>
> It would be good to see output of rte_dump_physmem_layout() on both host
> and guest, and check which address triggers the DMAR error (i.e. if the
> physical address is present in mappings for either DPDK process).
>
> --
>

Earlier I was focusing only on DMAR errors and I might have said 'it
worked' when I didn't notice them on host when dpdk was started on guest.
When trying to send packets out of that interface from guest I did see DMAR
errors. I am attaching information you requested.  I have enabled
log-level=8 and files contain dpdk EAL/PMD logs as well.

Snippets below

on host, DMAR fault address from dmesg

[351576.998109] DMAR: DRHD: handling fault status reg 702
[351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr
257617000 [fault reason 06] PTE Read access is not set

on guest (dump phys_mem_layout)

Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
...
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80
...

Thanks,
Ravi



> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-14 20:00                   ` Ravi Kerur
@ 2018-02-15 10:28                     ` Burakov, Anatoly
  2018-02-15 18:27                       ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-15 10:28 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev

On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
> 
> Earlier I was focusing only on DMAR errors and I might have said 'it 
> worked' when I didn't notice them on host when dpdk was started on 
> guest. When trying to send packets out of that interface from guest I 
> did see DMAR errors. I am attaching information you requested.  I have 
> enabled log-level=8 and files contain dpdk EAL/PMD logs as well.

Great, now we're on the same page.

> 
> Snippets below
> 
> on host, DMAR fault address from dmesg
> 
> [351576.998109] DMAR: DRHD: handling fault status reg 702
> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 257617000 [fault reason 06] PTE Read access is not set
> 
> on guest (dump phys_mem_layout)
> 
> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000, 
> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
> ...
> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 
> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600
> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 
> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80
> ...
> 

To me this looks like host (i.e. either QEMU or the PF driver) is trying 
to do DMA using guest-physical (and not host-physical). I'm not too 
well-versed in how QEMU works, but i'm pretty sure that's not supposed 
to happen.

Is PF also bound to DPDK, or are you using native Linux ixgbe driver?

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-15 10:28                     ` Burakov, Anatoly
@ 2018-02-15 18:27                       ` Ravi Kerur
  2018-02-15 20:53                         ` Ravi Kerur
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-15 18:27 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
>
>>
>> Earlier I was focusing only on DMAR errors and I might have said 'it
>> worked' when I didn't notice them on host when dpdk was started on guest.
>> When trying to send packets out of that interface from guest I did see DMAR
>> errors. I am attaching information you requested.  I have enabled
>> log-level=8 and files contain dpdk EAL/PMD logs as well.
>>
>
> Great, now we're on the same page.
>
>
>> Snippets below
>>
>> on host, DMAR fault address from dmesg
>>
>> [351576.998109] DMAR: DRHD: handling fault status reg 702
>> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 257617000 [fault reason 06] PTE Read access is not set
>>
>> on guest (dump phys_mem_layout)
>>
>> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000,
>> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
>> ...
>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
>> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600
>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
>> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80
>> ...
>>
>>
> To me this looks like host (i.e. either QEMU or the PF driver) is trying
> to do DMA using guest-physical (and not host-physical). I'm not too
> well-versed in how QEMU works, but i'm pretty sure that's not supposed to
> happen.
>
> Is PF also bound to DPDK, or are you using native Linux ixgbe driver?
>

Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF interfaces
disappear after it is bound to DPDK. If there is a way to use PF and VF
with DPDK let me know I can try it out. I am not sure how to move forward
on this, Is CPU/IXGBE PF driver playing a role? Following are the versions
I have

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                56
On-line CPU(s) list:   0-27
Off-line CPU(s) list:  28-55
Thread(s) per core:    1
Core(s) per socket:    14
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
Stepping:              2
CPU MHz:               2500.610
CPU max MHz:           3000.0000
CPU min MHz:           1200.0000
BogoMIPS:              4000.74
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              35840K
NUMA node0 CPU(s):     0-13
NUMA node1 CPU(s):     14-27

# ethtool -i enp4s0f0
driver: ixgbe
version: 5.3.3
firmware-version: 0x800007b8, 1.1018.0
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Thanks,
Ravi



>
> --
> Thanks,
> Anatoly
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-15 18:27                       ` Ravi Kerur
@ 2018-02-15 20:53                         ` Ravi Kerur
  2018-02-16  9:41                           ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Ravi Kerur @ 2018-02-15 20:53 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev

On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com> wrote:

>
>
> On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly <
> anatoly.burakov@intel.com> wrote:
>
>> On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
>>
>>>
>>> Earlier I was focusing only on DMAR errors and I might have said 'it
>>> worked' when I didn't notice them on host when dpdk was started on guest.
>>> When trying to send packets out of that interface from guest I did see DMAR
>>> errors. I am attaching information you requested.  I have enabled
>>> log-level=8 and files contain dpdk EAL/PMD logs as well.
>>>
>>
>> Great, now we're on the same page.
>>
>>
>>> Snippets below
>>>
>>> on host, DMAR fault address from dmesg
>>>
>>> [351576.998109] DMAR: DRHD: handling fault status reg 702
>>> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr
>>> 257617000 [fault reason 06] PTE Read access is not set
>>>
>>> on guest (dump phys_mem_layout)
>>>
>>> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000,
>>> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
>>> ...
>>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
>>> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600
>>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
>>> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80
>>> ...
>>>
>>>
>> To me this looks like host (i.e. either QEMU or the PF driver) is trying
>> to do DMA using guest-physical (and not host-physical). I'm not too
>> well-versed in how QEMU works, but i'm pretty sure that's not supposed to
>> happen.
>>
>> Is PF also bound to DPDK, or are you using native Linux ixgbe driver?
>>
>
> Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF interfaces
> disappear after it is bound to DPDK. If there is a way to use PF and VF
> with DPDK let me know I can try it out. I am not sure how to move forward
> on this, Is CPU/IXGBE PF driver playing a role? Following are the versions
> I have
>
> lscpu
> Architecture:          x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Little Endian
> CPU(s):                56
> On-line CPU(s) list:   0-27
> Off-line CPU(s) list:  28-55
> Thread(s) per core:    1
> Core(s) per socket:    14
> Socket(s):             2
> NUMA node(s):          2
> Vendor ID:             GenuineIntel
> CPU family:            6
> Model:                 63
> Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
> Stepping:              2
> CPU MHz:               2500.610
> CPU max MHz:           3000.0000
> CPU min MHz:           1200.0000
> BogoMIPS:              4000.74
> Virtualization:        VT-x
> L1d cache:             32K
> L1i cache:             32K
> L2 cache:              256K
> L3 cache:              35840K
> NUMA node0 CPU(s):     0-13
> NUMA node1 CPU(s):     14-27
>
> # ethtool -i enp4s0f0
> driver: ixgbe
> version: 5.3.3
> firmware-version: 0x800007b8, 1.1018.0
> bus-info: 0000:04:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> Thanks,
> Ravi
>
>
>

Debugging this I could co-relate doing interface link-up associated with
the dpdk inside the guest causes DMAR errors on host and an additional vflr
message.

[ 8135.861622] DMAR: DRHD: handling fault status reg 402
[ 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr
1b648a000 [fault reason 06] PTE Read access is not set
[ 8136.588074] ixgbe 0000:04:00.0: Issuing VFLR with pending transactions
[ 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0

Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from
'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it
possible that dpdk pmd vf driver is missing some fixes/porting from ixgbevf
driver since this issue is not seen when ixgbevf kernel driver is used?

Thanks,
Ravi


>> --
>> Thanks,
>> Anatoly
>>
>
>

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-15 20:53                         ` Ravi Kerur
@ 2018-02-16  9:41                           ` Burakov, Anatoly
  2019-01-15  7:07                             ` Hu, Xuekun
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2018-02-16  9:41 UTC (permalink / raw)
  To: Ravi Kerur; +Cc: dev, Ananyev, Konstantin, wenzhuo.lu

On 15-Feb-18 8:53 PM, Ravi Kerur wrote:
> 
> 
> On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com 
> <mailto:rkerur@gmail.com>> wrote:
> 
> 
> 
>     On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly
>     <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>         On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
> 
> 
>             Earlier I was focusing only on DMAR errors and I might have
>             said 'it worked' when I didn't notice them on host when dpdk
>             was started on guest. When trying to send packets out of
>             that interface from guest I did see DMAR errors. I am
>             attaching information you requested.  I have enabled
>             log-level=8 and files contain dpdk EAL/PMD logs as well.
> 
> 
>         Great, now we're on the same page.
> 
> 
>             Snippets below
> 
>             on host, DMAR fault address from dmesg
> 
>             [351576.998109] DMAR: DRHD: handling fault status reg 702
>             [351576.998113] DMAR: [DMA Read] Request device [04:10.0]
>             fault addr 257617000 [fault reason 06] PTE Read access is
>             not set
> 
>             on guest (dump phys_mem_layout)
> 
>             Segment 235: phys:0x257600000, len:2097152,
>             virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152,
>             nchannel:0, nrank:0
>             ...
>             PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
>             sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600
>             dma_addr=0x257617600
>             PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
>             sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80
>             dma_addr=0x25406fe80
>             ...
> 
> 
>         To me this looks like host (i.e. either QEMU or the PF driver)
>         is trying to do DMA using guest-physical (and not
>         host-physical). I'm not too well-versed in how QEMU works, but
>         i'm pretty sure that's not supposed to happen.
> 
>         Is PF also bound to DPDK, or are you using native Linux ixgbe
>         driver?
> 
> 
>     Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF
>     interfaces disappear after it is bound to DPDK. If there is a way to
>     use PF and VF with DPDK let me know I can try it out. I am not sure
>     how to move forward on this, Is CPU/IXGBE PF driver playing a role?
>     Following are the versions I have
> 
>     lscpu
>     Architecture:          x86_64
>     CPU op-mode(s):        32-bit, 64-bit
>     Byte Order:            Little Endian
>     CPU(s):                56
>     On-line CPU(s) list:   0-27
>     Off-line CPU(s) list:  28-55
>     Thread(s) per core:    1
>     Core(s) per socket:    14
>     Socket(s):             2
>     NUMA node(s):          2
>     Vendor ID:             GenuineIntel
>     CPU family:            6
>     Model:                 63
>     Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
>     Stepping:              2
>     CPU MHz:               2500.610
>     CPU max MHz:           3000.0000
>     CPU min MHz:           1200.0000
>     BogoMIPS:              4000.74
>     Virtualization:        VT-x
>     L1d cache:             32K
>     L1i cache:             32K
>     L2 cache:              256K
>     L3 cache:              35840K
>     NUMA node0 CPU(s):     0-13
>     NUMA node1 CPU(s):     14-27
> 
>     # ethtool -i enp4s0f0
>     driver: ixgbe
>     version: 5.3.3
>     firmware-version: 0x800007b8, 1.1018.0
>     bus-info: 0000:04:00.0
>     supports-statistics: yes
>     supports-test: yes
>     supports-eeprom-access: yes
>     supports-register-dump: yes
>     supports-priv-flags: yes
> 
>     Thanks,
>     Ravi
> 
> 
> 
> Debugging this I could co-relate doing interface link-up associated with 
> the dpdk inside the guest causes DMAR errors on host and an additional 
> vflr message.
> 
> [ 8135.861622] DMAR: DRHD: handling fault status reg 402
> [ 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 1b648a000 [fault reason 06] PTE Read access is not set
> [ 8136.588074] ixgbe 0000:04:00.0: Issuing VFLR with pending transactions
> [ 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0
> 
> Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from 
> 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it 
> possible that dpdk pmd vf driver is missing some fixes/porting from 
> ixgbevf driver since this issue is not seen when ixgbevf kernel driver 
> is used?
> 

Could very well be. +CC ixgbe maintainers which might be of further help 
debugging this issue.

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2018-02-16  9:41                           ` Burakov, Anatoly
@ 2019-01-15  7:07                             ` Hu, Xuekun
  2019-01-15 11:22                               ` Burakov, Anatoly
  0 siblings, 1 reply; 33+ messages in thread
From: Hu, Xuekun @ 2019-01-15  7:07 UTC (permalink / raw)
  To: Burakov, Anatoly, Ravi Kerur; +Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo

Hi, Ravi

Did you resolve this issue that VF used in guest with vIOMMU enabled? I googled, but still can't get the answer that it is driver or qemu vt-d emulation issue. 

Currently I met the same issue again that host reported DMAR error: 
[59939.130110] DMAR: DRHD: handling fault status reg 2
[59939.130116] DMAR: [DMA Read] Request device [83:10.0] fault addr 15f03d000 [fault reason 06] PTE Read access is not set
[59940.180859] ixgbe 0000:83:00.0: Issuing VFLR with pending transactions
[59940.180863] ixgbe 0000:83:00.0: Issuing VFLR for VF 0000:83:10.0
[59989.344683] ixgbe 0000:83:00.0 ens817f0: VF Reset msg received from vf 0

I'm using DPDK 18.11 in guest, and ixgbe.ko in host. 

Thx, Xuekun

-----Original Message-----
From: dev <dev-bounces@dpdk.org> On Behalf Of Burakov, Anatoly
Sent: Friday, February 16, 2018 5:42 PM
To: Ravi Kerur <rkerur@gmail.com>
Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>
Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

On 15-Feb-18 8:53 PM, Ravi Kerur wrote:
> 
> 
> On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com 
> <mailto:rkerur@gmail.com>> wrote:
> 
> 
> 
>     On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly
>     <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>         On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
> 
> 
>             Earlier I was focusing only on DMAR errors and I might have
>             said 'it worked' when I didn't notice them on host when dpdk
>             was started on guest. When trying to send packets out of
>             that interface from guest I did see DMAR errors. I am
>             attaching information you requested.  I have enabled
>             log-level=8 and files contain dpdk EAL/PMD logs as well.
> 
> 
>         Great, now we're on the same page.
> 
> 
>             Snippets below
> 
>             on host, DMAR fault address from dmesg
> 
>             [351576.998109] DMAR: DRHD: handling fault status reg 702
>             [351576.998113] DMAR: [DMA Read] Request device [04:10.0]
>             fault addr 257617000 [fault reason 06] PTE Read access is
>             not set
> 
>             on guest (dump phys_mem_layout)
> 
>             Segment 235: phys:0x257600000, len:2097152,
>             virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152,
>             nchannel:0, nrank:0
>             ...
>             PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
>             sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600
>             dma_addr=0x257617600
>             PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
>             sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80
>             dma_addr=0x25406fe80
>             ...
> 
> 
>         To me this looks like host (i.e. either QEMU or the PF driver)
>         is trying to do DMA using guest-physical (and not
>         host-physical). I'm not too well-versed in how QEMU works, but
>         i'm pretty sure that's not supposed to happen.
> 
>         Is PF also bound to DPDK, or are you using native Linux ixgbe
>         driver?
> 
> 
>     Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF
>     interfaces disappear after it is bound to DPDK. If there is a way to
>     use PF and VF with DPDK let me know I can try it out. I am not sure
>     how to move forward on this, Is CPU/IXGBE PF driver playing a role?
>     Following are the versions I have
> 
>     lscpu
>     Architecture:          x86_64
>     CPU op-mode(s):        32-bit, 64-bit
>     Byte Order:            Little Endian
>     CPU(s):                56
>     On-line CPU(s) list:   0-27
>     Off-line CPU(s) list:  28-55
>     Thread(s) per core:    1
>     Core(s) per socket:    14
>     Socket(s):             2
>     NUMA node(s):          2
>     Vendor ID:             GenuineIntel
>     CPU family:            6
>     Model:                 63
>     Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
>     Stepping:              2
>     CPU MHz:               2500.610
>     CPU max MHz:           3000.0000
>     CPU min MHz:           1200.0000
>     BogoMIPS:              4000.74
>     Virtualization:        VT-x
>     L1d cache:             32K
>     L1i cache:             32K
>     L2 cache:              256K
>     L3 cache:              35840K
>     NUMA node0 CPU(s):     0-13
>     NUMA node1 CPU(s):     14-27
> 
>     # ethtool -i enp4s0f0
>     driver: ixgbe
>     version: 5.3.3
>     firmware-version: 0x800007b8, 1.1018.0
>     bus-info: 0000:04:00.0
>     supports-statistics: yes
>     supports-test: yes
>     supports-eeprom-access: yes
>     supports-register-dump: yes
>     supports-priv-flags: yes
> 
>     Thanks,
>     Ravi
> 
> 
> 
> Debugging this I could co-relate doing interface link-up associated 
> with the dpdk inside the guest causes DMAR errors on host and an 
> additional vflr message.
> 
> [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [ 
> 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr
> 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074] 
> ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [ 
> 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0
> 
> Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from 
> 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it 
> possible that dpdk pmd vf driver is missing some fixes/porting from 
> ixgbevf driver since this issue is not seen when ixgbevf kernel driver 
> is used?
> 

Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue.

--
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2019-01-15  7:07                             ` Hu, Xuekun
@ 2019-01-15 11:22                               ` Burakov, Anatoly
  2019-01-15 13:07                                 ` Hu, Xuekun
  2019-01-21 13:18                                 ` Hu, Xuekun
  0 siblings, 2 replies; 33+ messages in thread
From: Burakov, Anatoly @ 2019-01-15 11:22 UTC (permalink / raw)
  To: Hu, Xuekun, Ravi Kerur; +Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo

On 15-Jan-19 7:07 AM, Hu, Xuekun wrote:
> Hi, Ravi
> 
> Did you resolve this issue that VF used in guest with vIOMMU enabled? I googled, but still can't get the answer that it is driver or qemu vt-d emulation issue.
> 
> Currently I met the same issue again that host reported DMAR error:
> [59939.130110] DMAR: DRHD: handling fault status reg 2
> [59939.130116] DMAR: [DMA Read] Request device [83:10.0] fault addr 15f03d000 [fault reason 06] PTE Read access is not set
> [59940.180859] ixgbe 0000:83:00.0: Issuing VFLR with pending transactions
> [59940.180863] ixgbe 0000:83:00.0: Issuing VFLR for VF 0000:83:10.0
> [59989.344683] ixgbe 0000:83:00.0 ens817f0: VF Reset msg received from vf 0
> 
> I'm using DPDK 18.11 in guest, and ixgbe.ko in host.
> 
> Thx, Xuekun
> 
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Burakov, Anatoly
> Sent: Friday, February 16, 2018 5:42 PM
> To: Ravi Kerur <rkerur@gmail.com>
> Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>
> Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
> 
> On 15-Feb-18 8:53 PM, Ravi Kerur wrote:
>>
>>
>> On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com
>> <mailto:rkerur@gmail.com>> wrote:
>>
>>
>>
>>      On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly
>>      <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>          On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
>>
>>
>>              Earlier I was focusing only on DMAR errors and I might have
>>              said 'it worked' when I didn't notice them on host when dpdk
>>              was started on guest. When trying to send packets out of
>>              that interface from guest I did see DMAR errors. I am
>>              attaching information you requested.  I have enabled
>>              log-level=8 and files contain dpdk EAL/PMD logs as well.
>>
>>
>>          Great, now we're on the same page.
>>
>>
>>              Snippets below
>>
>>              on host, DMAR fault address from dmesg
>>
>>              [351576.998109] DMAR: DRHD: handling fault status reg 702
>>              [351576.998113] DMAR: [DMA Read] Request device [04:10.0]
>>              fault addr 257617000 [fault reason 06] PTE Read access is
>>              not set
>>
>>              on guest (dump phys_mem_layout)
>>
>>              Segment 235: phys:0x257600000, len:2097152,
>>              virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152,
>>              nchannel:0, nrank:0
>>              ...
>>              PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
>>              sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600
>>              dma_addr=0x257617600
>>              PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
>>              sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80
>>              dma_addr=0x25406fe80
>>              ...
>>
>>
>>          To me this looks like host (i.e. either QEMU or the PF driver)
>>          is trying to do DMA using guest-physical (and not
>>          host-physical). I'm not too well-versed in how QEMU works, but
>>          i'm pretty sure that's not supposed to happen.
>>
>>          Is PF also bound to DPDK, or are you using native Linux ixgbe
>>          driver?
>>
>>
>>      Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF
>>      interfaces disappear after it is bound to DPDK. If there is a way to
>>      use PF and VF with DPDK let me know I can try it out. I am not sure
>>      how to move forward on this, Is CPU/IXGBE PF driver playing a role?
>>      Following are the versions I have
>>
>>      lscpu
>>      Architecture:          x86_64
>>      CPU op-mode(s):        32-bit, 64-bit
>>      Byte Order:            Little Endian
>>      CPU(s):                56
>>      On-line CPU(s) list:   0-27
>>      Off-line CPU(s) list:  28-55
>>      Thread(s) per core:    1
>>      Core(s) per socket:    14
>>      Socket(s):             2
>>      NUMA node(s):          2
>>      Vendor ID:             GenuineIntel
>>      CPU family:            6
>>      Model:                 63
>>      Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
>>      Stepping:              2
>>      CPU MHz:               2500.610
>>      CPU max MHz:           3000.0000
>>      CPU min MHz:           1200.0000
>>      BogoMIPS:              4000.74
>>      Virtualization:        VT-x
>>      L1d cache:             32K
>>      L1i cache:             32K
>>      L2 cache:              256K
>>      L3 cache:              35840K
>>      NUMA node0 CPU(s):     0-13
>>      NUMA node1 CPU(s):     14-27
>>
>>      # ethtool -i enp4s0f0
>>      driver: ixgbe
>>      version: 5.3.3
>>      firmware-version: 0x800007b8, 1.1018.0
>>      bus-info: 0000:04:00.0
>>      supports-statistics: yes
>>      supports-test: yes
>>      supports-eeprom-access: yes
>>      supports-register-dump: yes
>>      supports-priv-flags: yes
>>
>>      Thanks,
>>      Ravi
>>
>>
>>
>> Debugging this I could co-relate doing interface link-up associated
>> with the dpdk inside the guest causes DMAR errors on host and an
>> additional vflr message.
>>
>> [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [
>> 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074]
>> ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [
>> 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0
>>
>> Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from
>> 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it
>> possible that dpdk pmd vf driver is missing some fixes/porting from
>> ixgbevf driver since this issue is not seen when ixgbevf kernel driver
>> is used?
>>
> 
> Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue.
> 
> --
> Thanks,
> Anatoly
> 

You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76

There are apparently issues with some kernel versions that will manifest 
themselves as problems with using VF devices with IOMMU in a VM.

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2019-01-15 11:22                               ` Burakov, Anatoly
@ 2019-01-15 13:07                                 ` Hu, Xuekun
  2019-01-21 13:18                                 ` Hu, Xuekun
  1 sibling, 0 replies; 33+ messages in thread
From: Hu, Xuekun @ 2019-01-15 13:07 UTC (permalink / raw)
  To: Burakov, Anatoly, Ravi Kerur; +Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo

Thanks, I'm just using CentOS7.5 3.10!
I will try a newer kernel. Stay tuned.  

-----Original Message-----
From: Burakov, Anatoly 
Sent: Tuesday, January 15, 2019 7:22 PM
To: Hu, Xuekun <xuekun.hu@intel.com>; Ravi Kerur <rkerur@gmail.com>
Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>
Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

On 15-Jan-19 7:07 AM, Hu, Xuekun wrote:
> Hi, Ravi
> 
> Did you resolve this issue that VF used in guest with vIOMMU enabled? I googled, but still can't get the answer that it is driver or qemu vt-d emulation issue.
> 
> Currently I met the same issue again that host reported DMAR error:
> [59939.130110] DMAR: DRHD: handling fault status reg 2 [59939.130116] 
> DMAR: [DMA Read] Request device [83:10.0] fault addr 15f03d000 [fault 
> reason 06] PTE Read access is not set [59940.180859] ixgbe 
> 0000:83:00.0: Issuing VFLR with pending transactions [59940.180863] 
> ixgbe 0000:83:00.0: Issuing VFLR for VF 0000:83:10.0 [59989.344683] 
> ixgbe 0000:83:00.0 ens817f0: VF Reset msg received from vf 0
> 
> I'm using DPDK 18.11 in guest, and ixgbe.ko in host.
> 
> Thx, Xuekun
> 
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Burakov, Anatoly
> Sent: Friday, February 16, 2018 5:42 PM
> To: Ravi Kerur <rkerur@gmail.com>
> Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; 
> Lu, Wenzhuo <wenzhuo.lu@intel.com>
> Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
> 
> On 15-Feb-18 8:53 PM, Ravi Kerur wrote:
>>
>>
>> On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com 
>> <mailto:rkerur@gmail.com>> wrote:
>>
>>
>>
>>      On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly
>>      <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>          On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
>>
>>
>>              Earlier I was focusing only on DMAR errors and I might have
>>              said 'it worked' when I didn't notice them on host when dpdk
>>              was started on guest. When trying to send packets out of
>>              that interface from guest I did see DMAR errors. I am
>>              attaching information you requested.  I have enabled
>>              log-level=8 and files contain dpdk EAL/PMD logs as well.
>>
>>
>>          Great, now we're on the same page.
>>
>>
>>              Snippets below
>>
>>              on host, DMAR fault address from dmesg
>>
>>              [351576.998109] DMAR: DRHD: handling fault status reg 702
>>              [351576.998113] DMAR: [DMA Read] Request device [04:10.0]
>>              fault addr 257617000 [fault reason 06] PTE Read access is
>>              not set
>>
>>              on guest (dump phys_mem_layout)
>>
>>              Segment 235: phys:0x257600000, len:2097152,
>>              virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152,
>>              nchannel:0, nrank:0
>>              ...
>>              PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
>>              sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600
>>              dma_addr=0x257617600
>>              PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
>>              sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80
>>              dma_addr=0x25406fe80
>>              ...
>>
>>
>>          To me this looks like host (i.e. either QEMU or the PF driver)
>>          is trying to do DMA using guest-physical (and not
>>          host-physical). I'm not too well-versed in how QEMU works, but
>>          i'm pretty sure that's not supposed to happen.
>>
>>          Is PF also bound to DPDK, or are you using native Linux ixgbe
>>          driver?
>>
>>
>>      Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF
>>      interfaces disappear after it is bound to DPDK. If there is a way to
>>      use PF and VF with DPDK let me know I can try it out. I am not sure
>>      how to move forward on this, Is CPU/IXGBE PF driver playing a role?
>>      Following are the versions I have
>>
>>      lscpu
>>      Architecture:          x86_64
>>      CPU op-mode(s):        32-bit, 64-bit
>>      Byte Order:            Little Endian
>>      CPU(s):                56
>>      On-line CPU(s) list:   0-27
>>      Off-line CPU(s) list:  28-55
>>      Thread(s) per core:    1
>>      Core(s) per socket:    14
>>      Socket(s):             2
>>      NUMA node(s):          2
>>      Vendor ID:             GenuineIntel
>>      CPU family:            6
>>      Model:                 63
>>      Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
>>      Stepping:              2
>>      CPU MHz:               2500.610
>>      CPU max MHz:           3000.0000
>>      CPU min MHz:           1200.0000
>>      BogoMIPS:              4000.74
>>      Virtualization:        VT-x
>>      L1d cache:             32K
>>      L1i cache:             32K
>>      L2 cache:              256K
>>      L3 cache:              35840K
>>      NUMA node0 CPU(s):     0-13
>>      NUMA node1 CPU(s):     14-27
>>
>>      # ethtool -i enp4s0f0
>>      driver: ixgbe
>>      version: 5.3.3
>>      firmware-version: 0x800007b8, 1.1018.0
>>      bus-info: 0000:04:00.0
>>      supports-statistics: yes
>>      supports-test: yes
>>      supports-eeprom-access: yes
>>      supports-register-dump: yes
>>      supports-priv-flags: yes
>>
>>      Thanks,
>>      Ravi
>>
>>
>>
>> Debugging this I could co-relate doing interface link-up associated 
>> with the dpdk inside the guest causes DMAR errors on host and an 
>> additional vflr message.
>>
>> [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [ 
>> 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074] 
>> ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [ 
>> 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0
>>
>> Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from 
>> 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is 
>> it possible that dpdk pmd vf driver is missing some fixes/porting 
>> from ixgbevf driver since this issue is not seen when ixgbevf kernel 
>> driver is used?
>>
> 
> Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue.
> 
> --
> Thanks,
> Anatoly
> 

You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76

There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM.

--
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2019-01-15 11:22                               ` Burakov, Anatoly
  2019-01-15 13:07                                 ` Hu, Xuekun
@ 2019-01-21 13:18                                 ` Hu, Xuekun
  2019-01-21 13:39                                   ` Burakov, Anatoly
  1 sibling, 1 reply; 33+ messages in thread
From: Hu, Xuekun @ 2019-01-21 13:18 UTC (permalink / raw)
  To: Burakov, Anatoly, Ravi Kerur; +Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo


> You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76

> There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM.

Thanks, Anatoly. By updating host kernel to 4.18, the issue is gone. 😊 

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2019-01-21 13:18                                 ` Hu, Xuekun
@ 2019-01-21 13:39                                   ` Burakov, Anatoly
  2019-01-21 14:44                                     ` Thomas Monjalon
  0 siblings, 1 reply; 33+ messages in thread
From: Burakov, Anatoly @ 2019-01-21 13:39 UTC (permalink / raw)
  To: Hu, Xuekun, Ravi Kerur, Thomas Monjalon
  Cc: dev, Ananyev, Konstantin, Lu, Wenzhuo

On 21-Jan-19 1:18 PM, Hu, Xuekun wrote:
> 
>> You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76
> 
>> There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM.
> 
> Thanks, Anatoly. By updating host kernel to 4.18, the issue is gone. 😊
> 

@Thomas, does this look like something we should add to known issues? 
This question has been asked a few times already.

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 33+ messages in thread

* Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
  2019-01-21 13:39                                   ` Burakov, Anatoly
@ 2019-01-21 14:44                                     ` Thomas Monjalon
  0 siblings, 0 replies; 33+ messages in thread
From: Thomas Monjalon @ 2019-01-21 14:44 UTC (permalink / raw)
  To: Burakov, Anatoly
  Cc: Hu, Xuekun, Ravi Kerur, dev, Ananyev, Konstantin, Lu, Wenzhuo

21/01/2019 14:39, Burakov, Anatoly:
> On 21-Jan-19 1:18 PM, Hu, Xuekun wrote:
> > 
> >> You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76
> > 
> >> There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM.
> > 
> > Thanks, Anatoly. By updating host kernel to 4.18, the issue is gone. 😊
> > 
> 
> @Thomas, does this look like something we should add to known issues? 
> This question has been asked a few times already.

Yes sure

^ permalink raw reply	[flat|nested] 33+ messages in thread

end of thread, other threads:[~2019-01-21 14:45 UTC | newest]

Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-23 17:25 [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue Ravi Kerur
2018-01-24 10:31 ` Burakov, Anatoly
2018-01-24 19:13   ` Ravi Kerur
2018-01-25 10:49     ` Burakov, Anatoly
2018-01-29 22:35       ` Ravi Kerur
2018-01-31  9:59         ` Burakov, Anatoly
2018-01-31 21:51           ` Ravi Kerur
2018-02-01 10:10             ` Burakov, Anatoly
2018-02-01 19:26               ` Ravi Kerur
2018-02-02 10:28                 ` Burakov, Anatoly
2018-02-02 20:21                   ` Ravi Kerur
2018-02-02 20:51                     ` Ravi Kerur
2018-02-05 10:01                       ` Burakov, Anatoly
2018-02-06 17:55                         ` Ravi Kerur
2018-02-08 11:20                           ` Burakov, Anatoly
2018-02-09 17:41                             ` Ravi Kerur
2018-02-10 10:11                               ` Burakov, Anatoly
2018-02-10 10:58         ` Burakov, Anatoly
2018-02-10 17:53           ` Ravi Kerur
2018-02-12 10:13             ` Burakov, Anatoly
2018-02-12 22:00               ` Ravi Kerur
2018-02-13 14:31                 ` Burakov, Anatoly
2018-02-14 20:00                   ` Ravi Kerur
2018-02-15 10:28                     ` Burakov, Anatoly
2018-02-15 18:27                       ` Ravi Kerur
2018-02-15 20:53                         ` Ravi Kerur
2018-02-16  9:41                           ` Burakov, Anatoly
2019-01-15  7:07                             ` Hu, Xuekun
2019-01-15 11:22                               ` Burakov, Anatoly
2019-01-15 13:07                                 ` Hu, Xuekun
2019-01-21 13:18                                 ` Hu, Xuekun
2019-01-21 13:39                                   ` Burakov, Anatoly
2019-01-21 14:44                                     ` Thomas Monjalon

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).