Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Ravi Kerur <rkerur@gmail.com>]

On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rkerur@gmail.com> wrote:

>
>
> On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly <
> anatoly.burakov@intel.com> wrote:
>
>> On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
>>
>>>
>>>
>>> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly <
>>> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>>
>>>     On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
>>>
>>>
>>>         Hi Anatoly,
>>>
>>>         Thanks. I am following wiki link below which uses vIOMMU with
>>>         DPDK as a use-case and instantiate VM as specified with Q35
>>>         chipset in Qemu.
>>>
>>>         https://wiki.qemu.org/Features/VT-d
>>>         <https://wiki.qemu.org/Features/VT-d>
>>>
>>>         Qemu-version is 2.11
>>>         Host kernel 4.9
>>>         Guest kernel 4.4
>>>
>>>         I can only guess that guest kernel needs an upgrade in my setup
>>>         to work correctly, if versions on my setup rings a bell on not
>>>         having support kindly let me know.
>>>
>>>         When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on
>>>         guest I get following error
>>>         ...
>>>         vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored
>>>         ...
>>>
>>>         in guest.
>>>
>>>         Thanks.
>>>
>>>
>>>     AFAIK kernel 4.4 should have noiommu mode - it was introduced in
>>>     3.1x days. However, in order for that to work, kernel also has to be
>>>     built with this mode enabled. My guess is, whoever is the supplier
>>>     of your kernel, did not do that. You should double-check the kernel
>>>     configuration of your distribution.
>>>
>>>     However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode
>>>     - "regular" vfio should work fine. noiommu mode should only be
>>>     needed if you know you don't have IOMMU enabled in your kernel, and
>>>     even if you can't enable it, you can still use igb_uio.
>>>
>>> Hi Anatoly,
>>>
>>> Do you suggest I take this discussion to kvm/qemu mailing list as I am
>>> not sure which component has the issue? I check dmesg for BIOS physical
>>> memory map and address reported as fault by DMAR is reported by BIOS as
>>> usable on both host and vm.
>>>
>>> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr
>>> *33a128000 *[fault reason 06] PTE Read access is not set
>>>
>>> dmesg | grep BIOS
>>> [    0.000000] e820: BIOS-provided physical RAM map:
>>> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff]
>>> usable
>>> [    0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff]
>>> usable
>>> [    0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff]
>>> ACPI data
>>> [    0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff]
>>> ACPI NVS
>>> [    0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff]
>>> usable
>>> [    0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff]
>>> usable
>>> [    0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff]
>>> reserved
>>> [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff]
>>> reserved
>>> [*    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff]
>>> usable*
>>> *
>>> *
>>> Kindly let me know your inputs.
>>>
>>> Thanks.
>>>
>>>
>>>     --     Thanks,
>>>     Anatoly
>>>
>>>
>>>
>> The "PTE Read not set" error usually indicates that you are trying to use
>> a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio
>> when IOMMU is on). That, to me, indicates that you do have IOMMU emulation
>> enabled.
>>
>> I would go about it this way.
>>
>> First, i'd ensure that your VM has IOMMU emulation enabled and working.
>> You have mentioned that your QEMU version should have IOMMU emulation, so
>> let's assume that's the case.
>>
>> I am not sure of the exact command-line needed to activate the vIOMMU
>> emulation, but assuming your VM emulates an Intel processor, your kernel
>> command-line should have "iommu=on intel_iommu=on" in it. Check
>> /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above
>> values are not in there, add the above changes, do "update-grub" and reboot
>> your VM.
>>
>> If it already did have the necessary kernel configuration, do "dmesg |
>> grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU
>> is enabled and working in the kernel.
>>
>> After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it
>> should be working. Please bear in mind that all of that is how i would've
>> gone about it if i had similar problems on baremetal, but i'm hoping all of
>> it is applicable to VM's. So, either disable IOMMU and use igb_uio, or
>> enable IOMMU and use VFIO. Both will work.
>>
>>
> Thanks for your help so far on this. I have everything enabled as you have
> mentioned in your email. Below is the error from EAL when DPDK is run on a
> VM. I am referring EAL code to check why it is failing.
>
> ************VM DPDK errors***********
> EAL: Detected 14 lcore(s)
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> unreliable clock cycles !
> EAL: PCI device 0000:00:07.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 8086:10ed net_ixgbe_vf
> EAL:   using IOMMU type 1 (Type 1)
>
>
> *EAL:   RTE_IOVA_VA 140051367329792EAL:   cannot set up DMA remapping,
> error 14 (Bad address)EAL:   0000:00:07.0 DMA remapping failed, error 14
> (Bad address)*
> EAL: Requested device 0000:00:07.0 cannot be used
> EAL: PCI device 0000:03:00.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 1af4:1041 net_virtio
> EAL: PCI device 0000:04:00.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 1af4:1041 net_virtio
> EAL: PCI device 0000:05:00.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 1af4:1041 net_virtio
> EAL: No probed ethernet devices
> Interactive-mode selected
> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176,
> socket=0
>
>
> ********************VM************
> root@sriov-ravi-01:~# lsmod | grep vfio
>
>
>
>
> *vfio_pci               40960  1vfio_virqfd            16384  1
> vfio_pcivfio_iommu_type1       20480  1vfio                   28672  5
> vfio_iommu_type1,vfio_pciirqbypass              16384  3 kvm,vfio_pci*
> root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU
> [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR
> 00000001 BXPC 00000001)
> *[    0.000000] DMAR: IOMMU enabled*
> [    9.423839] DMAR: Host address width 39
> [    9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
> [    9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
> 12008c22260286 ecap f00f5e
> [    9.427314] DMAR: ATSR flags: 0x1
> [    9.428047] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU 0
> [    9.429255] DMAR-IR: Queued invalidation will be enabled to support
> x2apic and Intr-remapping.
> [    9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [   11.167334] DMAR: No RMRR found
> [   11.168855] DMAR: dmar0: Using Queued invalidation
> [   12.119060] DMAR: Setting RMRR:
> [   12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC
> [   12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
> 0xffffff]
> [   12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O
> root@sriov-ravi-01:~# dpdk-devbind -s
>
> Network devices using DPDK-compatible driver
> ============================================
> *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed'
> drv=vfio-pci unused=*
>
> Network devices using kernel driver
> ===================================
> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
>
> Other Network devices
> =====================
> <none>
>
> Crypto devices using DPDK-compatible driver
> ===========================================
> <none>
>
> Crypto devices using kernel driver
> ==================================
> <none>
>
> Other Crypto devices
> ====================
> <none>
>
> Eventdev devices using DPDK-compatible driver
> =============================================
> <none>
>
> Eventdev devices using kernel driver
> ====================================
> <none>
>
> Other Eventdev devices
> ======================
> <none>
>
> Mempool devices using DPDK-compatible driver
> ============================================
> <none>
>
> Mempool devices using kernel driver
> ===================================
> <none>
>
> Other Mempool devices
> =====================
> <none>
> root@sriov-ravi-01:~#
>
> root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i
> vfio
> [   77.676711] VFIO - User Level meta-driver version: 0.3
> [  149.013806] VFIO - User Level meta-driver version: 0.3
> [  275.876440] VFIO - User Level meta-driver version: 0.3
>
> Thanks,
> Ravi
>
>
>
Additonal information. On VM following message is seen

[  298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient
for the mapped address (7f7300000000)
[ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient
for the mapped address (7fa580000000)

ON VM, DMAR reports address width = 39
root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -e
IOMMU -e DMAR
[    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR
00000001 BXPC 00000001)
[    0.000000] DMAR: IOMMU enabled
[    9.209145] DMAR: Host address width 39

On Host, DMAR reports address width = 46
# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
00000001 INTL 20091013)
[    0.000000] DMAR: IOMMU enabled
[    0.519368] DMAR: Host address width 46
[    0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0

Searching for IOMMU errors found following link from RedHat which points to
DPDK 17.11 issue.

https://bugzilla.redhat.com/show_bug.cgi?id=1530957

Thanks.

> --
>> Thanks,
>> Anatoly
>>
>
>