Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Ravi Kerur <rkerur@gmail.com>]

On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
>
>>
>>
>> On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly <
>> anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>     On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
>>
>>
>>         Hi Anatoly,
>>
>>         Thanks. I am following wiki link below which uses vIOMMU with
>>         DPDK as a use-case and instantiate VM as specified with Q35
>>         chipset in Qemu.
>>
>>         https://wiki.qemu.org/Features/VT-d
>>         <https://wiki.qemu.org/Features/VT-d>
>>
>>         Qemu-version is 2.11
>>         Host kernel 4.9
>>         Guest kernel 4.4
>>
>>         I can only guess that guest kernel needs an upgrade in my setup
>>         to work correctly, if versions on my setup rings a bell on not
>>         having support kindly let me know.
>>
>>         When 'modprobe vfio enable_unsafe_noiommu_node=Y' is executed on
>>         guest I get following error
>>         ...
>>         vfio: unknown parameter 'enable_unsafe_noiommu_node' ignored
>>         ...
>>
>>         in guest.
>>
>>         Thanks.
>>
>>
>>     AFAIK kernel 4.4 should have noiommu mode - it was introduced in
>>     3.1x days. However, in order for that to work, kernel also has to be
>>     built with this mode enabled. My guess is, whoever is the supplier
>>     of your kernel, did not do that. You should double-check the kernel
>>     configuration of your distribution.
>>
>>     However, if you have vIOMMU in QEMU, you shouldn't need noiommu mode
>>     - "regular" vfio should work fine. noiommu mode should only be
>>     needed if you know you don't have IOMMU enabled in your kernel, and
>>     even if you can't enable it, you can still use igb_uio.
>>
>> Hi Anatoly,
>>
>> Do you suggest I take this discussion to kvm/qemu mailing list as I am
>> not sure which component has the issue? I check dmesg for BIOS physical
>> memory map and address reported as fault by DMAR is reported by BIOS as
>> usable on both host and vm.
>>
>> [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> *33a128000 *[fault reason 06] PTE Read access is not set
>>
>> dmesg | grep BIOS
>> [    0.000000] e820: BIOS-provided physical RAM map:
>> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009afff]
>> usable
>> [    0.000000] BIOS-e820: [mem 0x000000000009b000-0x000000000009ffff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007938afff]
>> usable
>> [    0.000000] BIOS-e820: [mem 0x000000007938b000-0x000000007994bfff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x000000007994c000-0x000000007999cfff]
>> ACPI data
>> [    0.000000] BIOS-e820: [mem 0x000000007999d000-0x0000000079f7dfff]
>> ACPI NVS
>> [    0.000000] BIOS-e820: [mem 0x0000000079f7e000-0x000000007bd37fff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x000000007bd38000-0x000000007bd38fff]
>> usable
>> [    0.000000] BIOS-e820: [mem 0x000000007bd39000-0x000000007bdbefff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x000000007bdbf000-0x000000007bffffff]
>> usable
>> [    0.000000] BIOS-e820: [mem 0x000000007c000000-0x000000008fffffff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff]
>> reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff]
>> reserved
>> [*    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000407fffffff]
>> usable*
>> *
>> *
>> Kindly let me know your inputs.
>>
>> Thanks.
>>
>>
>>     --     Thanks,
>>     Anatoly
>>
>>
>>
> The "PTE Read not set" error usually indicates that you are trying to use
> a non-IOMMU method when you have IOMMU enabled (i.e. trying to use igb_uio
> when IOMMU is on). That, to me, indicates that you do have IOMMU emulation
> enabled.
>
> I would go about it this way.
>
> First, i'd ensure that your VM has IOMMU emulation enabled and working.
> You have mentioned that your QEMU version should have IOMMU emulation, so
> let's assume that's the case.
>
> I am not sure of the exact command-line needed to activate the vIOMMU
> emulation, but assuming your VM emulates an Intel processor, your kernel
> command-line should have "iommu=on intel_iommu=on" in it. Check
> /etc/default/grub for GRUB_CMDLINE_LINUX_DEFAULT value, and if the above
> values are not in there, add the above changes, do "update-grub" and reboot
> your VM.
>
> If it already did have the necessary kernel configuration, do "dmesg |
> grep IOMMU" and look for "IOMMU Enabled". That should tell you that IOMMU
> is enabled and working in the kernel.
>
> After that, you can modprobe vfio and vfio-pci, bind NICs to it, and it
> should be working. Please bear in mind that all of that is how i would've
> gone about it if i had similar problems on baremetal, but i'm hoping all of
> it is applicable to VM's. So, either disable IOMMU and use igb_uio, or
> enable IOMMU and use VFIO. Both will work.
>
>
Thanks for your help so far on this. I have everything enabled as you have
mentioned in your email. Below is the error from EAL when DPDK is run on a
VM. I am referring EAL code to check why it is failing.

************VM DPDK errors***********
EAL: Detected 14 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable
clock cycles !
EAL: PCI device 0000:00:07.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10ed net_ixgbe_vf
EAL:   using IOMMU type 1 (Type 1)


*EAL:   RTE_IOVA_VA 140051367329792EAL:   cannot set up DMA remapping,
error 14 (Bad address)EAL:   0000:00:07.0 DMA remapping failed, error 14
(Bad address)*
EAL: Requested device 0000:00:07.0 cannot be used
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
EAL: No probed ethernet devices
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176,
socket=0


********************VM************
root@sriov-ravi-01:~# lsmod | grep vfio




*vfio_pci               40960  1vfio_virqfd            16384  1
vfio_pcivfio_iommu_type1       20480  1vfio                   28672  5
vfio_iommu_type1,vfio_pciirqbypass              16384  3 kvm,vfio_pci*
root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR
00000001 BXPC 00000001)
*[    0.000000] DMAR: IOMMU enabled*
[    9.423839] DMAR: Host address width 39
[    9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
12008c22260286 ecap f00f5e
[    9.427314] DMAR: ATSR flags: 0x1
[    9.428047] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU 0
[    9.429255] DMAR-IR: Queued invalidation will be enabled to support
x2apic and Intr-remapping.
[    9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode
[   11.167334] DMAR: No RMRR found
[   11.168855] DMAR: dmar0: Using Queued invalidation
[   12.119060] DMAR: Setting RMRR:
[   12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC
[   12.124787] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
0xffffff]
[   12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O
root@sriov-ravi-01:~# dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
*0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed'
drv=vfio-pci unused=*

Network devices using kernel driver
===================================
0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci

Other Network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other Crypto devices
====================
<none>

Eventdev devices using DPDK-compatible driver
=============================================
<none>

Eventdev devices using kernel driver
====================================
<none>

Other Eventdev devices
======================
<none>

Mempool devices using DPDK-compatible driver
============================================
<none>

Mempool devices using kernel driver
===================================
<none>

Other Mempool devices
=====================
<none>
root@sriov-ravi-01:~#

root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep -i
vfio
[   77.676711] VFIO - User Level meta-driver version: 0.3
[  149.013806] VFIO - User Level meta-driver version: 0.3
[  275.876440] VFIO - User Level meta-driver version: 0.3

Thanks,
Ravi


-- 
> Thanks,
> Anatoly
>