Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Ravi Kerur <rkerur@gmail.com>]

On Mon, Feb 5, 2018 at 2:01 AM, Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 02-Feb-18 8:51 PM, Ravi Kerur wrote:
>
>>
>>
>> On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rkerur@gmail.com <mailto:
>> rkerur@gmail.com>> wrote:
>>
>>
>>
>>     On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly
>>     <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>         On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
>>
>>
>>
>>             On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly
>>             <anatoly.burakov@intel.com
>>             <mailto:anatoly.burakov@intel.com>
>>             <mailto:anatoly.burakov@intel.com
>>
>>             <mailto:anatoly.burakov@intel.com>>> wrote:
>>
>>                  On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
>>
>>
>>                      Hi Anatoly,
>>
>>                      Thanks. I am following wiki link below which uses
>>             vIOMMU with
>>                      DPDK as a use-case and instantiate VM as specified
>>             with Q35
>>                      chipset in Qemu.
>>
>>             https://wiki.qemu.org/Features/VT-d
>>             <https://wiki.qemu.org/Features/VT-d>
>>                      <https://wiki.qemu.org/Features/VT-d
>>             <https://wiki.qemu.org/Features/VT-d>>
>>
>>                      Qemu-version is 2.11
>>                      Host kernel 4.9
>>                      Guest kernel 4.4
>>
>>                      I can only guess that guest kernel needs an upgrade
>>             in my setup
>>                      to work correctly, if versions on my setup rings a
>>             bell on not
>>                      having support kindly let me know.
>>
>>                      When 'modprobe vfio enable_unsafe_noiommu_node=Y'
>>             is executed on
>>                      guest I get following error
>>                      ...
>>                      vfio: unknown parameter
>>             'enable_unsafe_noiommu_node' ignored
>>                      ...
>>
>>                      in guest.
>>
>>                      Thanks.
>>
>>
>>                  AFAIK kernel 4.4 should have noiommu mode - it was
>>             introduced in
>>                  3.1x days. However, in order for that to work, kernel
>>             also has to be
>>                  built with this mode enabled. My guess is, whoever is
>>             the supplier
>>                  of your kernel, did not do that. You should
>>             double-check the kernel
>>                  configuration of your distribution.
>>
>>                  However, if you have vIOMMU in QEMU, you shouldn't need
>>             noiommu mode
>>                  - "regular" vfio should work fine. noiommu mode should
>>             only be
>>                  needed if you know you don't have IOMMU enabled in your
>>             kernel, and
>>                  even if you can't enable it, you can still use igb_uio.
>>
>>             Hi Anatoly,
>>
>>             Do you suggest I take this discussion to kvm/qemu mailing
>>             list as I am not sure which component has the issue? I check
>>             dmesg for BIOS physical memory map and address reported as
>>             fault by DMAR is reported by BIOS as usable on both host and
>> vm.
>>
>>             [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0]
>>             fault addr *33a128000 *[fault reason 06] PTE Read access is
>>             not set
>>
>>             dmesg | grep BIOS
>>             [    0.000000] e820: BIOS-provided physical RAM map:
>>             [    0.000000] BIOS-e820: [mem
>>             0x0000000000000000-0x000000000009afff] usable
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000000009b000-0x000000000009ffff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x00000000000e0000-0x00000000000fffff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x0000000000100000-0x000000007938afff] usable
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007938b000-0x000000007994bfff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007994c000-0x000000007999cfff] ACPI data
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007999d000-0x0000000079f7dfff] ACPI NVS
>>             [    0.000000] BIOS-e820: [mem
>>             0x0000000079f7e000-0x000000007bd37fff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007bd38000-0x000000007bd38fff] usable
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007bd39000-0x000000007bdbefff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007bdbf000-0x000000007bffffff] usable
>>             [    0.000000] BIOS-e820: [mem
>>             0x000000007c000000-0x000000008fffffff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x00000000fed1c000-0x00000000fed44fff] reserved
>>             [    0.000000] BIOS-e820: [mem
>>             0x00000000ff000000-0x00000000ffffffff] reserved
>>             [*    0.000000] BIOS-e820: [mem
>>             0x0000000100000000-0x000000407fffffff] usable*
>>             *
>>             *
>>             Kindly let me know your inputs.
>>
>>             Thanks.
>>
>>
>>                  --     Thanks,
>>                  Anatoly
>>
>>
>>
>>         The "PTE Read not set" error usually indicates that you are
>>         trying to use a non-IOMMU method when you have IOMMU enabled
>>         (i.e. trying to use igb_uio when IOMMU is on). That, to me,
>>         indicates that you do have IOMMU emulation enabled.
>>
>>         I would go about it this way.
>>
>>         First, i'd ensure that your VM has IOMMU emulation enabled and
>>         working. You have mentioned that your QEMU version should have
>>         IOMMU emulation, so let's assume that's the case.
>>
>>         I am not sure of the exact command-line needed to activate the
>>         vIOMMU emulation, but assuming your VM emulates an Intel
>>         processor, your kernel command-line should have "iommu=on
>>         intel_iommu=on" in it. Check /etc/default/grub for
>>         GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are
>>         not in there, add the above changes, do "update-grub" and reboot
>>         your VM.
>>
>>         If it already did have the necessary kernel configuration, do
>>         "dmesg | grep IOMMU" and look for "IOMMU Enabled". That should
>>         tell you that IOMMU is enabled and working in the kernel.
>>
>>         After that, you can modprobe vfio and vfio-pci, bind NICs to it,
>>         and it should be working. Please bear in mind that all of that
>>         is how i would've gone about it if i had similar problems on
>>         baremetal, but i'm hoping all of it is applicable to VM's. So,
>>         either disable IOMMU and use igb_uio, or enable IOMMU and use
>>         VFIO. Both will work.
>>
>>
>>     Thanks for your help so far on this. I have everything enabled as
>>     you have mentioned in your email. Below is the error from EAL when
>>     DPDK is run on a VM. I am referring EAL code to check why it is
>> failing.
>>
>>     ************VM DPDK errors***********
>>     EAL: Detected 14 lcore(s)
>>     EAL: Probing VFIO support...
>>     EAL: VFIO support initialized
>>     EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
>>     unreliable clock cycles !
>>     EAL: PCI device 0000:00:07.0 on NUMA socket -1
>>     EAL:   Invalid NUMA socket, default to 0
>>     EAL:   probe driver: 8086:10ed net_ixgbe_vf
>>     EAL:   using IOMMU type 1 (Type 1)
>>     *EAL:   RTE_IOVA_VA 140051367329792
>>     EAL:   cannot set up DMA remapping, error 14 (Bad address)
>>     EAL:   0000:00:07.0 DMA remapping failed, error 14 (Bad address)*
>>     EAL: Requested device 0000:00:07.0 cannot be used
>>     EAL: PCI device 0000:03:00.0 on NUMA socket -1
>>     EAL:   Invalid NUMA socket, default to 0
>>     EAL:   probe driver: 1af4:1041 net_virtio
>>     EAL: PCI device 0000:04:00.0 on NUMA socket -1
>>     EAL:   Invalid NUMA socket, default to 0
>>     EAL:   probe driver: 1af4:1041 net_virtio
>>     EAL: PCI device 0000:05:00.0 on NUMA socket -1
>>     EAL:   Invalid NUMA socket, default to 0
>>     EAL:   probe driver: 1af4:1041 net_virtio
>>     EAL: No probed ethernet devices
>>     Interactive-mode selected
>>     USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
>>     size=2176, socket=0
>>
>>
>>     ********************VM************
>>     root@sriov-ravi-01:~# lsmod | grep vfio
>>     *vfio_pci               40960  1
>>     vfio_virqfd            16384  1 vfio_pci
>>     vfio_iommu_type1       20480  1
>>     vfio                   28672  5 vfio_iommu_type1,vfio_pci
>>     irqbypass              16384  3 kvm,vfio_pci*
>>     root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU
>>     [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS
>>  BXPCDMAR 00000001 BXPC 00000001)
>>     *[    0.000000] DMAR: IOMMU enabled*
>>     [    9.423839] DMAR: Host address width 39
>>     [    9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
>>     [    9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
>>     12008c22260286 ecap f00f5e
>>     [    9.427314] DMAR: ATSR flags: 0x1
>>     [    9.428047] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU
>> 0
>>     [    9.429255] DMAR-IR: Queued invalidation will be enabled to
>>     support x2apic and Intr-remapping.
>>     [    9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode
>>     [   11.167334] DMAR: No RMRR found
>>     [   11.168855] DMAR: dmar0: Using Queued invalidation
>>     [   12.119060] DMAR: Setting RMRR:
>>     [   12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC
>>     [   12.124787] DMAR: Setting identity map for device 0000:00:1f.0
>>     [0x0 - 0xffffff]
>>     [   12.392826] DMAR: Intel(R) Virtualization Technology for Directed
>> I/O
>>     root@sriov-ravi-01:~# dpdk-devbind -s
>>
>>     Network devices using DPDK-compatible driver
>>     ============================================
>>     *0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed'
>>     drv=vfio-pci unused=*
>>
>>
>>     Network devices using kernel driver
>>     ===================================
>>     0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci
>>     *Active*
>>     0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
>>     0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
>>
>>     Other Network devices
>>     =====================
>>     <none>
>>
>>     Crypto devices using DPDK-compatible driver
>>     ===========================================
>>     <none>
>>
>>     Crypto devices using kernel driver
>>     ==================================
>>     <none>
>>
>>     Other Crypto devices
>>     ====================
>>     <none>
>>
>>     Eventdev devices using DPDK-compatible driver
>>     =============================================
>>     <none>
>>
>>     Eventdev devices using kernel driver
>>     ====================================
>>     <none>
>>
>>     Other Eventdev devices
>>     ======================
>>     <none>
>>
>>     Mempool devices using DPDK-compatible driver
>>     ============================================
>>     <none>
>>
>>     Mempool devices using kernel driver
>>     ===================================
>>     <none>
>>
>>     Other Mempool devices
>>     =====================
>>     <none>
>>     root@sriov-ravi-01:~#
>>
>>     root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg |
>>     grep -i vfio
>>     [   77.676711] VFIO - User Level meta-driver version: 0.3
>>     [  149.013806] VFIO - User Level meta-driver version: 0.3
>>     [  275.876440] VFIO - User Level meta-driver version: 0.3
>>
>>     Thanks,
>>     Ravi
>>
>>
>>
>> Additonal information. On VM following message is seen
>>
>> [  298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient
>> for the mapped address (7f7300000000)
>> [ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient
>> for the mapped address (7fa580000000)
>>
>> ON VM, DMAR reports address width = 39
>> root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep
>> -e IOMMU -e DMAR
>> [    0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS  BXPCDMAR
>> 00000001 BXPC 00000001)
>> [    0.000000] DMAR: IOMMU enabled
>> [    9.209145] DMAR: Host address width 39
>>
>> On Host, DMAR reports address width = 46
>> # dmesg | grep -e DMAR -e IOMMU
>> [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
>> 00000001 INTL 20091013)
>> [    0.000000] DMAR: IOMMU enabled
>> [    0.519368] DMAR: Host address width 46
>> [    0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
>>
>> Searching for IOMMU errors found following link from RedHat which points
>> to DPDK 17.11 issue.
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1530957
>>
>> Thanks.
>>
>>         --         Thanks,
>>         Anatoly
>>
>>
>>
>>
> Ah, yes, that. We have recently applied a patch dealing with exactly this:
>
> http://dpdk.org/dev/patchwork/patch/33650/
>
> Unfortunately, as far as i know, other than refusing to run (like the
> patch does), there's no way to fix this issue, so you'll have to disable
> IOMMU in your VM, and use either igb_uio or vfio in noiommu mode.
>
>
Hi Anatoly,

I am actually confused with the state of vIOMMU + DPDK. Can you please help
me clarify?

I tested following DPDK versions

(1) DPDK 17.11, exhibits the issue (IOMMU width as reported by RedHat and
solution is to prevent using the patch)
(2) DPDK 17.05.02 (stable release) using 'testpmd' I was able to bind a
device in VM with VFIO driver and no DMAR error message on host
(3) DPDK 16.11.02 (stable release) using 'testpmd' I was able to bind a
device in VM with VFIO driver and no DMAR error message on host

Clearly issue seen in 17.11 without the patch you mentioned is a regression
or the issue was masked in earlier DPDK version?  I did not test traffic
with any DPDK version because I wanted to first get DMAR errors on host
gone.

Our application 'v1' is integrated with DPDK 16.11 and 'v2' is integrated
with DPDK 17.05.01. In both 'v1' and 'v2' cases I don't see IOMMU width
error messages on VM, however, DMAR error messages are seen host. I am not
able to relate what causes DMAR error messages on host?


Thanks.

> --
> Thanks,
> Anatoly
>