Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Ravi Kerur <rkerur@gmail.com>]

On Sat, Feb 10, 2018 at 2:58 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 29-Jan-18 10:35 PM, Ravi Kerur wrote:
>
>> Hi Burakov,
>>
>> When using vfio-pci on host both VF and PF interfaces works fine with
>> dpdk i.e. I don't see DMAR fault messages anymore. However, when I attach a
>> VF interface to a VM and start DPDK with vfio-pci inside VM I still see
>> DMAR fault messages on host. Both host and VM are booted with
>> 'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work (I
>> think it's expected because of DMAR faults), however, when VF interface
>> uses ixgbevf driver ping works.
>>
>> Following are some details
>>
>> /*****************On VM***************/
>> dpdk-devbind -s
>>
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci
>> unused=ixgbevf
>>
>> Network devices using kernel driver
>> ===================================
>> 0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
>> 0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
>> 0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
>>
>> Other network devices
>> =====================
>> <none>
>>
>> Crypto devices using DPDK-compatible driver
>> ===========================================
>> <none>
>>
>> Crypto devices using kernel driver
>> ==================================
>> <none>
>>
>> Other crypto devices
>> ====================
>> <none>
>>
>>
>> 00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
>> Virtual Function (rev 01)
>>          Subsystem: Intel Corporation 82599 Ethernet Controller Virtual
>> Function
>>          Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
>> ParErr- Stepping- SERR- FastB2B- DisINTx+
>>          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>          Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K]
>>          Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K]
>>          Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
>>                  Vector table: BAR=3 offset=00000000
>>                  PBA: BAR=3 offset=00002000
>>          Capabilities: [a0] Express (v1) Root Complex Integrated
>> Endpoint, MSI 00
>>                  DevCap: MaxPayload 128 bytes, PhantFunc 0
>>                          ExtTag- RBE-
>>                  DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                          RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>>                          MaxPayload 128 bytes, MaxReadReq 128 bytes
>>                  DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>> TransPend-
>>          Capabilities: [100 v1] Advanced Error Reporting
>>                  UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
>> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                  UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
>> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                  UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
>> UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                  CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>> NonFatalErr-
>>                  CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
>> NonFatalErr-
>>                  AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap-
>> ChkEn-
>>          Kernel driver in use: vfio-pci
>>          Kernel modules: ixgbevf
>>
>> /***************on Host*************/
>> dmesg | grep DMAR
>> ...
>> [  978.268143] DMAR: DRHD: handling fault status reg 2
>> [  978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>> [ 1286.677726] DMAR: DRHD: handling fault status reg 102
>> [ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> fb663000 [fault reason 06] PTE Read access is not set
>> [ 1676.436145] DMAR: DRHD: handling fault status reg 202
>> [ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>> [ 1734.433649] DMAR: DRHD: handling fault status reg 302
>> [ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>> [ 2324.428938] DMAR: DRHD: handling fault status reg 402
>> [ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 7770c000 [fault reason 06] PTE Read access is not set
>> [ 2388.553640] DMAR: DRHD: handling fault status reg 502
>> [ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
>> 33a128000 [fault reason 06] PTE Read access is not set
>>
>>
>>
> Going back to this, i would like to suggest run a few tests to ensure that
> we have all information that we can gather.
>
> First of all, i'm assuming that you're using native ixgbe Linux driver on
> the host, and that you're only passing through the VF device to the VM
> using VFIO. Is my understanding correct here?
>
> Now, let's forget about the iommu=pt and igb_uio for a moment. Boot both
> your host and your VM with iommu=on and intel_iommu=on (or whatever
> command-line enables full IOMMU support on both host and guest) and do the
> same tests you've done before. Do you still see your issues?
>
> It would also be very useful to also try native Linux kernel driver on the
> guest *with traffic forwarding* and see how it works in your VM. Therefore
> i would suggest you to compile DPDK with PCAP support, bind your (VM)
> interface to native Linux driver, and use the interface via our pcap driver
> (creating a vdev should do the trick - please refer to PCAP PMD
> documentation [1]). Simple forwarding test should be enough - just make
> sure to pass traffic to and from DPDK in both cases, and that it doesn't
> give you any DMAR errors.
>
> We can go from there.
>
>
Let me just give you what has been tested and working/nonworking scenarios.
Some of your questions might get answered as well. Test bed is very simple
with 2 VF's created under IXGBE PF on host with one VF interface added to
ovs-bridge on host and another VF interface given to guest. Test
connectivity between VF's via ping.

Host and guest -- Kernel 4.9
Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the git (2.11.50))
DPDK -- 17.05.1 on host and guest
Host and guest -- booted with GRUB intel_iommu=on (which enables IOMMU).
Have tried with "iommu=on and intel_iommu=on" as well, but iommu=on is not
needed when intel_iommu=on is set.

Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver ping works
Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf driver ping
works
Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK vfio-pci driver,
DMAR errors seen on host, ping doesn't work

DPDK works fine on host with vfio-pci, however, has issues when used inside
the guest. Please let me know if more information is needed.

Thanks,
Ravi

[1] http://dpdk.org/doc/guides/nics/pcap_ring.html
>
> --
> Thanks,
> Anatoly
>