Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

["Hu, Xuekun" <xuekun.hu@intel.com>]

Thanks, I'm just using CentOS7.5 3.10!
I will try a newer kernel. Stay tuned.  

-----Original Message-----
From: Burakov, Anatoly 
Sent: Tuesday, January 15, 2019 7:22 PM
To: Hu, Xuekun <xuekun.hu@intel.com>; Ravi Kerur <rkerur@gmail.com>
Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Lu, Wenzhuo <wenzhuo.lu@intel.com>
Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

On 15-Jan-19 7:07 AM, Hu, Xuekun wrote:
> Hi, Ravi
> 
> Did you resolve this issue that VF used in guest with vIOMMU enabled? I googled, but still can't get the answer that it is driver or qemu vt-d emulation issue.
> 
> Currently I met the same issue again that host reported DMAR error:
> [59939.130110] DMAR: DRHD: handling fault status reg 2 [59939.130116] 
> DMAR: [DMA Read] Request device [83:10.0] fault addr 15f03d000 [fault 
> reason 06] PTE Read access is not set [59940.180859] ixgbe 
> 0000:83:00.0: Issuing VFLR with pending transactions [59940.180863] 
> ixgbe 0000:83:00.0: Issuing VFLR for VF 0000:83:10.0 [59989.344683] 
> ixgbe 0000:83:00.0 ens817f0: VF Reset msg received from vf 0
> 
> I'm using DPDK 18.11 in guest, and ixgbe.ko in host.
> 
> Thx, Xuekun
> 
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Burakov, Anatoly
> Sent: Friday, February 16, 2018 5:42 PM
> To: Ravi Kerur <rkerur@gmail.com>
> Cc: dev@dpdk.org; Ananyev, Konstantin <konstantin.ananyev@intel.com>; 
> Lu, Wenzhuo <wenzhuo.lu@intel.com>
> Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue
> 
> On 15-Feb-18 8:53 PM, Ravi Kerur wrote:
>>
>>
>> On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com 
>> <mailto:rkerur@gmail.com>> wrote:
>>
>>
>>
>>      On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly
>>      <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>>
>>          On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
>>
>>
>>              Earlier I was focusing only on DMAR errors and I might have
>>              said 'it worked' when I didn't notice them on host when dpdk
>>              was started on guest. When trying to send packets out of
>>              that interface from guest I did see DMAR errors. I am
>>              attaching information you requested.  I have enabled
>>              log-level=8 and files contain dpdk EAL/PMD logs as well.
>>
>>
>>          Great, now we're on the same page.
>>
>>
>>              Snippets below
>>
>>              on host, DMAR fault address from dmesg
>>
>>              [351576.998109] DMAR: DRHD: handling fault status reg 702
>>              [351576.998113] DMAR: [DMA Read] Request device [04:10.0]
>>              fault addr 257617000 [fault reason 06] PTE Read access is
>>              not set
>>
>>              on guest (dump phys_mem_layout)
>>
>>              Segment 235: phys:0x257600000, len:2097152,
>>              virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152,
>>              nchannel:0, nrank:0
>>              ...
>>              PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
>>              sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600
>>              dma_addr=0x257617600
>>              PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
>>              sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80
>>              dma_addr=0x25406fe80
>>              ...
>>
>>
>>          To me this looks like host (i.e. either QEMU or the PF driver)
>>          is trying to do DMA using guest-physical (and not
>>          host-physical). I'm not too well-versed in how QEMU works, but
>>          i'm pretty sure that's not supposed to happen.
>>
>>          Is PF also bound to DPDK, or are you using native Linux ixgbe
>>          driver?
>>
>>
>>      Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF
>>      interfaces disappear after it is bound to DPDK. If there is a way to
>>      use PF and VF with DPDK let me know I can try it out. I am not sure
>>      how to move forward on this, Is CPU/IXGBE PF driver playing a role?
>>      Following are the versions I have
>>
>>      lscpu
>>      Architecture:          x86_64
>>      CPU op-mode(s):        32-bit, 64-bit
>>      Byte Order:            Little Endian
>>      CPU(s):                56
>>      On-line CPU(s) list:   0-27
>>      Off-line CPU(s) list:  28-55
>>      Thread(s) per core:    1
>>      Core(s) per socket:    14
>>      Socket(s):             2
>>      NUMA node(s):          2
>>      Vendor ID:             GenuineIntel
>>      CPU family:            6
>>      Model:                 63
>>      Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
>>      Stepping:              2
>>      CPU MHz:               2500.610
>>      CPU max MHz:           3000.0000
>>      CPU min MHz:           1200.0000
>>      BogoMIPS:              4000.74
>>      Virtualization:        VT-x
>>      L1d cache:             32K
>>      L1i cache:             32K
>>      L2 cache:              256K
>>      L3 cache:              35840K
>>      NUMA node0 CPU(s):     0-13
>>      NUMA node1 CPU(s):     14-27
>>
>>      # ethtool -i enp4s0f0
>>      driver: ixgbe
>>      version: 5.3.3
>>      firmware-version: 0x800007b8, 1.1018.0
>>      bus-info: 0000:04:00.0
>>      supports-statistics: yes
>>      supports-test: yes
>>      supports-eeprom-access: yes
>>      supports-register-dump: yes
>>      supports-priv-flags: yes
>>
>>      Thanks,
>>      Ravi
>>
>>
>>
>> Debugging this I could co-relate doing interface link-up associated 
>> with the dpdk inside the guest causes DMAR errors on host and an 
>> additional vflr message.
>>
>> [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [ 
>> 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr
>> 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074] 
>> ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [ 
>> 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0
>>
>> Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from 
>> 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is 
>> it possible that dpdk pmd vf driver is missing some fixes/porting 
>> from ixgbevf driver since this issue is not seen when ixgbevf kernel 
>> driver is used?
>>
> 
> Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue.
> 
> --
> Thanks,
> Anatoly
> 

You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76

There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM.

--
Thanks,
Anatoly