Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue

[Ravi Kerur <rkerur@gmail.com>]

On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur <rkerur@gmail.com> wrote:

>
>
> On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly <
> anatoly.burakov@intel.com> wrote:
>
>> On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
>>
>>>
>>> Earlier I was focusing only on DMAR errors and I might have said 'it
>>> worked' when I didn't notice them on host when dpdk was started on guest.
>>> When trying to send packets out of that interface from guest I did see DMAR
>>> errors. I am attaching information you requested.  I have enabled
>>> log-level=8 and files contain dpdk EAL/PMD logs as well.
>>>
>>
>> Great, now we're on the same page.
>>
>>
>>> Snippets below
>>>
>>> on host, DMAR fault address from dmesg
>>>
>>> [351576.998109] DMAR: DRHD: handling fault status reg 702
>>> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr
>>> 257617000 [fault reason 06] PTE Read access is not set
>>>
>>> on guest (dump phys_mem_layout)
>>>
>>> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000,
>>> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
>>> ...
>>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0
>>> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600
>>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40
>>> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80
>>> ...
>>>
>>>
>> To me this looks like host (i.e. either QEMU or the PF driver) is trying
>> to do DMA using guest-physical (and not host-physical). I'm not too
>> well-versed in how QEMU works, but i'm pretty sure that's not supposed to
>> happen.
>>
>> Is PF also bound to DPDK, or are you using native Linux ixgbe driver?
>>
>
> Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF interfaces
> disappear after it is bound to DPDK. If there is a way to use PF and VF
> with DPDK let me know I can try it out. I am not sure how to move forward
> on this, Is CPU/IXGBE PF driver playing a role? Following are the versions
> I have
>
> lscpu
> Architecture:          x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Little Endian
> CPU(s):                56
> On-line CPU(s) list:   0-27
> Off-line CPU(s) list:  28-55
> Thread(s) per core:    1
> Core(s) per socket:    14
> Socket(s):             2
> NUMA node(s):          2
> Vendor ID:             GenuineIntel
> CPU family:            6
> Model:                 63
> Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
> Stepping:              2
> CPU MHz:               2500.610
> CPU max MHz:           3000.0000
> CPU min MHz:           1200.0000
> BogoMIPS:              4000.74
> Virtualization:        VT-x
> L1d cache:             32K
> L1i cache:             32K
> L2 cache:              256K
> L3 cache:              35840K
> NUMA node0 CPU(s):     0-13
> NUMA node1 CPU(s):     14-27
>
> # ethtool -i enp4s0f0
> driver: ixgbe
> version: 5.3.3
> firmware-version: 0x800007b8, 1.1018.0
> bus-info: 0000:04:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> Thanks,
> Ravi
>
>
>

Debugging this I could co-relate doing interface link-up associated with
the dpdk inside the guest causes DMAR errors on host and an additional vflr
message.

[ 8135.861622] DMAR: DRHD: handling fault status reg 402
[ 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr
1b648a000 [fault reason 06] PTE Read access is not set
[ 8136.588074] ixgbe 0000:04:00.0: Issuing VFLR with pending transactions
[ 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0

Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from
'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it
possible that dpdk pmd vf driver is missing some fixes/porting from ixgbevf
driver since this issue is not seen when ixgbevf kernel driver is used?

Thanks,
Ravi


>> --
>> Thanks,
>> Anatoly
>>
>
>