From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 48F162BD3 for ; Tue, 15 Jan 2019 12:22:22 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Jan 2019 03:22:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,481,1539673200"; d="scan'208";a="138424120" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.251.85.93]) ([10.251.85.93]) by fmsmga001.fm.intel.com with ESMTP; 15 Jan 2019 03:22:19 -0800 To: "Hu, Xuekun" , Ravi Kerur Cc: "dev@dpdk.org" , "Ananyev, Konstantin" , "Lu, Wenzhuo" References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <4a54fe58-02d8-07ef-b706-f747383f4985@intel.com> <3b6d6cfc-7e2e-70ee-e059-aa31dc23c93b@intel.com> <1ab980ce-16a5-4bf4-e673-b706e3126474@intel.com> <88A92D351643BA4CB23E303155170626635418D0@SHSMSX103.ccr.corp.intel.com> From: "Burakov, Anatoly" Message-ID: Date: Tue, 15 Jan 2019 11:22:18 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <88A92D351643BA4CB23E303155170626635418D0@SHSMSX103.ccr.corp.intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jan 2019 11:22:22 -0000 On 15-Jan-19 7:07 AM, Hu, Xuekun wrote: > Hi, Ravi > > Did you resolve this issue that VF used in guest with vIOMMU enabled? I googled, but still can't get the answer that it is driver or qemu vt-d emulation issue. > > Currently I met the same issue again that host reported DMAR error: > [59939.130110] DMAR: DRHD: handling fault status reg 2 > [59939.130116] DMAR: [DMA Read] Request device [83:10.0] fault addr 15f03d000 [fault reason 06] PTE Read access is not set > [59940.180859] ixgbe 0000:83:00.0: Issuing VFLR with pending transactions > [59940.180863] ixgbe 0000:83:00.0: Issuing VFLR for VF 0000:83:10.0 > [59989.344683] ixgbe 0000:83:00.0 ens817f0: VF Reset msg received from vf 0 > > I'm using DPDK 18.11 in guest, and ixgbe.ko in host. > > Thx, Xuekun > > -----Original Message----- > From: dev On Behalf Of Burakov, Anatoly > Sent: Friday, February 16, 2018 5:42 PM > To: Ravi Kerur > Cc: dev@dpdk.org; Ananyev, Konstantin ; Lu, Wenzhuo > Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue > > On 15-Feb-18 8:53 PM, Ravi Kerur wrote: >> >> >> On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur > > wrote: >> >> >> >> On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly >> > wrote: >> >> On 14-Feb-18 8:00 PM, Ravi Kerur wrote: >> >> >> Earlier I was focusing only on DMAR errors and I might have >> said 'it worked' when I didn't notice them on host when dpdk >> was started on guest. When trying to send packets out of >> that interface from guest I did see DMAR errors. I am >> attaching information you requested.  I have enabled >> log-level=8 and files contain dpdk EAL/PMD logs as well. >> >> >> Great, now we're on the same page. >> >> >> Snippets below >> >> on host, DMAR fault address from dmesg >> >> [351576.998109] DMAR: DRHD: handling fault status reg 702 >> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] >> fault addr 257617000 [fault reason 06] PTE Read access is >> not set >> >> on guest (dump phys_mem_layout) >> >> Segment 235: phys:0x257600000, len:2097152, >> virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152, >> nchannel:0, nrank:0 >> ... >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 >> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 >> dma_addr=0x257617600 >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 >> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 >> dma_addr=0x25406fe80 >> ... >> >> >> To me this looks like host (i.e. either QEMU or the PF driver) >> is trying to do DMA using guest-physical (and not >> host-physical). I'm not too well-versed in how QEMU works, but >> i'm pretty sure that's not supposed to happen. >> >> Is PF also bound to DPDK, or are you using native Linux ixgbe >> driver? >> >> >> Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF >> interfaces disappear after it is bound to DPDK. If there is a way to >> use PF and VF with DPDK let me know I can try it out. I am not sure >> how to move forward on this, Is CPU/IXGBE PF driver playing a role? >> Following are the versions I have >> >> lscpu >> Architecture:          x86_64 >> CPU op-mode(s):        32-bit, 64-bit >> Byte Order:            Little Endian >> CPU(s):                56 >> On-line CPU(s) list:   0-27 >> Off-line CPU(s) list:  28-55 >> Thread(s) per core:    1 >> Core(s) per socket:    14 >> Socket(s):             2 >> NUMA node(s):          2 >> Vendor ID:             GenuineIntel >> CPU family:            6 >> Model:                 63 >> Model name:            Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz >> Stepping:              2 >> CPU MHz:               2500.610 >> CPU max MHz:           3000.0000 >> CPU min MHz:           1200.0000 >> BogoMIPS:              4000.74 >> Virtualization:        VT-x >> L1d cache:             32K >> L1i cache:             32K >> L2 cache:              256K >> L3 cache:              35840K >> NUMA node0 CPU(s):     0-13 >> NUMA node1 CPU(s):     14-27 >> >> # ethtool -i enp4s0f0 >> driver: ixgbe >> version: 5.3.3 >> firmware-version: 0x800007b8, 1.1018.0 >> bus-info: 0000:04:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> supports-priv-flags: yes >> >> Thanks, >> Ravi >> >> >> >> Debugging this I could co-relate doing interface link-up associated >> with the dpdk inside the guest causes DMAR errors on host and an >> additional vflr message. >> >> [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [ >> 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074] >> ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [ >> 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0 >> >> Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from >> 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it >> possible that dpdk pmd vf driver is missing some fixes/porting from >> ixgbevf driver since this issue is not seen when ixgbevf kernel driver >> is used? >> > > Could very well be. +CC ixgbe maintainers which might be of further help debugging this issue. > > -- > Thanks, > Anatoly > You might also want to look here: https://bugs.dpdk.org/show_bug.cgi?id=76 There are apparently issues with some kernel versions that will manifest themselves as problems with using VF devices with IOMMU in a VM. -- Thanks, Anatoly