From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f44.google.com (mail-lf0-f44.google.com [209.85.215.44]) by dpdk.org (Postfix) with ESMTP id 04F821B299 for ; Thu, 15 Feb 2018 21:53:28 +0100 (CET) Received: by mail-lf0-f44.google.com with SMTP id g72so1373426lfg.5 for ; Thu, 15 Feb 2018 12:53:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ko7OEz8bZqPW43CBLYmu2zz5MQWgxO+dCDdABTzjX/Q=; b=XFVq1kqfX17uUXkVhATX2vucerxppvPV5a5nYS2+4em/TmhbyqzKyKYU5CmPyjfsa3 FcRGFDpM9sCjeYXLZ3fAqkL0umrtMLrUu0tCk6f/EswW3y1GhwNxKLnnguc5xsQARCdP 4rYuNbni+wEiXBpNchrBjB/FmMh27n906fDsgzSkehJFQEKduV7nSu+2NK9fGuCbKtzs 54Kipmi6/BI7k/MPzWRn5mTbonNJCSszrf05WSd+OpytIfEVE8OVZXfOR2Sbx1mI1CmE AsivNTOV7GT59XHHwD2a2eYM+wPvikKxlopEDAiOuBrgYUTq+LOeppnpP7/Qu3JiMwkD g7tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ko7OEz8bZqPW43CBLYmu2zz5MQWgxO+dCDdABTzjX/Q=; b=QBnyUw/nqGrKCrc9kIIIcmfKmVnE7itYzrFRR58tKGv/y9+fYJyfigm8GAMoH2f9Y0 9jrTvNznVeXC+wh8md/nLCnno0MEH+Jo4Hq/Jm+4eTByBzEWUxN+L2buJtKz8MSAk993 m73hq7Enfp6aHzIdI84vnFUBfiWAkTx0slLs7Tixrq5E+X7Jivx2tXBewsVCyoeV+lBo bMEhJdld6takghpzCJjoznShTIcOFpj/iaqM1mvq9dbkWJX8V/JUrlxiOT/53Mbb4cgq 0S5IdUlbxu5SdF/wX0Ry+qQoWQt8Ej55zXWUHOxRozTWFEWF48Tk9VvyLnGyrEZuQnHL xPBQ== X-Gm-Message-State: APf1xPCzXRmhgdj+v8D/4hGu9ArQegBAJTjj/csRqYY16qaCFtvqk1kI TAtWjMrCUziUkPxYe6v78j4rXhv0+a4ysAYjr6w= X-Google-Smtp-Source: AH8x226ACDD0PXrGIQvnj7egqEATLztUw8fD9jetg+sc6WXv/DaquN634Yz2SaDhOTvPopSVeFPUB+PBqmy2G7HLnzQ= X-Received: by 10.46.16.212 with SMTP id 81mr2533974ljq.119.1518728006012; Thu, 15 Feb 2018 12:53:26 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.80.12 with HTTP; Thu, 15 Feb 2018 12:53:25 -0800 (PST) In-Reply-To: References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <4a54fe58-02d8-07ef-b706-f747383f4985@intel.com> <3b6d6cfc-7e2e-70ee-e059-aa31dc23c93b@intel.com> From: Ravi Kerur Date: Thu, 15 Feb 2018 12:53:25 -0800 Message-ID: To: "Burakov, Anatoly" Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2018 20:53:28 -0000 On Thu, Feb 15, 2018 at 10:27 AM, Ravi Kerur wrote: > > > On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly < > anatoly.burakov@intel.com> wrote: > >> On 14-Feb-18 8:00 PM, Ravi Kerur wrote: >> >>> >>> Earlier I was focusing only on DMAR errors and I might have said 'it >>> worked' when I didn't notice them on host when dpdk was started on guest. >>> When trying to send packets out of that interface from guest I did see DMAR >>> errors. I am attaching information you requested. I have enabled >>> log-level=8 and files contain dpdk EAL/PMD logs as well. >>> >> >> Great, now we're on the same page. >> >> >>> Snippets below >>> >>> on host, DMAR fault address from dmesg >>> >>> [351576.998109] DMAR: DRHD: handling fault status reg 702 >>> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr >>> 257617000 [fault reason 06] PTE Read access is not set >>> >>> on guest (dump phys_mem_layout) >>> >>> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000, >>> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 >>> ... >>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 >>> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600 >>> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 >>> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80 >>> ... >>> >>> >> To me this looks like host (i.e. either QEMU or the PF driver) is trying >> to do DMA using guest-physical (and not host-physical). I'm not too >> well-versed in how QEMU works, but i'm pretty sure that's not supposed to >> happen. >> >> Is PF also bound to DPDK, or are you using native Linux ixgbe driver? >> > > Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF interfaces > disappear after it is bound to DPDK. If there is a way to use PF and VF > with DPDK let me know I can try it out. I am not sure how to move forward > on this, Is CPU/IXGBE PF driver playing a role? Following are the versions > I have > > lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 56 > On-line CPU(s) list: 0-27 > Off-line CPU(s) list: 28-55 > Thread(s) per core: 1 > Core(s) per socket: 14 > Socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 63 > Model name: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz > Stepping: 2 > CPU MHz: 2500.610 > CPU max MHz: 3000.0000 > CPU min MHz: 1200.0000 > BogoMIPS: 4000.74 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 35840K > NUMA node0 CPU(s): 0-13 > NUMA node1 CPU(s): 14-27 > > # ethtool -i enp4s0f0 > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > Thanks, > Ravi > > > Debugging this I could co-relate doing interface link-up associated with the dpdk inside the guest causes DMAR errors on host and an additional vflr message. [ 8135.861622] DMAR: DRHD: handling fault status reg 402 [ 8135.861627] DMAR: [DMA Read] Request device [04:10.0] fault addr 1b648a000 [fault reason 06] PTE Read access is not set [ 8136.588074] ixgbe 0000:04:00.0: Issuing VFLR with pending transactions [ 8136.588079] ixgbe 0000:04:00.0: Issuing VFLR for VF 0000:04:10.0 Looked at ixgbe driver code 'ixgbe_issue_vf_flr' is called from 'ixgbe_check_for_bad_vf' or 'ixgbe_io_error_detected' functions. Is it possible that dpdk pmd vf driver is missing some fixes/porting from ixgbevf driver since this issue is not seen when ixgbevf kernel driver is used? Thanks, Ravi >> -- >> Thanks, >> Anatoly >> > >