From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f47.google.com (mail-lf0-f47.google.com [209.85.215.47]) by dpdk.org (Postfix) with ESMTP id C72681B2CB for ; Thu, 15 Feb 2018 19:27:06 +0100 (CET) Received: by mail-lf0-f47.google.com with SMTP id w10so806317lfc.9 for ; Thu, 15 Feb 2018 10:27:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=BjZ7m0P6NmGc5kli7YU7YslwPbiunbN8pA6+DZKtYYw=; b=mf991VEkEWU93FdGucVs7Dn+9hWU77DYJ5D+3O3PoDKy3M4Cg8GfnpoZOn8Joo2m59 BBFSyqbxQgY/GmEi36oZHoWaZN0Fm/qT32+UCb+lsDcfzPB1AhgjcFf66S7QWZmTrwfv dHkoDkgexWCI9Ps+YHaect5PW3bga3XBuOmVZIuFe7FC9iDvY0LLAQeBCJFfXLDbnWbM v5aMZXOV6qu9iMnFCrBEzd2v/pQU5/qXQ1ty/KhRac1JbR4KFsMycLKF+8Xezs8mA0lz OdwmgzegQMS0+ZdRBvYez5EASpHP5RykfGCbYLd4T+7zW6JMxcgrk92Q8wCnTgGL8kX8 A6Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=BjZ7m0P6NmGc5kli7YU7YslwPbiunbN8pA6+DZKtYYw=; b=IBVPldGxyeW9vylore43aE3gfri5oErOyBYuKtoj6q0BWsZdNHuwjPOLsVTkI8YwOH HbDTuhEx4Fpb2Tra71bXGR3BstmRAU6A4mpVawUJoW/0Q8U2b3xM0R+ktss3RfpBMhjv lE1Y79Zmjl5MWuxp+1qQV8x5/r4Br9zSmbK2jC7xexiIiGprnTkU0ioJGp8L69hqs9r8 jvpbfJ04joO9UfguCDX3Ulp95wb4DTbVw2CAzXnUAzsJEcPaJ9SiTz6HDKAHFR5XtiM3 nhkm/plXJ+5h/o6zLBT/3d6pV+/IWIiEg1SFs3292hScDNHKRvgnvqL0zwsbnmySVM0z xF+w== X-Gm-Message-State: APf1xPBWmf2+AXu0BnGJUPDEiDZPXYywOFgiFZHSNka6gi66DE2V24GE UHhd9Z3Q99P1zjBHT3egToZk9cwZkjBlkkX8FQo= X-Google-Smtp-Source: AH8x227JX54BfKwISh0S5xJyQspPq7K+7b0rTPC6gVSL6KDZpeW9YGLxBcU4Y9LCEzvncM/m6cjcq1k0OXLr38COZfA= X-Received: by 10.25.157.80 with SMTP id g77mr1916239lfe.142.1518719226213; Thu, 15 Feb 2018 10:27:06 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.80.12 with HTTP; Thu, 15 Feb 2018 10:27:05 -0800 (PST) In-Reply-To: <3b6d6cfc-7e2e-70ee-e059-aa31dc23c93b@intel.com> References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <4a54fe58-02d8-07ef-b706-f747383f4985@intel.com> <3b6d6cfc-7e2e-70ee-e059-aa31dc23c93b@intel.com> From: Ravi Kerur Date: Thu, 15 Feb 2018 10:27:05 -0800 Message-ID: To: "Burakov, Anatoly" Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2018 18:27:06 -0000 On Thu, Feb 15, 2018 at 2:28 AM, Burakov, Anatoly wrote: > On 14-Feb-18 8:00 PM, Ravi Kerur wrote: > >> >> Earlier I was focusing only on DMAR errors and I might have said 'it >> worked' when I didn't notice them on host when dpdk was started on guest. >> When trying to send packets out of that interface from guest I did see DMAR >> errors. I am attaching information you requested. I have enabled >> log-level=8 and files contain dpdk EAL/PMD logs as well. >> > > Great, now we're on the same page. > > >> Snippets below >> >> on host, DMAR fault address from dmesg >> >> [351576.998109] DMAR: DRHD: handling fault status reg 702 >> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr >> 257617000 [fault reason 06] PTE Read access is not set >> >> on guest (dump phys_mem_layout) >> >> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000, >> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 >> ... >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 >> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600 >> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 >> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80 >> ... >> >> > To me this looks like host (i.e. either QEMU or the PF driver) is trying > to do DMA using guest-physical (and not host-physical). I'm not too > well-versed in how QEMU works, but i'm pretty sure that's not supposed to > happen. > > Is PF also bound to DPDK, or are you using native Linux ixgbe driver? > Thanks for your help. I cannot use PF with DPDK (vfio-pci), VF interfaces disappear after it is bound to DPDK. If there is a way to use PF and VF with DPDK let me know I can try it out. I am not sure how to move forward on this, Is CPU/IXGBE PF driver playing a role? Following are the versions I have lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 56 On-line CPU(s) list: 0-27 Off-line CPU(s) list: 28-55 Thread(s) per core: 1 Core(s) per socket: 14 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz Stepping: 2 CPU MHz: 2500.610 CPU max MHz: 3000.0000 CPU min MHz: 1200.0000 BogoMIPS: 4000.74 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 35840K NUMA node0 CPU(s): 0-13 NUMA node1 CPU(s): 14-27 # ethtool -i enp4s0f0 driver: ixgbe version: 5.3.3 firmware-version: 0x800007b8, 1.1018.0 bus-info: 0000:04:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes Thanks, Ravi > > -- > Thanks, > Anatoly >