From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) by dpdk.org (Postfix) with ESMTP id 0AEE71B1BF for ; Wed, 14 Feb 2018 21:01:01 +0100 (CET) Received: by mail-lf0-f51.google.com with SMTP id x196so31028177lfd.12 for ; Wed, 14 Feb 2018 12:01:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=u+beVMlVPEyweNSpMOF23pbv1OeUtdaGv3YPqstpK2U=; b=FKU5dsptPO7xzfLYvio5XCjt35hnJMbATGRlUMBdZ3Iqjlpy2fw6slpuHioRnOG3Kk oFl14J4ULg2rAdRD0F0PmDViH2DBbPmzPnoOwM5xaVzQIEQIId4SHDkgFMVRj2WxAMap zlJye8BH1vlZpxaHa0oyhDNcHdVilPM82QizMDvppeUCT1ZI5TnME78M2JBSZ8yXnGrq iqVBKcmRossKKN1YaXF30RblMWaiRxkjCSYlbEFgrKxxIn7oZBOfCShJvtlQWICt0gg9 W1pf0gkAS2iN7KEido8Zx0fg7g9tt8oTG4w5kjVi16U+OLNAle2KBhSTEPUcY0ESNGdK ATzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=u+beVMlVPEyweNSpMOF23pbv1OeUtdaGv3YPqstpK2U=; b=AFSD3UA/XfL9owNi7bdLkBEOo0sSu0zQpc5wGL1VQFkkvl8BEwIif6cGMxnbNoQ/rI 7XAHGtXGEtqdA7qwY3xlJxxOQoMc7OIynxyoMcSPUVoN7kdoe1SKrcPuhImNhElaRpVV 7FeC2ssxEbpbTjxVktmvskRGRrhDTnUZh9hCDng9pNf9I7ctm//D9QXIK6OPcROyPE4y zVfSqyfDcIThMs9oBaWp/nb3UudZ7LDKsmOKyQvd1L/8OkubVpWve30AVMEmYULx2EGG +8LeXF7X8bQiAY1GEcEG3gMALiI+cALTUh9jpA9YkTQTBID9GqAYynfu6NRZJEhhhp0T TtJw== X-Gm-Message-State: APf1xPAyZtcn3zMIT36Sj47BEEnbUqgJd+iYn57nqmTRT4hRX/sBw83s 4jChYTqmmYLUgL6iUsUhnDbQMFyME9HhqgDkb34= X-Google-Smtp-Source: AH8x226PphMDKMq63VFRPb/75E7Eh2wa1Ldb/MLS1qiRKxa5WYnsN5GLxHXQR6Xuf/VwJFR2WIhuHnfu3dtVX9HDrYE= X-Received: by 10.25.217.69 with SMTP id q66mr131568lfg.62.1518638460501; Wed, 14 Feb 2018 12:01:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.80.12 with HTTP; Wed, 14 Feb 2018 12:00:59 -0800 (PST) In-Reply-To: <4a54fe58-02d8-07ef-b706-f747383f4985@intel.com> References: <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com> <4a54fe58-02d8-07ef-b706-f747383f4985@intel.com> From: Ravi Kerur Date: Wed, 14 Feb 2018 12:00:59 -0800 Message-ID: To: "Burakov, Anatoly" Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] IXGBE, IOMMU DMAR DRHD handling fault issue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2018 20:01:01 -0000 On Tue, Feb 13, 2018 at 6:31 AM, Burakov, Anatoly wrote: > On 12-Feb-18 10:00 PM, Ravi Kerur wrote: > >> >> Let me just give you what has been tested and working/nonworking >> scenarios. Some of your questions might get answered as well. >> Test bed is very simple with 2 VF's created under IXGBE PF on >> host with one VF interface added to ovs-bridge on host and >> another VF interface given to guest. Test connectivity between >> VF's via ping. >> >> Host and guest -- Kernel 4.9 >> Host -- Qemu 2.11.50 (tried both released 2.11 and tip of the >> git (2.11.50)) >> DPDK -- 17.05.1 on host and guest >> Host and guest -- booted with GRUB intel_iommu=on (which enables >> IOMMU). Have tried with "iommu=on and intel_iommu=on" as well, >> but iommu=on is not needed when intel_iommu=on is set. >> >> Test-scenario-1: Host -- ixgbe_vf driver, Guest ixgbe_vf driver >> ping works >> Test-scenario-2: Host -- DPDK vfio-pci driver, Guest ixgbe_vf >> driver ping works >> Test-scenario-3: Host -- DPDK vfio-pci driver, Guest DPDK >> vfio-pci driver, DMAR errors seen on host, ping doesn't work >> >> >> OK, that makes it clearer, thanks. Does the third scenario work in >> other DPDK versions? >> >> >> No. Tried 16.11 same issue on guest and works fine on host. >> >> >> So now we've moved from "this worked on 16.11" to "this never worked". > > It would be good to see output of rte_dump_physmem_layout() on both host > and guest, and check which address triggers the DMAR error (i.e. if the > physical address is present in mappings for either DPDK process). > > -- > Earlier I was focusing only on DMAR errors and I might have said 'it worked' when I didn't notice them on host when dpdk was started on guest. When trying to send packets out of that interface from guest I did see DMAR errors. I am attaching information you requested. I have enabled log-level=8 and files contain dpdk EAL/PMD logs as well. Snippets below on host, DMAR fault address from dmesg [351576.998109] DMAR: DRHD: handling fault status reg 702 [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr 257617000 [fault reason 06] PTE Read access is not set on guest (dump phys_mem_layout) Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 ... PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600 PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80 ... Thanks, Ravi > Thanks, > Anatoly >