From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <Paul.Barrette@windriver.com>
Received: from mail7.windriver.com (mail7.windriver.com [128.224.252.3])
 by dpdk.org (Postfix) with ESMTP id 71A271F3
 for <dev@dpdk.org>; Mon, 12 Aug 2013 22:28:01 +0200 (CEST)
Received: from ALA-HCA.corp.ad.wrs.com (ala-hca.corp.ad.wrs.com
 [147.11.189.40])
 by mail7.windriver.com (8.14.5/8.14.3) with ESMTP id r7CKSR4i018711
 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL);
 Mon, 12 Aug 2013 16:28:28 -0400 (EDT)
Received: from yow-pbarrett-lx.wrs.com (128.224.146.11) by
 ALA-HCA.corp.ad.wrs.com (147.11.189.50) with Microsoft SMTP Server id
 14.2.342.3; Mon, 12 Aug 2013 13:28:27 -0700
Message-ID: <5209456A.9020200@windriver.com>
Date: Mon, 12 Aug 2013 16:28:26 -0400
From: Paul Barrette <paul.barrette@windriver.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130625 Thunderbird/17.0.7
MIME-Version: 1.0
To: jinho hwang <hwang.jinho@gmail.com>
References: <CAPQGAnGVzACKZDS5k894+-7kEZ2Sg6iSwkcj9+XOMYcuyGaeaQ@mail.gmail.com>
In-Reply-To: <CAPQGAnGVzACKZDS5k894+-7kEZ2Sg6iSwkcj9+XOMYcuyGaeaQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------060506000802090502050007"
Cc: dev <dev@dpdk.org>
Subject: Re: [dpdk-dev] DMAR fault
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 12 Aug 2013 20:28:01 -0000

--------------060506000802090502050007
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit


On 08/12/2013 04:19 PM, jinho hwang wrote:
> Hi All,
>
> I am using iommu to receive packets both from hypervisor and from VM. 
> KVM is used for the virtualization. However, after I deliver the 
> kernel options (iommu and pci realloc), I can not receive packets in 
> hypervisor, but VF works fine in VM. When I tried to receive packets 
> in hypervisor, dmesg shows the following:
>
> ixgbe 0000:03:00.1: complete
> ixgbe 0000:03:00.1: PCI INT A disabled
> igb_uio 0000:03:00.1: PCI INT A -> GSI 38 (level, low) -> IRQ 38
> igb_uio 0000:03:00.1: setting latency timer to 64
> igb_uio 0000:03:00.1: irq 87 for MSI/MSI-X
> uio device registered with irq 57
> DRHD: handling fault status reg 2
> DMAR:[DMA Read] Request device [03:00.1] fault addr *b9d0f000*
> DMAR:[fault reason 02] Present bit in context entry is clear
>
> 03:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit Dual 
> Port Backplane Connection (rev 01)
>         Subsystem: Intel Corporation Ethernet X520 10GbE Dual Port 
> KX4-KR Mezz
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort+ >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 38
>         Region 0: Memory at *d9400000* (64-bit, prefetchable) [size=4M]
>         Region 2: I/O ports at ece0 [size=32]
>         Region 4: Memory at d9bfc000 (64-bit, prefetchable) [size=16K]
>         Expansion ROM at <ignored> [disabled]
>         Capabilities: <access denied>
>         Kernel driver in use: igb_uio
>         Kernel modules: ixgbe
>
> We can see those addresses are not matched. So the kernel got fault. I 
> am wondering why this happens?
I have seen this happen when VT-d is enabled in the bios.  If you are 
using dpdk 1.4, add "iommu=pt" to your boot line.  Without it, no 
packets are received.

Pb
>
> One suspicion for this is BIOS. I am currently using BIOS version 3.0, 
> but the latest is 6.3.0. Does this affect the matter?
>
> Any help appreciated!
>
> Jinho
>


--------------060506000802090502050007
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <br>
    <div class="moz-cite-prefix">On 08/12/2013 04:19 PM, jinho hwang
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAPQGAnGVzACKZDS5k894+-7kEZ2Sg6iSwkcj9+XOMYcuyGaeaQ@mail.gmail.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=ISO-8859-1">
      <div dir="ltr">Hi All,<br>
        <br>
        I am using iommu to receive packets both from hypervisor and
        from VM. KVM is used for the virtualization. However, after I
        deliver the kernel options (iommu and pci realloc), I can not
        receive packets in hypervisor, but VF works fine in VM. When I
        tried to receive packets in hypervisor, dmesg shows the
        following:<br>
        <br>
        ixgbe 0000:03:00.1: complete &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br>
        ixgbe 0000:03:00.1: PCI INT A disabled &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br>
        igb_uio 0000:03:00.1: PCI INT A -&gt; GSI 38 (level, low) -&gt;
        IRQ 38<br>
        igb_uio 0000:03:00.1: setting latency timer to 64 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
        <br>
        igb_uio 0000:03:00.1: irq 87 for MSI/MSI-X &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br>
        uio device registered with irq 57 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
        <br>
        DRHD: handling fault status reg 2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
        <br>
        DMAR:[DMA Read] Request device [03:00.1] fault addr <b>b9d0f000</b>
        &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
        &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
        &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <br>
        DMAR:[fault reason 02] Present bit in context entry is clear<br>
        <br>
        03:00.1 Ethernet controller: Intel Corporation 82599EB 10
        Gigabit Dual Port Backplane Connection (rev 01)<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Subsystem: Intel Corporation Ethernet X520 10GbE Dual
        Port KX4-KR Mezz<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV-
        VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
        &gt;TAbort- &lt;TAbort- &lt;MAbort+ &gt;SERR- &lt;PERR- INTx-<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Latency: 0, Cache Line Size: 64 bytes &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Interrupt: pin A routed to IRQ 38 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Region 0: Memory at <b>d9400000</b> (64-bit,
        prefetchable) [size=4M]<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Region 2: I/O ports at ece0 [size=32] &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Region 4: Memory at d9bfc000 (64-bit, prefetchable)
        [size=16K]<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Expansion ROM at &lt;ignored&gt; [disabled] &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
        &nbsp; &nbsp; &nbsp;<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Capabilities: &lt;access denied&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
        &nbsp; &nbsp; &nbsp;<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Kernel driver in use: igb_uio &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br>
        &nbsp; &nbsp; &nbsp; &nbsp; Kernel modules: ixgbe &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br>
        <br>
        <div>We can see those addresses are not matched. So the kernel
          got fault. I am wondering why this happens?</div>
      </div>
    </blockquote>
    I have seen this happen when VT-d is enabled in the bios.&nbsp; If you
    are using dpdk 1.4, add "iommu=pt" to your boot line.&nbsp; Without it,
    no packets are received.<br>
    <br>
    Pb<br>
    <blockquote
cite="mid:CAPQGAnGVzACKZDS5k894+-7kEZ2Sg6iSwkcj9+XOMYcuyGaeaQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div><br>
        </div>
        <div>One suspicion for this is BIOS. I am currently using BIOS
          version 3.0, but the latest is 6.3.0. Does this affect the
          matter?&nbsp;</div>
        <div><br>
        </div>
        <div>Any help appreciated!</div>
        <div><br>
        </div>
        <div>Jinho &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>
          <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
            &nbsp; &nbsp; &nbsp;<br>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>

--------------060506000802090502050007--