From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from tyo201.gate.nec.co.jp (TYO201.gate.nec.co.jp [210.143.35.51]) by dpdk.org (Postfix) with ESMTP id 9AE451F7 for ; Mon, 29 Sep 2014 01:16:38 +0200 (CEST) Received: from mailgate3.nec.co.jp ([10.7.69.193]) by tyo201.gate.nec.co.jp (8.13.8/8.13.4) with ESMTP id s8SNN9ku019565; Mon, 29 Sep 2014 08:23:09 +0900 (JST) Received: from mailsv.nec.co.jp (imss62.nec.co.jp [10.7.69.157]) by mailgate3.nec.co.jp (8.11.7/3.7W-MAILGATE-NEC) with ESMTP id s8SNN9C23867; Mon, 29 Sep 2014 08:23:09 +0900 (JST) Received: from mail02.kamome.nec.co.jp (mail02.kamome.nec.co.jp [10.25.43.5]) by mailsv.nec.co.jp (8.13.8/8.13.4) with ESMTP id s8SNN9PT028387; Mon, 29 Sep 2014 08:23:09 +0900 (JST) Received: from bpxc99gp.gisp.nec.co.jp ([10.38.151.143] [10.38.151.143]) by mail02.kamome.nec.co.jp with ESMTP id BT-MMP-2168443; Mon, 29 Sep 2014 08:20:20 +0900 Received: from BPXM14GP.gisp.nec.co.jp ([169.254.1.136]) by BPXC15GP.gisp.nec.co.jp ([10.38.151.143]) with mapi id 14.03.0174.002; Mon, 29 Sep 2014 08:20:20 +0900 From: Hiroshi Shimamoto To: "Zhang, Jerry" , "dev@dpdk.org" Thread-Topic: DPDK doesn't work with iommu=pt Thread-Index: Ac/ZZvLQu0nCCbsaRVygc3CBHpGSYgBiRPVAACCMK0A= Date: Sun, 28 Sep 2014 23:20:19 +0000 Message-ID: <7F861DC0615E0C47A872E6F3C5FCDDBD02ADF477@BPXM14GP.gisp.nec.co.jp> References: <7F861DC0615E0C47A872E6F3C5FCDDBD02ADDCDD@BPXM14GP.gisp.nec.co.jp> In-Reply-To: Accept-Language: ja-JP, en-US Content-Language: ja-JP X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.205.5.123] Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Hayato Momma Subject: Re: [dpdk-dev] DPDK doesn't work with iommu=pt X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Sep 2014 23:16:39 -0000 Hi, > Subject: RE: DPDK doesn't work with iommu=3Dpt >=20 > Met the similar issue before. > VT-d enabled? If so you may need to contact HP to upgrade the BIOS or you= may disable VT-d and remove iommu=3Dpt intel_iommu=3Don > if you don't need VF function. we need VT-d and it's enabled. What we want to do is that use SR-IOV functionality and DPDK application co= ncurrently on the same box. thanks, Hiroshi >=20 > >-----Original Message----- > >From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hiroshi Shimamoto > >Sent: Friday, September 26, 2014 5:14 PM > >To: dev@dpdk.org > >Cc: Hayato Momma > >Subject: [dpdk-dev] DPDK doesn't work with iommu=3Dpt > > > >I encountered an issue that DPDK doesn't work with "iommu=3Dpt > >intel_iommu=3Don" > >on HP ProLiant DL380p Gen8 server. I'm using the following environment; > > > > HW: ProLiant DL380p Gen8 > > CPU: E5-2697 v2 > > OS: RHEL7 > > kernel: kernel-3.10.0-123 and the latest kernel 3.17-rc6+ > > DPDK: v1.7.1-53-gce5abac > > NIC: 82599ES > > > >When boot with "iommu=3Dpt intel_iommu=3Don", I got the below message an= d no > >packets are handled. > > > > [ 120.809611] dmar: DRHD: handling fault status reg 2 > > [ 120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault ad= dr > >aa010000 > > DMAR:[fault reason 02] Present bit in context entry is clear > > > >How to reproduce; > >just run testpmd > ># ./testpmd -c 0xf -n 4 -- -i > > > >Configuring Port 0 (socket 0) > >PMD: ixgbe_dev_tx_queue_setup(): sw_ring=3D0x7ffff54eafc0 > >hw_ring=3D0x7ffff4200000 dma_addr=3D0xaa000000 > >PMD: ixgbe_dev_tx_queue_setup(): Using full-featured tx code path > >PMD: ixgbe_dev_tx_queue_setup(): - txq_flags =3D 0 > >[IXGBE_SIMPLE_FLAGS=3Df01] > >PMD: ixgbe_dev_tx_queue_setup(): - tx_rs_thresh =3D 32 > >[RTE_PMD_IXGBE_TX_MAX_BURST=3D32] > >PMD: ixgbe_dev_rx_queue_setup(): sw_ring=3D0x7ffff54ea740 > >hw_ring=3D0x7ffff4210000 dma_addr=3D0xaa010000 > >PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc > >Preconditions: rxq->rx_free_thresh=3D0, RTE_PMD_IXGBE_RX_MAX_BURST=3D32 > >PMD: ixgbe_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are n= ot > >satisfied, Scattered Rx is requested, or > >RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC is not enabled (port=3D0, queue=3D0= ). > >PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc > >Preconditions: rxq->rx_free_thresh=3D0, RTE_PMD_IXGBE_RX_MAX_BURST=3D32 > > > >testpmd> start > > io packet forwarding - CRC stripping disabled - packets/burst=3D32 > > nb forwarding cores=3D1 - nb forwarding ports=3D2 > > RX queues=3D1 - RX desc=3D128 - RX free threshold=3D0 > > RX threshold registers: pthresh=3D8 hthresh=3D8 wthresh=3D0 > > TX queues=3D1 - TX desc=3D512 - TX free threshold=3D0 > > TX threshold registers: pthresh=3D32 hthresh=3D0 wthresh=3D0 > > TX RS bit threshold=3D0 - TXQ flags=3D0x0 > > > > > >and ping from another box to this server. > ># ping6 -I eth2 ff02::1 > > > >I got the below error message and no packet is received. > >I couldn't see any increase RX/TX count in testpmt statistics > > > >testpmd> show port stats 0 > > > > ######################## NIC statistics for port 0 > >######################## > > RX-packets: 6 RX-missed: 0 RX-bytes: 732 > > RX-badcrc: 0 RX-badlen: 0 RX-errors: 0 > > RX-nombuf: 0 > > TX-packets: 0 TX-errors: 0 TX-bytes: 0 > > > >################################################################# > >########### > >testpmd> show port stats 0 > > > > ######################## NIC statistics for port 0 > >######################## > > RX-packets: 6 RX-missed: 0 RX-bytes: 732 > > RX-badcrc: 0 RX-badlen: 0 RX-errors: 0 > > RX-nombuf: 0 > > TX-packets: 0 TX-errors: 0 TX-bytes: 0 > > > >################################################################# > >########### > > > > > >The fault addr in error message must be RX DMA descriptor > > > >error message > > [ 120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault ad= dr > >aa010000 > > > >log in testpmd > > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=3D0x7ffff54ea740 > >hw_ring=3D0x7ffff4210000 dma_addr=3D0xaa010000 > > > >I think the NIC received a packet in fifo and try to put into memory wit= h DMA. > >Before starting DMA, the NIC get the target address from RX descriptors = in > >RDBA register. > >But accessing RX descriptors failed in IOMMU unit and reported it to the= kernel. > > > > DMAR:[fault reason 02] Present bit in context entry is clear > > > >The error message looks there is no valid entry in IOMMU. > > > >I think the following issue is very similar, but using Ubuntu14.04 could= n't fix in my > >case. > >http://thread.gmane.org/gmane.comp.networking.dpdk.devel/2281 > > > >I tried Ubuntu14.04.1 and got the below error. > > > > [ 199.710191] dmar: DRHD: handling fault status reg 2 > > [ 199.710896] dmar: DMAR:[DMA Read] Request device [21:00.0] fault ad= dr > >7c24df000 > > [ 199.710896] DMAR:[fault reason 06] PTE Read access is not set > > > >Currently I could see this issue on HP ProLiant DL380p Gen8 only. > >Is there any idea? > >Has anyone noticed this issue? > > > >Note: we're thinking to use SR-IOV and DPDK app in the same box. > >The box has 2 NICs, one for SR-IOV and pass through to VM, one (no SR-IO= V) for > >DPDK app in host. > > > >thanks, > >Hiroshi