From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from tyo201.gate.nec.co.jp (TYO201.gate.nec.co.jp [210.143.35.51]) by dpdk.org (Postfix) with ESMTP id A2108678C for ; Fri, 26 Sep 2014 11:07:51 +0200 (CEST) Received: from mailgate3.nec.co.jp ([10.7.69.195]) by tyo201.gate.nec.co.jp (8.13.8/8.13.4) with ESMTP id s8Q9EAjt014353 for ; Fri, 26 Sep 2014 18:14:10 +0900 (JST) Received: from mailsv.nec.co.jp (imss63.nec.co.jp [10.7.69.158]) by mailgate3.nec.co.jp (8.11.7/3.7W-MAILGATE-NEC) with ESMTP id s8Q9EAx23617 for ; Fri, 26 Sep 2014 18:14:10 +0900 (JST) Received: from mail03.kamome.nec.co.jp (mail03.kamome.nec.co.jp [10.25.43.7]) by mailsv.nec.co.jp (8.13.8/8.13.4) with ESMTP id s8Q9EAwe024031 for ; Fri, 26 Sep 2014 18:14:10 +0900 (JST) Received: from bpxc99gp.gisp.nec.co.jp ([10.38.151.142] [10.38.151.142]) by mail01b.kamome.nec.co.jp with ESMTP id BT-MMP-2146015; Fri, 26 Sep 2014 18:13:53 +0900 Received: from BPXM14GP.gisp.nec.co.jp ([169.254.1.136]) by BPXC14GP.gisp.nec.co.jp ([10.38.151.142]) with mapi id 14.03.0174.002; Fri, 26 Sep 2014 18:13:52 +0900 From: Hiroshi Shimamoto To: "dev@dpdk.org" Thread-Topic: DPDK doesn't work with iommu=pt Thread-Index: Ac/ZZvLQu0nCCbsaRVygc3CBHpGSYg== Date: Fri, 26 Sep 2014 09:13:51 +0000 Message-ID: <7F861DC0615E0C47A872E6F3C5FCDDBD02ADDCDD@BPXM14GP.gisp.nec.co.jp> Accept-Language: ja-JP, en-US Content-Language: ja-JP X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.205.5.123] Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Hayato Momma Subject: [dpdk-dev] DPDK doesn't work with iommu=pt X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 09:07:52 -0000 I encountered an issue that DPDK doesn't work with "iommu=3Dpt intel_iommu= =3Don" on HP ProLiant DL380p Gen8 server. I'm using the following environment; HW: ProLiant DL380p Gen8 CPU: E5-2697 v2 OS: RHEL7=20 kernel: kernel-3.10.0-123 and the latest kernel 3.17-rc6+ DPDK: v1.7.1-53-gce5abac NIC: 82599ES When boot with "iommu=3Dpt intel_iommu=3Don", I got the below message and no packets are handled. [ 120.809611] dmar: DRHD: handling fault status reg 2 [ 120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr = aa010000 DMAR:[fault reason 02] Present bit in context entry is clear How to reproduce; just run testpmd # ./testpmd -c 0xf -n 4 -- -i Configuring Port 0 (socket 0) PMD: ixgbe_dev_tx_queue_setup(): sw_ring=3D0x7ffff54eafc0 hw_ring=3D0x7ffff= 4200000 dma_addr=3D0xaa000000 PMD: ixgbe_dev_tx_queue_setup(): Using full-featured tx code path PMD: ixgbe_dev_tx_queue_setup(): - txq_flags =3D 0 [IXGBE_SIMPLE_FLAGS=3Df= 01] PMD: ixgbe_dev_tx_queue_setup(): - tx_rs_thresh =3D 32 [RTE_PMD_IXGBE_TX_M= AX_BURST=3D32] PMD: ixgbe_dev_rx_queue_setup(): sw_ring=3D0x7ffff54ea740 hw_ring=3D0x7ffff= 4210000 dma_addr=3D0xaa010000 PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc Precond= itions: rxq->rx_free_thresh=3D0, RTE_PMD_IXGBE_RX_MAX_BURST=3D32 PMD: ixgbe_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are not = satisfied, Scattered Rx is requested, or RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALL= OC is not enabled (port=3D0, queue=3D0). PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc Precond= itions: rxq->rx_free_thresh=3D0, RTE_PMD_IXGBE_RX_MAX_BURST=3D32 testpmd> start io packet forwarding - CRC stripping disabled - packets/burst=3D32 nb forwarding cores=3D1 - nb forwarding ports=3D2 RX queues=3D1 - RX desc=3D128 - RX free threshold=3D0 RX threshold registers: pthresh=3D8 hthresh=3D8 wthresh=3D0 TX queues=3D1 - TX desc=3D512 - TX free threshold=3D0 TX threshold registers: pthresh=3D32 hthresh=3D0 wthresh=3D0 TX RS bit threshold=3D0 - TXQ flags=3D0x0 and ping from another box to this server. # ping6 -I eth2 ff02::1 I got the below error message and no packet is received. I couldn't see any increase RX/TX count in testpmt statistics testpmd> show port stats 0 ######################## NIC statistics for port 0 #####################= ### RX-packets: 6 RX-missed: 0 RX-bytes: 732 RX-badcrc: 0 RX-badlen: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 #########################################################################= ### testpmd> show port stats 0 ######################## NIC statistics for port 0 #####################= ### RX-packets: 6 RX-missed: 0 RX-bytes: 732 RX-badcrc: 0 RX-badlen: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 #########################################################################= ### The fault addr in error message must be RX DMA descriptor error message [ 120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr = aa010000 log in testpmd PMD: ixgbe_dev_rx_queue_setup(): sw_ring=3D0x7ffff54ea740 hw_ring=3D0x7ff= ff4210000 dma_addr=3D0xaa010000 I think the NIC received a packet in fifo and try to put into memory with D= MA. Before starting DMA, the NIC get the target address from RX descriptors in = RDBA register. But accessing RX descriptors failed in IOMMU unit and reported it to the ke= rnel. DMAR:[fault reason 02] Present bit in context entry is clear The error message looks there is no valid entry in IOMMU. I think the following issue is very similar, but using Ubuntu14.04 couldn't= fix in my case. http://thread.gmane.org/gmane.comp.networking.dpdk.devel/2281 I tried Ubuntu14.04.1 and got the below error. [ 199.710191] dmar: DRHD: handling fault status reg 2 [ 199.710896] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr = 7c24df000 [ 199.710896] DMAR:[fault reason 06] PTE Read access is not set Currently I could see this issue on HP ProLiant DL380p Gen8 only. Is there any idea? Has anyone noticed this issue? Note: we're thinking to use SR-IOV and DPDK app in the same box. The box has 2 NICs, one for SR-IOV and pass through to VM, one (no SR-IOV) = for DPDK app in host. thanks, Hiroshi