From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 538627E43 for ; Fri, 26 Sep 2014 16:41:22 +0200 (CEST) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP; 26 Sep 2014 07:41:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.04,604,1406617200"; d="scan'208";a="609132406" Received: from pgsmsx101.gar.corp.intel.com ([10.221.44.78]) by orsmga002.jf.intel.com with ESMTP; 26 Sep 2014 07:47:40 -0700 Received: from pgsmsx105.gar.corp.intel.com (10.221.44.96) by PGSMSX101.gar.corp.intel.com (10.221.44.78) with Microsoft SMTP Server (TLS) id 14.3.195.1; Fri, 26 Sep 2014 22:47:40 +0800 Received: from pgsmsx107.gar.corp.intel.com ([169.254.7.83]) by pgsmsx105.gar.corp.intel.com ([169.254.4.53]) with mapi id 14.03.0195.001; Fri, 26 Sep 2014 22:47:40 +0800 From: "Choi, Sy Jong" To: Hiroshi Shimamoto , "dev@dpdk.org" Thread-Topic: DPDK doesn't work with iommu=pt Thread-Index: Ac/ZZvLQu0nCCbsaRVygc3CBHpGSYgAMPSbg Date: Fri, 26 Sep 2014 14:47:39 +0000 Message-ID: <697F8B1B48670548A5BAB03E8283550F2D52706F@PGSMSX107.gar.corp.intel.com> References: <7F861DC0615E0C47A872E6F3C5FCDDBD02ADDCDD@BPXM14GP.gisp.nec.co.jp> In-Reply-To: <7F861DC0615E0C47A872E6F3C5FCDDBD02ADDCDD@BPXM14GP.gisp.nec.co.jp> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.30.20.205] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Hayato Momma Subject: Re: [dpdk-dev] DPDK doesn't work with iommu=pt X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 14:41:23 -0000 Hi Shimamoto-san, There are a lot of sighting relate to "DMAR:[fault reason 06] PTE Read acce= ss is not set" https://www.mail-archive.com/kvm@vger.kernel.org/msg106573.html This might be related to IOMMU, and kernel code. Here is what we know :- 1) Disabling VT-d in bios also removed the symptom 2) Switch to another OS distribution also removed the symptom 3) even different HW we will not see the symptom. In my case, switch from E= ngineering board to EPSD board. Regards, Choi, Sy Jong Platform Application Engineer -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hiroshi Shimamoto Sent: Friday, September 26, 2014 5:14 PM To: dev@dpdk.org Cc: Hayato Momma Subject: [dpdk-dev] DPDK doesn't work with iommu=3Dpt I encountered an issue that DPDK doesn't work with "iommu=3Dpt intel_iommu= =3Don" on HP ProLiant DL380p Gen8 server. I'm using the following environment; HW: ProLiant DL380p Gen8 CPU: E5-2697 v2 OS: RHEL7 kernel: kernel-3.10.0-123 and the latest kernel 3.17-rc6+ DPDK: v1.7.1-53-gce5abac NIC: 82599ES When boot with "iommu=3Dpt intel_iommu=3Don", I got the below message and n= o packets are handled. [ 120.809611] dmar: DRHD: handling fault status reg 2 [ 120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr = aa010000 DMAR:[fault reason 02] Present bit in context entry is clear How to reproduce; just run testpmd # ./testpmd -c 0xf -n 4 -- -i Configuring Port 0 (socket 0) PMD: ixgbe_dev_tx_queue_setup(): sw_ring=3D0x7ffff54eafc0 hw_ring=3D0x7ffff= 4200000 dma_addr=3D0xaa000000 PMD: ixgbe_dev_tx_queue_setup(): Using full-featured tx code path PMD: ixgbe_dev_tx_queue_setup(): - txq_flags =3D 0 [IXGBE_SIMPLE_FLAGS=3Df= 01] PMD: ixgbe_dev_tx_queue_setup(): - tx_rs_thresh =3D 32 [RTE_PMD_IXGBE_TX_M= AX_BURST=3D32] PMD: ixgbe_dev_rx_queue_setup(): sw_ring=3D0x7ffff54ea740 hw_ring=3D0x7ffff= 4210000 dma_addr=3D0xaa010000 PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc Precond= itions: rxq->rx_free_thresh=3D0, RTE_PMD_IXGBE_RX_MAX_BURST=3D32 PMD: ixgbe_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are not = satisfied, Scattered Rx is requested, or RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALL= OC is not enabled (port=3D0, queue=3D0). PMD: check_rx_burst_bulk_alloc_preconditions(): Rx Burst Bulk Alloc Precond= itions: rxq->rx_free_thresh=3D0, RTE_PMD_IXGBE_RX_MAX_BURST=3D32 testpmd> start io packet forwarding - CRC stripping disabled - packets/burst=3D32 nb forwarding cores=3D1 - nb forwarding ports=3D2 RX queues=3D1 - RX desc=3D128 - RX free threshold=3D0 RX threshold registers: pthresh=3D8 hthresh=3D8 wthresh=3D0 TX queues=3D1 - TX desc=3D512 - TX free threshold=3D0 TX threshold registers: pthresh=3D32 hthresh=3D0 wthresh=3D0 TX RS bit threshold=3D0 - TXQ flags=3D0x0 and ping from another box to this server. # ping6 -I eth2 ff02::1 I got the below error message and no packet is received. I couldn't see any increase RX/TX count in testpmt statistics testpmd> show port stats 0 ######################## NIC statistics for port 0 #####################= ### RX-packets: 6 RX-missed: 0 RX-bytes: 732 RX-badcrc: 0 RX-badlen: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 #########################################################################= ### testpmd> show port stats 0 ######################## NIC statistics for port 0 #####################= ### RX-packets: 6 RX-missed: 0 RX-bytes: 732 RX-badcrc: 0 RX-badlen: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 #########################################################################= ### The fault addr in error message must be RX DMA descriptor error message [ 120.809635] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr = aa010000 log in testpmd PMD: ixgbe_dev_rx_queue_setup(): sw_ring=3D0x7ffff54ea740 hw_ring=3D0x7ff= ff4210000 dma_addr=3D0xaa010000 I think the NIC received a packet in fifo and try to put into memory with D= MA. Before starting DMA, the NIC get the target address from RX descriptors in = RDBA register. But accessing RX descriptors failed in IOMMU unit and reported it to the ke= rnel. DMAR:[fault reason 02] Present bit in context entry is clear The error message looks there is no valid entry in IOMMU. I think the following issue is very similar, but using Ubuntu14.04 couldn't= fix in my case. http://thread.gmane.org/gmane.comp.networking.dpdk.devel/2281 I tried Ubuntu14.04.1 and got the below error. [ 199.710191] dmar: DRHD: handling fault status reg 2 [ 199.710896] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr = 7c24df000 [ 199.710896] DMAR:[fault reason 06] PTE Read access is not set Currently I could see this issue on HP ProLiant DL380p Gen8 only. Is there any idea? Has anyone noticed this issue? Note: we're thinking to use SR-IOV and DPDK app in the same box. The box has 2 NICs, one for SR-IOV and pass through to VM, one (no SR-IOV) = for DPDK app in host. thanks, Hiroshi