From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 6B0371F1C for ; Mon, 29 Oct 2018 10:36:24 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Oct 2018 02:36:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,439,1534834800"; d="scan'208";a="101545229" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by fmsmga004.fm.intel.com with ESMTP; 29 Oct 2018 02:36:21 -0700 Received: from fmsmsx111.amr.corp.intel.com (10.18.116.5) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.408.0; Mon, 29 Oct 2018 02:36:21 -0700 Received: from shsmsx104.ccr.corp.intel.com (10.239.4.70) by fmsmsx111.amr.corp.intel.com (10.18.116.5) with Microsoft SMTP Server (TLS) id 14.3.408.0; Mon, 29 Oct 2018 02:36:20 -0700 Received: from shsmsx102.ccr.corp.intel.com ([169.254.2.84]) by SHSMSX104.ccr.corp.intel.com ([169.254.5.117]) with mapi id 14.03.0415.000; Mon, 29 Oct 2018 17:36:19 +0800 From: "Yao, Lei A" To: Thomas Monjalon CC: Alejandro Lucero , "dev@dpdk.org" , "Xu, Qian Q" , "Lin, Xueqin" , "Burakov, Anatoly" Thread-Topic: [dpdk-dev] [PATCH v3 0/6] use IOVAs check based on DMA mask Thread-Index: AQHUbwHVCqPw4mslyE66+0cqAeEf9aU14R4g//+BeYCAAJNOUA== Date: Mon, 29 Oct 2018 09:36:18 +0000 Message-ID: <2DBBFF226F7CF64BAFCA79B681719D954502B7E1@shsmsx102.ccr.corp.intel.com> References: <1538743527-8285-1-git-send-email-alejandro.lucero@netronome.com> <4805099.7RpDHruce7@xps> <2DBBFF226F7CF64BAFCA79B681719D954502B75A@shsmsx102.ccr.corp.intel.com> <1593678.TTmrtHRuFR@xps> In-Reply-To: <1593678.TTmrtHRuFR@xps> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiOTI3NGJkMTctNTljYi00ZjNkLTljMjMtZjE3M2FmOWY4MjQ1IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoibk03VVZKVTE4NFFpZDlZd2d4SjFJN1BhYzE3c3ZOY0Y2OXhDNncwY09BK1htYXQ2c0hGMVoyckwwZ0d2azJRRiJ9 x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v3 0/6] use IOVAs check based on DMA mask X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 09:36:25 -0000 > -----Original Message----- > From: Thomas Monjalon [mailto:thomas@monjalon.net] > Sent: Monday, October 29, 2018 4:43 PM > To: Yao, Lei A > Cc: Alejandro Lucero ; dev@dpdk.org; > Xu, Qian Q ; Lin, Xueqin ; > Burakov, Anatoly > Subject: Re: [dpdk-dev] [PATCH v3 0/6] use IOVAs check based on DMA > mask >=20 > 29/10/2018 09:23, Yao, Lei A: > > Hi, Lucero, Thomas > > > > This patch set will cause deadlock during memory initialization. > > rte_memseg_walk and try_expand_heap both will lock > > the file &mcfg->memory_hotplug_lock. So dead lock will occur. > > > > #0 rte_memseg_walk > > #1 <-rte_eal_check_dma_mask > > #2 <-alloc_pages_on_heap > > #3 <-try_expand_heap_primary > > #4 <-try_expand_heap > > > > Log as following: > > EAL: TSC frequency is ~2494156 KHz > > EAL: Master lcore 0 is ready (tid=3D7ffff7fe3c00;cpuset=3D[0]) > > [New Thread 0x7ffff5e0d700 (LWP 330350)] > > EAL: lcore 1 is ready (tid=3D7ffff5e0d700;cpuset=3D[1]) > > EAL: Trying to obtain current memory policy. > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > EAL: Restoring previous memory policy: 0 > > > > Could you have a check on this? A lot of test cases in our validation > > team fail because of this. Thanks a lot! >=20 > Can we just call rte_memseg_walk_thread_unsafe()? >=20 > +Cc Anatoly Hi, Thomas I change to rte_memseg_walk_thread_unsafe(), still Can't work.=20 EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: memseg iova 140000000, len 40000000, out of range EAL: using dma mask ffffffffffffffff EAL: alloc_pages_on_heap(): couldn't allocate memory due to DMA mask EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 1 EAL: Restoring previous memory policy: 0 EAL: memseg iova 1bc0000000, len 40000000, out of range EAL: using dma mask ffffffffffffffff EAL: alloc_pages_on_heap(): couldn't allocate memory due to DMA mask error allocating rte services array EAL: FATAL: rte_service_init() failed EAL: rte_service_init() failed PANIC in main(): BRs Lei >=20 >=20 > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon > > > 05/10/2018 14:45, Alejandro Lucero: > > > > I sent a patchset about this to be applied on 17.11 stable. The mem= ory > > > > code has had main changes since that version, so here it is the pat= chset > > > > adjusted to current master repo. > > > > > > > > This patchset adds, mainly, a check for ensuring IOVAs are within a > > > > restricted range due to addressing limitations with some devices. T= here > > > > are two known cases: NFP and IOMMU VT-d emulation. > > > > > > > > With this check IOVAs out of range are detected and PMDs can abort > > > > initialization. For the VT-d case, IOVA VA mode is allowed as long = as > > > > IOVAs are within the supported range, avoiding to forbid IOVA VA by > > > > default. > > > > > > > > For the addressing limitations known cases, there are just 40(NFP) = or > > > > 39(VT-d) bits for handling IOVAs. When using IOVA PA, those limitat= ions > > > > imply 1TB(NFP) or 512M(VT-d) as upper limits, which is likely enoug= h for > > > > most systems. With machines using more memory, the added check > will > > > > ensure IOVAs within the range. > > > > > > > > With IOVA VA, and because the way the Linux kernel serves mmap call= s > > > > in 64 bits systems, 39 or 40 bits are not enough. It is possible to > > > > give an address hint with a lower starting address than the default= one > > > > used by the kernel, and then ensuring the mmap uses that hint or hi= nt > plus > > > > some offset. With 64 bits systems, the process virtual address spac= e is > > > > large enoguh for doing the hugepages mmaping within the supported > > > range > > > > when those addressing limitations exist. This patchset also adds a > change > > > > for using such a hint making the use of IOVA VA a more than likely > > > > possibility when there are those addressing limitations. > > > > > > > > The check is not done by default but just when it is required. This > > > > patchset adds the check for NFP initialization and for setting the = IOVA > > > > mode is an emulated VT-d is detected. Also, because the recent > patchset > > > > adding dynamic memory allocation, the check is also invoked for > ensuring > > > > the new memsegs are within the required range. > > > > > > > > This patchset could be applied to stable 18.05. > > > > > > Applied, thanks >=20 >=20