From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 1E6CFAFCD for ; Fri, 20 Jun 2014 16:43:12 +0200 (CEST) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP; 20 Jun 2014 07:37:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.01,514,1400050800"; d="scan'208";a="560741178" Received: from irsmsx101.ger.corp.intel.com ([163.33.3.153]) by orsmga002.jf.intel.com with ESMTP; 20 Jun 2014 07:42:46 -0700 Received: from irsmsx151.ger.corp.intel.com (163.33.192.59) by IRSMSX101.ger.corp.intel.com (163.33.3.153) with Microsoft SMTP Server (TLS) id 14.3.123.3; Fri, 20 Jun 2014 15:42:45 +0100 Received: from irsmsx101.ger.corp.intel.com ([169.254.1.245]) by IRSMSX151.ger.corp.intel.com ([169.254.4.152]) with mapi id 14.03.0123.003; Fri, 20 Jun 2014 15:42:44 +0100 From: "Burakov, Anatoly" To: "Gooch, Stephen (Wind River)" , "Richardson, Bruce" , "dev@dpdk.org" Thread-Topic: mmap() hint address Thread-Index: Ac+HSjModLMx+SPRQcmV/p2lSa3M/AAA1WOwAHrFVkAA1pSt4AAAkgSg Date: Fri, 20 Jun 2014 14:42:44 +0000 Message-ID: References: <9205DC19ECCD944CA2FAC59508A772BABCEFF60C@ALA-MBA.corp.ad.wrs.com> <59AF69C657FD0841A61C55336867B5B01AA361A6@IRSMSX103.ger.corp.intel.com> <9205DC19ECCD944CA2FAC59508A772BABCF0F702@ALA-MBA.corp.ad.wrs.com> In-Reply-To: <9205DC19ECCD944CA2FAC59508A772BABCF0F702@ALA-MBA.corp.ad.wrs.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [163.33.239.182] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] mmap() hint address X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2014 14:43:14 -0000 Hi Stephen, You may want to look at the local tailqs patchset I've made. This may fix y= our issues as well. http://dpdk.org/ml/archives/dev/2014-June/003573.html I'm planning to respin a v4 of it, with an addition of using --base-virtadd= r flag to also control where rte_config is mapped as well as hugepages. Thanks, Anatoly > -----Original Message----- > From: Gooch, Stephen [mailto:stephen.gooch@windriver.com] > Sent: Friday, June 20, 2014 3:37 PM > To: Burakov, Anatoly; Richardson, Bruce; dev@dpdk.org > Subject: RE: mmap() hint address >=20 > Hello, >=20 > One item I should have included is this device is running 32-bit 2.6.27, = quite > old, and sharing 4GB of RAM with a number of applications. We were able= to > find the issue. In the failure case vDSO is mapped lower (toward [heap]) > than normal. As a result , .rte_config was mapped into the pre-mapped pc= i > uio resource virtual address range. >=20 > The fix: (1) move uio mmap() out of the narrow range at the bottom of the > memory maps and (2) creating spacing between the uio maps and rte_config > mmap(). It works with all huge page settings tested. >=20 > - Stephen >=20 > -----Original Message----- > From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com] > Sent: Monday, June 16, 2014 1:00 AM > To: RICHARDSON, BRUCE; Gooch, Stephen; dev@dpdk.org > Subject: RE: mmap() hint address >=20 > Hi Bruce, Stephen, >=20 > > > Hello, > > > > > > I have seen a case where a secondary DPDK process tries to map uio > > > resource in which mmap() normally sends the corresponding virtual > > > address as a hint address. However on some instances mmap() returns > > > a virtual address that is not the hint address, and it result in > > > rte_panic() and the secondary process goes defunct. > > > > > > This happens from time to time on an embedded device when > > nr_hugepages is > > > set to 128, but never when nr_hugepage is set to 256 on the same > device. > > My > > > question is, if mmap() can find the correct memory regions when > > > hugepages is set to 256, would it not require less resources (and > > > therefore be more likely to > > > pass) at a lower value such as 128? > > > > > > Any ideas what would cause this mmap() behavior at a lower > > > nr_hugepage value? > > > > > > - Stephen > > > > Hi Stephen, > > > > That's a strange one! > > I don't know for definite why this is happening, but here is one > > possible theory. :-) > > > > It could be due to the size of the memory blocks that are getting > mmapped. > > When you use 256 pages, the blocks of memory getting mapped may well > > be larger (depending on how fragmented in memory the 2MB pages are), > > and so may be getting mapped at a higher set of address ranges where > > there is more free memory. This set of address ranges is then free in > > the secondary process and it is similarly able to map the memory. > > With the 128 hugepages, you may be looking for smaller amounts of > > memory and so the addresses get mapped in at a different spot in the > > virtual address space, one that may be more heavily used. Then when > > the secondary process tries to duplicate the mappings, it already has > > memory in that region in use and the mapping fails. > > In short - one theory is that having bigger blocks to map causes the > > memory to be mapped to a different location in memory which is free > > from conflicts in the secondary process. > > > > So, how to confirm or refute this, and generally debug this issue? > > Well, in general we would need to look at the messages printed out at > > startup in the primary process to see how big of blocks it is trying > > to map in each case, and where they end up in the virtual address-space= . >=20 > As I remember, OVDK project has had vaguely similar issues (only they wer= e > trying to map hugepages into the space that QEMU has already occupied). > This resulted in us adding a --base-virtaddr EAL command-line flag that w= ould > specify the start virtual address where primary process would start mappi= ng > pages. I guess you can try that as well (just remember that it needs to b= e > done in the primary process, because the secondary one just copies the > mappings and succeeds or fails to do so). >=20 > Best regards, > Anatoly Burakov > DPDK SW Engineer >=20 >=20 >=20 >=20