From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 39D86A04C2; Thu, 14 Nov 2019 10:49:54 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 963252BC8; Thu, 14 Nov 2019 10:49:53 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id 9C9BF2BAE; Thu, 14 Nov 2019 10:49:51 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Nov 2019 01:49:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,304,1569308400"; d="scan'208";a="203019841" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.252.7.220]) ([10.252.7.220]) by fmsmga008.fm.intel.com with ESMTP; 14 Nov 2019 01:49:49 -0800 To: Venumadhav Josyula Cc: Bruce Richardson , users@dpdk.org, dev@dpdk.org, Venumadhav Josyula References: <20191113091927.GA1501@bricha3-MOBL.ger.corp.intel.com> <70f4e9f0-70f7-aa4a-6c5d-c24308d196c2@intel.com> From: "Burakov, Anatoly" Message-ID: <133b1b07-77bd-330a-e42c-2a8ad40628b6@intel.com> Date: Thu, 14 Nov 2019 09:49:48 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] time taken for allocation of mempool. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 14-Nov-19 8:12 AM, Venumadhav Josyula wrote: > Hi Oliver,Bruce, > > * we were using --SOCKET-MEM Eal flag. > * We did not wanted to avoid going back to legacy mode. > * we also wanted to avoid 1G huge-pages. > > Thanks for your inputs. > > Hi Anatoly, > > We were using vfio with iommu, but by default it s iova-mode=pa, after > changing to iova-mode=va via EAL it kind of helped us to bring down > allocation time(s) for mempools drastically. The time taken was brought > from ~4.4 sec to 0.165254 sec. > > Thanks and regards > Venu That's great to hear. As a final note, --socket-mem is no longer necessary, because 18.11 will allocate memory as needed. It is however still advisable to use it if you see yourself end up in a situation where the runtime allocation could conceivably fail (such as if you have other applications running on your system, and DPDK has to compete for hugepage memory). I would also suggest using --limit-mem if you desire to limit the maximum amount of memory DPDK will be able to allocate. This will make DPDK behave similarly to older releases in that it will not attempt to allocate more memory than you allow it. > > > On Wed, 13 Nov 2019 at 22:56, Burakov, Anatoly > > wrote: > > On 13-Nov-19 9:19 AM, Bruce Richardson wrote: > > On Wed, Nov 13, 2019 at 10:37:57AM +0530, Venumadhav Josyula wrote: > >> Hi , > >> We are using 'rte_mempool_create' for allocation of flow memory. > This has > >> been there for a while. We just migrated to dpdk-18.11 from > dpdk-17.05. Now > >> here is problem statement > >> > >> Problem statement : > >> In new dpdk ( 18.11 ), the 'rte_mempool_create' take > approximately ~4.4 sec > >> for allocation compared to older dpdk (17.05). We have som 8-9 > mempools for > >> our entire product. We do upfront allocation for all of them ( > i.e. when > >> dpdk application is coming up). Our application is run to > completion model. > >> > >> Questions:- > >> i)  is that acceptable / has anybody seen such a thing ? > >> ii) What has changed between two dpdk versions ( 18.11 v/s 17.05 > ) from > >> memory perspective ? > >> > >> Any pointer are welcome. > >> > > Hi, > > > > from 17.05 to 18.11 there was a change in default memory model > for DPDK. In > > 17.05 all DPDK memory was allocated statically upfront and that > used for > > the memory pools. With 18.11, no large blocks of memory are > allocated at > > init time, instead the memory is requested from the kernel as it > is needed > > by the app. This will make the initial startup of an app faster, > but the > > allocation of new objects like mempools slower, and it could be > this you > > are seeing. > > > > Some things to try: > > 1. Use "--socket-mem" EAL flag to do an upfront allocation of > memory for use > > by your memory pools and see if it improves things. > > 2. Try using "--legacy-mem" flag to revert to the old memory model. > > > > Regards, > > /Bruce > > > > I would also add to this the fact that the mempool will, by default, > attempt to allocate IOVA-contiguous memory, with a fallback to non-IOVA > contiguous memory whenever getting IOVA-contiguous memory isn't > possible. > > If you are running in IOVA as PA mode (such as would be the case if you > are using igb_uio kernel driver), then, since it is now impossible to > preallocate large PA-contiguous chunks in advance, what will likely > happen in this case is, mempool will try to allocate IOVA-contiguous > memory, fail and retry with non-IOVA contiguous memory (essentially > allocating memory twice). For large mempools (or large number of > mempools) that can take a bit of time. > > The obvious workaround is using VFIO and IOVA as VA mode. This will > cause the allocator to be able to get IOVA-contiguous memory at the > outset, and allocation will complete faster. > > The other two alternatives, already suggested in this thread by Bruce > and Olivier, are: > > 1) use bigger page sizes (such as 1G) > 2) use legacy mode (and lose out on all of the benefits provided by the > new memory model) > > The recommended solution is to use VFIO/IOMMU, and IOVA as VA mode. > > -- > Thanks, > Anatoly > -- Thanks, Anatoly