From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 32A08A04C2; Thu, 14 Nov 2019 09:13:01 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 39AB83423; Thu, 14 Nov 2019 09:13:00 +0100 (CET) Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by dpdk.org (Postfix) with ESMTP id 112D22F4F; Thu, 14 Nov 2019 09:12:59 +0100 (CET) Received: by mail-lj1-f195.google.com with SMTP id e9so5607392ljp.13; Thu, 14 Nov 2019 00:12:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ISSAxhaZ4KnIkXtH2CoxypdU602qMhvHLvYn1L1yLXQ=; b=PVXT30h13uhfwR4gu1KJN8XVp4LxI1/zjumm54xpu1dKFmhSzZKVkit9BFXEugzNgT dfETuRfa2UQEd7zHwpwx/2ihPFBKIMYp0lg2Wuyxlf3aiGZgjs4KRL3Qd6pFhzbR2aO/ sssb2wOOO3huOD3ZV1N8VVL0rnAv+wcISL3NLCPdxho8lSfacbMuybZ6VMATzAmFt5Bt DkUWHRqyy5JcRBDoyMwyEdIbY+QVu9yJOUFRBb3U5wWhtjOGHRrRSczX4bGud/Z+whN6 7kv7PuYXmJ+ouYmJ7ZZVv3kbIAUg91mL11Z6JQWeIbpdAt6z90eM28V8wcUzJbUwus1w A+/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ISSAxhaZ4KnIkXtH2CoxypdU602qMhvHLvYn1L1yLXQ=; b=nXNIHOipOi+07UbCcgKCIEZexeElroQkHtKVdvmzTUNK0lllcDo88nb8mtqvZdUPUZ TM4rbAiz5gaBTTKzHV2J5zGQGmk2BvwzpPTw0yr/+DRGceltU2oyR8OMFsUpvIbX8AYd jo9oKpFJHfS1juv9+yTfQYh3TDXtMv1dgbSDAe0NYhnokYiUb0CQYSOjLDr6FwJyZYdv 6Ytb39Cxr/R/M2YcBmrYk9bwciDr36bwvIDIH4kvFFBYlzaWyHsFisP33w3cA29hO90u S6LbYxtw+iFXw4BzspImVNfs6VvpxvpZt740OXe63my9kukZEwj/QNzKTok6Z/cmZ7SV uaGQ== X-Gm-Message-State: APjAAAXQZDzRCBiApsIULeFBvDsBuBEpT4THN7mHUpYWOAW25sQE9wGG +KOaXx4s7uHfBUkXXDr1sJ8LwzfgMQsdA1JbXO0= X-Google-Smtp-Source: APXvYqyEijoClQ2+ruadVCwkCOgNdA3yQvrGZru7GQy7ptW5hm0PxaU2FDd9wLH1GEcQxoY+BxavMo+qmqsWOaZw6TU= X-Received: by 2002:a2e:998a:: with SMTP id w10mr5625732lji.66.1573719178349; Thu, 14 Nov 2019 00:12:58 -0800 (PST) MIME-Version: 1.0 References: <20191113091927.GA1501@bricha3-MOBL.ger.corp.intel.com> <70f4e9f0-70f7-aa4a-6c5d-c24308d196c2@intel.com> In-Reply-To: <70f4e9f0-70f7-aa4a-6c5d-c24308d196c2@intel.com> From: Venumadhav Josyula Date: Thu, 14 Nov 2019 13:42:45 +0530 Message-ID: To: "Burakov, Anatoly" Cc: Bruce Richardson , users@dpdk.org, dev@dpdk.org, Venumadhav Josyula Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] time taken for allocation of mempool. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Oliver,Bruce, - we were using --SOCKET-MEM Eal flag. - We did not wanted to avoid going back to legacy mode. - we also wanted to avoid 1G huge-pages. Thanks for your inputs. Hi Anatoly, We were using vfio with iommu, but by default it s iova-mode=pa, after changing to iova-mode=va via EAL it kind of helped us to bring down allocation time(s) for mempools drastically. The time taken was brought from ~4.4 sec to 0.165254 sec. Thanks and regards Venu On Wed, 13 Nov 2019 at 22:56, Burakov, Anatoly wrote: > On 13-Nov-19 9:19 AM, Bruce Richardson wrote: > > On Wed, Nov 13, 2019 at 10:37:57AM +0530, Venumadhav Josyula wrote: > >> Hi , > >> We are using 'rte_mempool_create' for allocation of flow memory. This > has > >> been there for a while. We just migrated to dpdk-18.11 from dpdk-17.05. > Now > >> here is problem statement > >> > >> Problem statement : > >> In new dpdk ( 18.11 ), the 'rte_mempool_create' take approximately ~4.4 > sec > >> for allocation compared to older dpdk (17.05). We have som 8-9 mempools > for > >> our entire product. We do upfront allocation for all of them ( i.e. when > >> dpdk application is coming up). Our application is run to completion > model. > >> > >> Questions:- > >> i) is that acceptable / has anybody seen such a thing ? > >> ii) What has changed between two dpdk versions ( 18.11 v/s 17.05 ) from > >> memory perspective ? > >> > >> Any pointer are welcome. > >> > > Hi, > > > > from 17.05 to 18.11 there was a change in default memory model for DPDK. > In > > 17.05 all DPDK memory was allocated statically upfront and that used for > > the memory pools. With 18.11, no large blocks of memory are allocated at > > init time, instead the memory is requested from the kernel as it is > needed > > by the app. This will make the initial startup of an app faster, but the > > allocation of new objects like mempools slower, and it could be this you > > are seeing. > > > > Some things to try: > > 1. Use "--socket-mem" EAL flag to do an upfront allocation of memory for > use > > by your memory pools and see if it improves things. > > 2. Try using "--legacy-mem" flag to revert to the old memory model. > > > > Regards, > > /Bruce > > > > I would also add to this the fact that the mempool will, by default, > attempt to allocate IOVA-contiguous memory, with a fallback to non-IOVA > contiguous memory whenever getting IOVA-contiguous memory isn't possible. > > If you are running in IOVA as PA mode (such as would be the case if you > are using igb_uio kernel driver), then, since it is now impossible to > preallocate large PA-contiguous chunks in advance, what will likely > happen in this case is, mempool will try to allocate IOVA-contiguous > memory, fail and retry with non-IOVA contiguous memory (essentially > allocating memory twice). For large mempools (or large number of > mempools) that can take a bit of time. > > The obvious workaround is using VFIO and IOVA as VA mode. This will > cause the allocator to be able to get IOVA-contiguous memory at the > outset, and allocation will complete faster. > > The other two alternatives, already suggested in this thread by Bruce > and Olivier, are: > > 1) use bigger page sizes (such as 1G) > 2) use legacy mode (and lose out on all of the benefits provided by the > new memory model) > > The recommended solution is to use VFIO/IOMMU, and IOVA as VA mode. > > -- > Thanks, > Anatoly >