From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1A694A04C4; Wed, 13 Nov 2019 22:06:45 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B18222B8B; Wed, 13 Nov 2019 22:06:44 +0100 (CET) Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by dpdk.org (Postfix) with ESMTP id DB33B2B87; Wed, 13 Nov 2019 22:06:43 +0100 (CET) Received: by mail-lj1-f194.google.com with SMTP id d5so4199435ljl.4; Wed, 13 Nov 2019 13:06:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QPocVuNx9zN3KgREMiPpZIBh3EKdnw4M7IVeiZgOdqE=; b=OQYoKMENC3zpmx1i7kB9U5LKUq18BvD6lt7KgdaG9NEEEu0m9lRtgNvtv8B9G4znPo XCUqsQXku//l4fSo3dQfas5gXeB99hWFs/71KeSoCE7jl9ZOLmuev3bQYVEy5irvgE5W zKfbMbyRbeZ3mnLKH7P3rmDHxxD+4XU4Ht5bxadKmVvau5969WYZom/W8u/BKmUdOwou KTPMy6AGZKOMVfLBgFopkJ+NZFCSEZwkMf6IgLGxdDNII1vN2T39I9e8u5YSzj6Vc7Sm LpA/rvT8HOiccC5MqUf2kVREMW9Re7+xkuxOghjr7LgrU+P40mOVOdGZTcNbk7t/2O89 02TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QPocVuNx9zN3KgREMiPpZIBh3EKdnw4M7IVeiZgOdqE=; b=SVX/p2kAIUOFUVhJTo4hTXVvW8o8ymAk7/ChIq84TeZ1vb+egt3zgA1HZ3xZQhOxzA dfUma0Y2wsRYJdsPCIY6nPF4bgahmgOEHChw+lJUu49lHboipsH+Qmgih8vl84oNrq2m zh2ds0Qw69kvdil/IBP/5BVdXk439TAoU6f7Zwrx7bxLCWGmTXp+Ll9/qE4g+KwB+pLq EhW0Wh9nhwD/uwUrfOS7tYCnVQbQUUooedHFi0fPymjciakyQw0rrK1ToPFfeBeQFiSG vK99JhO2prCoMxeMuaWWgYAwW4vQ3JyBm4y37106w3Htv7FGg5CPswe3OlVrZ0Bxndyq p1og== X-Gm-Message-State: APjAAAWObHiJCu2n6W/Ffnmt4ZkeLTb5S3MPDFn5FHzMSlPIj9FitYhs H5rHktxRmZeiiOJRLg2a2NGkozBiIXljOEH2Gx8= X-Google-Smtp-Source: APXvYqytZ8JU+P0POiclW58kC1g7LVer+06JHv2EsvPLFXR0ZgBzc6foo/Rc52qsH1a/8CLz/JB+TAtzdpAoblQgMEs= X-Received: by 2002:a2e:3805:: with SMTP id f5mr3906819lja.220.1573679203208; Wed, 13 Nov 2019 13:06:43 -0800 (PST) MIME-Version: 1.0 References: <CA+i0PGV9DxiwwyL-AXCuhMcndZ=11Yk+t6KOub-R7yYuaB1qzQ@mail.gmail.com> <20191113091927.GA1501@bricha3-MOBL.ger.corp.intel.com> <70f4e9f0-70f7-aa4a-6c5d-c24308d196c2@intel.com> In-Reply-To: <70f4e9f0-70f7-aa4a-6c5d-c24308d196c2@intel.com> From: Venumadhav Josyula <vjosyula@gmail.com> Date: Thu, 14 Nov 2019 02:31:05 +0530 Message-ID: <CA+i0PGVfe1AKaR-JZB+8LkFSWaQeLtcJqCHH09ekNcz1pbgF7w@mail.gmail.com> To: "Burakov, Anatoly" <anatoly.burakov@intel.com> Cc: Bruce Richardson <bruce.richardson@intel.com>, users@dpdk.org, dev@dpdk.org, Venumadhav Josyula <vjosyula@parallelwireless.com> Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] time taken for allocation of mempool. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> Hi Anatoly, By default w/o specifying --iova-mode option is iova-mode=pa by default ? Thanks Venu On Wed, 13 Nov, 2019, 10:56 pm Burakov, Anatoly, <anatoly.burakov@intel.com> wrote: > On 13-Nov-19 9:19 AM, Bruce Richardson wrote: > > On Wed, Nov 13, 2019 at 10:37:57AM +0530, Venumadhav Josyula wrote: > >> Hi , > >> We are using 'rte_mempool_create' for allocation of flow memory. This > has > >> been there for a while. We just migrated to dpdk-18.11 from dpdk-17.05. > Now > >> here is problem statement > >> > >> Problem statement : > >> In new dpdk ( 18.11 ), the 'rte_mempool_create' take approximately ~4.4 > sec > >> for allocation compared to older dpdk (17.05). We have som 8-9 mempools > for > >> our entire product. We do upfront allocation for all of them ( i.e. when > >> dpdk application is coming up). Our application is run to completion > model. > >> > >> Questions:- > >> i) is that acceptable / has anybody seen such a thing ? > >> ii) What has changed between two dpdk versions ( 18.11 v/s 17.05 ) from > >> memory perspective ? > >> > >> Any pointer are welcome. > >> > > Hi, > > > > from 17.05 to 18.11 there was a change in default memory model for DPDK. > In > > 17.05 all DPDK memory was allocated statically upfront and that used for > > the memory pools. With 18.11, no large blocks of memory are allocated at > > init time, instead the memory is requested from the kernel as it is > needed > > by the app. This will make the initial startup of an app faster, but the > > allocation of new objects like mempools slower, and it could be this you > > are seeing. > > > > Some things to try: > > 1. Use "--socket-mem" EAL flag to do an upfront allocation of memory for > use > > by your memory pools and see if it improves things. > > 2. Try using "--legacy-mem" flag to revert to the old memory model. > > > > Regards, > > /Bruce > > > > I would also add to this the fact that the mempool will, by default, > attempt to allocate IOVA-contiguous memory, with a fallback to non-IOVA > contiguous memory whenever getting IOVA-contiguous memory isn't possible. > > If you are running in IOVA as PA mode (such as would be the case if you > are using igb_uio kernel driver), then, since it is now impossible to > preallocate large PA-contiguous chunks in advance, what will likely > happen in this case is, mempool will try to allocate IOVA-contiguous > memory, fail and retry with non-IOVA contiguous memory (essentially > allocating memory twice). For large mempools (or large number of > mempools) that can take a bit of time. > > The obvious workaround is using VFIO and IOVA as VA mode. This will > cause the allocator to be able to get IOVA-contiguous memory at the > outset, and allocation will complete faster. > > The other two alternatives, already suggested in this thread by Bruce > and Olivier, are: > > 1) use bigger page sizes (such as 1G) > 2) use legacy mode (and lose out on all of the benefits provided by the > new memory model) > > The recommended solution is to use VFIO/IOMMU, and IOVA as VA mode. > > -- > Thanks, > Anatoly >