From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 817B8A059F for ; Fri, 10 Apr 2020 14:14:02 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6B73B1D40D; Fri, 10 Apr 2020 14:14:02 +0200 (CEST) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 85AA61D381; Fri, 10 Apr 2020 14:14:00 +0200 (CEST) IronPort-SDR: 17tIkxcfQrnIabLPLzZsYTqYUZffG4hqV0vPtctgrLi+P4Mkme5o9MCE2zAPwZfyXIf7D1rWwO 5y2auaWe942w== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Apr 2020 05:13:59 -0700 IronPort-SDR: J40LgcCP672JtHMrZ1xIJoa+FZhBdAXQaXw1M+ZN5t2zUsUxhAv8IlBHjPq8l5qdbWBpiy3sq5 F3haGak9HshA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,366,1580803200"; d="scan'208";a="260466949" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.251.81.66]) ([10.251.81.66]) by orsmga008.jf.intel.com with ESMTP; 10 Apr 2020 05:13:57 -0700 To: Bao-Long Tran , olivier.matz@6wind.com, arybchenko@solarflare.com Cc: dev@dpdk.org, users@dpdk.org References: <0B83F781-A4DB-4775-95CA-E6C2A6838D72@niometrics.com> From: "Burakov, Anatoly" Message-ID: <034f6201-b381-98fd-0637-31f6e2f253e5@intel.com> Date: Fri, 10 Apr 2020 13:13:57 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <0B83F781-A4DB-4775-95CA-E6C2A6838D72@niometrics.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-users] Memory requirement calculation to create mempools with external memory X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" On 04-Apr-20 5:27 AM, Bao-Long Tran wrote: > Hi, > > My goal is to create mbuf pools using external memory with rte_malloc_heap_*(). > The initial implementation goes like this: > - Create an empty heap rte_malloc_heap_create() > - Calculate memory required > - Provision and add the calculated single memory chunk to > the heap rte_malloc_heap_memory_add() > - Create mbuf pool from the heap > > The problematic step is the calculation step. AFAIK there's no generic way to > know how much memory is required to create a particular mempool. The example > I found in testpmd (see below) is quite pessimistic as stated, which leads to > up to 1G overprovision if you're using 1G hugepages. > > I wonder under what circumstances can I make this memory requirement > calculation deterministic. I.e., let's say I have total control over the > underlying hugepages, including page size, iova, alignment, contiguousness > (both iova and va), etc, how can I know exactly how much memory to allocate? > > Anatoly, I don't quite understand your comment in the code below about the > hdr_mem provision in particular. Why do we need to give hdr_mem special > treatment? Further explanation is much appreciated. Hi, When you're creating a mempool, you're not just creating space for mbufs - you're also reserving space for mempool headers and mempool chunk areas. The way they are calculated are IIRC specific to the mempool driver (with default implementation provided), so without knowing the exact layout of memory and how much memory we'll be using up for them (i.e. how many headers it'll take, how big is the mbuf, etc.) and without knowing which specific mempool driver we will be using it is impossible to predict exact mempool memory usage. The memchunks will come from the main memory even if you're reserving from external memory. This is an implementation detail that has to do with the fact that memchunks are allocated using a regular rte_zmalloc() call. Perhaps we should change that, and the zmalloc call should be called on the same socket the mempool sits on, but i don't know the mempool code well enough to understand the implications of such a change. > > Thanks, > BL > > /* extremely pessimistic estimation of memory required to create a mempool */ > static int > calc_mem_size(uint32_t nb_mbufs, uint32_t mbuf_sz, size_t pgsz, size_t *out) > { > unsigned int n_pages, mbuf_per_pg, leftover; > uint64_t total_mem, mbuf_mem, obj_sz; > > /* there is no good way to predict how much space the mempool will > * occupy because it will allocate chunks on the fly, and some of those > * will come from default DPDK memory while some will come from our > * external memory, so just assume 128MB will be enough for everyone. > */ > uint64_t hdr_mem = 128 << 20; > > /* account for possible non-contiguousness */ > obj_sz = rte_mempool_calc_obj_size(mbuf_sz, 0, NULL); > if (obj_sz > pgsz) { > TESTPMD_LOG(ERR, "Object size is bigger than page size\n"); > return -1; > } > > mbuf_per_pg = pgsz / obj_sz; > leftover = (nb_mbufs % mbuf_per_pg) > 0; > n_pages = (nb_mbufs / mbuf_per_pg) + leftover; > > mbuf_mem = n_pages * pgsz; > > total_mem = RTE_ALIGN(hdr_mem + mbuf_mem, pgsz); > > if (total_mem > SIZE_MAX) { > TESTPMD_LOG(ERR, "Memory size too big\n"); > return -1; > } > *out = (size_t)total_mem; > > return 0; > } > -- Thanks, Anatoly