From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f173.google.com (mail-qk0-f173.google.com [209.85.220.173]) by dpdk.org (Postfix) with ESMTP id 5B32A530F for ; Tue, 15 Mar 2016 02:47:30 +0100 (CET) Received: by mail-qk0-f173.google.com with SMTP id x1so1306446qkc.1 for ; Mon, 14 Mar 2016 18:47:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=jQtWKyc8KB1OSGhOWp/i9OgK+jGqCfFnLNMUFox95No=; b=URqt6BF1a/XjEFTip61z4Q+BXP+hqd8VDvLrr2uwPQDxl/s8MRYY8r3SWrP+h20tGE WM7tt7I4XWsJuLlYHvRLQpVUTqjc+sCMYttD0Wtd/RCQ4bT4cAhuG7KrxnsvjcMbREYZ ZRCbME0cGMLSRQuTF8HaA+CXMKCPrl7rDsL1U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=jQtWKyc8KB1OSGhOWp/i9OgK+jGqCfFnLNMUFox95No=; b=g/gzn3EbirYTnt8/jxnybgLpK1adRJrZgowc5ps8qfOCZMAz/TuSpw3tyhQS6WIwp6 F+ZHQO+9jDwdk9MehdfhtPNXjWEhBcehqrH/lvp8k0oWpIiQgYlFh80qr1c5OupdB3ft CddbFMzprBb4FxUa/SpZTnZ3NfgzMoLoVtLDtVbhSMI1zUEf4r+IIa2nFsw0xUWhWihI 4LMgpXnHF8OgEAGbeKwF9e7Euv8MEuzJWcRps6uJ6IiN6sb/yAiXlXoLcFdPO+kg0JZe Xb6/hsKiGA7dAQfTWSGoF9JbM62BfWMwoe+L/nNAj2KayIHR2oUqE4dUslPI3FhwU0HN LiGw== X-Gm-Message-State: AD7BkJJ7J7IBkwApGLlYW0s5dlkmru4q8B97R57UJriP1yGzsoTzX4OAUrr0HWIaZAog7NrpIhmU1kcXVgasZihO MIME-Version: 1.0 X-Received: by 10.55.55.138 with SMTP id e132mr34141854qka.11.1458006449916; Mon, 14 Mar 2016 18:47:29 -0700 (PDT) Received: by 10.140.98.7 with HTTP; Mon, 14 Mar 2016 18:47:29 -0700 (PDT) In-Reply-To: References: Date: Mon, 14 Mar 2016 18:47:29 -0700 Message-ID: From: John Boyle To: Saurabh Mishra Cc: users@dpdk.org, dev@dpdk.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Huge pages to be allocated based on number of mbufs X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Mar 2016 01:47:30 -0000 Hi Saurabh, I don't know all the details of your setup, but I'm guessing that you may have run into the hugepage fragmentation issue. Try calling rte_malloc_dump_stats(stdout, "dummy") right before the mempool creation. Output might look like this: Socket:0 Heap_size:2147472192, Free_size:2047523584, Alloc_size:99948608, Greatest_free_size:130023360, Alloc_count:82, Free_count:179, (That would be after a successful allocation of a ~99MB mbuf pool.) The mbuf_pool gets allocated with a single giant call to the internal malloc_heap_alloc function. If the "Greatest_free_size" is smaller than the mbuf_pool you're trying to create, then the alloc will fail. Now, if the total free size is smaller or is not much larger than what you're trying to allocate, then you would be advised to give it more hugepages. On the other hand, if the total "Free_size" is much larger than what you need, but the "Greatest_free_size" is considerably smaller (in the above example, the largest free slab is 130 MB despite nearly 2GB being available), then you have a considerably fragmented heap. How do you get a fragmented heap during the initialization phase of the program? The heap is created by mmapping a bunch of hugepages, noticing which ones happen to have adjacent physical addresses, and then the contiguous chunks become the separate available slabs in the heap. If the system has just been booted, then you are likely to end up with a nice large slab into which you can fit a huge mbuf_pool. If the system's been running for a while, it's more likely to be fragmented, in which case you may get something like the example I pasted above. At Pure Storage, we ended up solving this by reserving a single 1GB hugepage, which can't be fragmented. -- John Boyle *Science is what we understand well enough to explain to a computer. Art is everything else we do.* --Knuth On Mon, Mar 14, 2016 at 10:54 AM, Saurabh Mishra wrote: > Hi, > > We are planning to support virtio, vmxnet3, ixgbe, i40e, bxn2x and SR-IOV > on some of them with DPDK. > > We have seen that even if we give correct number of mbufs given the number > hugepages reserved, rte_eth_tx_queue_setup() may still fail with no enough > memory (I saw this on i40evf but worked on virtio and vmxnet3). > > We like to know what's the recommended way to determine how many hugepages > we should allocate given the number of mbufs such that queue setup APIs > also don't fail. > > Since we will be running on low-end systems too we need to be careful about > reserving hugepages. > > Thanks, > /Saurabh >