From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8E3CBA00C2 for ; Wed, 9 Feb 2022 23:40:23 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 45E9041101; Wed, 9 Feb 2022 23:40:22 +0100 (CET) Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by mails.dpdk.org (Postfix) with ESMTP id AFED840140 for ; Wed, 9 Feb 2022 23:40:20 +0100 (CET) Received: by mail-pj1-f47.google.com with SMTP id r64-20020a17090a43c600b001b8854e682eso3789837pjg.0 for ; Wed, 09 Feb 2022 14:40:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ytrDq5e7tu8jkB0E5j6IpfuejY1MQsRQ1sZl7FrvlgI=; b=KfTMyVJqR19KLmqz5kXlF85D/zL6b0O827dpKE2G9zh6n8cgXlZFkg4erAwwx9V7xZ p55e98R2VMh7yL4ydcRPd2LkhQIwPpojTeDiuAXokR9upPkwACNtobTLfECLNNTe5vJO peaQOx7LbkWiiK1fl1aUdbgOojEARkOZEC5iUqt92Pg7H6+H0jCx8qzMAjDCek5zeZm3 urVZL5BhxYv3iBVhPa6krhln+QCrs4kk1fvDyRhXP15iU2e6w9SSRQkFDHsycy5ZXGFK 0bWPNmAp+djGPMiSUFP7RST0qFNeZ0c00GRssBuF8R5omrfHyGrGzr0kjCeVj3Mulb32 nLMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ytrDq5e7tu8jkB0E5j6IpfuejY1MQsRQ1sZl7FrvlgI=; b=P8jLYHxy2Vo3KEIOQX/G8o+wdpRLH8UO7T2i42MggeSllSqPT0L+TjRvja6ttWMKKa W9laJzmClHJD6Yld9FbBKeo1HehaOuYULWnqJZUNu4GPvs7kjpqMIuEoa5Xi3gHPZryB fndr3V9pTp27P4xCi/D4myhA/WcD7O0HeoTRiEDGMl5PZsz9fHgYoOQD6h/895F6TqF6 z8dR2VsWkeDNrn1VlbHs5HcDLPFIPkJ3xfwxpaIzyGYyA2FNKiRP+VfjgNuljVR1kiIn 4AYLZVlM8MAha6e0REJVdPw0s/N4Vhd43IocCt19FoAq8+6NalznXRJjZfWTWsQW5KDC IHDg== X-Gm-Message-State: AOAM531+WB8xra+sZm/RZn9980KiPZ1i/gbWRdEt73beBRakraDYwag8 0jdICcRpPnUo44De+os4FF8ndQ== X-Google-Smtp-Source: ABdhPJxdqQe4qVTg7e379Q9cOmcx+L/iDTGlqPuE2VMNgUsUFw4c2maB/1NzduqdHAeuaQy8/NWLYA== X-Received: by 2002:a17:90b:3810:: with SMTP id mq16mr6058277pjb.26.1644446419772; Wed, 09 Feb 2022 14:40:19 -0800 (PST) Received: from hermes.local (204-195-112-199.wavecable.com. [204.195.112.199]) by smtp.gmail.com with ESMTPSA id g4sm7109933pgw.9.2022.02.09.14.40.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Feb 2022 14:40:19 -0800 (PST) Date: Wed, 9 Feb 2022 14:40:16 -0800 From: Stephen Hemminger To: Antonio Di Bacco Cc: users@dpdk.org Subject: Re: Max size for rte_mempool_create Message-ID: <20220209144016.19b75a9f@hermes.local> In-Reply-To: References: <20220209134446.403cf13f@hermes.local> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Wed, 9 Feb 2022 22:58:43 +0100 Antonio Di Bacco wrote: > Thanks! I already reserve huge pages from kernel command line . I reserve 6 > 1G hugepages. Is there any other reason for the ENOMEM? > > On Wed, 9 Feb 2022 at 22:44, Stephen Hemminger > wrote: > > > On Wed, 9 Feb 2022 22:20:34 +0100 > > Antonio Di Bacco wrote: > > > > > I have a system with two numa sockets. Each numa socket has 8GB of RAM. > > > I reserve a total of 6 hugepages (1G). > > > > > > When I try to create a mempool (API rte_mempool_create) of 530432 mbufs > > > (each one with 9108 bytes) I get a ENOMEM error. > > > > > > In theory this mempool should be around 4.8GB and the hugepages are > > enough > > > to hold it. > > > Why is this failing ? > > > > This is likely becaus the hugepages have to be contiguous and > > the kernel has to that many free pages (especially true with 1G pages). > > Therefore it is recommended to > > configure and reserve huge pages on kernel command line during boot. > > Your calculations look about right: elementsize = sizeof(struct rte_mbuf) + private_size + RTE_PKTMBUF_HEADROOM + mbuf_size; object = rte_mempool_calc_objsize(elementsize, 0, NULL); With mbuf_size of 9108 and typical DPDK defaults: elementsize = 128 + 128 + 9108 = 9364 mempool rounds 9364 up to cacheline (64) = 9408 mempool object header = 192 objectsize = 9408 + 192 = 9600 bytes per object Total size of mempool requested = 530432 * 9600 = 4.74G If this a Numa machine you may need to make sure that the kernel has decided to put the hugepages on the right socket. Perhaps it decided to split them across sockets?