From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <users-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 8E3CBA00C2
for <public@inbox.dpdk.org>; Wed, 9 Feb 2022 23:40:23 +0100 (CET)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 45E9041101;
Wed, 9 Feb 2022 23:40:22 +0100 (CET)
Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com
[209.85.216.47]) by mails.dpdk.org (Postfix) with ESMTP id AFED840140
for <users@dpdk.org>; Wed, 9 Feb 2022 23:40:20 +0100 (CET)
Received: by mail-pj1-f47.google.com with SMTP id
r64-20020a17090a43c600b001b8854e682eso3789837pjg.0
for <users@dpdk.org>; Wed, 09 Feb 2022 14:40:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=networkplumber-org.20210112.gappssmtp.com; s=20210112;
h=date:from:to:cc:subject:message-id:in-reply-to:references
:mime-version:content-transfer-encoding;
bh=ytrDq5e7tu8jkB0E5j6IpfuejY1MQsRQ1sZl7FrvlgI=;
b=KfTMyVJqR19KLmqz5kXlF85D/zL6b0O827dpKE2G9zh6n8cgXlZFkg4erAwwx9V7xZ
p55e98R2VMh7yL4ydcRPd2LkhQIwPpojTeDiuAXokR9upPkwACNtobTLfECLNNTe5vJO
peaQOx7LbkWiiK1fl1aUdbgOojEARkOZEC5iUqt92Pg7H6+H0jCx8qzMAjDCek5zeZm3
urVZL5BhxYv3iBVhPa6krhln+QCrs4kk1fvDyRhXP15iU2e6w9SSRQkFDHsycy5ZXGFK
0bWPNmAp+djGPMiSUFP7RST0qFNeZ0c00GRssBuF8R5omrfHyGrGzr0kjCeVj3Mulb32
nLMA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to
:references:mime-version:content-transfer-encoding;
bh=ytrDq5e7tu8jkB0E5j6IpfuejY1MQsRQ1sZl7FrvlgI=;
b=P8jLYHxy2Vo3KEIOQX/G8o+wdpRLH8UO7T2i42MggeSllSqPT0L+TjRvja6ttWMKKa
W9laJzmClHJD6Yld9FbBKeo1HehaOuYULWnqJZUNu4GPvs7kjpqMIuEoa5Xi3gHPZryB
fndr3V9pTp27P4xCi/D4myhA/WcD7O0HeoTRiEDGMl5PZsz9fHgYoOQD6h/895F6TqF6
z8dR2VsWkeDNrn1VlbHs5HcDLPFIPkJ3xfwxpaIzyGYyA2FNKiRP+VfjgNuljVR1kiIn
4AYLZVlM8MAha6e0REJVdPw0s/N4Vhd43IocCt19FoAq8+6NalznXRJjZfWTWsQW5KDC
IHDg==
X-Gm-Message-State: AOAM531+WB8xra+sZm/RZn9980KiPZ1i/gbWRdEt73beBRakraDYwag8
0jdICcRpPnUo44De+os4FF8ndQ==
X-Google-Smtp-Source: ABdhPJxdqQe4qVTg7e379Q9cOmcx+L/iDTGlqPuE2VMNgUsUFw4c2maB/1NzduqdHAeuaQy8/NWLYA==
X-Received: by 2002:a17:90b:3810:: with SMTP id
mq16mr6058277pjb.26.1644446419772;
Wed, 09 Feb 2022 14:40:19 -0800 (PST)
Received: from hermes.local (204-195-112-199.wavecable.com. [204.195.112.199])
by smtp.gmail.com with ESMTPSA id
g4sm7109933pgw.9.2022.02.09.14.40.19
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Wed, 09 Feb 2022 14:40:19 -0800 (PST)
Date: Wed, 9 Feb 2022 14:40:16 -0800
From: Stephen Hemminger <stephen@networkplumber.org>
To: Antonio Di Bacco <a.dibacco.ks@gmail.com>
Cc: users@dpdk.org
Subject: Re: Max size for rte_mempool_create
Message-ID: <20220209144016.19b75a9f@hermes.local>
In-Reply-To: <CAO8pfFnS0sOA4cmQ-CR1Gpc+QM3Fuh6L0ZEMm+d2RZ4BEd3SgA@mail.gmail.com>
References: <CAO8pfFmjrm-qkZJh4GYvNd-6jqoyB-HB1PQCrd1oCu6eh_FSdw@mail.gmail.com>
<20220209134446.403cf13f@hermes.local>
<CAO8pfFnS0sOA4cmQ-CR1Gpc+QM3Fuh6L0ZEMm+d2RZ4BEd3SgA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/users>,
<mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/users>,
<mailto:users-request@dpdk.org?subject=subscribe>
Errors-To: users-bounces@dpdk.org
On Wed, 9 Feb 2022 22:58:43 +0100
Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
> Thanks! I already reserve huge pages from kernel command line . I reserve 6
> 1G hugepages. Is there any other reason for the ENOMEM?
>
> On Wed, 9 Feb 2022 at 22:44, Stephen Hemminger <stephen@networkplumber.org>
> wrote:
>
> > On Wed, 9 Feb 2022 22:20:34 +0100
> > Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
> >
> > > I have a system with two numa sockets. Each numa socket has 8GB of RAM.
> > > I reserve a total of 6 hugepages (1G).
> > >
> > > When I try to create a mempool (API rte_mempool_create) of 530432 mbufs
> > > (each one with 9108 bytes) I get a ENOMEM error.
> > >
> > > In theory this mempool should be around 4.8GB and the hugepages are
> > enough
> > > to hold it.
> > > Why is this failing ?
> >
> > This is likely becaus the hugepages have to be contiguous and
> > the kernel has to that many free pages (especially true with 1G pages).
> > Therefore it is recommended to
> > configure and reserve huge pages on kernel command line during boot.
> >
Your calculations look about right:
elementsize = sizeof(struct rte_mbuf)
+ private_size
+ RTE_PKTMBUF_HEADROOM
+ mbuf_size;
object = rte_mempool_calc_objsize(elementsize, 0, NULL);
With mbuf_size of 9108 and typical DPDK defaults:
elementsize = 128 + 128 + 9108 = 9364
mempool rounds 9364 up to cacheline (64) = 9408
mempool object header = 192
objectsize = 9408 + 192 = 9600 bytes per object
Total size of mempool requested = 530432 * 9600 = 4.74G
If this a Numa machine you may need to make sure that the kernel
has decided to put the hugepages on the right socket.
Perhaps it decided to split them across sockets?