From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <users-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 91807A04A8
for <public@inbox.dpdk.org>; Wed, 2 Mar 2022 02:37:11 +0100 (CET)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 5121440696;
Wed, 2 Mar 2022 02:37:11 +0100 (CET)
Received: from mx0a-00196b01.pphosted.com (mx0b-00196b01.pphosted.com
[67.231.157.166])
by mails.dpdk.org (Postfix) with ESMTP id E60E140040
for <users@dpdk.org>; Wed, 2 Mar 2022 02:37:09 +0100 (CET)
Received: from pps.filterd (m0072399.ppops.net [127.0.0.1])
by mx0b-00196b01.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 221LTTJF014993;
Tue, 1 Mar 2022 20:37:09 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com;
h=from : to : cc :
subject : date : message-id : references : in-reply-to : content-type :
content-transfer-encoding : mime-version; s=netscout.com.09.24.2020;
bh=PoUvC5phf0GjBrWKZS11qM56d9dfE8HN0FPNCZHDRj0=;
b=ExqhYCSSKfYbBDnJzxj12Hxepd7tneC2HvnJhuop1vPUG02XND5l0ek1sGnu8ZD3lFhn
fgCvu6TokXKgXHK6s4XhS/pzK7aPbFFAjbVsEFJ0H8RfDYXBtodKIkZXXZaMepnxmh+n
rKTeq73ZHbbeMjI223Up1Pk2Qabn/5rAphPeogAF4jR4nLW45FTLzE+uIDwTR/drRp/x
KPxSFanLN1KXANuRG+KQ8Sekhgr2YBkhVX9M9IphJg/Bpvebco9YJ6h/M/EVHQrG9G/A
8+GKsID3ccDFPNJzYc2WYycxfsLqL15B0HUPKwb5IplVkwFng/zA2slWczvZNKklve5G KQ==
Received: from nam12-bn8-obe.outbound.protection.outlook.com
(mail-bn8nam12lp2168.outbound.protection.outlook.com [104.47.55.168])
by mx0b-00196b01.pphosted.com (PPS) with ESMTPS id 3egp0au8xt-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Tue, 01 Mar 2022 20:37:08 -0500
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=EMgregj/fJrjLW6NHbIQgJ8cP9Gwr/JD3tTRaCKpk+2raXnlcKq531JvE/BerSNi8NOuC8ZKxRziFVIwvPWT2hRR7pizMEBwFHl+QoVfkyRtaBeW1KZoSiP77D07fUngfvTpgC+a416omTrof1zurwaeG04DyUGxIwYS4ZNRTBh+VjlMTzby9KHhAVQzsrDXIviSUrACIBHeTEfkESQH8Kxg145NKVDVgvYspudvaX0zLsqqjqWXK1YiQuZzfjheV7IlTO4ZJmNynE1IK7K2jfXU4PUBZhixvsdxx1wXswT3CkfcAU9xZkWEuuAojezXMelRr5zWAjGWS5shcrgtbA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
bh=PoUvC5phf0GjBrWKZS11qM56d9dfE8HN0FPNCZHDRj0=;
b=DoglMj0opHo4AyoAU+3z7E3ZQYb9yWb6jeArqcgbclQ6rhYkp7G41AdlIvwCrvAUMD/OPRBcS4A1l29jE7e1c7WDSQlX15Rbwa+61+te7Hq+JKKIe3sMV5e1G5cIZGl9HQjfZT4hazdg47vo+AWZ1QAkPaAk5AlcyaeOQcZmushNsqepME5bRadpZoRov6cmk1iZY33RHTAqVDvO5ImW0YPi3RowQzsDZezsHzEAP1g7gXkfx9APNEUpTGh2zQEz6NWWBpeST9qc+37rBhA1ni+2CqpoqV4LhB6d3+BsgNtl1OSAkhs1YKp32GFix+wapRCaKkZB1SQ4A5YQ0R62cg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com;
dkim=pass header.d=netscout.com; arc=none
Received: from SJ0PR01MB6399.prod.exchangelabs.com (2603:10b6:a03:298::11) by
CY4PR01MB3208.prod.exchangelabs.com (2603:10b6:903:e5::14) with
Microsoft
SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.5017.26; Wed, 2 Mar 2022 01:37:04 +0000
Received: from SJ0PR01MB6399.prod.exchangelabs.com
([fe80::75a9:94c4:a954:dc0b]) by SJ0PR01MB6399.prod.exchangelabs.com
([fe80::75a9:94c4:a954:dc0b%9]) with mapi id 15.20.5017.027; Wed, 2 Mar 2022
01:37:02 +0000
From: "Lombardo, Ed" <Ed.Lombardo@netscout.com>
To: Stephen Hemminger <stephen@networkplumber.org>, Cliff Burdick
<shaklee3@gmail.com>
CC: "users@dpdk.org" <users@dpdk.org>
Subject: RE: How to increase mbuf size in dpdk version 17.11
Thread-Topic: How to increase mbuf size in dpdk version 17.11
Thread-Index: AdgtgvmYWBNf+aRRSn6QDROyekYkIAAI3y0AAAMjDRAAAF9XgAACaCoAAAXAv5A=
Date: Wed, 2 Mar 2022 01:37:01 +0000
Message-ID: <SJ0PR01MB6399C4B7293DCF4431EB9D4C8F039@SJ0PR01MB6399.prod.exchangelabs.com>
References: <SJ0PR01MB639925215C86846311B241C78F029@SJ0PR01MB6399.prod.exchangelabs.com>
<20220301115638.62387935@hermes.local>
<SJ0PR01MB639963B00671DFC3851E79108F029@SJ0PR01MB6399.prod.exchangelabs.com>
<CA+Gp1namcMUUBf16WyhNJ4LHrpUf8C=ZhtPJZgkT3_eo0KN2ZQ@mail.gmail.com>
<20220301144602.73c8ff95@hermes.local>
In-Reply-To: <20220301144602.73c8ff95@hermes.local>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: d1e13fd2-2084-41ce-845e-08d9fbed26bf
x-ms-traffictypediagnostic: CY4PR01MB3208:EE_
x-microsoft-antispam-prvs: <CY4PR01MB32085CF54B884B346476AE9C8F039@CY4PR01MB3208.prod.exchangelabs.com>
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: 5GUME0MFtfCxnOsb5n54I02xnpEFEqkdPlU0wb4eMm9qLOCpXh4d0dnN6+Ya6IQNZqD9zgyZ3Dlm/tiZCYd/SzB70f4xvjccBr8JALVW3c75UlIpUJnLuAmoQVctxws+HvnGyPCJsCCR0602gRpW+DmHkGgT/VTtBiQEUdmcU0h0GwAhLpKgHpSjvTuJ8Cnr1TCLPEr6A/xTQsTEmItavj8NW7gt6qGRuM+V8TBuX+DP94bK/GS5u3e9bshY3oi6RxFNTyNSvRJ8m3zbhoT5l172oZZy1JMTVOwhietPzmYCikyBk9ovEgasaOhj0Hmt0SVb1TrRshj42eAP3KMjigSNEAwOtdrt+MxJmxq6XyXURx9GZ+5qvx3JSu7dEAn1AUYsn0/hoLY+Bvn+3MNizJGSBVYwjrwXruCQlTNC6GR/OsPbXwgkR29smq/lcqvr9sPd9aGWPQyJ4JwrInAExGTqL6WkSbBMIkEcMi673t/c0L4795dItmVLZRZxjYAnjEHfPVsSmHNNsY+u1bm0rOcRBYPAfYY+d9U0L+Jq4ltjC2AJYqTqoWRHAmoXbA2aDGMQ00l4ICo2LKZAnHGs2NwqcNadgnBDaa1Kpb9nNfBuKpVw9wPqnBexPPXCNz+yLY6OrOkRsCmfbmv8R8JHQL84PKM+c+zVYAmK7P4qBfotraxBMKeKuMPBVBANCfDt7B4e8HMDV4/fVyQFVmLEUQ==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
IPV:NLI; SFV:NSPM; H:SJ0PR01MB6399.prod.exchangelabs.com; PTR:; CAT:NONE;
SFS:(13230001)(366004)(4326008)(6506007)(71200400001)(7696005)(9686003)(122000001)(508600001)(55016003)(83380400001)(110136005)(86362001)(38100700002)(52536014)(316002)(8936002)(5660300002)(38070700005)(2906002)(33656002)(186003)(66446008)(66476007)(66946007)(66556008)(76116006)(64756008)(8676002)(53546011);
DIR:OUT; SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?pTQ0VxuM2RXeQr9gEyMoIYnJYm1nUEhEX+nZ7vPBaDNwj4UzXCeU7m2A7qCI?=
=?us-ascii?Q?QlYgJuDbNxGjEgIGdCBai7QpQwxJQv+io++z+Y1K4HycHSiactRRgZxgd+C3?=
=?us-ascii?Q?AVI64vlkrmt/n1Ca+2Hp59wxYuPkhUjDN4xJyCp5UPe1x8CB7tpYTPCC/2yi?=
=?us-ascii?Q?1+GrLII/lJ+RZtxXkY7I+nXqBqyhhX51K1VZInCKYX3SfrgpnC7/dMm5MV9O?=
=?us-ascii?Q?bsmqQ9VC6P/wfRrV9IVgvJlkvRiZLZxjeuGf+HNYdwCMIj+//sSO3Pb7dFRd?=
=?us-ascii?Q?bYBpac0Kg4B1cUGi6gxQgG3Bp+LIEQGbxaZei1IYDcnzWOnpZr4LKfe/YtZu?=
=?us-ascii?Q?bS9abq9AdbIOYFqk0qxJ9MG9W/MkXkCzp9Uw4GT0+SsAfy/Wf7EJyyO66Ni3?=
=?us-ascii?Q?czL/bAlFGxaG3nMIJVL2++SUuSBMuLO2jlvfDWchUS7P41qsbLX+2kO3Y5LO?=
=?us-ascii?Q?DgYIn9St4QKFznvQgQsQ+Ojw6UwjUTlJapgSCB8U86Pb3wFDeOAJ8ISvc6cG?=
=?us-ascii?Q?s9eeCVOEnoaXZ/egXjFF37yMKlXfLoEesGKFOyu368jMwE86BWc6tUzu0yqJ?=
=?us-ascii?Q?PuRPWdCK+zj6zQJ2kZGw9FPkPHXXL7jKHY6+k6T8Plv/Cfnap/ORRLbEzfiv?=
=?us-ascii?Q?qieKmcBtV+Fzgj974ekN9t14E6da6mc5xLI3+vCCChQQBn86dm0thqkbsIr9?=
=?us-ascii?Q?/qaIRNoTrDmNLCPqCgT1dIiR6BW5clKVqUzlkDdLIfShDmmmLs8w3Dbl3dwf?=
=?us-ascii?Q?Wqq/3skBXJ0WfcmDaCUcO0xduiPuRhZDBrfqAdqC+0uHkLgUtaTcGXPaGmYt?=
=?us-ascii?Q?QYC3Mv9PwW6Td+4q0+S+uh4MU89j6pfitrsu1c/i9jdUlShUXcg4XS+7ZJ1p?=
=?us-ascii?Q?gY4MZdug0QHDfLXbZ0tpXvrt3+EzP61SmOwbd1w7V7SNkOqpfi+HK5Jm3C4Z?=
=?us-ascii?Q?UjVGWrkdKd8xmZUqJYfBsn/edzwTapQWcyw18p1iAnl2+5q4LAa/a5fjb+Ov?=
=?us-ascii?Q?yvNkQNuBl7ldS1WmvulvbHknPegirCwzJ+3Zn6r8KCTWHqiqB+473y3TQ2gQ?=
=?us-ascii?Q?7zaqFuACFR1vDjs7RSj35u7kIuflv4b+eH9JuGJsPgLpgCKtQjWhONsmjcXE?=
=?us-ascii?Q?82YIQO0fIz/pUCIAZCRuf9yuT6z4WlEnEHSIMB2dH6ZE+gflE/UsqHG0eMv0?=
=?us-ascii?Q?GKJohEZiQZI/VsziigsHHou8egzQg/RADctImtRKI5taiqnwxIe5InsoLacj?=
=?us-ascii?Q?dNZs4lVEkYP/N3luickS422rqZH/aghz4W/M4AzowEQ0pCAqsTlJcfYdjSuS?=
=?us-ascii?Q?jgKcouZSC2LRm7usGd6uSj55gnFLGl5HuPcAjIV3r/+TTSLte2DctBsEEQwI?=
=?us-ascii?Q?ILsKUIQanAtRcdW06qYSW2qIVpGUkxrIJXSv9VE9MhrsUtEY2QOA3yh+cMw5?=
=?us-ascii?Q?pByegbmaGVl9G8hXxsWZ+LzcJSU2tZN1E2IyKXup4hD1Zw4UEXZrFz4afrk+?=
=?us-ascii?Q?NMCRYnc8OnHnzLEtRfxFZq6kf8q21AB5TdS2QkkMyQ+LurQIxZv3JoLIGCKC?=
=?us-ascii?Q?n6E8+CexikARivl/cYJWYwC+U5XdEPlnzmfOsaHqwmOoNoVhyaW0hTFPY3EO?=
=?us-ascii?Q?tUQi+oGUUlFOeoLY6q/58qHGXCDCdxJoMkz4PmI8q8e4z4EAwJVDs+FW4DxV?=
=?us-ascii?Q?UxIyp+fzJ5uc9wCsYUgEjNqg4aA=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: netscout.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR01MB6399.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d1e13fd2-2084-41ce-845e-08d9fbed26bf
X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Mar 2022 01:37:02.3823 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: mX0vXFD3veCsm8nH5Uu17nHh4c58s6igfJASfvIaNudsZneCpRKRuOwgcq1OHtIec1KjoX84HHhNI9X611tO/w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR01MB3208
X-Proofpoint-GUID: WLFfXNqjX34kcH5-BHEXj-UBINpvug6N
X-Proofpoint-ORIG-GUID: WLFfXNqjX34kcH5-BHEXj-UBINpvug6N
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy
score=0 malwarescore=0
clxscore=1011 bulkscore=0 adultscore=0 phishscore=0 lowpriorityscore=0
mlxlogscore=999 mlxscore=0 priorityscore=1501 spamscore=0 impostorscore=0
suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
engine=8.12.0-2201110000 definitions=main-2203020006
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/users>,
<mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/users>,
<mailto:users-request@dpdk.org?subject=subscribe>
Errors-To: users-bounces@dpdk.org
Here is the output from rte_mempool_dump() after creating the mbuf " mbuf_p=
ool_create (mbuf_seg_size=3D16512, nb_mbuf=3D32768, socket_id=3D0)":
nb_mbuf_per_pool =3D 32768
mb_size =3D 16640
16512 * 32768 =3D 541,065,216
mempool <mbuf_pool_socket_0>@0x17f811400
flags=3D10
pool=3D0x17f791180
iova=3D0x80fe11400
nb_mem_chunks=3D1
size=3D32768
populated_size=3D32768
header_size=3D64
elt_size=3D16640
trailer_size=3D0
total_obj_size=3D16704
private_data_size=3D64
avg bytes/object=3D16704.000000
internal cache infos:
cache_size=3D250
cache_count[0]=3D0
...
cache_count[126]=3D0
cache_count[127]=3D0
total_cache_count=3D0
common_pool_count=3D32768
no statistics available
-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org>=20
Sent: Tuesday, March 1, 2022 5:46 PM
To: Cliff Burdick <shaklee3@gmail.com>
Cc: Lombardo, Ed <Ed.Lombardo@netscout.com>; users@dpdk.org
Subject: Re: How to increase mbuf size in dpdk version 17.11
External Email: This message originated outside of NETSCOUT. Do not click l=
inks or open attachments unless you recognize the sender and know the conte=
nt is safe.
On Tue, 1 Mar 2022 13:37:07 -0800
Cliff Burdick <shaklee3@gmail.com> wrote:
> Can you verify how many buffers you're allocating? I don't see how=20
> many you're allocating in this thread.
>=20
> On Tue, Mar 1, 2022 at 1:30 PM Lombardo, Ed <Ed.Lombardo@netscout.com>
> wrote:
>=20
> > Hi Stephen,
> > The VM is configured to have 32 GB of memory.
> > Will dpdk consume the 2GB of hugepage memory for the mbufs?
> > I don't mind having less mbufs with mbuf size of 16K vs original=20
> > mbuf size of 2K.
> >
> > Thanks,
> > Ed
> >
> > -----Original Message-----
> > From: Stephen Hemminger <stephen@networkplumber.org>
> > Sent: Tuesday, March 1, 2022 2:57 PM
> > To: Lombardo, Ed <Ed.Lombardo@netscout.com>
> > Cc: users@dpdk.org
> > Subject: Re: How to increase mbuf size in dpdk version 17.11
> >
> > External Email: This message originated outside of NETSCOUT. Do not=20
> > click links or open attachments unless you recognize the sender and=20
> > know the content is safe.
> >
> > On Tue, 1 Mar 2022 18:34:22 +0000
> > "Lombardo, Ed" <Ed.Lombardo@netscout.com> wrote:
> > =20
> > > Hi,
> > > I have an application built with dpdk 17.11.
> > > During initialization I want to change the mbuf size from 2K to 16K.
> > > I want to receive packet sizes of 8K or more in one mbuf.
> > >
> > > The VM running the application is configured to have 2G hugepages.
> > >
> > > I tried many things and I get an error when a packet arrives.
> > >
> > > I read online that there is #define DEFAULT_MBUF_DATA_SIZE that I
> > changed from 2176 to ((2048*8)+128), where 128 is for headroom. =20
> > > The call to rte_pktmbuf_pool_create() returns success with my changes=
.
> > > From the rte_mempool_dump() - "rx_nombuf" - Total number of Rx=20
> > > mbuf
> > allocation failures. This value increments each time a packet arrives.=
=20
> > >
> > > Is there any reference document explaining what causes this error?
> > > Is there a user guide I should follow to make the mbuf size=20
> > > change,
> > starting with the hugepage value? =20
> > >
> > > Thanks,
> > > Ed
> >
> > Did you check that you have enough memory in the system for the=20
> > larger footprint?
> > Using 16K per mbuf is going to cause lots of memory to be consumed.
A little maths you can fill in your own values.
Assuming you want 16K of data.
You need at a minimum [1]
num_rxq :=3D total number of receive queues
num_rxd :=3D number of receive descriptors per receive queue
num_txq :=3D total number of transmit queues (assume all can be full)
num_txd :=3D number of transmit descriptors
num_mbufs =3D num_rxq * num_rxd + num_txq * num_txd + num_cores * burst=
_size
Assuming you are using code copy/pasted from some example like l3fwd.
With 4 Rxq
num_mbufs =3D 4 * 1024 + 4 * 1024 + 4 * 32 =3D 8320
Each mbuf element requires [2]
elt_size =3D sizeof(struct rte_mbuf) + HEADROOM + mbuf_size
=3D 128 + 128 + 16K =3D 16640
obj_size =3D rte_mempool_calc_obj_size(elt_size, 0, NULL)
=3D 16832
So total pool is
num_mbufs * obj_size =3D 8320 * 16832 =3D 140,042,240 ~ 139M
[1] Some devices line bnxt need multiple buffers per packet.
[2] Often applications want additional space per mbuf for meta-data.
=20