From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 91807A04A8 for ; Wed, 2 Mar 2022 02:37:11 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5121440696; Wed, 2 Mar 2022 02:37:11 +0100 (CET) Received: from mx0a-00196b01.pphosted.com (mx0b-00196b01.pphosted.com [67.231.157.166]) by mails.dpdk.org (Postfix) with ESMTP id E60E140040 for ; Wed, 2 Mar 2022 02:37:09 +0100 (CET) Received: from pps.filterd (m0072399.ppops.net [127.0.0.1]) by mx0b-00196b01.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 221LTTJF014993; Tue, 1 Mar 2022 20:37:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=netscout.com.09.24.2020; bh=PoUvC5phf0GjBrWKZS11qM56d9dfE8HN0FPNCZHDRj0=; b=ExqhYCSSKfYbBDnJzxj12Hxepd7tneC2HvnJhuop1vPUG02XND5l0ek1sGnu8ZD3lFhn fgCvu6TokXKgXHK6s4XhS/pzK7aPbFFAjbVsEFJ0H8RfDYXBtodKIkZXXZaMepnxmh+n rKTeq73ZHbbeMjI223Up1Pk2Qabn/5rAphPeogAF4jR4nLW45FTLzE+uIDwTR/drRp/x KPxSFanLN1KXANuRG+KQ8Sekhgr2YBkhVX9M9IphJg/Bpvebco9YJ6h/M/EVHQrG9G/A 8+GKsID3ccDFPNJzYc2WYycxfsLqL15B0HUPKwb5IplVkwFng/zA2slWczvZNKklve5G KQ== Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2168.outbound.protection.outlook.com [104.47.55.168]) by mx0b-00196b01.pphosted.com (PPS) with ESMTPS id 3egp0au8xt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 01 Mar 2022 20:37:08 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EMgregj/fJrjLW6NHbIQgJ8cP9Gwr/JD3tTRaCKpk+2raXnlcKq531JvE/BerSNi8NOuC8ZKxRziFVIwvPWT2hRR7pizMEBwFHl+QoVfkyRtaBeW1KZoSiP77D07fUngfvTpgC+a416omTrof1zurwaeG04DyUGxIwYS4ZNRTBh+VjlMTzby9KHhAVQzsrDXIviSUrACIBHeTEfkESQH8Kxg145NKVDVgvYspudvaX0zLsqqjqWXK1YiQuZzfjheV7IlTO4ZJmNynE1IK7K2jfXU4PUBZhixvsdxx1wXswT3CkfcAU9xZkWEuuAojezXMelRr5zWAjGWS5shcrgtbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PoUvC5phf0GjBrWKZS11qM56d9dfE8HN0FPNCZHDRj0=; b=DoglMj0opHo4AyoAU+3z7E3ZQYb9yWb6jeArqcgbclQ6rhYkp7G41AdlIvwCrvAUMD/OPRBcS4A1l29jE7e1c7WDSQlX15Rbwa+61+te7Hq+JKKIe3sMV5e1G5cIZGl9HQjfZT4hazdg47vo+AWZ1QAkPaAk5AlcyaeOQcZmushNsqepME5bRadpZoRov6cmk1iZY33RHTAqVDvO5ImW0YPi3RowQzsDZezsHzEAP1g7gXkfx9APNEUpTGh2zQEz6NWWBpeST9qc+37rBhA1ni+2CqpoqV4LhB6d3+BsgNtl1OSAkhs1YKp32GFix+wapRCaKkZB1SQ4A5YQ0R62cg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from SJ0PR01MB6399.prod.exchangelabs.com (2603:10b6:a03:298::11) by CY4PR01MB3208.prod.exchangelabs.com (2603:10b6:903:e5::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.26; Wed, 2 Mar 2022 01:37:04 +0000 Received: from SJ0PR01MB6399.prod.exchangelabs.com ([fe80::75a9:94c4:a954:dc0b]) by SJ0PR01MB6399.prod.exchangelabs.com ([fe80::75a9:94c4:a954:dc0b%9]) with mapi id 15.20.5017.027; Wed, 2 Mar 2022 01:37:02 +0000 From: "Lombardo, Ed" To: Stephen Hemminger , Cliff Burdick CC: "users@dpdk.org" Subject: RE: How to increase mbuf size in dpdk version 17.11 Thread-Topic: How to increase mbuf size in dpdk version 17.11 Thread-Index: AdgtgvmYWBNf+aRRSn6QDROyekYkIAAI3y0AAAMjDRAAAF9XgAACaCoAAAXAv5A= Date: Wed, 2 Mar 2022 01:37:01 +0000 Message-ID: References: <20220301115638.62387935@hermes.local> <20220301144602.73c8ff95@hermes.local> In-Reply-To: <20220301144602.73c8ff95@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d1e13fd2-2084-41ce-845e-08d9fbed26bf x-ms-traffictypediagnostic: CY4PR01MB3208:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 5GUME0MFtfCxnOsb5n54I02xnpEFEqkdPlU0wb4eMm9qLOCpXh4d0dnN6+Ya6IQNZqD9zgyZ3Dlm/tiZCYd/SzB70f4xvjccBr8JALVW3c75UlIpUJnLuAmoQVctxws+HvnGyPCJsCCR0602gRpW+DmHkGgT/VTtBiQEUdmcU0h0GwAhLpKgHpSjvTuJ8Cnr1TCLPEr6A/xTQsTEmItavj8NW7gt6qGRuM+V8TBuX+DP94bK/GS5u3e9bshY3oi6RxFNTyNSvRJ8m3zbhoT5l172oZZy1JMTVOwhietPzmYCikyBk9ovEgasaOhj0Hmt0SVb1TrRshj42eAP3KMjigSNEAwOtdrt+MxJmxq6XyXURx9GZ+5qvx3JSu7dEAn1AUYsn0/hoLY+Bvn+3MNizJGSBVYwjrwXruCQlTNC6GR/OsPbXwgkR29smq/lcqvr9sPd9aGWPQyJ4JwrInAExGTqL6WkSbBMIkEcMi673t/c0L4795dItmVLZRZxjYAnjEHfPVsSmHNNsY+u1bm0rOcRBYPAfYY+d9U0L+Jq4ltjC2AJYqTqoWRHAmoXbA2aDGMQ00l4ICo2LKZAnHGs2NwqcNadgnBDaa1Kpb9nNfBuKpVw9wPqnBexPPXCNz+yLY6OrOkRsCmfbmv8R8JHQL84PKM+c+zVYAmK7P4qBfotraxBMKeKuMPBVBANCfDt7B4e8HMDV4/fVyQFVmLEUQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR01MB6399.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(4326008)(6506007)(71200400001)(7696005)(9686003)(122000001)(508600001)(55016003)(83380400001)(110136005)(86362001)(38100700002)(52536014)(316002)(8936002)(5660300002)(38070700005)(2906002)(33656002)(186003)(66446008)(66476007)(66946007)(66556008)(76116006)(64756008)(8676002)(53546011); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?pTQ0VxuM2RXeQr9gEyMoIYnJYm1nUEhEX+nZ7vPBaDNwj4UzXCeU7m2A7qCI?= =?us-ascii?Q?QlYgJuDbNxGjEgIGdCBai7QpQwxJQv+io++z+Y1K4HycHSiactRRgZxgd+C3?= =?us-ascii?Q?AVI64vlkrmt/n1Ca+2Hp59wxYuPkhUjDN4xJyCp5UPe1x8CB7tpYTPCC/2yi?= =?us-ascii?Q?1+GrLII/lJ+RZtxXkY7I+nXqBqyhhX51K1VZInCKYX3SfrgpnC7/dMm5MV9O?= =?us-ascii?Q?bsmqQ9VC6P/wfRrV9IVgvJlkvRiZLZxjeuGf+HNYdwCMIj+//sSO3Pb7dFRd?= =?us-ascii?Q?bYBpac0Kg4B1cUGi6gxQgG3Bp+LIEQGbxaZei1IYDcnzWOnpZr4LKfe/YtZu?= =?us-ascii?Q?bS9abq9AdbIOYFqk0qxJ9MG9W/MkXkCzp9Uw4GT0+SsAfy/Wf7EJyyO66Ni3?= =?us-ascii?Q?czL/bAlFGxaG3nMIJVL2++SUuSBMuLO2jlvfDWchUS7P41qsbLX+2kO3Y5LO?= =?us-ascii?Q?DgYIn9St4QKFznvQgQsQ+Ojw6UwjUTlJapgSCB8U86Pb3wFDeOAJ8ISvc6cG?= =?us-ascii?Q?s9eeCVOEnoaXZ/egXjFF37yMKlXfLoEesGKFOyu368jMwE86BWc6tUzu0yqJ?= =?us-ascii?Q?PuRPWdCK+zj6zQJ2kZGw9FPkPHXXL7jKHY6+k6T8Plv/Cfnap/ORRLbEzfiv?= =?us-ascii?Q?qieKmcBtV+Fzgj974ekN9t14E6da6mc5xLI3+vCCChQQBn86dm0thqkbsIr9?= =?us-ascii?Q?/qaIRNoTrDmNLCPqCgT1dIiR6BW5clKVqUzlkDdLIfShDmmmLs8w3Dbl3dwf?= =?us-ascii?Q?Wqq/3skBXJ0WfcmDaCUcO0xduiPuRhZDBrfqAdqC+0uHkLgUtaTcGXPaGmYt?= =?us-ascii?Q?QYC3Mv9PwW6Td+4q0+S+uh4MU89j6pfitrsu1c/i9jdUlShUXcg4XS+7ZJ1p?= =?us-ascii?Q?gY4MZdug0QHDfLXbZ0tpXvrt3+EzP61SmOwbd1w7V7SNkOqpfi+HK5Jm3C4Z?= =?us-ascii?Q?UjVGWrkdKd8xmZUqJYfBsn/edzwTapQWcyw18p1iAnl2+5q4LAa/a5fjb+Ov?= =?us-ascii?Q?yvNkQNuBl7ldS1WmvulvbHknPegirCwzJ+3Zn6r8KCTWHqiqB+473y3TQ2gQ?= =?us-ascii?Q?7zaqFuACFR1vDjs7RSj35u7kIuflv4b+eH9JuGJsPgLpgCKtQjWhONsmjcXE?= =?us-ascii?Q?82YIQO0fIz/pUCIAZCRuf9yuT6z4WlEnEHSIMB2dH6ZE+gflE/UsqHG0eMv0?= =?us-ascii?Q?GKJohEZiQZI/VsziigsHHou8egzQg/RADctImtRKI5taiqnwxIe5InsoLacj?= =?us-ascii?Q?dNZs4lVEkYP/N3luickS422rqZH/aghz4W/M4AzowEQ0pCAqsTlJcfYdjSuS?= =?us-ascii?Q?jgKcouZSC2LRm7usGd6uSj55gnFLGl5HuPcAjIV3r/+TTSLte2DctBsEEQwI?= =?us-ascii?Q?ILsKUIQanAtRcdW06qYSW2qIVpGUkxrIJXSv9VE9MhrsUtEY2QOA3yh+cMw5?= =?us-ascii?Q?pByegbmaGVl9G8hXxsWZ+LzcJSU2tZN1E2IyKXup4hD1Zw4UEXZrFz4afrk+?= =?us-ascii?Q?NMCRYnc8OnHnzLEtRfxFZq6kf8q21AB5TdS2QkkMyQ+LurQIxZv3JoLIGCKC?= =?us-ascii?Q?n6E8+CexikARivl/cYJWYwC+U5XdEPlnzmfOsaHqwmOoNoVhyaW0hTFPY3EO?= =?us-ascii?Q?tUQi+oGUUlFOeoLY6q/58qHGXCDCdxJoMkz4PmI8q8e4z4EAwJVDs+FW4DxV?= =?us-ascii?Q?UxIyp+fzJ5uc9wCsYUgEjNqg4aA=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SJ0PR01MB6399.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: d1e13fd2-2084-41ce-845e-08d9fbed26bf X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Mar 2022 01:37:02.3823 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: mX0vXFD3veCsm8nH5Uu17nHh4c58s6igfJASfvIaNudsZneCpRKRuOwgcq1OHtIec1KjoX84HHhNI9X611tO/w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR01MB3208 X-Proofpoint-GUID: WLFfXNqjX34kcH5-BHEXj-UBINpvug6N X-Proofpoint-ORIG-GUID: WLFfXNqjX34kcH5-BHEXj-UBINpvug6N X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 malwarescore=0 clxscore=1011 bulkscore=0 adultscore=0 phishscore=0 lowpriorityscore=0 mlxlogscore=999 mlxscore=0 priorityscore=1501 spamscore=0 impostorscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2203020006 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Here is the output from rte_mempool_dump() after creating the mbuf " mbuf_p= ool_create (mbuf_seg_size=3D16512, nb_mbuf=3D32768, socket_id=3D0)": nb_mbuf_per_pool =3D 32768 mb_size =3D 16640 16512 * 32768 =3D 541,065,216 mempool @0x17f811400 flags=3D10 pool=3D0x17f791180 iova=3D0x80fe11400 nb_mem_chunks=3D1 size=3D32768 populated_size=3D32768 header_size=3D64 elt_size=3D16640 trailer_size=3D0 total_obj_size=3D16704 private_data_size=3D64 avg bytes/object=3D16704.000000 internal cache infos: cache_size=3D250 cache_count[0]=3D0 ... cache_count[126]=3D0 cache_count[127]=3D0 total_cache_count=3D0 common_pool_count=3D32768 no statistics available -----Original Message----- From: Stephen Hemminger =20 Sent: Tuesday, March 1, 2022 5:46 PM To: Cliff Burdick Cc: Lombardo, Ed ; users@dpdk.org Subject: Re: How to increase mbuf size in dpdk version 17.11 External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. On Tue, 1 Mar 2022 13:37:07 -0800 Cliff Burdick wrote: > Can you verify how many buffers you're allocating? I don't see how=20 > many you're allocating in this thread. >=20 > On Tue, Mar 1, 2022 at 1:30 PM Lombardo, Ed > wrote: >=20 > > Hi Stephen, > > The VM is configured to have 32 GB of memory. > > Will dpdk consume the 2GB of hugepage memory for the mbufs? > > I don't mind having less mbufs with mbuf size of 16K vs original=20 > > mbuf size of 2K. > > > > Thanks, > > Ed > > > > -----Original Message----- > > From: Stephen Hemminger > > Sent: Tuesday, March 1, 2022 2:57 PM > > To: Lombardo, Ed > > Cc: users@dpdk.org > > Subject: Re: How to increase mbuf size in dpdk version 17.11 > > > > External Email: This message originated outside of NETSCOUT. Do not=20 > > click links or open attachments unless you recognize the sender and=20 > > know the content is safe. > > > > On Tue, 1 Mar 2022 18:34:22 +0000 > > "Lombardo, Ed" wrote: > > =20 > > > Hi, > > > I have an application built with dpdk 17.11. > > > During initialization I want to change the mbuf size from 2K to 16K. > > > I want to receive packet sizes of 8K or more in one mbuf. > > > > > > The VM running the application is configured to have 2G hugepages. > > > > > > I tried many things and I get an error when a packet arrives. > > > > > > I read online that there is #define DEFAULT_MBUF_DATA_SIZE that I > > changed from 2176 to ((2048*8)+128), where 128 is for headroom. =20 > > > The call to rte_pktmbuf_pool_create() returns success with my changes= . > > > From the rte_mempool_dump() - "rx_nombuf" - Total number of Rx=20 > > > mbuf > > allocation failures. This value increments each time a packet arrives.= =20 > > > > > > Is there any reference document explaining what causes this error? > > > Is there a user guide I should follow to make the mbuf size=20 > > > change, > > starting with the hugepage value? =20 > > > > > > Thanks, > > > Ed > > > > Did you check that you have enough memory in the system for the=20 > > larger footprint? > > Using 16K per mbuf is going to cause lots of memory to be consumed. A little maths you can fill in your own values. Assuming you want 16K of data. You need at a minimum [1] num_rxq :=3D total number of receive queues num_rxd :=3D number of receive descriptors per receive queue num_txq :=3D total number of transmit queues (assume all can be full) num_txd :=3D number of transmit descriptors num_mbufs =3D num_rxq * num_rxd + num_txq * num_txd + num_cores * burst= _size Assuming you are using code copy/pasted from some example like l3fwd. With 4 Rxq num_mbufs =3D 4 * 1024 + 4 * 1024 + 4 * 32 =3D 8320 Each mbuf element requires [2] elt_size =3D sizeof(struct rte_mbuf) + HEADROOM + mbuf_size =3D 128 + 128 + 16K =3D 16640 obj_size =3D rte_mempool_calc_obj_size(elt_size, 0, NULL) =3D 16832 So total pool is num_mbufs * obj_size =3D 8320 * 16832 =3D 140,042,240 ~ 139M [1] Some devices line bnxt need multiple buffers per packet. [2] Often applications want additional space per mbuf for meta-data. =20