From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BB44E45DAF for ; Wed, 27 Nov 2024 00:50:28 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5CDE640EE3; Wed, 27 Nov 2024 00:50:28 +0100 (CET) Received: from sonic315-21.consmr.mail.ne1.yahoo.com (sonic315-21.consmr.mail.ne1.yahoo.com [66.163.190.147]) by mails.dpdk.org (Postfix) with ESMTP id 844FF402A7 for ; Wed, 27 Nov 2024 00:50:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1732665026; bh=VY4IROFWJ7Wqq6Yt8u1STO2BH30eLpSvloH/vn31CCE=; h=Date:From:To:Cc:In-Reply-To:References:Subject:From:Subject:Reply-To; b=qEubb+QUdO1ZMOqGxfi3EqGdZzhSspE8xwyl5ak5jBBU8/j06S8pI1jfSU7mozp08HafZIyaif69Ph6e7gVWZomD3BSrzBRD3QotAPkcpnpezwEKCqULP9JsksarWPbfGTc4zNxsg2ibBgpD5VT8+lD/9C299UVpu9id5rJViulf4a1Q34t2OELIqGLwMVpouB0HF4jucjsXb2Rm4JEyCHaVd4j+ediK66bjYI05BMhJdBvc27pOeC4J2+tMd9Gvw3STBgfoUM3/iNTWAguV/Tf+AA1akjuHbC78X7Zv/tULSehMiCqNmieiqPvA9l9Z0iOGkJDB0JtLSHzd/dyogA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1732665026; bh=H7FAJp//Fp7pxCEX2N0g5fSasZ9RkQCA9JlwgUTe3vU=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=W03SZhkNAxvQdLb//mn2N60ymY3b6kMlv0MF2ucG/b/QeKo0ZdDsuk5hdJFoWz1r5LpsnAkLi772eHluAqcZiu/sKqrgd8YrtCCfN0GoHoPgBBfL/4iWQ197LBDHuWoINnYpIYeLL3KGrMj4eCpVIOZj0ol3XZh8KUEQFlUArbr8nfBu3jzIMPFJ+HEihDRV8UIu83SW7cYOrIiMsjqusduxz8bxvEzPfWF0eW2VTuDENGzv2VuSuYmuGOSl7ykiW4pYYIQEtfYqTxfZNpMCJu/rb7ycuJcRo5rsJm5gjRw2yxbuGfvl8Vt/KGSB37zjRB/T1RmPdRUSoG/y6uEXSw== X-YMail-OSG: tB.6mTUVM1nlbGZ3YyWIAVrRM4MPR1qunPAWwxDCdE5SoDdzVRm9Gi2t8kY_MYs ruVB4d_X9c9GEt37jSVapkRKAhe97C7chtx1elP80yguDVs_m298zy0we2dqgCwsa80aXx_1_YxN _VoPYCUZOB9h1cPKcsvCLadJu2I4D0Hh6e7_F2LVaSyRxGhpvJTPI9pcLmt3OrHKmsmxO72OLghq 7N_yDXW9mPQ.hSuXQN4gHJ03A477Gw_G1BY7TvzTHB_BGi5AP37vYgZiuJkyya7tYwmJqBNwEkdf 3AcwB2br038Bk5rhBSvYnTAPf1ECrjjtnr_wOmizijFkeQzkpw.4aQYN1HpcEe1VHE6lIVqtS.Jd SjpyoqkUOhDv_BhjTORSpotpOd2GDseycPDxOnio3Vd5d0sroN0RMNLaLTqO8Aiky_Ss9NqcfuG7 7W.Uo19jjsMqHat6Sj0j95PYJ7RhkfnfwLQ3hJHzHAiBeQ.QpN3FBFBTa3nCkwZL4Z7aTP97XEDH mhIVAfY0MzIxbz7GQQ5KVhDCp1AWuE4Sy28624zCM7Iah61bjl7MTDArlTM9E23oZ29ra8P808Ly 2OLT6UaDn6t2NBOUhQnG1_Ko_mRVNo5NS5v_c6mnw61UtYcSPFxlBaMzqmKzfiSX2TvgaWbYG166 3NVqHr1mtR2GS7RzFP3haxG2pLoKLdoSy4ZhEUL1C0eCqd1HnnFWRI2_mhNdeJTbopCWttFikIc8 2h4kUoPlt3redw63On8nfYR7Bf68FAE_VDgsYR5AU6vFWP5n7toG8.e0GO1ZSSftr6SqMh3YGYgH oc33aDsCRc.LAH0tWsvhrl460dxUIOauvWW_dmmaVFqDnDfCNHHNTlyeDy6rur3D1yRBuGHFsjEc O93DCqSmSoz6qcgfG015ZtYiPWFuQJ1Z94tLTIOhlDMvDwisg6Tr98OIgahCqVi6yM60esoNffYi ZBWDq0nXqy9oND1EKu28qZWh_i_uShzdyozO5DCHzuoAOq8iZPsZ7iMI8OG1h4h8058R8wD_ws.z PK.n0dWoK8f0TUg_0wzbV4Dc_FaqPrUXFNxc_r_K7Y6hw51qZZFESGuU6vXByAhV.GsCvU8yTbD1 ugM1KjjNmfkX8QFH2PM7MUB.ciIqYrOY_eKvxq1CMVM6cwJBZrTjrNLX99VZ2hquLvUoQqoKiPF2 fdVsJFF2f3BTAhpl6IHDEhu5_6Ah_bHXAPk7WUDHBCotyF9U1sfooCOAVS1bw8xuyEV3uZtcMK37 vbOduEK3EibTNYTgb5I2HWWHNxsqt_z2wyuYe88eVqRPw3HaAW9rtjma6L5XhQBPPCt92htg.AvV bIC.f7DD7L.gZgWd9t3LeiHzW8tKVX0oRzz2KPguWKbvQRpk9.MJeoClRrwqZvYVbUgZ.C4Y2OQe qPaDGO6728oNK3IKZOZXTGAvAn1Wu5oi4Kyd2fI.ZiS711PKMYq1hpBTZ2R0e_ynF6HLlJ7u0Oxa HPqicBsqPiYg9lZBGSbaWXwl1uzqg2eB0Ie_9VbAzCjMUrfgm2zywfKCVAFsGixjR0vM8ZmVPG7O ipcgioI7tSd6v_1exOrou1VTJkxTXlspdQiTqsxAVesaXBkJBzQpUf.64jIOS50UPizB.t0PN6.N _NNdzDynZ2L3usMP9tIfvq0IYNdyL_dVrlNS_0ov1RmhK1CSSNgxik5aMh984fIAwW9DdE8l1XwK McZf_UF3c0Xikdui6ptxi6ylznAxVqJJuTB_bJtdshx7bPG0Ybbp2nVl9IT2TfwYztJWFynVkGL3 jQMbMUZvHb486P9EuZ3TU5kXreylrnpHPWas9gRDAsOVamUP7OGAbh8BnMOrqT949NefFdWU38uX 1AJvYM.wsV346QaAuFkjAVWCoeYIPYtECUp7Sm0_A91HhHmt1M7Nanmp3nJDOeTKHRIZctFMR3rH R9ZPI.aQuPeUSh.2x8isyMCSg6HSP9okq6Ptwkm01IzQnJHb2mRU7qY99_D7lvBRm6meaA1SAHsr Fq1aSh40ELrv4yFuyF6shJf184Ptqwo3fKbbUopED_HBJgzPS8C.S4ot8RWlR3b8baVa5G1GMqiv uW7.7EcXmlOYnZtkh3N812XigooAuNPV3vPODg1H8jp7gYsl3vW0jI4g3N60bE2cNnLG0nMkaFtv JWCQqlaLTIavjeEANbYiaJtz4B5sIEsPnvhUcCxrm18NSZlmYHFarnznQjvf7BXkhpV6RJaW5nh. jnnjzDdIrBsap5eiPtNHuGGsxEaUd0n9GCzd_99IfE6B5dkcfubAK9h7ig0U- X-Sonic-MF: X-Sonic-ID: 9c4026ec-93cf-4ff3-9eda-686a59428156 Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.ne1.yahoo.com with HTTP; Tue, 26 Nov 2024 23:50:26 +0000 Date: Tue, 26 Nov 2024 23:50:25 +0000 (UTC) From: amit sehas To: Stephen Hemminger Cc: "users@dpdk.org" Message-ID: <341904647.3308895.1732665025670@mail.yahoo.com> In-Reply-To: <450859216.2968663.1732643879391@mail.yahoo.com> References: <67781150.1429748.1732243135675.ref@mail.yahoo.com> <67781150.1429748.1732243135675@mail.yahoo.com> <20241122084557.726e38e7@hermes.local> <450859216.2968663.1732643879391@mail.yahoo.com> Subject: Re: rte_pktmbuf_alloc() out of rte_mbufs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailer: WebService/1.1.22941 YMailNorrin X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Dumping the stats every 10 minutes suggest that there is no slow leak of bu= ffers. The problem arises when the system is under stress and starts perfor= ming extra disk i/o. In this situation dpdk accumulates the buffers and doe= s not return them back to the mempool right away thereby accumulating all t= he 4k buffers allocated to the queue. rte_eth_tx_buffer_flush() should be flushing the buffers and returning them= to the mempool ... is there any additional API that can make sure that thi= s happens. regards On Tuesday, November 26, 2024 at 09:57:59 AM PST, amit sehas wrote:=20 rte_mempool_dump() with debugging enabled finds the following data, below i= see that put_bulk is=C2=A040671864 and=C2=A0get_success_bulk is=C2=A040675= 959, the difference between these is 4095, which is exactly the number of buffers. I will try to dig into the meaning of put_bu= lk and get_success_bulk to determine if there is some kind of buffer leak that is occurring ... some a= mount of code review did not indicate an obvious issue . mempool @0x16c4e2b00 =C2=A0 flags=3D10 =C2=A0 socket_id=3D-1 =C2=A0 pool=3D0x16c4da840 =C2=A0 iova=3D0x3ac4e2b00 =C2=A0 nb_mem_chunks=3D1 =C2=A0 size=3D4095 =C2=A0 populated_size=3D4095 =C2=A0 header_size=3D64 =C2=A0 elt_size=3D2176 =C2=A0 trailer_size=3D128 =C2=A0 total_obj_size=3D2368 =C2=A0 private_data_size=3D64 =C2=A0 ops_index=3D0 =C2=A0 ops_name: =C2=A0 avg bytes/object=3D2368.578266 =C2=A0 stats: =C2=A0 =C2=A0 put_bulk=3D40671864 =C2=A0 =C2=A0 put_objs=3D40671864 =C2=A0 =C2=A0 put_common_pool_bulk=3D4095 =C2=A0 =C2=A0 put_common_pool_objs=3D4095 =C2=A0 =C2=A0 get_common_pool_bulk=3D455 =C2=A0 =C2=A0 get_common_pool_objs=3D4095 =C2=A0 =C2=A0 get_success_bulk=3D40675959 =C2=A0 =C2=A0 get_success_objs=3D40675959 =C2=A0 =C2=A0 get_fail_bulk=3D1 =C2=A0 =C2=A0 get_fail_objs=3D1 On Friday, November 22, 2024 at 08:46:00 AM PST, Stephen Hemminger wrote:=20 On Fri, 22 Nov 2024 02:38:55 +0000 (UTC) amit sehas wrote: > I am frequently running into out of mbufs when allocating packets. When t= his happens is there a way to dump counts of which buffers are where so we = know what is going on? >=20 > I know that each rte_mbuf pool also has per cpu core cache to speed up al= loc/free, and some of the buffers will end up there and if one were to neve= r utilize a particular core for a particular mpool perhaps those mbufs are = lost ... that is my rough guess ... >=20 > How do you debug out of mbufs issue? >=20 > regards The function rte_mempool_dump() will tell you some information about the st= atus of a particular mempool. If you enable mempool statistics you can get more info. The best way to size a memory pool is to account for all the possible place= s mbuf's can be waiting. Something like: =C2=A0 Num Port * Num RxQ * Num RxD + Num Port * Num TxQ * Num TxD + Num Lc= ores * Burst Size + Num Lcores * Cache size Often running out of mbufs is because of failure to free an recveived mbuf,= or a buggy driver.