From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EF6E345DAF for ; Tue, 26 Nov 2024 18:58:02 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7ED1A4278C; Tue, 26 Nov 2024 18:58:02 +0100 (CET) Received: from sonic319-20.consmr.mail.ne1.yahoo.com (sonic319-20.consmr.mail.ne1.yahoo.com [66.163.188.82]) by mails.dpdk.org (Postfix) with ESMTP id A46614278A for ; Tue, 26 Nov 2024 18:58:01 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1732643880; bh=IxppU7sVXi56b46N51PpeDadTMbKN8vN+6c4YSZFWt8=; h=Date:From:To:Cc:In-Reply-To:References:Subject:From:Subject:Reply-To; b=Cv++YPDhuvsYfqt4+bN6WJHPJOu3BijQtG8Hhsw/4Nh2jNfpRukZxE5iggnSkcL1Wjm2yfDRFD2rYNzMyJnuZ4HRaDP1XbJ3cwPFx4cjZrHIZY2cd++KB+d7mLjzu4gPzJ+BGjfH7PJiMJ2hr0ttFlAgFiBNZTcuvCgzMeliZRWiFvViZ1ltJg0xmonO6X7IVuNwMgJHg3fLGCxL/d8QQspkcMaYtEor+BdbfjjJp7jh1XFyu6/gKnmb+PdTZtKTAi2ci7mrcCL89YKBiz43VMUx9DUNCR8vgHGcMCxwIsBZenC4M5Bl7rcL0fpVhu8DeJYicRcY8uLjPabXmgTp4A== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1732643880; bh=fRn0f/phaxwuk4LAnLE0G0Xqa+n57vkCkUy22zBdwdL=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=UqJBC1vqFANEdNshm3yFq/Q45GLA9RsV+evQSBqb7mWPDSYqTwaftjD1GpiwDceHQa/mYNjb4yotA6zytrIUn+e+ktI3lmc9yLKzjlA/AmD1fpyZB/jSW9uUW39t094TOa+C20SotsI3SwZsUUyPjZZ6tr9wdG4b2V/Nv8CGIFH7gh4csR9NeP2CKNg7OKBGwS5ijI4A97Yh/kIHp7z48KukPsBcWj5YWmIIAvEgl62SIPK9bQcSFwhHpKjcUh8ZqPFKmytwSnCsGbcy4xrej7Jc1sZ+baqWfpo0jIZnOhD58WKEoo+nOU6ta9HYZuq+JugZhaRVP1SZdtb6QwxUNg== X-YMail-OSG: TWP8alMVM1nOMTsXUNJcjV5yxRjoMFdhL5s.LfDZpAWH1oRHL.oQOOfHB6ay_7p g_sQ8uzEMakAuXTCZLwvv9tkJxKL9ogr3FoRVnoGSlOZIfNc7PEtTWaQUuC36a7ju3Td5ekhEs06 3ycVAmUeJGzYLAlLt.vj2iDNVGI3jSe0kobRSKiDMrh5AVYRaX7RhKUsA3sanrF3TLwLDNjszLIu Zjrg.E1gRUvy3ycPx19PQ6PDhdU3QLF.BcvAkSALeQHtQtRKv9KqzU_70apCvfG_bXpUIelEe21K iwIPFja4gX5TR9Z.0HU7C4RD2Karsz6jmU1ynoGmJD8Lzih1Nyhvau2JgpikNFL9U26Q2mr3JkTD jrqoBelAzyZ3SQYocO1bU707m7rGcb6OOQInfUuU0._Lc0eWZg9Csmiv_W0LHXuFAarZjxD1pzBq 2XPxOv.oIdbJ__JvQEjzHPAMV2P2esSFytz4PM7LJ20uWrnhYtkX.GtBMlFz.Qm828Elzk2toU9u jVNMKw3LWOSDYaZerU2pUdHbvMqXPz_S83UjWGDVUYbraCs3E7Y5RgV8E3ylyPMujlZn4zLB93Zk boLlusMWKf.yZHvHUg_hBHZDjCFNil9dX9aRNkmdxUEhLMSCVrxr10ot5jw7xb_IsqeAbMS_mPWR ilQNF448FxLquhUB2XFvyxmYytW06lRX.ha8bFsGojb0OKMxyppcAcRPFLW6Pg4NBJbUp67bzJzx wGrLO..k_UpnGCWeeMurki_S3L.tZEB9mXCtdWkgblacMfvC68hTTYJ_1x1QSIm5KdqfMn86ALae c1L3b86OCye7fq9WkWWyBnSv5.6a0QZOjm5cygpVu4B786rao1gh9djXfVLVHTk5LmdQPj9JHtwI GeJCPLjGXF19jB2Zg1AyCZ4uDd89QmuWxqmFwlV3DVDdvJ17vmJ95qbdtgmUctoGv7tefsv1vT4p lPS9OhP._cFUDnDZv_MZAN0omr.r_dyj2xKTimQAGdhq1uVZszrFPWB99sAM.3j39JdN0tT0CeM1 8X65sd57VE2b9Mv3FZqiBrUYr8swQ530hY15_lK6xAnKJ.jPmY0t4dSL4vfuJJXZN6z0fuP7gdQ6 Tix1jlMZ0DUvr0EqZx5_EDFASa9KHPEX12G0OFR0ntWllNwSZ74E.xp2p1dCoAZqzZm6MQKenH0n OX4TlbTqfCWWZzwTMan4E1e0g6WkQ2_OGjBcaUh3S.yJf1hUYn3GOswf8mi3fSMkKv3rTmjaOjGI JUktrHC_AJA1Cn8YDnBb0Mgk6D3mUQljAdcarA9_SsMkJBKO9XUWrYEDyztl56IE_5Wq.gWcgMm8 3VAhxyjb6FsN9GevftWVwmMLxl1pyJAtnowe.kyRd9zfSSF2tCbZGlr.Mla8UEgIKRoVIlTr2v9w YIiP1T0TBxyI4KKFDWCZYeqcYXhLosxIXEesPP9C2_FAxW5qiuLkzJCqRm.YUehpCIgGdei0VpJa qpP_3eHevdGzstAuXrDPzVupndK2vfUckqcVeUQM_9SLIT6YGdUhBneOd4YlO0aufWs4YIQXVSiV w5vbMZBZht_iUMDRkaFbN9fZAe2KF2xWFkVNJCr0nxuWYOngPGdXyhwjVjdOpd3X_MjoUS2WU8Zy K3M8GxBP3Ls6QPAkvpjXRpB2NALvIkvWMO0XvnGv0bvmmvKUezbyDGEQ1qEZCtYZao.lcgFOVuba yI33Wn5bNRyUOHcej.BGHmKwMoZr5Epu_mpjJPEEJOUcgrYyoWfXEJSpFq7wQ6UDXcnQV0Qy.eUN LARnPh3Z4qd4s89fZSO275I80tFHMTnCnIwMh7i_xcS_KYFw84u_.6hHUezhhdGtmr0EkTiUjh.q TbcZIU08up72pSeqh_XIuI4DhzmG2yfr2DWOgVQJ7epRJF8.kmLAnp5wIFcCSY1nkk8zBnDrRmaO elbo_dxsdM.WrukYi_vi8kG4u8b15svyuUiWvMaiDkhtvLIt6rCzS1zaAhkdd4LY7DKMwC3OIJFm 5_4HAiE3nPj0DJ7uEwLG8gt9YySVvPZIt4gCE4gupaPqooB7fDxHNSHL5VVvwAUXvMQ8462roGW5 pP4b__0DydclDfw1gJgo7o2lkvWHe9NSj6PSSd_oEEmsSz5Mv9qufl.Cu5FFg.3N5hqkekfJA89q 7yAXSHgUy.G.7OIc9XLEwdNcwQvpVErfAmv31LZqbTsWmgsETwxmEmxAPmcY8N81wLcYmPfHBaKD w4HjZYzdvnfuDfmYrsS9QrIuRaj49wqCp9jjW9NEwFDzgQ9nOsGmGlzbBAew- X-Sonic-MF: X-Sonic-ID: f5234ea6-325d-4482-ba18-bf1389fefdb9 Received: from sonic.gate.mail.ne1.yahoo.com by sonic319.consmr.mail.ne1.yahoo.com with HTTP; Tue, 26 Nov 2024 17:58:00 +0000 Date: Tue, 26 Nov 2024 17:57:59 +0000 (UTC) From: amit sehas To: Stephen Hemminger Cc: "users@dpdk.org" Message-ID: <450859216.2968663.1732643879391@mail.yahoo.com> In-Reply-To: <20241122084557.726e38e7@hermes.local> References: <67781150.1429748.1732243135675.ref@mail.yahoo.com> <67781150.1429748.1732243135675@mail.yahoo.com> <20241122084557.726e38e7@hermes.local> Subject: Re: rte_pktmbuf_alloc() out of rte_mbufs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailer: WebService/1.1.22941 YMailNorrin X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org rte_mempool_dump() with debugging enabled finds the following data, below i= see that put_bulk is=C2=A040671864 and=C2=A0get_success_bulk is=C2=A040675= 959, the difference between these is 4095, which is exactly the number of buffers. I will try to dig into the meaning of put_bu= lk and get_success_bulk to determine if there is some kind of buffer leak that is occurring ... some a= mount of code review did not indicate an obvious issue . mempool @0x16c4e2b00 =C2=A0 flags=3D10 =C2=A0 socket_id=3D-1 =C2=A0 pool=3D0x16c4da840 =C2=A0 iova=3D0x3ac4e2b00 =C2=A0 nb_mem_chunks=3D1 =C2=A0 size=3D4095 =C2=A0 populated_size=3D4095 =C2=A0 header_size=3D64 =C2=A0 elt_size=3D2176 =C2=A0 trailer_size=3D128 =C2=A0 total_obj_size=3D2368 =C2=A0 private_data_size=3D64 =C2=A0 ops_index=3D0 =C2=A0 ops_name: =C2=A0 avg bytes/object=3D2368.578266 =C2=A0 stats: =C2=A0 =C2=A0 put_bulk=3D40671864 =C2=A0 =C2=A0 put_objs=3D40671864 =C2=A0 =C2=A0 put_common_pool_bulk=3D4095 =C2=A0 =C2=A0 put_common_pool_objs=3D4095 =C2=A0 =C2=A0 get_common_pool_bulk=3D455 =C2=A0 =C2=A0 get_common_pool_objs=3D4095 =C2=A0 =C2=A0 get_success_bulk=3D40675959 =C2=A0 =C2=A0 get_success_objs=3D40675959 =C2=A0 =C2=A0 get_fail_bulk=3D1 =C2=A0 =C2=A0 get_fail_objs=3D1 On Friday, November 22, 2024 at 08:46:00 AM PST, Stephen Hemminger wrote:=20 On Fri, 22 Nov 2024 02:38:55 +0000 (UTC) amit sehas wrote: > I am frequently running into out of mbufs when allocating packets. When t= his happens is there a way to dump counts of which buffers are where so we = know what is going on? >=20 > I know that each rte_mbuf pool also has per cpu core cache to speed up al= loc/free, and some of the buffers will end up there and if one were to neve= r utilize a particular core for a particular mpool perhaps those mbufs are = lost ... that is my rough guess ... >=20 > How do you debug out of mbufs issue? >=20 > regards The function rte_mempool_dump() will tell you some information about the st= atus of a particular mempool. If you enable mempool statistics you can get more info. The best way to size a memory pool is to account for all the possible place= s mbuf's can be waiting. Something like: =C2=A0 Num Port * Num RxQ * Num RxD + Num Port * Num TxQ * Num TxD + Num Lc= ores * Burst Size + Num Lcores * Cache size Often running out of mbufs is because of failure to free an recveived mbuf,= or a buggy driver.