From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0521E45D76 for ; Wed, 27 Nov 2024 02:21:10 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D46A64027A; Wed, 27 Nov 2024 02:21:10 +0100 (CET) Received: from sonic315-21.consmr.mail.ne1.yahoo.com (sonic315-21.consmr.mail.ne1.yahoo.com [66.163.190.147]) by mails.dpdk.org (Postfix) with ESMTP id A54604026C for ; Wed, 27 Nov 2024 02:21:08 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1732670467; bh=l6lfUZmb72KhIOC8VAJJBuPr3nX50s4fecnEACnNc0g=; h=Date:From:To:Cc:In-Reply-To:References:Subject:From:Subject:Reply-To; b=SDQfQkV2QfgQQ2+QILU8P/zRsKrgSnkJG38hLLq1+B/pKWjFFNc0YMBX7+DVmoBmtXtcBS1XmEeMsPKpD6Se2hwGnypdwwOowSwSwQXZM0b1Miyl8gvz59kKA+kyC3V+7Y1Lq5/YjojOC7VcyYJEnMTVb4WlpnbGX5iQGc5Ws/YKJtSr8lEBoaufB9tx1fHxdBbMKbzhYXDMQpLChD6VfvisILfHObLjqbU0+4DQ5Q9XK1CntM+oXFOOIDRv3OzqbvIh5cRGh7djzw7OX58mfd1FlCcv2zpxv2laAXzTeWqRpkiO8RCo3E5cFElgkFgZ03GkznhzfdXUZL1/RI5ZSg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1732670467; bh=MSrUXWD0HdZ0p2WQFtt4HWZnXw2cmOGoh+S0IKRs5ec=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=AuzTwupVUIKUiPoLjpjXvka7teJMFEo+ZkjJHMPipQy1ArCKAmBKT4iAq0sazl0s7ftchjmTJXr49ZC4Wz+2XlT+DaVz82Hax+rH0I+zr1Ehslw1WmvlPqCwqfw0CJjUy20sG8toqM6ORSu0kmWKrJyoeVrMLPBL5lludYvYUkgb6eTOVppVgqNmKYgqTQxEEchF/EgA8ubiAMA6YfhpWQJdX6y92aabEMh6p7AuE60lHzC2/FeHlb53So6V8N77IprtVGZ+Jfdi18RMIdr6JEHjJjfWbwY/MfE/0JamxNgG0/GqbY2cd8k+Sl2BBwbGB+ihZ5zsCBpYAun3KNtCZw== X-YMail-OSG: Um.ka1AVM1lwuNhwylOm0iqrL6bwIaTFPAHSMA788jT2.H8ViU0FYfXuio5REkU rbnFgJLCpFxY_5JiOMcRCF0lbI00Zl43pdI9fimP.6.8M88QfZAjIuTURd_5KFJfLDM8faG4BHMd _LiprJIYEBxuz5ruRUHkFzN2D1H.9E85fTBybtJm4MCDyzfssLX85Rpo6rx2mHchl92R5jw6oz0_ 1kz1oiPNZS8kLi0y8NuW1qU9WYocRMBarIailnw5iKuD7kO9EuL_lVoadSNItxkcJt1FvCqJ.lL4 m2ARiunNJ54GVwmFqA.i8.oGOTEfkW8PXy4pUM8pEdMflOBCywViSx_MEBVN8woGIABot9yrJHeO zPi5ZYqe6EdWJbKmZycFca16wzmjhazEajxcgLUUteJ87dyYCUunLAjLyYChm.RkDX_xPxl0VihD GIC14nOfL_OHaClZjcPMqX7AqzcEaWYBx1Jlfx5eeSRl0IWWxfL1x2OrEFjrSbJydO93HWML20zK UpMAeBhz93izCGduKMRSIv_rY37MKACts5MJDto1j4vY1Zi_km9xbquN4o1OX4nKWlF2J7_JUVOL xnp6gGhQDsEWQyaGgB_Srl6bG1Um.0oL0Y3oKHlqtQeDKl1_u1BLQ0eOiqn7v2lpAjIZCY91ofHn jh7fz03F.qLZHTDOlKy4nwnX3ijpdrY93cxMn95x1692a0W2xBQmLOMLOg0nfOsZQEB_Ybc9LvrD OND5xHYKAuxdfY9wPZ.QUKaeA_8hfmwKxTktntk5deZHsskmZN0b2.Osh3z5WuKFferIPxAdHFOv kKfwI_dn9MTpBpGB0zVzeufXKBuA4tAZ7iHz7B6_fNxN12L3.xHYiYL5KYKn1ulfmZ1GIlVkY43H 4Qd._yu3mlYrnnKT1QiXmYzQiSwXo4zV97Ztc2TtpdWaMeqPAtC_pKoYFkTm5QFsPWwBaqKL69GU auKzA1T75boyT98UgJaUsIrLt9X8XiH4nOPdwW9S0AnuiG_RKavYkRU1_2S3v6_q90pq9uCmBCxE fN8niyJgeNlTsxOck6HqyxhOF6pXoddyC31qbccrrerWvNnrii3vcFXuuKidAoS6aWe3QAvGah9o d5z9_Od4YlDUAsZxYNuIRAcLFynR_Lol3oCnlbm0g7raA01.b4YKBsqP_5.tIeTZy0ptLqV.lJdP xhpCCLw55xZRhowDmITLRC9lyVqarAgsXpiRL3ApEThwaNSiF._4UvAu2bjSkcGtSQgUNUL090vI eqLgZwF.hjZhT0swRDGV6WP2gG5H99BKmRHm6DqKw_lAe2.AT8QouEkMwd.DYH7I5PWPmYwFN96l UTnZPmPF1RiXuKGqcA3G6mwL6PERtxEKwdfjY4nWWLigHDpq0mDMAQQLbsNLAgTqSq84Hl_Dao7r Kl.MPqOvMnOX3hu1p7KD9Zzk70LArxdZB3UMZR8Uz.FugbjR6c7efbGW.XiY7w_R7hL8_lEAwpUt OiCZIlXpCRyf0HCpuRUw9aT0.lbYxTr1Uq.LAk7KL.ewsVqAOSj24nUNnGfEX2NzsnwMYZ2wJS.P DvT1VY8W9YBEfBre8jIrjjUxJ_X7wdp1hnoV5qlJgu3vRxDiCwG1vPhsuLoipnb3JjEu4cldqRBX aM0.TQbixORKrQXNBvN.Sd87mkiTzf7kyrci1VR6NSJdIiMvqcbYC5RrRTA4RbsZrEKlN7XOLmwp hNddtjU9FfB4Zi5NtrM1E6G39STjeGu.i0pTDWVfQ18ioR0mkc1d3ciTQ3Betw3K74dO7HHZlb3Z nzcU_FEHudIgwRLl.xKj02.d7HwWVFU7_Dr_jjW_OFHru2nliQeAxbl0dDMw8wfj8zYoSubssG9U 6DDvu.wP1hpNzXPbCR5zzmApFAkUbcsexrmQjGFlSZUgkoO0mTuYds6yXA9iUZoD1Hb08mkL2ZQH bOZHYXx34jQlcSApUUceql_Bwv4C0gpPgjR.JdexX2eIjwhvqABJbdMAcJPMGquGcbd20qqe329o 79Gp4PVP8Immrj1amarO6S02mhCy3R2QlQI8X51b61ckzQ.gif3_zxVgFX.mYLrtgJIaL4LT1ifr efP51EzGAsEbLeXcACdguJ0iwfoy7h_7V6KQyQa2BXjw3SlDa_td.ZCb4nOb_TNhOst.doVyPhxF 6FbmTAbYfO1eLP1LWkfhhpfMOgakiV1RNb4Y_Vpu75MN7KYVjz8dl02YkCeD2RegvpeS7HEcPrwy vMBsZfqE2u11o1Lf4y25USxDblali6OziWtBufWPWAkLHuGeUU.LTzmThiRYrx1N0nJc- X-Sonic-MF: X-Sonic-ID: 73ea8af2-bf30-4232-9253-35cb08c335c7 Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.ne1.yahoo.com with HTTP; Wed, 27 Nov 2024 01:21:07 +0000 Date: Wed, 27 Nov 2024 01:21:04 +0000 (UTC) From: amit sehas To: Stephen Hemminger Cc: "users@dpdk.org" Message-ID: <1128887693.3323512.1732670464665@mail.yahoo.com> In-Reply-To: <20241126165106.6e13ac9c@hermes.local> References: <67781150.1429748.1732243135675.ref@mail.yahoo.com> <67781150.1429748.1732243135675@mail.yahoo.com> <20241122084557.726e38e7@hermes.local> <450859216.2968663.1732643879391@mail.yahoo.com> <341904647.3308895.1732665025670@mail.yahoo.com> <20241126165106.6e13ac9c@hermes.local> Subject: Re: rte_pktmbuf_alloc() out of rte_mbufs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailer: WebService/1.1.22941 YMailNorrin X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org looking at code for rte_eth_tx_buffer_flush(), in the error case where some= buffers were not sent, it has a default callback which will enqueue the bu= ffers back to the mempool, and in the non error case as well when the call = is done it will enqueue the buffers back to the mempool ... this is what we have relied on for ever, otherwise we would not be able to = utilize it at all ... in the transmit case we never explicitly free the buf= fers ...and we are able to run the product through 100s of millions of pack= et transmits, its only under special circumstances that we run into the sai= d issue .. I hope i am not misunderstanding something.=C2=A0 regards On Tuesday, November 26, 2024 at 04:51:09 PM PST, Stephen Hemminger wrote:=20 On Tue, 26 Nov 2024 23:50:25 +0000 (UTC) amit sehas wrote: > Dumping the stats every 10 minutes suggest that there is no slow leak of = buffers. The problem arises when the system is under stress and starts perf= orming extra disk i/o. In this situation dpdk accumulates the buffers and d= oes not return them back to the mempool right away thereby accumulating all= the 4k buffers allocated to the queue. >=20 > rte_eth_tx_buffer_flush() should be flushing the buffers and returning th= em to the mempool ... is there any additional API that can make sure that t= his happens. If you read the code in rte_ethdev.h The rte_eth_tx_buffer_flush is just does a send of the packets that applica= tion has aggregated via rte_eth_tx_buffer. It does nothing vis-a-vis mempools are causing the driver (PMD) to complete= transmits. There are some tuneables such as tx_free_thresh which control when driver s= hould start freeing sent mbufs.=20 Have you isolated the CPU's used by DPDK threads? Is the application stalling because it starts swapping. You may have to mlo= ckall to keep the pages of application from swapping out.