From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6E97E45D31 for ; Wed, 27 Nov 2024 01:51:10 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4982A402E0; Wed, 27 Nov 2024 01:51:10 +0100 (CET) Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by mails.dpdk.org (Postfix) with ESMTP id 7AC50402A7 for ; Wed, 27 Nov 2024 01:51:09 +0100 (CET) Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-7fbd70f79f2so4172153a12.2 for ; Tue, 26 Nov 2024 16:51:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1732668668; x=1733273468; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=XpQYkcKCNgbufToBz8S/Oo4aF2Dgi4HolU7/xUvE53Q=; b=lMVGMjK2NeOciOdFYa7M1EPj8ZHFJXiSkxbfRJ58Q1a2v9f8vgFWg1YVjT/j5XhMFu xIioNwAi2R1nz9hWmYoHbxbogdLAnTYtXgkM3MTy3R3qmeQ/KcJsJC+WJB0N13MCH7UW 3i4NmyMZCFliMRSqYwkqwXcrxWkFOksafBfoaWyKeuvd+x181btQsyXn0BXSx3v5TL4g 8YQUwof3n9F0iI6jHi3ROyeOqCxrk7tv7ZUMVgBGDCWwu+r/IYFg8eS2kteDIMhCKYos phQiRcP/sqM+U+BF24vf5Q2ElL+d1p5uMqvBKjrxV1LFYwkiOtEC8uZgonhKrJRGWpt7 Impw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732668668; x=1733273468; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XpQYkcKCNgbufToBz8S/Oo4aF2Dgi4HolU7/xUvE53Q=; b=JhP3FQZVkV8CHoFlLQIaV6gHTKT/uzEc4S1Hm+qT8ON9ZKHrxIihXIu7WBq1wbqUfJ IkIQZoaAa+C+/JAv4P4mUscQCzRHjbhlRNVeu6lrJVxSuItg2dGUQBMYPLUVIJ2BtiQd Qy1FeY6lHUHl3F1MNvm/MyCStoHUl2rJU6CVJULOuvRMEH6wvi+SDHh6sxjHGU+EX8EZ rup7u+m/eEt3TOwKtAbFkUB+b1/Q0DLcwx/oJr0znTF/Cs55jYv5wLMa7eiO0HUyy1qK WLVWLC9STRVDFQ/bMFOhF0Gd26jj2CkJJl/QXES9jlm4KCenV6Ac3vW0ZdFbInoKoEce uvxQ== X-Gm-Message-State: AOJu0Yy2icCBHZUwNJi2vRv9kxosUkyxIDeZBe61eT+qLMjcU7qFGKpv sLgTssu9witt6eiEK/54PQ9OdGOZZNkH+RsPPvftQ3vC8STcurv+X3UcU/0KQ58= X-Gm-Gg: ASbGnct+Ke8IJucIsQRFwB9AxlQBpb+NC5AIu0XZ5lSwvkjG09xyzKQCzs4/+o0tLOc qVzMOXYUUTSUh4vPIYB+zaL5BWyHJw/uKhy/YOnHiO7MYjl7Vsh91qBvU200xW16QqOD50XCMXX Q/xgIsPkoL98AtZVa/GA3q85ld+EiYP1OeBPLJnL64nIHHs0NO7c36IgsjFq4ZPJLnEIKjk20jW +2zYrdMVsUGK/cDdnnXa53LSuIbMXRKjSMsTfdIFDLq5gHcVGWeQwSY5D6xCHbxjs5Tzz/fBdhz gDsLn0q2/jrRZ73U72F02bzRTM4= X-Google-Smtp-Source: AGHT+IEhOWgsEfBMWUA8/sjBa5U9BcnEsSBjOA8yUPdHWrB4KtoJQjiGj0wSkZUjAG+N+62lBP3w2w== X-Received: by 2002:a05:6a20:6a28:b0:1e0:d6d5:39c3 with SMTP id adf61e73a8af0-1e0e0aada91mr2360018637.8.1732668668561; Tue, 26 Nov 2024 16:51:08 -0800 (PST) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-724e0fca304sm8872572b3a.175.2024.11.26.16.51.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Nov 2024 16:51:08 -0800 (PST) Date: Tue, 26 Nov 2024 16:51:06 -0800 From: Stephen Hemminger To: amit sehas Cc: "users@dpdk.org" Subject: Re: rte_pktmbuf_alloc() out of rte_mbufs Message-ID: <20241126165106.6e13ac9c@hermes.local> In-Reply-To: <341904647.3308895.1732665025670@mail.yahoo.com> References: <67781150.1429748.1732243135675.ref@mail.yahoo.com> <67781150.1429748.1732243135675@mail.yahoo.com> <20241122084557.726e38e7@hermes.local> <450859216.2968663.1732643879391@mail.yahoo.com> <341904647.3308895.1732665025670@mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Tue, 26 Nov 2024 23:50:25 +0000 (UTC) amit sehas wrote: > Dumping the stats every 10 minutes suggest that there is no slow leak of buffers. The problem arises when the system is under stress and starts performing extra disk i/o. In this situation dpdk accumulates the buffers and does not return them back to the mempool right away thereby accumulating all the 4k buffers allocated to the queue. > > rte_eth_tx_buffer_flush() should be flushing the buffers and returning them to the mempool ... is there any additional API that can make sure that this happens. If you read the code in rte_ethdev.h The rte_eth_tx_buffer_flush is just does a send of the packets that application has aggregated via rte_eth_tx_buffer. It does nothing vis-a-vis mempools are causing the driver (PMD) to complete transmits. There are some tuneables such as tx_free_thresh which control when driver should start freeing sent mbufs. Have you isolated the CPU's used by DPDK threads? Is the application stalling because it starts swapping. You may have to mlockall to keep the pages of application from swapping out.