From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F1B3443F37; Fri, 3 May 2024 20:58:06 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 702E7402D4; Fri, 3 May 2024 20:58:06 +0200 (CEST) Received: from dkmailrelay1.smartsharesystems.com (smartserver.smartsharesystems.com [77.243.40.215]) by mails.dpdk.org (Postfix) with ESMTP id 6D85A402D1 for ; Fri, 3 May 2024 20:58:05 +0200 (CEST) Received: from smartserver.smartsharesystems.com (smartserver.smartsharesys.local [192.168.4.10]) by dkmailrelay1.smartsharesystems.com (Postfix) with ESMTP id 5B003219BB; Fri, 3 May 2024 20:58:04 +0200 (CEST) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: RE: Need help with reducing VIRT memory Date: Fri, 3 May 2024 20:58:00 +0200 X-MimeOLE: Produced By Microsoft Exchange V6.5 Message-ID: <98CBD80474FA8B44BF855DF32C47DC35E9F41A@smartserver.smartshare.dk> In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Need help with reducing VIRT memory Thread-Index: AdqdceJJbw0ES+TWQhmVRbyx/hsDxgAE8/Hw References: <20240502230352.7bb1ef0d@sovereign> From: =?iso-8859-1?Q?Morten_Br=F8rup?= To: "Bruce Richardson" , "Lombardo, Ed" Cc: "Dmitry Kozlyuk" , , X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > From: Bruce Richardson [mailto:bruce.richardson@intel.com] > Sent: Friday, 3 May 2024 17.52 >=20 > On Fri, May 03, 2024 at 04:27:39PM +0100, Bruce Richardson wrote: > > On Fri, May 03, 2024 at 02:48:12PM +0000, Lombardo, Ed wrote: > > > Hi Dmitry, > > > I am not clear on the DPDK memory layout and how to tweak these > #define values. > > > > > > #define RTE_MAX_MEMSEG_PER_LIST 8192 > > > #define RTE_MAX_MEM_MB_PER_LIST 32768 > > > #define RTE_MAX_MEMSEG_PER_TYPE 32768 > > > #define RTE_MAX_MEM_MB_PER_TYPE 65536 > > > > > > I want to limit how much DPDK grabs for memory, but grabs what it > absolutely needs for our application. > > > > Hi, > > > > This is what DPDK does. What is being shown in the VIRT figures is = the > > address space reservation DPDK has made, but not what memory it > actually uses. > > Only sufficient hugepage memory to meet the demands of your app = should > be > > mapped by DPDK, the rest is unused address space that is not taking = up > any > > actual memory. > > On DPDK 17.11, DPDK allocates the command line specified amount of = hugemem (--socket-mem ). E.g. if a hardware appliance with 8 GB RAM running Linux has been = boot-time configured with 6 GB hugemem (e.g. 4 * 1 GB gigantic hugepages = and 1024 * 2 MB hugepages), and DPDK allocates all 6 GB hugemem at EAL = init, no hugepages will be available for other applications, regardless = if DPDK actually uses this memory or not. This is especially relevant for "embedded" systems with only FLASH and = RAM, and no swap. I don't know if other DPDK versions behave differently. I haven't looked = into this in detail. I certainly don't hope recent DPDK versions assume swap is available, = and blindly allocate obscene amounts of memory. Configuring overcommitted hugepages might help with the problem of DPDK = allocating all available hugepages: Instead of reserving a fixed number of hugepages at boot-time, by = setting nr_hugepages, set nr_overcommit_hugepages to allow applications = to dynamically allocate hugepages, where the number of hugepages in the = system grows and shrinks with the amount of hugemem allocated by = applications. Just set it to a sufficiently large number. Please note that the Linux kernel does not support overcommitting 1 GB = gigantic hugepages, only 2 MB hugepages. > By way of illustration, here is the memory output for a testpmd = process > on > my system. I got this by running "top -b -p " >=20 > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 2336969 bruce 20 0 256.2g 26432 19712 S 93.8 0.0 5:28.13 > dpdk-testpmd >=20 > If we look at the memory relevant columns, indeed VIRT shows a huge > value - > 256G in my case. However, the actual RAM used by testpmd is given in = the > "RES" (resident??) column, showing that testpmd actually is only using > 26,432kB of memory in this instance, of which 19,712kB is shared = memory > (mostly hugepages). In fact, testpmd actually has even more hugepage > memory > than that mapped into it, but they must not be actually in use. > [Anatoly, > can you confirm that this would be the case when using vfio-pci i.e. = no > physical addresses to query?] >=20 > Regards, > /Bruce