From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8E19A45C98; Mon, 11 Nov 2024 17:54:31 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 28033400D6; Mon, 11 Nov 2024 17:54:31 +0100 (CET) Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by mails.dpdk.org (Postfix) with ESMTP id A71AB40041 for ; Mon, 11 Nov 2024 17:54:29 +0100 (CET) Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-20ca1b6a80aso50513735ad.2 for ; Mon, 11 Nov 2024 08:54:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1731344069; x=1731948869; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=0Tod2iQV3uhtnd+RN+6+t5+WpWg4sLyrecvQuRxFLGk=; b=E4s1rXasdrx3KQdA798BhFr5/4je42yrSZQ1P4ujaa4gnV23UqFlx1W3cak1dAVwOz iIiHLVBZm60/tZASskfoJPfV6TEx0dygXsXG+JacBJW29I8LTL4OLoOM1LwcrnVc/DPz oWxeMJp5LyWlxJVztztkO53lmamg1pQd5tpwGHbOHYrrLpGkAqjtohmNDqrncUvsyJdI yg+DpvB4nbL+FEEokQgcP5awhrsXRdnycCYObSh3Wz42c23IFTL0/ORVJBgXtfMGbPSy w6uIkHdLNX6451zll3pQ93YIcwUvXvtD2wuGPPfkBguxFOd1zKuEQQh6NDFoxa/eQu9N vSlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731344069; x=1731948869; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0Tod2iQV3uhtnd+RN+6+t5+WpWg4sLyrecvQuRxFLGk=; b=tqpvtX37J63tXyEEbI5dzFLwvTibgrAX+pXT7ka5HoAK0xg3ZAmWcfz6w9Y+dtJZqL SS1V2GtAtWLJfbZvDLAqMAuy5KRIbYs6lP9CxuqdlxV3F/Ko0farU/xVfPWzm+UWPQC2 9zi1zukeET+amFsQT6HNfF0Pw7FmyQ9o6r/+25UZvzJ1k/mZUFIDPofZnZYdVavmylaL wYddNL4U4RHETQh5M7dD3Ux0BSFLwMqv2on+ZTi1ghjb260aFYsCzJ4U68k9V/Z2piaU R62QFbFAXKJ5MSGP/VdQtqtkhh5Xtrk89eV4P5YDtkAr+e2jQyUobTdWH0OnkadmnQpD XkzQ== X-Forwarded-Encrypted: i=1; AJvYcCXTgma2ceUYD0aFigdg7tKhtGtZAdVbHLMJzoO5AaYzxx64UVYJeKkTbTApc88qC3uiaIY=@dpdk.org X-Gm-Message-State: AOJu0Yy9I44lcaewbnDq0TQUAU3gXJ98kZXcDW2OmCqIQ4612QyJIwow pdrFjFKlPNJ+DMLKR8A1fhOKO1b8xo38isWRXj3Hb54mYYb3mgyTFWw4RmJeNR0= X-Google-Smtp-Source: AGHT+IG8H6XOAfF57oGK4Dv58H2+kDluaF0hXYEy8oL6FGHuiel1Rz8HXZanvlv6Tg3p9gTUbHJtng== X-Received: by 2002:a17:902:da92:b0:20b:61ec:7d3c with SMTP id d9443c01a7336-21183d7b929mr189498055ad.49.1731344068673; Mon, 11 Nov 2024 08:54:28 -0800 (PST) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21177e4278dsm77635505ad.159.2024.11.11.08.54.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2024 08:54:28 -0800 (PST) Date: Mon, 11 Nov 2024 08:54:26 -0800 From: Stephen Hemminger To: Mattias =?UTF-8?B?UsO2bm5ibG9t?= Cc: Morten =?UTF-8?B?QnLDuHJ1cA==?= , dev@dpdk.org, Mattias =?UTF-8?B?UsO2bm5ibG9t?= , David Marchand , thomas@monjalon.net, Bruce Richardson , Chengwen Feng , Konstantin Ananyev Subject: Re: [PATCH] config: limit lcore variable maximum size to 4k Message-ID: <20241111085426.484fc96e@hermes.local> In-Reply-To: References: <20241108181732.173263-1-david.marchand@redhat.com> <98CBD80474FA8B44BF855DF32C47DC35E9F8B3@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35E9F8B4@smartserver.smartshare.dk> <6d5abcd2-dab0-4e20-8141-d233c19cc350@lysator.liu.se> <98CBD80474FA8B44BF855DF32C47DC35E9F8B8@smartserver.smartshare.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Mon, 11 Nov 2024 08:22:46 +0100 Mattias R=C3=B6nnblom wrote: > On 2024-11-09 00:52, Morten Br=C3=B8rup wrote: > >> From: Mattias R=C3=B6nnblom [mailto:hofors@lysator.liu.se] > >> Sent: Friday, 8 November 2024 23.23 > >> > >> On 2024-11-08 20:53, Morten Br=C3=B8rup wrote: =20 > >>>> From: Morten Br=C3=B8rup [mailto:mb@smartsharesystems.com] > >>>> Sent: Friday, 8 November 2024 19.35 > >>>> =20 > >>>>> From: David Marchand [mailto:david.marchand@redhat.com] > >>>>> Sent: Friday, 8 November 2024 19.18 > >>>>> > >>>>> OVS locks all pages to avoid page faults while processing packets. = =20 > >>> > >>> It sounds smart, so I just took a look at how it does this. I'm not = =20 > >> sure, but it seems like it only locks pages that are actually mapped > >> (current and future). =20 > >>> =20 > >> > >> mlockall(MLOCK_CURRENT) will bring in the whole BSS, it seems. Plus all > >> the rest like unused parts of the execution stacks, the data section > >> and > >> unused code (text) in the binary and all libraries it has linked to. > >> > >> It makes a simple (e.g., a unit test) DPDK 24.07 program use ~33x more > >> residential memory. After lcore variables, the same MLOCK_CURRENT-ed > >> program is ~30% larger than before. So, a relatively modest increase. = =20 > >=20 > > Thank you for testing this, Mattias. > > What are the absolute numbers, i.e. in KB, to get an idea of the number= s I should be looking for? > > =20 >=20 > Hello world type program with static linking. Default DPDK config. x86_64. >=20 > DPDK version MAX_LCORE_VAR EAL params mlock RSS [MB] > 22.11 - --no-huge -m 1000 no 22 > 24.11 1048576 --no-huge -m 1000 no 22 > 24.11 1048576 --no-huge -m 1000 yes 1576 > 24.11 4096 --no-huge -m 1000 yes 1445 > 22.11 - - yes 333* > 24.11 1048576 - yes 542* > 24.11 4096 - yes 411* >=20 > * Excluding huge pages >=20 > If you are more selective what libraries you bring in, the footprint=20 > will be lower. How large a fraction is effectively unavoidable, I don't=20 > know. The relative increase will depends on how much memory the=20 > application uses, obviously. The hello world app doesn't have any=20 > app-level state. >=20 > > I wonder why the footprint grows at all... Intuitively the same variabl= es should consume approximately the same amount of RAM, regardless how they= are allocated. > > Speculating... =20 >=20 > lcore variables use malloc(), which in turn does not bring in memory=20 > pages unless they are needed. Much of the lcore buffer will be unused,=20 > and not resident. I covered this, including some example calculation of=20 > the space savings, in an earlier thread. It may be in the programmer's=20 > guide as well, I don't remember. I suspect that glibc malloc assumes a virtual memory backed model. It is lazy about reclaiming memory and grows in large chunks. This is one of the reasons malloc() is faster than rte_malloc() for allocation. https://sourceware.org/glibc/wiki/MallocInternals