From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AF7C845CB7; Fri, 8 Nov 2024 23:23:13 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4AE514067C; Fri, 8 Nov 2024 23:23:13 +0100 (CET) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by mails.dpdk.org (Postfix) with ESMTP id 16A9C40263 for ; Fri, 8 Nov 2024 23:23:12 +0100 (CET) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id C1EF21D9FE for ; Fri, 8 Nov 2024 23:23:11 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id B55FE1D96A; Fri, 8 Nov 2024 23:23:11 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on hermod.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.2 required=5.0 tests=ALL_TRUSTED,AWL, T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Score: -1.2 Received: from [192.168.1.85] (h-62-63-215-114.A163.priv.bahnhof.se [62.63.215.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 776221DA94; Fri, 8 Nov 2024 23:23:09 +0100 (CET) Message-ID: <6d5abcd2-dab0-4e20-8141-d233c19cc350@lysator.liu.se> Date: Fri, 8 Nov 2024 23:23:09 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] config: limit lcore variable maximum size to 4k To: =?UTF-8?Q?Morten_Br=C3=B8rup?= , David Marchand , dev@dpdk.org, =?UTF-8?Q?Mattias_R=C3=B6nnblom?= Cc: thomas@monjalon.net, Bruce Richardson , Stephen Hemminger , Chengwen Feng , Konstantin Ananyev References: <20241108181732.173263-1-david.marchand@redhat.com> <98CBD80474FA8B44BF855DF32C47DC35E9F8B3@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35E9F8B4@smartserver.smartshare.dk> Content-Language: en-US From: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9F8B4@smartserver.smartshare.dk> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV using ClamSMTP X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2024-11-08 20:53, Morten Brørup wrote: >> From: Morten Brørup [mailto:mb@smartsharesystems.com] >> Sent: Friday, 8 November 2024 19.35 >> >>> From: David Marchand [mailto:david.marchand@redhat.com] >>> Sent: Friday, 8 November 2024 19.18 >>> >>> OVS locks all pages to avoid page faults while processing packets. > > It sounds smart, so I just took a look at how it does this. I'm not sure, but it seems like it only locks pages that are actually mapped (current and future). > mlockall(MLOCK_CURRENT) will bring in the whole BSS, it seems. Plus all the rest like unused parts of the execution stacks, the data section and unused code (text) in the binary and all libraries it has linked to. It makes a simple (e.g., a unit test) DPDK 24.07 program use ~33x more residential memory. After lcore variables, the same MLOCK_CURRENT-ed program is ~30% larger than before. So, a relatively modest increase. The numbers are less drastic, obviously, for many real-world programs, which have large packet pools and other memory hogs. >>> 1M for each lcore translates to allocating 128M with default build >>> options on x86. >>> This resulted in OOM while running unit tests in parallel. > > Is the root cause the lcore variables library itself, or the unit test using a lot of memory for testing the lcore variables? > We don't want to fix the library if the problem is elsewhere. > >>> >>> At the moment, the more demanding DPDK user of lcore variable is >>> rte_service, with a 2112 bytes object. >>> >>> Limit the lcore variable maximum size to 4k which looks more >>> reasonable. >> >> 4 KB is not future proof. >> >> Here's an example where 16 KB is cutting it close: >> https://inbox.dpdk.org/dev/98CBD80474FA8B44BF855DF32C47DC35E9F7D0@smart >> server.smartshare.dk/ >> >> Depends on how we are going to use it. 4 KB suffices if we only want to >> use it for "small" structures. >> >> Would 64 KB work as a compromise? >