From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1084746332 for ; Mon, 3 Mar 2025 23:05:57 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D672F40156; Mon, 3 Mar 2025 23:05:56 +0100 (CET) Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) by mails.dpdk.org (Postfix) with ESMTP id 7957240041 for ; Mon, 3 Mar 2025 23:05:55 +0100 (CET) Received: by mail-lf1-f42.google.com with SMTP id 2adb3069b0e04-5493b5bc6e8so5801054e87.2 for ; Mon, 03 Mar 2025 14:05:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741039555; x=1741644355; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=tXBNhRjfk2FymVjRd+Z/ulMcdFQt+EIrwNLUq7CGJC8=; b=VyxAPFuC/3Rs9HZYf4gPodHgW0XsInl3zjrIs1p74GjJpuROR32N68brQWGmh3kbpM F938po22P0XO18BmiQCXG2sz3I+HRsEfUjb9rMWghoe1+gVfCnh+XPunb2KJCfByk8aU 5csD4eDDE0Jnw0+YeTEfIF/L5/TVxe5Va6RrbM9OqUVz/cJ9SDhTf3RUlXkneqoZuNLa Q0euj7CtF41WTy0zJYiomKN5rVLSH89fBXmCr0YO09NE+HEDrirzZPiFdJ04bsP2+kdX ILMPP/6qJTN+dGXVs/Gemg6SMwCl9DMxdFYNePzcLBxF8HgCkxSfYdtAowJuVMPJ/zKd MJRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741039555; x=1741644355; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tXBNhRjfk2FymVjRd+Z/ulMcdFQt+EIrwNLUq7CGJC8=; b=RA39EI82uXwKG7Si86B+/eG6vSAQII56zH7lrn67gg/phyc4YOgEKSGPMoECDO0bR4 qNkUbOQe2NTSL2nsXjz0xOZe07WKkddrDRX5+JBGroJAt5QXU6zl+1teGh2IlnPVH0SI bhKotfWzG91jsdqhEhwBFVpkSUo6Fj4Eu+iVQOg1zHrjpWl8Gq2+DyQSnGieMq78dMJz lnvmG2p3GXIg9p5gMTuuIStvfsxguLPfSUbYLCbNbum/VT3Pr+k6PdSyDXlEI72oUSZn lS7O43uCRdnQ0Q4NMxbb38nWRxP+8lqYBoW/ebjZia1gtWDPFvCNdbKd8IjRmo/3kAEQ xqXQ== X-Gm-Message-State: AOJu0YxM6SFv4xvnmQ2giDRtuiuF9iVh/8nowEUdFu23Pomb490rnBQF 3JjRnhW1eBTtGjKJg28wWG4+End2Zr4NuItsX3abCbSx+58rvwM+ X-Gm-Gg: ASbGncu3qV8I9EmXScRXcpbyK5ZCnZXLMdmnhV7TzWfuzLWssG2F5lDuakxVVmvAbi6 3sZEtLFMvcj/qyZOBImx0e5gFYzg53rG/0DtOyA32f7bMCnXUvjgWN/88mCVzt1h9NCwscEyf2C GmhsyO5nvagAOA7V5NGnMdKICfZwqBkUh1r0Bs1GDRc/Pl02lTvJQssv92gf09C2vtSBxGAX6Y0 +JO2FIh9fChUAQOi17GL/cQP7WN2PkBFjdFbcFaJm1nLI+HgRpX5K7gn7OwLC0adHWqD1pfQNYP d1M/bQwrUst5xwV/M2kvOnky5FNByQ/boCVX7kNtqudhH7qqMlwCi9AosWyDdbk9sIPKDRPuBWk 9+vK1aumAEGqKfVHe X-Google-Smtp-Source: AGHT+IGYkHXD4Ssy0GSasqhTtNlCUo/omD23ZAYLEUd+7uVYj8eBarANLUJv/bqMwXw9s0UfFYGCCg== X-Received: by 2002:a05:6512:104c:b0:549:6f5a:27b1 with SMTP id 2adb3069b0e04-5496f5a2a7amr1407197e87.30.1741039554500; Mon, 03 Mar 2025 14:05:54 -0800 (PST) Received: from sovereign (broadband-109-173-43-194.ip.moscow.rt.ru. [109.173.43.194]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5495900d63bsm856647e87.135.2025.03.03.14.05.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 14:05:53 -0800 (PST) Date: Tue, 4 Mar 2025 01:05:51 +0300 From: Dmitry Kozlyuk To: Lucas Cc: users@dpdk.org Subject: Re: Increase DPDK's virtual memory to more than 512 GiB Message-ID: <20250304010551.6a650521@sovereign> In-Reply-To: References: <20250220012828.21ae8b76@sovereign> <20250220142143.7cbecabd@sovereign> <20250220175505.5116b4e0@sovereign> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org 2025-03-03 15:52 (UTC+0100), Lucas: > Hi Dmitry, > > Excuse my late reply. > My system is configured with 1G huge page size upon boot and then later on > I issue `sysctl -w vm.nr_hugepages=700` > It appears that the swapping was caused by some daemon that decided to > start up and allocate quite some memory. I no longer observe this effect. > I included a FlameGraph (recorded with perf) from program startup that does > a memory allocation for 500 GB (MBUFS + priv data): mempool allocation > with 218'531'468 MBUFs. > Most time is spent in mmap and in memsets. > > Let me know what you think and perhaps if there are ways to improve the > loading time. Thanks for the profile, Lucas. [1] suggests that allocation would take about (500 / 32) * 2.16 = 33.75 sec. You can run "malloc_perf_autotest" in "dpdk-test" to estimate for your HW. I don't know any simple and secure way to speed up the kernel part. Restart can be made faster as [1] suggests, but not the first start. Try investigating other parts of the profile that take place in user space. [1]: https://inbox.dpdk.org/dev/20211230143744.3550098-1-dkozlyuk@nvidia.com/