From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C22B043F39; Fri, 3 May 2024 22:54:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 81D7A402D4; Fri, 3 May 2024 22:54:46 +0200 (CEST) Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by mails.dpdk.org (Postfix) with ESMTP id 9DEDF402D1 for ; Fri, 3 May 2024 22:54:45 +0200 (CEST) Received: by mail-lj1-f178.google.com with SMTP id 38308e7fff4ca-2e1a02af4a5so1265491fa.2 for ; Fri, 03 May 2024 13:54:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714769685; x=1715374485; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=yzcam93e7XtYAwohREXQA/9kopb9IKU5X30UJLXNXVk=; b=mKw+1vMD1QEoBUx0EdQZZQpQxJLhRV3I5MaWJymsbWOO07Y5gxXQ84XE8Eu4C/GQpL /soaO+DBTDkuG30JTd+bpOweDUsqmWqtdt7xeQ91I52NmHvaCoJ+R/Z9tj6J45gKaYI3 yeT687i6epfdeu0JhSBujcW4apkY/q/yucfJgN06AxAESTp13m5rXjS1qhAKZH+47Sz3 RU1O633C6oc+/a5t5pV8O4jtSNFzaC9LfPrqls0cNP3pZXndAil7TIju5ln6f/gMPISg +Un+BbwKuK+VZNB+5Yt3/Ma8MA16URixbGr/JK3cLk1pBSqO7i1ODL/SOcyH2d/9E/oa s+pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714769685; x=1715374485; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yzcam93e7XtYAwohREXQA/9kopb9IKU5X30UJLXNXVk=; b=CsUId3KowM3AZvma3LR1JTa2xidgi8CLooTkW/RrPJXw3eImHqgnEQB982piJuExcY qE3z0RVz7R7KKIlWRni2J9pmgQzbpiZNl6pjJCGpocQpzf3IPLEVfBMzL8XL7OQvoErb lkW2vOz4G6aSZkiRpmQ58Gww8lgPScrLY7XEdlVZmVnfLQrqXx6t2lE/M68fGRN8awtL FDyjkiEtX56YzmYcqynIewFqJXgokSSaMZZyLhCF1J+UkWdrOuzdYpqlAktM6o31JwYe AEH/y6crGnWBZhv/qHtWYc2AtmsIGcdC1Tl9p/WE+rwL2MqfStT+i1DWH5EmPm9dERqw ZZIQ== X-Gm-Message-State: AOJu0YzknAI7y/7Z3BYeDpi9UfhN/9eOmDd/bGl4N+JrUKFeFEyx82a5 E5SpUuxhQMyUmlf6E6UzXntt4EvZa8AfpnuWunbD153rluROdGxD X-Google-Smtp-Source: AGHT+IEI+cjDvnw6dswHmLB+I7jRAqhd9otr/C02DcuskXQ/hUXDQbZa3Znd7H4DU1uhgA8ZK1bbww== X-Received: by 2002:a2e:9857:0:b0:2e1:a3dc:ed8 with SMTP id e23-20020a2e9857000000b002e1a3dc0ed8mr2390471ljj.4.1714769684530; Fri, 03 May 2024 13:54:44 -0700 (PDT) Received: from sovereign (broadband-109-173-110-33.ip.moscow.rt.ru. [109.173.110.33]) by smtp.gmail.com with ESMTPSA id d20-20020a2e96d4000000b002df6de7283asm648051ljj.126.2024.05.03.13.54.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 May 2024 13:54:43 -0700 (PDT) Date: Fri, 3 May 2024 23:54:42 +0300 From: Dmitry Kozlyuk To: "Lombardo, Ed" Cc: "dev@dpdk.org" Subject: Re: Need help with reducing VIRT memory Message-ID: <20240503235442.70b4b056@sovereign> In-Reply-To: References: <20240502230352.7bb1ef0d@sovereign> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 2024-05-03 14:48 (UTC+0000), Lombardo, Ed: > Hi Dmitry, > I am not clear on the DPDK memory layout and how to tweak these #define values. Given your statement: > Currently we configure 2x1G hugepages and single socket. I suggest the following: #define RTE_MAX_MEMSEG_LISTS 2 // see item 5 below #define RTE_MAX_MEM_MB_PER_LIST 2024 // see item 4 below #define RTE_MAX_MEMSEG_PER_LIST 1024 // ditto #define RTE_MAX_MEM_MB_PER_TYPE 2048 // see item 2 below #define RTE_MAX_MEMSEG_PER_TYPE 1024 // see item 3 below Explanation: 1. There is one memory type: socket 0 + 1G hugepages. 2. There are 2 GB = 2048 MB memory of this type. 3. On x86, there may be 1G or 2M hugepages. If the latter size is ever used, 2 GB need 1024 of 2M hugepages. The cost of allowing 1024 memsegs instead of 2 is negligible. 4. Each hugepage is represented by a "memory segment" in DPDK. Memory segment list (MSL) is a virtually continuous span of memory segments with equal size. DPDK cannot allocate a block larger than an entire MSL. In order to be able to allocate a block of any size (subject to the limitations of the only memory type = 2GB), MSL must be as large as the amount of memory of this type. Hence, MSL maximums = memory type maximums. 5. There may be two memory types (socket 0 + 2M, socket 0 + 1G), so you need at least two MSLs available. In my tests, VIRT = 3142 MB for testpmd with these settings. > I want to limit how much DPDK grabs for memory, but grabs what it > absolutely needs for our application. I don't want DPDK to plan for any > hot-plug or dynamic adjustments of hugepages. Our configuration is static, > we allocate hugepages, discover ports and initialize couple ring buffers > and 1 Rx Q and 4 Tx Qs per port. Max of 4 ports. This is all irrelevant to EAL reservations of address space. > My goal is to reduce our application VIRT memory by 80%. Not clear which config variables to adjust. I hope that the above addresses your need. However, I suggest considering what me and Bruce have been explaining: VIRT usage is not RAM usage and high VIRT is not an issue by itself. It is not recent DPDK that is broken or unsuitable for your hardware. It is something in your application or setup that is incompatible with large virtual address space reservations. If you could provide more details about what is failing for you (which system call, which what parameters, etc.), it would be much appreciated, because the issue is rare and maybe DPDK could account for this case and work out of the box in future.