From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 99DBB46276 for ; Thu, 20 Feb 2025 12:21:50 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 677AA4014F; Thu, 20 Feb 2025 12:21:50 +0100 (CET) Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) by mails.dpdk.org (Postfix) with ESMTP id 77DB440041 for ; Thu, 20 Feb 2025 12:21:49 +0100 (CET) Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-54298ec925bso1370216e87.3 for ; Thu, 20 Feb 2025 03:21:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740050509; x=1740655309; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=xehY5mRh6g3KQVmzqpQeQaDD+lsH+fDb9WJK1UA1rxs=; b=Tz0zy59vqgTrqEh/vc+Nq3JijejJPtoBgWdsBLJcWNr462SGbj4ClOD1S5W7I0gqMQ kmedpXFHG8IMnf8tRBYNGrGXKUnUC5DXgkGYQNRBkEz3UAY/QleetZ+3eiBnZ53U5Enj d0il264G+lrldzsgHS/HtD7s1DvoELq6l/WxzUzk2pq2Hz5BJDu0i0ptujY7KiSW6xcc QYyYj3v9kXfkE9yQTYfiNSg/K8FQR1qHCA/035mWGI+mp9DtB81WWo71dYpMtg+PgtDA GiyQDCqniyY7tE7+lEz3G07Gny7q54+hqPgsOlpnTo/b4F2j956x1tfaztL9fuUUuEQU krcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740050509; x=1740655309; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xehY5mRh6g3KQVmzqpQeQaDD+lsH+fDb9WJK1UA1rxs=; b=I0e+z5fZa6ddlmyaboGtRxmxKG89XYQLcJPBPohmrc/9vAW0hKrpFRHNhkfkTf4pXp prfRP/6O1bAKMr7gTpdMb9wlV76Pr7PlFK/cGPbw0gyBZGZIE2DpD4+97mIIoZX+xims s9ZCkxCjVXa/2ChRBFJ6RYj43x9eGLUkAqBULiDSlcMQQ3fGCY6eZhTV0T7kjUGmyK9B kii2HHrLFM3uRqK+5vW/B6Fx+aDUBbZ+HJQNt//EyMdgMmtDKwnUbSudv5VaKGYr6wFK itw10dKH6amLZ9HuRXv1Y/TCwJkrO+kcUP7K3MX4pjRS4jrHq7OkspAb1j1PXlhUKKYb EXiQ== X-Gm-Message-State: AOJu0YyroV4gjfzKPXlMKRMkkgqUuZcKrTpzU/G45F/IqHS7Jg4B4fbv dKDjr30SM3HtuSqD/WUb0UKLdUe9CotXb/9SD9JkbI+Ss0BrsvPE X-Gm-Gg: ASbGncvfFujQeMk3EdcRHaJxmwY5cB2F96W/kg4/7HG45Tf/L3fzIoccoUrOh51oqlo WhPqugwk65D/9ZC6SKPU5j/Qt6oBnV38nD33X7e+K2yiepg0NgxA0z6Y3G6cUtAkGamnunmSJuQ 6H2m9VW4W8QDHsvkmbfGRkiP9mKG1EWw+NXiFMbr8oQXVzArv0rDV2FH4URJJYgnzPI99nWoskq ZHjNZjbw9IFs+IQux8TUXzJ9IueYnkUE6uJaHHGc6Y51MmgZf/i2x6fNsjay1hnMrEZ/uPHcfWx XqzN/sEV4vE20xTOWRbEYF+KWmsgdeMiOSO087iD47Y8j4bB5KZe40Afal/jFpI= X-Google-Smtp-Source: AGHT+IGK/BPt3sQE5knR8/MgwTtcddWAEHXm17SphNW55Oxey7vF+0SSFeiY9KeDyAW+NO5oWwrk8w== X-Received: by 2002:a05:6512:31cd:b0:545:f0a:bf50 with SMTP id 2adb3069b0e04-5462ef19790mr2617336e87.35.1740050508335; Thu, 20 Feb 2025 03:21:48 -0800 (PST) Received: from sovereign (broadband-109-173-43-194.ip.moscow.rt.ru. [109.173.43.194]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-54531d76843sm1669010e87.84.2025.02.20.03.21.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2025 03:21:46 -0800 (PST) Date: Thu, 20 Feb 2025 14:21:43 +0300 From: Dmitry Kozlyuk To: Lucas Cc: users@dpdk.org Subject: Re: Increase DPDK's virtual memory to more than 512 GiB Message-ID: <20250220142143.7cbecabd@sovereign> In-Reply-To: References: <20250220012828.21ae8b76@sovereign> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Lucas, Please don't send private replies to discussions in the mailing list. 2025-02-20 12:00 (UTC+0100), Lucas: > Hi Dmitry, >=20 > Thank you for your detailed instructions. > I have followed them in the following way: >=20 > - In config/rte_config.h set RTE_MAX_MEM_MB_PER_LIST to 524288 > - In config/rte_config.h set RTE_MAX_MEM_MB_PER_TYPE to 524288 > - To change RTE_MAX_MEM_MB, I have to change the configure set in the > meson build system. To do so, I changed > https://elixir.bootlin.com/dpdk/v24.11.1/source/config/meson.build#L362 > I replaced "dpdk_conf.set('RTE_MAX_MEM_MB', 524288)" with > "dpdk_conf.set('RTE_MAX_MEM_MB', 8388608)". As I have 8 NUMA nodes, wi= th 2 > huge pages sizes: 8 NUMA nodes =C3=97 2 hugepage sizes =C3=97 512 GiB = =3D 8388 GiB >=20 > With these changes, my program can create a mempool with 275'735'294 > MBUFS, comprising 2'176 (bytes of MBUF size) =C3=97 275'735'294 =3D > 599'999'999'744 bytes ~ 600 GB, but fails later, as I also need an extra > rte_ring to hold pointers to the MBUFs. In HTOP, a virtual memory size of > 4'097G is reported. > However, with a smaller amount, 229'779'411 MBUFs, it works (i.e. 500 GB). > And I can also allocate a ring of the same size (I use RING_F_EXACT_SZ, so > in reality it is more). >=20 > I have tried increasing the limits further to be able to allocate more: >=20 > - In config/rte_config.h set RTE_MAX_MEM_MB_PER_LIST to 1048576 > - In config/rte_config.h set RTE_MAX_MEM_MB_PER_TYPE to 1048576 > - Set RTE_MAX_MEM_MB =3D 16'777'216 >=20 > The virtual memory allocated is now 8192G, 8192G, and I can allocate 600 = GB > for the mempool (275'735'294 MBUFs), but the ring allocation fails (fails > with 'Cannot allocate memory'). Allocation of the mempool now takes seven > minutes. It would be interesting to profile this one your issue is resolved. I expect that hugepage allocation takes only about 10 seconds of this, while the rest is mempool initialization. > How would I be able to also allocate a ring of the same size? > I verified that the number of MBUFs I need rounded up to the next power of > 2 is still smaller than the max size of an unsigned int on my platform > (x86_64, so 32 bit unsigned int). > I have 670 hugepages of 1 GB available. Is this too little? In principle > the ring takes 64 bits =C3=97 # entries =3D memory. In this case, that wo= uld be: > next power of 2 is 536'870'912 =C3=97 8 bytes (64 bites) =3D 4.3 GB. > With 670 GB available, and roughly 600 GB for the mempool, this should fi= t. > Could it be that supporting structures take the rest of the memory? Mempool adds headers and padding to objects within, so it probably takes more memory than calculated. You can use rte_mempool_mem_iter() and rte_mempool_calc_obj_size() to check. You can check exact memory usage with rte_malloc_get_socket_stats().