From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <users-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5AF3143CB0 for <public@inbox.dpdk.org>; Thu, 14 Mar 2024 03:51:17 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D568C40297; Thu, 14 Mar 2024 03:51:16 +0100 (CET) Received: from mx0a-00196b01.pphosted.com (mx0a-00196b01.pphosted.com [67.231.149.170]) by mails.dpdk.org (Postfix) with ESMTP id CAADC400D5 for <users@dpdk.org>; Thu, 14 Mar 2024 03:51:14 +0100 (CET) Received: from pps.filterd (m0072398.ppops.net [127.0.0.1]) by mx0a-00196b01.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 42DMo4Tb016719; Wed, 13 Mar 2024 22:51:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= from:to:cc:subject:date:message-id:references:in-reply-to :content-type:content-transfer-encoding:mime-version; s= netscout.com.09.24.2020; bh=wt4356ZTynNH+moFyboI1o3J1+86d5JznP+j Q0vc+ow=; b=zysLh7pdYoqnmjfdxpBhDjENjHPhawvibzneifeDuv08z3vW9p3P Vin0EFdjejPvWpoF5G75mJc+lZhlJX7wxjsOlDbwJJ052M4+pqsOPDPMdPCbU3p/ /V4ZJj8dZK5IjbSm1BtR4zACoxUsuewCpbYY7R87VOxhzw/ilBLO0+NXPj716tIT z7YL9oxH5LX5Vm2pGhyl2BwZAtCIWHdwM1cgrRGz0g3F56DjNAYqE6yjXkufI+yZ q7ri+yYHAajudWTGtu6uXhdwD6GDlM8s/shIViCehm/XqMNHC4SK4TP4zf/whh0l 4czOHtNRzCvYC/HvbYnS308n6KrJmwV6og== Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2169.outbound.protection.outlook.com [104.47.59.169]) by mx0a-00196b01.pphosted.com (PPS) with ESMTPS id 3wrmssr9pq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Mar 2024 22:51:13 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=b1M7Iiv11MwwJ9NRw884EOCjojkhoYaSq0N8jv+bd1YSea1BTr7/rhumjI7925egXBKulvs1fet7XIXFBH7vE8vGZpsPNEP4bO4FJkebPWt4rofYwi+RvB3K6jz0/tKWx7oN//oJFQXVDFGi/sbn3PWnzRs+e9VA4enrbs1hOUQYUyIka7Z3WoyloSQryBCKxV3g7AfmLRK28e/IGYEe804SDFV+8iuX+kZ2F07JSWm5OdO8mv0PcmmJjsvo7dFll2+LmCXeAyKAlECdXPBEtkAKTUEMoBJKYYN0aGmsn7ERZFnNwmUX9+KBHYepVC+/t6EA6CNL+KlnIa33DwufWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wt4356ZTynNH+moFyboI1o3J1+86d5JznP+jQ0vc+ow=; b=S5M9VkwYobEhc5zuJCOv/XPTWQbF5JRMqpFDdglIsbKXiPXiGcRUMa2hSf7udwPpLONRnPnBO2cPFmPEiYwv0Qtp+iYlF9wexsihW6mVrQhCP38UvQ/2iIGqeew6kWkg9JDM1KyHE/feWtrKnXtdUXCqBcQ2bcM6udwaIXRBEC7Pj0ABZSuU5eKPRf5c1jcXLt0YwDBbxENC2PCq0fjjwubVQ36xDmg8wolRlWWrnHiHlFRlNVW4/tibty79A4ZnbSRwX2RUc+cPCDMVmY9Uswrb4dUAGdxYZGQJd9w1IS1IjnckbwBMBjTmof5u09A/uh6gn03Pwqv4EXAeOS8s/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by SA1PR01MB6749.prod.exchangelabs.com (2603:10b6:806:1a1::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.21; Thu, 14 Mar 2024 02:51:10 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::44f2:5149:e583:9bf1]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::44f2:5149:e583:9bf1%4]) with mapi id 15.20.7386.017; Thu, 14 Mar 2024 02:51:09 +0000 From: "Lombardo, Ed" <Ed.Lombardo@netscout.com> To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> CC: users <users@dpdk.org> Subject: RE: DPDK 22.11.2 requires too much VIRT memory, how to reduce Thread-Topic: DPDK 22.11.2 requires too much VIRT memory, how to reduce Thread-Index: AdoP/ZNzzEBMQqwdS1qgWUB1z9DFpQAF4HsAAHkN8NAAHXTkAAAIxu1AAAF4RgAANJhDYAATjLCAAAA2iYAYgDkW8A== Date: Thu, 14 Mar 2024 02:51:09 +0000 Message-ID: <CH3PR01MB8470C9675763E14954D6E3B88F292@CH3PR01MB8470.prod.exchangelabs.com> References: <PH0PR01MB67301F1FBFD731EE7A48AD288FABA@PH0PR01MB6730.prod.exchangelabs.com> <CAEYuUWDsPoTZXxv9SHdLbwDio7e36P__+j=ZGoL0=VzxAJG2WQ@mail.gmail.com> <SA1PR01MB6734BB0F8B1343BDC0E6BE958FA8A@SA1PR01MB6734.prod.exchangelabs.com> <CAEYuUWCMHc4RyBKziEpDZryZuczQEzWW25AbOj71gnwkTcX2ZQ@mail.gmail.com> <PH0PR01MB67302322AE1F6236E6C1DE218FA8A@PH0PR01MB6730.prod.exchangelabs.com> <20231109020550.365e80bc@sovereign> <PH0PR01MB6730BD9ACA0534A2BBBB2E5B8FAEA@PH0PR01MB6730.prod.exchangelabs.com> <20231110123133.2adbd2cc@sovereign> <20231110123739.21fbcb01@sovereign> In-Reply-To: <20231110123739.21fbcb01@sovereign> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|SA1PR01MB6749:EE_ x-ms-office365-filtering-correlation-id: 0d4eab8d-55b5-4a40-d902-08dc43d19a64 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: N4X8nHG/p03+Y9i5ylzDOUDh451NseLYngCL8EjPzxAGLNtET3n9yHzsGa+cuBkiK7JCUxvDlypCtUZubz0Q72XVu3JGYd1EgpNSZp/bvE2f4/qASPKUmM2EQYYKBuPfHBImZisADq+JBaMaiQZCnm9v33/Nd8ydBS2Mm/17nZ3LEfdjExg8dtbVyLUcppNWLJI9a2EDPtdHqwL4VCZ04WoPVTw05mBBgUP16rPpI02OAs7VJEgChx6YIjzK9bzhhS/8RdY/iAKP5T5JWerCNpTCkhnqayOhJJA35nWIwWwrRZgOV96Yd0RkncXlO3mB/a1V90xoGanSnNtrnn9mXtq5HFonUWbHbN6Slko/CjCeAYlxGRdO+oMp1TYozfQUe6EhiAV8yiZQv3lUJHTfL76O9QcoSDfO7Pn7EwQ63hNrqrxLYagry0vGijMFvTjxZhp8SHrEhPPSkAWKsSVssRGwm79PwFTJJI7JWauAf65eaue6bQjxw7w4LJOtgem7AuS9/t0pzdhtNce7wAd5Wn41zxh0XTiWHHB+TaqhqmExMkO7lx+CYiEHNyXkg/gJb8BSl0dn8z2xvEakPLDWbr2OFNRQLTo+FkfU4CuQ2jEKE8kmVsQmQueeP+zTdACn71FW5PnbuZJtVJAZK3HOCMmKSpm2okuoZNTAGhs67fSzHeyvdvb3x95I5UBjRkdcT+n/lCu+gkeSRH9ehNykSA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230031)(376005)(1800799015)(38070700009); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?7acPe5P9GPdlGgiRufCWtXCU8/UroM0se3Sm2tWWAlwHauX1MzXxtklV51H/?= =?us-ascii?Q?WnuMwq0s/d0bfg0E/8AhV15ko3SN2CERsGQ8719QXnWwfWj7JBCIoGA7Yhre?= =?us-ascii?Q?YHVuIVeNtgaJC249Qcdxtxb/u8N5TEdE/co24z8ddQOo7AVmk6b4ath7fUb8?= =?us-ascii?Q?LOo4UiMmYZOdScwnkvLAh10lcOLiK/LvZHMFoZyHVmnkjfvOOYvD42JNhRGO?= =?us-ascii?Q?zQSX7nu+U9pcHt5RiuJRYn9yEtO2nn6pnsYJodMSsfdM4ee6cI1vyf0rfFWS?= =?us-ascii?Q?f7STq5qeUo6Hdoqguox2CJhdc1kBJ3ZJb0Z9EueBA5fU1Tr9ezQssYVa8KkW?= =?us-ascii?Q?nSPvtjedwFc5/sjFmoMyAqIP5uE4o9fNoR5yTudC3yBhww2WrIwiGuuwf8p3?= =?us-ascii?Q?QQTJbXYv0Qha4uOnRbgwnkOqXzOEWJPTyNpY/25uX08W75KGub194nE6/OB6?= =?us-ascii?Q?H5nSsrQaq/MPerACpC0bOgJ1aMNmurzMVuVqs7fun8xh/i+k4y5EpOP6NdlJ?= =?us-ascii?Q?O7MzjfqguAWmuQEe20fpkQqPXO71cqDKTXyoX/Z6qQ86aZk25uTUJKrn0JDK?= =?us-ascii?Q?6aVVHytcgBK71PEaSZGpYFMJoxPMZ04ruKPC9dPfcGu7pEiBjSOn6kYKp9ih?= =?us-ascii?Q?V56XLewDbVSaKgjs1/YaEffO4gI+6tYyUm2Xkf9mj8a59MnUzg37BL1ZZ8us?= =?us-ascii?Q?TFOtwTGiz/G2c9KeAEAP2ziQATs/uQxhmO7Bs6/ubBjOMOs2GEk2uzIVn5Sm?= =?us-ascii?Q?eii+X1G9GCGOAKl7q0SNKfzBNgZBu74j60mImEOlVns6kHcI/iwlA0NKTlJZ?= =?us-ascii?Q?79MsXXOfVFlKNkAHzJfvCPilppI2UjqBXx7jEaFiyREcBoICPRHV9MSFjOcS?= =?us-ascii?Q?veGkrcvYpEk5+0HkCrttQFTRDy7ZW8vseCDP/gMp9U3iQES/BiIzAzEk2hJu?= =?us-ascii?Q?w612l57g2CVKZ3UXlLsBQ6jt8aB3Y2EIWHCH/8KBLOQRTlD2HJ7LcrPVMm3I?= =?us-ascii?Q?QbVfOT4YbWDQso5OrPcFlwtTVnqDzffoUsB2bXeNaeJe1EWu/0S1gHVlgajz?= =?us-ascii?Q?gQFamSqRu3GnDg0gXKDXJCkqqOaDRLjLnfTOHn9ijXhVOvS5ZPYITImkpL6U?= =?us-ascii?Q?/Oqz82RBdpntudbFAV0GZ2uaMU+FWwkNc49l50/uBr2l2JBG1b6Bf25s6+oC?= =?us-ascii?Q?ZuEdZia16t/w41pX1ens3TjnJUxgWIIbiuQSpS+h8bu6J+7GYC7xdu+zPFb3?= =?us-ascii?Q?Sr90oUBbsP5roHqMqc7pHSHHmuqyWZ7njBvvScLIQOHffHEEq6zlHm0TZ7s0?= =?us-ascii?Q?Um+Punoxs/Jt8O2Ob749JOaguX4PFMqvJ0XCORLZuzy4csoIjtq/VDwC7vvf?= =?us-ascii?Q?MAvBfuF2iABI9RxHx9APaEKL3pKUl4sjzyYI/Ck0RcpqP3gNiFVx1US6GCh7?= =?us-ascii?Q?s/XPe+knDXfQ3GcbKdnCx7HI/mpSmj8DqYs+oVhKE1i3SvaVYCPIPsYEWqbC?= =?us-ascii?Q?2OZaNDsb04lQKjmMDkRd7nVLNwf+gqaQiQ1p66DGqTtYEB3khC5WLr5/rWLn?= =?us-ascii?Q?Kbnl87fPa2iJ+u1W0gdMdWWfO6Se7moxxwcGYtVp?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: lnh2CSuKEaYtKHxB26OfBt370cUnGmiKOXFGxF0CO2JZ+UaH2HPtiGyRNIrYZXFJJjwk7bJRSKVDJGN9+zKZj8zfRF4N2S2VhtPzrYSbyrG2fcKAG1UQKoadqEq28qVRiX26VBR2VwW3zDtXsuPMvgl/NY03Wz1CpxRA26d9YHuYsuM6VA5Tu4tnebo5HoHSXPeB6VAG8hyNJ+b7Vuu7zY8F+m8QgEpb9tO9b9vdVxOddMox2bOfeR0Ji2a2BqnrkY48KviQPlK3BzeKs5K1AfG8lC0QKlj+l6fDkUp3fetzV07BXOy1AGLRcigONR+r6nA8SNQZWmEQV4i0PjDkrpOxDCg4V20na6j2j8strShxTV0iflT/WuShbD2fWWqbGmuURC7C/Kemf7hThUcBpWqdBIJaN1ECcfAlCQanTx3tNGByq1Zd0de/ZUtAopdY2Brw4NrNlb6NaFJQI+kLRrmGysZHxxPr2qrCmBwUSBBaA/fclUpCTCJR7jMOWO/RN3JiSlgd0OcSTwIykicqV0xY8B5H3xBUkhugXUUTrAkRK1QUBNkldwemTF8spYnSr8rACNBvloe+wlHSI3PJTIwNJbDObbESAhpAjH98kKWQ0iCWQZx31aiQm/Qz7UjE X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0d4eab8d-55b5-4a40-d902-08dc43d19a64 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Mar 2024 02:51:09.6718 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: vJvWGm2KlWx0PC17Fj1ufpi1HYq9QGGyYL66piLIZ/6/0WwXHPnGUrKu4KvAIT9Xi+goiVZ255qR1oBAaoINJg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR01MB6749 X-Proofpoint-ORIG-GUID: iHHB7E7HKlocJDxENblbjJG-0SGc72C_ X-Proofpoint-GUID: iHHB7E7HKlocJDxENblbjJG-0SGc72C_ X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 impostorscore=0 phishscore=0 spamscore=0 priorityscore=1501 adultscore=0 suspectscore=0 mlxscore=0 bulkscore=0 clxscore=1011 mlxlogscore=999 lowpriorityscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2402120000 definitions=main-2403140017 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions <users.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/users>, <mailto:users-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/users/> List-Post: <mailto:users@dpdk.org> List-Help: <mailto:users-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/users>, <mailto:users-request@dpdk.org?subject=subscribe> Errors-To: users-bounces@dpdk.org Hi, I am revisiting this issue I have been living with. The workaround I am us= ing is to not memlock memory in our application. =20 Error is "VIRT memory is too high and mmap fails, Cannot allocate memory (1= 2)." I tried DPDK 23.11 and I see the same issue I see with DPDK 22.11. =20 In meson I tried setting -Db_lto=3Dtrue -Dbuildtype=3Dminsize but did not h= elp my issue. We have only 16 GB of memory and I setup 2x1GB=3D2GB hugepage size, (legacy= mode EAL setting). Also running Oracle91 kernel 5.14.0-284. I see same the issue on virtual VM on Intel host, and on hardware bare meta= l with Atom processor. (Both running Oracle91) Can I reduce VIRT memory if I switch from static libraries to shared librar= ies? Any help will be greatly appreciated. Thanks, Ed -----Original Message----- From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>=20 Sent: Friday, November 10, 2023 4:38 AM To: Lombardo, Ed <Ed.Lombardo@netscout.com> Cc: users <users@dpdk.org> Subject: Re: DPDK 22.11.2 requires too much VIRT memory, how to reduce External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. 2023-11-10 12:31 (UTC+0300), Dmitry Kozlyuk: > Hi Ed, >=20 > 2023-11-10 00:16 (UTC+0000), Lombardo, Ed: > > I finally finished testing all the options and found the VIRT value=20 > > can be reduced from 66 GB to 16 GB with --legacy-mem setting in EAL ini= t argument. >=20 > Right. > By default, DPDK can use up to 64 GB of hugepage memory, so it=20 > reserves 64 GB of VIRT (but does not map most of it), RES should be=20 > low until the app actually allocates something. > In legacy mode, DPDK maps all available hugepage memory at startup, in=20 > your case 16 GB, VIRT and RES should be close. >=20 > > So I therefore had to increase the VM memory from 16 GB to 24 GB=20 > > (instead of 80 GB without this setting). >=20 > I don't understand why you have to do that. > Possible VIRT is not limited by available RAM. > DPDK should be able to reserve 64 GB of VIRT on a machine with 16 GB=20 > of RAM, it will just be unable to map more than 16 GB (obviously). Sorry, I've sent the message early by mistake. > > I wonder what do we give up with this setting? Most importantly, in legacy mode DPDK will consume all available hugepages = at startup and will not free them back to the system until the all is termi= nated. The default dynamic mode allocates and frees physical RAM on demand. Some advanced DPDK memory API don't work in legacy mode. > >=20 > > All the other settings I tried and combinations of these had no=20 > > impact (socket-limit=3D2048, single-file-segments, no-shconf, and=20 > > no-telemetry) on VIRT memory. Right, they should not. DPDK assumes that VIRT reservation is almost free and unlimited. May it be that your system somehow limits it?