From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5B6F043D99; Fri, 3 May 2024 16:48:19 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1370B402D1; Fri, 3 May 2024 16:48:18 +0200 (CEST) Received: from mx0a-00196b01.pphosted.com (mx0b-00196b01.pphosted.com [67.231.157.166]) by mails.dpdk.org (Postfix) with ESMTP id 64B80402D0 for ; Fri, 3 May 2024 16:48:16 +0200 (CEST) Received: from pps.filterd (m0072399.ppops.net [127.0.0.1]) by mx0b-00196b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 4436uosK032086; Fri, 3 May 2024 10:48:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netscout.com; h= from:to:cc:subject:date:message-id:references:in-reply-to :content-type:content-transfer-encoding:mime-version; s= netscout.com.09.24.2020; bh=KoGCU0BfaKX0BWVj6aPwZhyup41X6YYmcUcA t+HYEKc=; b=eR/gUDhyve5xiYu3Xx8ax3oFeT1DfOYUvjEhD3ukRsmt2Fv6vQwC 1T2M/dh5cCykePDvBtfhG4yCK7j/FtWQVdBHokX2bXAdyqThSzpzC3Pf6RB014nI MSsLoVY3izpQrDm9e5HMtMRrKUwLWN1AbW42EV0a8O/AVKtFdUtUsxdWa1KhR3Lp ez+WkpGMeU4PCibqGZg8sFWEdAs7Wg5DWIlMhGLO9qXdkrx3LTjXz7w6CRkGTFRg UQLDT+CNWEvChmBGh2br+pTJe4LmSa2BVcG0cJYgjENseY2xqOJOUwaTDXDEqkiW jJFb7VXld1pXeDB5slNHGRbKFqabJySTNw== Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2168.outbound.protection.outlook.com [104.47.55.168]) by mx0b-00196b01.pphosted.com (PPS) with ESMTPS id 3xrtkvg50h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 03 May 2024 10:48:15 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Np7qRtqBLOh/Qby3vV8GKCSV2UxS2HClyrtolFGCYEnLd3Nsysdv0lmtYgJgCA4iy8OmXRZNoiT2lf0mepxXBcFA02qIku54xEMBq2FCpneEXpq8/uZQko6b9MkjoWMqMcLqlz8dNpF61EUiGY6OEabQGL9zn5ne+LVlZoT9RInFmjHw+XYvReR7Xvy6aqGyErZEsYueBwYAsSSppgknZWIdH+bZOYxLYS/s/6qaoq4ttf5mjWEb0QixNbuQ4V6tz8vXKCd21kvuRhWN4MPMLGmWPe91rirbs07UzViqtTxBDh3/JKtOik1+XjDvY9xvJ9Z6PJG0BNsiluPbo+hO3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KoGCU0BfaKX0BWVj6aPwZhyup41X6YYmcUcAt+HYEKc=; b=DzYRym5xeKbyv66nPB3/4VphoqAP8sQwnaZRwklHffZ0aikqGUVQT+ScKhE6zKly8lW+khWkLaPhaJAfh3f/nSY8yQOdQJyHo+Q7UidNEe5K+6MSJ32XnSVt9k2b9gaGadbd8NFuTBtoCuhnUDh6UCerm9LQ0+ORI3r67BKyvbGfqrCiGDWZUk/MY5vPMvX8tVbPP4vWXyL5cSJxtc53tnuJZI8xfGgS3e+JE6rptW6AqURWfLmcpj45Edm3MehJiDZEsJOkbKOi/1svsJNiyCH/N5ftdphucMPjRmB3UiY8QwrvROmgBN2747KJ3dcNIZwliMCWBpkAabCqspHM3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netscout.com; dmarc=pass action=none header.from=netscout.com; dkim=pass header.d=netscout.com; arc=none Received: from CH3PR01MB8470.prod.exchangelabs.com (2603:10b6:610:1a4::21) by SA0PR01MB6265.prod.exchangelabs.com (2603:10b6:806:ea::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7544.30; Fri, 3 May 2024 14:48:12 +0000 Received: from CH3PR01MB8470.prod.exchangelabs.com ([fe80::44f2:5149:e583:9bf1]) by CH3PR01MB8470.prod.exchangelabs.com ([fe80::44f2:5149:e583:9bf1%5]) with mapi id 15.20.7544.023; Fri, 3 May 2024 14:48:12 +0000 From: "Lombardo, Ed" To: Dmitry Kozlyuk CC: "dev@dpdk.org" Subject: RE: Need help with reducing VIRT memory Thread-Topic: Need help with reducing VIRT memory Thread-Index: Adqcu3BiXzYpqqoETgaEa9S0ksiqXwAEGosAAAF8anAAJajQIA== Date: Fri, 3 May 2024 14:48:12 +0000 Message-ID: References: <20240502230352.7bb1ef0d@sovereign> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR01MB8470:EE_|SA0PR01MB6265:EE_ x-ms-office365-filtering-correlation-id: af3699f5-afa3-442e-c28d-08dc6b800e71 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230031|1800799015|366007|376005|38070700009; x-microsoft-antispam-message-info: =?us-ascii?Q?QuWCdkeqmjuHiSaUIfyuNu0bHlSVwnRTtyORWueyuQFxWOvY+EorM6P47yuk?= =?us-ascii?Q?G8i0us8q0+XdIUlb4Wl1d4q8S4hwfQ9AxNVwsuSunk1z8MNMXLIOG0NpmHp/?= =?us-ascii?Q?AGJzGv73H7SK8OpbhWAmfkhAUdkelZqvHzmhvRFs/qUl45gDuXtihxsRsopj?= =?us-ascii?Q?1dGoe4QeVwFLRoUte5r2DiJCI6DHPYg7CgY8KE0KIK/ByFK2sI+CUFGWqLge?= =?us-ascii?Q?OyBMEZEkA5RUlu2tsHwf59NUSR3k/gfUgHtTxFPLB94O+POCaPAHY1mM5SFZ?= =?us-ascii?Q?tgkgksnF8ETdJemhAzvEp6sFOLef0fuXnvisFjNdjZyx5whN+Ega4zyxR6F2?= =?us-ascii?Q?HJBcVGN94iuSklW6CD1eBjAbNgfIEoZ/pO2xQ2Qx+eWaZzkfMBp9POGhMeto?= =?us-ascii?Q?SzEmmCUnz/BduXnQyBXom8EdwCIAygg4YB5iR9Gb0Qg0XynsIoIl01yEtAW+?= =?us-ascii?Q?Ro1YFo/2a4Wt3QZuGPyZduQTlb81QQscKbFw/CGq2XZG+wiuIUxyYZ02MUQ4?= =?us-ascii?Q?40dwTcd6kchTDXgzYhBch09bF+vhZIF/2owIc87SXZYNV5gvHPFo8i70FE8r?= =?us-ascii?Q?DYRVhaFf2BEcB6nkNskg9JvPTDXZ9UdCPsuzooA5mpOgsfBA18FTkqv0pv0L?= =?us-ascii?Q?ZAhtJac/IjcOWaplTCGue5gZXdOJYJSc5i5rwICq2TW9qxBKg5btl9x/L7C2?= =?us-ascii?Q?C4GIy1BmwvsiWrCwJaglQT814wkTBO1sx8hWzag7+Qv08/68owu5j1luGHaM?= =?us-ascii?Q?/RXSPlWVqLrgPnf6yVF4zGxYPhPqthZmAFowkxjGtsNHglKwQbubh9eGL5t4?= =?us-ascii?Q?TI8JdhrR0mTUPwxwQE1WCQBBR5j1KGKbpPl8PxO7AG+OnZOF+7B/Yc7RHOlv?= =?us-ascii?Q?oz2FPbLo7z8xp2lNN4XJ6k5xpS0g3XUBeL5TWGAgKFll4v1qD1PR4XYhiEUT?= =?us-ascii?Q?B0Mq5cTsoM6+NACz4hiUCA//NHUJAiFomPBJOvmtTYczpHG3iQVmHdqatUN+?= =?us-ascii?Q?l6Pc+F/55y6hylmQGa+UzVNIMcZdGYx459vQ6W1k4lrUPHq4Eyqcsb7KWMUh?= =?us-ascii?Q?RVN+jOhXFbndtzMJ/DP7HZ3OFsNKjwjXoIH/Hjcr6yj0WaFFe0uwhGL/lbMr?= =?us-ascii?Q?Nk+rETgndsUs/vdDMHWUY0TqxP2cPaAvSWDNuB+b+rsiKQXAo2+PP7DbiaAj?= =?us-ascii?Q?QgXUc0DEU/+vbKo5/GXxSdMZO55E8ERDok0TJgaO+W27V7G1oRMd57V/d1dh?= =?us-ascii?Q?Dq3l1CNS4dOUiCftqgN/l8uqvo6D4exZP4O6ZEtdnQ=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR01MB8470.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230031)(1800799015)(366007)(376005)(38070700009); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?mFc/G3gmUAQZPCB1TvnzZgXUCPneMFR4utgY/zkdmMW+3TqaOOz8p45FL8kD?= =?us-ascii?Q?lDnOXPGZYoMy2cHmUudstLQtJHCGGo5WF5XCDPWXGzY5acCqAYutpmOX9fLB?= =?us-ascii?Q?7huVuIVbOTiW0dzNiFgEyfhYTsmQ5U2FkBjb382xDbLvis5wRLmhZDH5cqFH?= =?us-ascii?Q?U6QrV6krIeowjdZ6BZ/8F+oU/HoLAEY3DBLlTRRlcE7WmaZCbby+U3XZWD8a?= =?us-ascii?Q?Ho17zpcoGR6miO/Pv0jeYhHAo7IuE88qi/7azUVcokKLz4+zfpWHbXtQo21s?= =?us-ascii?Q?pnw0dpDgpCuL+eZM9V0zKTMfUvG7ZqkM4CdBsbkZyQHmuQGYmCVLnuPI8akN?= =?us-ascii?Q?aZYgAmF028IVtwtNd8BQZGuxkwknF1XUZ0ShnHDmzTUMbmSjDOwWMTNAmXr6?= =?us-ascii?Q?7piVOciHdmTNYrBBRbo+gj6VjSljRZbwJHTxuCJKp4Qeinw9gxSxoU1Dy9Cx?= =?us-ascii?Q?ffm/eyjLEZoalDLpXkMrgTnQVHu8PtAu62jyAR4SMn9uV5scrGLKR8klUEsB?= =?us-ascii?Q?U1hv+F4JUtm702UqHQzhBDILhWD8kyhxL9NiTyNLgCYKvxhGsGCvCfllR4lE?= =?us-ascii?Q?w90w4UwcUB3mwxnjp+1zhBEMB32NXHZb62wDVLnsigNhavE18Y0YDdvqvsrR?= =?us-ascii?Q?KxEcwg0FrtaDi1Ig365IrtE4FXj7ZpK3lTSJOod5aujRds9gpdOToSOHeXvY?= =?us-ascii?Q?Uuu7psSAop09951GSuygsl3qdiEAM5MKa/3SuZGQlHeXWr1sJAjsuZQ+FvQU?= =?us-ascii?Q?3FgVTx51jD5S++luuEhKUTwxCS3Jz9wcKIGb/LzM/aB98h7DwVsqycJlouEs?= =?us-ascii?Q?9cYxIxP1IE/zxnxpdHw2GixAbDGpsb2dOHRq1QO7tXNlbUjKmEILnn/c73w4?= =?us-ascii?Q?8n6UKRI4MZg0uF9hfquQxuv8/fe+2qy9ik1K19TiHMSljzg7LwV0fdmLcmrS?= =?us-ascii?Q?q0/0enMmpUgZuDlhbjBj868wIgTl1X8IVF1EZTvZguxf/ZotLR386yKDz6y0?= =?us-ascii?Q?GNfJ7hMu61a6o0S2BFG8ExKJjubd+bkK/CMXwMX8qEjgdJeX9QrVXK5WyHes?= =?us-ascii?Q?f7pyYtM84F0jxenQOKeqGXwovlOuhNQE+KB21gjMXo8IxEc0YLTHcXiK8FTm?= =?us-ascii?Q?IZeWayWk7VZbCJVMSx4+YgIDQWYaQzm3ddDx19jck2PBJXXfoZjSapNyIvvC?= =?us-ascii?Q?LqK3MO2j1nJJpiJBLnem0Svv0DDWbM3VIbxgGIoOZiOGvUOudWR+0HHIf4AE?= =?us-ascii?Q?j/TVOjkGnnWziXW9a+/09RjI4lQXrN64zDZfjosMDeA/Kibd1BFp3KEktwFd?= =?us-ascii?Q?zUebyVE/bouVpUhPN6BmOemzmVA5g7hIPaYUqQVae1hi1raMplLr4xO9fVWI?= =?us-ascii?Q?NkCBjPSaSEzq9aspQn9qYuLfEXZEZZC6LXllzMghkxdIQfbyPU0OPvYBlHcE?= =?us-ascii?Q?mEsTI8P11kNlgu0a/zBOOOfWf1ogsVa8lp8WXiDTn4b7oECEaefcv80O4QA0?= =?us-ascii?Q?11t4Elk830IxrFYcGJqmKismhK9B6J/BLRyNYMXM0LAg/9SS2G3WtDcWOKpF?= =?us-ascii?Q?2kINqB3NE1fxIjEI2D0Y2r37+khIPsHe4N6IWqpw?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: jcA9sR4bhH3/GqLE+S7bkgfV6GOvWGETywcOM7vfmFo0wpNLx6XCjB/SC7Lgg9zfiQ1f6Xz+HVKRWFDmuMyZ6smf8juQ/mRRWruWXBjSafeMgI+jf31AmFuR0I5asKULW17oTXvqjz6++m0O+CCZYnrLUtk0AfbVpRB08spSLjDonm15JrmqFQ7Bs1wE5VFkI7eFrBx8bXcYDZLt+96iCFaelboFyGJKnRt30lDxx0xtF0j0cyGXtcN6XaJbNaILWxA0N49Eu8WNnM3U3/3+FcTcvoUURZTOGrLscMKoJh/gZ3XxuHrxVDLU6/6cJt1w1Po/GylieGzKs1aXI9VgeIElJRJfx8c7fXmp7a0YdP4rxIqxg3ZtjrHz3bWMC4JMbtMZKggxgEZ8W1/+iSATlCSZTbW6MDixreZzLm/gxC6cG4NJosK249r6MNM/4wNckwKew24STJGhxBDsClKkkNOSQgtGr8JjSPyttHHT6KYvM56wHjmzyNjGHj28qWvN5kdxgAcxr9Ms5FY2E9rMNlG06qr5iOeDa3Y88WmllzvvM+R5IRDJuGXxzI3ah+Uw1jkKEuqjkqCLDV/dJwAHVsvHFlv/jO938N+flRzse4tyczyFc3CHNMuqXShlGKAd X-OriginatorOrg: netscout.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR01MB8470.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: af3699f5-afa3-442e-c28d-08dc6b800e71 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 May 2024 14:48:12.1913 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 54f11205-d4aa-4809-bd36-0b542199c5b2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: REvdEQAHBN8x/Z4ktVBY/5ua75tH0lkgBY3izteTJZVlzY2X0hvOlFNdLki4nzb93XseTgCHo7oTmy9lcZp7Hg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR01MB6265 X-Proofpoint-GUID: YrzhPnC7XpAv98oNU0no9MOFAlJNt1-a X-Proofpoint-ORIG-GUID: YrzhPnC7XpAv98oNU0no9MOFAlJNt1-a X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 mlxscore=0 lowpriorityscore=0 adultscore=0 clxscore=1015 phishscore=0 impostorscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2404010003 definitions=main-2405030104 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Dmitry, I am not clear on the DPDK memory layout and how to tweak these #define val= ues. =20 #define RTE_MAX_MEMSEG_PER_LIST 8192 #define RTE_MAX_MEM_MB_PER_LIST 32768 #define RTE_MAX_MEMSEG_PER_TYPE 32768 #define RTE_MAX_MEM_MB_PER_TYPE 65536 I want to limit how much DPDK grabs for memory, but grabs what it absolutel= y needs for our application. I don't want DPDK to plan for any hot-plug or= dynamic adjustments of hugepages. Our configuration is static, we allocat= e hugepages, discover ports and initialize couple ring buffers and 1 Rx Q a= nd 4 Tx Qs per port. Max of 4 ports. My goal is to reduce our application VIRT memory by 80%. Not clear which c= onfig variables to adjust. What do you suggest, is there a formula? Regards, Ed -----Original Message----- From: Lombardo, Ed=20 Sent: Thursday, May 2, 2024 4:59 PM To: Dmitry Kozlyuk Cc: dev@dpdk.org Subject: RE: Need help with reducing VIRT memory Hi Dmitry, Yes. I will look into=20 The VM gives the following: [root@vSTREAMO9-234 bin]# prlimit -v RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes I saw these optional configurations in my google searching, but was hesitan= t because of unknown impact. I will try this path next. It would be valuable to have meson be able to configure DPDK for low memory= devices. Thank you for replying. Regards, Ed -----Original Message----- From: Dmitry Kozlyuk Sent: Thursday, May 2, 2024 4:04 PM To: Lombardo, Ed Cc: dev@dpdk.org Subject: Re: Need help with reducing VIRT memory External Email: This message originated outside of NETSCOUT. Do not click l= inks or open attachments unless you recognize the sender and know the conte= nt is safe. Hi Ed, I presume it's a revival of this thread: https://urldefense.com/v3/__http://inbox.dpdk.org/users/CH3PR01MB8470C9675= 763E14954D6E3B88F292@CH3PR01MB8470.prod.exchangelabs.com/__;!!Nzg7nt7_!EBEN= SvDUv9LcvbAw6vLkHgh11JdjksApffakP2JU30OJaPLCkZPZVzk4U8rRTJsxJVCeRubpFQy7TB2= -R6jDbrk4QpU$=20 2024-05-02 19:05 (UTC+0000), Lombardo, Ed: [...] > My situation is as follows: > We were on DPDK 17.11.6 and upgraded to DPDK22.11.2 to support the Intel = E810. Also upgraded from CentOS7 to Oracle 91. > In DPDK 22.11.2 the VIRT memory has skyrocketed and causes major issues w= ith our application that runs on 16 GB of memory (virtual appliances and lo= w end remote appliances). >=20 > I have also tried DPDK 23.11 and 24.03 and I also see across the DPDK=20 > versions that the VIRT memory has increase by a factor of 5 over DPDK > 17.11 >=20 > Our application process shows VIRT memory is 7.6 GB with DPDK 17.11.6,=20 > and with DPDK 22.11.2, 23.11.2 and 24.03 the VIRT memory is ~36.6GB >=20 > Our application minimum configuration needs to run with 16 GB memory when= DPDK is enabled. As explained in the linked thread, high reserved VIRT does not mean high memory demand. Running DPDK on a 16 GB machine is perfectly viable. Something in your app or setup conflicts with the high VIRT reservation. > Our application uses memory locking and rlimit settings to get optimal pe= rformance. Does it use address space limit (prlimit -v)? If so, this limit may be hit. With default build options, DPDK reserves at least 32 GB of VIRT (one list)= per memory type (a combination of NUMA node + hugepage size). This is controlled by defines in : #define RTE_MAX_MEMSEG_LISTS 128 #define RTE_MAX_MEMSEG_PER_LIST 8192 #define RTE_MAX_MEM_MB_PER_LIST 32768 #define RTE_MAX_MEMSEG_PER_TYPE 32768 #define RTE_MAX_MEM_MB_PER_TYPE 65536 Try reducing those and rebuilding DPDK if the above is your case. Reducing these values will limit the amount of memory available to DPDK all= ocator in your app.