From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9D54343F31; Fri, 3 May 2024 17:52:17 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 19DDB402DD; Fri, 3 May 2024 17:52:17 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) by mails.dpdk.org (Postfix) with ESMTP id D78D2402D0 for ; Fri, 3 May 2024 17:52:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1714751535; x=1746287535; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=6NKeRUPNXnEGXEpJ85Z+sGxoxzdSz6Xyaeo5UqbkkcI=; b=f6LRwruDI0lR+6UUOCZMNdlEM4tDz4ascDbZenXhMuEJbvn2nhwo8yj1 qTuiTh/1wlCE/FDQJ7hXRThRsD/HAS5nrgJU/zTdzFGIpde3dtd36nOxn dz+hAP4hj0NeVIQWaEmWHNckYWWcgvafJeiR0iZLYTf8TM2aHu6EnJ3H8 JX+mhltFywHtPHnwVjH4/aj/LxPzTofDqdRfxeZKhv2MuJYy4V/xcSlG1 Ao34LZrQQwjbcp76OClv/s1BbO+GHqSEE3TVPV3GIKEjy+c8oocgr8R/W jk6Y3G9HF1hcdYB3SUQGoFZWngKcmVQDsk5qfu4LlV/11xFjFq92Q3hoQ w==; X-CSE-ConnectionGUID: fKasqcD4Q+uBDr2wibQnLg== X-CSE-MsgGUID: AOFTaIeNT6qeeOvfthbBWA== X-IronPort-AV: E=McAfee;i="6600,9927,11063"; a="21248831" X-IronPort-AV: E=Sophos;i="6.07,251,1708416000"; d="scan'208";a="21248831" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 May 2024 08:52:14 -0700 X-CSE-ConnectionGUID: dnugasoRQsaXy4uMr6cBgw== X-CSE-MsgGUID: 77+fS9pCSaeXch1dLJIkKw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,251,1708416000"; d="scan'208";a="27570221" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by fmviesa010.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 03 May 2024 08:52:13 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 3 May 2024 08:52:13 -0700 Received: from orsmsx602.amr.corp.intel.com (10.22.229.15) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 3 May 2024 08:52:12 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Fri, 3 May 2024 08:52:12 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.101) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Fri, 3 May 2024 08:52:12 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LsP780Q1lKVzhHhcSmhpqBrYsWsYqoWdp4G+LtcFa3bV/+qyukJ9u4vlBLg/hCZhvGJ5D7sFhwa1L+HTqYz8/fBrEF8PM21Er9LxHnj1GCMhbX30XVji58Q03jyg1ULeiVSiKD4penorthrq+uk+UvyUTgoVeOP6CEswW5VnWCencnwZc7vBlq9oR+NhcXUktvY/RqtXMd38Bj78bN6zl/kbJ6CdFAMXvlV1iRqCH0WCEdKLqDVU7lOkC7VZ/D9tqE3aHCieic6s6/+FXJCAwSgC+eTVJO4pIMA8Li+c3+83/R3I/PUb7Hts2V8wT1lXDq/+I+DIdGKShPRua/pv6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mnked7KAxPYH4eXczjN6UMlwGRJktRSFgsXxaXjoRmo=; b=TrhmGoOi9jYCTQaGOCYLvqkhmTV+Gz1PKnyStQdCXZw9HWuV2ueFFZ5TZybdlFcRScxWvYpqQAHD9Lqil6HrSHjVsVeMQC6fz4HU1yOrFiQth5Um3iS5jJ6pqL0YDE7eSiIro5DzLsjIJhSuzgRPNVwlIO77iSfglyB8Ib0qAONrJRGG3JYrzIgolH1EPZVYzV4eq5gaf6ZJNFbnVJxUHLFgWNg+pMFdHYBcC+C/VpVLyTr5bFxrpQLKrXvVxpKIYwgaVHQP4TaAif/3zHAI/8mu7b4j80Qmko8GhrW64mKIJ+UhUOR+PiLFANYImz1TD7Guo7zqQ7yKcfnp/CPb/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by SJ0PR11MB4944.namprd11.prod.outlook.com (2603:10b6:a03:2ae::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7544.29; Fri, 3 May 2024 15:52:10 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::487e:e20c:ad88:9c0f]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::487e:e20c:ad88:9c0f%7]) with mapi id 15.20.7544.023; Fri, 3 May 2024 15:52:10 +0000 Date: Fri, 3 May 2024 16:52:07 +0100 From: Bruce Richardson To: "Lombardo, Ed" CC: Dmitry Kozlyuk , "dev@dpdk.org" , Subject: Re: Need help with reducing VIRT memory Message-ID: References: <20240502230352.7bb1ef0d@sovereign> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: DU2PR04CA0043.eurprd04.prod.outlook.com (2603:10a6:10:234::18) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|SJ0PR11MB4944:EE_ X-MS-Office365-Filtering-Correlation-Id: 9b5675fc-1252-4fed-337a-08dc6b88fe43 X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|366007|376005; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ipMmeDUiP/hvKeTHJxoDDiaApIN+bII5DyXSIlx8EmavLMr0FMy2OKyHCQIf?= =?us-ascii?Q?GdrUx0kcFcpZ2AeojyEbSJ4yrMbWnMNVPkRwtPOMsvsW2eaT5rqJv/9huXPL?= =?us-ascii?Q?AGD5DkinJ+rKS+3C405wAdSgu4dKb3qF2EOzlRHd0Js98lrCWRhTTTjPb1PW?= =?us-ascii?Q?+sYsnQwR+PmldJxIOza8eTcXnlI67nKWrn4j1UjXUEy7PE9T3rO8xfQO2fn0?= =?us-ascii?Q?uLHUUA8N5OJugrCNSuYdSncDHZWe41XCiGm71wz4Fncxo/a2dpDTvimZgsQq?= =?us-ascii?Q?yBkIZ3OYBQkyGrgheE1KUMLQQK5Un+XiYmjJ/V9afsGGc6FM6KSE2uKcc3c1?= =?us-ascii?Q?9MNCR6iDeOJTUye9V4ZJyOLg4BACa+xJUYBktzSTfApK/lDacwCLKIugAFrs?= =?us-ascii?Q?yIRR1plcxjQFgvzgTD1KEq3sGJ1b6h6weD1L+eCd/wOLCxvE5IqGZuEZcoOA?= =?us-ascii?Q?i9VZAih7vQOUAuz6EoBZvEkPRZLu3TwRJK7/wuxITcQs5CfXb/T2xjQoJX3w?= =?us-ascii?Q?J8KCjUi6FJSsnznAnjdaNMwESlEFtkTf+7ThAKVLZJEiu+q7SNgKXqbsSuQR?= =?us-ascii?Q?z3Qqx2sYDuJPRRJUDe6aX8H32vgN75Sq/SkRCgEujhZrfo3XR6iapcv9sIiK?= =?us-ascii?Q?d7zbJNs8xjp2sFgtuqXTlLu3gnXawqiug651OdRiROAdqs9mshsyWporG5SC?= =?us-ascii?Q?RQy68udaLgxsINvQYw3ygzpuvOB3XkFwHSdDNjppln2+UXro157iGRm3tigB?= =?us-ascii?Q?Z7UoKSvDtCPN99KVWgtuy2MT7jwBUC1TMIK1t43LQZMj7NyQ9Tbn6pihcI3b?= =?us-ascii?Q?H6ppTf3wnVgyTbLv/72xZ8GCBhXuPi/4P3JcGrYts4AJzOChzRdFMWpyi8TL?= =?us-ascii?Q?EZwdOAvG/n6vMNssRXkqeKwzaGw7yCIEcugBzSWhsLpN+4K6KGGxliJzpJ6Z?= =?us-ascii?Q?Ju5As6pQBAKP5tAktmhPwtKM5/bct4mF20eUGkzbGWFUTvEvooiipoZ5lq5V?= =?us-ascii?Q?M4IyB4m5FWeOg0DEt4c1JNo31VxxxRQWPZetwDX73BprGa4YgnVfbs4a4dzb?= =?us-ascii?Q?1E4HNlRReOCyaf3kKaB734lD031Y0rGVcII/vF1XUk3QCMJBTEKHgXS5Bp6Q?= =?us-ascii?Q?bJkemIokqRuZA2mcEVTv8ciBM/P0/ed/5mvkVP9UnqCdgKhbZUSFim3DX3ZN?= =?us-ascii?Q?tSiI0epl/YwH0RmAd5WPX/+e3aKyqKNc2IzlZJMWKs6d+3ZTaVBehza1zYRG?= =?us-ascii?Q?x+StyMW2K/MvYZGU26yxZdUjmKdXEWxpaOgfLvS1Zw=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(1800799015)(366007)(376005); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?qJSwXEDQjJKwS1Y5fjM3jSmyBbqtaaNMiHuFdOgLGCtWh7jNaKZktrfZBvWM?= =?us-ascii?Q?JzQZSRIBRiULUUaa+bsCoif2RYTHL75oH1rA2Msd9i5fWufT3CyYNFwm1cZl?= =?us-ascii?Q?i+7aZJgoL0vFVsSzyBBD4ioGe1BMIrNDO7bgi9KSuveM7g0WlGDXg6oR1ayM?= =?us-ascii?Q?KCvcm2R1bxkBnJUXIYzAcCTaZ/xrAGE6uRUZtbimCejTGKHGAq2sW7bxSzbb?= =?us-ascii?Q?Z9IT3KyxvVckX/tdHCxRqKBpJlXTz7G1txtqbz7CHNFgs+XVnjXvwXF/G9Kv?= =?us-ascii?Q?TGsikyeWZDYunY8Y3/iec4e+r6gsu/FPzmouzJvgI2UF6Ma+oeNi0OEsKIPh?= =?us-ascii?Q?+spllyZBhrS8Z9e1T2WQVMizUovAIrZWZAtWJnwuxIsUOLSKtjUUdqkPIOL6?= =?us-ascii?Q?6GD0b2VjoTtThreo3OAYPU+t2JQglXNfFoJAo4F3iyWBoQCouBKZnjBFlTAQ?= =?us-ascii?Q?RRAzNeBNcXgMZ1cK/s96mz/odY5z/t/KPSvnPThjmAP5sypT88C9wuo1Ic9/?= =?us-ascii?Q?hnE/xcfv/iu8tQxIWQ3QakYLoL1IjJsuh+/zLHC6Ibrz4jjUbtQjJYwSvtrF?= =?us-ascii?Q?oEZcvUb7JC4JF5MFW22FxPAYEaC81zYh0KotJoltw8clujkSt5OHQ9bI3gmP?= =?us-ascii?Q?Byli8K84ZqKngMOq59+v1q6+S8Ozvre0WTyhlcfe/ITXAU3dsTKPEQvUz9oJ?= =?us-ascii?Q?uBJ7otOfNut3zMW5884bN/YuE+ZtVMAMeBQ3zqpopBFQIJKwS3kHlU0aHnf2?= =?us-ascii?Q?4JH0FlXDH76IIptfN+eNpxzZIP8xLYsHQ74aTd+zKj1DAfxfvRodlcYYaUzp?= =?us-ascii?Q?64vTtZC8mVbsn7vwq0dto7KSdRyD6IBFQjTbwPIsgTbDGC8m6nnFKspzjndT?= =?us-ascii?Q?JPSZV7o6zND5A7eL40FQKUjYfX6WtGJKALiv1/+WO6caK2OZrc6d5bmUDJxE?= =?us-ascii?Q?Pi/HTWpB2HWdXyKpPSJxq1nGlW37kIpEZB5Z6Tc92yovLK/5f+7zNhvKQRdT?= =?us-ascii?Q?HGMxGJmIgw4Bgsx2fnufUVO8R+D9a7zf4XzXhiOKbju4zRU1sxan06t/N5vT?= =?us-ascii?Q?Hsm4Icw9FAlS8Ef1DzNGAbmOVaBit9kE5qkTL1x9fF/1x4SjZ4Zxvzs6EXC+?= =?us-ascii?Q?zoAtHmA/aOdrBn/peO3GoOr1V2i5EZve/mrJkGsRH+ALmmG0IjheFw4wFU81?= =?us-ascii?Q?cKXEXLZ0kfPlxt3mggRyh34KDD5GMI9igV7waqBrGG7NJLwihJlOLxTXXC2o?= =?us-ascii?Q?MVuk0eMd83KVqntEEAenzDAtdsifIbTetEtWfFoNOCr+NqhzJTwQg37S+fPy?= =?us-ascii?Q?jCvFo/ikOIJRJHtvx71c+7Biv7iiSytWf8kJFnRDricM2pZTmXwAhVWoUAVM?= =?us-ascii?Q?RSjNxWUJBqbtf/T71Vm9Wgx4fueeT3v6o3UheYIS67KckYdIF9vS1Q0dmPQH?= =?us-ascii?Q?bseXCGRHMLQxar3wC9xoLlDbpnGEHn3wqQkJOjL0UKxcIaRo/I/BJ/cAIxNI?= =?us-ascii?Q?EvlY6pOfh597gG97SsrbumXeedEi5OlYPyaqggjYGLJQHZQICqtRNhpeWEo6?= =?us-ascii?Q?uW748Oorqa5v9zro2ULNVDO2Z0oFKZ2W2fpkffSwls4GCbf0uTL//2pvFT0J?= =?us-ascii?Q?9Q=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 9b5675fc-1252-4fed-337a-08dc6b88fe43 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2024 15:52:10.7457 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bfI3Okp1ghCvT0NMrje0mt02uWWS3vnGdZG/mQGqTh1s6H6J/HuaZ+fMNsltW+kJ26kjm4NV1mDpbzGU5awhCFhaTcjhD1UJNiWL9a6oeSQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB4944 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, May 03, 2024 at 04:27:39PM +0100, Bruce Richardson wrote: > On Fri, May 03, 2024 at 02:48:12PM +0000, Lombardo, Ed wrote: > > Hi Dmitry, > > I am not clear on the DPDK memory layout and how to tweak these #define values. > > > > #define RTE_MAX_MEMSEG_PER_LIST 8192 > > #define RTE_MAX_MEM_MB_PER_LIST 32768 > > #define RTE_MAX_MEMSEG_PER_TYPE 32768 > > #define RTE_MAX_MEM_MB_PER_TYPE 65536 > > > > I want to limit how much DPDK grabs for memory, but grabs what it absolutely needs for our application. > > Hi, > > This is what DPDK does. What is being shown in the VIRT figures is the > address space reservation DPDK has made, but not what memory it actually uses. > Only sufficient hugepage memory to meet the demands of your app should be > mapped by DPDK, the rest is unused address space that is not taking up any > actual memory. > By way of illustration, here is the memory output for a testpmd process on my system. I got this by running "top -b -p " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2336969 bruce 20 0 256.2g 26432 19712 S 93.8 0.0 5:28.13 dpdk-testpmd If we look at the memory relevant columns, indeed VIRT shows a huge value - 256G in my case. However, the actual RAM used by testpmd is given in the "RES" (resident??) column, showing that testpmd actually is only using 26,432kB of memory in this instance, of which 19,712kB is shared memory (mostly hugepages). In fact, testpmd actually has even more hugepage memory than that mapped into it, but they must not be actually in use. [Anatoly, can you confirm that this would be the case when using vfio-pci i.e. no physical addresses to query?] Regards, /Bruce