From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <users-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B5061A0547 for <public@inbox.dpdk.org>; Wed, 29 Sep 2021 14:40:41 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 46129410E5; Wed, 29 Sep 2021 14:40:41 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 8EFB640685 for <users@dpdk.org>; Wed, 29 Sep 2021 14:40:39 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10121"; a="310478376" X-IronPort-AV: E=Sophos;i="5.85,332,1624345200"; d="scan'208";a="310478376" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Sep 2021 05:40:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,332,1624345200"; d="scan'208";a="438434870" Received: from orsmsx604.amr.corp.intel.com ([10.22.229.17]) by orsmga003.jf.intel.com with ESMTP; 29 Sep 2021 05:40:35 -0700 Received: from orsmsx608.amr.corp.intel.com (10.22.229.21) by ORSMSX604.amr.corp.intel.com (10.22.229.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Wed, 29 Sep 2021 05:40:35 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX608.amr.corp.intel.com (10.22.229.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Wed, 29 Sep 2021 05:40:34 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12 via Frontend Transport; Wed, 29 Sep 2021 05:40:34 -0700 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.173) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2242.12; Wed, 29 Sep 2021 05:40:34 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aH3KF+Mkv5uKma0/f+KFegFpNuYXL/yMhIe1NF0Qw7NYt0SLWEF4QFlNGVKWwt2ZAy4IEvahwFOgMeCgtrE0uXpkZjXwC0Qj1TK5jrtQcnQxLzmhxu5s8ux///Nns6/IgSj+IvsKeO94cAOD/7msi6FbZnYAYLG9bZDFbDokDrY+gWs3O8bCDUBzoOrH0ZwU4UFRSFalzl4KGo0GRIMYMne/J3kinjb/V/2dxVE/zW55NgrLZNcYqZgNES3tF/1OaqV6ONJuSbrNQ/yx4QvQ3LTpdbzgh2s6Aq3ipODaKcMVPjjStkPDMN5azAgq7veK0DHHLd8jPSVJlqM59wL08g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=fGb/qFw+/kv3rUiQTLi/Tch6CklklS44XfYfbA0o5Ko=; b=M0CNOaYl6HjGUpYy0Dm6kayxZTIljZOtbHX0nOvhU4sOdg3bSFYWyKaBXhzD7xvTIX73AN9naVh4t8dNEWkWt6gRPLqpg0u6pFvkC4b96qg79d+K5g/nVVn9SMSuD0/6j+sTxrrLFIcGERegXNx3MygJJuiL0uuQeU++DHgKenKydJl2S+Ego5PJa3jI1z/4J3S4VWf+0IGBluj4o4FHw1FzVQajd2NF6XtdhYMdf36K2xBw+kATuurTyW9KMLRe79pVzQf2bGrkfyt4XhWfP89D+FpY9tyaJXfqR1sLMgi7rew5QWtQcwQRn4KEFlxfBzO2m1RTT2g2tExWedxl3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fGb/qFw+/kv3rUiQTLi/Tch6CklklS44XfYfbA0o5Ko=; b=iPagLjA7cmQO0VVLq1A+Iw7k7vyxK8kt9psmbGKgQ+k+aabZ5ZRtA2lq9TsxUzvUTRNXJ4CLQVBvPM8TzQmx0cr/lFcwkWlCC3Cyv1jipSQ4aYRqGMxtFBjirxnEhtELaGg2muo1JkgeTX3OmUgf/kFA/1Rn+9dKdPq76bRCRsw= Received: from PH0PR11MB5093.namprd11.prod.outlook.com (2603:10b6:510:3e::23) by PH0PR11MB5112.namprd11.prod.outlook.com (2603:10b6:510:3b::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.14; Wed, 29 Sep 2021 12:40:33 +0000 Received: from PH0PR11MB5093.namprd11.prod.outlook.com ([fe80::d145:710a:1bac:7e91]) by PH0PR11MB5093.namprd11.prod.outlook.com ([fe80::d145:710a:1bac:7e91%9]) with mapi id 15.20.4566.014; Wed, 29 Sep 2021 12:40:33 +0000 From: "Burakov, Anatoly" <anatoly.burakov@intel.com> To: Thomas Monjalon <thomas@monjalon.net>, Jiany Wu <wujianyue000@gmail.com> CC: "users@dpdk.org" <users@dpdk.org>, "Richardson, Bruce" <bruce.richardson@intel.com>, "olivier.matz@6wind.com" <olivier.matz@6wind.com>, "dmitry.kozliuk@gmail.com" <dmitry.kozliuk@gmail.com>, "stephen@networkplumber.org" <stephen@networkplumber.org>, "Mcnamara, John" <john.mcnamara@intel.com> Subject: RE: [dpdk-users] can we reserve hugepage and not release Thread-Topic: [dpdk-users] can we reserve hugepage and not release Thread-Index: AQHXtRlji2zqhAaLHUSLD0Nvrcen9au6ymqggAAQwwCAABdZwA== Date: Wed, 29 Sep 2021 12:40:33 +0000 Message-ID: <PH0PR11MB50935FEA5233F5DC875B99A3F7A99@PH0PR11MB5093.namprd11.prod.outlook.com> References: <CAJxJ_jgsN+PHbBtY8Ju5V5xm2NGQwNNtfw9PJScfVs9thZmzuw@mail.gmail.com> <3927535.NYFkNA2spg@thomas> <PH0PR11MB5093B5D0808FA9C81FFB99D3F7A99@PH0PR11MB5093.namprd11.prod.outlook.com> <7391021.896Q4A1nnV@thomas> In-Reply-To: <7391021.896Q4A1nnV@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 13c48874-86b1-411d-0442-08d983465423 x-ms-traffictypediagnostic: PH0PR11MB5112: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: <PH0PR11MB511228DA9AD5B60D2F441107F7A99@PH0PR11MB5112.namprd11.prod.outlook.com> x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 1bQ1B8nUigCSH6MuUvhP0VARKLOwDnD00jpcxyCXz2gEggLd0gM9v/Ia9UN7XnxYJkjv7DHq4Z4HT6OJyKYsefxuqyOf0SdmitbBG5ArFWLL7tgFpg92dQjL4xcK3F6Ud6npemDHY0VQn08RyKnGNdPgsIn4DTADdO/q1O9guGY0ftIa/7ShJlP/r3rAOvK2IMlzjBPiyFFVaAiaD+5KzuspKBNP51g6EBuJdPdMOZavU25+q3XGdOnNsMSNj2w77zPf1HR2Vkz+Wn6Gt+BkhU5gQXKt98fkt2Ev9gJs2b7QKqEMk0jpRpVeRsArHTA6Hm+IBO4LW4ncaaE7GcrMpmYjTOEbXfSaHIZP3XHqe0KqlxB9qg9n2gHAAnqEmSSRWYsA0zb+U4OrkpD44kwcXevCqNPi7GUxfvBDu5tKBHS9AszmqanhiTlajYjHvKbUFG6Qx6CSkwyusjHVrbwDrWvh2Vajw4nwetFIXtjJkZgu5gPcji+4ASLaPbX4wP8zIE174CjXpk1YkJLN1+/gJgQVxc+qQQJTdc7NzkdDZ3wKbQwSlJLhooL61uiKLWs1YHc6IgWOM9huOkdRUSydah32IGbbGUbak2bSchuagUbSgb55PhW7QoT+Kv4/PIX1Dy4sEwFelrWqB/FR+vdsZ0Rm+yfTvoY2Tj6XLnLoc7pNHJugOgGCFwP6qmAUpLrSFJAtbyIU0Qrusodsc9Mowg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR11MB5093.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(8676002)(122000001)(33656002)(2906002)(9686003)(5660300002)(107886003)(110136005)(54906003)(7696005)(66556008)(66476007)(186003)(64756008)(66446008)(4326008)(53546011)(66946007)(76116006)(38100700002)(8936002)(52536014)(71200400001)(6506007)(55016002)(38070700005)(26005)(86362001)(83380400001)(508600001)(316002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?mIeXTOTuFRmfZXaYJ0d/UROt1OIqZDbWm25QQA4cWXUP+fEHw3rE9UzVhdgb?= =?us-ascii?Q?6emXiVrWjagqLY6Ya7LUE5LAZg1AnjytDhkTOVUjG1wGshOmSHDYjxwYVuUA?= =?us-ascii?Q?Rn+lqnD4ZC70zoQvY62ZzBikcH6CSMBIEAWDlPA/HL/3lSIev8D6qSmA3l1y?= =?us-ascii?Q?4pLBsk2o39WbNJnMoI/8biKHgbzwzJtASA+5sPBrQQwcGlZ3G/zPLk0VVuN+?= =?us-ascii?Q?+kiANisvRQZRgs9E1g9KJR0url9z9m+WXkqOsyMTdDBAJq/JxtdlCwrcL3Oi?= =?us-ascii?Q?EuPobdRunN4gjw0Yns7uerpiCDkawCQzqMeg0ZPF6wufL8F/FC8xdFX2Qheh?= =?us-ascii?Q?WJK8GnuYKUYp9r9D+bKVFAe2EjjSG/YTbaxGepTaTZ7M2BUtyg1QI8boykoH?= =?us-ascii?Q?Dz/uR2OYvFUoVsyvqNX4EC6RaDT+nNV9JegGb8IxUb+4WXKH04Vjs/MyN33A?= =?us-ascii?Q?eL0SSy/GpD/AtuO2ePDis5QJotprSvJfBwQU5sdw6zmpSi+yhikPY++AyvRg?= =?us-ascii?Q?XJD3bJqhyM3iyPKB+ScBVXK6lUBEr1dLvBlUc1ImRcMDZtmtV8cuenq+KwAU?= =?us-ascii?Q?+yFxoyLruBcillXSC/0vzuVbCP/aeCHyPxNbM3oqkcE/vVu9lbw17iaNJdAn?= =?us-ascii?Q?hjPgDo0wtglt+HWNOh97xOOeh5OFbGfgGSeY0VDAc1ASISG5QzyLUfloB+0B?= =?us-ascii?Q?3+pJusjYPBEc9w/1Hy2ROL+dheE/0tfOhZJjdySf+iHyfnPzWVAvllMeIrFp?= =?us-ascii?Q?+sLKAlPtj8dWDfBHg+2jVnwNwbFNEfl1ruAhKE2x6eiOS+78n4RUZ0TVZvz6?= =?us-ascii?Q?s0i0hxxuKR2YjgLSKuOlry9Z5iJCi0ibaUZcK1mXoG2HAoyZSyKqYR2QDD3d?= =?us-ascii?Q?BtaqkK0Y9F0NlYu1G77PfLLKKgK1149/zBsY2hlJwNcUFE1XMZHTFGKjXO8f?= =?us-ascii?Q?HmVHQHp6LY0mYFo1muJmMXMyMezX1hdTqM9Jxeiyz3fVqPlA1tanBwbJ+cIY?= =?us-ascii?Q?Ua1Rr7RJ0rMLKFyc+uv56kpHENye+oFfYeTct1MW5qqc08QUrxPhd3mes76I?= =?us-ascii?Q?4FTtE95zao6sb4jI0OrYdIdXwz3R4rtAiamwQ6vzofDU+NVaNidmdhHT/b3e?= =?us-ascii?Q?RQGYITcHQhYZLIMc9G+UH9M/tQwbmfNCexUZ+DUo7DxkrRoy0EgILFYvt0lT?= =?us-ascii?Q?qCHOrgYmioXkZVUFKHL6eIqq+iT2ZkIhyuKRAb2t+gLAmNflP+VDBT2+H87n?= =?us-ascii?Q?ZmKlf0qbYtXOo9b2aIppt8mB7fcsoiPnoE28EMEOTPeIaVlfcT1UlXeMfhTa?= =?us-ascii?Q?x23yXHpU/3Nt1fTjNY5ttUxZ?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5093.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 13c48874-86b1-411d-0442-08d983465423 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Sep 2021 12:40:33.0413 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: IYwiMsglClAsL/cCJCxjFR7LwuOErsEOu8RCl3DrNVlYOwjKqnzwPSRA5rVKYXhneYLFKQmd/yqEFwWTA0fmw+h5F/iXCmXpitBmF+JndsM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5112 X-OriginatorOrg: intel.com X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions <users.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/users>, <mailto:users-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/users/> List-Post: <mailto:users@dpdk.org> List-Help: <mailto:users-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/users>, <mailto:users-request@dpdk.org?subject=subscribe> Errors-To: users-bounces@dpdk.org > -----Original Message----- > From: Thomas Monjalon <thomas@monjalon.net> > Sent: Wednesday, September 29, 2021 12:11 PM > To: Jiany Wu <wujianyue000@gmail.com>; Burakov, Anatoly > <anatoly.burakov@intel.com> > Cc: users@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; > olivier.matz@6wind.com; dmitry.kozliuk@gmail.com; > stephen@networkplumber.org; Mcnamara, John > <john.mcnamara@intel.com> > Subject: Re: [dpdk-users] can we reserve hugepage and not release >=20 > 29/09/2021 12:16, Burakov, Anatoly: > > From: Thomas Monjalon <thomas@monjalon.net> > > > 18/09/2021 04:37, Jiany Wu: > > > > Hello, > > > > > > > > I met a scenario that, need to start and stop the container many > > > > times for the hugepage. But after several times container start > > > > and stop, the hugepage is not able to reserve. > > > > Hugepage size is 2MB, and HW only support 2MB, can't support 1GB. > > > > Is there anyway to make sure the hugepage is still kept continuous? > > > > Thanks indeed. > > > > > > Interesting question. > > > I think we need to address it in the DPDK documentation. > > > > > > Anatoly, Stephen, Bruce, any advice please? > > > > > > > Hi, > > > > From description, I don't quite understand what's the issue here. Is th= e > problem about "contiguousness of memory", or is it about inability to res= erve > more hugepages? >=20 > I think the issue is that sometimes some pages are not properly released,= so > we cannot reserve them again. > That's something I experienced myself. > Any trick to reset hugepages state? [[AB]]=20 Under normal circumstances, the hugepage files would just be deleted and re= created the next time a primary process initializes even if there are lefto= ver pages. That said, assuming the pages are not "released" because DPDK has left a fe= w files behind and a new container doesn't have permissions to delete them = (or can't for some other reason), then there is very little DPDK can do oth= er than track down all memory leaks. For example, using RX callback feature= is intentionally causing a memory leak because by default the API will not= free the callback structure as we cannot guarantee that there are no Datap= lane threads still using that structure. Individual drivers/libraries might= also leak memory due to e.g. secondary process usage. For example, in the = general case, if something is meant to be shared by primary and secondary, = but primary cannot free() it because it might still be used by secondaries,= and secondaries have no way of verifying whether the resource is being use= d by other processes, or can be freed. Put it simply, sometimes DPDK may leak memory either out of necessity, or e= ven intentionally, to avoid potential use-after-free errors, and there's no= general way to solve this problem that I'm aware of. This is the kind of s= tuff --in-memory mode was introduced to address, as when hugepages are in m= emory, there can never be leaks because we never touch the filesystem in th= e first place, so if there is a recommended way to address this at all, it = would be --in-memory mode.=20 >=20 > > How are hugepages assigned to your container? > > Have you tried using --in-memory mode? >=20 >=20