From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5F68942925; Wed, 12 Apr 2023 10:44:19 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 03ABA40FAE; Wed, 12 Apr 2023 10:44:19 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id C2D574067B for ; Wed, 12 Apr 2023 10:44:16 +0200 (CEST) Received: from dggpeml100020.china.huawei.com (unknown [172.30.72.53]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4PxGQq4l6MzKyFL for ; Wed, 12 Apr 2023 16:41:39 +0800 (CST) Received: from dggpeml500021.china.huawei.com (7.185.36.21) by dggpeml100020.china.huawei.com (7.185.36.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 12 Apr 2023 16:44:14 +0800 Received: from dggpeml500021.china.huawei.com ([7.185.36.21]) by dggpeml500021.china.huawei.com ([7.185.36.21]) with mapi id 15.01.2507.023; Wed, 12 Apr 2023 16:44:14 +0800 From: "wuchangsheng (C)" To: "anatoly.burakov@intel.com" CC: "dev@dpdk.org" , "jiangheng (G)" , "Yanan (Euler)" , "Suweifeng (Weifeng, EulerOS)" Subject: [DPDK] heap memory fragmentation issue Thread-Topic: [DPDK] heap memory fragmentation issue Thread-Index: AdltGnbHkADfDfN2S9CgD+rz+TFNnw== Date: Wed, 12 Apr 2023 08:44:14 +0000 Message-ID: Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.136.112.45] Content-Type: multipart/alternative; boundary="_000_ba5b1a9815394b868b3fac8f4e1f3b27huaweicom_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --_000_ba5b1a9815394b868b3fac8f4e1f3b27huaweicom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello: When using rte_malloc and rte_free to request and release memory repeat= edly, the usage of large pages gradually increases. Checking the relevant source code shows that memory requests and releases a= re started from the head of the freelist chain list of the heap. Memory fra= gmentation seems to result from this, which is considered because the memor= y recently released may be in the cache, and requesting this memory at the = time of allocation may achieve higher performance? How does the community consider the heap's memory fragmentation issue? Is t= here a future plan for memory fragmentation optimization? --_000_ba5b1a9815394b868b3fac8f4e1f3b27huaweicom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello:

When using rte_malloc and rte_fr= ee to request and release memory repeatedly, the usage of large pages gradu= ally increases.

Checking the relevant = source code shows that memory requests and releases are started from the he= ad of the freelist chain list of the heap. Memory fragmentation seems to re= sult from this, which is considered because the memory recently released may be in the cache, and requesting t= his memory at the time of allocation may achieve higher performance?

How does the community= consider the heap's memory fragmentation issue? Is there a future plan for= memory fragmentation optimization?

--_000_ba5b1a9815394b868b3fac8f4e1f3b27huaweicom_-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2129742974; Tue, 18 Apr 2023 04:48:24 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A154740EDF; Tue, 18 Apr 2023 04:48:23 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id 973BE40E09 for ; Tue, 18 Apr 2023 04:48:21 +0200 (CEST) Received: from dggpeml500017.china.huawei.com (unknown [7.185.36.243]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4Q0pCf2TsCzSsyn for ; Tue, 18 Apr 2023 10:44:14 +0800 (CST) Received: from dggpeml500021.china.huawei.com (7.185.36.21) by dggpeml500017.china.huawei.com (7.185.36.243) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Tue, 18 Apr 2023 10:48:19 +0800 Received: from dggpeml500021.china.huawei.com ([7.185.36.21]) by dggpeml500021.china.huawei.com ([7.185.36.21]) with mapi id 15.01.2507.023; Tue, 18 Apr 2023 10:48:19 +0800 From: "wuchangsheng (C)" To: "anatoly.burakov@intel.com" CC: "dev@dpdk.org" , "jiangheng (G)" , "Yanan (Euler)" , "Suweifeng (Weifeng, EulerOS)" Subject: [DPDK] heap memory fragmentation issue Thread-Topic: [DPDK] heap memory fragmentation issue Thread-Index: AdltGnbHkADfDfN2S9CgD+rz+TFNnwEhSPfg Date: Tue, 18 Apr 2023 02:48:18 +0000 Message-ID: <448c5b93631a4ef191b57efb10d9424f@huawei.com> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.136.112.45] Content-Type: multipart/alternative; boundary="_000_448c5b93631a4ef191b57efb10d9424fhuaweicom_" MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --_000_448c5b93631a4ef191b57efb10d9424fhuaweicom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable ping Hello: When using rte_malloc and rte_free to request and release memory repeated= ly, the usage of large pages gradually increases. Checking the relevant source code shows that memory requests and releases a= re started from the head of the freelist chain list of the heap. Memory fra= gmentation seems to result from this, which is considered because the memor= y recently released may be in the cache, and requesting this memory at the = time of allocation may achieve higher performance? How does the community consider the heap's memory fragmentation issue? Is t= here a future plan for memory fragmentation optimization? --_000_448c5b93631a4ef191b57efb10d9424fhuaweicom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

ping

&n= bsp;

Hello:

When using rte_malloc and rte_free to r= equest and release memory repeatedly, the usage of large pages gradually in= creases.

How does the community= consider the heap's memory fragmentation issue? Is there a future plan for= memory fragmentation optimization?

--_000_448c5b93631a4ef191b57efb10d9424fhuaweicom_-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 097F842981; Tue, 18 Apr 2023 23:35:36 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7E2F44021F; Tue, 18 Apr 2023 23:35:35 +0200 (CEST) Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) by mails.dpdk.org (Postfix) with ESMTP id CF1BD4014F for ; Tue, 18 Apr 2023 23:35:33 +0200 (CEST) Received: by mail-lj1-f173.google.com with SMTP id 38308e7fff4ca-2a8c30ac7e3so18323961fa.1 for ; Tue, 18 Apr 2023 14:35:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681853733; x=1684445733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=lcpTgjevrpO0XF3HW39Dkp3T+JmApscYqv8bTKTPD4c=; b=jMHDORN47lJPJGZDSmMVWOshFTd9UiUjJW2IH8kgTVlaqrwspez/JbkcgwSP2AiOjw qKMJ2q079/tm4AD2xHO4bRq2WOxhF/QjjPrRTL2gNbF4B3fySq2Egdgv36lXQQX7qqoe cMLkxUQMvDrWv9cMgE4DPOnRXJgxKkOriCC3hJ15uj8s12a9upeMtARdYTUN9o/OP0Cq yMETN419Rm46sUrPhCZmk2NJ+GCjtSpvHl2CsD7MW8L3lhdA8SsSDag54IVEAIQmdia+ BNcMN8TqYCFSIH0qPJNogCqNIDPft+kFLVPHm7D/R6YdEkLbdpJIkjI8xxwX+EWt37ne BYBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681853733; x=1684445733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lcpTgjevrpO0XF3HW39Dkp3T+JmApscYqv8bTKTPD4c=; b=P5sY4pHUbz7IkB4Ct8vcVsFLR6HccxW8Bx2VTkTAbJM9upkRhcLwwXG29ugrhPBXr2 swI+wyOo2WvRyOq6FwTFh2hr7B3tUDBJxBi3LJslEb78BMBsljaDjJFRry3JkPHslcN8 8r8x/K+NIiTVDhRt4hFe1MrgVwRUsdckTDK3pf+QLETey92ruhKf7k+7w8gpQkwVwOLf PxUkB3oWHxdVx6JfsRLHIlK5F8tvqsgrOYFBqtLfZ5DS5WFhf7YTBkglG1JUB+FIpQXr GmW1Pc5RkpeIYZkOZJYeBTYxRTvmqgDJ1i05CGL1XDzSqGfB+TnLNFJrmPlnRnRiTdSA 6Hrg== X-Gm-Message-State: AAQBX9e0a45grP6hPEvskN7ZlCmnyT2Lh0zQTRMkaieO+Vyvj7ZjzwzS EPM51ioSRj5dK+idcH92TdY= X-Google-Smtp-Source: AKy350auM4jm1dx4v+E3e2WSH1W8HG85tl3K39SHuDcb1LBAQNmRLCON4DU9ox2qIRhXHKoN6hHBFQ== X-Received: by 2002:a19:f70b:0:b0:4ed:cb37:7d95 with SMTP id z11-20020a19f70b000000b004edcb377d95mr1301883lfe.44.1681853733009; Tue, 18 Apr 2023 14:35:33 -0700 (PDT) Received: from sovereign (broadband-109-173-110-33.ip.moscow.rt.ru. [109.173.110.33]) by smtp.gmail.com with ESMTPSA id z14-20020a19f70e000000b004eb12850c40sm2481725lfe.14.2023.04.18.14.35.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 14:35:32 -0700 (PDT) Date: Wed, 19 Apr 2023 00:35:31 +0300 From: Dmitry Kozlyuk To: "wuchangsheng (C)" Cc: "anatoly.burakov@intel.com" , "dev@dpdk.org" , "jiangheng (G)" , "Yanan (Euler)" , "Suweifeng (Weifeng, EulerOS)" Subject: Re: [DPDK] heap memory fragmentation issue Message-ID: <20230419003531.6f492cce@sovereign> In-Reply-To: References: X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi, 2023-04-12 08:44 (UTC+0000), wuchangsheng (C): > When using rte_malloc and rte_free to request and release memory repeated= ly, the usage of large pages gradually increases. Do you have a repro? > Checking the relevant source code shows that memory requests and releases > are started from the head of the freelist chain list of the heap. > Memory fragmentation seems to result from this, which is considered becau= se > the memory recently released may be in the cache, and requesting this > memory at the time of allocation may achieve higher performance? Could you please elaborate? DPDK uses "first fit" algorithm to select the free block, is that what you mean by "memory fragmentation seems to result from this"? > How does the community consider the heap's memory fragmentation issue? Is > there a future plan for memory fragmentation optimization? =46rom my experience with a production app, fragmentation is indeed an issue. Unfortunately, fragmentation often happens within a single 1G hugepage. It is impossible to coalesce free fragments of a hugepage into a contiguous block, like an OS would do with disjoint pages. Often the application can move its objects in memory, though. DPDK could provide means for applications to defragment memory this way. I don't think DPDK should drive the defragmentation process, because only the application knows when, which, and how much memory it is reasonable to move around. DPDK could provide facilities to analyze memory fragmentation degree (now possible by tracking objects and inspecting memseg, but hard) and to allocate memory at the desired range (API missing). From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 163F242981; Wed, 19 Apr 2023 00:43:55 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 884A040DF8; Wed, 19 Apr 2023 00:43:54 +0200 (CEST) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mails.dpdk.org (Postfix) with ESMTP id A1AD74021F for ; Wed, 19 Apr 2023 00:43:52 +0200 (CEST) Received: by mail-pl1-f180.google.com with SMTP id p17so16997136pla.3 for ; Tue, 18 Apr 2023 15:43:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20221208.gappssmtp.com; s=20221208; t=1681857831; x=1684449831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=MwsMTFLu1DhOlTghmSojcuJXONLWl7vU4400PB3Ew34=; b=0uGhSLx7dpermoydEQCFfiRuopCI0O2OcS6m+grNansr4u1GVZK8M5IeHwBPXrx8jo IHbBo0fKiJeubAY1wvaR5nQkIz8uGB7mNmXONzxAjog6+nIgc3Y0X51naeFea9QERlxv rvRGS3gP3QeVRB0RovLrAi567Q2ev6jrQduYNT6YHEpri9wSWqPpzamDs+ZCWv51DxSO cCmO3ANALCXgAzmCq5guK2ALCjwKvjRbuPChTvNrySva3x+yPx7ZN0RYF7Geb7hyKSp5 3E7WyJfyt4R8ZQtO8+cOOQP5Tb8zladrBafN2aYWU3pUPQq8n6CnMaWcmDawp7yngQpx Usaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681857831; x=1684449831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MwsMTFLu1DhOlTghmSojcuJXONLWl7vU4400PB3Ew34=; b=YyjJfg6++9njjXNmlmWpYboc8RTnmcaq7ETegVvgUO+gSc1pR+cxJTYrI3d1h8YCxT o6Eks+VAihzQYCBINjIbfYejuxZ9NL7hYye5eh+Mndw7zAO1+fzyfjv11v0GuBOND7en Ydg8oYjvYAPtVFp63dy6+Lh2g0J7gqutA6jtje85sybwZCnJ7Bf11kXr53M3GnjbNyTI 5h+xx9xM3+EDAZQg59B6xi0Gx6YAd+uHCNLZOxVkcnPx++yDJpeckPYjgyCRCwOtIqDd B63UrN9aKb761N9klynt9vNmXznjrQfK4WY6S3g22kdTp4RV8LXoGPzfniGAkG2DWUef Relg== X-Gm-Message-State: AAQBX9cNeQ+oqZCcGj4Pf1Ak1jSUUGvQerTaQcptS4LuRvFqk+DIINOT h6GGzX98mR42dBYrNdpAdNsY2g== X-Google-Smtp-Source: AKy350b3ZHm2hFoS6ilwAGueOm3OD5/3XlImjkyaBJ53U06DBLV3OIQ5wC/ZmijY+1gq1SpW5BBpHg== X-Received: by 2002:a05:6a21:6d8e:b0:ef:f558:b80 with SMTP id wl14-20020a056a216d8e00b000eff5580b80mr1237980pzb.58.1681857831546; Tue, 18 Apr 2023 15:43:51 -0700 (PDT) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id z16-20020aa791d0000000b0062bc045bf4fsm10178563pfa.19.2023.04.18.15.43.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 15:43:51 -0700 (PDT) Date: Tue, 18 Apr 2023 15:43:49 -0700 From: Stephen Hemminger To: Dmitry Kozlyuk Cc: "wuchangsheng (C)" , "anatoly.burakov@intel.com" , "dev@dpdk.org" , "jiangheng (G)" , "Yanan (Euler)" , "Suweifeng (Weifeng, EulerOS)" Subject: Re: [DPDK] heap memory fragmentation issue Message-ID: <20230418154349.10b4261e@hermes.local> In-Reply-To: <20230419003531.6f492cce@sovereign> References: <20230419003531.6f492cce@sovereign> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, 19 Apr 2023 00:35:31 +0300 Dmitry Kozlyuk wrote: > Hi, > > 2023-04-12 08:44 (UTC+0000), wuchangsheng (C): > > When using rte_malloc and rte_free to request and release memory repeatedly, the usage of large pages gradually increases. > > Do you have a repro? > > > Checking the relevant source code shows that memory requests and releases > > are started from the head of the freelist chain list of the heap. > > Memory fragmentation seems to result from this, which is considered because > > the memory recently released may be in the cache, and requesting this > > memory at the time of allocation may achieve higher performance? > > Could you please elaborate? > DPDK uses "first fit" algorithm to select the free block, > is that what you mean by "memory fragmentation seems to result from this"? > > > How does the community consider the heap's memory fragmentation issue? Is > > there a future plan for memory fragmentation optimization? > > From my experience with a production app, fragmentation is indeed an issue. The problem is that simple first-fit is often not good enough. Memory allocation algorithms are much researched area of operating system design. You will find lots of lectures online like https://www.scs.stanford.edu/15au-cs140/notes/mem_allocation.pdf If interested, some other malloc libraries: https://en.wikipedia.org/wiki/C_dynamic_memory_allocation Glibc https://sourceware.org/glibc/wiki/MallocInternals Dimalloc Small (<256) use simple best fit power of two with splitting Medium bitwise trie Large use mmap JeMalloc https://jemalloc.net/ What FreeBSD uses, avoids fragmentation Mimalloc https://microsoft.github.io/mimalloc/ Uses free list sharding (ie free list per page) https://www.microsoft.com/en-us/research/uploads/prod/2019/06/mimalloc-tr-v1.pdf If someone wants to go deeper, then looking into some combination of these would be good. Some of these already are huge page aware so probably DPDK could stop reinventing this.