From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 163F242981; Wed, 19 Apr 2023 00:43:55 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 884A040DF8; Wed, 19 Apr 2023 00:43:54 +0200 (CEST) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mails.dpdk.org (Postfix) with ESMTP id A1AD74021F for ; Wed, 19 Apr 2023 00:43:52 +0200 (CEST) Received: by mail-pl1-f180.google.com with SMTP id p17so16997136pla.3 for ; Tue, 18 Apr 2023 15:43:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20221208.gappssmtp.com; s=20221208; t=1681857831; x=1684449831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=MwsMTFLu1DhOlTghmSojcuJXONLWl7vU4400PB3Ew34=; b=0uGhSLx7dpermoydEQCFfiRuopCI0O2OcS6m+grNansr4u1GVZK8M5IeHwBPXrx8jo IHbBo0fKiJeubAY1wvaR5nQkIz8uGB7mNmXONzxAjog6+nIgc3Y0X51naeFea9QERlxv rvRGS3gP3QeVRB0RovLrAi567Q2ev6jrQduYNT6YHEpri9wSWqPpzamDs+ZCWv51DxSO cCmO3ANALCXgAzmCq5guK2ALCjwKvjRbuPChTvNrySva3x+yPx7ZN0RYF7Geb7hyKSp5 3E7WyJfyt4R8ZQtO8+cOOQP5Tb8zladrBafN2aYWU3pUPQq8n6CnMaWcmDawp7yngQpx Usaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681857831; x=1684449831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MwsMTFLu1DhOlTghmSojcuJXONLWl7vU4400PB3Ew34=; b=YyjJfg6++9njjXNmlmWpYboc8RTnmcaq7ETegVvgUO+gSc1pR+cxJTYrI3d1h8YCxT o6Eks+VAihzQYCBINjIbfYejuxZ9NL7hYye5eh+Mndw7zAO1+fzyfjv11v0GuBOND7en Ydg8oYjvYAPtVFp63dy6+Lh2g0J7gqutA6jtje85sybwZCnJ7Bf11kXr53M3GnjbNyTI 5h+xx9xM3+EDAZQg59B6xi0Gx6YAd+uHCNLZOxVkcnPx++yDJpeckPYjgyCRCwOtIqDd B63UrN9aKb761N9klynt9vNmXznjrQfK4WY6S3g22kdTp4RV8LXoGPzfniGAkG2DWUef Relg== X-Gm-Message-State: AAQBX9cNeQ+oqZCcGj4Pf1Ak1jSUUGvQerTaQcptS4LuRvFqk+DIINOT h6GGzX98mR42dBYrNdpAdNsY2g== X-Google-Smtp-Source: AKy350b3ZHm2hFoS6ilwAGueOm3OD5/3XlImjkyaBJ53U06DBLV3OIQ5wC/ZmijY+1gq1SpW5BBpHg== X-Received: by 2002:a05:6a21:6d8e:b0:ef:f558:b80 with SMTP id wl14-20020a056a216d8e00b000eff5580b80mr1237980pzb.58.1681857831546; Tue, 18 Apr 2023 15:43:51 -0700 (PDT) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id z16-20020aa791d0000000b0062bc045bf4fsm10178563pfa.19.2023.04.18.15.43.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 15:43:51 -0700 (PDT) Date: Tue, 18 Apr 2023 15:43:49 -0700 From: Stephen Hemminger To: Dmitry Kozlyuk Cc: "wuchangsheng (C)" , "anatoly.burakov@intel.com" , "dev@dpdk.org" , "jiangheng (G)" , "Yanan (Euler)" , "Suweifeng (Weifeng, EulerOS)" Subject: Re: [DPDK] heap memory fragmentation issue Message-ID: <20230418154349.10b4261e@hermes.local> In-Reply-To: <20230419003531.6f492cce@sovereign> References: <20230419003531.6f492cce@sovereign> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, 19 Apr 2023 00:35:31 +0300 Dmitry Kozlyuk wrote: > Hi, > > 2023-04-12 08:44 (UTC+0000), wuchangsheng (C): > > When using rte_malloc and rte_free to request and release memory repeatedly, the usage of large pages gradually increases. > > Do you have a repro? > > > Checking the relevant source code shows that memory requests and releases > > are started from the head of the freelist chain list of the heap. > > Memory fragmentation seems to result from this, which is considered because > > the memory recently released may be in the cache, and requesting this > > memory at the time of allocation may achieve higher performance? > > Could you please elaborate? > DPDK uses "first fit" algorithm to select the free block, > is that what you mean by "memory fragmentation seems to result from this"? > > > How does the community consider the heap's memory fragmentation issue? Is > > there a future plan for memory fragmentation optimization? > > From my experience with a production app, fragmentation is indeed an issue. The problem is that simple first-fit is often not good enough. Memory allocation algorithms are much researched area of operating system design. You will find lots of lectures online like https://www.scs.stanford.edu/15au-cs140/notes/mem_allocation.pdf If interested, some other malloc libraries: https://en.wikipedia.org/wiki/C_dynamic_memory_allocation Glibc https://sourceware.org/glibc/wiki/MallocInternals Dimalloc Small (<256) use simple best fit power of two with splitting Medium bitwise trie Large use mmap JeMalloc https://jemalloc.net/ What FreeBSD uses, avoids fragmentation Mimalloc https://microsoft.github.io/mimalloc/ Uses free list sharding (ie free list per page) https://www.microsoft.com/en-us/research/uploads/prod/2019/06/mimalloc-tr-v1.pdf If someone wants to go deeper, then looking into some combination of these would be good. Some of these already are huge page aware so probably DPDK could stop reinventing this.