From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 17BD7A04C7; Wed, 16 Sep 2020 06:33:37 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 680B71C21B; Wed, 16 Sep 2020 06:32:35 +0200 (CEST) Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) by dpdk.org (Postfix) with ESMTP id 5E4DB1C1E3 for ; Wed, 16 Sep 2020 06:32:32 +0200 (CEST) Received: by mail-lf1-f52.google.com with SMTP id b12so5437598lfp.9 for ; Tue, 15 Sep 2020 21:32:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=HKGNnbzoTUtSyuv3STUYkMHdSpQ3RNPwMGybHhQwuTY=; b=PzR727z0wVOiqyVbvqQubzyYwDmpv/QBFom8RNGUqgdSPaNf4bne74GxMKtAAf1Caq uT1xaXaHYxZEUKWdxa3TNaA5jdL1ustGz6DG1Aep7n9ObZuhU4oOfCj8cgpZuSzrhM2u t1Ne2kCQbc+eAsS4gQLh0qg7te4n1h+7pE77oqaKdQbwVaOgzzez5bSWS0+8tKBb6h22 jAUcs2FxUlOiz9bM12Y5TPMBIibWLDKoy808sx5zHY1gIRyF6foToU/9K2/lBBtjwV5p CWlj9JlQdepZ3w8Kf5igv2a//hhIWKJpbd59jeVF3FZI0dtPo5Rb06jVCtNIQwenzAN4 HcRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HKGNnbzoTUtSyuv3STUYkMHdSpQ3RNPwMGybHhQwuTY=; b=k2FzH4T4/1VCP+8C1/RTXk2uJ/sPczBmChKsaJSuVmOfAr7s+fZGrhszUwzgau1BN2 EzhFLqgjyaR8ysAWEJEphpv5Qr6M3Ia4eknRMZp9+q4tLKlTxBRRJj4Hfd36+j1S7poi j9evAdpK88CP0paGsUXBFQB7/Fli2dSqhli3vb/IuYbXyOjtw9r4rJ6VGFV9OUTpi9Pz 6xtlDhltkMAhxo4Hv2jGDHPV/cF+IYcYDmkHRITrOj1win7sYefqpo2226COG515VJ1Q uqSCajmzK7OoXU5gqU+oMHZDvlx+VZ9Y6FzUiZ7xtcUyT7PxUwKoCETkExu5Qj6JFKV8 eL0g== X-Gm-Message-State: AOAM531KZVkPfuogflZtfo7VhiBuZsej7Ui7aRQFdw6uQXT6CsxSyKtI y2ouD6aDMGz8Plg6KJfnzFwZIU9cM8lsQ4D+gjM= X-Google-Smtp-Source: ABdhPJzXmEpr0CEMbwb1XRNkjvN+nqMBHCJ+K0dRjM2GwSYD8QNsc7djV6MruYkozCyTbauP+DaXcBu26LQucqLaBO0= X-Received: by 2002:ac2:43c2:: with SMTP id u2mr7700431lfl.586.1600230751497; Tue, 15 Sep 2020 21:32:31 -0700 (PDT) MIME-Version: 1.0 References: <20200710102837.GB684@bricha3-MOBL.ger.corp.intel.com> <75824075-9690-814a-1849-1107504ce344@intel.com> In-Reply-To: From: Kamaraj P Date: Wed, 16 Sep 2020 10:02:18 +0530 Message-ID: To: "Burakov, Anatoly" Cc: Bruce Richardson , dev@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] DPDK hugepage memory fragmentation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Anatoly, We just dump the memory contents when it fails to allocate the memory. rte_malloc_dump_stats(); rte_dump_physmem_layout(); rte_memzone_dump(); We could see the fragmented memory ----------- MALLOC_STATS ----------- Heap id:0 Heap name:socket_0 Heap_size:134217728, Free_size:133604096, Alloc_size:613632, Greatest_free_size:8388608, Alloc_count:48, Free_count:53, Heap id:1 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:2 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:3 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:4 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:5 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:6 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:7 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:8 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:9 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:10 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:11 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:12 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:13 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:14 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:15 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:16 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:17 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:18 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:19 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:20 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:21 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:22 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:23 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:24 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:25 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:26 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:27 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:28 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:29 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:30 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, Heap id:31 Heap name: Heap_size:0, Free_size:0, Alloc_size:0, Greatest_free_size:0, Alloc_count:0, Free_count:0, --------- END_MALLOC_STATS --------- ----------- MEMORY_SEGMENTS ----------- Segment 4-0: IOVA:0x2a5e00000, len:2097152, virt:0x2200200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:15 Segment 4-1: IOVA:0x2a6000000, len:2097152, virt:0x2200400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:19 Segment 4-2: IOVA:0x2a6200000, len:2097152, virt:0x2200600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:20 Segment 4-4: IOVA:0x2a6600000, len:2097152, virt:0x2200a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:21 Segment 4-6: IOVA:0x2a7000000, len:2097152, virt:0x2200e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:22 Segment 4-8: IOVA:0x2a7600000, len:2097152, virt:0x2201200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:23 Segment 4-10: IOVA:0x2a8000000, len:2097152, virt:0x2201600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:24 Segment 4-11: IOVA:0x2a8200000, len:2097152, virt:0x2201800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:25 Segment 4-12: IOVA:0x2a8400000, len:2097152, virt:0x2201a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:26 Segment 4-13: IOVA:0x2a8600000, len:2097152, virt:0x2201c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:27 Segment 4-15: IOVA:0x2a8c00000, len:2097152, virt:0x2202000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:28 Segment 4-16: IOVA:0x2a8e00000, len:2097152, virt:0x2202200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:29 Segment 4-18: IOVA:0x2a9400000, len:2097152, virt:0x2202600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:30 Segment 4-20: IOVA:0x2aa400000, len:2097152, virt:0x2202a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:31 Segment 4-21: IOVA:0x2aa600000, len:2097152, virt:0x2202c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:32 Segment 4-23: IOVA:0x2aac00000, len:2097152, virt:0x2203000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:33 Segment 4-25: IOVA:0x2abc00000, len:2097152, virt:0x2203400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:34 Segment 4-26: IOVA:0x2abe00000, len:2097152, virt:0x2203600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:35 Segment 4-28: IOVA:0x2b0400000, len:2097152, virt:0x2203a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:36 Segment 4-29: IOVA:0x2b0600000, len:2097152, virt:0x2203c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:37 Segment 4-30: IOVA:0x2b0800000, len:2097152, virt:0x2203e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:38 Segment 4-31: IOVA:0x2b0a00000, len:2097152, virt:0x2204000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:39 Segment 4-33: IOVA:0x2b1200000, len:2097152, virt:0x2204400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:40 Segment 4-35: IOVA:0x2b2600000, len:2097152, virt:0x2204800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:41 Segment 4-37: IOVA:0x2b2a00000, len:2097152, virt:0x2204c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:42 Segment 4-39: IOVA:0x2b5000000, len:2097152, virt:0x2205000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:43 Segment 4-41: IOVA:0x2b5600000, len:2097152, virt:0x2205400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:44 Segment 4-43: IOVA:0x2b6800000, len:2097152, virt:0x2205800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:45 Segment 4-45: IOVA:0x2b7c00000, len:2097152, virt:0x2205c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:46 Segment 4-47: IOVA:0x2b8e00000, len:2097152, virt:0x2206000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:47 Segment 4-49: IOVA:0x2b9a00000, len:2097152, virt:0x2206400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:48 Segment 4-50: IOVA:0x2b9c00000, len:2097152, virt:0x2206600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:49 Segment 4-51: IOVA:0x2b9e00000, len:2097152, virt:0x2206800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:50 Segment 4-53: IOVA:0x2ba400000, len:2097152, virt:0x2206c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:51 Segment 4-55: IOVA:0x2bbe00000, len:2097152, virt:0x2207000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:52 Segment 4-57: IOVA:0x2bc200000, len:2097152, virt:0x2207400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:53 Segment 4-59: IOVA:0x2c1c00000, len:2097152, virt:0x2207800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:54 Segment 4-60: IOVA:0x2c1e00000, len:2097152, virt:0x2207a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:55 Segment 4-62: IOVA:0x2d1a00000, len:2097152, virt:0x2207e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:56 Segment 4-64: IOVA:0x2d5000000, len:2097152, virt:0x2208200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:57 Segment 4-130: IOVA:0x2d7000000, len:2097152, virt:0x2210600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:58 Segment 4-132: IOVA:0x2d8000000, len:2097152, virt:0x2210a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:59 Segment 4-134: IOVA:0x2d8400000, len:2097152, virt:0x2210e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:60 Segment 4-136: IOVA:0x2db400000, len:2097152, virt:0x2211200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:61 Segment 4-138: IOVA:0x2dc600000, len:2097152, virt:0x2211600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:62 Segment 4-139: IOVA:0x2dc800000, len:2097152, virt:0x2211800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:63 Segment 4-140: IOVA:0x2dca00000, len:2097152, virt:0x2211a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:64 Segment 4-142: IOVA:0x2de800000, len:2097152, virt:0x2211e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:65 Segment 4-143: IOVA:0x2dea00000, len:2097152, virt:0x2212000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:66 Segment 4-145: IOVA:0x3d8c00000, len:2097152, virt:0x2212400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:67 Segment 4-147: IOVA:0x3d9400000, len:2097152, virt:0x2212800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:68 Segment 4-149: IOVA:0x3d9c00000, len:2097152, virt:0x2212c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:69 Segment 4-151: IOVA:0x3e2200000, len:2097152, virt:0x2213000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:70 Segment 4-153: IOVA:0x3e5a00000, len:2097152, virt:0x2213400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:71 Segment 4-155: IOVA:0x3e6000000, len:2097152, virt:0x2213800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:72 Segment 4-157: IOVA:0x3e9e00000, len:2097152, virt:0x2213c00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:73 Segment 4-159: IOVA:0x3f0a00000, len:2097152, virt:0x2214000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:74 Segment 4-160: IOVA:0x3f0c00000, len:2097152, virt:0x2214200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:75 Segment 4-162: IOVA:0x3f1c00000, len:2097152, virt:0x2214600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:76 Segment 4-164: IOVA:0x3f2400000, len:2097152, virt:0x2214a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:77 Segment 4-166: IOVA:0x3f2e00000, len:2097152, virt:0x2214e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:78 Segment 4-168: IOVA:0x3f3200000, len:2097152, virt:0x2215200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:79 Segment 4-169: IOVA:0x3f3400000, len:2097152, virt:0x2215400000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:80 Segment 4-171: IOVA:0x3f3a00000, len:2097152, virt:0x2215800000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:81 --------- END_MEMORY_SEGMENTS --------- ------------ MEMORY_ZONES ------------- Zone 0: name:, len:0x35840, virt:0x22159b4740, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 1: name:, len:0x3000, virt:0x22159ac000, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 2: name:, len:0x3000, virt:0x22159a7000, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 3: name:, len:0x9000, virt:0x221599dfc0, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 4: name:, len:0x2000, virt:0x221599b000, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 5: name:, len:0x1000, virt:0x2215999fc0, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 6: name:, len:0x3000, virt:0x2215992000, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 7: name:, len:0x3000, virt:0x221598d000, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 8: name:, len:0x9000, virt:0x2215983fc0, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 9: name:, len:0x2000, virt:0x2215981000, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 10: name:, len:0x1000, virt:0x221597ffc0, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 11: name:, len:0x3000, virt:0x2215978000, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 12: name:, len:0x3000, virt:0x2215973000, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 13: name:, len:0x9000, virt:0x2215969fc0, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 14: name:, len:0x2000, virt:0x2215967000, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 Zone 15: name:, len:0x1000, virt:0x2215965fc0, socket_id:0, flags:0 physical segments used: addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000 ---------- END_MEMORY_ZONES ----------- Could you please suggest is there any rte_mem_lib to align the memory pages ? Thanks, Kamaraj On Tue, Jul 28, 2020 at 3:40 PM Burakov, Anatoly wrote: > On 27-Jul-20 4:30 PM, Kamaraj P wrote: > > Hi Anatoly, > > Since we do not have the driver support of SRIOv with VFIO, we are using > > IGB_UIO . > > I believe it's coming :) > > > Basically our application is crashing due to the buffer allocation > > failure. I believe because it didn't get a contiguous memory location > > and fails to allocate the memory. > > Again, "crashing due to buffer allocation failure" is not very > descriptive. When allocation fails, EAL will produce an error log, so if > your failures are indeed due to memory allocation failures, an EAL log > will tell you if it's actually the case (and enabling debug level > logging will tell you more). > > By default, all memory allocations will *not* be contiguous and > therefore will not fail if the memory is not contiguous. In order for > such an allocation to fail, you actually have to run out of memory. > > If there is indeed a place where you are specifically requesting > contiguous memory, it will be signified by a call to memzone reserve API > with a RTE_MEMZONE_IOVA_CONTIG flag (or a call to > rte_eth_dma_zone_reserve(), if your driver makes use of that API). So if > you're not willing to provide any logs to help with debugging, i would > at least suggest you grep your codebase for the above two things, and > put GDB breakpoints right after the calls to either memzone reserve API > or a ethdev DMA zone reserve API. > > To summarize: a regular allocation *will not fail* if memory is non > contiguous, so you can disregard those. If you find all places where > you're requesting *contiguous* memory (which should be at most one or > two), you'll be in a better position to determine whether this is what's > causing the failures. > > > Is there any API, I can use to dump before our application dies ? > > Please let me know. > > Not sure what you mean by that, but you could use > rte_dump_physmem_layout() function to dump your hugepage layout. That > said, i believe a debugger is, in most cases, a better way to diagnose > the issue. > > > > > Thanks, > > Kamaraj > > > > > > On Mon, Jul 13, 2020 at 2:57 PM Burakov, Anatoly > > > wrote: > > > > On 11-Jul-20 8:51 AM, Kamaraj P wrote: > > > Hello Anatoly/Bruce, > > > > > > We are using the 18_11 version of DPDK and we are using igb_uio. > > > The way we observe an issue here is that, after we tried multiple > > > iterations of start/stop of container application(which has DPDK), > > > we were not able to allocate the memory for port during the init. > > > We thought that it could be an issue of not getting continuous > > > allocation hence it fails. > > > > > > Is there an API where I can check if the memory is fragmented > > before we > > > invoke an allocation ? > > > Or do we have any such mechanism to defragment the memory > allocation > > > once we exist from the application ? > > > Please advise. > > > > > > > This is unlikely due to fragmentation, because the only way for > > 18.11 to > > be affected my memory fragmentation is 1) if you're using legacy mem > > mode, or 2) you're using IOVA as PA mode and you need huge amounts of > > contiguous memory. (you are using igb_uio, so you would be in IOVA > > as PA > > mode) > > > > NICs very rarely, if ever, allocate more than a 2M-page worth of > > contiguous memory, because their descriptor rings aren't that big, > and > > they'll usually get all the IOVA-contiguous space they need even in > the > > face of heavily fragmented memory. Similarly, while 18.11 mempools > will > > request IOVA-contiguous memory first, they have a fallback to using > > non-contiguous memory and thus too work just fine in the face of high > > memory fragmentation. > > > > The nature of the 18.11 memory subsystem is such that IOVA layout is > > decoupled from VA layout, so fragmentation does not affect DPDK as > much > > as it has for previous versions. The first thing i'd suggest is using > > VFIO as opposed to igb_uio, as it's safer to use in a container > > environment, and it's less susceptible to memory fragmentation issues > > because it can remap memory to appear IOVA-contiguous. > > > > Could you please provide detailed logs of the init process? You can > add > > '--log-level=eal,8' to the EAL command-line to enable debug logging > in > > the EAL. > > > > > Thanks, > > > Kamaraj > > > > > > > > > > > > On Fri, Jul 10, 2020 at 9:14 PM Burakov, Anatoly > > > > > > >> wrote: > > > > > > On 10-Jul-20 11:28 AM, Bruce Richardson wrote: > > > > On Fri, Jul 10, 2020 at 02:52:16PM +0530, Kamaraj P wrote: > > > >> Hello All, > > > >> > > > >> We are running to run DPDK based application in a > > container mode, > > > >> When we do multiple start/stop of our container > > application, the > > > DPDK > > > >> initialization seems to be failing. > > > >> This is because the hugepage memory fragementated and is > not > > > able to find > > > >> the continuous allocation of the memory to initialize the > > buffer > > > in the > > > >> dpdk init. > > > >> > > > >> As part of the cleanup of the container, we do call > > > rte_eal_cleanup() to > > > >> cleanup the memory w.r.t our application. However after > > > iterations we still > > > >> see the memory allocation failure due to the > > fragmentation issue. > > > >> > > > >> We also tried to set the "--huge-unlink" as an argument > > before > > > when we > > > >> called the rte_eal_init() and it did not help. > > > >> > > > >> Could you please suggest if there is an option or any > > existing > > > patches > > > >> available to clean up the memory to avoid fragmentation > > issues > > > in the > > > >> future. > > > >> > > > >> Please advise. > > > >> > > > > What version of DPDK are you using, and what kernel driver > > for NIC > > > > interfacing are you using? > > > > DPDK versions since 18.05 should be more forgiving of > > fragmented > > > memory, > > > > especially if using the vfio-pci kernel driver. > > > > > > > > > > This sounds odd, to be honest. > > > > > > Unless you're allocating huge chunks of IOVA-contiguous > memory, > > > fragmentation shouldn't be an issue. How did you determine > > that this > > > was > > > in fact due to fragmentation? > > > > > > > Regards, > > > > /Bruce > > > > > > > > > > > > > -- > > > Thanks, > > > Anatoly > > > > > > > > > -- > > Thanks, > > Anatoly > > > > > -- > Thanks, > Anatoly >