DPDK patches and discussions
 help / color / mirror / Atom feed
From: Kamaraj P <pkamaraj@gmail.com>
To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
Cc: Bruce Richardson <bruce.richardson@intel.com>, dev@dpdk.org
Subject: Re: [dpdk-dev] DPDK hugepage memory fragmentation
Date: Wed, 16 Sep 2020 10:02:18 +0530
Message-ID: <CAG8PAargDvknkbvTxfW4NsSoufJ5q4rGyjYK++bOH9KOW9=3zA@mail.gmail.com> (raw)
In-Reply-To: <ce80fe59-04d4-b848-0fd8-c40f68c784ff@intel.com>

Hi Anatoly,

We just dump the memory contents when it fails to allocate the memory.

rte_malloc_dump_stats();
rte_dump_physmem_layout();
rte_memzone_dump();

We could see the fragmented memory

----------- MALLOC_STATS -----------
Heap id:0
Heap name:socket_0
Heap_size:134217728,
Free_size:133604096,
Alloc_size:613632,
Greatest_free_size:8388608,
Alloc_count:48,
Free_count:53,
Heap id:1
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:2
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:3
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:4
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:5
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:6
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:7
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:8
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:9
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:10
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:11
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:12
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:13
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:14
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:15
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:16
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:17
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:18
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:19
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:20
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:21
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:22
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:23
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:24
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:25
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:26
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:27
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:28
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:29
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:30
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
Heap id:31
Heap name:
Heap_size:0,
Free_size:0,
Alloc_size:0,
Greatest_free_size:0,
Alloc_count:0,
Free_count:0,
--------- END_MALLOC_STATS ---------
----------- MEMORY_SEGMENTS -----------
Segment 4-0: IOVA:0x2a5e00000, len:2097152, virt:0x2200200000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:15
Segment 4-1: IOVA:0x2a6000000, len:2097152, virt:0x2200400000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:19
Segment 4-2: IOVA:0x2a6200000, len:2097152, virt:0x2200600000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:20
Segment 4-4: IOVA:0x2a6600000, len:2097152, virt:0x2200a00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:21
Segment 4-6: IOVA:0x2a7000000, len:2097152, virt:0x2200e00000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:22
Segment 4-8: IOVA:0x2a7600000, len:2097152, virt:0x2201200000, socket_id:0,
hugepage_sz:2097152, nchannel:0, nrank:0 fd:23
Segment 4-10: IOVA:0x2a8000000, len:2097152, virt:0x2201600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:24
Segment 4-11: IOVA:0x2a8200000, len:2097152, virt:0x2201800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:25
Segment 4-12: IOVA:0x2a8400000, len:2097152, virt:0x2201a00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:26
Segment 4-13: IOVA:0x2a8600000, len:2097152, virt:0x2201c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:27
Segment 4-15: IOVA:0x2a8c00000, len:2097152, virt:0x2202000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:28
Segment 4-16: IOVA:0x2a8e00000, len:2097152, virt:0x2202200000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:29
Segment 4-18: IOVA:0x2a9400000, len:2097152, virt:0x2202600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:30
Segment 4-20: IOVA:0x2aa400000, len:2097152, virt:0x2202a00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:31
Segment 4-21: IOVA:0x2aa600000, len:2097152, virt:0x2202c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:32
Segment 4-23: IOVA:0x2aac00000, len:2097152, virt:0x2203000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:33
Segment 4-25: IOVA:0x2abc00000, len:2097152, virt:0x2203400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:34
Segment 4-26: IOVA:0x2abe00000, len:2097152, virt:0x2203600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:35
Segment 4-28: IOVA:0x2b0400000, len:2097152, virt:0x2203a00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:36
Segment 4-29: IOVA:0x2b0600000, len:2097152, virt:0x2203c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:37
Segment 4-30: IOVA:0x2b0800000, len:2097152, virt:0x2203e00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:38
Segment 4-31: IOVA:0x2b0a00000, len:2097152, virt:0x2204000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:39
Segment 4-33: IOVA:0x2b1200000, len:2097152, virt:0x2204400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:40
Segment 4-35: IOVA:0x2b2600000, len:2097152, virt:0x2204800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:41
Segment 4-37: IOVA:0x2b2a00000, len:2097152, virt:0x2204c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:42
Segment 4-39: IOVA:0x2b5000000, len:2097152, virt:0x2205000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:43
Segment 4-41: IOVA:0x2b5600000, len:2097152, virt:0x2205400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:44
Segment 4-43: IOVA:0x2b6800000, len:2097152, virt:0x2205800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:45
Segment 4-45: IOVA:0x2b7c00000, len:2097152, virt:0x2205c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:46
Segment 4-47: IOVA:0x2b8e00000, len:2097152, virt:0x2206000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:47
Segment 4-49: IOVA:0x2b9a00000, len:2097152, virt:0x2206400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:48
Segment 4-50: IOVA:0x2b9c00000, len:2097152, virt:0x2206600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:49
Segment 4-51: IOVA:0x2b9e00000, len:2097152, virt:0x2206800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:50
Segment 4-53: IOVA:0x2ba400000, len:2097152, virt:0x2206c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:51
Segment 4-55: IOVA:0x2bbe00000, len:2097152, virt:0x2207000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:52
Segment 4-57: IOVA:0x2bc200000, len:2097152, virt:0x2207400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:53
Segment 4-59: IOVA:0x2c1c00000, len:2097152, virt:0x2207800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:54
Segment 4-60: IOVA:0x2c1e00000, len:2097152, virt:0x2207a00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:55
Segment 4-62: IOVA:0x2d1a00000, len:2097152, virt:0x2207e00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:56
Segment 4-64: IOVA:0x2d5000000, len:2097152, virt:0x2208200000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:57
Segment 4-130: IOVA:0x2d7000000, len:2097152, virt:0x2210600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:58
Segment 4-132: IOVA:0x2d8000000, len:2097152, virt:0x2210a00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:59
Segment 4-134: IOVA:0x2d8400000, len:2097152, virt:0x2210e00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:60
Segment 4-136: IOVA:0x2db400000, len:2097152, virt:0x2211200000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:61
Segment 4-138: IOVA:0x2dc600000, len:2097152, virt:0x2211600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:62
Segment 4-139: IOVA:0x2dc800000, len:2097152, virt:0x2211800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:63
Segment 4-140: IOVA:0x2dca00000, len:2097152, virt:0x2211a00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:64
Segment 4-142: IOVA:0x2de800000, len:2097152, virt:0x2211e00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:65
Segment 4-143: IOVA:0x2dea00000, len:2097152, virt:0x2212000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:66
Segment 4-145: IOVA:0x3d8c00000, len:2097152, virt:0x2212400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:67
Segment 4-147: IOVA:0x3d9400000, len:2097152, virt:0x2212800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:68
Segment 4-149: IOVA:0x3d9c00000, len:2097152, virt:0x2212c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:69
Segment 4-151: IOVA:0x3e2200000, len:2097152, virt:0x2213000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:70
Segment 4-153: IOVA:0x3e5a00000, len:2097152, virt:0x2213400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:71
Segment 4-155: IOVA:0x3e6000000, len:2097152, virt:0x2213800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:72
Segment 4-157: IOVA:0x3e9e00000, len:2097152, virt:0x2213c00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:73
Segment 4-159: IOVA:0x3f0a00000, len:2097152, virt:0x2214000000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:74
Segment 4-160: IOVA:0x3f0c00000, len:2097152, virt:0x2214200000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:75
Segment 4-162: IOVA:0x3f1c00000, len:2097152, virt:0x2214600000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:76
Segment 4-164: IOVA:0x3f2400000, len:2097152, virt:0x2214a00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:77
Segment 4-166: IOVA:0x3f2e00000, len:2097152, virt:0x2214e00000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:78
Segment 4-168: IOVA:0x3f3200000, len:2097152, virt:0x2215200000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:79
Segment 4-169: IOVA:0x3f3400000, len:2097152, virt:0x2215400000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:80
Segment 4-171: IOVA:0x3f3a00000, len:2097152, virt:0x2215800000,
socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 fd:81
--------- END_MEMORY_SEGMENTS ---------
------------ MEMORY_ZONES -------------
Zone 0: name:<rte_eth_dev_data>, len:0x35840, virt:0x22159b4740,
socket_id:0, flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 1: name:<port0_vq0>, len:0x3000, virt:0x22159ac000, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 2: name:<port0_vq1>, len:0x3000, virt:0x22159a7000, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 3: name:<port0_vq1_hdr>, len:0x9000, virt:0x221599dfc0, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 4: name:<port0_vq2>, len:0x2000, virt:0x221599b000, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 5: name:<port0_vq2_hdr>, len:0x1000, virt:0x2215999fc0, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 6: name:<port1_vq0>, len:0x3000, virt:0x2215992000, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 7: name:<port1_vq1>, len:0x3000, virt:0x221598d000, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 8: name:<port1_vq1_hdr>, len:0x9000, virt:0x2215983fc0, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 9: name:<port1_vq2>, len:0x2000, virt:0x2215981000, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 10: name:<port1_vq2_hdr>, len:0x1000, virt:0x221597ffc0, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 11: name:<port2_vq0>, len:0x3000, virt:0x2215978000, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 12: name:<port2_vq1>, len:0x3000, virt:0x2215973000, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 13: name:<port2_vq1_hdr>, len:0x9000, virt:0x2215969fc0, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 14: name:<port2_vq2>, len:0x2000, virt:0x2215967000, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
Zone 15: name:<port2_vq2_hdr>, len:0x1000, virt:0x2215965fc0, socket_id:0,
flags:0
physical segments used:
  addr: 0x2215800000 iova: 0x3f3a00000 len: 0x200000 pagesz: 0x200000
---------- END_MEMORY_ZONES -----------

Could you please suggest is there any rte_mem_lib to align the memory pages
?

Thanks,
Kamaraj

On Tue, Jul 28, 2020 at 3:40 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 27-Jul-20 4:30 PM, Kamaraj P wrote:
> > Hi Anatoly,
> > Since we do not have the driver support of SRIOv with VFIO, we are using
> > IGB_UIO .
>
> I believe it's coming :)
>
> > Basically our application is crashing due to the buffer allocation
> > failure. I believe because it didn't get a contiguous memory location
> > and fails to allocate the memory.
>
> Again, "crashing due to buffer allocation failure" is not very
> descriptive. When allocation fails, EAL will produce an error log, so if
> your failures are indeed due to memory allocation failures, an EAL log
> will tell you if it's actually the case (and enabling debug level
> logging will tell you more).
>
> By default, all memory allocations will *not* be contiguous and
> therefore will not fail if the memory is not contiguous. In order for
> such an allocation to fail, you actually have to run out of memory.
>
> If there is indeed a place where you are specifically requesting
> contiguous memory, it will be signified by a call to memzone reserve API
> with a RTE_MEMZONE_IOVA_CONTIG flag (or a call to
> rte_eth_dma_zone_reserve(), if your driver makes use of that API). So if
> you're not willing to provide any logs to help with debugging, i would
> at least suggest you grep your codebase for the above two things, and
> put GDB breakpoints right after the calls to either memzone reserve API
> or a ethdev DMA zone reserve API.
>
> To summarize: a regular allocation *will not fail* if memory is non
> contiguous, so you can disregard those. If you find all places where
> you're requesting *contiguous* memory (which should be at most one or
> two), you'll be in a better position to determine whether this is what's
> causing the failures.
>
> > Is there any API, I can use to dump before our application dies ?
> > Please let me know.
>
> Not sure what you mean by that, but you could use
> rte_dump_physmem_layout() function to dump your hugepage layout. That
> said, i believe a debugger is, in most cases, a better way to diagnose
> the issue.
>
> >
> > Thanks,
> > Kamaraj
> >
> >
> > On Mon, Jul 13, 2020 at 2:57 PM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> >
> >     On 11-Jul-20 8:51 AM, Kamaraj P wrote:
> >      > Hello Anatoly/Bruce,
> >      >
> >      > We are using the 18_11 version of DPDK and we are using igb_uio.
> >      > The way we observe an issue here is that, after we tried multiple
> >      > iterations of start/stop of container application(which has DPDK),
> >      > we were not able to allocate the memory for port during the init.
> >      > We thought that it could be an issue of not getting continuous
> >      > allocation hence it fails.
> >      >
> >      > Is there an API where I can check if the memory is fragmented
> >     before we
> >      > invoke an allocation ?
> >      > Or do we have any such mechanism to defragment the memory
> allocation
> >      > once we exist from the application ?
> >      > Please advise.
> >      >
> >
> >     This is unlikely due to fragmentation, because the only way for
> >     18.11 to
> >     be affected my memory fragmentation is 1) if you're using legacy mem
> >     mode, or 2) you're using IOVA as PA mode and you need huge amounts of
> >     contiguous memory. (you are using igb_uio, so you would be in IOVA
> >     as PA
> >     mode)
> >
> >     NICs very rarely, if ever, allocate more than a 2M-page worth of
> >     contiguous memory, because their descriptor rings aren't that big,
> and
> >     they'll usually get all the IOVA-contiguous space they need even in
> the
> >     face of heavily fragmented memory. Similarly, while 18.11 mempools
> will
> >     request IOVA-contiguous memory first, they have a fallback to using
> >     non-contiguous memory and thus too work just fine in the face of high
> >     memory fragmentation.
> >
> >     The nature of the 18.11 memory subsystem is such that IOVA layout is
> >     decoupled from VA layout, so fragmentation does not affect DPDK as
> much
> >     as it has for previous versions. The first thing i'd suggest is using
> >     VFIO as opposed to igb_uio, as it's safer to use in a container
> >     environment, and it's less susceptible to memory fragmentation issues
> >     because it can remap memory to appear IOVA-contiguous.
> >
> >     Could you please provide detailed logs of the init process? You can
> add
> >     '--log-level=eal,8' to the EAL command-line to enable debug logging
> in
> >     the EAL.
> >
> >      > Thanks,
> >      > Kamaraj
> >      >
> >      >
> >      >
> >      > On Fri, Jul 10, 2020 at 9:14 PM Burakov, Anatoly
> >      > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
> >     <mailto:anatoly.burakov@intel.com
> >     <mailto:anatoly.burakov@intel.com>>> wrote:
> >      >
> >      >     On 10-Jul-20 11:28 AM, Bruce Richardson wrote:
> >      >      > On Fri, Jul 10, 2020 at 02:52:16PM +0530, Kamaraj P wrote:
> >      >      >> Hello All,
> >      >      >>
> >      >      >> We are running to run DPDK based application in a
> >     container mode,
> >      >      >> When we do multiple start/stop of our container
> >     application, the
> >      >     DPDK
> >      >      >> initialization seems to be failing.
> >      >      >> This is because the hugepage memory fragementated and is
> not
> >      >     able to find
> >      >      >> the continuous allocation of the memory to initialize the
> >     buffer
> >      >     in the
> >      >      >> dpdk init.
> >      >      >>
> >      >      >> As part of the cleanup of the container, we do call
> >      >     rte_eal_cleanup() to
> >      >      >> cleanup the memory w.r.t our application. However after
> >      >     iterations we still
> >      >      >> see the memory allocation failure due to the
> >     fragmentation issue.
> >      >      >>
> >      >      >> We also tried to set the "--huge-unlink" as an argument
> >     before
> >      >     when we
> >      >      >> called the rte_eal_init() and it did not help.
> >      >      >>
> >      >      >> Could you please suggest if there is an option or any
> >     existing
> >      >     patches
> >      >      >> available to clean up the memory to avoid fragmentation
> >     issues
> >      >     in the
> >      >      >> future.
> >      >      >>
> >      >      >> Please advise.
> >      >      >>
> >      >      > What version of DPDK are you using, and what kernel driver
> >     for NIC
> >      >      > interfacing are you using?
> >      >      > DPDK versions since 18.05 should be more forgiving of
> >     fragmented
> >      >     memory,
> >      >      > especially if using the vfio-pci kernel driver.
> >      >      >
> >      >
> >      >     This sounds odd, to be honest.
> >      >
> >      >     Unless you're allocating huge chunks of IOVA-contiguous
> memory,
> >      >     fragmentation shouldn't be an issue. How did you determine
> >     that this
> >      >     was
> >      >     in fact due to fragmentation?
> >      >
> >      >      > Regards,
> >      >      > /Bruce
> >      >      >
> >      >
> >      >
> >      >     --
> >      >     Thanks,
> >      >     Anatoly
> >      >
> >
> >
> >     --
> >     Thanks,
> >     Anatoly
> >
>
>
> --
> Thanks,
> Anatoly
>

  reply	other threads:[~2020-09-16  4:33 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-10  9:22 Kamaraj P
2020-07-10 10:28 ` Bruce Richardson
2020-07-10 15:44   ` Burakov, Anatoly
2020-07-11  7:51     ` Kamaraj P
2020-07-13  9:27       ` Burakov, Anatoly
2020-07-27 15:30         ` Kamaraj P
2020-07-28 10:10           ` Burakov, Anatoly
2020-09-16  4:32             ` Kamaraj P [this message]
2020-09-16 11:19               ` Burakov, Anatoly
2020-09-16 11:47                 ` Burakov, Anatoly

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAG8PAargDvknkbvTxfW4NsSoufJ5q4rGyjYK++bOH9KOW9=3zA@mail.gmail.com' \
    --to=pkamaraj@gmail.com \
    --cc=anatoly.burakov@intel.com \
    --cc=bruce.richardson@intel.com \
    --cc=dev@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git