[dpdk-dev] struct malloc_elem overrun/corruption

[dpdk-dev] struct malloc_elem overrun/corruption

[He Huang]

Hi,

I’ve been troubleshooting a possible memory allocator corruption:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffefdf0700 (LWP 1079)]
0x00000000004794ee in malloc_elem_free_list_insert (elem=0x7ff82d265000) at dpdk/lib/librte_eal/common/malloc_elem.c:292
292             LIST_INSERT_HEAD(&elem->heap->free_head[idx], elem, free_list);
(gdb) bt
#0  0x00000000004794ee in malloc_elem_free_list_insert (elem=0x7ff82d265000) at dpdk/lib/librte_eal/common/malloc_elem.c:292
#1  0x0000000000479971 in malloc_elem_free (elem=0x7ff82d265000) at dpdk/lib/librte_eal/common/malloc_elem.c:448
#2  0x000000000047b054 in malloc_heap_free (elem=0x7ff82d265fc0) at dpdk/lib/librte_eal/common/malloc_heap.c:628
#3  0x00000000004787f5 in rte_free (addr=0x7ff82d266000) at dpdk/lib/librte_eal/common/rte_malloc.c:32

Looked like the 1st field of struct malloc_elem (i.e. the heap pointer: struct malloc_heap *heap) was corrupted. Everything else looked good:
(gdb) p *elem
$2 = {
  heap = 0x9e0,
  prev = 0x7ff82d254fc0,
  next = 0x7ff84ce9a000,
  free_list = {
    le_next = 0x7ff873c89000,
    le_prev = 0x7ff82bcbf018
  },
  msl = 0x7ffff7f3d07c,
  state = ELEM_FREE,
  pad = 0,
  size = 532893696
}
(gdb) p *elem->prev
$3 = {
  heap = 0x7ffff7f3f67c,
  prev = 0x7ff82ce14000,
  next = 0x7ff82d265000,
  free_list = {
   le_next = 0x0,
    le_prev = 0x0
  },
  msl = 0x7ffff7f3d07c,
  state = ELEM_BUSY,
  pad = 0,
  size = 65600
}

I haven’t completely ruled out my own code had a buffer overrun and corrupted the first field of malloc_elem object yet, but I’m beginning to look at it as a possible DPDK internal corruption. The DPDK code isn’t the latest but it had malloc fixes up to commit 9554dbb50a8a22942128a0e5bcb52243a4f723ab.

Ideas/suggestions greatly appreciated! BTW it’s DMA memory so I couldn’t just use malloc/free and debug with standard memory debuggers.

Thanks,
Isaac

Re: [dpdk-dev] struct malloc_elem overrun/corruption

[Burakov, Anatoly]

On 26-Nov-18 11:44 PM, He Huang wrote:
> Hi,
> 
> I’ve been troubleshooting a possible memory allocator corruption:
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7fffefdf0700 (LWP 1079)]
> 0x00000000004794ee in malloc_elem_free_list_insert (elem=0x7ff82d265000) at dpdk/lib/librte_eal/common/malloc_elem.c:292
> 292             LIST_INSERT_HEAD(&elem->heap->free_head[idx], elem, free_list);
> (gdb) bt
> #0  0x00000000004794ee in malloc_elem_free_list_insert (elem=0x7ff82d265000) at dpdk/lib/librte_eal/common/malloc_elem.c:292
> #1  0x0000000000479971 in malloc_elem_free (elem=0x7ff82d265000) at dpdk/lib/librte_eal/common/malloc_elem.c:448
> #2  0x000000000047b054 in malloc_heap_free (elem=0x7ff82d265fc0) at dpdk/lib/librte_eal/common/malloc_heap.c:628
> #3  0x00000000004787f5 in rte_free (addr=0x7ff82d266000) at dpdk/lib/librte_eal/common/rte_malloc.c:32
> 
> Looked like the 1st field of struct malloc_elem (i.e. the heap pointer: struct malloc_heap *heap) was corrupted. Everything else looked good:
> (gdb) p *elem
> $2 = {
>    heap = 0x9e0,
>    prev = 0x7ff82d254fc0,
>    next = 0x7ff84ce9a000,
>    free_list = {
>      le_next = 0x7ff873c89000,
>      le_prev = 0x7ff82bcbf018
>    },
>    msl = 0x7ffff7f3d07c,
>    state = ELEM_FREE,
>    pad = 0,
>    size = 532893696
> }
> (gdb) p *elem->prev
> $3 = {
>    heap = 0x7ffff7f3f67c,
>    prev = 0x7ff82ce14000,
>    next = 0x7ff82d265000,
>    free_list = {
>     le_next = 0x0,
>      le_prev = 0x0
>    },
>    msl = 0x7ffff7f3d07c,
>    state = ELEM_BUSY,
>    pad = 0,
>    size = 65600
> }
> 
> I haven’t completely ruled out my own code had a buffer overrun and corrupted the first field of malloc_elem object yet, but I’m beginning to look at it as a possible DPDK internal corruption. The DPDK code isn’t the latest but it had malloc fixes up to commit 9554dbb50a8a22942128a0e5bcb52243a4f723ab.
> 
> Ideas/suggestions greatly appreciated! BTW it’s DMA memory so I couldn’t just use malloc/free and debug with standard memory debuggers.
> 
> Thanks,
> Isaac
> 

Hi Isaac,

You might want to look into enabling malloc debug options.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] struct malloc_elem overrun/corruption

[Burakov, Anatoly]

On 27-Nov-18 10:38 AM, Burakov, Anatoly wrote:
> On 26-Nov-18 11:44 PM, He Huang wrote:
>> Hi,
>>
>> I’ve been troubleshooting a possible memory allocator corruption:
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7fffefdf0700 (LWP 1079)]
>> 0x00000000004794ee in malloc_elem_free_list_insert 
>> (elem=0x7ff82d265000) at dpdk/lib/librte_eal/common/malloc_elem.c:292
>> 292             LIST_INSERT_HEAD(&elem->heap->free_head[idx], elem, 
>> free_list);
>> (gdb) bt
>> #0  0x00000000004794ee in malloc_elem_free_list_insert 
>> (elem=0x7ff82d265000) at dpdk/lib/librte_eal/common/malloc_elem.c:292
>> #1  0x0000000000479971 in malloc_elem_free (elem=0x7ff82d265000) at 
>> dpdk/lib/librte_eal/common/malloc_elem.c:448
>> #2  0x000000000047b054 in malloc_heap_free (elem=0x7ff82d265fc0) at 
>> dpdk/lib/librte_eal/common/malloc_heap.c:628
>> #3  0x00000000004787f5 in rte_free (addr=0x7ff82d266000) at 
>> dpdk/lib/librte_eal/common/rte_malloc.c:32
>>
>> Looked like the 1st field of struct malloc_elem (i.e. the heap 
>> pointer: struct malloc_heap *heap) was corrupted. Everything else 
>> looked good:
>> (gdb) p *elem
>> $2 = {
>>    heap = 0x9e0,
>>    prev = 0x7ff82d254fc0,
>>    next = 0x7ff84ce9a000,
>>    free_list = {
>>      le_next = 0x7ff873c89000,
>>      le_prev = 0x7ff82bcbf018
>>    },
>>    msl = 0x7ffff7f3d07c,
>>    state = ELEM_FREE,
>>    pad = 0,
>>    size = 532893696
>> }
>> (gdb) p *elem->prev
>> $3 = {
>>    heap = 0x7ffff7f3f67c,
>>    prev = 0x7ff82ce14000,
>>    next = 0x7ff82d265000,
>>    free_list = {
>>     le_next = 0x0,
>>      le_prev = 0x0
>>    },
>>    msl = 0x7ffff7f3d07c,
>>    state = ELEM_BUSY,
>>    pad = 0,
>>    size = 65600
>> }
>>
>> I haven’t completely ruled out my own code had a buffer overrun and 
>> corrupted the first field of malloc_elem object yet, but I’m beginning 
>> to look at it as a possible DPDK internal corruption. The DPDK code 
>> isn’t the latest but it had malloc fixes up to commit 
>> 9554dbb50a8a22942128a0e5bcb52243a4f723ab.
>>
>> Ideas/suggestions greatly appreciated! BTW it’s DMA memory so I 
>> couldn’t just use malloc/free and debug with standard memory debuggers.
>>
>> Thanks,
>> Isaac
>>
> 
> Hi Isaac,
> 
> You might want to look into enabling malloc debug options.
> 

Also, it might be useful to apply this commit and check if it still happens:

71aae4b421da9b741d1fb73a190a9facfec555b9

http://patches.dpdk.org/patch/48080/

-- 
Thanks,
Anatoly