dpdk address sanitizer

dpdk address sanitizer

[Juan Pablo L.]

[-- Attachment #1: Type: text/plain, Size: 3732 bytes --]

Hello, I am new to dpdk ... i would like to trace memory usage and detect memory leaks, valgrind as well as address sanitizer (gcc) report some memory loss at application end. For the life of me, i cannot figure it out ... i just write a simple program that has the rte_eal_init + rte_eal_cleanup and i get the following error (also tried helloworld from examples, with same results):


==3399==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6ca3efb480 at pc 0x7f6ca7162b61 bp 0x7f6ca3efb450 sp 0x7f6ca3efac00

WRITE of size 24 at 0x7f6ca3efb480 thread T-1

#0 0x7f6ca7162b60 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.8+0x61b60)

#1 0x7f6ca71d9337 in __sanitizer::UnsetAlternateSignalStack() (/lib64/libasan.so.8+0xd8337)

#2 0x7f6ca71c90f4 in __asan::AsanThread::Destroy() (/lib64/libasan.so.8+0xc80f4)

#3 0x7f6ca679b000 in __GI___nptl_deallocate_tsd (/lib64/libc.so.6+0x8a000)

#4 0x7f6ca679dc9d in start_thread (/lib64/libc.so.6+0x8cc9d)

#5 0x7f6ca68235df in __GI___clone3 (/lib64/libc.so.6+0x1125df)


Address 0x7f6ca3efb480 is a wild pointer inside of access range of size 0x000000000018.

SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.8+0x61b60) in __interceptor_sigaltstack.part.0

Shadow bytes around the buggy address:

0x0fee147d7640: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00

0x0fee147d7650: 00 00 00 00 00 00 00 00 00 06 f2 f2 f2 f2 00 00

0x0fee147d7660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0x0fee147d7670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0x0fee147d7680: 00 00 00 00 00 00 00 00 00 00 00 04 f3 f3 f3 f3

=>0x0fee147d7690:[f3]f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00

0x0fee147d76a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0x0fee147d76b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0x0fee147d76c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0x0fee147d76d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0x0fee147d76e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Shadow byte legend (one shadow byte represents 8 application bytes):

Addressable: 00

Partially addressable: 01 02 03 04 05 06 07

Heap left redzone: fa

Freed heap region: fd

Stack left redzone: f1

Stack mid redzone: f2

Stack right redzone: f3

Stack after return: f5

Stack use after scope: f8

Global redzone: f9

Global init order: f6

Poisoned by user: f7

Container overflow: fc

Array cookie: ac

Intra object redzone: bb

ASan internal: fe

Left alloca redzone: ca

Right alloca redzone: cb

==3399==ABORTING


I am not sure what I m doing wrong but it is very frustrating. On top of that, I try other scenarios and see if I can just "ignore" that and still detect other memory leaks but it does not work. I get memory from rte_malloc and don't free it and I still get the above report only, I do not get any report from the memory I leaked intentionally ... no difference what so ever .... I tried the same with the helloworld example and I get the same results ....


Please, if anyone can shed some light and point me in any direction ... I have already searched google for 2 days with nothing new ... surely, I m doing something wrong but I believe I have read everything I can find on google related to tracing memory in dpdk and there is very little information about it and mostly dismiss valgrind and favor the address sanitizer ...


specs:

Fedora 36 server edition (KVM vm)

gcc (GCC) 12.1.1 20220507 (Red Hat 12.1.1-1)

dpdk 22.03 (compiled with: -Dbuildtype=debug -Dprefix=/path/to/lib/dpdk-22.03/debug/ -Dtests=false -Denable_kmods=true -Denable_trace_fp=true -Db_sanitize=address)

thank you very much!

[-- Attachment #2: Type: text/html, Size: 6577 bytes --]

Re: dpdk address sanitizer

[David Marchand]

Hello,

On Fri, Jun 17, 2022 at 6:08 AM Juan Pablo L.
<jpablolorenzetti@hotmail.com> wrote:
>
> Hello, I am new to dpdk ... i would like to trace memory usage and detect memory leaks, valgrind as well as address sanitizer (gcc) report some memory loss at application end. For the life of me, i cannot figure it out ... i just write a simple program that has the rte_eal_init + rte_eal_cleanup and i get the following error (also tried helloworld from examples, with same results):
>
> ==3399==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6ca3efb480 at pc 0x7f6ca7162b61 bp 0x7f6ca3efb450 sp 0x7f6ca3efac00
> WRITE of size 24 at 0x7f6ca3efb480 thread T-1
> #0 0x7f6ca7162b60 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.8+0x61b60)
> #1 0x7f6ca71d9337 in __sanitizer::UnsetAlternateSignalStack() (/lib64/libasan.so.8+0xd8337)
> #2 0x7f6ca71c90f4 in __asan::AsanThread::Destroy() (/lib64/libasan.so.8+0xc80f4)
> #3 0x7f6ca679b000 in __GI___nptl_deallocate_tsd (/lib64/libc.so.6+0x8a000)
> #4 0x7f6ca679dc9d in start_thread (/lib64/libc.so.6+0x8cc9d)
> #5 0x7f6ca68235df in __GI___clone3 (/lib64/libc.so.6+0x1125df)
>
> Address 0x7f6ca3efb480 is a wild pointer inside of access range of size 0x000000000018.
> SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.8+0x61b60) in __interceptor_sigaltstack.part.0
> Shadow bytes around the buggy address:
> 0x0fee147d7640: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00
> 0x0fee147d7650: 00 00 00 00 00 00 00 00 00 06 f2 f2 f2 f2 00 00
> 0x0fee147d7660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d7670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d7680: 00 00 00 00 00 00 00 00 00 00 00 04 f3 f3 f3 f3
> =>0x0fee147d7690:[f3]f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> ==3399==ABORTING
>
> I am not sure what I m doing wrong but it is very frustrating. On top of that, I try other scenarios and see if I can just "ignore" that and still detect other memory leaks but it does not work. I get memory from rte_malloc and don't free it and I still get the above report only, I do not get any report from the memory I leaked intentionally ... no difference what so ever .... I tried the same with the helloworld example and I get the same results ....

I experienced the same issue recently on Fedora 36.
I did not investigate.

I think I waived this warning, by setting
ASAN_OPTIONS="use_sigaltstack=0" in the environment.
HTH.

-- 
David Marchand

Re: dpdk address sanitizer

[Stephen Hemminger]

On Fri, 17 Jun 2022 07:45:33 +0200
David Marchand <david.marchand@redhat.com> wrote:

> Hello,
> 
> On Fri, Jun 17, 2022 at 6:08 AM Juan Pablo L.
> <jpablolorenzetti@hotmail.com> wrote:
> >
> > Hello, I am new to dpdk ... i would like to trace memory usage and detect memory leaks, valgrind as well as address sanitizer (gcc) report some memory loss at application end. For the life of me, i cannot figure it out ... i just write a simple program that has the rte_eal_init + rte_eal_cleanup and i get the following error (also tried helloworld from examples, with same results):
> >
> > ==3399==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6ca3efb480 at pc 0x7f6ca7162b61 bp 0x7f6ca3efb450 sp 0x7f6ca3efac00
> > WRITE of size 24 at 0x7f6ca3efb480 thread T-1
> > #0 0x7f6ca7162b60 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.8+0x61b60)
> > #1 0x7f6ca71d9337 in __sanitizer::UnsetAlternateSignalStack() (/lib64/libasan.so.8+0xd8337)
> > #2 0x7f6ca71c90f4 in __asan::AsanThread::Destroy() (/lib64/libasan.so.8+0xc80f4)
> > #3 0x7f6ca679b000 in __GI___nptl_deallocate_tsd (/lib64/libc.so.6+0x8a000)
> > #4 0x7f6ca679dc9d in start_thread (/lib64/libc.so.6+0x8cc9d)
> > #5 0x7f6ca68235df in __GI___clone3 (/lib64/libc.so.6+0x1125df)
> >
> > Address 0x7f6ca3efb480 is a wild pointer inside of access range of size 0x000000000018.
> > SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.8+0x61b60) in __interceptor_sigaltstack.part.0
> > Shadow bytes around the buggy address:
> > 0x0fee147d7640: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00
> > 0x0fee147d7650: 00 00 00 00 00 00 00 00 00 06 f2 f2 f2 f2 00 00
> > 0x0fee147d7660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d7670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d7680: 00 00 00 00 00 00 00 00 00 00 00 04 f3 f3 f3 f3  
> > =>0x0fee147d7690:[f3]f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00  
> > 0x0fee147d76a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d76b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d76c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d76d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d76e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > Shadow byte legend (one shadow byte represents 8 application bytes):
> > Addressable: 00
> > Partially addressable: 01 02 03 04 05 06 07
> > Heap left redzone: fa
> > Freed heap region: fd
> > Stack left redzone: f1
> > Stack mid redzone: f2
> > Stack right redzone: f3
> > Stack after return: f5
> > Stack use after scope: f8
> > Global redzone: f9
> > Global init order: f6
> > Poisoned by user: f7
> > Container overflow: fc
> > Array cookie: ac
> > Intra object redzone: bb
> > ASan internal: fe
> > Left alloca redzone: ca
> > Right alloca redzone: cb
> > ==3399==ABORTING
> >
> > I am not sure what I m doing wrong but it is very frustrating. On top of that, I try other scenarios and see if I can just "ignore" that and still detect other memory leaks but it does not work. I get memory from rte_malloc and don't free it and I still get the above report only, I do not get any report from the memory I leaked intentionally ... no difference what so ever .... I tried the same with the helloworld example and I get the same results ....  
> 
> I experienced the same issue recently on Fedora 36.
> I did not investigate.
> 
> I think I waived this warning, by setting
> ASAN_OPTIONS="use_sigaltstack=0" in the environment.
> HTH.
> 

Looks like the alternate signal stack allocated inside address sanitizer is not big enough.

Re: dpdk address sanitizer

[Juan Pablo L.]

[-- Attachment #1: Type: text/plain, Size: 3859 bytes --]

Thank you very much, I will try that.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: David Marchand <david.marchand@redhat.com>
Sent: Thursday, June 16, 2022 11:45:33 PM
To: Juan Pablo L. <jpablolorenzetti@hotmail.com>
Cc: users@dpdk.org <users@dpdk.org>; Burakov, Anatoly <anatoly.burakov@intel.com>; Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; dev <dev@dpdk.org>; ci@dpdk.org <ci@dpdk.org>
Subject: Re: dpdk address sanitizer

Hello,

On Fri, Jun 17, 2022 at 6:08 AM Juan Pablo L.
<jpablolorenzetti@hotmail.com> wrote:
>
> Hello, I am new to dpdk ... i would like to trace memory usage and detect memory leaks, valgrind as well as address sanitizer (gcc) report some memory loss at application end. For the life of me, i cannot figure it out ... i just write a simple program that has the rte_eal_init + rte_eal_cleanup and i get the following error (also tried helloworld from examples, with same results):
>
> ==3399==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6ca3efb480 at pc 0x7f6ca7162b61 bp 0x7f6ca3efb450 sp 0x7f6ca3efac00
> WRITE of size 24 at 0x7f6ca3efb480 thread T-1
> #0 0x7f6ca7162b60 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.8+0x61b60)
> #1 0x7f6ca71d9337 in __sanitizer::UnsetAlternateSignalStack() (/lib64/libasan.so.8+0xd8337)
> #2 0x7f6ca71c90f4 in __asan::AsanThread::Destroy() (/lib64/libasan.so.8+0xc80f4)
> #3 0x7f6ca679b000 in __GI___nptl_deallocate_tsd (/lib64/libc.so.6+0x8a000)
> #4 0x7f6ca679dc9d in start_thread (/lib64/libc.so.6+0x8cc9d)
> #5 0x7f6ca68235df in __GI___clone3 (/lib64/libc.so.6+0x1125df)
>
> Address 0x7f6ca3efb480 is a wild pointer inside of access range of size 0x000000000018.
> SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.8+0x61b60) in __interceptor_sigaltstack.part.0
> Shadow bytes around the buggy address:
> 0x0fee147d7640: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00
> 0x0fee147d7650: 00 00 00 00 00 00 00 00 00 06 f2 f2 f2 f2 00 00
> 0x0fee147d7660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d7670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d7680: 00 00 00 00 00 00 00 00 00 00 00 04 f3 f3 f3 f3
> =>0x0fee147d7690:[f3]f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0fee147d76e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> ==3399==ABORTING
>
> I am not sure what I m doing wrong but it is very frustrating. On top of that, I try other scenarios and see if I can just "ignore" that and still detect other memory leaks but it does not work. I get memory from rte_malloc and don't free it and I still get the above report only, I do not get any report from the memory I leaked intentionally ... no difference what so ever .... I tried the same with the helloworld example and I get the same results ....

I experienced the same issue recently on Fedora 36.
I did not investigate.

I think I waived this warning, by setting
ASAN_OPTIONS="use_sigaltstack=0" in the environment.
HTH.

--
David Marchand


[-- Attachment #2: Type: text/html, Size: 4994 bytes --]

Re: dpdk address sanitizer

[Juan Pablo L.]

[-- Attachment #1: Type: text/plain, Size: 4256 bytes --]

Thank you for your feedback, i will look into that .. any suggestions on what technique I can use to find memory leaks, invalid accesses, etc etc .... ?

thanks!!!
________________________________
From: Stephen Hemminger <stephen@networkplumber.org>
Sent: Friday, June 17, 2022 4:17 PM
To: David Marchand <david.marchand@redhat.com>
Cc: Juan Pablo L. <jpablolorenzetti@hotmail.com>; users@dpdk.org <users@dpdk.org>; Burakov, Anatoly <anatoly.burakov@intel.com>; Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; dev <dev@dpdk.org>; ci@dpdk.org <ci@dpdk.org>
Subject: Re: dpdk address sanitizer

On Fri, 17 Jun 2022 07:45:33 +0200
David Marchand <david.marchand@redhat.com> wrote:

> Hello,
>
> On Fri, Jun 17, 2022 at 6:08 AM Juan Pablo L.
> <jpablolorenzetti@hotmail.com> wrote:
> >
> > Hello, I am new to dpdk ... i would like to trace memory usage and detect memory leaks, valgrind as well as address sanitizer (gcc) report some memory loss at application end. For the life of me, i cannot figure it out ... i just write a simple program that has the rte_eal_init + rte_eal_cleanup and i get the following error (also tried helloworld from examples, with same results):
> >
> > ==3399==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6ca3efb480 at pc 0x7f6ca7162b61 bp 0x7f6ca3efb450 sp 0x7f6ca3efac00
> > WRITE of size 24 at 0x7f6ca3efb480 thread T-1
> > #0 0x7f6ca7162b60 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.8+0x61b60)
> > #1 0x7f6ca71d9337 in __sanitizer::UnsetAlternateSignalStack() (/lib64/libasan.so.8+0xd8337)
> > #2 0x7f6ca71c90f4 in __asan::AsanThread::Destroy() (/lib64/libasan.so.8+0xc80f4)
> > #3 0x7f6ca679b000 in __GI___nptl_deallocate_tsd (/lib64/libc.so.6+0x8a000)
> > #4 0x7f6ca679dc9d in start_thread (/lib64/libc.so.6+0x8cc9d)
> > #5 0x7f6ca68235df in __GI___clone3 (/lib64/libc.so.6+0x1125df)
> >
> > Address 0x7f6ca3efb480 is a wild pointer inside of access range of size 0x000000000018.
> > SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.8+0x61b60) in __interceptor_sigaltstack.part.0
> > Shadow bytes around the buggy address:
> > 0x0fee147d7640: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00
> > 0x0fee147d7650: 00 00 00 00 00 00 00 00 00 06 f2 f2 f2 f2 00 00
> > 0x0fee147d7660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d7670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d7680: 00 00 00 00 00 00 00 00 00 00 00 04 f3 f3 f3 f3
> > =>0x0fee147d7690:[f3]f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d76a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d76b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d76c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d76d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0fee147d76e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > Shadow byte legend (one shadow byte represents 8 application bytes):
> > Addressable: 00
> > Partially addressable: 01 02 03 04 05 06 07
> > Heap left redzone: fa
> > Freed heap region: fd
> > Stack left redzone: f1
> > Stack mid redzone: f2
> > Stack right redzone: f3
> > Stack after return: f5
> > Stack use after scope: f8
> > Global redzone: f9
> > Global init order: f6
> > Poisoned by user: f7
> > Container overflow: fc
> > Array cookie: ac
> > Intra object redzone: bb
> > ASan internal: fe
> > Left alloca redzone: ca
> > Right alloca redzone: cb
> > ==3399==ABORTING
> >
> > I am not sure what I m doing wrong but it is very frustrating. On top of that, I try other scenarios and see if I can just "ignore" that and still detect other memory leaks but it does not work. I get memory from rte_malloc and don't free it and I still get the above report only, I do not get any report from the memory I leaked intentionally ... no difference what so ever .... I tried the same with the helloworld example and I get the same results ....
>
> I experienced the same issue recently on Fedora 36.
> I did not investigate.
>
> I think I waived this warning, by setting
> ASAN_OPTIONS="use_sigaltstack=0" in the environment.
> HTH.
>

Looks like the alternate signal stack allocated inside address sanitizer is not big enough.

[-- Attachment #2: Type: text/html, Size: 5948 bytes --]