confusion about dpdk asan

confusion about dpdk asan

[杨佳昊]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

Hello 
I am tring to run dpdk example with asan.
I have compile dpdk and example as directed by link https://github.com/DPDK/dpdk/blob/v21.11/doc/guides/prog_guide/asan.rst
But I always get result as follow:
=================================================================
==20078== ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000d28b92 sp 0x7ffc6a983b90 bp 0x7ffc6a983bb0 T0)
AddressSanitizer can not provide additional info.
    #0 0xd28b91 (/root/dpdk-21.11/build/examples/dpdk-helloworld+0xd28b91)
    #1 0xd28fc1 (/root/dpdk-21.11/build/examples/dpdk-helloworld+0xd28fc1)
    #2 0xd2901d (/root/dpdk-21.11/build/examples/dpdk-helloworld+0xd2901d)
    #3 0x56dac3 (/root/dpdk-21.11/build/examples/dpdk-helloworld+0x56dac3)
    #4 0x7ff4b3eed554 (/usr/lib64/libc-2.17.so<http://libc-2.17.so>+0x22554)
    #5 0x56d958 (/root/dpdk-21.11/build/examples/dpdk-helloworld+0x56d958)
==20078== ABORTING

It's not same as what i see in asan.rst<https://github.com/DPDK/dpdk/blob/v21.11/doc/guides/prog_guide/asan.rst>.This result doesn't look right. 
My env:
Gcc :9.3.0
Libasan: 4.8.5
Looking forward to hearing from you

[-- Attachment #2: Type: text/html, Size: 6221 bytes --]

Re: confusion about dpdk asan

[David Marchand]

Hello,

On Thu, Sep 14, 2023 at 9:51 AM 杨佳昊 <yang_jiahao@venusgroup.com.cn> wrote:
>
> Hello
> I am tring to run dpdk example with asan.
> I have compile dpdk and example as directed by link https://github.com/DPDK/dpdk/blob/v21.11/doc/guides/prog_guide/asan.rst
> But I always get result as follow:
> =================================================================
> ==20078== ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000d28b92 sp 0x7ffc6a983b90 bp 0x7ffc6a983bb0 T0)
> AddressSanitizer can not provide additional info.
>     #0 0xd28b91 (/root/dpdk-21.11/build/examples/dpdk-helloworld+0xd28b91)
>     #1 0xd28fc1 (/root/dpdk-21.11/build/examples/dpdk-helloworld+0xd28fc1)
>     #2 0xd2901d (/root/dpdk-21.11/build/examples/dpdk-helloworld+0xd2901d)
>     #3 0x56dac3 (/root/dpdk-21.11/build/examples/dpdk-helloworld+0x56dac3)
>     #4 0x7ff4b3eed554 (/usr/lib64/libc-2.17.so+0x22554)
>     #5 0x56d958 (/root/dpdk-21.11/build/examples/dpdk-helloworld+0x56d958)
> ==20078== ABORTING
>
> It's not same as what i see in asan.rst.This result doesn't look right.
> My env:
> Gcc :9.3.0
> Libasan: 4.8.5

Hard to tell what is wrong here (especially without debug symbols).

Some quick questions for now:
Are you triggering this segfault with ASan disabled (iow: simply
configuring your build env with meson setup <build_dir>)?

Did you change something in DPDK or the example sources?


-- 
David Marchand

Re: confusion about dpdk asan

[David Marchand]

On Thu, Sep 14, 2023 at 10:59 AM 杨佳昊 <yang_jiahao@venusgroup.com.cn> wrote:
>
> Thank you for your reply.

Please, let's keep this discussion on the ml.
Anybody can help and this thread may help others having issues with ASan.
Let's try to understand if something is wrong in your setup or if
there is a bug in the doc or example code.


You mentionned initially those versions:
Gcc :9.3.0
Libasan: 4.8.5

Which distribution is it?
It looks close to a Ubuntu 20.04, but latest 20.04 has a more recent
version of gcc (9.4 with libasan 5.x)


> Sorry for not describing it in detail.
> I compile dpdk with cmd:
> meson setup -Db_sanitize=address build -Dmax_numa_nodes=1 -Dexamples=helloworld

Well, this segfault is strange and I am suspecting an issue out of
DPDK (in libasan itself?).
As I mentionned in my first mail, what happens if you compile without
-Db_sanitize=address and run the example?


If you don't reproduce the segfault without ASan, my next request is
to try to run with some debug traces in ASan:
ASAN_OPTIONS=verbosity=1 ./build/examples/dpdk-helloworld etc...

Alternatively, recompile with debug symbols and run the example
through gdb to get a better backtrace.


> I added code as what asan.txt do:
>
> char *p = rte_zmalloc(NULL, 9, 0);
> if (!p) {
>     printf("rte_zmalloc error.\n");
>     return -1;
> }
> p[9] = 'a';
>
> But the result is exactly as described in the previous email.

I tried adding this block on the 21.11 branch, in a Ubuntu 20.04.6 container.
As I mentionned above, this container has a gcc 9.4 version, coming
with a libasan 5 and I get the expected output from ASan.


-- 
David Marchand

Re: confusion about dpdk asan

[Stephen Hemminger]

On Thu, 14 Sep 2023 10:25:29 +0200
David Marchand <david.marchand@redhat.com> wrote:

> > It's not same as what i see in asan.rst.This result doesn't look right.
> > My env:
> > Gcc :9.3.0
> > Libasan: 4.8.5  

In Debian libasan seems tied to compiler version.
For Debian stable GCC is 12.2 and libasan is 12.2

Re: confusion about dpdk asan

[Stephen Hemminger]

On Thu, 14 Sep 2023 12:42:20 +0200
David Marchand <david.marchand@redhat.com> wrote:

> On Thu, Sep 14, 2023 at 10:59 AM 杨佳昊 <yang_jiahao@venusgroup.com.cn> wrote:
> >
> > Thank you for your reply.  
> 
> Please, let's keep this discussion on the ml.
> Anybody can help and this thread may help others having issues with ASan.
> Let's try to understand if something is wrong in your setup or if
> there is a bug in the doc or example code.
> 
> 
> You mentionned initially those versions:
> Gcc :9.3.0
> Libasan: 4.8.5
> 
> Which distribution is it?
> It looks close to a Ubuntu 20.04, but latest 20.04 has a more recent
> version of gcc (9.4 with libasan 5.x)
> 
> 
> > Sorry for not describing it in detail.
> > I compile dpdk with cmd:
> > meson setup -Db_sanitize=address build -Dmax_numa_nodes=1 -Dexamples=helloworld  
> 
> Well, this segfault is strange and I am suspecting an issue out of
> DPDK (in libasan itself?).
> As I mentionned in my first mail, what happens if you compile without
> -Db_sanitize=address and run the example?
> 
> 
> If you don't reproduce the segfault without ASan, my next request is
> to try to run with some debug traces in ASan:
> ASAN_OPTIONS=verbosity=1 ./build/examples/dpdk-helloworld etc...
> 
> Alternatively, recompile with debug symbols and run the example
> through gdb to get a better backtrace.
> 
> 
> > I added code as what asan.txt do:
> >
> > char *p = rte_zmalloc(NULL, 9, 0);
> > if (!p) {
> >     printf("rte_zmalloc error.\n");
> >     return -1;
> > }
> > p[9] = 'a';
> >
> > But the result is exactly as described in the previous email.  
> 
> I tried adding this block on the 21.11 branch, in a Ubuntu 20.04.6 container.
> As I mentionned above, this container has a gcc 9.4 version, coming
> with a libasan 5 and I get the expected output from ASan.

Tried some code with those flags on Debian stable which has Gcc 12.2.

1. Testpmd fails because it can't setup heap.  Looks like asan won't work
   with hugepages.

AL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: alloc_pages_on_heap(): couldn't allocate memory due to IOVA exceeding limits of current DMA mask
EAL: alloc_pages_on_heap(): Please try initializing EAL with --iova-mode=pa parameter
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
EAL: Error - exiting with code: 1
  Cause: Cannot init EAL: Cannot allocate memory

2. Same for hello world
# ./build/examples/dpdk-helloworld
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: alloc_pages_on_heap(): couldn't allocate memory due to IOVA exceeding limits of current DMA mask
EAL: alloc_pages_on_heap(): Please try initializing EAL with --iova-mode=pa parameter
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
PANIC in main():
Cannot init EAL
0: /lib/x86_64-linux-gnu/libasan.so.8 (7f8616c00000+0x51f31) [7f8616c51f31]
1: ./build/examples/dpdk-helloworld (rte_dump_stack+0xf5) [55a9b8763045]
2: ./build/examples/dpdk-helloworld (__rte_panic+0x11d) [55a9b7c44958]
3: ./build/examples/dpdk-helloworld (55a9b75fb000+0x611a51) [55a9b7c0ca51]
4: /lib/x86_64-linux-gnu/libc.so.6 (7f861561f000+0x271ca) [7f86156461ca]
5: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0x85) [7f8615646285]
6: ./build/examples/dpdk-helloworld (_start+0x21) [55a9b81cffd1]
Aborted