[dpdk-dev] Valgrind and DPDK - does it work ?

[dpdk-dev] Valgrind and DPDK - does it work ?

[Morten Jagd Christensen]

Hi all,

 

I am interested to hear if anyone here have had any luck running Valgrind on
DPDK applications?

 

I tried to use this some time ago, but Valgrind does not seem to work
together with hugepages – or 

I am doing something wrong ;-)

 

I would like to hear form anyone having success or failure with Valgrind and
DPDK.

 

Best regards

 

/Morten

 

This is an extract of a printout from running my application normally

>sudo ./build/l47app –c 0xffffffff –n 4

EAL: Detected a total of 32 lcores

EAL: Setting up memory...

EAL: Ask a virtual area of 0x280000000 bytes

EAL: Virtual area found at 0x7f7b80000000 (size = 0x280000000)

EAL: Ask a virtual area of 0x280000000 bytes

EAL: Virtual area found at 0x7f78c0000000 (size = 0x280000000)

EAL: Requesting 10 pages of size 1024MB from socket 0

EAL: Requesting 10 pages of size 1024MB from socket 1

 

This is what I get when running Valgrind

>sudo valgrind --leak-check=full ./build/l47server

EAL: Detected a total of 32 lcores

EAL: Setting up memory...

EAL: map_all_hugepages(): mmap failed: Invalid argument

EAL: Failed to mmap 1024 MB hugepages

 

And here is an strace of the above command 

rt_sigprocmask(SIG_SETMASK, ~[ILL TRAP BUS FPE KILL SEGV STOP], NULL, 8) = 0

gettid()                                = 29696

read(1028, "C", 1)                      = 1

close(6)                                = 0

gettid()                                = 29696

write(1029, "D", 1)                     = 1

rt_sigprocmask(SIG_SETMASK, [], ~[ILL TRAP BUS FPE KILL SEGV STOP], 8) = 0

write(1, "EAL: Setting up memory...\n", 26EAL: Setting up memory...

) = 26

rt_sigprocmask(SIG_SETMASK, ~[ILL TRAP BUS FPE KILL SEGV STOP], NULL, 8) = 0

gettid()                                = 29696

read(1028, "D", 1)                      = 1

mmap(0x805b51000, 16384, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = 0x805b51000

getpid()                                = 29696

getpid()                                = 29696

gettid()                                = 29696

write(1029, "E", 1)                     = 1

rt_sigprocmask(SIG_SETMASK, [], ~[ILL TRAP BUS FPE KILL SEGV STOP], 8) = 0

open("/dev/hugepages/rtemap_0", O_RDWR|O_CREAT, 0755) = 6

rt_sigprocmask(SIG_SETMASK, ~[ILL TRAP BUS FPE KILL SEGV STOP], NULL, 8) = 0

gettid()                                = 29696

read(1028, "E", 1)                      = 1

mmap(0x39fe0000, 1073741824, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, 6,
0) = -1 EINVAL (Invalid argument)

gettid()                                = 29696

write(1029, "F", 1)                     = 1

rt_sigprocmask(SIG_SETMASK, [], ~[ILL TRAP BUS FPE KILL SEGV STOP], 8) = 0

write(1, "EAL: map_all_hugepages(): mmap f"..., 56EAL: map_all_hugepages():
mmap failed: Invalid argument

) = 56

rt_sigprocmask(SIG_SETMASK, ~[ILL TRAP BUS FPE KILL SEGV STOP], NULL, 8) = 0

gettid()                                = 29696

read(1028, "F", 1)                      = 1

close(6)                                = 0

gettid()                                = 29696

write(1029, "G", 1)                     = 1

rt_sigprocmask(SIG_SETMASK, [], ~[ILL TRAP BUS FPE KILL SEGV STOP], 8) = 0

write(1, "EAL: Failed to mmap 1024 MB huge"..., 38EAL: Failed to mmap 1024
MB hugepages

 

 

Morten Jagd Christensen
VP Product Management
Xena Networks
Læderstræde 34, 3rd Floor
1201 Copenhagen K
Denmark

Ph: (+45) 51 54 51 87
 <mailto:mjc@xenanetworks.com> mjc@xenanetworks.com

 

Re: [dpdk-dev] Valgrind and DPDK - does it work ?

[Marc Sune]

On 16/09/14 13:42, Morten Jagd Christensen wrote:
> Hi all,
>
>   
>
> I am interested to hear if anyone here have had any luck running Valgrind on
> DPDK applications?
>
>   
>
> I tried to use this some time ago, but Valgrind does not seem to work
> together with hugepages – or
>
> I am doing something wrong ;-)
>
>   
>
> I would like to hear form anyone having success or failure with Valgrind and
> DPDK.

Found some time to have a close look. I also wanted to check a DPDK app 
against valgrind. It works!

I downloaded and compiled valgrind from sources (3.10.0) and applied 
(manually) this patch:

https://bugs.kde.org/attachment.cgi?id=85950&action=edit

(Applied around line 2216)

 From this post:

http://valgrind.10908.n7.nabble.com/mpich-unable-to-munmap-hugepages-td49150.html

Happy debugging
Marc

Re: [dpdk-dev] Valgrind and DPDK - does it work ?

[Marc Sune]

On 07/11/14 01:22, Marc Sune wrote:
> On 16/09/14 13:42, Morten Jagd Christensen wrote:
>> Hi all,
>>
>>
>> I am interested to hear if anyone here have had any luck running 
>> Valgrind on
>> DPDK applications?
>>
>>
>> I tried to use this some time ago, but Valgrind does not seem to work
>> together with hugepages – or
>>
>> I am doing something wrong ;-)
>>
>>
>> I would like to hear form anyone having success or failure with 
>> Valgrind and
>> DPDK.
>
> Found some time to have a close look. I also wanted to check a DPDK 
> app against valgrind. It works!
>
> I downloaded and compiled valgrind from sources (3.10.0) and applied 
> (manually) this patch:
>
> https://bugs.kde.org/attachment.cgi?id=85950&action=edit
>
> (Applied around line 2216)
>
> From this post:
>
> http://valgrind.10908.n7.nabble.com/mpich-unable-to-munmap-hugepages-td49150.html 
>
>
> Happy debugging
> Marc
>
Btw, I will debug closer my DPDK application, but a quick start and stop 
shows already some warnings:

==27534== Thread 2:
==27534== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==27534==    at 0x65E6DBA: epoll_ctl (syscall-template.S:82)
==27534==    by 0x58DAEC: eal_intr_thread_main (in 
/home/marc/xdpd/build/src/xdpd/xdpd)
==27534==    by 0x5DD7B4F: start_thread (pthread_create.c:304)
==27534==    by 0x65E67BC: clone (clone.S:112)
==27534==  Address 0x7717d78 is on thread 2's stack
==27534==  in frame #1, created by eal_intr_thread_main (???)
==27534==

Regards
Marc

Re: [dpdk-dev] Valgrind and DPDK - does it work ?

[Matthew Hall]

On Fri, Nov 07, 2014 at 01:22:49AM +0100, Marc Sune wrote:
> Found some time to have a close look. I also wanted to check a DPDK app
> against valgrind. It works!
> 
> I downloaded and compiled valgrind from sources (3.10.0) and applied
> (manually) this patch:
> 
> https://bugs.kde.org/attachment.cgi?id=85950&action=edit
> 
> (Applied around line 2216)
> 
> From this post:
> 
> http://valgrind.10908.n7.nabble.com/mpich-unable-to-munmap-hugepages-td49150.html
> 
> Happy debugging
> Marc

Marc,

This is just AMAZING!!! I have wished for it for many years for DPDK, ever 
since I used it in beta before it went GA.

Would it be possible to post your modification of Valgrind in Github, 
Bitbucket, or some other repo? I'd like to try this out on my app, too.

Also, not sure if anybody sent this upstream to Valgrind, but if not, we 
really should, so it just works by default from now on.

Thanks,
Matthew.

Re: [dpdk-dev] Valgrind and DPDK - does it work ?

[Marc Sune]

On 07/11/14 02:04, Matthew Hall wrote:
> On Fri, Nov 07, 2014 at 01:22:49AM +0100, Marc Sune wrote:
>> Found some time to have a close look. I also wanted to check a DPDK app
>> against valgrind. It works!
>>
>> I downloaded and compiled valgrind from sources (3.10.0) and applied
>> (manually) this patch:
>>
>> https://bugs.kde.org/attachment.cgi?id=85950&action=edit
>>
>> (Applied around line 2216)
>>
>>  From this post:
>>
>> http://valgrind.10908.n7.nabble.com/mpich-unable-to-munmap-hugepages-td49150.html
>>
>> Happy debugging
>> Marc
> Marc,
>
> This is just AMAZING!!! I have wished for it for many years for DPDK, ever
> since I used it in beta before it went GA.
Thanks

Most kudos though (if not all), should go to T. Janjusic. I just found 
it and backported to the latest stable:

https://bugs.kde.org/show_bug.cgi?id=333051

>
> Would it be possible to post your modification of Valgrind in Github,
> Bitbucket, or some other repo? I'd like to try this out on my app, too.

Sure, it is a good idea.

I just did a quick&dirty import of the latest stable release 3.10.0. No 
git-svn , or anything fancy (github import failed, silently ignoring 
files...)  no gitignore too. And then I applied the patch:

https://github.com/bisdn/valgrind-hugepages

Please note the comment in README.md, I cannot provide any support or 
take any responsibility. We will also delete this repo once there is a 
stable release.

But you can at least try it, should work ;)

>
> Also, not sure if anybody sent this upstream to Valgrind, but if not, we
> really should, so it just works by default from now on.
It already is, there is simply no stable release yet (see Phillipe comment):

https://bugs.kde.org/show_bug.cgi?id=333051#c8

Cheers
Marc
>
> Thanks,
> Matthew.