[dpdk-dev] Segmentation fault in rte_eal_hugepage_attach

[dpdk-dev] Segmentation fault in rte_eal_hugepage_attach

[Rick LaMont]

My DPDK application works fine when it's the primary process but crashes
whenever --proc-type=secondary. The segmentation fault occurs in this call
to mmap() within rte_eal_hugepage_attach():

        /*
         * fdzero is mmapped to get a contiguous block of virtual
         * addresses of the appropriate memseg size.
         * use mmap to get identical addresses as the primary process.
         */
        base_addr = mmap(mcfg->memseg[s].addr, mcfg->memseg[s].len,
                 PROT_READ, MAP_PRIVATE | MAP_FIXED, fd_zero, 0);

I've confirmed that addr and len match the values in rte_eal_hugepage_init()
of the primary process (1 gigabyte). The target platform is a 32-bit embedded
system running a Yocto distribution. I've confirmed that other applications
such as mp_simple work as both primary and secondary on the same platform.
The problem only occurs with a larger application to which I'm adding DPDK
capabilities.

Any advice on how to troubleshoot this? I've been looking at it for a week
already and am running out of ideas for things to test.

Thanks,


Rick LaMont          | The storm that I thought would blow over
Dot C Software, Inc. | Clouds the light of the love that I found

Re: [dpdk-dev] Segmentation fault in rte_eal_hugepage_attach

[Bruce Richardson]

On Tue, Dec 16, 2014 at 11:12:36PM -0500, Rick LaMont wrote:
> My DPDK application works fine when it's the primary process but crashes
> whenever --proc-type=secondary. The segmentation fault occurs in this call
> to mmap() within rte_eal_hugepage_attach():
> 
>         /*
>          * fdzero is mmapped to get a contiguous block of virtual
>          * addresses of the appropriate memseg size.
>          * use mmap to get identical addresses as the primary process.
>          */
>         base_addr = mmap(mcfg->memseg[s].addr, mcfg->memseg[s].len,
>                  PROT_READ, MAP_PRIVATE | MAP_FIXED, fd_zero, 0);
> 
> I've confirmed that addr and len match the values in rte_eal_hugepage_init()
> of the primary process (1 gigabyte). The target platform is a 32-bit embedded
> system running a Yocto distribution. I've confirmed that other applications
> such as mp_simple work as both primary and secondary on the same platform.
> The problem only occurs with a larger application to which I'm adding DPDK
> capabilities.
> 
> Any advice on how to troubleshoot this? I've been looking at it for a week
> already and am running out of ideas for things to test.
> 
> Thanks,
> 
> 
> Rick LaMont          | The storm that I thought would blow over
> Dot C Software, Inc. | Clouds the light of the love that I found

Hi,

getting multi-process support working on 32-bits is a lot more difficult than
with 64-bit due to the reduced addresss space range available. The most likely
cause of failure in mapping the memory is that the range requested is already
used by another memory mapping in the processes address space. 
Two possible paths to look at this:
1) examine the /proc/<pid>/maps file for the secondary process to see what is
using the address range and if it's something you can do something about.
2) If you do find a free 1G range inside the secondary process address space, you
can try hinting that address to the primary process to see if it can map the
memory there, allowing the secondary process to duplicate. [See "base-virtaddr"
EAL option]

Regards,
/Bruce

Re: [dpdk-dev] Segmentation fault in rte_eal_hugepage_attach

[Rick LaMont]

> The most likely cause of failure in mapping the memory is that the range
> requested is already used by another memory mapping in the processes address
> space. Two possible paths to look at this:
> 1) examine the /proc/<pid>/maps file for the secondary process

Brilliant advice, Bruce! This was indeed the cause. The secondary process
had some shared libraries mapped right in the middle of the viable 1G range.
The most expedient workaround was to link the primary process against those
same shared libraries, as you advised Etai here:

    http://www.dpdk.info/ml/archives/dev/2014-April/001829.html

Perhaps not the most sustainable solution...

Thank you very much,


Rick LaMont          | After forty years of toil
Dot C Software, Inc. | He just up and walked away