[dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread

[dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread

[Mario Gianni]

Hi all, I have a problem since I updated to 1.7.0 version,
I got a multi-process, multi-threaded application,
In my application first I launch a master process, then I launch a secondary process with multiple threads in it
Well, when the number of lcores reserved for the secondary process exceeds a certain number (eg. 4) i got an error in rte_eal_init() on the secondary process when it tries to map PCI memory:
 
EAL: pci_map_resource(): cannot mmap(12, 0x7ffff2e96000, 0x800000, 0x1000): Success (0x7ffff559b000)
EAL: Cannot mmap device resource
EAL: Error - exiting with code: 1
    Cause: Requested device 0000:01:00.0 cannot be used

Can you help me?

Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread

[Bruce Richardson]

On Fri, Oct 24, 2014 at 01:21:08PM +0200, Mario Gianni wrote:
> Hi all, I have a problem since I updated to 1.7.0 version,
> I got a multi-process, multi-threaded application,
> In my application first I launch a master process, then I launch a secondary process with multiple threads in it
> Well, when the number of lcores reserved for the secondary process exceeds a certain number (eg. 4) i got an error in rte_eal_init() on the secondary process when it tries to map PCI memory:
>  
> EAL: pci_map_resource(): cannot mmap(12, 0x7ffff2e96000, 0x800000, 0x1000): Success (0x7ffff559b000)
> EAL: Cannot mmap device resource
> EAL: Error - exiting with code: 1
>     Cause: Requested device 0000:01:00.0 cannot be used
> 
> Can you help me?

This could be because the additional memory/stack space used by the pthreads 
for the cores in the secondary process is overlapping the space used in the 
primary process for hugepage or device memory. You could perhaps try adding 
a few cores to the primary process's coremask (and not using those cores) 
and see if it helps things. 
Alternatively there is a base-virtaddr parameter that can be passed to the 
primary process to try and adjust the starting address for it mapping 
memory. If you look at where it starts mapping memory right now, and then 
try hinting to it to maps the pages at a slightly higher or lower address 
and see if it helps.

/Bruce

Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread

[Mario Gianni]

Hi Bruce, 
thank you for your answer, adding cores to the primary mask didn't help, instead it helped manually passing the --base-virtaddr parameter, setting it to the first value of Virtual Area that EAL finds when it starts the primary process.
 
Honestly I don't understand why it works in this way, in the experimental phase this could be a patch, but in the final program I have to automate this process, do you have any suggestions?
For example is there a way to find the virtual area before starting the primary process?
 
Mario
 

Sent: Friday, October 24, 2014 at 2:08 PM
From: "Bruce Richardson" <bruce.richardson@intel.com>
To: "Mario Gianni" <m.gianni@engineer.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread
On Fri, Oct 24, 2014 at 01:21:08PM +0200, Mario Gianni wrote:
> Hi all, I have a problem since I updated to 1.7.0 version,
> I got a multi-process, multi-threaded application,
> In my application first I launch a master process, then I launch a secondary process with multiple threads in it
> Well, when the number of lcores reserved for the secondary process exceeds a certain number (eg. 4) i got an error in rte_eal_init() on the secondary process when it tries to map PCI memory:
>
> EAL: pci_map_resource(): cannot mmap(12, 0x7ffff2e96000, 0x800000, 0x1000): Success (0x7ffff559b000)
> EAL: Cannot mmap device resource
> EAL: Error - exiting with code: 1
> Cause: Requested device 0000:01:00.0 cannot be used
>
> Can you help me?

This could be because the additional memory/stack space used by the pthreads
for the cores in the secondary process is overlapping the space used in the
primary process for hugepage or device memory. You could perhaps try adding
a few cores to the primary process's coremask (and not using those cores)
and see if it helps things.
Alternatively there is a base-virtaddr parameter that can be passed to the
primary process to try and adjust the starting address for it mapping
memory. If you look at where it starts mapping memory right now, and then
try hinting to it to maps the pages at a slightly higher or lower address
and see if it helps.

/Bruce

Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread

[Bruce Richardson]

On Fri, Oct 24, 2014 at 03:04:26PM +0200, Mario Gianni wrote:
> Hi Bruce, 
> thank you for your answer, adding cores to the primary mask didn't help, instead it helped manually passing the --base-virtaddr parameter, setting it to the first value of Virtual Area that EAL finds when it starts the primary process.
>  
> Honestly I don't understand why it works in this way, in the experimental phase this could be a patch, but in the final program I have to automate this process, do you have any suggestions?
> For example is there a way to find the virtual area before starting the primary process?
>  
> Mario

In multi-process, there is a requirement that we can map the hugepage memory 
and the NIC BARs to the same virtual addresses in both processes. Mostly 
this works ok, but occasionally it needs help due to the memory regions 
being chosen in the primary process being used by something else 
pre-eal_init in the secondary process. Anything from additional threads, to 
having an additional shared library linked in can affect the amount of 
memory used by the secondary process and therefore affect the chances that 
we won't be able to get an exact mapping. As far as I know there is no way 
to pre-compute how much memory a given process will use, or what memory 
regions will be free in it, by the time rte_eal_init() is called.

If you just need multiple processes, which don't need to be individually 
spawned, then perhaps consider using fork() to spawn the processes, since 
that will guarantee you idential mappings without issues. The downside 
obviously is that you need to have all processes use the same binary, 
something not required for DPDK multi-process support.  

/Bruce

>  
> 
> Sent: Friday, October 24, 2014 at 2:08 PM
> From: "Bruce Richardson" <bruce.richardson@intel.com>
> To: "Mario Gianni" <m.gianni@engineer.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread
> On Fri, Oct 24, 2014 at 01:21:08PM +0200, Mario Gianni wrote:
> > Hi all, I have a problem since I updated to 1.7.0 version,
> > I got a multi-process, multi-threaded application,
> > In my application first I launch a master process, then I launch a secondary process with multiple threads in it
> > Well, when the number of lcores reserved for the secondary process exceeds a certain number (eg. 4) i got an error in rte_eal_init() on the secondary process when it tries to map PCI memory:
> >
> > EAL: pci_map_resource(): cannot mmap(12, 0x7ffff2e96000, 0x800000, 0x1000): Success (0x7ffff559b000)
> > EAL: Cannot mmap device resource
> > EAL: Error - exiting with code: 1
> > Cause: Requested device 0000:01:00.0 cannot be used
> >
> > Can you help me?
> 
> This could be because the additional memory/stack space used by the pthreads
> for the cores in the secondary process is overlapping the space used in the
> primary process for hugepage or device memory. You could perhaps try adding
> a few cores to the primary process's coremask (and not using those cores)
> and see if it helps things.
> Alternatively there is a base-virtaddr parameter that can be passed to the
> primary process to try and adjust the starting address for it mapping
> memory. If you look at where it starts mapping memory right now, and then
> try hinting to it to maps the pages at a slightly higher or lower address
> and see if it helps.
> 
> /Bruce

Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread

[Mario Gianni]

So you are telling me that in order to implement multi-process I should better use the l2fwd_fork example instead of client_server_mp.
In fact if I use the client_server_mp with a lot of mp_client threads it gives me the error.
If instead I use the l2fwd_fork example it doesn't give me the error.

One more question at this point:
Assume that I use l2fwd_fork, when I launch the secondary process, how do I assign the lcore coremask associated with that process? 

Mario
 
 

Sent: Friday, October 24, 2014 at 3:39 PM
From: "Bruce Richardson" <bruce.richardson@intel.com>
To: "Mario Gianni" <m.gianni@engineer.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread
On Fri, Oct 24, 2014 at 03:04:26PM +0200, Mario Gianni wrote:
> Hi Bruce,
> thank you for your answer, adding cores to the primary mask didn't help, instead it helped manually passing the --base-virtaddr parameter, setting it to the first value of Virtual Area that EAL finds when it starts the primary process.
>  
> Honestly I don't understand why it works in this way, in the experimental phase this could be a patch, but in the final program I have to automate this process, do you have any suggestions?
> For example is there a way to find the virtual area before starting the primary process?
>  
> Mario

In multi-process, there is a requirement that we can map the hugepage memory
and the NIC BARs to the same virtual addresses in both processes. Mostly
this works ok, but occasionally it needs help due to the memory regions
being chosen in the primary process being used by something else
pre-eal_init in the secondary process. Anything from additional threads, to
having an additional shared library linked in can affect the amount of
memory used by the secondary process and therefore affect the chances that
we won't be able to get an exact mapping. As far as I know there is no way
to pre-compute how much memory a given process will use, or what memory
regions will be free in it, by the time rte_eal_init() is called.

If you just need multiple processes, which don't need to be individually
spawned, then perhaps consider using fork() to spawn the processes, since
that will guarantee you idential mappings without issues. The downside
obviously is that you need to have all processes use the same binary,
something not required for DPDK multi-process support.

/Bruce

>  
>
> Sent: Friday, October 24, 2014 at 2:08 PM
> From: "Bruce Richardson" <bruce.richardson@intel.com>
> To: "Mario Gianni" <m.gianni@engineer.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread
> On Fri, Oct 24, 2014 at 01:21:08PM +0200, Mario Gianni wrote:
> > Hi all, I have a problem since I updated to 1.7.0 version,
> > I got a multi-process, multi-threaded application,
> > In my application first I launch a master process, then I launch a secondary process with multiple threads in it
> > Well, when the number of lcores reserved for the secondary process exceeds a certain number (eg. 4) i got an error in rte_eal_init() on the secondary process when it tries to map PCI memory:
> >
> > EAL: pci_map_resource(): cannot mmap(12, 0x7ffff2e96000, 0x800000, 0x1000): Success (0x7ffff559b000)
> > EAL: Cannot mmap device resource
> > EAL: Error - exiting with code: 1
> > Cause: Requested device 0000:01:00.0 cannot be used
> >
> > Can you help me?
>
> This could be because the additional memory/stack space used by the pthreads
> for the cores in the secondary process is overlapping the space used in the
> primary process for hugepage or device memory. You could perhaps try adding
> a few cores to the primary process's coremask (and not using those cores)
> and see if it helps things.
> Alternatively there is a base-virtaddr parameter that can be passed to the
> primary process to try and adjust the starting address for it mapping
> memory. If you look at where it starts mapping memory right now, and then
> try hinting to it to maps the pages at a slightly higher or lower address
> and see if it helps.
>
> /Bruce

Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-process/multi-thread

[Richardson, Bruce]

> -----Original Message-----
> From: Mario Gianni [mailto:m.gianni@engineer.com]
> Sent: Friday, October 24, 2014 4:03 PM
> To: Richardson, Bruce
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] Cannot mmap device resource in DPDK 1.7.0 multi-
> process/multi-thread
> 
> So you are telling me that in order to implement multi-process I should better
> use the l2fwd_fork example instead of client_server_mp.
> In fact if I use the client_server_mp with a lot of mp_client threads it gives me
> the error.
> If instead I use the l2fwd_fork example it doesn't give me the error.
> 
> One more question at this point:
> Assume that I use l2fwd_fork, when I launch the secondary process, how do I
> assign the lcore coremask associated with that process?
> 

With the l2fwd_fork example, the forked processes are not secondary proceses in the sense of the other DPDK multiprocess models - with each one having its own core mask and set of threads. Instead, each forked process is a single thread based off the core-mask provided at process startup. It's a different way of doing things, but one which provides separate processes, while guaranteeing that all processes can share the same hugepage memory at the same address without relying on having the kernel map things at specific points in the address space.

/Bruce