[dpdk-dev] propose a solution for mapping same virtual address space to asymmetric processes

[dpdk-dev] propose a solution for mapping same virtual address space to asymmetric processes

[Nissim Nisimov]

Hi all,

The below will try to suggest a modification to the initialization of Environment Abstraction Layer (AKA EAL) so it will be able to allocate memory zones from same virtual memory addresses even if the primary process is not similar to the secondary processes.

Problem:
The DPDK Primary/Secondary model requires that the exact same hugepage memory mappings be present in all applications.
An issue may occur when the Primary and secondary processes are not symmetric in such way that the code has big differences (for example, Primary process is a traffic distributer and secondary is a worker).
The result may be that specific virtual address region in the first process won't be available in the second process.


Suggested solution:
Map all related rte and uio sections somewhere close to the end of huge pages memory (that mean rte_eal_memory_init() should be called before rte_config_init() in primary process)
According to our observations there will be more probability to success when allocating the above sections after huge pages section (actually uio is already allocated after the huge pages area)

It solved our problem when trying to work with a primary traffic distributer which is a very "light" process and few secondary worker processes.


Please share your thoughts on this before I will try to commit our patch for review

Thanks,
Nissim

Re: [dpdk-dev] propose a solution for mapping same virtual address space to asymmetric processes

[Richardson, Bruce]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Nissim Nisimov
> Sent: Tuesday, October 13, 2015 4:40 PM
> To: 'dev@dpdk.org'
> Subject: [dpdk-dev] propose a solution for mapping same virtual address
> space to asymmetric processes
> 
> Hi all,
> 
> The below will try to suggest a modification to the initialization of
> Environment Abstraction Layer (AKA EAL) so it will be able to allocate
> memory zones from same virtual memory addresses even if the primary
> process is not similar to the secondary processes.
> 
> Problem:
> The DPDK Primary/Secondary model requires that the exact same hugepage
> memory mappings be present in all applications.
> An issue may occur when the Primary and secondary processes are not
> symmetric in such way that the code has big differences (for example,
> Primary process is a traffic distributer and secondary is a worker).
> The result may be that specific virtual address region in the first
> process won't be available in the second process.
> 
> 
> Suggested solution:
> Map all related rte and uio sections somewhere close to the end of huge
> pages memory (that mean rte_eal_memory_init() should be called before
> rte_config_init() in primary process) According to our observations there
> will be more probability to success when allocating the above sections
> after huge pages section (actually uio is already allocated after the huge
> pages area)
> 
> It solved our problem when trying to work with a primary traffic
> distributer which is a very "light" process and few secondary worker
> processes.
> 
> 
> Please share your thoughts on this before I will try to commit our patch
> for review
> 
> Thanks,
> Nissim

Hi,

out of interest, have you tried fixing the issue using the "--base-virtaddr" EAL flag to hint a base address to the primary process? It was put into the code some time ago to help solve exactly this problem.

/Bruce

Re: [dpdk-dev] propose a solution for mapping same virtual address space to asymmetric processes

[Nissim Nisimov]

Hi Bruce,

Using "--base-virtaddr" requires knowledge on the huge pages wanted address going to be used and might vary on different uses of the application.

We suggest a more generic solution which wont require any previous knowledge and will be "bullet proof" as much as possible.

Regards,
Nissim

On Oct 13, 2015 18:49, "Richardson, Bruce" <bruce.richardson@intel.com> wrote:


> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Nissim Nisimov
> Sent: Tuesday, October 13, 2015 4:40 PM
> To: 'dev@dpdk.org'
> Subject: [dpdk-dev] propose a solution for mapping same virtual address
> space to asymmetric processes
>
> Hi all,
>
> The below will try to suggest a modification to the initialization of
> Environment Abstraction Layer (AKA EAL) so it will be able to allocate
> memory zones from same virtual memory addresses even if the primary
> process is not similar to the secondary processes.
>
> Problem:
> The DPDK Primary/Secondary model requires that the exact same hugepage
> memory mappings be present in all applications.
> An issue may occur when the Primary and secondary processes are not
> symmetric in such way that the code has big differences (for example,
> Primary process is a traffic distributer and secondary is a worker).
> The result may be that specific virtual address region in the first
> process won't be available in the second process.
>
>
> Suggested solution:
> Map all related rte and uio sections somewhere close to the end of huge
> pages memory (that mean rte_eal_memory_init() should be called before
> rte_config_init() in primary process) According to our observations there
> will be more probability to success when allocating the above sections
> after huge pages section (actually uio is already allocated after the huge
> pages area)
>
> It solved our problem when trying to work with a primary traffic
> distributer which is a very "light" process and few secondary worker
> processes.
>
>
> Please share your thoughts on this before I will try to commit our patch
> for review
>
> Thanks,
> Nissim

Hi,

out of interest, have you tried fixing the issue using the "--base-virtaddr" EAL flag to hint a base address to the primary process? It was put into the code some time ago to help solve exactly this problem.

/Bruce