Re: [dpdk-dev] dev Digest, Vol 45, Issue 9

Re: [dpdk-dev] dev Digest, Vol 45, Issue 9

[Assaad, Sami (Sami)]

Hello Bruce, 

As you stated previously, I cannot guarantee the proper mapping of the virtual addresses between my primary and secondary processes. I have one primary process and up to 42 secondary processes. The application as a whole works very well. However, closing the application/restarting it continuously results with certain clients (secondary processes) failing to memory map the primary virtual addresses into /dev/zero (fd_zero). [Error "Cound not mmap 8573157376 bytes in /dev/zero to requested address ..."]

So, I disabled ASLR on my virtual machine (VM). Rebooted it with ASLR permanently disabled and ran the application. None of the clients (secondary processes) was able in memory mapping the virtual addresses derived from the primary process. Rte_eal_config_reattach() fails with the error "Cannot mmap memory for rte_config".

My experience with ASLR is limited. Must I rebuild the software with ASLR disabled or am I missing EAL configuration?
How about SELinux - must it be enabled or disabled?
Or ... must I manually configure the virtual addressing for both the primary & secondary processes? An example would help.

I am running the primary process with the option --proc-type=primary and the 40+ clients with --proc-type=secondary.

- My OS is CentOS6.6
- Using KVM/QEMU
- DPDK 1.8.0
- Haswell

Thanks in advance!

Best Regards,
Sami Assaad. 
 

-----Original Message-----

Message: 2
Date: Mon, 15 Jun 2015 11:21:39 +0100
From: Bruce Richardson <bruce.richardson@intel.com>
To: "Assaad, Sami (Sami)" <sami.assaad@alcatel-lucent.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] DPDK and ASLR
Message-ID: <20150615102138.GB3872@bricha3-MOBL3>
Content-Type: text/plain; charset=us-ascii

On Fri, Jun 12, 2015 at 10:53:58PM +0000, Assaad, Sami (Sami) wrote:
> When I operate a DPDK based application, the EAL always reports the following:
> EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
> EAL:      This may cause issues with mapping memory into secondary processes.
> 
> Our application is DPDK client/server based and runs properly.
> 
> My questions are:
> 
> *         Is this warning of any importance?

Yes, it's there for a reason. With ASLR, the position of the hugepage (and other) memory in your DPDK primary process virtual address space will move about from one run to another, and the same with the secondary process. Because of this, you may occasionally get instances where your application fails to run because an essential piece of memory is mapped at address X in the primary, while something else is mapped at address X in the secondary process. How frequently, if ever, this happens will vary from application to application.

If ASLR is disabled, the memory mappings created in the primary and secondary processes will be identical and repeatable from one run to another, so you can know that if a set of processes starts once, it will start a second time. With ASLR enabled, that guarantee cannot be made.


> 
> *         Should ASLR be disabled?
> 

That is a questions we can't answer for you. ASLR is a security feature in the OS so you should be aware of the implications of disabling it. However, if you need absolute guarantees of repeatabiltiy of mappings from one multi-process run to another, the only way get that - that I am aware of - is to disable ASLR. If an occasional random failure at startup is ok, then ASLR can safely be left on.

> *         Does ASLR affect DPDK performance?

No, it only affects the repeatability of memory mappings at DPDK start-up.

Hope this clarifies things.

/Bruce

Re: [dpdk-dev] dev Digest, Vol 45, Issue 9

[Bruce Richardson]

On Wed, Jul 08, 2015 at 11:18:04PM +0000, Assaad, Sami (Sami) wrote:
> Hello Bruce, 
> 
> As you stated previously, I cannot guarantee the proper mapping of the virtual addresses between my primary and secondary processes. I have one primary process and up to 42 secondary processes. The application as a whole works very well. However, closing the application/restarting it continuously results with certain clients (secondary processes) failing to memory map the primary virtual addresses into /dev/zero (fd_zero). [Error "Cound not mmap 8573157376 bytes in /dev/zero to requested address ..."]
> 
> So, I disabled ASLR on my virtual machine (VM). Rebooted it with ASLR permanently disabled and ran the application. None of the clients (secondary processes) was able in memory mapping the virtual addresses derived from the primary process. Rte_eal_config_reattach() fails with the error "Cannot mmap memory for rte_config".
> 
> My experience with ASLR is limited. Must I rebuild the software with ASLR disabled or am I missing EAL configuration?
> How about SELinux - must it be enabled or disabled?
> Or ... must I manually configure the virtual addressing for both the primary & secondary processes? An example would help.
> 
> I am running the primary process with the option --proc-type=primary and the 40+ clients with --proc-type=secondary.
> 
> - My OS is CentOS6.6
> - Using KVM/QEMU
> - DPDK 1.8.0
> - Haswell
> 
> Thanks in advance!
> 
> Best Regards,
> Sami Assaad. 
>  

This unfortunately can happen. To work around this we added in the EAL flag
"--base-virtaddr", to allow you to hint a suitable base address in the primary
process so that the address mappings can succeed in the secondary processes.
[What is happening with ASLR off is that the mappings are repeatable - either
always successful or always fail - sadly you hit the latter].
One tip might be to look at the output of the secondary processes to see what
address the failing file is getting mapped at, and use that as a base address
hint to the primary process. You may have to play about with a few values before
you get it working.
[One other possibility that sometimes works is to pass in additional cores in the
coremask to the primary process. Having more threads, even if they are idle, can
also adjust the address layout due to the addition per-thread stack space.]

/Bruce

> 
> -----Original Message-----
> 
> Message: 2
> Date: Mon, 15 Jun 2015 11:21:39 +0100
> From: Bruce Richardson <bruce.richardson@intel.com>
> To: "Assaad, Sami (Sami)" <sami.assaad@alcatel-lucent.com>
> Cc: "dev@dpdk.org" <dev@dpdk.org>
> Subject: Re: [dpdk-dev] DPDK and ASLR
> Message-ID: <20150615102138.GB3872@bricha3-MOBL3>
> Content-Type: text/plain; charset=us-ascii
> 
> On Fri, Jun 12, 2015 at 10:53:58PM +0000, Assaad, Sami (Sami) wrote:
> > When I operate a DPDK based application, the EAL always reports the following:
> > EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
> > EAL:      This may cause issues with mapping memory into secondary processes.
> > 
> > Our application is DPDK client/server based and runs properly.
> > 
> > My questions are:
> > 
> > *         Is this warning of any importance?
> 
> Yes, it's there for a reason. With ASLR, the position of the hugepage (and other) memory in your DPDK primary process virtual address space will move about from one run to another, and the same with the secondary process. Because of this, you may occasionally get instances where your application fails to run because an essential piece of memory is mapped at address X in the primary, while something else is mapped at address X in the secondary process. How frequently, if ever, this happens will vary from application to application.
> 
> If ASLR is disabled, the memory mappings created in the primary and secondary processes will be identical and repeatable from one run to another, so you can know that if a set of processes starts once, it will start a second time. With ASLR enabled, that guarantee cannot be made.
> 
> 
> > 
> > *         Should ASLR be disabled?
> > 
> 
> That is a questions we can't answer for you. ASLR is a security feature in the OS so you should be aware of the implications of disabling it. However, if you need absolute guarantees of repeatabiltiy of mappings from one multi-process run to another, the only way get that - that I am aware of - is to disable ASLR. If an occasional random failure at startup is ok, then ASLR can safely be left on.
> 
> > *         Does ASLR affect DPDK performance?
> 
> No, it only affects the repeatability of memory mappings at DPDK start-up.
> 
> Hope this clarifies things.
> 
> /Bruce
> 

Re: [dpdk-dev] dev Digest, Vol 45, Issue 9

[Assaad, Sami (Sami)]

Thank you!

Best Regards,
Sami.

-----Original Message-----
From: Bruce Richardson [mailto:bruce.richardson@intel.com] 
Sent: Thursday, July 09, 2015 6:10 AM
To: Assaad, Sami (Sami)
Cc: dev@dpdk.org
Subject: Re: dev Digest, Vol 45, Issue 9

On Wed, Jul 08, 2015 at 11:18:04PM +0000, Assaad, Sami (Sami) wrote:
> Hello Bruce,
> 
> As you stated previously, I cannot guarantee the proper mapping of the 
> virtual addresses between my primary and secondary processes. I have 
> one primary process and up to 42 secondary processes. The application 
> as a whole works very well. However, closing the 
> application/restarting it continuously results with certain clients 
> (secondary processes) failing to memory map the primary virtual 
> addresses into /dev/zero (fd_zero). [Error "Cound not mmap 8573157376 
> bytes in /dev/zero to requested address ..."]
> 
> So, I disabled ASLR on my virtual machine (VM). Rebooted it with ASLR permanently disabled and ran the application. None of the clients (secondary processes) was able in memory mapping the virtual addresses derived from the primary process. Rte_eal_config_reattach() fails with the error "Cannot mmap memory for rte_config".
> 
> My experience with ASLR is limited. Must I rebuild the software with ASLR disabled or am I missing EAL configuration?
> How about SELinux - must it be enabled or disabled?
> Or ... must I manually configure the virtual addressing for both the primary & secondary processes? An example would help.
> 
> I am running the primary process with the option --proc-type=primary and the 40+ clients with --proc-type=secondary.
> 
> - My OS is CentOS6.6
> - Using KVM/QEMU
> - DPDK 1.8.0
> - Haswell
> 
> Thanks in advance!
> 
> Best Regards,
> Sami Assaad. 
>  

This unfortunately can happen. To work around this we added in the EAL flag "--base-virtaddr", to allow you to hint a suitable base address in the primary process so that the address mappings can succeed in the secondary processes.
[What is happening with ASLR off is that the mappings are repeatable - either always successful or always fail - sadly you hit the latter].
One tip might be to look at the output of the secondary processes to see what address the failing file is getting mapped at, and use that as a base address hint to the primary process. You may have to play about with a few values before you get it working.
[One other possibility that sometimes works is to pass in additional cores in the coremask to the primary process. Having more threads, even if they are idle, can also adjust the address layout due to the addition per-thread stack space.]

/Bruce

> 
> -----Original Message-----
> 
> Message: 2
> Date: Mon, 15 Jun 2015 11:21:39 +0100
> From: Bruce Richardson <bruce.richardson@intel.com>
> To: "Assaad, Sami (Sami)" <sami.assaad@alcatel-lucent.com>
> Cc: "dev@dpdk.org" <dev@dpdk.org>
> Subject: Re: [dpdk-dev] DPDK and ASLR
> Message-ID: <20150615102138.GB3872@bricha3-MOBL3>
> Content-Type: text/plain; charset=us-ascii
> 
> On Fri, Jun 12, 2015 at 10:53:58PM +0000, Assaad, Sami (Sami) wrote:
> > When I operate a DPDK based application, the EAL always reports the following:
> > EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
> > EAL:      This may cause issues with mapping memory into secondary processes.
> > 
> > Our application is DPDK client/server based and runs properly.
> > 
> > My questions are:
> > 
> > *         Is this warning of any importance?
> 
> Yes, it's there for a reason. With ASLR, the position of the hugepage (and other) memory in your DPDK primary process virtual address space will move about from one run to another, and the same with the secondary process. Because of this, you may occasionally get instances where your application fails to run because an essential piece of memory is mapped at address X in the primary, while something else is mapped at address X in the secondary process. How frequently, if ever, this happens will vary from application to application.
> 
> If ASLR is disabled, the memory mappings created in the primary and secondary processes will be identical and repeatable from one run to another, so you can know that if a set of processes starts once, it will start a second time. With ASLR enabled, that guarantee cannot be made.
> 
> 
> > 
> > *         Should ASLR be disabled?
> > 
> 
> That is a questions we can't answer for you. ASLR is a security feature in the OS so you should be aware of the implications of disabling it. However, if you need absolute guarantees of repeatabiltiy of mappings from one multi-process run to another, the only way get that - that I am aware of - is to disable ASLR. If an occasional random failure at startup is ok, then ASLR can safely be left on.
> 
> > *         Does ASLR affect DPDK performance?
> 
> No, it only affects the repeatability of memory mappings at DPDK start-up.
> 
> Hope this clarifies things.
> 
> /Bruce
>