[dpdk-dev] DDPK use of MAP_FIXED in mmap

[dpdk-dev] DDPK use of MAP_FIXED in mmap

[Karmarkar Suyash]

Hello,

In DPDK when we use mmap why are we passing the MAP_FIXED flag when Linux man page itself says that the option is discouraged? Any specific reason for passing the MAP_FIXED flag?


http://linux.die.net/man/2/mmap

MAP_FIXED
Don't interpret addr as a hint: place the mapping at exactly that address. addr must be a multiple of the page size. If the memory region specified by addr and len overlaps pages of any existing mapping(s), then the overlapped part of the existing mapping(s) will be discarded. If the specified address cannot be used, mmap() will fail. Because requiring a fixed address for a mapping is less portable, the use of this option is discouraged.


Regards
Suyash Karmarkar

Re: [dpdk-dev] DDPK use of MAP_FIXED in mmap

[Bruce Richardson]

On Mon, Dec 08, 2014 at 07:02:38PM +0000, Karmarkar Suyash wrote:
> Hello,
> 
> In DPDK when we use mmap why are we passing the MAP_FIXED flag when Linux man page itself says that the option is discouraged? Any specific reason for passing the MAP_FIXED flag?
> 
> 
> http://linux.die.net/man/2/mmap
> 
> MAP_FIXED
> Don't interpret addr as a hint: place the mapping at exactly that address. addr must be a multiple of the page size. If the memory region specified by addr and len overlaps pages of any existing mapping(s), then the overlapped part of the existing mapping(s) will be discarded. If the specified address cannot be used, mmap() will fail. Because requiring a fixed address for a mapping is less portable, the use of this option is discouraged.
> 
> 
> Regards
> Suyash Karmarkar

I won't comment on every occurance of "MAP_FIXED" in DPDK, but it's main use is
when mapping the hugepages into memory inside EAL init. In this case, we are ok to
use it, as we take good care to ensure that our mapping space is free. What we do
is, once we know how many contiguous hugepages we need to map, we request a mapping
from /dev/zero for that particular size. We then record the address of the mapping
we get, and then unmap /dev/zero again - thereby freeing up the entire address
range. At this point, we then use MAP_FIXED to explicitly mmap in the hugepages
into this region that we have just freed up - thereby guaranteeing contiguous
hugepage mappings in the correct order. [The reason for doing things this way is
that we found on some systems - particularly with 32-bit code, the regular mmaps
of pages we being done in reverse order, meaning each page became it's own segment].

On the other hand, it's also good to note where we don't use MAP_FIXED. We don't
use map fixed when initializing a secondary process and are mapping the hugepage
memory into it. In this case, although we know where the memory has to be placed,
we don't know if it is safe to use or not. Instead of using MAP_FIXED, we instead
hint to the kernel our preferred address and check if the request was satisfied
at that address.

Hope this clarifies things a bit,
/Bruce

Re: [dpdk-dev] DDPK use of MAP_FIXED in mmap

[Neil Horman]

On Mon, Dec 08, 2014 at 07:02:38PM +0000, Karmarkar Suyash wrote:
> Hello,
> 
> In DPDK when we use mmap why are we passing the MAP_FIXED flag when Linux man page itself says that the option is discouraged? Any specific reason for passing the MAP_FIXED flag?
> 
Because theres nothing wrong with doing so.  The man page says its discouraged
because it creates less portable applications (due to the fact that not all
operating systems support MAP_FIXED).  Given that we currently only support BSD
and Linux however, and given that MAP_FIXED was added to POSIX for
XSI compliant systems, it seems like a reasonable thing to use, as we will most
likely never run into a system that won't support it
Neil

> 
> http://linux.die.net/man/2/mmap
> 
> MAP_FIXED
> Don't interpret addr as a hint: place the mapping at exactly that address. addr must be a multiple of the page size. If the memory region specified by addr and len overlaps pages of any existing mapping(s), then the overlapped part of the existing mapping(s) will be discarded. If the specified address cannot be used, mmap() will fail. Because requiring a fixed address for a mapping is less portable, the use of this option is discouraged.
> 
> 
> Regards
> Suyash Karmarkar
> 

Re: [dpdk-dev] DDPK use of MAP_FIXED in mmap

[Bruce Richardson]

On Tue, Dec 09, 2014 at 10:26:34AM -0500, Neil Horman wrote:
> On Mon, Dec 08, 2014 at 07:02:38PM +0000, Karmarkar Suyash wrote:
> > Hello,
> > 
> > In DPDK when we use mmap why are we passing the MAP_FIXED flag when Linux man page itself says that the option is discouraged? Any specific reason for passing the MAP_FIXED flag?
> > 
> Because theres nothing wrong with doing so.  The man page says its discouraged
> because it creates less portable applications (due to the fact that not all
> operating systems support MAP_FIXED).  Given that we currently only support BSD
> and Linux however, and given that MAP_FIXED was added to POSIX for
> XSI compliant systems, it seems like a reasonable thing to use, as we will most
> likely never run into a system that won't support it
> Neil
> 

Whatever about portability, I thing the best reason to avoid it is given in the
quote from the map page given below: "If the memory region specified by addr and
len overlaps pages of any existing mapping(s), then the overlapped part of the
existing mapping(s) will be discarded." i.e. an mmap with MAP_FIXED can silently
discard an existing mapping. This can lead to some strange behaviour for the
unwary.

/Bruce

> > 
> > http://linux.die.net/man/2/mmap
> > 
> > MAP_FIXED
> > Don't interpret addr as a hint: place the mapping at exactly that address. addr must be a multiple of the page size. If the memory region specified by addr and len overlaps pages of any existing mapping(s), then the overlapped part of the existing mapping(s) will be discarded. If the specified address cannot be used, mmap() will fail. Because requiring a fixed address for a mapping is less portable, the use of this option is discouraged.
> > 
> > 
> > Regards
> > Suyash Karmarkar
> > 

Re: [dpdk-dev] DDPK use of MAP_FIXED in mmap

[Neil Horman]

On Tue, Dec 09, 2014 at 04:04:11PM +0000, Bruce Richardson wrote:
> On Tue, Dec 09, 2014 at 10:26:34AM -0500, Neil Horman wrote:
> > On Mon, Dec 08, 2014 at 07:02:38PM +0000, Karmarkar Suyash wrote:
> > > Hello,
> > > 
> > > In DPDK when we use mmap why are we passing the MAP_FIXED flag when Linux man page itself says that the option is discouraged? Any specific reason for passing the MAP_FIXED flag?
> > > 
> > Because theres nothing wrong with doing so.  The man page says its discouraged
> > because it creates less portable applications (due to the fact that not all
> > operating systems support MAP_FIXED).  Given that we currently only support BSD
> > and Linux however, and given that MAP_FIXED was added to POSIX for
> > XSI compliant systems, it seems like a reasonable thing to use, as we will most
> > likely never run into a system that won't support it
> > Neil
> > 
> 
> Whatever about portability, I thing the best reason to avoid it is given in the
> quote from the map page given below: "If the memory region specified by addr and
> len overlaps pages of any existing mapping(s), then the overlapped part of the
> existing mapping(s) will be discarded." i.e. an mmap with MAP_FIXED can silently
> discard an existing mapping. This can lead to some strange behaviour for the
> unwary.
> 
> /Bruce
> 
That can definately produce some odd behavior, but thats completely avoidable.
An application can introspect its own address mapping to avoid such overlaps.

Of course, if we can avoid using MAP_FIXED, it just makes things a bit easier :)
Neil

> > > 
> > > http://linux.die.net/man/2/mmap
> > > 
> > > MAP_FIXED
> > > Don't interpret addr as a hint: place the mapping at exactly that address. addr must be a multiple of the page size. If the memory region specified by addr and len overlaps pages of any existing mapping(s), then the overlapped part of the existing mapping(s) will be discarded. If the specified address cannot be used, mmap() will fail. Because requiring a fixed address for a mapping is less portable, the use of this option is discouraged.
> > > 
> > > 
> > > Regards
> > > Suyash Karmarkar
> > > 
> 

Re: [dpdk-dev] DDPK use of MAP_FIXED in mmap

[Xie, Huawei]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> Sent: Tuesday, December 09, 2014 2:55 AM
> To: Karmarkar Suyash
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] DDPK use of MAP_FIXED in mmap
> 
> On Mon, Dec 08, 2014 at 07:02:38PM +0000, Karmarkar Suyash wrote:
> > Hello,
> >
> > In DPDK when we use mmap why are we passing the MAP_FIXED flag when
> Linux man page itself says that the option is discouraged? Any specific reason
> for passing the MAP_FIXED flag?
> >
> >
> > http://linux.die.net/man/2/mmap
> >
> > MAP_FIXED
> > Don't interpret addr as a hint: place the mapping at exactly that address. addr
> must be a multiple of the page size. If the memory region specified by addr and
> len overlaps pages of any existing mapping(s), then the overlapped part of the
> existing mapping(s) will be discarded. If the specified address cannot be used,
> mmap() will fail. Because requiring a fixed address for a mapping is less portable,
> the use of this option is discouraged.
> >
> >
> > Regards
> > Suyash Karmarkar
> 
> I won't comment on every occurance of "MAP_FIXED" in DPDK, but it's main use
> is
> when mapping the hugepages into memory inside EAL init. In this case, we are
> ok to
> use it, as we take good care to ensure that our mapping space is free. What we
> do
> is, once we know how many contiguous hugepages we need to map, we request
> a mapping
> from /dev/zero for that particular size. We then record the address of the
> mapping
> we get, and then unmap /dev/zero again - thereby freeing up the entire address
> range. At this point, we then use MAP_FIXED to explicitly mmap in the
> hugepages
> into this region that we have just freed up - thereby guaranteeing contiguous
> hugepage mappings in the correct order. [The reason for doing things this way is
> that we found on some systems - particularly with 32-bit code, the regular
> mmaps
> of pages we being done in reverse order, meaning each page became it's own
> segment].
> 
> On the other hand, it's also good to note where we don't use MAP_FIXED. We
> don't
> use map fixed when initializing a secondary process and are mapping the
> hugepage
> memory into it. In this case, although we know where the memory has to be
> placed,
> we don't know if it is safe to use or not. Instead of using MAP_FIXED, we instead
> hint to the kernel our preferred address and check if the request was satisfied
> at that address.
> 
> Hope this clarifies things a bit,

Don't know if kernel always use the hinted address if the target region is free for mmap.
The safe way is if we are absolutely sure the region is free(like through checking maps file), use MMAP_FIXED(in secondary process).

> /Bruce