[dpdk-dev] Huge mapping secondary process linux

[dpdk-dev] Huge mapping secondary process linux

[Jonas Pfefferle1]

Hi @all,

I'm trying to make sense of the hugepage memory mappings in
librte_eal/linuxapp/eal/eal_memory.c:
* In rte_eal_hugepage_attach (line 1347) when we try to do a private
mapping on /dev/zero (line 1393) why do we not use MAP_FIXED if we need the
addresses to be identical with the primary process?
* On POWER we have this weird business going on where we use MAP_HUGETLB
because according to this commit:

commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
Date:   Thu Apr 6 15:36:09 2017 +0530

    eal/ppc: fix mmap for memory initialization

    On IBM POWER platform, when mapping /dev/zero file to hugepage memory
    space, mmap will not respect the requested address hint. This will
cause
    the memory initialization for the second process fails. This patch adds
    the required mmap flags to make it work. Beside this, users need to set
    the nr_overcommit_hugepages to expand the VA range. When
    doing the initialization, users need to set both nr_hugepages and
    nr_overcommit_hugepages to the same value, like 64, 128, etc.

mmap address hints are not respected. Looking at the mmap code in the
kernel this is not true entirely however under some circumstances the hint
can be ignored (
http://elixir.free-electrons.com/linux/latest/source/arch/powerpc/mm/mmap.c#L103
). However I believe we can remove the extra case for PPC if we use
MAP_FIXED when doing the secondary process mappings because we need them to
be identical anyway. We could also use MAP_FIXED when doing the primary
process mappings resp. get_virtual_area if we want to have any guarantees
when specifying a base address. Any thoughts?

Thanks,
Jonas

Re: [dpdk-dev] Huge mapping secondary process linux

[Burakov, Anatoly]

On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
> 
> 
> Hi @all,
> 
> I'm trying to make sense of the hugepage memory mappings in
> librte_eal/linuxapp/eal/eal_memory.c:
> * In rte_eal_hugepage_attach (line 1347) when we try to do a private
> mapping on /dev/zero (line 1393) why do we not use MAP_FIXED if we need the
> addresses to be identical with the primary process?
> * On POWER we have this weird business going on where we use MAP_HUGETLB
> because according to this commit:
> 
> commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
> Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> Date:   Thu Apr 6 15:36:09 2017 +0530
> 
>      eal/ppc: fix mmap for memory initialization
> 
>      On IBM POWER platform, when mapping /dev/zero file to hugepage memory
>      space, mmap will not respect the requested address hint. This will
> cause
>      the memory initialization for the second process fails. This patch adds
>      the required mmap flags to make it work. Beside this, users need to set
>      the nr_overcommit_hugepages to expand the VA range. When
>      doing the initialization, users need to set both nr_hugepages and
>      nr_overcommit_hugepages to the same value, like 64, 128, etc.
> 
> mmap address hints are not respected. Looking at the mmap code in the
> kernel this is not true entirely however under some circumstances the hint
> can be ignored (
> http://elixir.free-electrons.com/linux/latest/source/arch/powerpc/mm/mmap.c#L103
> ). However I believe we can remove the extra case for PPC if we use
> MAP_FIXED when doing the secondary process mappings because we need them to
> be identical anyway. We could also use MAP_FIXED when doing the primary
> process mappings resp. get_virtual_area if we want to have any guarantees
> when specifying a base address. Any thoughts?
> 
> Thanks,
> Jonas
> 
hi Jonas,

MAP_FIXED is not used because it's dangerous, it unmaps anything that is 
already mapped into that space. We would rather know that we can't map 
something than unwittingly unmap something that was mapped before.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] Huge mapping secondary process linux

[Jonas Pfefferle1]

"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017 04:06:44
PM:

> From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
> Cc: chaozhu@linux.vnet.ibm.com, bruce.richardson@intel.com
> Date: 10/27/2017 04:06 PM
> Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>
> On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
> >
> >
> > Hi @all,
> >
> > I'm trying to make sense of the hugepage memory mappings in
> > librte_eal/linuxapp/eal/eal_memory.c:
> > * In rte_eal_hugepage_attach (line 1347) when we try to do a private
> > mapping on /dev/zero (line 1393) why do we not use MAP_FIXED if we need
the
> > addresses to be identical with the primary process?
> > * On POWER we have this weird business going on where we use
MAP_HUGETLB
> > because according to this commit:
> >
> > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
> > Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> > Date:   Thu Apr 6 15:36:09 2017 +0530
> >
> >      eal/ppc: fix mmap for memory initialization
> >
> >      On IBM POWER platform, when mapping /dev/zero file to hugepage
memory
> >      space, mmap will not respect the requested address hint. This will
> > cause
> >      the memory initialization for the second process fails. This patch
adds
> >      the required mmap flags to make it work. Beside this, users need
to set
> >      the nr_overcommit_hugepages to expand the VA range. When
> >      doing the initialization, users need to set both nr_hugepages and
> >      nr_overcommit_hugepages to the same value, like 64, 128, etc.
> >
> > mmap address hints are not respected. Looking at the mmap code in the
> > kernel this is not true entirely however under some circumstances the
hint
> > can be ignored (
> > https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-

> siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
> BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
> > ). However I believe we can remove the extra case for PPC if we use
> > MAP_FIXED when doing the secondary process mappings because we need
them to
> > be identical anyway. We could also use MAP_FIXED when doing the primary
> > process mappings resp. get_virtual_area if we want to have any
guarantees
> > when specifying a base address. Any thoughts?
> >
> > Thanks,
> > Jonas
> >
> hi Jonas,
>
> MAP_FIXED is not used because it's dangerous, it unmaps anything that is
> already mapped into that space. We would rather know that we can't map
> something than unwittingly unmap something that was mapped before.

Ok, I see. Maybe we can add a check to the primary process's memory
mappings whether the hint has been respected or not? At least warn if it
hasn't.

>
> --
> Thanks,
> Anatoly
>

Re: [dpdk-dev] Huge mapping secondary process linux

[Burakov, Anatoly]

On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017 
> 04:06:44 PM:
> 
>  > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>  > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
>  > Cc: chaozhu@linux.vnet.ibm.com, bruce.richardson@intel.com
>  > Date: 10/27/2017 04:06 PM
>  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>  >
>  > On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
>  > >
>  > >
>  > > Hi @all,
>  > >
>  > > I'm trying to make sense of the hugepage memory mappings in
>  > > librte_eal/linuxapp/eal/eal_memory.c:
>  > > * In rte_eal_hugepage_attach (line 1347) when we try to do a private
>  > > mapping on /dev/zero (line 1393) why do we not use MAP_FIXED if we 
> need the
>  > > addresses to be identical with the primary process?
>  > > * On POWER we have this weird business going on where we use 
> MAP_HUGETLB
>  > > because according to this commit:
>  > >
>  > > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
>  > > Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
>  > > Date:   Thu Apr 6 15:36:09 2017 +0530
>  > >
>  > >      eal/ppc: fix mmap for memory initialization
>  > >
>  > >      On IBM POWER platform, when mapping /dev/zero file to hugepage 
> memory
>  > >      space, mmap will not respect the requested address hint. This will
>  > > cause
>  > >      the memory initialization for the second process fails. This 
> patch adds
>  > >      the required mmap flags to make it work. Beside this, users 
> need to set
>  > >      the nr_overcommit_hugepages to expand the VA range. When
>  > >      doing the initialization, users need to set both nr_hugepages and
>  > >      nr_overcommit_hugepages to the same value, like 64, 128, etc.
>  > >
>  > > mmap address hints are not respected. Looking at the mmap code in the
>  > > kernel this is not true entirely however under some circumstances 
> the hint
>  > > can be ignored (
>  > > https://urldefense.proofpoint.com/v2/url?
>  > 
> u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-
>  > siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
>  > pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
>  > BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
>  > > ). However I believe we can remove the extra case for PPC if we use
>  > > MAP_FIXED when doing the secondary process mappings because we need 
> them to
>  > > be identical anyway. We could also use MAP_FIXED when doing the primary
>  > > process mappings resp. get_virtual_area if we want to have any 
> guarantees
>  > > when specifying a base address. Any thoughts?
>  > >
>  > > Thanks,
>  > > Jonas
>  > >
>  > hi Jonas,
>  >
>  > MAP_FIXED is not used because it's dangerous, it unmaps anything that is
>  > already mapped into that space. We would rather know that we can't map
>  > something than unwittingly unmap something that was mapped before.
> 
> Ok, I see. Maybe we can add a check to the primary process's memory 
> mappings whether the hint has been respected or not? At least warn if it 
> hasn't.

Hi Jonas,

I'm unfamiliar with POWER platform, so i'm afraid you'd have to explain 
a bit more what you mean by "hint has been respected" :)


-- 
Thanks,
Anatoly

Re: [dpdk-dev] Huge mapping secondary process linux

[Jonas Pfefferle1]

"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017 04:44:52
PM:

> From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com, dev@dpdk.org
> Date: 10/27/2017 04:45 PM
> Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>
> On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
> > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017
> > 04:06:44 PM:
> >
> >  > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> >  > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
> >  > Cc: chaozhu@linux.vnet.ibm.com, bruce.richardson@intel.com
> >  > Date: 10/27/2017 04:06 PM
> >  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> >  >
> >  > On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
> >  > >
> >  > >
> >  > > Hi @all,
> >  > >
> >  > > I'm trying to make sense of the hugepage memory mappings in
> >  > > librte_eal/linuxapp/eal/eal_memory.c:
> >  > > * In rte_eal_hugepage_attach (line 1347) when we try to do a
private
> >  > > mapping on /dev/zero (line 1393) why do we not use MAP_FIXED if we

> > need the
> >  > > addresses to be identical with the primary process?
> >  > > * On POWER we have this weird business going on where we use
> > MAP_HUGETLB
> >  > > because according to this commit:
> >  > >
> >  > > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
> >  > > Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> >  > > Date:   Thu Apr 6 15:36:09 2017 +0530
> >  > >
> >  > >      eal/ppc: fix mmap for memory initialization
> >  > >
> >  > >      On IBM POWER platform, when mapping /dev/zero file to
hugepage
> > memory
> >  > >      space, mmap will not respect the requested address hint.This
will
> >  > > cause
> >  > >      the memory initialization for the second process fails. This
> > patch adds
> >  > >      the required mmap flags to make it work. Beside this, users
> > need to set
> >  > >      the nr_overcommit_hugepages to expand the VA range. When
> >  > >      doing the initialization, users need to set both nr_hugepages
and
> >  > >      nr_overcommit_hugepages to the same value, like 64, 128, etc.
> >  > >
> >  > > mmap address hints are not respected. Looking at the mmap code in
the
> >  > > kernel this is not true entirely however under some circumstances
> > the hint
> >  > > can be ignored (
> >  > > https://urldefense.proofpoint.com/v2/url?
> >  >
> >
>
u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-

> >  > siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> >  > pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
> >  >
BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
> >  > > ). However I believe we can remove the extra case for PPC if we
use
> >  > > MAP_FIXED when doing the secondary process mappings because we
need
> > them to
> >  > > be identical anyway. We could also use MAP_FIXED when doing the
primary
> >  > > process mappings resp. get_virtual_area if we want to have any
> > guarantees
> >  > > when specifying a base address. Any thoughts?
> >  > >
> >  > > Thanks,
> >  > > Jonas
> >  > >
> >  > hi Jonas,
> >  >
> >  > MAP_FIXED is not used because it's dangerous, it unmaps anything
that is
> >  > already mapped into that space. We would rather know that we can't
map
> >  > something than unwittingly unmap something that was mapped before.
> >
> > Ok, I see. Maybe we can add a check to the primary process's memory
> > mappings whether the hint has been respected or not? At least warn if
it
> > hasn't.
>
> Hi Jonas,
>
> I'm unfamiliar with POWER platform, so i'm afraid you'd have to explain
> a bit more what you mean by "hint has been respected" :)

Hi Anatoly,

What I meant was the mmap address hint:

"If addr is not NULL, then the kernel takes it as a hint
 about where to place the mapping; on Linux, the mapping will be
 created at a nearby page boundary."

This is actually not true on POWER. It can happen that the address hint is
ignored and you get any address back that fits your mapping.

Thanks,
Jonas

>
>
> --
> Thanks,
> Anatoly
>

Re: [dpdk-dev] Huge mapping secondary process linux

[Jonas Pfefferle1]

"dev" <dev-bounces@dpdk.org> wrote on 10/27/2017 04:58:01 PM:

> From: "Jonas Pfefferle1" <JPF@zurich.ibm.com>
> To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com, dev@dpdk.org
> Date: 10/27/2017 04:58 PM
> Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> Sent by: "dev" <dev-bounces@dpdk.org>
>
>
> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017
04:44:52
> PM:
>
> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com,
dev@dpdk.org
> > Date: 10/27/2017 04:45 PM
> > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> >
> > On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
> > > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017
> > > 04:06:44 PM:
> > >
> > >  > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > >  > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
> > >  > Cc: chaozhu@linux.vnet.ibm.com, bruce.richardson@intel.com
> > >  > Date: 10/27/2017 04:06 PM
> > >  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > >  >
> > >  > On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
> > >  > >
> > >  > >
> > >  > > Hi @all,
> > >  > >
> > >  > > I'm trying to make sense of the hugepage memory mappings in
> > >  > > librte_eal/linuxapp/eal/eal_memory.c:
> > >  > > * In rte_eal_hugepage_attach (line 1347) when we try to do a
> private
> > >  > > mapping on /dev/zero (line 1393) why do we not use MAP_FIXED if
we
>
> > > need the
> > >  > > addresses to be identical with the primary process?
> > >  > > * On POWER we have this weird business going on where we use
> > > MAP_HUGETLB
> > >  > > because according to this commit:
> > >  > >
> > >  > > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
> > >  > > Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> > >  > > Date:   Thu Apr 6 15:36:09 2017 +0530
> > >  > >
> > >  > >      eal/ppc: fix mmap for memory initialization
> > >  > >
> > >  > >      On IBM POWER platform, when mapping /dev/zero file to
> hugepage
> > > memory
> > >  > >      space, mmap will not respect the requested address
hint.This
> will
> > >  > > cause
> > >  > >      the memory initialization for the second process fails.
This
> > > patch adds
> > >  > >      the required mmap flags to make it work. Beside this, users
> > > need to set
> > >  > >      the nr_overcommit_hugepages to expand the VA range. When
> > >  > >      doing the initialization, users need to set both
nr_hugepages
> and
> > >  > >      nr_overcommit_hugepages to the same value, like 64, 128,
etc.
> > >  > >
> > >  > > mmap address hints are not respected. Looking at the mmap code
in
> the
> > >  > > kernel this is not true entirely however under some
circumstances
> > > the hint
> > >  > > can be ignored (
> > >  > > https://urldefense.proofpoint.com/v2/url?
> > >  >
> > >
> >
>
u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-

>
> > >  > siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> > >  > pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
> > >  >
> BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
> > >  > > ). However I believe we can remove the extra case for PPC if we
> use
> > >  > > MAP_FIXED when doing the secondary process mappings because we
> need
> > > them to
> > >  > > be identical anyway. We could also use MAP_FIXED when doing the
> primary
> > >  > > process mappings resp. get_virtual_area if we want to have any
> > > guarantees
> > >  > > when specifying a base address. Any thoughts?
> > >  > >
> > >  > > Thanks,
> > >  > > Jonas
> > >  > >
> > >  > hi Jonas,
> > >  >
> > >  > MAP_FIXED is not used because it's dangerous, it unmaps anything
> that is
> > >  > already mapped into that space. We would rather know that we can't
> map
> > >  > something than unwittingly unmap something that was mapped before.
> > >
> > > Ok, I see. Maybe we can add a check to the primary process's memory
> > > mappings whether the hint has been respected or not? At least warn if
> it
> > > hasn't.
> >
> > Hi Jonas,
> >
> > I'm unfamiliar with POWER platform, so i'm afraid you'd have to explain
> > a bit more what you mean by "hint has been respected" :)
>
> Hi Anatoly,
>
> What I meant was the mmap address hint:
>
> "If addr is not NULL, then the kernel takes it as a hint
>  about where to place the mapping; on Linux, the mapping will be
>  created at a nearby page boundary."
>
> This is actually not true on POWER. It can happen that the address hint
is
> ignored and you get any address back that fits your mapping.
>
> Thanks,
> Jonas

Actually looking through the kernel code this is also not guaranteed on
x86.
(
http://elixir.free-electrons.com/linux/latest/source/arch/x86/kernel/sys_x86_64.c#L165
)

So in any case the address hint can be ignored by the kernel and you get
any address that fits your mapping.
My suggestion is to check when we do the initial mapping in
get_virtual_area if the hint was respected or not, i.e. if the returned
address == PAGE_ALIGN(address_hint).

Thanks,
Jonas

>
> >
> >
> > --
> > Thanks,
> > Anatoly
> >
>

Re: [dpdk-dev] Huge mapping secondary process linux

[Burakov, Anatoly]

On 27-Oct-17 4:16 PM, Jonas Pfefferle1 wrote:
> "dev" <dev-bounces@dpdk.org> wrote on 10/27/2017 04:58:01 PM:
> 
>  > From: "Jonas Pfefferle1" <JPF@zurich.ibm.com>
>  > To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>  > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com, dev@dpdk.org
>  > Date: 10/27/2017 04:58 PM
>  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>  > Sent by: "dev" <dev-bounces@dpdk.org>
>  >
>  >
>  > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017 
> 04:44:52
>  > PM:
>  >
>  > > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>  > > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
>  > > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com, 
> dev@dpdk.org
>  > > Date: 10/27/2017 04:45 PM
>  > > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>  > >
>  > > On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
>  > > > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017
>  > > > 04:06:44 PM:
>  > > >
>  > > >  > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>  > > >  > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
>  > > >  > Cc: chaozhu@linux.vnet.ibm.com, bruce.richardson@intel.com
>  > > >  > Date: 10/27/2017 04:06 PM
>  > > >  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>  > > >  >
>  > > >  > On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
>  > > >  > >
>  > > >  > >
>  > > >  > > Hi @all,
>  > > >  > >
>  > > >  > > I'm trying to make sense of the hugepage memory mappings in
>  > > >  > > librte_eal/linuxapp/eal/eal_memory.c:
>  > > >  > > * In rte_eal_hugepage_attach (line 1347) when we try to do a
>  > private
>  > > >  > > mapping on /dev/zero (line 1393) why do we not use MAP_FIXED 
> if we
>  >
>  > > > need the
>  > > >  > > addresses to be identical with the primary process?
>  > > >  > > * On POWER we have this weird business going on where we use
>  > > > MAP_HUGETLB
>  > > >  > > because according to this commit:
>  > > >  > >
>  > > >  > > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
>  > > >  > > Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
>  > > >  > > Date:   Thu Apr 6 15:36:09 2017 +0530
>  > > >  > >
>  > > >  > >      eal/ppc: fix mmap for memory initialization
>  > > >  > >
>  > > >  > >      On IBM POWER platform, when mapping /dev/zero file to
>  > hugepage
>  > > > memory
>  > > >  > >      space, mmap will not respect the requested address 
> hint.This
>  > will
>  > > >  > > cause
>  > > >  > >      the memory initialization for the second process fails. 
> This
>  > > > patch adds
>  > > >  > >      the required mmap flags to make it work. Beside this, users
>  > > > need to set
>  > > >  > >      the nr_overcommit_hugepages to expand the VA range. When
>  > > >  > >      doing the initialization, users need to set both 
> nr_hugepages
>  > and
>  > > >  > >      nr_overcommit_hugepages to the same value, like 64, 
> 128, etc.
>  > > >  > >
>  > > >  > > mmap address hints are not respected. Looking at the mmap 
> code in
>  > the
>  > > >  > > kernel this is not true entirely however under some 
> circumstances
>  > > > the hint
>  > > >  > > can be ignored (
>  > > >  > > https://urldefense.proofpoint.com/v2/url?
>  > > >  >
>  > > >
>  > >
>  > 
> u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-
>  >
>  > > >  > siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
>  > > >  > pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
>  > > >  >
>  > BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
>  > > >  > > ). However I believe we can remove the extra case for PPC if we
>  > use
>  > > >  > > MAP_FIXED when doing the secondary process mappings because we
>  > need
>  > > > them to
>  > > >  > > be identical anyway. We could also use MAP_FIXED when doing the
>  > primary
>  > > >  > > process mappings resp. get_virtual_area if we want to have any
>  > > > guarantees
>  > > >  > > when specifying a base address. Any thoughts?
>  > > >  > >
>  > > >  > > Thanks,
>  > > >  > > Jonas
>  > > >  > >
>  > > >  > hi Jonas,
>  > > >  >
>  > > >  > MAP_FIXED is not used because it's dangerous, it unmaps anything
>  > that is
>  > > >  > already mapped into that space. We would rather know that we can't
>  > map
>  > > >  > something than unwittingly unmap something that was mapped before.
>  > > >
>  > > > Ok, I see. Maybe we can add a check to the primary process's memory
>  > > > mappings whether the hint has been respected or not? At least warn if
>  > it
>  > > > hasn't.
>  > >
>  > > Hi Jonas,
>  > >
>  > > I'm unfamiliar with POWER platform, so i'm afraid you'd have to explain
>  > > a bit more what you mean by "hint has been respected" :)
>  >
>  > Hi Anatoly,
>  >
>  > What I meant was the mmap address hint:
>  >
>  > "If addr is not NULL, then the kernel takes it as a hint
>  >  about where to place the mapping; on Linux, the mapping will be
>  >  created at a nearby page boundary."
>  >
>  > This is actually not true on POWER. It can happen that the address 
> hint is
>  > ignored and you get any address back that fits your mapping.
>  >
>  > Thanks,
>  > Jonas
> 
> Actually looking through the kernel code this is also not guaranteed on x86.
> (http://elixir.free-electrons.com/linux/latest/source/arch/x86/kernel/sys_x86_64.c#L165)
> 
> So in any case the address hint can be ignored by the kernel and you get 
> any address that fits your mapping.
> My suggestion is to check when we do the initial mapping in 
> get_virtual_area if the hint was respected or not, i.e. if the returned 
> address == PAGE_ALIGN(address_hint).
> 

I'm not sure i see the issue here. So, just to make sure i understand 
things correctly:

Whenever we don't request a specific base address through base_address 
EAL parameter, none of this matters - we always ask for memory in 
arbitrary memory locations, correct?

It's also not an issue with secondary processes because we do check 
returned mmap address to see whether it's the same as we requested, correct?

It's only whenever we *do* specify a base_address, we provide an address 
hint to mmap to, but we don't check if the address we got from mmap is 
one in the vicinity of our requested base address, correct? We don't 
check, and the kernel can ignore address hint, so we're not guaranteed 
to respect the base_address flag.

I'm not sure this is a serious issue, because as far as i'm concerned, 
this flag is advisory - we only promise to *attempt* to map things at 
that particular address, not that it will succeed. If the kernel simply 
cannot find an address to satisfy our address hint, or ignores it for 
other reasons - well, tough, nothing we can do about that. I'm not sure 
putting a check like this, where we can't even predict an "expected" 
address is a good idea.

Am i getting this right?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] Huge mapping secondary process linux

[Jonas Pfefferle1]

"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 27/10/2017 
18:00:27:

> From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com, dev@dpdk.org
> Date: 27/10/2017 18:00
> Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> 
> On 27-Oct-17 4:16 PM, Jonas Pfefferle1 wrote:
> > "dev" <dev-bounces@dpdk.org> wrote on 10/27/2017 04:58:01 PM:
> > 
> >  > From: "Jonas Pfefferle1" <JPF@zurich.ibm.com>
> >  > To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> >  > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com, 
dev@dpdk.org
> >  > Date: 10/27/2017 04:58 PM
> >  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> >  > Sent by: "dev" <dev-bounces@dpdk.org>
> >  >
> >  >
> >  > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017 
> > 04:44:52
> >  > PM:
> >  >
> >  > > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> >  > > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> >  > > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com, 
> > dev@dpdk.org
> >  > > Date: 10/27/2017 04:45 PM
> >  > > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> >  > >
> >  > > On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
> >  > > > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 
10/27/2017
> >  > > > 04:06:44 PM:
> >  > > >
> >  > > > Â > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> >  > > > Â > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
> >  > > > Â > Cc: chaozhu@linux.vnet.ibm.com, bruce.richardson@intel.com
> >  > > > Â > Date: 10/27/2017 04:06 PM
> >  > > > Â > Subject: Re: [dpdk-dev] Huge mapping secondary process 
linux
> >  > > > Â >
> >  > > > Â > On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
> >  > > > Â > >
> >  > > > Â > >
> >  > > > Â > > Hi @all,
> >  > > > Â > >
> >  > > > Â > > I'm trying to make sense of the hugepage memory mappings 
in
> >  > > > Â > > librte_eal/linuxapp/eal/eal_memory.c:
> >  > > > Â > > * In rte_eal_hugepage_attach (line 1347) when we try to 
do a
> >  > private
> >  > > > Â > > mapping on /dev/zero (line 1393) why do we not use 
MAP_FIXED 
> > if we
> >  >
> >  > > > need the
> >  > > > Â > > addresses to be identical with the primary process?
> >  > > > Â > > * On POWER we have this weird business going on where we 
use
> >  > > > MAP_HUGETLB
> >  > > > Â > > because according to this commit:
> >  > > > Â > >
> >  > > > Â > > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
> >  > > > Â > > Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> >  > > > Â > > Date: Â  Thu Apr 6 15:36:09 2017 +0530
> >  > > > Â > >
> >  > > > Â > > Â  Â  Â eal/ppc: fix mmap for memory initialization
> >  > > > Â > >
> >  > > > Â > > Â  Â  Â On IBM POWER platform, when mapping /dev/zero 
file to
> >  > hugepage
> >  > > > memory
> >  > > > Â > > Â  Â  Â space, mmap will not respect the requested 
address 
> > hint.This
> >  > will
> >  > > > Â > > cause
> >  > > > Â > > Â  Â  Â the memory initialization for the second 
> process fails. 
> > This
> >  > > > patch adds
> >  > > > Â > > Â  Â  Â the required mmap flags to make it work. 
> Beside this, users
> >  > > > need to set
> >  > > > Â > > Â  Â  Â the nr_overcommit_hugepages to expand the VA 
> range. When
> >  > > > Â > > Â  Â  Â doing the initialization, users need to set both 
> > nr_hugepages
> >  > and
> >  > > > Â > > Â  Â  Â nr_overcommit_hugepages to the same value, like 
64, 
> > 128, etc.
> >  > > > Â > >
> >  > > > Â > > mmap address hints are not respected. Looking at the mmap 

> > code in
> >  > the
> >  > > > Â > > kernel this is not true entirely however under some 
> > circumstances
> >  > > > the hint
> >  > > > Â > > can be ignored (
> >  > > > Â > > https://urldefense.proofpoint.com/v2/url?
> >  > > > Â >
> >  > > >
> >  > >
> >  > 
> > 
> 
u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-
> >  >
> >  > > > Â > siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> >  > > > Â > pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
> >  > > > Â >
> >  > 
BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
> >  > > > Â > > ). However I believe we can remove the extra case forPPC 
if we
> >  > use
> >  > > > Â > > MAP_FIXED when doing the secondary process mappings 
because we
> >  > need
> >  > > > them to
> >  > > > Â > > be identical anyway. We could also use MAP_FIXED 
whendoing the
> >  > primary
> >  > > > Â > > process mappings resp. get_virtual_area if we want to 
have any
> >  > > > guarantees
> >  > > > Â > > when specifying a base address. Any thoughts?
> >  > > > Â > >
> >  > > > Â > > Thanks,
> >  > > > Â > > Jonas
> >  > > > Â > >
> >  > > > Â > hi Jonas,
> >  > > > Â >
> >  > > > Â > MAP_FIXED is not used because it's dangerous, it unmaps 
anything
> >  > that is
> >  > > > Â > already mapped into that space. We would rather know 
> that we can't
> >  > map
> >  > > > Â > something than unwittingly unmap something that was 
> mapped before.
> >  > > >
> >  > > > Ok, I see. Maybe we can add a check to the primary process's 
memory
> >  > > > mappings whether the hint has been respected or not? At least 
warn if
> >  > it
> >  > > > hasn't.
> >  > >
> >  > > Hi Jonas,
> >  > >
> >  > > I'm unfamiliar with POWER platform, so i'm afraid you'd have to 
explain
> >  > > a bit more what you mean by "hint has been respected" :)
> >  >
> >  > Hi Anatoly,
> >  >
> >  > What I meant was the mmap address hint:
> >  >
> >  > "If addr is not NULL, then the kernel takes it as a hint
> >  > Â about where to place the mapping; on Linux, the mapping will be
> >  > Â created at a nearby page boundary."
> >  >
> >  > This is actually not true on POWER. It can happen that the address 
> > hint is
> >  > ignored and you get any address back that fits your mapping.
> >  >
> >  > Thanks,
> >  > Jonas
> > 
> > Actually looking through the kernel code this is also not guaranteed 
on x86.
> > (https://urldefense.proofpoint.com/v2/url?
> 
u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_x86_kernel_sys-5Fx86-5F64.c-23L165&d=DwID-
> g&c=jf_iaSHvJObTbx-siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> 
pXjigIjRW0&m=iqakzG7nSXLfvDHyS9IV5E9DWPnNcv19zcsl3MKMdvI&s=VqzZpcTaCUMmNieZ3WyUw-
> jsnNP-hAcW487Mumv6xPw&e=)
> > 
> > So in any case the address hint can be ignored by the kernel and you 
get 
> > any address that fits your mapping.
> > My suggestion is to check when we do the initial mapping in 
> > get_virtual_area if the hint was respected or not, i.e. if the 
returned 
> > address == PAGE_ALIGN(address_hint).
> > 
> 
> I'm not sure i see the issue here. So, just to make sure i understand 
> things correctly:
> 
> Whenever we don't request a specific base address through base_address 
> EAL parameter, none of this matters - we always ask for memory in 
> arbitrary memory locations, correct?
> 
> It's also not an issue with secondary processes because we do check 
> returned mmap address to see whether it's the same as we requested, 
correct?
> 
> It's only whenever we *do* specify a base_address, we provide an address 

> hint to mmap to, but we don't check if the address we got from mmap is 
> one in the vicinity of our requested base address, correct? We don't 
> check, and the kernel can ignore address hint, so we're not guaranteed 
> to respect the base_address flag.
> 
> I'm not sure this is a serious issue, because as far as i'm concerned, 
> this flag is advisory - we only promise to *attempt* to map things at 
> that particular address, not that it will succeed. If the kernel simply 
> cannot find an address to satisfy our address hint, or ignores it for 
> other reasons - well, tough, nothing we can do about that. I'm not sure 
> putting a check like this, where we can't even predict an "expected" 
> address is a good idea.
> 
> Am i getting this right?

The problem is when we specify a base address we want it to be used. If it 
is
not respected we basically end up with the case like we would have never 
specified it.
This very likely leads to not being able to run a secondary process 
because
we will not be able to map the addresses from our primary process and that 
is why we
introduced the base address parameter in the first place.

> 
> -- 
> Thanks,
> Anatoly
> 

Re: [dpdk-dev] Huge mapping secondary process linux

[Chao Zhu]

 

 

From: Jonas Pfefferle1 [mailto:JPF@zurich.ibm.com] 
Sent: 2017年10月28日 3:23
To: Burakov, Anatoly <anatoly.burakov@intel.com>
Cc: bruce.richardson@intel.com; chaozhu@linux.vnet.ibm.com; dev@dpdk.org
Subject: Re: [dpdk-dev] Huge mapping secondary process linux

 

"Burakov, Anatoly" < <mailto:anatoly.burakov@intel.com> anatoly.burakov@intel.com> wrote on 27/10/2017 18:00:27:

> From: "Burakov, Anatoly" < <mailto:anatoly.burakov@intel.com> anatoly.burakov@intel.com>
> To: Jonas Pfefferle1 < <mailto:JPF@zurich.ibm.com> JPF@zurich.ibm.com>
> Cc:  <mailto:bruce.richardson@intel.com> bruce.richardson@intel.com,  <mailto:chaozhu@linux.vnet.ibm.com> chaozhu@linux.vnet.ibm.com,  <mailto:dev@dpdk.org> dev@dpdk.org
> Date: 27/10/2017 18:00
> Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> 
> On 27-Oct-17 4:16 PM, Jonas Pfefferle1 wrote:
> > "dev" < <mailto:dev-bounces@dpdk.org> dev-bounces@dpdk.org> wrote on 10/27/2017 04:58:01 PM:
> > 
> >  > From: "Jonas Pfefferle1" < <mailto:JPF@zurich.ibm.com> JPF@zurich.ibm.com>
> >  > To: "Burakov, Anatoly" < <mailto:anatoly.burakov@intel.com> anatoly.burakov@intel.com>
> >  > Cc:  <mailto:bruce.richardson@intel.com> bruce.richardson@intel.com,  <mailto:chaozhu@linux.vnet.ibm.com> chaozhu@linux.vnet.ibm.com,  <mailto:dev@dpdk.org> dev@dpdk.org
> >  > Date: 10/27/2017 04:58 PM
> >  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> >  > Sent by: "dev" < <mailto:dev-bounces@dpdk.org> dev-bounces@dpdk.org>
> >  >
> >  >
> >  > "Burakov, Anatoly" < <mailto:anatoly.burakov@intel.com> anatoly.burakov@intel.com> wrote on 10/27/2017 
> > 04:44:52
> >  > PM:
> >  >
> >  > > From: "Burakov, Anatoly" < <mailto:anatoly.burakov@intel.com> anatoly.burakov@intel.com>
> >  > > To: Jonas Pfefferle1 < <mailto:JPF@zurich.ibm.com> JPF@zurich.ibm.com>
> >  > > Cc:  <mailto:bruce.richardson@intel.com> bruce.richardson@intel.com,  <mailto:chaozhu@linux.vnet.ibm.com> chaozhu@linux.vnet.ibm.com, 
> >  <mailto:dev@dpdk.org> dev@dpdk.org
> >  > > Date: 10/27/2017 04:45 PM
> >  > > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> >  > >
> >  > > On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
> >  > > > "Burakov, Anatoly" < <mailto:anatoly.burakov@intel.com> anatoly.burakov@intel.com> wrote on 10/27/2017
> >  > > > 04:06:44 PM:
> >  > > >
> >  > > > Â > From: "Burakov, Anatoly" < <mailto:anatoly.burakov@intel.com> anatoly.burakov@intel.com>
> >  > > > Â > To: Jonas Pfefferle1 < <mailto:JPF@zurich.ibm.com> JPF@zurich.ibm.com>,  <mailto:dev@dpdk.org> dev@dpdk.org
> >  > > > Â > Cc:  <mailto:chaozhu@linux.vnet.ibm.com> chaozhu@linux.vnet.ibm.com,  <mailto:bruce.richardson@intel.com> bruce.richardson@intel.com
> >  > > > Â > Date: 10/27/2017 04:06 PM
> >  > > > Â > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> >  > > > Â >
> >  > > > Â > On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
> >  > > > Â > >
> >  > > > Â > >
> >  > > > Â > > Hi @all,
> >  > > > Â > >
> >  > > > Â > > I'm trying to make sense of the hugepage memory mappings in
> >  > > > Â > > librte_eal/linuxapp/eal/eal_memory.c:
> >  > > > Â > > * In rte_eal_hugepage_attach (line 1347) when we try to do a
> >  > private
> >  > > > Â > > mapping on /dev/zero (line 1393) why do we not use MAP_FIXED 
> > if we
> >  >
> >  > > > need the
> >  > > > Â > > addresses to be identical with the primary process?
> >  > > > Â > > * On POWER we have this weird business going on where we use
> >  > > > MAP_HUGETLB
> >  > > > Â > > because according to this commit:
> >  > > > Â > >
> >  > > > Â > > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
> >  > > > Â > > Author: Chao Zhu < <mailto:chaozhu@linux.vnet.ibm.com> chaozhu@linux.vnet.ibm.com>
> >  > > > Â > > Date: Â  Thu Apr 6 15:36:09 2017 +0530
> >  > > > Â > >
> >  > > > Â > > Â  Â  Â eal/ppc: fix mmap for memory initialization
> >  > > > Â > >
> >  > > > Â > > Â  Â  Â On IBM POWER platform, when mapping /dev/zero file to
> >  > hugepage
> >  > > > memory
> >  > > > Â > > Â  Â  Â space, mmap will not respect the requested address 
> > hint.This
> >  > will
> >  > > > Â > > cause
> >  > > > Â > > Â  Â  Â the memory initialization for the second 
> process fails. 
> > This
> >  > > > patch adds
> >  > > > Â > > Â  Â  Â the required mmap flags to make it work. 
> Beside this, users
> >  > > > need to set
> >  > > > Â > > Â  Â  Â the nr_overcommit_hugepages to expand the VA 
> range. When
> >  > > > Â > > Â  Â  Â doing the initialization, users need to set both 
> > nr_hugepages
> >  > and
> >  > > > Â > > Â  Â  Â nr_overcommit_hugepages to the same value, like 64, 
> > 128, etc.
> >  > > > Â > >
> >  > > > Â > > mmap address hints are not respected. Looking at the mmap 
> > code in
> >  > the
> >  > > > Â > > kernel this is not true entirely however under some 
> > circumstances
> >  > > > the hint
> >  > > > Â > > can be ignored (
> >  > > > Â > >  <https://urldefense.proofpoint.com/v2/url?> https://urldefense.proofpoint.com/v2/url?
> >  > > > Â >
> >  > > >
> >  > >
> >  > 
> > 
> u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-
> >  >
> >  > > > Â > siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> >  > > > Â > pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
> >  > > > Â >
> >  > BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
> >  > > > Â > > ). However I believe we can remove the extra case forPPC if we
> >  > use
> >  > > > Â > > MAP_FIXED when doing the secondary process mappings because we
> >  > need
> >  > > > them to
> >  > > > Â > > be identical anyway. We could also use MAP_FIXED whendoing the
> >  > primary
> >  > > > Â > > process mappings resp. get_virtual_area if we want to have any
> >  > > > guarantees
> >  > > > Â > > when specifying a base address. Any thoughts?
> >  > > > Â > >
> >  > > > Â > > Thanks,
> >  > > > Â > > Jonas
> >  > > > Â > >
> >  > > > Â > hi Jonas,
> >  > > > Â >
> >  > > > Â > MAP_FIXED is not used because it's dangerous, it unmaps anything
> >  > that is
> >  > > > Â > already mapped into that space. We would rather know 
> that we can't
> >  > map
> >  > > > Â > something than unwittingly unmap something that was 
> mapped before.
> >  > > >
> >  > > > Ok, I see. Maybe we can add a check to the primary process's memory
> >  > > > mappings whether the hint has been respected or not? At least warn if
> >  > it
> >  > > > hasn't.
> >  > >
> >  > > Hi Jonas,
> >  > >
> >  > > I'm unfamiliar with POWER platform, so i'm afraid you'd have to explain
> >  > > a bit more what you mean by "hint has been respected" :)
> >  >
> >  > Hi Anatoly,
> >  >
> >  > What I meant was the mmap address hint:
> >  >
> >  > "If addr is not NULL, then the kernel takes it as a hint
> >  > Â about where to place the mapping; on Linux, the mapping will be
> >  > Â created at a nearby page boundary."
> >  >
> >  > This is actually not true on POWER. It can happen that the address 
> > hint is
> >  > ignored and you get any address back that fits your mapping.
> >  >
> >  > Thanks,
> >  > Jonas
> > 
> > Actually looking through the kernel code this is also not guaranteed on x86.
> > ( <https://urldefense.proofpoint.com/v2/url?> https://urldefense.proofpoint.com/v2/url?
> u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_x86_kernel_sys-5Fx86-5F64.c-23L165&d=DwID-
> g&c=jf_iaSHvJObTbx-siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> pXjigIjRW0&m=iqakzG7nSXLfvDHyS9IV5E9DWPnNcv19zcsl3MKMdvI&s=VqzZpcTaCUMmNieZ3WyUw-
> jsnNP-hAcW487Mumv6xPw&e=)
> > 
> > So in any case the address hint can be ignored by the kernel and you get 
> > any address that fits your mapping.
> > My suggestion is to check when we do the initial mapping in 
> > get_virtual_area if the hint was respected or not, i.e. if the returned 
> > address == PAGE_ALIGN(address_hint).
> > 
> 
> I'm not sure i see the issue here. So, just to make sure i understand 
> things correctly:
> 
> Whenever we don't request a specific base address through base_address 
> EAL parameter, none of this matters - we always ask for memory in 
> arbitrary memory locations, correct?
> 
> It's also not an issue with secondary processes because we do check 
> returned mmap address to see whether it's the same as we requested, correct?
> 
> It's only whenever we *do* specify a base_address, we provide an address 
> hint to mmap to, but we don't check if the address we got from mmap is 
> one in the vicinity of our requested base address, correct? We don't 
> check, and the kernel can ignore address hint, so we're not guaranteed 
> to respect the base_address flag.
> 
> I'm not sure this is a serious issue, because as far as i'm concerned, 
> this flag is advisory - we only promise to *attempt* to map things at 
> that particular address, not that it will succeed. If the kernel simply 
> cannot find an address to satisfy our address hint, or ignores it for 
> other reasons - well, tough, nothing we can do about that. I'm not sure 
> putting a check like this, where we can't even predict an "expected" 
> address is a good idea.
> 
> Am i getting this right?

The problem is when we specify a base address we want it to be used. If it is
not respected we basically end up with the case like we would have never specified it.
This very likely leads to not being able to run a secondary process because
we will not be able to map the addresses from our primary process and that is why we
introduced the base address parameter in the first place.

> 
> -- 
> Thanks,
> Anatoly
> 

The reason why I put the patch there is that when mapping hugepage on POWER, the kernel will never respect the address hints when doing mmap unless we expand the address space or unmap all the hugepages.  This is a big difference when compared with x86. And it affects the mapping of  the secondary process. I agree that the hints is advisory. Just want to see if there are better solutions.

Re: [dpdk-dev] Huge mapping secondary process linux

[Jonas Pfefferle1]

"Chao Zhu" <chaozhu@linux.vnet.ibm.com> wrote on 11/07/2017 09:25:26 AM:

> From: "Chao Zhu" <chaozhu@linux.vnet.ibm.com>
> To: "'Jonas Pfefferle1'" <JPF@zurich.ibm.com>, "'Burakov, Anatoly'"
> <anatoly.burakov@intel.com>
> Cc: <bruce.richardson@intel.com>, <dev@dpdk.org>
> Date: 11/07/2017 11:00 AM
> Subject: RE: [dpdk-dev] Huge mapping secondary process linux
>
>
>
> From: Jonas Pfefferle1 [mailto:JPF@zurich.ibm.com]
> Sent: 2017年10月28日 3:23
> To: Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: bruce.richardson@intel.com; chaozhu@linux.vnet.ibm.com; dev@dpdk.org
> Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>
> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 27/10/2017
18:00:27:
>
> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com,
dev@dpdk.org
> > Date: 27/10/2017 18:00
> > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> >
> > On 27-Oct-17 4:16 PM, Jonas Pfefferle1 wrote:
> > > "dev" <dev-bounces@dpdk.org> wrote on 10/27/2017 04:58:01 PM:
> > >
> > >  > From: "Jonas Pfefferle1" <JPF@zurich.ibm.com>
> > >  > To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > >  > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com,
> dev@dpdk.org
> > >  > Date: 10/27/2017 04:58 PM
> > >  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > >  > Sent by: "dev" <dev-bounces@dpdk.org>
> > >  >
> > >  >
> > >  > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017

> > > 04:44:52
> > >  > PM:
> > >  >
> > >  > > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > >  > > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> > >  > > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com,
> > > dev@dpdk.org
> > >  > > Date: 10/27/2017 04:45 PM
> > >  > > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > >  > >
> > >  > > On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
> > >  > > > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on
10/27/2017
> > >  > > > 04:06:44 PM:
> > >  > > >
> > >  > > > Â > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > >  > > > Â > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
> > >  > > > Â > Cc: chaozhu@linux.vnet.ibm.com, bruce.richardson@intel.com
> > >  > > > Â > Date: 10/27/2017 04:06 PM
> > >  > > > Â > Subject: Re: [dpdk-dev] Huge mapping secondary process
linux
> > >  > > > Â >
> > >  > > > Â > On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
> > >  > > > Â > >
> > >  > > > Â > >
> > >  > > > Â > > Hi @all,
> > >  > > > Â > >
> > >  > > > Â > > I'm trying to make sense of the hugepage memory mappings
in
> > >  > > > Â > > librte_eal/linuxapp/eal/eal_memory.c:
> > >  > > > Â > > * In rte_eal_hugepage_attach (line 1347) when we try to
do a
> > >  > private
> > >  > > > Â > > mapping on /dev/zero (line 1393) why do we not use
MAP_FIXED
> > > if we
> > >  >
> > >  > > > need the
> > >  > > > Â > > addresses to be identical with the primary process?
> > >  > > > Â > > * On POWER we have this weird business going on where we
use
> > >  > > > MAP_HUGETLB
> > >  > > > Â > > because according to this commit:
> > >  > > > Â > >
> > >  > > > Â > > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
> > >  > > > Â > > Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> > >  > > > Â > > Date: Â  Thu Apr 6 15:36:09 2017 +0530
> > >  > > > Â > >
> > >  > > > Â > > Â  Â  Â eal/ppc: fix mmap for memory initialization
> > >  > > > Â > >
> > >  > > > Â > > Â  Â  Â On IBM POWER platform, when mapping /dev/zero
file to
> > >  > hugepage
> > >  > > > memory
> > >  > > > Â > > Â  Â  Â space, mmap will not respect the requested
address
> > > hint.This
> > >  > will
> > >  > > > Â > > cause
> > >  > > > Â > > Â  Â  Â the memory initialization for the second
> > process fails.
> > > This
> > >  > > > patch adds
> > >  > > > Â > > Â  Â  Â the required mmap flags to make it work.
> > Beside this, users
> > >  > > > need to set
> > >  > > > Â > > Â  Â  Â the nr_overcommit_hugepages to expand the VA
> > range. When
> > >  > > > Â > > Â  Â  Â doing the initialization, users need to set both

> > > nr_hugepages
> > >  > and
> > >  > > > Â > > Â  Â  Â nr_overcommit_hugepages to the same value, like
64,
> > > 128, etc.
> > >  > > > Â > >
> > >  > > > Â > > mmap address hints are not respected. Looking at the
mmap
> > > code in
> > >  > the
> > >  > > > Â > > kernel this is not true entirely however under some
> > > circumstances
> > >  > > > the hint
> > >  > > > Â > > can be ignored (
> > >  > > > Â > > https://urldefense.proofpoint.com/v2/url?
> > >  > > > Â >
> > >  > > >
> > >  > >
> > >  >
> > >
> >
>
u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-

> > >  >
> > >  > > > Â > siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> > >  > > > Â > pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
> > >  > > > Â >
> > >  >
> BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
> > >  > > > Â > > ). However I believe we can remove the extra case
> forPPC if we
> > >  > use
> > >  > > > Â > > MAP_FIXED when doing the secondary process
mappingsbecause we
> > >  > need
> > >  > > > them to
> > >  > > > Â > > be identical anyway. We could also use MAP_FIXED
> whendoing the
> > >  > primary
> > >  > > > Â > > process mappings resp. get_virtual_area if we want
> to have any
> > >  > > > guarantees
> > >  > > > Â > > when specifying a base address. Any thoughts?
> > >  > > > Â > >
> > >  > > > Â > > Thanks,
> > >  > > > Â > > Jonas
> > >  > > > Â > >
> > >  > > > Â > hi Jonas,
> > >  > > > Â >
> > >  > > > Â > MAP_FIXED is not used because it's dangerous, it
> unmaps anything
> > >  > that is
> > >  > > > Â > already mapped into that space. We would rather know
> > that we can't
> > >  > map
> > >  > > > Â > something than unwittingly unmap something that was
> > mapped before.
> > >  > > >
> > >  > > > Ok, I see. Maybe we can add a check to the primary process's
memory
> > >  > > > mappings whether the hint has been respected or not? At
> least warn if
> > >  > it
> > >  > > > hasn't.
> > >  > >
> > >  > > Hi Jonas,
> > >  > >
> > >  > > I'm unfamiliar with POWER platform, so i'm afraid you'd
> have to explain
> > >  > > a bit more what you mean by "hint has been respected" :)
> > >  >
> > >  > Hi Anatoly,
> > >  >
> > >  > What I meant was the mmap address hint:
> > >  >
> > >  > "If addr is not NULL, then the kernel takes it as a hint
> > >  > Â about where to place the mapping; on Linux, the mapping will be
> > >  > Â created at a nearby page boundary."
> > >  >
> > >  > This is actually not true on POWER. It can happen that the address

> > > hint is
> > >  > ignored and you get any address back that fits your mapping.
> > >  >
> > >  > Thanks,
> > >  > Jonas
> > >
> > > Actually looking through the kernel code this is also not
> guaranteed on x86.
> > > (https://urldefense.proofpoint.com/v2/url?
> >
>
u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_x86_kernel_sys-5Fx86-5F64.c-23L165&d=DwID-

> > g&c=jf_iaSHvJObTbx-siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> >
>
pXjigIjRW0&m=iqakzG7nSXLfvDHyS9IV5E9DWPnNcv19zcsl3MKMdvI&s=VqzZpcTaCUMmNieZ3WyUw-

> > jsnNP-hAcW487Mumv6xPw&e=)
> > >
> > > So in any case the address hint can be ignored by the kernel and you
get
> > > any address that fits your mapping.
> > > My suggestion is to check when we do the initial mapping in
> > > get_virtual_area if the hint was respected or not, i.e. if the
returned
> > > address == PAGE_ALIGN(address_hint).
> > >
> >
> > I'm not sure i see the issue here. So, just to make sure i understand
> > things correctly:
> >
> > Whenever we don't request a specific base address through base_address
> > EAL parameter, none of this matters - we always ask for memory in
> > arbitrary memory locations, correct?
> >
> > It's also not an issue with secondary processes because we do check
> > returned mmap address to see whether it's the same as we requested,
correct?
> >
> > It's only whenever we *do* specify a base_address, we provide an
address
> > hint to mmap to, but we don't check if the address we got from mmap is
> > one in the vicinity of our requested base address, correct? We don't
> > check, and the kernel can ignore address hint, so we're not guaranteed
> > to respect the base_address flag.
> >
> > I'm not sure this is a serious issue, because as far as i'm concerned,
> > this flag is advisory - we only promise to *attempt* to map things at
> > that particular address, not that it will succeed. If the kernel simply

> > cannot find an address to satisfy our address hint, or ignores it for
> > other reasons - well, tough, nothing we can do about that. I'm not sure

> > putting a check like this, where we can't even predict an "expected"
> > address is a good idea.
> >
> > Am i getting this right?
>
> The problem is when we specify a base address we want it to be used. If
it is
> not respected we basically end up with the case like we would have
> never specified it.
> This very likely leads to not being able to run a secondary process
because
> we will not be able to map the addresses from our primary process
> and that is why we
> introduced the base address parameter in the first place.
>
> >
> > --
> > Thanks,
> > Anatoly
> >
> The reason why I put the patch there is that when mapping hugepage
> on POWER, the kernel will never respect the address hints when doing
> mmap unless we expand the address space or unmap all the hugepages.
> This is a big difference when compared with x86. And it affects the
> mapping of  the secondary process. I agree that the hints is
> advisory. Just want to see if there are better solutions.


This is not true. I looked through the kernel code and the address
hint is treated almost the same on both platforms:

PPC:
https://elixir.free-electrons.com/linux/latest/source/arch/powerpc/mm/mmap.c#L143
Line 169/170

x86:
https://elixir.free-electrons.com/linux/latest/source/arch/x86/kernel/sys_x86_64.c#L165
Line 189/190

The only thing that might differ is the virtual address layout
(e.g. due to different page size etc) and that might lead to the same
value for base-virtaddr not working on both x86 and POWER.
However I tested with different address hints and you easily can
find addresses where the address hint is indeed respected.
That is also why I send in a patch to remove the HUGETLB flags on
the mmap.

Thanks,
Jonas

Re: [dpdk-dev] Huge mapping secondary process linux

[Chao Zhu]

 

 

From: Jonas Pfefferle1 [mailto:JPF@zurich.ibm.com] 
Sent: 2017年11月7日 18:16
To: Chao Zhu <chaozhu@linux.vnet.ibm.com>
Cc: 'Burakov, Anatoly' <anatoly.burakov@intel.com>; bruce.richardson@intel.com; dev@dpdk.org
Subject: RE: [dpdk-dev] Huge mapping secondary process linux

 

"Chao Zhu" <chaozhu@linux.vnet.ibm.com <mailto:chaozhu@linux.vnet.ibm.com> > wrote on 11/07/2017 09:25:26 AM:

> From: "Chao Zhu" <chaozhu@linux.vnet.ibm.com <mailto:chaozhu@linux.vnet.ibm.com> >
> To: "'Jonas Pfefferle1'" <JPF@zurich.ibm.com <mailto:JPF@zurich.ibm.com> >, "'Burakov, Anatoly'" 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> >
> Cc: <bruce.richardson@intel.com <mailto:bruce.richardson@intel.com> >, <dev@dpdk.org <mailto:dev@dpdk.org> >
> Date: 11/07/2017 11:00 AM
> Subject: RE: [dpdk-dev] Huge mapping secondary process linux
> 
>  
>  
> From: Jonas Pfefferle1 [mailto:JPF@zurich.ibm.com] 
> Sent: 2017年10月28日 3:23
> To: Burakov, Anatoly <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> >
> Cc: bruce.richardson@intel.com <mailto:bruce.richardson@intel.com> ; chaozhu@linux.vnet.ibm.com <mailto:chaozhu@linux.vnet.ibm.com> ; dev@dpdk.org <mailto:dev@dpdk.org> 
> Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>  
> "Burakov, Anatoly" <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> > wrote on 27/10/2017 18:00:27:
> 
> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> >
> > To: Jonas Pfefferle1 <JPF@zurich.ibm.com <mailto:JPF@zurich.ibm.com> >
> > Cc: bruce.richardson@intel.com <mailto:bruce.richardson@intel.com> , chaozhu@linux.vnet.ibm.com <mailto:chaozhu@linux.vnet.ibm.com> , dev@dpdk.org <mailto:dev@dpdk.org> 
> > Date: 27/10/2017 18:00
> > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > 
> > On 27-Oct-17 4:16 PM, Jonas Pfefferle1 wrote:
> > > "dev" <dev-bounces@dpdk.org <mailto:dev-bounces@dpdk.org> > wrote on 10/27/2017 04:58:01 PM:
> > > 
> > >  > From: "Jonas Pfefferle1" <JPF@zurich.ibm.com <mailto:JPF@zurich.ibm.com> >
> > >  > To: "Burakov, Anatoly" <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> >
> > >  > Cc: bruce.richardson@intel.com <mailto:bruce.richardson@intel.com> , chaozhu@linux.vnet.ibm.com <mailto:chaozhu@linux.vnet.ibm.com> , 
> dev@dpdk.org <mailto:dev@dpdk.org> 
> > >  > Date: 10/27/2017 04:58 PM
> > >  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > >  > Sent by: "dev" <dev-bounces@dpdk.org <mailto:dev-bounces@dpdk.org> >
> > >  >
> > >  >
> > >  > "Burakov, Anatoly" <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> > wrote on 10/27/2017 
> > > 04:44:52
> > >  > PM:
> > >  >
> > >  > > From: "Burakov, Anatoly" <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> >
> > >  > > To: Jonas Pfefferle1 <JPF@zurich.ibm.com <mailto:JPF@zurich.ibm.com> >
> > >  > > Cc: bruce.richardson@intel.com <mailto:bruce.richardson@intel.com> , chaozhu@linux.vnet.ibm.com <mailto:chaozhu@linux.vnet.ibm.com> , 
> > > dev@dpdk.org <mailto:dev@dpdk.org> 
> > >  > > Date: 10/27/2017 04:45 PM
> > >  > > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > >  > >
> > >  > > On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
> > >  > > > "Burakov, Anatoly" <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> > wrote on 10/27/2017
> > >  > > > 04:06:44 PM:
> > >  > > >
> > >  > > > Â > From: "Burakov, Anatoly" <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com> >
> > >  > > > Â > To: Jonas Pfefferle1 <JPF@zurich.ibm.com <mailto:JPF@zurich.ibm.com> >, dev@dpdk.org <mailto:dev@dpdk.org> 
> > >  > > > Â > Cc: chaozhu@linux.vnet.ibm.com <mailto:chaozhu@linux.vnet.ibm.com> , bruce.richardson@intel.com <mailto:bruce.richardson@intel.com> 
> > >  > > > Â > Date: 10/27/2017 04:06 PM
> > >  > > > Â > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > >  > > > Â >
> > >  > > > Â > On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
> > >  > > > Â > >
> > >  > > > Â > >
> > >  > > > Â > > Hi @all,
> > >  > > > Â > >
> > >  > > > Â > > I'm trying to make sense of the hugepage memory mappings in
> > >  > > > Â > > librte_eal/linuxapp/eal/eal_memory.c:
> > >  > > > Â > > * In rte_eal_hugepage_attach (line 1347) when we try to do a
> > >  > private
> > >  > > > Â > > mapping on /dev/zero (line 1393) why do we not use MAP_FIXED 
> > > if we
> > >  >
> > >  > > > need the
> > >  > > > Â > > addresses to be identical with the primary process?
> > >  > > > Â > > * On POWER we have this weird business going on where we use
> > >  > > > MAP_HUGETLB
> > >  > > > Â > > because according to this commit:
> > >  > > > Â > >
> > >  > > > Â > > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
> > >  > > > Â > > Author: Chao Zhu <chaozhu@linux.vnet.ibm.com <mailto:chaozhu@linux.vnet.ibm.com> >
> > >  > > > Â > > Date: Â  Thu Apr 6 15:36:09 2017 +0530
> > >  > > > Â > >
> > >  > > > Â > > Â  Â  Â eal/ppc: fix mmap for memory initialization
> > >  > > > Â > >
> > >  > > > Â > > Â  Â  Â On IBM POWER platform, when mapping /dev/zero file to
> > >  > hugepage
> > >  > > > memory
> > >  > > > Â > > Â  Â  Â space, mmap will not respect the requested address 
> > > hint.This
> > >  > will
> > >  > > > Â > > cause
> > >  > > > Â > > Â  Â  Â the memory initialization for the second 
> > process fails. 
> > > This
> > >  > > > patch adds
> > >  > > > Â > > Â  Â  Â the required mmap flags to make it work. 
> > Beside this, users
> > >  > > > need to set
> > >  > > > Â > > Â  Â  Â the nr_overcommit_hugepages to expand the VA 
> > range. When
> > >  > > > Â > > Â  Â  Â doing the initialization, users need to set both 
> > > nr_hugepages
> > >  > and
> > >  > > > Â > > Â  Â  Â nr_overcommit_hugepages to the same value, like 64, 
> > > 128, etc.
> > >  > > > Â > >
> > >  > > > Â > > mmap address hints are not respected. Looking at the mmap 
> > > code in
> > >  > the
> > >  > > > Â > > kernel this is not true entirely however under some 
> > > circumstances
> > >  > > > the hint
> > >  > > > Â > > can be ignored (
> > >  > > > Â > > https://urldefense.proofpoint.com/v2/url?
> > >  > > > Â >
> > >  > > >
> > >  > >
> > >  > 
> > > 
> > 
> u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-
> > >  >
> > >  > > > Â > siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> > >  > > > Â > pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
> > >  > > > Â >
> > >  > 
> BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
> > >  > > > Â > > ). However I believe we can remove the extra case 
> forPPC if we
> > >  > use
> > >  > > > Â > > MAP_FIXED when doing the secondary process mappingsbecause we
> > >  > need
> > >  > > > them to
> > >  > > > Â > > be identical anyway. We could also use MAP_FIXED 
> whendoing the
> > >  > primary
> > >  > > > Â > > process mappings resp. get_virtual_area if we want 
> to have any
> > >  > > > guarantees
> > >  > > > Â > > when specifying a base address. Any thoughts?
> > >  > > > Â > >
> > >  > > > Â > > Thanks,
> > >  > > > Â > > Jonas
> > >  > > > Â > >
> > >  > > > Â > hi Jonas,
> > >  > > > Â >
> > >  > > > Â > MAP_FIXED is not used because it's dangerous, it 
> unmaps anything
> > >  > that is
> > >  > > > Â > already mapped into that space. We would rather know 
> > that we can't
> > >  > map
> > >  > > > Â > something than unwittingly unmap something that was 
> > mapped before.
> > >  > > >
> > >  > > > Ok, I see. Maybe we can add a check to the primary process's memory
> > >  > > > mappings whether the hint has been respected or not? At 
> least warn if
> > >  > it
> > >  > > > hasn't.
> > >  > >
> > >  > > Hi Jonas,
> > >  > >
> > >  > > I'm unfamiliar with POWER platform, so i'm afraid you'd 
> have to explain
> > >  > > a bit more what you mean by "hint has been respected" :)
> > >  >
> > >  > Hi Anatoly,
> > >  >
> > >  > What I meant was the mmap address hint:
> > >  >
> > >  > "If addr is not NULL, then the kernel takes it as a hint
> > >  > Â about where to place the mapping; on Linux, the mapping will be
> > >  > Â created at a nearby page boundary."
> > >  >
> > >  > This is actually not true on POWER. It can happen that the address 
> > > hint is
> > >  > ignored and you get any address back that fits your mapping.
> > >  >
> > >  > Thanks,
> > >  > Jonas
> > > 
> > > Actually looking through the kernel code this is also not 
> guaranteed on x86.
> > > (https://urldefense.proofpoint.com/v2/url?
> > 
> u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_x86_kernel_sys-5Fx86-5F64.c-23L165&d=DwID-
> > g&c=jf_iaSHvJObTbx-siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> > 
> pXjigIjRW0&m=iqakzG7nSXLfvDHyS9IV5E9DWPnNcv19zcsl3MKMdvI&s=VqzZpcTaCUMmNieZ3WyUw-
> > jsnNP-hAcW487Mumv6xPw&e=)
> > > 
> > > So in any case the address hint can be ignored by the kernel and you get 
> > > any address that fits your mapping.
> > > My suggestion is to check when we do the initial mapping in 
> > > get_virtual_area if the hint was respected or not, i.e. if the returned 
> > > address == PAGE_ALIGN(address_hint).
> > > 
> > 
> > I'm not sure i see the issue here. So, just to make sure i understand 
> > things correctly:
> > 
> > Whenever we don't request a specific base address through base_address 
> > EAL parameter, none of this matters - we always ask for memory in 
> > arbitrary memory locations, correct?
> > 
> > It's also not an issue with secondary processes because we do check 
> > returned mmap address to see whether it's the same as we requested, correct?
> > 
> > It's only whenever we *do* specify a base_address, we provide an address 
> > hint to mmap to, but we don't check if the address we got from mmap is 
> > one in the vicinity of our requested base address, correct? We don't 
> > check, and the kernel can ignore address hint, so we're not guaranteed 
> > to respect the base_address flag.
> > 
> > I'm not sure this is a serious issue, because as far as i'm concerned, 
> > this flag is advisory - we only promise to *attempt* to map things at 
> > that particular address, not that it will succeed. If the kernel simply 
> > cannot find an address to satisfy our address hint, or ignores it for 
> > other reasons - well, tough, nothing we can do about that. I'm not sure 
> > putting a check like this, where we can't even predict an "expected" 
> > address is a good idea.
> > 
> > Am i getting this right?
> 
> The problem is when we specify a base address we want it to be used. If it is
> not respected we basically end up with the case like we would have 
> never specified it.
> This very likely leads to not being able to run a secondary process because
> we will not be able to map the addresses from our primary process 
> and that is why we
> introduced the base address parameter in the first place.
> 
> > 
> > -- 
> > Thanks,
> > Anatoly
> > 
> The reason why I put the patch there is that when mapping hugepage 
> on POWER, the kernel will never respect the address hints when doing
> mmap unless we expand the address space or unmap all the hugepages. 
> This is a big difference when compared with x86. And it affects the 
> mapping of  the secondary process. I agree that the hints is 
> advisory. Just want to see if there are better solutions.


This is not true. I looked through the kernel code and the address
hint is treated almost the same on both platforms: 

PPC:  <https://elixir.free-electrons.com/linux/latest/source/arch/powerpc/mm/mmap.c#L143> https://elixir.free-electrons.com/linux/latest/source/arch/powerpc/mm/mmap.c#L143
Line 169/170

x86:  <https://elixir.free-electrons.com/linux/latest/source/arch/x86/kernel/sys_x86_64.c#L165> https://elixir.free-electrons.com/linux/latest/source/arch/x86/kernel/sys_x86_64.c#L165
Line 189/190

The only thing that might differ is the virtual address layout
(e.g. due to different page size etc) and that might lead to the same 
value for base-virtaddr not working on both x86 and POWER.
However I tested with different address hints and you easily can
find addresses where the address hint is indeed respected. 
That is also why I send in a patch to remove the HUGETLB flags on
the mmap.

Thanks,
Jonas

You can take a look at this. https://bugzilla.linux.ibm.com/show_bug.cgi?id=141628

It’s quite interesting.

Re: [dpdk-dev] Huge mapping secondary process linux

[Jonas Pfefferle1]

"Chao Zhu" <chaozhu@linux.vnet.ibm.com> wrote on 11/09/2017 04:08:36 AM:

> From: "Chao Zhu" <chaozhu@linux.vnet.ibm.com>
> To: "'Jonas Pfefferle1'" <JPF@zurich.ibm.com>
> Cc: "'Burakov, Anatoly'" <anatoly.burakov@intel.com>,
> <bruce.richardson@intel.com>, <dev@dpdk.org>
> Date: 11/09/2017 04:08 AM
> Subject: RE: [dpdk-dev] Huge mapping secondary process linux
>
>
>
> From: Jonas Pfefferle1 [mailto:JPF@zurich.ibm.com]
> Sent: 2017年11月7日 18:16
> To: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> Cc: 'Burakov, Anatoly' <anatoly.burakov@intel.com>;
> bruce.richardson@intel.com; dev@dpdk.org
> Subject: RE: [dpdk-dev] Huge mapping secondary process linux
>
> "Chao Zhu" <chaozhu@linux.vnet.ibm.com> wrote on 11/07/2017 09:25:26 AM:
>
> > From: "Chao Zhu" <chaozhu@linux.vnet.ibm.com>
> > To: "'Jonas Pfefferle1'" <JPF@zurich.ibm.com>, "'Burakov, Anatoly'"
> > <anatoly.burakov@intel.com>
> > Cc: <bruce.richardson@intel.com>, <dev@dpdk.org>
> > Date: 11/07/2017 11:00 AM
> > Subject: RE: [dpdk-dev] Huge mapping secondary process linux
> >
> >
> >
> > From: Jonas Pfefferle1 [mailto:JPF@zurich.ibm.com]
> > Sent: 2017年10月28日 3:23
> > To: Burakov, Anatoly <anatoly.burakov@intel.com>
> > Cc: bruce.richardson@intel.com; chaozhu@linux.vnet.ibm.com;
dev@dpdk.org
> > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> >
> > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on
27/10/201718:00:27:
> >
> > > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> > > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com,
dev@dpdk.org
> > > Date: 27/10/2017 18:00
> > > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > >
> > > On 27-Oct-17 4:16 PM, Jonas Pfefferle1 wrote:
> > > > "dev" <dev-bounces@dpdk.org> wrote on 10/27/2017 04:58:01 PM:
> > > >
> > > >  > From: "Jonas Pfefferle1" <JPF@zurich.ibm.com>
> > > >  > To: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > > >  > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com,
> > dev@dpdk.org
> > > >  > Date: 10/27/2017 04:58 PM
> > > >  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > > >  > Sent by: "dev" <dev-bounces@dpdk.org>
> > > >  >
> > > >  >
> > > >  > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on
10/27/2017
> > > > 04:44:52
> > > >  > PM:
> > > >  >
> > > >  > > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > > >  > > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>
> > > >  > > Cc: bruce.richardson@intel.com, chaozhu@linux.vnet.ibm.com,
> > > > dev@dpdk.org
> > > >  > > Date: 10/27/2017 04:45 PM
> > > >  > > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
> > > >  > >
> > > >  > > On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
> > > >  > > > "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote
on10/27/2017
> > > >  > > > 04:06:44 PM:
> > > >  > > >
> > > >  > > > Â > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > > >  > > > Â > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
> > > >  > > > Â > Cc: chaozhu@linux.vnet.ibm.com,
bruce.richardson@intel.com
> > > >  > > > Â > Date: 10/27/2017 04:06 PM
> > > >  > > > Â > Subject: Re: [dpdk-dev] Huge mapping secondary process
linux
> > > >  > > > Â >
> > > >  > > > Â > On 27-Oct-17 1:43 PM, Jonas Pfefferle1 wrote:
> > > >  > > > Â > >
> > > >  > > > Â > >
> > > >  > > > Â > > Hi @all,
> > > >  > > > Â > >
> > > >  > > > Â > > I'm trying to make sense of the hugepage memory
mappings in
> > > >  > > > Â > > librte_eal/linuxapp/eal/eal_memory.c:
> > > >  > > > Â > > * In rte_eal_hugepage_attach (line 1347) when we
> try to do a
> > > >  > private
> > > >  > > > Â > > mapping on /dev/zero (line 1393) why do we not
> use MAP_FIXED
> > > > if we
> > > >  >
> > > >  > > > need the
> > > >  > > > Â > > addresses to be identical with the primary process?
> > > >  > > > Â > > * On POWER we have this weird business going on
> where we use
> > > >  > > > MAP_HUGETLB
> > > >  > > > Â > > because according to this commit:
> > > >  > > > Â > >
> > > >  > > > Â > > commit 284ae3e9ff9a92575c28c858efd2c85c8de6d440
> > > >  > > > Â > > Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
> > > >  > > > Â > > Date: Â  Thu Apr 6 15:36:09 2017 +0530
> > > >  > > > Â > >
> > > >  > > > Â > > Â  Â  Â eal/ppc: fix mmap for memory initialization
> > > >  > > > Â > >
> > > >  > > > Â > > Â  Â  Â On IBM POWER platform, when mapping /dev/
> zero file to
> > > >  > hugepage
> > > >  > > > memory
> > > >  > > > Â > > Â  Â  Â space, mmap will not respect the requested
address
> > > > hint.This
> > > >  > will
> > > >  > > > Â > > cause
> > > >  > > > Â > > Â  Â  Â the memory initialization for the second
> > > process fails.
> > > > This
> > > >  > > > patch adds
> > > >  > > > Â > > Â  Â  Â the required mmap flags to make it work.
> > > Beside this, users
> > > >  > > > need to set
> > > >  > > > Â > > Â  Â  Â the nr_overcommit_hugepages to expand the VA
> > > range. When
> > > >  > > > Â > > Â  Â  Â doing the initialization, users need to set
both
> > > > nr_hugepages
> > > >  > and
> > > >  > > > Â > > Â  Â  Â nr_overcommit_hugepages to the same
> value, like 64,
> > > > 128, etc.
> > > >  > > > Â > >
> > > >  > > > Â > > mmap address hints are not respected. Looking at the
mmap
> > > > code in
> > > >  > the
> > > >  > > > Â > > kernel this is not true entirely however under some
> > > > circumstances
> > > >  > > > the hint
> > > >  > > > Â > > can be ignored (
> > > >  > > > Â > > https://urldefense.proofpoint.com/v2/url?
> > > >  > > > Â >
> > > >  > > >
> > > >  > >
> > > >  >
> > > >
> > >
> >
>
u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_powerpc_mm_mmap.c-23L103&d=DwICaQ&c=jf_iaSHvJObTbx-

> > > >  >
> > > >  > > > Â > siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> > > >  > > > Â > pXjigIjRW0&m=cttQcHlAYixhsYS3lz-
> > > >  > > > Â >
> > > >  >
> >
BAdEeg4dpbwGdPnj2R3I8Do0&s=Gp0TIjUtIed05Jgb7XnlocpCYZdFXZXiH0LqIWiNMhA&e=
> > > >  > > > Â > > ). However I believe we can remove the extra case
> > forPPC if we
> > > >  > use
> > > >  > > > Â > > MAP_FIXED when doing the secondary process
> mappingsbecause we
> > > >  > need
> > > >  > > > them to
> > > >  > > > Â > > be identical anyway. We could also use MAP_FIXED
> > whendoing the
> > > >  > primary
> > > >  > > > Â > > process mappings resp. get_virtual_area if we want
> > to have any
> > > >  > > > guarantees
> > > >  > > > Â > > when specifying a base address. Any thoughts?
> > > >  > > > Â > >
> > > >  > > > Â > > Thanks,
> > > >  > > > Â > > Jonas
> > > >  > > > Â > >
> > > >  > > > Â > hi Jonas,
> > > >  > > > Â >
> > > >  > > > Â > MAP_FIXED is not used because it's dangerous, it
> > unmaps anything
> > > >  > that is
> > > >  > > > Â > already mapped into that space. We would rather know
> > > that we can't
> > > >  > map
> > > >  > > > Â > something than unwittingly unmap something that was
> > > mapped before.
> > > >  > > >
> > > >  > > > Ok, I see. Maybe we can add a check to the primary
> process's memory
> > > >  > > > mappings whether the hint has been respected or not? At
> > least warn if
> > > >  > it
> > > >  > > > hasn't.
> > > >  > >
> > > >  > > Hi Jonas,
> > > >  > >
> > > >  > > I'm unfamiliar with POWER platform, so i'm afraid you'd
> > have to explain
> > > >  > > a bit more what you mean by "hint has been respected" :)
> > > >  >
> > > >  > Hi Anatoly,
> > > >  >
> > > >  > What I meant was the mmap address hint:
> > > >  >
> > > >  > "If addr is not NULL, then the kernel takes it as a hint
> > > >  > Â about where to place the mapping; on Linux, the mapping will
be
> > > >  > Â created at a nearby page boundary."
> > > >  >
> > > >  > This is actually not true on POWER. It can happen that the
address
> > > > hint is
> > > >  > ignored and you get any address back that fits your mapping.
> > > >  >
> > > >  > Thanks,
> > > >  > Jonas
> > > >
> > > > Actually looking through the kernel code this is also not
> > guaranteed on x86.
> > > > (https://urldefense.proofpoint.com/v2/url?
> > >
> >
>
u=http-3A__elixir.free-2Delectrons.com_linux_latest_source_arch_x86_kernel_sys-5Fx86-5F64.c-23L165&d=DwID-

> > > g&c=jf_iaSHvJObTbx-siA1ZOg&r=rOdXhRsgn8Iur7bDE0vgwvo6TC8OpoDN-
> > >
> >
>
pXjigIjRW0&m=iqakzG7nSXLfvDHyS9IV5E9DWPnNcv19zcsl3MKMdvI&s=VqzZpcTaCUMmNieZ3WyUw-

> > > jsnNP-hAcW487Mumv6xPw&e=)
> > > >
> > > > So in any case the address hint can be ignored by the kernel
> and you get
> > > > any address that fits your mapping.
> > > > My suggestion is to check when we do the initial mapping in
> > > > get_virtual_area if the hint was respected or not, i.e. if
thereturned
> > > > address == PAGE_ALIGN(address_hint).
> > > >
> > >
> > > I'm not sure i see the issue here. So, just to make sure i understand

> > > things correctly:
> > >
> > > Whenever we don't request a specific base address through
base_address
> > > EAL parameter, none of this matters - we always ask for memory in
> > > arbitrary memory locations, correct?
> > >
> > > It's also not an issue with secondary processes because we do check
> > > returned mmap address to see whether it's the same as we
> requested, correct?
> > >
> > > It's only whenever we *do* specify a base_address, we provide an
address
> > > hint to mmap to, but we don't check if the address we got from mmap
is
> > > one in the vicinity of our requested base address, correct? We don't
> > > check, and the kernel can ignore address hint, so we're not
guaranteed
> > > to respect the base_address flag.
> > >
> > > I'm not sure this is a serious issue, because as far as i'm
concerned,
> > > this flag is advisory - we only promise to *attempt* to map things at

> > > that particular address, not that it will succeed. If the kernel
simply
> > > cannot find an address to satisfy our address hint, or ignores it for

> > > other reasons - well, tough, nothing we can do about that. I'm not
sure
> > > putting a check like this, where we can't even predict an "expected"
> > > address is a good idea.
> > >
> > > Am i getting this right?
> >
> > The problem is when we specify a base address we want it to be
> used. If it is
> > not respected we basically end up with the case like we would have
> > never specified it.
> > This very likely leads to not being able to run a secondary process
because
> > we will not be able to map the addresses from our primary process
> > and that is why we
> > introduced the base address parameter in the first place.
> >
> > >
> > > --
> > > Thanks,
> > > Anatoly
> > >
> > The reason why I put the patch there is that when mapping hugepage
> > on POWER, the kernel will never respect the address hints when doing
> > mmap unless we expand the address space or unmap all the hugepages.
> > This is a big difference when compared with x86. And it affects the
> > mapping of  the secondary process. I agree that the hints is
> > advisory. Just want to see if there are better solutions.
>
>
> This is not true. I looked through the kernel code and the address
> hint is treated almost the same on both platforms:
>
> PPC: https://elixir.free-electrons.com/linux/latest/source/arch/
> powerpc/mm/mmap.c#L143
> Line 169/170
>
> x86: https://elixir.free-electrons.com/linux/latest/source/arch/x86/
> kernel/sys_x86_64.c#L165
> Line 189/190
>
> The only thing that might differ is the virtual address layout
> (e.g. due to different page size etc) and that might lead to the same
> value for base-virtaddr not working on both x86 and POWER.
> However I tested with different address hints and you easily can
> find addresses where the address hint is indeed respected.
> That is also why I send in a patch to remove the HUGETLB flags on
> the mmap.
>
> Thanks,
> Jonas
> You can take a look at this. https://bugzilla.linux.ibm.com/
> show_bug.cgi?id=141628
> It’s quite interesting.

Interesting indeed. I misunderstood the problem I thought the
get_virtual_area mmap adress hint is not respected when
the real problem is the address hint when mapping the
hugepages. Still I hope we can find a better solution.
Aside from that I still believe warning on the address
hint being respected or not is a good idea.

Re: [dpdk-dev] Huge mapping secondary process linux

[Tan, Jianfeng]

On 10/27/2017 10:44 PM, Burakov, Anatoly wrote:
> On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
>> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017 
>> 04:06:44 PM:
>>
>>  > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>>  > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
>>  > Cc: chaozhu@linux.vnet.ibm.com, bruce.richardson@intel.com
>>  > Date: 10/27/2017 04:06 PM
>>  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>>  ...
>>  > >
>>  > hi Jonas,
>>  >
>>  > MAP_FIXED is not used because it's dangerous, it unmaps anything 
>> that is
>>  > already mapped into that space. We would rather know that we can't 
>> map
>>  > something than unwittingly unmap something that was mapped before.
>>
>> Ok, I see. Maybe we can add a check to the primary process's memory 
>> mappings whether the hint has been respected or not? At least warn if 
>> it hasn't.
>
> Hi Jonas,
>
> I'm unfamiliar with POWER platform, so i'm afraid you'd have to 
> explain a bit more what you mean by "hint has been respected" :)

Actually, I also met this case on x86 once that kernel does not respect 
the "addr" parameter even that memory region is not occupied. I am not 
sure if it can be reproduced now, anyway, send here FYI: we run primary 
on the host, run secondary in a container.

I'll agree at least we need to check if the final addr is the same of 
the parameter addr, and warn if it's not.

Thanks,
Jianfeng

Re: [dpdk-dev] Huge mapping secondary process linux

[Burakov, Anatoly]

On 27-Oct-17 4:48 PM, Tan, Jianfeng wrote:
> 
> 
> On 10/27/2017 10:44 PM, Burakov, Anatoly wrote:
>> On 27-Oct-17 3:28 PM, Jonas Pfefferle1 wrote:
>>> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 10/27/2017 
>>> 04:06:44 PM:
>>>
>>>  > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>>>  > To: Jonas Pfefferle1 <JPF@zurich.ibm.com>, dev@dpdk.org
>>>  > Cc: chaozhu@linux.vnet.ibm.com, bruce.richardson@intel.com
>>>  > Date: 10/27/2017 04:06 PM
>>>  > Subject: Re: [dpdk-dev] Huge mapping secondary process linux
>>>  ...
>>>  > >
>>>  > hi Jonas,
>>>  >
>>>  > MAP_FIXED is not used because it's dangerous, it unmaps anything 
>>> that is
>>>  > already mapped into that space. We would rather know that we can't 
>>> map
>>>  > something than unwittingly unmap something that was mapped before.
>>>
>>> Ok, I see. Maybe we can add a check to the primary process's memory 
>>> mappings whether the hint has been respected or not? At least warn if 
>>> it hasn't.
>>
>> Hi Jonas,
>>
>> I'm unfamiliar with POWER platform, so i'm afraid you'd have to 
>> explain a bit more what you mean by "hint has been respected" :)
> 
> Actually, I also met this case on x86 once that kernel does not respect 
> the "addr" parameter even that memory region is not occupied. I am not 
> sure if it can be reproduced now, anyway, send here FYI: we run primary 
> on the host, run secondary in a container.
> 
> I'll agree at least we need to check if the final addr is the same of 
> the parameter addr, and warn if it's not.
> 
> Thanks,
> Jianfeng
> 

We could put in a warning saying that the address we got is *lower* than 
the address we expected to get, but i'm not sure throwing a warning 
because our assumption about kernel's behavior was incorrect is worth it.

-- 
Thanks,
Anatoly