[dpdk-dev] [PATCH] mem: warn if address hint is not respected

[dpdk-dev] [PATCH] mem: warn if address hint is not respected

[Jonas Pfefferle]

Print a warning if the --base-virtaddr hint is not respected
since this might lead to problems when mapping memory in
the secondary process.

Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
---
 lib/librte_eal/linuxapp/eal/eal_memory.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index ddf88c5..8273c58 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -256,33 +256,44 @@ static void *
 get_virtual_area(size_t *size, size_t hugepage_sz)
 {
 	void *addr;
+	void *addr_hint;
 	int fd;
 	long aligned_addr;
 
 	if (internal_config.base_virtaddr != 0) {
-		addr = (void*) (uintptr_t) (internal_config.base_virtaddr +
-				baseaddr_offset);
+		int page_size = sysconf(_SC_PAGE_SIZE);
+		addr_hint = (void *) (uintptr_t)
+			(internal_config.base_virtaddr + baseaddr_offset);
+		addr_hint = RTE_PTR_ALIGN_FLOOR(addr_hint, page_size);
+	} else {
+		addr_hint = NULL;
 	}
-	else addr = NULL;
 
 	RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
 
+
 	fd = open("/dev/zero", O_RDONLY);
 	if (fd < 0){
 		RTE_LOG(ERR, EAL, "Cannot open /dev/zero\n");
 		return NULL;
 	}
 	do {
-		addr = mmap(addr,
-				(*size) + hugepage_sz, PROT_READ,
+		addr = mmap(addr_hint, (*size) + hugepage_sz, PROT_READ,
 #ifdef RTE_ARCH_PPC_64
 				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
 #else
 				MAP_PRIVATE,
 #endif
 				fd, 0);
-		if (addr == MAP_FAILED)
+		if (addr == MAP_FAILED) {
 			*size -= hugepage_sz;
+		} else if (addr_hint != NULL && addr != addr_hint) {
+			RTE_LOG(WARNING, EAL, "WARNING! Base virtual address "
+				"hint (%p != %p) not respected!\n",
+				addr_hint, addr);
+			RTE_LOG(WARNING, EAL, "   This may cause issues with "
+				"mapping memory into secondary processes\n");
+		}
 	} while (addr == MAP_FAILED && *size > 0);
 
 	if (addr == MAP_FAILED) {
-- 
2.7.4

Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected

[Thomas Monjalon]

31/10/2017 10:08, Jonas Pfefferle:
> Print a warning if the --base-virtaddr hint is not respected
> since this might lead to problems when mapping memory in
> the secondary process.
> 
> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>

Anatoly, please review this patch.
It does not seem to fix something, so it is candidate for 18.02.

Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected

[Burakov, Anatoly]

On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> 31/10/2017 10:08, Jonas Pfefferle:
>> Print a warning if the --base-virtaddr hint is not respected
>> since this might lead to problems when mapping memory in
>> the secondary process.
>>
>> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
> 
> Anatoly, please review this patch.
> It does not seem to fix something, so it is candidate for 18.02.
> 

For some reason my Thunderbird ate the original email, so i'll reply to 
this one.

One nitpick would be that we're calling get_virtual_area many times and 
it would probably be a good idea to make pagesize static and call 
sysconf only once. Otherwise,

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected

[Jonas Pfefferle1]

"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 11/07/2017 02:54:24
PM:

> From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> To: Thomas Monjalon <thomas@monjalon.net>
> Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
jianfeng.tan@intel.com
> Date: 11/07/2017 02:54 PM
> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
respected
>
> On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> > 31/10/2017 10:08, Jonas Pfefferle:
> >> Print a warning if the --base-virtaddr hint is not respected
> >> since this might lead to problems when mapping memory in
> >> the secondary process.
> >>
> >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
> >
> > Anatoly, please review this patch.
> > It does not seem to fix something, so it is candidate for 18.02.
> >
>
> For some reason my Thunderbird ate the original email, so i'll reply to
> this one.
>
> One nitpick would be that we're calling get_virtual_area many times and
> it would probably be a good idea to make pagesize static and call
> sysconf only once. Otherwise,

We should address this in a separate patch and introduce a pagesize
function
for everyone to use. sysconf is used like this all over the place.

>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>
> --
> Thanks,
> Anatoly
>

Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected

[Xueming(Steven) Li]

Hi Jonas,

Seems you are trying to use --base-virtaddr to resolve address conflicts 
in secondary, I'm wondering how this happened and how to avoid it:
1. what's your hugepage side? Hugepage mmap is size aligned, maybe 1G works?
2. is there more libs loaded in secondary process or memory usage before 
EAL init?
3. Since address allocated in one direction, I'm thinking to reserve a 
larger "hop" address space as MAP_ANONYMOUS, allocate hugepage, then release 
"hop". That essentially reserve an address space big enough for secondary,
and most important the hop size is easy to estimate than --base-virtaddr.

Thanks,
Xueming(Steven)

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jonas Pfefferle1
> Sent: Wednesday, November 8, 2017 7:52 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected
> 
> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 11/07/2017
> 02:54:24
> PM:
> 
> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> > To: Thomas Monjalon <thomas@monjalon.net>
> > Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
> jianfeng.tan@intel.com
> > Date: 11/07/2017 02:54 PM
> > Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
> respected
> >
> > On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> > > 31/10/2017 10:08, Jonas Pfefferle:
> > >> Print a warning if the --base-virtaddr hint is not respected since
> > >> this might lead to problems when mapping memory in the secondary
> > >> process.
> > >>
> > >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
> > >
> > > Anatoly, please review this patch.
> > > It does not seem to fix something, so it is candidate for 18.02.
> > >
> >
> > For some reason my Thunderbird ate the original email, so i'll reply
> > to this one.
> >
> > One nitpick would be that we're calling get_virtual_area many times
> > and it would probably be a good idea to make pagesize static and call
> > sysconf only once. Otherwise,
> 
> We should address this in a separate patch and introduce a pagesize
> function for everyone to use. sysconf is used like this all over the place.
> 
> >
> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> >
> > --
> > Thanks,
> > Anatoly
> >

Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected

[Jonas Pfefferle]

Hi Xueming,

Correct --base-virtaddr was introduced for that purpose. There are 
multiple reasons why the address layout of the secondary process might 
look different: reasons you mentioned in 2), ASLR etc. I believe there 
is no way to avoid this in real world use cases. The reason for this 
particular patch is that the address hint (--base-virtaddr) is 
sometimes not respected and the kernel falls back to just reserving 
any address it can find to satisfy the mapping (see discussion on the 
patch), i.e. effectively rendering --base-virtaddr useless.

Regards,
Jonas (new email address)


  On Tue, 26 Dec 2017 15:56:10 +0000
  "Xueming(Steven) Li" <xuemingl@mellanox.com> wrote:
> Hi Jonas,
> 
> Seems you are trying to use --base-virtaddr to resolve address 
>conflicts 
> in secondary, I'm wondering how this happened and how to avoid it:
> 1. what's your hugepage side? Hugepage mmap is size aligned, maybe 
>1G works?
> 2. is there more libs loaded in secondary process or memory usage 
>before 
> EAL init?
> 3. Since address allocated in one direction, I'm thinking to reserve 
>a 
> larger "hop" address space as MAP_ANONYMOUS, allocate hugepage, then 
>release 
> "hop". That essentially reserve an address space big enough for 
>secondary,
> and most important the hop size is easy to estimate than 
>--base-virtaddr.
> 
> Thanks,
> Xueming(Steven)
> 
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jonas 
>>Pfefferle1
>> Sent: Wednesday, November 8, 2017 7:52 PM
>> To: Burakov, Anatoly <anatoly.burakov@intel.com>
>> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
>> <thomas@monjalon.net>
>> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not 
>>respected
>> 
>> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 11/07/2017
>> 02:54:24
>> PM:
>> 
>> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>> > To: Thomas Monjalon <thomas@monjalon.net>
>> > Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
>> jianfeng.tan@intel.com
>> > Date: 11/07/2017 02:54 PM
>> > Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
>> respected
>> >
>> > On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
>> > > 31/10/2017 10:08, Jonas Pfefferle:
>> > >> Print a warning if the --base-virtaddr hint is not respected 
>>since
>> > >> this might lead to problems when mapping memory in the 
>>secondary
>> > >> process.
>> > >>
>> > >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
>> > >
>> > > Anatoly, please review this patch.
>> > > It does not seem to fix something, so it is candidate for 18.02.
>> > >
>> >
>> > For some reason my Thunderbird ate the original email, so i'll 
>>reply
>> > to this one.
>> >
>> > One nitpick would be that we're calling get_virtual_area many 
>>times
>> > and it would probably be a good idea to make pagesize static and 
>>call
>> > sysconf only once. Otherwise,
>> 
>> We should address this in a separate patch and introduce a pagesize
>> function for everyone to use. sysconf is used like this all over the 
>>place.
>> 
>> >
>> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>> >
>> > --
>> > Thanks,
>> > Anatoly
>> >

Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected

[Xueming(Steven) Li]

So the idea of item 3 might sound and lead to seldom usage of '--base-virtaddress'.
Reserve an address hole big enough before hugepage almost cost nothing.

> -----Original Message-----
> From: Jonas Pfefferle [mailto:pepperjo@japf.ch]
> Sent: Wednesday, January 3, 2018 5:22 PM
> To: Xueming(Steven) Li <xuemingl@mellanox.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>
> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
> <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected
> 
> Hi Xueming,
> 
> Correct --base-virtaddr was introduced for that purpose. There are
> multiple reasons why the address layout of the secondary process might
> look different: reasons you mentioned in 2), ASLR etc. I believe there is
> no way to avoid this in real world use cases. The reason for this
> particular patch is that the address hint (--base-virtaddr) is sometimes
> not respected and the kernel falls back to just reserving any address it
> can find to satisfy the mapping (see discussion on the patch), i.e.
> effectively rendering --base-virtaddr useless.
> 
> Regards,
> Jonas (new email address)
> 
> 
>   On Tue, 26 Dec 2017 15:56:10 +0000
>   "Xueming(Steven) Li" <xuemingl@mellanox.com> wrote:
> > Hi Jonas,
> >
> > Seems you are trying to use --base-virtaddr to resolve address
> >conflicts  in secondary, I'm wondering how this happened and how to
> >avoid it:
> > 1. what's your hugepage side? Hugepage mmap is size aligned, maybe 1G
> >works?
> > 2. is there more libs loaded in secondary process or memory usage
> >before  EAL init?
> > 3. Since address allocated in one direction, I'm thinking to reserve a
> >larger "hop" address space as MAP_ANONYMOUS, allocate hugepage, then
> >release  "hop". That essentially reserve an address space big enough
> >for secondary,  and most important the hop size is easy to estimate
> >than --base-virtaddr.
> >
> > Thanks,
> > Xueming(Steven)
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jonas
> >>Pfefferle1
> >> Sent: Wednesday, November 8, 2017 7:52 PM
> >> To: Burakov, Anatoly <anatoly.burakov@intel.com>
> >> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
> >><thomas@monjalon.net>
> >> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
> >>respected
> >>
> >> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 11/07/2017
> >> 02:54:24
> >> PM:
> >>
> >> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
> >> > To: Thomas Monjalon <thomas@monjalon.net>
> >> > Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
> >> jianfeng.tan@intel.com
> >> > Date: 11/07/2017 02:54 PM
> >> > Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
> >> respected
> >> >
> >> > On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> >> > > 31/10/2017 10:08, Jonas Pfefferle:
> >> > >> Print a warning if the --base-virtaddr hint is not respected
> >>since
> >> > >> this might lead to problems when mapping memory in the
> >>secondary
> >> > >> process.
> >> > >>
> >> > >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
> >> > >
> >> > > Anatoly, please review this patch.
> >> > > It does not seem to fix something, so it is candidate for 18.02.
> >> > >
> >> >
> >> > For some reason my Thunderbird ate the original email, so i'll
> >>reply
> >> > to this one.
> >> >
> >> > One nitpick would be that we're calling get_virtual_area many
> >>times
> >> > and it would probably be a good idea to make pagesize static and
> >>call
> >> > sysconf only once. Otherwise,
> >>
> >> We should address this in a separate patch and introduce a pagesize
> >>function for everyone to use. sysconf is used like this all over the
> >>place.
> >>
> >> >
> >> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> >> >
> >> > --
> >> > Thanks,
> >> > Anatoly
> >> >

Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected

[Jonas Pfefferle]

I don't believe that it would work since you cannot assume that mmap 
addresses will just increase. Especially with ASLR anything can 
happen. I don't understand why you want to get rid of --base-virtaddr 
I think it is a valid solution for the problem.

  On Wed, 3 Jan 2018 09:37:00 +0000
  "Xueming(Steven) Li" <xuemingl@mellanox.com> wrote:
> So the idea of item 3 might sound and lead to seldom usage of 
>'--base-virtaddress'.
> Reserve an address hole big enough before hugepage almost cost 
>nothing.
> 
>> -----Original Message-----
>> From: Jonas Pfefferle [mailto:pepperjo@japf.ch]
>> Sent: Wednesday, January 3, 2018 5:22 PM
>> To: Xueming(Steven) Li <xuemingl@mellanox.com>; Burakov, Anatoly
>> <anatoly.burakov@intel.com>
>> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
>> <thomas@monjalon.net>
>> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not 
>>respected
>> 
>> Hi Xueming,
>> 
>> Correct --base-virtaddr was introduced for that purpose. There are
>> multiple reasons why the address layout of the secondary process 
>>might
>> look different: reasons you mentioned in 2), ASLR etc. I believe 
>>there is
>> no way to avoid this in real world use cases. The reason for this
>> particular patch is that the address hint (--base-virtaddr) is 
>>sometimes
>> not respected and the kernel falls back to just reserving any 
>>address it
>> can find to satisfy the mapping (see discussion on the patch), i.e.
>> effectively rendering --base-virtaddr useless.
>> 
>> Regards,
>> Jonas (new email address)
>> 
>> 
>>   On Tue, 26 Dec 2017 15:56:10 +0000
>>   "Xueming(Steven) Li" <xuemingl@mellanox.com> wrote:
>> > Hi Jonas,
>> >
>> > Seems you are trying to use --base-virtaddr to resolve address
>> >conflicts  in secondary, I'm wondering how this happened and how to
>> >avoid it:
>> > 1. what's your hugepage side? Hugepage mmap is size aligned, maybe 
>>1G
>> >works?
>> > 2. is there more libs loaded in secondary process or memory usage
>> >before  EAL init?
>> > 3. Since address allocated in one direction, I'm thinking to 
>>reserve a
>> >larger "hop" address space as MAP_ANONYMOUS, allocate hugepage, 
>>then
>> >release  "hop". That essentially reserve an address space big 
>>enough
>> >for secondary,  and most important the hop size is easy to estimate
>> >than --base-virtaddr.
>> >
>> > Thanks,
>> > Xueming(Steven)
>> >
>> >> -----Original Message-----
>> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jonas
>> >>Pfefferle1
>> >> Sent: Wednesday, November 8, 2017 7:52 PM
>> >> To: Burakov, Anatoly <anatoly.burakov@intel.com>
>> >> Cc: dev@dpdk.org; jianfeng.tan@intel.com; Thomas Monjalon
>> >><thomas@monjalon.net>
>> >> Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is not
>> >>respected
>> >>
>> >> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote on 
>>11/07/2017
>> >> 02:54:24
>> >> PM:
>> >>
>> >> > From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
>> >> > To: Thomas Monjalon <thomas@monjalon.net>
>> >> > Cc: dev@dpdk.org, Jonas Pfefferle <jpf@zurich.ibm.com>,
>> >> jianfeng.tan@intel.com
>> >> > Date: 11/07/2017 02:54 PM
>> >> > Subject: Re: [dpdk-dev] [PATCH] mem: warn if address hint is 
>>not
>> >> respected
>> >> >
>> >> > On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
>> >> > > 31/10/2017 10:08, Jonas Pfefferle:
>> >> > >> Print a warning if the --base-virtaddr hint is not respected
>> >>since
>> >> > >> this might lead to problems when mapping memory in the
>> >>secondary
>> >> > >> process.
>> >> > >>
>> >> > >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
>> >> > >
>> >> > > Anatoly, please review this patch.
>> >> > > It does not seem to fix something, so it is candidate for 
>>18.02.
>> >> > >
>> >> >
>> >> > For some reason my Thunderbird ate the original email, so i'll
>> >>reply
>> >> > to this one.
>> >> >
>> >> > One nitpick would be that we're calling get_virtual_area many
>> >>times
>> >> > and it would probably be a good idea to make pagesize static 
>>and
>> >>call
>> >> > sysconf only once. Otherwise,
>> >>
>> >> We should address this in a separate patch and introduce a 
>>pagesize
>> >>function for everyone to use. sysconf is used like this all over 
>>the
>> >>place.
>> >>
>> >> >
>> >> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>> >> >
>> >> > --
>> >> > Thanks,
>> >> > Anatoly
>> >> >
> 

Re: [dpdk-dev] [PATCH] mem: warn if address hint is not respected

[Thomas Monjalon]

07/11/2017 14:54, Burakov, Anatoly:
> On 06-Nov-17 8:26 PM, Thomas Monjalon wrote:
> > 31/10/2017 10:08, Jonas Pfefferle:
> >> Print a warning if the --base-virtaddr hint is not respected
> >> since this might lead to problems when mapping memory in
> >> the secondary process.
> >>
> >> Signed-off-by: Jonas Pfefferle <jpf@zurich.ibm.com>
> > 
> > Anatoly, please review this patch.
> > It does not seem to fix something, so it is candidate for 18.02.
> > 
> 
> For some reason my Thunderbird ate the original email, so i'll reply to 
> this one.
> 
> One nitpick would be that we're calling get_virtual_area many times and 
> it would probably be a good idea to make pagesize static and call 
> sysconf only once. Otherwise,
> 
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

Applied, thanks