DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [RFC PATCH 0/2] Reduce DPDK initialization time
@ 2015-11-18  3:27 Zhihong Wang
  2015-11-18  3:27 ` [dpdk-dev] [RFC PATCH 1/2] lib/librte_eal: Reduce timer " Zhihong Wang
                   ` (2 more replies)
  0 siblings, 3 replies; 25+ messages in thread
From: Zhihong Wang @ 2015-11-18  3:27 UTC (permalink / raw)
  To: dev

This RFC patch aims to reduce DPDK initialization time, which is important in cases such as micro service.

Changes are:

1. Reduce timer initialization time

2. Remove unnecessary hugepage zero-filling operations

With this patch:

1. Timer initialization time can be reduced by 4/10 second

2. Memory initialization time can be reduced nearly by half

The 2nd topic has been brought up before in this thread:
http://dpdk.org/dev/patchwork/patch/4219/

Zhihong Wang (2):
  lib/librte_eal: Reduce timer initialization time
  lib/librte_eal: Remove unnecessary hugepage zero-filling

 lib/librte_eal/linuxapp/eal/eal_memory.c | 5 +----
 lib/librte_eal/linuxapp/eal/eal_timer.c  | 2 +-
 2 files changed, 2 insertions(+), 5 deletions(-)

-- 
2.5.0

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [dpdk-dev] [RFC PATCH 1/2] lib/librte_eal: Reduce timer initialization time
  2015-11-18  3:27 [dpdk-dev] [RFC PATCH 0/2] Reduce DPDK initialization time Zhihong Wang
@ 2015-11-18  3:27 ` Zhihong Wang
  2015-11-18  3:27 ` [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling Zhihong Wang
  2015-12-24  8:34 ` [dpdk-dev] [RFC PATCH 0/2] Reduce DPDK initialization time Qiu, Michael
  2 siblings, 0 replies; 25+ messages in thread
From: Zhihong Wang @ 2015-11-18  3:27 UTC (permalink / raw)
  To: dev

Changing from 1/2 second to 1/10 doesn't compromise the precision, and a 4/10 second is worth saving.

Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_timer.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_timer.c b/lib/librte_eal/linuxapp/eal/eal_timer.c
index e0642de..4de0353 100644
--- a/lib/librte_eal/linuxapp/eal/eal_timer.c
+++ b/lib/librte_eal/linuxapp/eal/eal_timer.c
@@ -271,7 +271,7 @@ get_tsc_freq(void)
 #ifdef CLOCK_MONOTONIC_RAW
 #define NS_PER_SEC 1E9
 
-	struct timespec sleeptime = {.tv_nsec = 5E8 }; /* 1/2 second */
+	struct timespec sleeptime = {.tv_nsec = 1E8 }; /* 1/10 second */
 
 	struct timespec t_start, t_end;
 	uint64_t tsc_hz;
-- 
2.5.0

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18  3:27 [dpdk-dev] [RFC PATCH 0/2] Reduce DPDK initialization time Zhihong Wang
  2015-11-18  3:27 ` [dpdk-dev] [RFC PATCH 1/2] lib/librte_eal: Reduce timer " Zhihong Wang
@ 2015-11-18  3:27 ` Zhihong Wang
  2015-11-18 10:39   ` Mcnamara, John
  2015-12-24  8:34 ` [dpdk-dev] [RFC PATCH 0/2] Reduce DPDK initialization time Qiu, Michael
  2 siblings, 1 reply; 25+ messages in thread
From: Zhihong Wang @ 2015-11-18  3:27 UTC (permalink / raw)
  To: dev

The kernel fills new allocated (huge) pages with zeros.
DPDK just has to touch the pages to trigger the allocation.

Signed-off-by: Zhihong Wang <zhihong.wang@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_memory.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 0de75cd..af823dc 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -410,7 +410,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 
 		if (orig) {
 			hugepg_tbl[i].orig_va = virtaddr;
-			memset(virtaddr, 0, hugepage_sz);
+			memset(virtaddr, 0, 8);
 		}
 		else {
 			hugepg_tbl[i].final_va = virtaddr;
@@ -592,9 +592,6 @@ remap_all_hugepages(struct hugepage_file *hugepg_tbl, struct hugepage_info *hpi)
 			}
 		}
 
-		/* zero out the whole segment */
-		memset(hugepg_tbl[page_idx].final_va, 0, total_size);
-
 		page_idx++;
 	}
 
-- 
2.5.0

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18  3:27 ` [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling Zhihong Wang
@ 2015-11-18 10:39   ` Mcnamara, John
  2015-11-18 10:44     ` Wang, Zhihong
  0 siblings, 1 reply; 25+ messages in thread
From: Mcnamara, John @ 2015-11-18 10:39 UTC (permalink / raw)
  To: Wang, Zhihong, dev



> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
> Sent: Wednesday, November 18, 2015 3:27 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
> hugepage zero-filling
> 
> The kernel fills new allocated (huge) pages with zeros.
> DPDK just has to touch the pages to trigger the allocation.
> 
> ...
>  		if (orig) {
>  			hugepg_tbl[i].orig_va = virtaddr;
> -			memset(virtaddr, 0, hugepage_sz);
> +			memset(virtaddr, 0, 8);
>  		}

Probably worth adding a one or two line comment here to avoid someone thinking that it is a bug at some later stage. The text in the commit message above is suitable.

John.
-- 

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18 10:39   ` Mcnamara, John
@ 2015-11-18 10:44     ` Wang, Zhihong
  2015-11-18 12:07       ` Xie, Huawei
  0 siblings, 1 reply; 25+ messages in thread
From: Wang, Zhihong @ 2015-11-18 10:44 UTC (permalink / raw)
  To: Mcnamara, John, dev


> -----Original Message-----
> From: Mcnamara, John
> Sent: Wednesday, November 18, 2015 6:40 PM
> To: Wang, Zhihong <zhihong.wang@intel.com>; dev@dpdk.org
> Subject: RE: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
> hugepage zero-filling
> 
> 
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
> > Sent: Wednesday, November 18, 2015 3:27 AM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
> > hugepage zero-filling
> >
> > The kernel fills new allocated (huge) pages with zeros.
> > DPDK just has to touch the pages to trigger the allocation.
> >
> > ...
> >  		if (orig) {
> >  			hugepg_tbl[i].orig_va = virtaddr;
> > -			memset(virtaddr, 0, hugepage_sz);
> > +			memset(virtaddr, 0, 8);
> >  		}
> 
> Probably worth adding a one or two line comment here to avoid someone
> thinking that it is a bug at some later stage. The text in the commit message
> above is suitable.
> 

Good suggestion! Will add it :)

> John.
> --

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18 10:44     ` Wang, Zhihong
@ 2015-11-18 12:07       ` Xie, Huawei
  2015-11-18 16:00         ` Stephen Hemminger
                           ` (2 more replies)
  0 siblings, 3 replies; 25+ messages in thread
From: Xie, Huawei @ 2015-11-18 12:07 UTC (permalink / raw)
  To: Wang, Zhihong, Mcnamara, John, dev

On 11/18/2015 6:45 PM, Wang, Zhihong wrote:
>> -----Original Message-----
>> From: Mcnamara, John
>> Sent: Wednesday, November 18, 2015 6:40 PM
>> To: Wang, Zhihong <zhihong.wang@intel.com>; dev@dpdk.org
>> Subject: RE: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
>> hugepage zero-filling
>>
>>
>>
>>> -----Original Message-----
>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
>>> Sent: Wednesday, November 18, 2015 3:27 AM
>>> To: dev@dpdk.org
>>> Subject: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
>>> hugepage zero-filling
>>>
>>> The kernel fills new allocated (huge) pages with zeros.
>>> DPDK just has to touch the pages to trigger the allocation.
I think we shouldn't reply on the assumption that kernel has zeroed the
memory. Kernel zeroes the memory mostly to avoid information leakage.It
could also achieve this by setting each bit to 1.
What we indeed need to check is later DPDK initialization code doesn't
assume the memory has been zeroed. Otherwise zero only that part of the
memory. Does this makes sense?

>>> ...
>>>  		if (orig) {
>>>  			hugepg_tbl[i].orig_va = virtaddr;
>>> -			memset(virtaddr, 0, hugepage_sz);
>>> +			memset(virtaddr, 0, 8);
>>>  		}
>> Probably worth adding a one or two line comment here to avoid someone
>> thinking that it is a bug at some later stage. The text in the commit message
>> above is suitable.
>>
> Good suggestion! Will add it :)
>
>> John.
>> --
>


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18 12:07       ` Xie, Huawei
@ 2015-11-18 16:00         ` Stephen Hemminger
  2015-11-18 16:13           ` Richardson, Bruce
  2015-11-19  3:54         ` Wang, Zhihong
  2015-11-19  9:14         ` Sergio Gonzalez Monroy
  2 siblings, 1 reply; 25+ messages in thread
From: Stephen Hemminger @ 2015-11-18 16:00 UTC (permalink / raw)
  To: Xie, Huawei; +Cc: dev

On Wed, 18 Nov 2015 12:07:54 +0000
"Xie, Huawei" <huawei.xie@intel.com> wrote:

> >>> The kernel fills new allocated (huge) pages with zeros.
> >>> DPDK just has to touch the pages to trigger the allocation.  
> I think we shouldn't reply on the assumption that kernel has zeroed the
> memory. Kernel zeroes the memory mostly to avoid information leakage.It
> could also achieve this by setting each bit to 1.
> What we indeed need to check is later DPDK initialization code doesn't
> assume the memory has been zeroed. Otherwise zero only that part of the
> memory. Does this makes sense?

If all new pages are zero, why does DPDK have to pre-touch the pages
at all?

I thought there as some optimization to initialize hugepages since
Oracle has same problem with their Shared Global Area which was why
hugpages were invented anyway

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18 16:00         ` Stephen Hemminger
@ 2015-11-18 16:13           ` Richardson, Bruce
  2015-11-18 19:09             ` Stephen Hemminger
  0 siblings, 1 reply; 25+ messages in thread
From: Richardson, Bruce @ 2015-11-18 16:13 UTC (permalink / raw)
  To: Stephen Hemminger, Xie, Huawei; +Cc: dev



> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen Hemminger
> Sent: Wednesday, November 18, 2015 4:00 PM
> To: Xie, Huawei <huawei.xie@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
> hugepage zero-filling
> 
> On Wed, 18 Nov 2015 12:07:54 +0000
> "Xie, Huawei" <huawei.xie@intel.com> wrote:
> 
> > >>> The kernel fills new allocated (huge) pages with zeros.
> > >>> DPDK just has to touch the pages to trigger the allocation.
> > I think we shouldn't reply on the assumption that kernel has zeroed
> > the memory. Kernel zeroes the memory mostly to avoid information
> > leakage.It could also achieve this by setting each bit to 1.
> > What we indeed need to check is later DPDK initialization code doesn't
> > assume the memory has been zeroed. Otherwise zero only that part of
> > the memory. Does this makes sense?
> 
> If all new pages are zero, why does DPDK have to pre-touch the pages at
> all?

The pages won't actually be mapped into the processes address space until accessed. 

/Bruce

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18 16:13           ` Richardson, Bruce
@ 2015-11-18 19:09             ` Stephen Hemminger
  2015-11-19  2:15               ` Wang, Zhihong
  2015-11-20 12:15               ` Bruce Richardson
  0 siblings, 2 replies; 25+ messages in thread
From: Stephen Hemminger @ 2015-11-18 19:09 UTC (permalink / raw)
  To: Richardson, Bruce; +Cc: dev

On Wed, 18 Nov 2015 16:13:32 +0000
"Richardson, Bruce" <bruce.richardson@intel.com> wrote:

> 
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen Hemminger
> > Sent: Wednesday, November 18, 2015 4:00 PM
> > To: Xie, Huawei <huawei.xie@intel.com>
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
> > hugepage zero-filling
> > 
> > On Wed, 18 Nov 2015 12:07:54 +0000
> > "Xie, Huawei" <huawei.xie@intel.com> wrote:
> > 
> > > >>> The kernel fills new allocated (huge) pages with zeros.
> > > >>> DPDK just has to touch the pages to trigger the allocation.
> > > I think we shouldn't reply on the assumption that kernel has zeroed
> > > the memory. Kernel zeroes the memory mostly to avoid information
> > > leakage.It could also achieve this by setting each bit to 1.
> > > What we indeed need to check is later DPDK initialization code doesn't
> > > assume the memory has been zeroed. Otherwise zero only that part of
> > > the memory. Does this makes sense?
> > 
> > If all new pages are zero, why does DPDK have to pre-touch the pages at
> > all?
> 
> The pages won't actually be mapped into the processes address space until accessed. 
> 
> /Bruce

Isn't that what mmap MAP_POPULATE flag (not currently used) will do.

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18 19:09             ` Stephen Hemminger
@ 2015-11-19  2:15               ` Wang, Zhihong
  2015-11-19  6:04                 ` Xie, Huawei
  2015-11-20 12:15               ` Bruce Richardson
  1 sibling, 1 reply; 25+ messages in thread
From: Wang, Zhihong @ 2015-11-19  2:15 UTC (permalink / raw)
  To: Stephen Hemminger, Richardson, Bruce; +Cc: dev



> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen Hemminger
> Sent: Thursday, November 19, 2015 3:09 AM
> To: Richardson, Bruce <bruce.richardson@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
> hugepage zero-filling
> 
> On Wed, 18 Nov 2015 16:13:32 +0000
> "Richardson, Bruce" <bruce.richardson@intel.com> wrote:
> 
> >
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen
> > > Hemminger
> > > Sent: Wednesday, November 18, 2015 4:00 PM
> > > To: Xie, Huawei <huawei.xie@intel.com>
> > > Cc: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
> > > unnecessary hugepage zero-filling
> > >
> > > On Wed, 18 Nov 2015 12:07:54 +0000
> > > "Xie, Huawei" <huawei.xie@intel.com> wrote:
> > >
> > > > >>> The kernel fills new allocated (huge) pages with zeros.
> > > > >>> DPDK just has to touch the pages to trigger the allocation.
> > > > I think we shouldn't reply on the assumption that kernel has
> > > > zeroed the memory. Kernel zeroes the memory mostly to avoid
> > > > information leakage.It could also achieve this by setting each bit to 1.
> > > > What we indeed need to check is later DPDK initialization code
> > > > doesn't assume the memory has been zeroed. Otherwise zero only
> > > > that part of the memory. Does this makes sense?
> > >
> > > If all new pages are zero, why does DPDK have to pre-touch the pages
> > > at all?
> >
> > The pages won't actually be mapped into the processes address space until
> accessed.
> >
> > /Bruce
> 
> Isn't that what mmap MAP_POPULATE flag (not currently used) will do.

Yes, the MAP_POPULATE does literally the same thing.
This flag is implemented since Linux 2.5.46 according to Linux man page, guess that's why DPDK fault the page tables manually in the first place. :)

I think we can use this flag since it makes the code clearer.

/Zhihong

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18 12:07       ` Xie, Huawei
  2015-11-18 16:00         ` Stephen Hemminger
@ 2015-11-19  3:54         ` Wang, Zhihong
  2015-11-19  6:09           ` Xie, Huawei
  2015-11-19  9:14         ` Sergio Gonzalez Monroy
  2 siblings, 1 reply; 25+ messages in thread
From: Wang, Zhihong @ 2015-11-19  3:54 UTC (permalink / raw)
  To: Xie, Huawei, Mcnamara, John, dev



> -----Original Message-----
> From: Xie, Huawei
> Sent: Wednesday, November 18, 2015 8:08 PM
> To: Wang, Zhihong <zhihong.wang@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
> hugepage zero-filling
> 
> On 11/18/2015 6:45 PM, Wang, Zhihong wrote:
> >> -----Original Message-----
> >> From: Mcnamara, John
> >> Sent: Wednesday, November 18, 2015 6:40 PM
> >> To: Wang, Zhihong <zhihong.wang@intel.com>; dev@dpdk.org
> >> Subject: RE: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
> >> unnecessary hugepage zero-filling
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
> >>> Sent: Wednesday, November 18, 2015 3:27 AM
> >>> To: dev@dpdk.org
> >>> Subject: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
> >>> unnecessary hugepage zero-filling
> >>>
> >>> The kernel fills new allocated (huge) pages with zeros.
> >>> DPDK just has to touch the pages to trigger the allocation.
> I think we shouldn't reply on the assumption that kernel has zeroed the memory.

I understand the concern.
In my opinion application shouldn't assume malloced memory to be zero-filled. So it should be okay for DPDK even if the kernel doesn't zero the page at all.

I agree that we should check if any code accidentally make that assumption. Currently there's rte_pktmbuf_init() for packet mbuf initialization.

/Zhihong


> Kernel zeroes the memory mostly to avoid information leakage.It could also
> achieve this by setting each bit to 1.
> What we indeed need to check is later DPDK initialization code doesn't assume
> the memory has been zeroed. Otherwise zero only that part of the memory.
> Does this makes sense?
> 
> >>> ...
> >>>  		if (orig) {
> >>>  			hugepg_tbl[i].orig_va = virtaddr;
> >>> -			memset(virtaddr, 0, hugepage_sz);
> >>> +			memset(virtaddr, 0, 8);
> >>>  		}
> >> Probably worth adding a one or two line comment here to avoid someone
> >> thinking that it is a bug at some later stage. The text in the commit
> >> message above is suitable.
> >>
> > Good suggestion! Will add it :)
> >
> >> John.
> >> --
> >

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-19  2:15               ` Wang, Zhihong
@ 2015-11-19  6:04                 ` Xie, Huawei
  2015-11-19  6:32                   ` Wang, Zhihong
  0 siblings, 1 reply; 25+ messages in thread
From: Xie, Huawei @ 2015-11-19  6:04 UTC (permalink / raw)
  To: Wang, Zhihong, Stephen Hemminger, Richardson, Bruce; +Cc: dev

On 11/19/2015 10:16 AM, Wang, Zhihong wrote:
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen Hemminger
>> Sent: Thursday, November 19, 2015 3:09 AM
>> To: Richardson, Bruce <bruce.richardson@intel.com>
>> Cc: dev@dpdk.org
>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
>> hugepage zero-filling
>>
>> On Wed, 18 Nov 2015 16:13:32 +0000
>> "Richardson, Bruce" <bruce.richardson@intel.com> wrote:
>>
>>>
>>>> -----Original Message-----
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen
>>>> Hemminger
>>>> Sent: Wednesday, November 18, 2015 4:00 PM
>>>> To: Xie, Huawei <huawei.xie@intel.com>
>>>> Cc: dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
>>>> unnecessary hugepage zero-filling
>>>>
>>>> On Wed, 18 Nov 2015 12:07:54 +0000
>>>> "Xie, Huawei" <huawei.xie@intel.com> wrote:
>>>>
>>>>>>>> The kernel fills new allocated (huge) pages with zeros.
>>>>>>>> DPDK just has to touch the pages to trigger the allocation.
>>>>> I think we shouldn't reply on the assumption that kernel has
>>>>> zeroed the memory. Kernel zeroes the memory mostly to avoid
>>>>> information leakage.It could also achieve this by setting each bit to 1.
>>>>> What we indeed need to check is later DPDK initialization code
>>>>> doesn't assume the memory has been zeroed. Otherwise zero only
>>>>> that part of the memory. Does this makes sense?
>>>> If all new pages are zero, why does DPDK have to pre-touch the pages
>>>> at all?
>>> The pages won't actually be mapped into the processes address space until
>> accessed.
>>> /Bruce
>> Isn't that what mmap MAP_POPULATE flag (not currently used) will do.
> Yes, the MAP_POPULATE does literally the same thing.
> This flag is implemented since Linux 2.5.46 according to Linux man page, guess that's why DPDK fault the page tables manually in the first place. :)
>
> I think we can use this flag since it makes the code clearer.
The manual says MAP_POPULATE is only supported for private mappings
since Linux 2.6.23.
>
> /Zhihong
>
>


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-19  3:54         ` Wang, Zhihong
@ 2015-11-19  6:09           ` Xie, Huawei
  0 siblings, 0 replies; 25+ messages in thread
From: Xie, Huawei @ 2015-11-19  6:09 UTC (permalink / raw)
  To: Wang, Zhihong, Mcnamara, John, dev

On 11/19/2015 11:54 AM, Wang, Zhihong wrote:
>
>> -----Original Message-----
>> From: Xie, Huawei
>> Sent: Wednesday, November 18, 2015 8:08 PM
>> To: Wang, Zhihong <zhihong.wang@intel.com>; Mcnamara, John
>> <john.mcnamara@intel.com>; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
>> hugepage zero-filling
>>
>> On 11/18/2015 6:45 PM, Wang, Zhihong wrote:
>>>> -----Original Message-----
>>>> From: Mcnamara, John
>>>> Sent: Wednesday, November 18, 2015 6:40 PM
>>>> To: Wang, Zhihong <zhihong.wang@intel.com>; dev@dpdk.org
>>>> Subject: RE: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
>>>> unnecessary hugepage zero-filling
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
>>>>> Sent: Wednesday, November 18, 2015 3:27 AM
>>>>> To: dev@dpdk.org
>>>>> Subject: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
>>>>> unnecessary hugepage zero-filling
>>>>>
>>>>> The kernel fills new allocated (huge) pages with zeros.
>>>>> DPDK just has to touch the pages to trigger the allocation.
>> I think we shouldn't reply on the assumption that kernel has zeroed the memory.
> I understand the concern.
> In my opinion application shouldn't assume malloced memory to be zero-filled. So it should be okay for DPDK even if the kernel doesn't zero the page at all.
For malloc, we shouldn't make this assumption because it might allocate
just freed memory from the heap. Hugetlbfs is different. Let us listen
to other people's opinion.
It will make life much easier if we could make this assumption.
>
> I agree that we should check if any code accidentally make that assumption. Currently there's rte_pktmbuf_init() for packet mbuf initialization.
>
> /Zhihong
>
>
>> Kernel zeroes the memory mostly to avoid information leakage.It could also
>> achieve this by setting each bit to 1.
>> What we indeed need to check is later DPDK initialization code doesn't assume
>> the memory has been zeroed. Otherwise zero only that part of the memory.
>> Does this makes sense?
>>
>>>>> ...
>>>>>  		if (orig) {
>>>>>  			hugepg_tbl[i].orig_va = virtaddr;
>>>>> -			memset(virtaddr, 0, hugepage_sz);
>>>>> +			memset(virtaddr, 0, 8);
>>>>>  		}
>>>> Probably worth adding a one or two line comment here to avoid someone
>>>> thinking that it is a bug at some later stage. The text in the commit
>>>> message above is suitable.
>>>>
>>> Good suggestion! Will add it :)
>>>
>>>> John.
>>>> --
>


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-19  6:04                 ` Xie, Huawei
@ 2015-11-19  6:32                   ` Wang, Zhihong
  2015-11-19  9:18                     ` Sergio Gonzalez Monroy
  2015-11-23  2:54                     ` Xie, Huawei
  0 siblings, 2 replies; 25+ messages in thread
From: Wang, Zhihong @ 2015-11-19  6:32 UTC (permalink / raw)
  To: Xie, Huawei, Stephen Hemminger, Richardson, Bruce; +Cc: dev


> -----Original Message-----
> From: Xie, Huawei
> Sent: Thursday, November 19, 2015 2:05 PM
> To: Wang, Zhihong <zhihong.wang@intel.com>; Stephen Hemminger
> <stephen@networkplumber.org>; Richardson, Bruce
> <bruce.richardson@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
> hugepage zero-filling
> 
> On 11/19/2015 10:16 AM, Wang, Zhihong wrote:
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen
> >> Hemminger
> >> Sent: Thursday, November 19, 2015 3:09 AM
> >> To: Richardson, Bruce <bruce.richardson@intel.com>
> >> Cc: dev@dpdk.org
> >> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
> >> unnecessary hugepage zero-filling
> >>
> >> On Wed, 18 Nov 2015 16:13:32 +0000
> >> "Richardson, Bruce" <bruce.richardson@intel.com> wrote:
> >>
> >>>
> >>>> -----Original Message-----
> >>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen
> >>>> Hemminger
> >>>> Sent: Wednesday, November 18, 2015 4:00 PM
> >>>> To: Xie, Huawei <huawei.xie@intel.com>
> >>>> Cc: dev@dpdk.org
> >>>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
> >>>> unnecessary hugepage zero-filling
> >>>>
> >>>> On Wed, 18 Nov 2015 12:07:54 +0000
> >>>> "Xie, Huawei" <huawei.xie@intel.com> wrote:
> >>>>
> >>>>>>>> The kernel fills new allocated (huge) pages with zeros.
> >>>>>>>> DPDK just has to touch the pages to trigger the allocation.
> >>>>> I think we shouldn't reply on the assumption that kernel has
> >>>>> zeroed the memory. Kernel zeroes the memory mostly to avoid
> >>>>> information leakage.It could also achieve this by setting each bit to 1.
> >>>>> What we indeed need to check is later DPDK initialization code
> >>>>> doesn't assume the memory has been zeroed. Otherwise zero only
> >>>>> that part of the memory. Does this makes sense?
> >>>> If all new pages are zero, why does DPDK have to pre-touch the
> >>>> pages at all?
> >>> The pages won't actually be mapped into the processes address space
> >>> until
> >> accessed.
> >>> /Bruce
> >> Isn't that what mmap MAP_POPULATE flag (not currently used) will do.
> > Yes, the MAP_POPULATE does literally the same thing.
> > This flag is implemented since Linux 2.5.46 according to Linux man
> > page, guess that's why DPDK fault the page tables manually in the
> > first place. :)
> >
> > I think we can use this flag since it makes the code clearer.
> The manual says MAP_POPULATE is only supported for private mappings since
> Linux 2.6.23.

I've done check before and MAP_SHARED | MAP_POPULATE worked together correctly. Is there any implicit complication here?


> >
> > /Zhihong
> >
> >

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18 12:07       ` Xie, Huawei
  2015-11-18 16:00         ` Stephen Hemminger
  2015-11-19  3:54         ` Wang, Zhihong
@ 2015-11-19  9:14         ` Sergio Gonzalez Monroy
  2015-11-23  3:46           ` Xie, Huawei
  2 siblings, 1 reply; 25+ messages in thread
From: Sergio Gonzalez Monroy @ 2015-11-19  9:14 UTC (permalink / raw)
  To: Xie, Huawei, Wang, Zhihong, Mcnamara, John, dev

On 18/11/2015 12:07, Xie, Huawei wrote:
> On 11/18/2015 6:45 PM, Wang, Zhihong wrote:
>>> -----Original Message-----
>>> From: Mcnamara, John
>>> Sent: Wednesday, November 18, 2015 6:40 PM
>>> To: Wang, Zhihong <zhihong.wang@intel.com>; dev@dpdk.org
>>> Subject: RE: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
>>> hugepage zero-filling
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
>>>> Sent: Wednesday, November 18, 2015 3:27 AM
>>>> To: dev@dpdk.org
>>>> Subject: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
>>>> hugepage zero-filling
>>>>
>>>> The kernel fills new allocated (huge) pages with zeros.
>>>> DPDK just has to touch the pages to trigger the allocation.
> I think we shouldn't reply on the assumption that kernel has zeroed the
> memory. Kernel zeroes the memory mostly to avoid information leakage.It
> could also achieve this by setting each bit to 1.
> What we indeed need to check is later DPDK initialization code doesn't
> assume the memory has been zeroed. Otherwise zero only that part of the
> memory. Does this makes sense?

Why cannot we rely on the kernel zeroing the memory ?
If that behavior were to change, then we can zero out the memory ourselves.

Bruce pointed out to me that the semantics have changed a bit since we 
introduced
rte_memzone_free.
Before that, all memzone reserve were zero out by default.
Is there code relying on that? I'm not sure, but it still is good 
practice to do it.

I submitted an RFC regarding this:
http://dpdk.org/ml/archives/dev/2015-November/028416.html

The idea would be to keep the available memory we are managing zeroed at 
all times.

Sergio
>>>> ...
>>>>   		if (orig) {
>>>>   			hugepg_tbl[i].orig_va = virtaddr;
>>>> -			memset(virtaddr, 0, hugepage_sz);
>>>> +			memset(virtaddr, 0, 8);
>>>>   		}
>>> Probably worth adding a one or two line comment here to avoid someone
>>> thinking that it is a bug at some later stage. The text in the commit message
>>> above is suitable.
>>>
>> Good suggestion! Will add it :)
>>
>>> John.
>>> --

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-19  6:32                   ` Wang, Zhihong
@ 2015-11-19  9:18                     ` Sergio Gonzalez Monroy
  2015-11-23  2:54                     ` Xie, Huawei
  1 sibling, 0 replies; 25+ messages in thread
From: Sergio Gonzalez Monroy @ 2015-11-19  9:18 UTC (permalink / raw)
  To: Wang, Zhihong, Xie, Huawei, Stephen Hemminger, Richardson, Bruce; +Cc: dev

On 19/11/2015 06:32, Wang, Zhihong wrote:
>> -----Original Message-----
>> From: Xie, Huawei
>> Sent: Thursday, November 19, 2015 2:05 PM
>> To: Wang, Zhihong <zhihong.wang@intel.com>; Stephen Hemminger
>> <stephen@networkplumber.org>; Richardson, Bruce
>> <bruce.richardson@intel.com>
>> Cc: dev@dpdk.org
>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
>> hugepage zero-filling
>>
>> On 11/19/2015 10:16 AM, Wang, Zhihong wrote:
>>>> -----Original Message-----
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen
>>>> Hemminger
>>>> Sent: Thursday, November 19, 2015 3:09 AM
>>>> To: Richardson, Bruce <bruce.richardson@intel.com>
>>>> Cc: dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
>>>> unnecessary hugepage zero-filling
>>>>
>>>> On Wed, 18 Nov 2015 16:13:32 +0000
>>>> "Richardson, Bruce" <bruce.richardson@intel.com> wrote:
>>>>
>>>>>> -----Original Message-----
>>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen
>>>>>> Hemminger
>>>>>> Sent: Wednesday, November 18, 2015 4:00 PM
>>>>>> To: Xie, Huawei <huawei.xie@intel.com>
>>>>>> Cc: dev@dpdk.org
>>>>>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
>>>>>> unnecessary hugepage zero-filling
>>>>>>
>>>>>> On Wed, 18 Nov 2015 12:07:54 +0000
>>>>>> "Xie, Huawei" <huawei.xie@intel.com> wrote:
>>>>>>
>>>>>>>>>> The kernel fills new allocated (huge) pages with zeros.
>>>>>>>>>> DPDK just has to touch the pages to trigger the allocation.
>>>>>>> I think we shouldn't reply on the assumption that kernel has
>>>>>>> zeroed the memory. Kernel zeroes the memory mostly to avoid
>>>>>>> information leakage.It could also achieve this by setting each bit to 1.
>>>>>>> What we indeed need to check is later DPDK initialization code
>>>>>>> doesn't assume the memory has been zeroed. Otherwise zero only
>>>>>>> that part of the memory. Does this makes sense?
>>>>>> If all new pages are zero, why does DPDK have to pre-touch the
>>>>>> pages at all?
>>>>> The pages won't actually be mapped into the processes address space
>>>>> until
>>>> accessed.
>>>>> /Bruce
>>>> Isn't that what mmap MAP_POPULATE flag (not currently used) will do.
>>> Yes, the MAP_POPULATE does literally the same thing.
>>> This flag is implemented since Linux 2.5.46 according to Linux man
>>> page, guess that's why DPDK fault the page tables manually in the
>>> first place. :)
>>>
>>> I think we can use this flag since it makes the code clearer.
>> The manual says MAP_POPULATE is only supported for private mappings since
>> Linux 2.6.23.
> I've done check before and MAP_SHARED | MAP_POPULATE worked together correctly. Is there any implicit complication here?
>

None that I can see.

Sergio
>>> /Zhihong
>>>
>>>

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-18 19:09             ` Stephen Hemminger
  2015-11-19  2:15               ` Wang, Zhihong
@ 2015-11-20 12:15               ` Bruce Richardson
  1 sibling, 0 replies; 25+ messages in thread
From: Bruce Richardson @ 2015-11-20 12:15 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: dev

On Wed, Nov 18, 2015 at 11:09:06AM -0800, Stephen Hemminger wrote:
> On Wed, 18 Nov 2015 16:13:32 +0000
> "Richardson, Bruce" <bruce.richardson@intel.com> wrote:
> 
> > 
> > 
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen Hemminger
> > > Sent: Wednesday, November 18, 2015 4:00 PM
> > > To: Xie, Huawei <huawei.xie@intel.com>
> > > Cc: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
> > > hugepage zero-filling
> > > 
> > > On Wed, 18 Nov 2015 12:07:54 +0000
> > > "Xie, Huawei" <huawei.xie@intel.com> wrote:
> > > 
> > > > >>> The kernel fills new allocated (huge) pages with zeros.
> > > > >>> DPDK just has to touch the pages to trigger the allocation.
> > > > I think we shouldn't reply on the assumption that kernel has zeroed
> > > > the memory. Kernel zeroes the memory mostly to avoid information
> > > > leakage.It could also achieve this by setting each bit to 1.
> > > > What we indeed need to check is later DPDK initialization code doesn't
> > > > assume the memory has been zeroed. Otherwise zero only that part of
> > > > the memory. Does this makes sense?
> > > 
> > > If all new pages are zero, why does DPDK have to pre-touch the pages at
> > > all?
> > 
> > The pages won't actually be mapped into the processes address space until accessed. 
> > 
> > /Bruce
> 
> Isn't that what mmap MAP_POPULATE flag (not currently used) will do.

Hadn't seen that flag before, but it does indeed look to do exactly what we
want.

/Bruce

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-19  6:32                   ` Wang, Zhihong
  2015-11-19  9:18                     ` Sergio Gonzalez Monroy
@ 2015-11-23  2:54                     ` Xie, Huawei
  2015-11-23 10:18                       ` Bruce Richardson
  1 sibling, 1 reply; 25+ messages in thread
From: Xie, Huawei @ 2015-11-23  2:54 UTC (permalink / raw)
  To: Wang, Zhihong, Stephen Hemminger, Richardson, Bruce; +Cc: dev

On 11/19/2015 2:32 PM, Wang, Zhihong wrote:
>> -----Original Message-----
>> From: Xie, Huawei
>> Sent: Thursday, November 19, 2015 2:05 PM
>> To: Wang, Zhihong <zhihong.wang@intel.com>; Stephen Hemminger
>> <stephen@networkplumber.org>; Richardson, Bruce
>> <bruce.richardson@intel.com>
>> Cc: dev@dpdk.org
>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary
>> hugepage zero-filling
>>
>> On 11/19/2015 10:16 AM, Wang, Zhihong wrote:
>>>> -----Original Message-----
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen
>>>> Hemminger
>>>> Sent: Thursday, November 19, 2015 3:09 AM
>>>> To: Richardson, Bruce <bruce.richardson@intel.com>
>>>> Cc: dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
>>>> unnecessary hugepage zero-filling
>>>>
>>>> On Wed, 18 Nov 2015 16:13:32 +0000
>>>> "Richardson, Bruce" <bruce.richardson@intel.com> wrote:
>>>>
>>>>>> -----Original Message-----
>>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Stephen
>>>>>> Hemminger
>>>>>> Sent: Wednesday, November 18, 2015 4:00 PM
>>>>>> To: Xie, Huawei <huawei.xie@intel.com>
>>>>>> Cc: dev@dpdk.org
>>>>>> Subject: Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
>>>>>> unnecessary hugepage zero-filling
>>>>>>
>>>>>> On Wed, 18 Nov 2015 12:07:54 +0000
>>>>>> "Xie, Huawei" <huawei.xie@intel.com> wrote:
>>>>>>
>>>>>>>>>> The kernel fills new allocated (huge) pages with zeros.
>>>>>>>>>> DPDK just has to touch the pages to trigger the allocation.
>>>>>>> I think we shouldn't reply on the assumption that kernel has
>>>>>>> zeroed the memory. Kernel zeroes the memory mostly to avoid
>>>>>>> information leakage.It could also achieve this by setting each bit to 1.
>>>>>>> What we indeed need to check is later DPDK initialization code
>>>>>>> doesn't assume the memory has been zeroed. Otherwise zero only
>>>>>>> that part of the memory. Does this makes sense?
>>>>>> If all new pages are zero, why does DPDK have to pre-touch the
>>>>>> pages at all?
>>>>> The pages won't actually be mapped into the processes address space
>>>>> until
>>>> accessed.
>>>>> /Bruce
>>>> Isn't that what mmap MAP_POPULATE flag (not currently used) will do.
>>> Yes, the MAP_POPULATE does literally the same thing.
>>> This flag is implemented since Linux 2.5.46 according to Linux man
>>> page, guess that's why DPDK fault the page tables manually in the
>>> first place. :)
>>>
>>> I think we can use this flag since it makes the code clearer.
>> The manual says MAP_POPULATE is only supported for private mappings since
>> Linux 2.6.23.
> I've done check before and MAP_SHARED | MAP_POPULATE worked together correctly. Is there any implicit complication here?
I checked shared mapping with MAP_POPULATE between two processes. It
works. Then i did some search, find the manual is also ambiguous to
others and thus have been changed, :).
Before: MAP_POPULATE is only supported for private mappings since Linux
2.6.23.
After: MAP_POPULATE is supported for private mappings only since Linux
2.6.23.
http://stackoverflow.com/questions/23502361/linux-mmap-with-map-populate-man-page-seems-to-give-wrong-info
>
>>> /Zhihong
>>>
>>>
>


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-19  9:14         ` Sergio Gonzalez Monroy
@ 2015-11-23  3:46           ` Xie, Huawei
  2015-11-23  4:07             ` Stephen Hemminger
  0 siblings, 1 reply; 25+ messages in thread
From: Xie, Huawei @ 2015-11-23  3:46 UTC (permalink / raw)
  To: Gonzalez Monroy, Sergio, Wang, Zhihong, Mcnamara, John, dev

On 11/19/2015 5:15 PM, Gonzalez Monroy, Sergio wrote:
> On 18/11/2015 12:07, Xie, Huawei wrote:
>> On 11/18/2015 6:45 PM, Wang, Zhihong wrote:
>>>> -----Original Message-----
>>>> From: Mcnamara, John
>>>> Sent: Wednesday, November 18, 2015 6:40 PM
>>>> To: Wang, Zhihong <zhihong.wang@intel.com>; dev@dpdk.org
>>>> Subject: RE: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
>>>> unnecessary
>>>> hugepage zero-filling
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhihong Wang
>>>>> Sent: Wednesday, November 18, 2015 3:27 AM
>>>>> To: dev@dpdk.org
>>>>> Subject: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove
>>>>> unnecessary
>>>>> hugepage zero-filling
>>>>>
>>>>> The kernel fills new allocated (huge) pages with zeros.
>>>>> DPDK just has to touch the pages to trigger the allocation.
>> I think we shouldn't reply on the assumption that kernel has zeroed the
>> memory. Kernel zeroes the memory mostly to avoid information leakage.It
>> could also achieve this by setting each bit to 1.
>> What we indeed need to check is later DPDK initialization code doesn't
>> assume the memory has been zeroed. Otherwise zero only that part of the
>> memory. Does this makes sense?
>
> Why cannot we rely on the kernel zeroing the memory ?
> If that behavior were to change, then we can zero out the memory
> ourselves.
It is undocumented kernel behavior. My opinion is if not a big burden,
zero out the needed memory ourselves, otherwise resort to this kernel
behavior.

>
> Bruce pointed out to me that the semantics have changed a bit since we
> introduced
> rte_memzone_free.
> Before that, all memzone reserve were zero out by default.
> Is there code relying on that? I'm not sure, but it still is good
> practice to do it.
>
> I submitted an RFC regarding this:
> http://dpdk.org/ml/archives/dev/2015-November/028416.html
>
> The idea would be to keep the available memory we are managing zeroed
> at all times.
>
> Sergio
>>>>> ...
>>>>>           if (orig) {
>>>>>               hugepg_tbl[i].orig_va = virtaddr;
>>>>> -            memset(virtaddr, 0, hugepage_sz);
>>>>> +            memset(virtaddr, 0, 8);
>>>>>           }
>>>> Probably worth adding a one or two line comment here to avoid someone
>>>> thinking that it is a bug at some later stage. The text in the
>>>> commit message
>>>> above is suitable.
>>>>
>>> Good suggestion! Will add it :)
>>>
>>>> John.
>>>> -- 
>
>


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-23  3:46           ` Xie, Huawei
@ 2015-11-23  4:07             ` Stephen Hemminger
  2015-11-23  5:05               ` Xie, Huawei
  0 siblings, 1 reply; 25+ messages in thread
From: Stephen Hemminger @ 2015-11-23  4:07 UTC (permalink / raw)
  To: Xie, Huawei; +Cc: dev

On Mon, 23 Nov 2015 03:46:31 +0000
"Xie, Huawei" <huawei.xie@intel.com> wrote:

> >
> > Why cannot we rely on the kernel zeroing the memory ?
> > If that behavior were to change, then we can zero out the memory
> > ourselves.  
> It is undocumented kernel behavior. My opinion is if not a big burden,
> zero out the needed memory ourselves, otherwise resort to this kernel
> behavior.

Really, I think it is more an oversight of missing documentation,
the kernel has always (and will continue) to zero out memory that is given
to a process. If it didn't it would be a massive security hole.

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-23  4:07             ` Stephen Hemminger
@ 2015-11-23  5:05               ` Xie, Huawei
  2015-11-23  6:52                 ` Stephen Hemminger
  0 siblings, 1 reply; 25+ messages in thread
From: Xie, Huawei @ 2015-11-23  5:05 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: dev

On 11/23/2015 12:07 PM, Stephen Hemminger wrote:
> On Mon, 23 Nov 2015 03:46:31 +0000
> "Xie, Huawei" <huawei.xie@intel.com> wrote:
>
>>> Why cannot we rely on the kernel zeroing the memory ?
>>> If that behavior were to change, then we can zero out the memory
>>> ourselves.  
>> It is undocumented kernel behavior. My opinion is if not a big burden,
>> zero out the needed memory ourselves, otherwise resort to this kernel
>> behavior.
> Really, I think it is more an oversight of missing documentation,
> the kernel has always (and will continue) to zero out memory that is given
> to a process. If it didn't it would be a massive security hole.
Agree. I believe this behavior will not change in future. For the
security issue, kernel could also set all bits like to 1. Just wonder if
this is best practice and whether there are other user space programs
rely on this behavior.


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-23  5:05               ` Xie, Huawei
@ 2015-11-23  6:52                 ` Stephen Hemminger
  2015-11-25 18:24                   ` Xie, Huawei
  0 siblings, 1 reply; 25+ messages in thread
From: Stephen Hemminger @ 2015-11-23  6:52 UTC (permalink / raw)
  To: Xie, Huawei; +Cc: dev

On Mon, 23 Nov 2015 05:05:21 +0000
"Xie, Huawei" <huawei.xie@intel.com> wrote:

> On 11/23/2015 12:07 PM, Stephen Hemminger wrote:
> > On Mon, 23 Nov 2015 03:46:31 +0000
> > "Xie, Huawei" <huawei.xie@intel.com> wrote:
> >
> >>> Why cannot we rely on the kernel zeroing the memory ?
> >>> If that behavior were to change, then we can zero out the memory
> >>> ourselves.  
> >> It is undocumented kernel behavior. My opinion is if not a big burden,
> >> zero out the needed memory ourselves, otherwise resort to this kernel
> >> behavior.
> > Really, I think it is more an oversight of missing documentation,
> > the kernel has always (and will continue) to zero out memory that is given
> > to a process. If it didn't it would be a massive security hole.
> Agree. I believe this behavior will not change in future. For the
> security issue, kernel could also set all bits like to 1. Just wonder if
> this is best practice and whether there are other user space programs
> rely on this behavior.
> 

Glibc almost certainly depends on this, because heap is grown by mmaping addtional
memory.

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-23  2:54                     ` Xie, Huawei
@ 2015-11-23 10:18                       ` Bruce Richardson
  0 siblings, 0 replies; 25+ messages in thread
From: Bruce Richardson @ 2015-11-23 10:18 UTC (permalink / raw)
  To: Xie, Huawei; +Cc: dev

On Mon, Nov 23, 2015 at 02:54:54AM +0000, Xie, Huawei wrote:
 I checked shared mapping with MAP_POPULATE between two processes. It
> works. Then i did some search, find the manual is also ambiguous to
> others and thus have been changed, :).
> Before: MAP_POPULATE is only supported for private mappings since Linux
> 2.6.23.
> After: MAP_POPULATE is supported for private mappings only since Linux
> 2.6.23.
> http://stackoverflow.com/questions/23502361/linux-mmap-with-map-populate-man-page-seems-to-give-wrong-info
> >

Doesn't matter much for us, as we use shared rather than private mappings for the
hugepages.

/Bruce

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling
  2015-11-23  6:52                 ` Stephen Hemminger
@ 2015-11-25 18:24                   ` Xie, Huawei
  0 siblings, 0 replies; 25+ messages in thread
From: Xie, Huawei @ 2015-11-25 18:24 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: dev

On 11/23/2015 2:52 PM, Stephen Hemminger wrote:
> On Mon, 23 Nov 2015 05:05:21 +0000
> "Xie, Huawei" <huawei.xie@intel.com> wrote:
>
>> On 11/23/2015 12:07 PM, Stephen Hemminger wrote:
>>> On Mon, 23 Nov 2015 03:46:31 +0000
>>> "Xie, Huawei" <huawei.xie@intel.com> wrote:
>>>
>>>>> Why cannot we rely on the kernel zeroing the memory ?
>>>>> If that behavior were to change, then we can zero out the memory
>>>>> ourselves.  
>>>> It is undocumented kernel behavior. My opinion is if not a big burden,
>>>> zero out the needed memory ourselves, otherwise resort to this kernel
>>>> behavior.
>>> Really, I think it is more an oversight of missing documentation,
>>> the kernel has always (and will continue) to zero out memory that is given
>>> to a process. If it didn't it would be a massive security hole.
>> Agree. I believe this behavior will not change in future. For the
>> security issue, kernel could also set all bits like to 1. Just wonder if
>> this is best practice and whether there are other user space programs
>> rely on this behavior.
>>
> Glibc almost certainly depends on this, because heap is grown by mmaping addtional
> memory.
Thanks. It is OK, but still don't feel good, :). It is like that we
require the clear_user_page(and other unix, windows equivalent) is
always memset(ptr, 0 , PAGE_SIZE). To me, memset(ptr, 1, PAGE_SIZE)
doesn't make difference.


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [dpdk-dev] [RFC PATCH 0/2] Reduce DPDK initialization time
  2015-11-18  3:27 [dpdk-dev] [RFC PATCH 0/2] Reduce DPDK initialization time Zhihong Wang
  2015-11-18  3:27 ` [dpdk-dev] [RFC PATCH 1/2] lib/librte_eal: Reduce timer " Zhihong Wang
  2015-11-18  3:27 ` [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling Zhihong Wang
@ 2015-12-24  8:34 ` Qiu, Michael
  2 siblings, 0 replies; 25+ messages in thread
From: Qiu, Michael @ 2015-12-24  8:34 UTC (permalink / raw)
  To: Wang, Zhihong, dev; +Cc: Mata, Jhoanna E

On 11/18/2015 6:30 PM, Zhihong Wang wrote:
> This RFC patch aims to reduce DPDK initialization time, which is important in cases such as micro service.
>
> Changes are:
>
> 1. Reduce timer initialization time
>
> 2. Remove unnecessary hugepage zero-filling operations
>
> With this patch:
>
> 1. Timer initialization time can be reduced by 4/10 second
>
> 2. Memory initialization time can be reduced nearly by half
>
> The 2nd topic has been brought up before in this thread:
> http://dpdk.org/dev/patchwork/patch/4219/
>
> Zhihong Wang (2):
>   lib/librte_eal: Reduce timer initialization time
>   lib/librte_eal: Remove unnecessary hugepage zero-filling
>
>  lib/librte_eal/linuxapp/eal/eal_memory.c | 5 +----
>  lib/librte_eal/linuxapp/eal/eal_timer.c  | 2 +-
>  2 files changed, 2 insertions(+), 5 deletions(-)
>

As I tested with 8192 hugepages(size 2M), one nic 82599 bind, using time
to get the seconds used:
with this patch:

echo quit | time ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 -- -i
2.15 user
5.55 system
0:07.82 elapsed

Without patch:
echo quit | time ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 -- -i
3.18 user
5.63 system
0:09.32 elapsed

1.5s saved,  16% improvement, I don't know if this is good enough, but
indeed save lots of time.

Thanks,
Michael

^ permalink raw reply	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2015-12-24  8:34 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-18  3:27 [dpdk-dev] [RFC PATCH 0/2] Reduce DPDK initialization time Zhihong Wang
2015-11-18  3:27 ` [dpdk-dev] [RFC PATCH 1/2] lib/librte_eal: Reduce timer " Zhihong Wang
2015-11-18  3:27 ` [dpdk-dev] [RFC PATCH 2/2] lib/librte_eal: Remove unnecessary hugepage zero-filling Zhihong Wang
2015-11-18 10:39   ` Mcnamara, John
2015-11-18 10:44     ` Wang, Zhihong
2015-11-18 12:07       ` Xie, Huawei
2015-11-18 16:00         ` Stephen Hemminger
2015-11-18 16:13           ` Richardson, Bruce
2015-11-18 19:09             ` Stephen Hemminger
2015-11-19  2:15               ` Wang, Zhihong
2015-11-19  6:04                 ` Xie, Huawei
2015-11-19  6:32                   ` Wang, Zhihong
2015-11-19  9:18                     ` Sergio Gonzalez Monroy
2015-11-23  2:54                     ` Xie, Huawei
2015-11-23 10:18                       ` Bruce Richardson
2015-11-20 12:15               ` Bruce Richardson
2015-11-19  3:54         ` Wang, Zhihong
2015-11-19  6:09           ` Xie, Huawei
2015-11-19  9:14         ` Sergio Gonzalez Monroy
2015-11-23  3:46           ` Xie, Huawei
2015-11-23  4:07             ` Stephen Hemminger
2015-11-23  5:05               ` Xie, Huawei
2015-11-23  6:52                 ` Stephen Hemminger
2015-11-25 18:24                   ` Xie, Huawei
2015-12-24  8:34 ` [dpdk-dev] [RFC PATCH 0/2] Reduce DPDK initialization time Qiu, Michael

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).