Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Thomas Monjalon]

Hi,

2016-08-10 23:30, Verkamp, Daniel:
> It seems that with DPDK 16.07, rte_zmalloc() and related functions no
> longer return zeroed memory reliably on FreeBSD.
> 
> I notice that commit b78c9175118f7d61022ddc5c62ce54a1bd73cea5 ("mem: do
> not zero out memory on zmalloc") removed the explicit memset() that used
> to ensure the buffer was zeroed; its log message says:
> 
> "Zeroing out memory on rte_zmalloc_socket is not required anymore since
> all allocated memory is already zeroed."

On Linux, the memory is zeroed by the kernel.
Then the zero value is maintained in the rte_malloc pool by rte_free.

> However, I don't see how this is guaranteed (at least for FreeBSD), and
> it is not true in practice.  I've attached a minimized reproducer program -
> running it twice in a row fails reliably for me.
> 
> Is there a missing step in FreeBSD, or is it a more general problem for
> other platforms?

I guess the initial value from the kernel has been verified only on Linux.
We could re-add a memset for FreeBSD.

Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Harris, James R]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> Sent: Thursday, August 11, 2016 12:05 AM
> To: users@dpdk.org; dev@dpdk.org; Gonzalez Monroy, Sergio; Richardson,
> Bruce
> Cc: Verkamp, Daniel
> Subject: Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed
> memory on FreeBSD
> 
> Hi,
> 
> 2016-08-10 23:30, Verkamp, Daniel:
> > It seems that with DPDK 16.07, rte_zmalloc() and related functions no
> > longer return zeroed memory reliably on FreeBSD.
> >
> > I notice that commit b78c9175118f7d61022ddc5c62ce54a1bd73cea5 ("mem:
> do
> > not zero out memory on zmalloc") removed the explicit memset() that
> used
> > to ensure the buffer was zeroed; its log message says:
> >
> > "Zeroing out memory on rte_zmalloc_socket is not required anymore since
> > all allocated memory is already zeroed."
> 
> On Linux, the memory is zeroed by the kernel.
> Then the zero value is maintained in the rte_malloc pool by rte_free.
> 
> > However, I don't see how this is guaranteed (at least for FreeBSD), and
> > it is not true in practice.  I've attached a minimized reproducer program -
> > running it twice in a row fails reliably for me.
> >
> > Is there a missing step in FreeBSD, or is it a more general problem for
> > other platforms?
> 
> I guess the initial value from the kernel has been verified only on Linux.
> We could re-add a memset for FreeBSD.

The problem is that the FreeBSD contigmem driver does not re-zero the huge
pages each time they are mmap'd - they are only zeroed when contigmem
initially loads.  I will push a patch for this shortly.

Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Sergio Gonzalez Monroy]

On 15/08/2016 18:23, Harris, James R wrote:
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
>> Sent: Thursday, August 11, 2016 12:05 AM
>> To: users@dpdk.org; dev@dpdk.org; Gonzalez Monroy, Sergio; Richardson,
>> Bruce
>> Cc: Verkamp, Daniel
>> Subject: Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed
>> memory on FreeBSD
>>
>> Hi,
>>
>> 2016-08-10 23:30, Verkamp, Daniel:
>>> It seems that with DPDK 16.07, rte_zmalloc() and related functions no
>>> longer return zeroed memory reliably on FreeBSD.
>>>
>>> I notice that commit b78c9175118f7d61022ddc5c62ce54a1bd73cea5 ("mem:
>> do
>>> not zero out memory on zmalloc") removed the explicit memset() that
>> used
>>> to ensure the buffer was zeroed; its log message says:
>>>
>>> "Zeroing out memory on rte_zmalloc_socket is not required anymore since
>>> all allocated memory is already zeroed."
>> On Linux, the memory is zeroed by the kernel.
>> Then the zero value is maintained in the rte_malloc pool by rte_free.
>>
>>> However, I don't see how this is guaranteed (at least for FreeBSD), and
>>> it is not true in practice.  I've attached a minimized reproducer program -
>>> running it twice in a row fails reliably for me.
>>>
>>> Is there a missing step in FreeBSD, or is it a more general problem for
>>> other platforms?
>> I guess the initial value from the kernel has been verified only on Linux.
>> We could re-add a memset for FreeBSD.
> The problem is that the FreeBSD contigmem driver does not re-zero the huge
> pages each time they are mmap'd - they are only zeroed when contigmem
> initially loads.  I will push a patch for this shortly.

So that is the case where we run the app more than once, right?
I missed that, I only ran it once.

Sergio

Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Harris, James R]

> -----Original Message-----
> From: Gonzalez Monroy, Sergio
> Sent: Tuesday, August 16, 2016 12:37 AM
> To: Harris, James R; Thomas Monjalon; users@dpdk.org; dev@dpdk.org;
> Richardson, Bruce
> Cc: Verkamp, Daniel
> Subject: Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed
> memory on FreeBSD
> 
> On 15/08/2016 18:23, Harris, James R wrote:
> >

<snip>

> > The problem is that the FreeBSD contigmem driver does not re-zero the
> huge
> > pages each time they are mmap'd - they are only zeroed when contigmem
> > initially loads.  I will push a patch for this shortly.
> 
> So that is the case where we run the app more than once, right?
> I missed that, I only ran it once.

Correct - it works OK the first time.  The problem only occurs if you run the app
more than once (unless you unload and reload contigmem before running the
app a second time).

-Jim

Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Lewis Donzis]

I’m curious about how/whether this got resolved.  The 16.07.1 code doesn’t appear to have this fixed.

Is it still forthcoming?

Thanks,
lew

Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Sergio Gonzalez Monroy]

On 03/11/2016 20:04, Lewis Donzis wrote:
> I’m curious about how/whether this got resolved.  The 16.07.1 code doesn’t appear to have this fixed.
>
> Is it still forthcoming?
>
> Thanks,
> lew

It should have been fixed in 16.07.1.
http://dpdk.org/browse/dpdk-stable/commit/?id=82f931805506efbb8b5046e9045bec8f04bbabf6

Do you have an easy test to reproduce? can you reproduce it with any of 
the app/examples?

Sergio

Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Lewis Donzis]

> On Nov 4, 2016, at 11:38 AM, Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com> wrote:
> 
> On 03/11/2016 20:04, Lewis Donzis wrote:
>> I’m curious about how/whether this got resolved.  The 16.07.1 code doesn’t appear to have this fixed.
>> 
>> Is it still forthcoming?
>> 
>> Thanks,
>> lew
> 
> It should have been fixed in 16.07.1.
> http://dpdk.org/browse/dpdk-stable/commit/?id=82f931805506efbb8b5046e9045bec8f04bbabf6
> 
> Do you have an easy test to reproduce? can you reproduce it with any of the app/examples?

Oh, sorry, I was looking at the rte_zmalloc() code, assuming it would be changed there, but it looks like the commit was to change contigmem instead, which seems perfectly logical and reasonable.

It’s entirely possible that the folks here who tested 16.07.1 forgot to replace the contigmem driver, so perhaps that is the problem.

We’ll check it out and report back.

Thanks!
lew

Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Lewis Donzis]

> On Nov 4, 2016, at 11:38 AM, Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com> wrote:
> It should have been fixed in 16.07.1.
> http://dpdk.org/browse/dpdk-stable/commit/?id=82f931805506efbb8b5046e9045bec8f04bbabf6

Hi, Sergio.

We confirmed that it works perfectly with 16.07.1 with the new contigmem driver.

Thanks!
lew