[dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Verkamp, Daniel]

Hi,

It seems that with DPDK 16.07, rte_zmalloc() and related functions no
longer return zeroed memory reliably on FreeBSD.

I notice that commit b78c9175118f7d61022ddc5c62ce54a1bd73cea5 ("mem: do
not zero out memory on zmalloc") removed the explicit memset() that used
to ensure the buffer was zeroed; its log message says:

"Zeroing out memory on rte_zmalloc_socket is not required anymore since
all allocated memory is already zeroed."

However, I don't see how this is guaranteed (at least for FreeBSD), and
it is not true in practice.  I've attached a minimized reproducer program - running it twice in a row fails reliably for me.

Is there a missing step in FreeBSD, or is it a more general problem for
other platforms?

Thanks,
-- Daniel

Re: [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Verkamp, Daniel]

On Wed, 2016-08-10 at 23:30 +0000, Verkamp, Daniel wrote:
> I've attached a minimized reproducer
> program - running it twice in a row fails reliably for me.

It seems the attachment got stripped, so here it is inline:

#include <stdio.h>
#include <rte_config.h>
#include <rte_eal.h>
#include <rte_malloc.h>

int main(int argc, char **argv)
{
        unsigned char *buf;
        int i;
        size_t size = 5200;

        rte_eal_init(argc, argv);

        buf = rte_zmalloc("test", size, 64);
        printf("Got buf %p\n", buf);

        for (i = 0; i < size; i++) {
                if (buf[i] != 0) {
                        printf("byte %d is %x\n", i, buf[i]);
                        return 1;
                }
        }

        printf("All bytes cleared\n");
        printf("Writing garbage\n");

        for (i = 0; i < size; i++) {
                buf[i] = 0x5A;
        }

        return 0;
}

Re: [dpdk-users] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Thomas Monjalon]

Hi,

2016-08-10 23:30, Verkamp, Daniel:
> It seems that with DPDK 16.07, rte_zmalloc() and related functions no
> longer return zeroed memory reliably on FreeBSD.
> 
> I notice that commit b78c9175118f7d61022ddc5c62ce54a1bd73cea5 ("mem: do
> not zero out memory on zmalloc") removed the explicit memset() that used
> to ensure the buffer was zeroed; its log message says:
> 
> "Zeroing out memory on rte_zmalloc_socket is not required anymore since
> all allocated memory is already zeroed."

On Linux, the memory is zeroed by the kernel.
Then the zero value is maintained in the rte_malloc pool by rte_free.

> However, I don't see how this is guaranteed (at least for FreeBSD), and
> it is not true in practice.  I've attached a minimized reproducer program -
> running it twice in a row fails reliably for me.
> 
> Is there a missing step in FreeBSD, or is it a more general problem for
> other platforms?

I guess the initial value from the kernel has been verified only on Linux.
We could re-add a memset for FreeBSD.

Re: [dpdk-users] [dpdk-dev] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Harris, James R]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> Sent: Thursday, August 11, 2016 12:05 AM
> To: users@dpdk.org; dev@dpdk.org; Gonzalez Monroy, Sergio; Richardson,
> Bruce
> Cc: Verkamp, Daniel
> Subject: Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed
> memory on FreeBSD
> 
> Hi,
> 
> 2016-08-10 23:30, Verkamp, Daniel:
> > It seems that with DPDK 16.07, rte_zmalloc() and related functions no
> > longer return zeroed memory reliably on FreeBSD.
> >
> > I notice that commit b78c9175118f7d61022ddc5c62ce54a1bd73cea5 ("mem:
> do
> > not zero out memory on zmalloc") removed the explicit memset() that
> used
> > to ensure the buffer was zeroed; its log message says:
> >
> > "Zeroing out memory on rte_zmalloc_socket is not required anymore since
> > all allocated memory is already zeroed."
> 
> On Linux, the memory is zeroed by the kernel.
> Then the zero value is maintained in the rte_malloc pool by rte_free.
> 
> > However, I don't see how this is guaranteed (at least for FreeBSD), and
> > it is not true in practice.  I've attached a minimized reproducer program -
> > running it twice in a row fails reliably for me.
> >
> > Is there a missing step in FreeBSD, or is it a more general problem for
> > other platforms?
> 
> I guess the initial value from the kernel has been verified only on Linux.
> We could re-add a memset for FreeBSD.

The problem is that the FreeBSD contigmem driver does not re-zero the huge
pages each time they are mmap'd - they are only zeroed when contigmem
initially loads.  I will push a patch for this shortly.

Re: [dpdk-users] [dpdk-dev] rte_zmalloc() returning non-zeroed memory on FreeBSD

[Harris, James R]

> -----Original Message-----
> From: Gonzalez Monroy, Sergio
> Sent: Tuesday, August 16, 2016 12:37 AM
> To: Harris, James R; Thomas Monjalon; users@dpdk.org; dev@dpdk.org;
> Richardson, Bruce
> Cc: Verkamp, Daniel
> Subject: Re: [dpdk-dev] [dpdk-users] rte_zmalloc() returning non-zeroed
> memory on FreeBSD
> 
> On 15/08/2016 18:23, Harris, James R wrote:
> >

<snip>

> > The problem is that the FreeBSD contigmem driver does not re-zero the
> huge
> > pages each time they are mmap'd - they are only zeroed when contigmem
> > initially loads.  I will push a patch for this shortly.
> 
> So that is the case where we run the app more than once, right?
> I missed that, I only ran it once.

Correct - it works OK the first time.  The problem only occurs if you run the app
more than once (unless you unload and reload contigmem before running the
app a second time).

-Jim