[dpdk-dev] A (possible) problem with `--no-huge` option

[dpdk-dev] A (possible) problem with `--no-huge` option

[Ilya Matveychikov]

Hi guys,

I have a problem while running DPDK with `--no-huge` option. It seems that the problem occurs since commit cdc242f260e766bd95a658b5e0686a62ec04f5b0 and that is the change that affects me:

+	if ((page & 0x7fffffffffffffULL) == 0)
+		return RTE_BAD_PHYS_ADDR;
+

What I did is to try to create memory pool using rte_pktmbuf_pool_create(). I dig into the issue and found that in my case “page" value is 0x0080000000000000 which means that the page is not present and “soft-dirty” (according to kernel’s documentation):

   * Bits 0-54  page frame number (PFN) if present
   * Bits 0-4   swap type if swapped
   * Bits 5-54  swap offset if swapped
   * Bit  55    pte is soft-dirty (see Documentation/vm/soft-dirty.txt)
   * Bit  56    page exclusively mapped (since 4.2)
   * Bits 57-60 zero
   * Bit  61    page is file-page or shared-anon (since 3.5)
   * Bit  62    page swapped
   * Bit  63    page present

So, before the change mentioned all “works” fine and such pages were not handled. But now the check causes rte_mempool_populate_default to fail with -EINVAL...
Can anyone familiar with the memory pool allocation helps with the issue?

Thanks in advice,
Ilya Matveychikov.

Re: [dpdk-dev] A (possible) problem with `--no-huge` option

[Olivier Matz]

Hi Ilya,

On Sun, 14 May 2017 14:34:14 +0400, Ilya Matveychikov <matvejchikov@gmail.com> wrote:
> Hi guys,
> 
> I have a problem while running DPDK with `--no-huge` option. It seems that the problem occurs since commit cdc242f260e766bd95a658b5e0686a62ec04f5b0 and that is the change that affects me:
> 
> +	if ((page & 0x7fffffffffffffULL) == 0)
> +		return RTE_BAD_PHYS_ADDR;
> +
> 
> What I did is to try to create memory pool using rte_pktmbuf_pool_create(). I dig into the issue and found that in my case “page" value is 0x0080000000000000 which means that the page is not present and “soft-dirty” (according to kernel’s documentation):
> 
>    * Bits 0-54  page frame number (PFN) if present
>    * Bits 0-4   swap type if swapped
>    * Bits 5-54  swap offset if swapped
>    * Bit  55    pte is soft-dirty (see Documentation/vm/soft-dirty.txt)
>    * Bit  56    page exclusively mapped (since 4.2)
>    * Bits 57-60 zero
>    * Bit  61    page is file-page or shared-anon (since 3.5)
>    * Bit  62    page swapped
>    * Bit  63    page present
> 
> So, before the change mentioned all “works” fine and such pages were not handled. But now the check causes rte_mempool_populate_default to fail with -EINVAL...
> Can anyone familiar with the memory pool allocation helps with the issue?
> 
> Thanks in advice,
> Ilya Matveychikov.
> 

I can reproduce the issue:

  make config T=x86_64-native-linuxapp-gcc
  make -j32 EXTRA_CFLAGS="-O0 -g"
  mkdir -p /mnt/huge
  mount -t hugetlbfs nodev /mnt/huge
  echo 256 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

  # ok
  ./build/app/testpmd -l 2,4 --log-level 8 --vdev=eth_null0 -- --no-numa --total-num-mbufs=4096 -i --port-topology=chained

  # fail
  ./build/app/testpmd --no-huge -l 2,4 --log-level 8 --vdev=eth_null0 -- --no-numa --total-num-mbufs=4096 -i --port-topology=chained


I confirm that rte_mem_virt2phy() returns RTE_BAD_PHYS_ADDR,
which makes rte_mempool_populate_virt() to fail.

Reverting cdc242f260e7 ("eal/linux: support running as unprivileged user")
fixes the problem. Actually, it makes rte_mem_virt2phy() return 0 instead
of RTE_BAD_PHYS_ADDR, which is seen as a valid address.

I think querying the physical address when using --no-huge does not make
sense because the memory is not locked, and could be swapped.

Another strange thing, when using --no-huge, the physical address returned
when allocating a memzone is the virtual address.

I see several solutions to fix the issue:

1/ Always set physical addresses to RTE_BAD_PHYS_ADDR when started
   with --no-huge. We consider that the physical address is invalid
   in that case and must not be used.

   This impacts rte_mem_virt2phy() and memzone_reserve*() functions.

   In rte_mempool_populate_virt(), don't expect a physical address
   if the application is started with --no-huge.

2/ Change rte_mem_virt2phy() to return the virtual address when we
   ask for the physical address when started with --no-huge. This is
   wrong, but consistent with what is done in memzones today.

   In rte_mem_virt2phy(), add at the beginning:

     if (!rte_eal_has_hugepages())
         return (intptr_t)virtaddr;

3/ lock pages in memory by reverting
   729f17a932dd ("mem: revert page locking when not using hugepages")

   This would make the physical address available.
   As explained in the commit log, this would also break the ability to
   start dpdk with --no-huge for non-root users.


I think 1/ is better. I'm sending a patch in reply to this mail.
Ilya, please let me know if it fixes your issue.

Regards,
Olivier

[dpdk-dev] [PATCH] eal: don't advertise a physical address when no hugepages

[Olivier Matz]

When populating a mempool with a virtual memory area, the mempool
library expects to be able to get the physical address of each page.

When started with --no-huge, the physical addresses may not be available
because the pages are not locked in memory. It sometimes returns
RTE_BAD_PHYS_ADDR, which makes the mempool_populate() function to fail.

This was working before the commit cdc242f260e7 ("eal/linux: support
running as unprivileged user"), because rte_mem_virt2phy() was returning
0 instead of RTE_BAD_PHYS_ADDR, which was seen as a valid physical
address.

Since --no-huge is a debug function that breaks the support of physical
drivers, always set physical addresses to RTE_BAD_PHYS_ADDR in memzones
or in rte_mem_virt2phy(), and ensure that mempool won't complain in that
case.

Fixes: cdc242f260e7 ("eal/linux: support running as unprivileged user")

CC: stable@dpdk.org
Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
---
 lib/librte_eal/common/eal_common_memzone.c | 5 ++++-
 lib/librte_eal/linuxapp/eal/eal_memory.c   | 7 +++++++
 lib/librte_mempool/rte_mempool.c           | 2 +-
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_memzone.c b/lib/librte_eal/common/eal_common_memzone.c
index 3026e36b8..c465c8fc2 100644
--- a/lib/librte_eal/common/eal_common_memzone.c
+++ b/lib/librte_eal/common/eal_common_memzone.c
@@ -251,7 +251,10 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 
 	mcfg->memzone_cnt++;
 	snprintf(mz->name, sizeof(mz->name), "%s", name);
-	mz->phys_addr = rte_malloc_virt2phy(mz_addr);
+	if (rte_eal_has_hugepages())
+		mz->phys_addr = rte_malloc_virt2phy(mz_addr);
+	else
+		mz->phys_addr = RTE_BAD_PHYS_ADDR;
 	mz->addr = mz_addr;
 	mz->len = (requested_len == 0 ? elem->size : requested_len);
 	mz->hugepage_sz = elem->ms->hugepage_sz;
diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 9c9baf628..1c99852f6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -108,6 +108,13 @@ test_phys_addrs_available(void)
 	if (rte_xen_dom0_supported())
 		return;
 
+	if (!rte_eal_has_hugepages()) {
+		RTE_LOG(ERR, EAL,
+			"Started without hugepages support, physical addresses not available\n");
+		phys_addrs_available = false;
+		return;
+	}
+
 	physaddr = rte_mem_virt2phy(&tmp);
 	if (physaddr == RTE_BAD_PHYS_ADDR) {
 		RTE_LOG(ERR, EAL,
diff --git a/lib/librte_mempool/rte_mempool.c b/lib/librte_mempool/rte_mempool.c
index f65310f60..6fc3c9c7c 100644
--- a/lib/librte_mempool/rte_mempool.c
+++ b/lib/librte_mempool/rte_mempool.c
@@ -476,7 +476,7 @@ rte_mempool_populate_virt(struct rte_mempool *mp, char *addr,
 		/* required for xen_dom0 to get the machine address */
 		paddr = rte_mem_phy2mch(-1, paddr);
 
-		if (paddr == RTE_BAD_PHYS_ADDR) {
+		if (paddr == RTE_BAD_PHYS_ADDR && rte_eal_has_hugepages()) {
 			ret = -EINVAL;
 			goto fail;
 		}
-- 
2.11.0

Re: [dpdk-dev] [PATCH] eal: don't advertise a physical address when no hugepages

[Jan Blunck]

On Fri, Jun 9, 2017 at 10:29 AM, Olivier Matz <olivier.matz@6wind.com> wrote:
> When populating a mempool with a virtual memory area, the mempool
> library expects to be able to get the physical address of each page.
>
> When started with --no-huge, the physical addresses may not be available
> because the pages are not locked in memory. It sometimes returns
> RTE_BAD_PHYS_ADDR, which makes the mempool_populate() function to fail.
>
> This was working before the commit cdc242f260e7 ("eal/linux: support
> running as unprivileged user"), because rte_mem_virt2phy() was returning
> 0 instead of RTE_BAD_PHYS_ADDR, which was seen as a valid physical
> address.
>
> Since --no-huge is a debug function that breaks the support of physical
> drivers, always set physical addresses to RTE_BAD_PHYS_ADDR in memzones
> or in rte_mem_virt2phy(), and ensure that mempool won't complain in that
> case.
>
> Fixes: cdc242f260e7 ("eal/linux: support running as unprivileged user")
>
> CC: stable@dpdk.org
> Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
> ---
>  lib/librte_eal/common/eal_common_memzone.c | 5 ++++-
>  lib/librte_eal/linuxapp/eal/eal_memory.c   | 7 +++++++
>  lib/librte_mempool/rte_mempool.c           | 2 +-
>  3 files changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/lib/librte_eal/common/eal_common_memzone.c b/lib/librte_eal/common/eal_common_memzone.c
> index 3026e36b8..c465c8fc2 100644
> --- a/lib/librte_eal/common/eal_common_memzone.c
> +++ b/lib/librte_eal/common/eal_common_memzone.c
> @@ -251,7 +251,10 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
>
>         mcfg->memzone_cnt++;
>         snprintf(mz->name, sizeof(mz->name), "%s", name);
> -       mz->phys_addr = rte_malloc_virt2phy(mz_addr);
> +       if (rte_eal_has_hugepages())
> +               mz->phys_addr = rte_malloc_virt2phy(mz_addr);
> +       else
> +               mz->phys_addr = RTE_BAD_PHYS_ADDR;

Since you set phys_addrs_available to false rte_malloc_virt2phy()
anyway returns RTE_BAD_PHYS_ADDR so I believe the conditional isn't
necessary here.

Rest of the patch looks good to me.

>         mz->addr = mz_addr;
>         mz->len = (requested_len == 0 ? elem->size : requested_len);
>         mz->hugepage_sz = elem->ms->hugepage_sz;
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index 9c9baf628..1c99852f6 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -108,6 +108,13 @@ test_phys_addrs_available(void)
>         if (rte_xen_dom0_supported())
>                 return;
>
> +       if (!rte_eal_has_hugepages()) {
> +               RTE_LOG(ERR, EAL,
> +                       "Started without hugepages support, physical addresses not available\n");
> +               phys_addrs_available = false;
> +               return;
> +       }
> +
>         physaddr = rte_mem_virt2phy(&tmp);
>         if (physaddr == RTE_BAD_PHYS_ADDR) {
>                 RTE_LOG(ERR, EAL,
> diff --git a/lib/librte_mempool/rte_mempool.c b/lib/librte_mempool/rte_mempool.c
> index f65310f60..6fc3c9c7c 100644
> --- a/lib/librte_mempool/rte_mempool.c
> +++ b/lib/librte_mempool/rte_mempool.c
> @@ -476,7 +476,7 @@ rte_mempool_populate_virt(struct rte_mempool *mp, char *addr,
>                 /* required for xen_dom0 to get the machine address */
>                 paddr = rte_mem_phy2mch(-1, paddr);
>
> -               if (paddr == RTE_BAD_PHYS_ADDR) {
> +               if (paddr == RTE_BAD_PHYS_ADDR && rte_eal_has_hugepages()) {
>                         ret = -EINVAL;
>                         goto fail;
>                 }
> --
> 2.11.0
>

Re: [dpdk-dev] [PATCH] eal: don't advertise a physical address when no hugepages

[Olivier Matz]

Hi Jan,

On Sat, 10 Jun 2017 10:31:22 +0200, Jan Blunck <jblunck@infradead.org> wrote:
> On Fri, Jun 9, 2017 at 10:29 AM, Olivier Matz <olivier.matz@6wind.com> wrote:
> > When populating a mempool with a virtual memory area, the mempool
> > library expects to be able to get the physical address of each page.
> >
> > When started with --no-huge, the physical addresses may not be available
> > because the pages are not locked in memory. It sometimes returns
> > RTE_BAD_PHYS_ADDR, which makes the mempool_populate() function to fail.
> >
> > This was working before the commit cdc242f260e7 ("eal/linux: support
> > running as unprivileged user"), because rte_mem_virt2phy() was returning
> > 0 instead of RTE_BAD_PHYS_ADDR, which was seen as a valid physical
> > address.
> >
> > Since --no-huge is a debug function that breaks the support of physical
> > drivers, always set physical addresses to RTE_BAD_PHYS_ADDR in memzones
> > or in rte_mem_virt2phy(), and ensure that mempool won't complain in that
> > case.
> >
> > Fixes: cdc242f260e7 ("eal/linux: support running as unprivileged user")
> >
> > CC: stable@dpdk.org
> > Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
> > ---
> >  lib/librte_eal/common/eal_common_memzone.c | 5 ++++-
> >  lib/librte_eal/linuxapp/eal/eal_memory.c   | 7 +++++++
> >  lib/librte_mempool/rte_mempool.c           | 2 +-
> >  3 files changed, 12 insertions(+), 2 deletions(-)
> >
> > diff --git a/lib/librte_eal/common/eal_common_memzone.c b/lib/librte_eal/common/eal_common_memzone.c
> > index 3026e36b8..c465c8fc2 100644
> > --- a/lib/librte_eal/common/eal_common_memzone.c
> > +++ b/lib/librte_eal/common/eal_common_memzone.c
> > @@ -251,7 +251,10 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
> >
> >         mcfg->memzone_cnt++;
> >         snprintf(mz->name, sizeof(mz->name), "%s", name);
> > -       mz->phys_addr = rte_malloc_virt2phy(mz_addr);
> > +       if (rte_eal_has_hugepages())
> > +               mz->phys_addr = rte_malloc_virt2phy(mz_addr);
> > +       else
> > +               mz->phys_addr = RTE_BAD_PHYS_ADDR;  
> 
> Since you set phys_addrs_available to false rte_malloc_virt2phy()
> anyway returns RTE_BAD_PHYS_ADDR so I believe the conditional isn't
> necessary here.
> 
> Rest of the patch looks good to me.

The variable phys_addrs_available only impacts rte_mem_virt2phy().
Here, for memzones allocation, rte_malloc_virt2phy() is used, and
it gets its physical address by retrieving it from the memseg structure.

With the full patch, "dump_memzone" displays something like:
  Zone 0: name:<rte_eth_dev_data>, phys:0xffffffffffffffff, len:0x30100, [...]
  ...

If I strip the memzone part, it displays:
  Zone 0: name:<rte_eth_dev_data>, phys:0x7fe382c62640, len:0x30100, [...]
  ...

So I think we should either keep the patch as is, or change the memseg
and malloc part like this (it's maybe better):

  --- a/lib/librte_eal/common/rte_malloc.c
  +++ b/lib/librte_eal/common/rte_malloc.c
  @@ -254,5 +254,7 @@ rte_malloc_virt2phy(const void *addr)
          const struct malloc_elem *elem = malloc_elem_from_data(addr);
          if (elem == NULL)
                  return 0;
  +       if (elem->ms->phys_addr == RTE_BAD_PHYS_ADDR)
  +               return RTE_BAD_PHYS_ADDR;
          return elem->ms->phys_addr + ((uintptr_t)addr - (uintptr_t)elem->ms->addr);
   }
  diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
  index 1c99852..2a401ca 100644
  --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
  +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
  @@ -973,7 +973,7 @@ rte_eal_hugepage_init(void)
                                          strerror(errno));
                          return -1;
                  }
  -               mcfg->memseg[0].phys_addr = (phys_addr_t)(uintptr_t)addr;
  +               mcfg->memseg[0].phys_addr = RTE_BAD_PHYS_ADDR;
                  mcfg->memseg[0].addr = addr;
                  mcfg->memseg[0].hugepage_sz = RTE_PGSIZE_4K;
                  mcfg->memseg[0].len = internal_config.memory;


Let me know what you are ok with this and I'll send a v2.

Thanks,
Olivier

Re: [dpdk-dev] [PATCH] eal: don't advertise a physical address when no hugepages

[Jan Blunck]

On Fri, Jun 23, 2017 at 10:11 AM, Olivier Matz <olivier.matz@6wind.com> wrote:
> Hi Jan,
>
> On Sat, 10 Jun 2017 10:31:22 +0200, Jan Blunck <jblunck@infradead.org> wrote:
>> On Fri, Jun 9, 2017 at 10:29 AM, Olivier Matz <olivier.matz@6wind.com> wrote:
>> > When populating a mempool with a virtual memory area, the mempool
>> > library expects to be able to get the physical address of each page.
>> >
>> > When started with --no-huge, the physical addresses may not be available
>> > because the pages are not locked in memory. It sometimes returns
>> > RTE_BAD_PHYS_ADDR, which makes the mempool_populate() function to fail.
>> >
>> > This was working before the commit cdc242f260e7 ("eal/linux: support
>> > running as unprivileged user"), because rte_mem_virt2phy() was returning
>> > 0 instead of RTE_BAD_PHYS_ADDR, which was seen as a valid physical
>> > address.
>> >
>> > Since --no-huge is a debug function that breaks the support of physical
>> > drivers, always set physical addresses to RTE_BAD_PHYS_ADDR in memzones
>> > or in rte_mem_virt2phy(), and ensure that mempool won't complain in that
>> > case.
>> >
>> > Fixes: cdc242f260e7 ("eal/linux: support running as unprivileged user")
>> >
>> > CC: stable@dpdk.org
>> > Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
>> > ---
>> >  lib/librte_eal/common/eal_common_memzone.c | 5 ++++-
>> >  lib/librte_eal/linuxapp/eal/eal_memory.c   | 7 +++++++
>> >  lib/librte_mempool/rte_mempool.c           | 2 +-
>> >  3 files changed, 12 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/lib/librte_eal/common/eal_common_memzone.c b/lib/librte_eal/common/eal_common_memzone.c
>> > index 3026e36b8..c465c8fc2 100644
>> > --- a/lib/librte_eal/common/eal_common_memzone.c
>> > +++ b/lib/librte_eal/common/eal_common_memzone.c
>> > @@ -251,7 +251,10 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
>> >
>> >         mcfg->memzone_cnt++;
>> >         snprintf(mz->name, sizeof(mz->name), "%s", name);
>> > -       mz->phys_addr = rte_malloc_virt2phy(mz_addr);
>> > +       if (rte_eal_has_hugepages())
>> > +               mz->phys_addr = rte_malloc_virt2phy(mz_addr);
>> > +       else
>> > +               mz->phys_addr = RTE_BAD_PHYS_ADDR;
>>
>> Since you set phys_addrs_available to false rte_malloc_virt2phy()
>> anyway returns RTE_BAD_PHYS_ADDR so I believe the conditional isn't
>> necessary here.
>>
>> Rest of the patch looks good to me.
>
> The variable phys_addrs_available only impacts rte_mem_virt2phy().
> Here, for memzones allocation, rte_malloc_virt2phy() is used, and
> it gets its physical address by retrieving it from the memseg structure.
>
> With the full patch, "dump_memzone" displays something like:
>   Zone 0: name:<rte_eth_dev_data>, phys:0xffffffffffffffff, len:0x30100, [...]
>   ...
>
> If I strip the memzone part, it displays:
>   Zone 0: name:<rte_eth_dev_data>, phys:0x7fe382c62640, len:0x30100, [...]
>   ...
>
> So I think we should either keep the patch as is, or change the memseg
> and malloc part like this (it's maybe better):
>
>   --- a/lib/librte_eal/common/rte_malloc.c
>   +++ b/lib/librte_eal/common/rte_malloc.c
>   @@ -254,5 +254,7 @@ rte_malloc_virt2phy(const void *addr)
>           const struct malloc_elem *elem = malloc_elem_from_data(addr);
>           if (elem == NULL)
>                   return 0;
>   +       if (elem->ms->phys_addr == RTE_BAD_PHYS_ADDR)
>   +               return RTE_BAD_PHYS_ADDR;
>           return elem->ms->phys_addr + ((uintptr_t)addr - (uintptr_t)elem->ms->addr);
>    }
>   diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
>   index 1c99852..2a401ca 100644
>   --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
>   +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
>   @@ -973,7 +973,7 @@ rte_eal_hugepage_init(void)
>                                           strerror(errno));
>                           return -1;
>                   }
>   -               mcfg->memseg[0].phys_addr = (phys_addr_t)(uintptr_t)addr;
>   +               mcfg->memseg[0].phys_addr = RTE_BAD_PHYS_ADDR;
>                   mcfg->memseg[0].addr = addr;
>                   mcfg->memseg[0].hugepage_sz = RTE_PGSIZE_4K;
>                   mcfg->memseg[0].len = internal_config.memory;
>
>
> Let me know what you are ok with this and I'll send a v2.
>

This approach looks better to me.

Thanks,
Jan

Re: [dpdk-dev] [PATCH] eal: don't advertise a physical address when no hugepages

[santosh]

Hi Olivier,

On Friday 23 June 2017 10:38 PM, Jan Blunck wrote:

> On Fri, Jun 23, 2017 at 10:11 AM, Olivier Matz <olivier.matz@6wind.com> wrote:
>> Hi Jan,
>>
>> On Sat, 10 Jun 2017 10:31:22 +0200, Jan Blunck <jblunck@infradead.org> wrote:
>>> On Fri, Jun 9, 2017 at 10:29 AM, Olivier Matz <olivier.matz@6wind.com> wrote:
>>>> When populating a mempool with a virtual memory area, the mempool
>>>> library expects to be able to get the physical address of each page.
>>>>
>>>> When started with --no-huge, the physical addresses may not be available
>>>> because the pages are not locked in memory. It sometimes returns
>>>> RTE_BAD_PHYS_ADDR, which makes the mempool_populate() function to fail.
>>>>
>>>> This was working before the commit cdc242f260e7 ("eal/linux: support
>>>> running as unprivileged user"), because rte_mem_virt2phy() was returning
>>>> 0 instead of RTE_BAD_PHYS_ADDR, which was seen as a valid physical
>>>> address.
>>>>
>>>> Since --no-huge is a debug function that breaks the support of physical
>>>> drivers, always set physical addresses to RTE_BAD_PHYS_ADDR in memzones
>>>> or in rte_mem_virt2phy(), and ensure that mempool won't complain in that
>>>> case.
>>>>
>>>> Fixes: cdc242f260e7 ("eal/linux: support running as unprivileged user")
>>>>
>>>> CC: stable@dpdk.org
>>>> Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
>>>> ---
>>>>  lib/librte_eal/common/eal_common_memzone.c | 5 ++++-
>>>>  lib/librte_eal/linuxapp/eal/eal_memory.c   | 7 +++++++
>>>>  lib/librte_mempool/rte_mempool.c           | 2 +-
>>>>  3 files changed, 12 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/lib/librte_eal/common/eal_common_memzone.c b/lib/librte_eal/common/eal_common_memzone.c
>>>> index 3026e36b8..c465c8fc2 100644
>>>> --- a/lib/librte_eal/common/eal_common_memzone.c
>>>> +++ b/lib/librte_eal/common/eal_common_memzone.c
>>>> @@ -251,7 +251,10 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
>>>>
>>>>         mcfg->memzone_cnt++;
>>>>         snprintf(mz->name, sizeof(mz->name), "%s", name);
>>>> -       mz->phys_addr = rte_malloc_virt2phy(mz_addr);
>>>> +       if (rte_eal_has_hugepages())
>>>> +               mz->phys_addr = rte_malloc_virt2phy(mz_addr);
>>>> +       else
>>>> +               mz->phys_addr = RTE_BAD_PHYS_ADDR;
>>> Since you set phys_addrs_available to false rte_malloc_virt2phy()
>>> anyway returns RTE_BAD_PHYS_ADDR so I believe the conditional isn't
>>> necessary here.
>>>
>>> Rest of the patch looks good to me.
>> The variable phys_addrs_available only impacts rte_mem_virt2phy().
>> Here, for memzones allocation, rte_malloc_virt2phy() is used, and
>> it gets its physical address by retrieving it from the memseg structure.
>>
>> With the full patch, "dump_memzone" displays something like:
>>   Zone 0: name:<rte_eth_dev_data>, phys:0xffffffffffffffff, len:0x30100, [...]
>>   ...
>>
>> If I strip the memzone part, it displays:
>>   Zone 0: name:<rte_eth_dev_data>, phys:0x7fe382c62640, len:0x30100, [...]
>>   ...
>>
>> So I think we should either keep the patch as is, or change the memseg
>> and malloc part like this (it's maybe better):
>>
>>   --- a/lib/librte_eal/common/rte_malloc.c
>>   +++ b/lib/librte_eal/common/rte_malloc.c
>>   @@ -254,5 +254,7 @@ rte_malloc_virt2phy(const void *addr)
>>           const struct malloc_elem *elem = malloc_elem_from_data(addr);
>>           if (elem == NULL)
>>                   return 0;
>>   +       if (elem->ms->phys_addr == RTE_BAD_PHYS_ADDR)
>>   +               return RTE_BAD_PHYS_ADDR;
>>           return elem->ms->phys_addr + ((uintptr_t)addr - (uintptr_t)elem->ms->addr);
>>    }
>>   diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
>>   index 1c99852..2a401ca 100644
>>   --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
>>   +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
>>   @@ -973,7 +973,7 @@ rte_eal_hugepage_init(void)
>>                                           strerror(errno));
>>                           return -1;
>>                   }
>>   -               mcfg->memseg[0].phys_addr = (phys_addr_t)(uintptr_t)addr;
>>   +               mcfg->memseg[0].phys_addr = RTE_BAD_PHYS_ADDR;
>>                   mcfg->memseg[0].addr = addr;
>>                   mcfg->memseg[0].hugepage_sz = RTE_PGSIZE_4K;
>>                   mcfg->memseg[0].len = internal_config.memory;
>>
>>
>> Let me know what you are ok with this and I'll send a v2.
>>
> This approach looks better to me.
>
> hanks,
> Jan

Approach LGTM, though small comment:
I think we also need to fix error return description for API rte_malloc_virt2phy.
It says 'NULL' on error. It should be 0 or RTE_BAD_PHYS_ADDR.

In fact, we should remove '0' as error return and keep RTE_BAD_PHYS_ADDR as the error value.
If so then change may look like:

if (elem == NULL || elem->ms->phys_addr == RTE_BAD_PHYS_ADDR)
    return RTE_BAD_PHYS_ADDR;

Provided that return value '0' considered as error value in current code.

Having said that, few drivers using rte_malloc_virt2phy without an error check.
I guess now they must check return value before using phys_addr_t.

Re: [dpdk-dev] [PATCH] eal: don't advertise a physical address when no hugepages

[Adrien Mazarguil]

On Fri, Jun 09, 2017 at 10:29:37AM +0200, Olivier Matz wrote:
> When populating a mempool with a virtual memory area, the mempool
> library expects to be able to get the physical address of each page.
> 
> When started with --no-huge, the physical addresses may not be available
> because the pages are not locked in memory. It sometimes returns
> RTE_BAD_PHYS_ADDR, which makes the mempool_populate() function to fail.
> 
> This was working before the commit cdc242f260e7 ("eal/linux: support
> running as unprivileged user"), because rte_mem_virt2phy() was returning
> 0 instead of RTE_BAD_PHYS_ADDR, which was seen as a valid physical
> address.
> 
> Since --no-huge is a debug function that breaks the support of physical
> drivers, always set physical addresses to RTE_BAD_PHYS_ADDR in memzones
> or in rte_mem_virt2phy(), and ensure that mempool won't complain in that
> case.
> 
> Fixes: cdc242f260e7 ("eal/linux: support running as unprivileged user")
> 
> CC: stable@dpdk.org
> Signed-off-by: Olivier Matz <olivier.matz@6wind.com>

Tested-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>

-- 
Adrien Mazarguil
6WIND

[dpdk-dev] [PATCH v2] eal: don't advertise a physical address when no hugepages

[Olivier Matz]

When populating a mempool with a virtual memory area, the mempool
library expects to be able to get the physical address of each page.

When started with --no-huge, the physical addresses may not be available
because the pages are not locked in memory. It sometimes returns
RTE_BAD_PHYS_ADDR, which makes the mempool_populate() function to fail.

This was working before the commit cdc242f260e7 ("eal/linux: support
running as unprivileged user"), because rte_mem_virt2phy() was returning
0 instead of RTE_BAD_PHYS_ADDR, which was seen as a valid physical
address.

Since --no-huge is a debug function that breaks the support of physical
drivers, always set physical addresses to RTE_BAD_PHYS_ADDR in memzones
or in rte_mem_virt2phy(), and ensure that mempool won't complain in that
case.

Fixes: cdc242f260e7 ("eal/linux: support running as unprivileged user")

CC: stable@dpdk.org
Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
---

v1 -> v2:
- initialize memory segments (rte_memseg) with RTE_BAD_PHYS_ADDR
  if started with --no-huge
- return RTE_BAD_PHYS_ADDR when rte_malloc_virt2phy() if the
  memory segment has no physical address
- drop change in eal_common_memzone.c: the physical address is
  now set to RTE_BAD_PHYS_ADDR as per return value of
  rte_malloc_virt2phy()


 lib/librte_eal/common/include/rte_malloc.h | 2 +-
 lib/librte_eal/common/rte_malloc.c         | 4 +++-
 lib/librte_eal/linuxapp/eal/eal_memory.c   | 9 ++++++++-
 lib/librte_mempool/rte_mempool.c           | 2 +-
 4 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/common/include/rte_malloc.h b/lib/librte_eal/common/include/rte_malloc.h
index 61aac322b..3d37f79b8 100644
--- a/lib/librte_eal/common/include/rte_malloc.h
+++ b/lib/librte_eal/common/include/rte_malloc.h
@@ -329,7 +329,7 @@ rte_malloc_set_limit(const char *type, size_t max);
  * @param addr
  *   Address obtained from a previous rte_malloc call
  * @return
- *   NULL on error
+ *   RTE_BAD_PHYS_ADDR on error
  *   otherwise return physical address of the buffer
  */
 phys_addr_t
diff --git a/lib/librte_eal/common/rte_malloc.c b/lib/librte_eal/common/rte_malloc.c
index f4a883529..5c0627bf4 100644
--- a/lib/librte_eal/common/rte_malloc.c
+++ b/lib/librte_eal/common/rte_malloc.c
@@ -253,6 +253,8 @@ rte_malloc_virt2phy(const void *addr)
 {
 	const struct malloc_elem *elem = malloc_elem_from_data(addr);
 	if (elem == NULL)
-		return 0;
+		return RTE_BAD_PHYS_ADDR;
+	if (elem->ms->phys_addr == RTE_BAD_PHYS_ADDR)
+		return RTE_BAD_PHYS_ADDR;
 	return elem->ms->phys_addr + ((uintptr_t)addr - (uintptr_t)elem->ms->addr);
 }
diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 647d89c58..040f24a43 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -112,6 +112,13 @@ test_phys_addrs_available(void)
 	if (rte_xen_dom0_supported())
 		return;
 
+	if (!rte_eal_has_hugepages()) {
+		RTE_LOG(ERR, EAL,
+			"Started without hugepages support, physical addresses not available\n");
+		phys_addrs_available = false;
+		return;
+	}
+
 	physaddr = rte_mem_virt2phy(&tmp);
 	if (physaddr == RTE_BAD_PHYS_ADDR) {
 		RTE_LOG(ERR, EAL,
@@ -1054,7 +1061,7 @@ rte_eal_hugepage_init(void)
 					strerror(errno));
 			return -1;
 		}
-		mcfg->memseg[0].phys_addr = (phys_addr_t)(uintptr_t)addr;
+		mcfg->memseg[0].phys_addr = RTE_BAD_PHYS_ADDR;
 		mcfg->memseg[0].addr = addr;
 		mcfg->memseg[0].hugepage_sz = RTE_PGSIZE_4K;
 		mcfg->memseg[0].len = internal_config.memory;
diff --git a/lib/librte_mempool/rte_mempool.c b/lib/librte_mempool/rte_mempool.c
index f65310f60..6fc3c9c7c 100644
--- a/lib/librte_mempool/rte_mempool.c
+++ b/lib/librte_mempool/rte_mempool.c
@@ -476,7 +476,7 @@ rte_mempool_populate_virt(struct rte_mempool *mp, char *addr,
 		/* required for xen_dom0 to get the machine address */
 		paddr = rte_mem_phy2mch(-1, paddr);
 
-		if (paddr == RTE_BAD_PHYS_ADDR) {
+		if (paddr == RTE_BAD_PHYS_ADDR && rte_eal_has_hugepages()) {
 			ret = -EINVAL;
 			goto fail;
 		}
-- 
2.11.0

Re: [dpdk-dev] [PATCH v2] eal: don't advertise a physical address when no hugepages

[Jan Blunck]

On Mon, Jul 3, 2017 at 12:04 PM, Olivier Matz <olivier.matz@6wind.com> wrote:
> When populating a mempool with a virtual memory area, the mempool
> library expects to be able to get the physical address of each page.
>
> When started with --no-huge, the physical addresses may not be available
> because the pages are not locked in memory. It sometimes returns
> RTE_BAD_PHYS_ADDR, which makes the mempool_populate() function to fail.
>
> This was working before the commit cdc242f260e7 ("eal/linux: support
> running as unprivileged user"), because rte_mem_virt2phy() was returning
> 0 instead of RTE_BAD_PHYS_ADDR, which was seen as a valid physical
> address.
>
> Since --no-huge is a debug function that breaks the support of physical
> drivers, always set physical addresses to RTE_BAD_PHYS_ADDR in memzones
> or in rte_mem_virt2phy(), and ensure that mempool won't complain in that
> case.
>
> Fixes: cdc242f260e7 ("eal/linux: support running as unprivileged user")
>
> CC: stable@dpdk.org
> Signed-off-by: Olivier Matz <olivier.matz@6wind.com>

Reviewed-by: Jan Blunck <jblunck@infradead.org>

> ---
>
> v1 -> v2:
> - initialize memory segments (rte_memseg) with RTE_BAD_PHYS_ADDR
>   if started with --no-huge
> - return RTE_BAD_PHYS_ADDR when rte_malloc_virt2phy() if the
>   memory segment has no physical address
> - drop change in eal_common_memzone.c: the physical address is
>   now set to RTE_BAD_PHYS_ADDR as per return value of
>   rte_malloc_virt2phy()
>
>
>  lib/librte_eal/common/include/rte_malloc.h | 2 +-
>  lib/librte_eal/common/rte_malloc.c         | 4 +++-
>  lib/librte_eal/linuxapp/eal/eal_memory.c   | 9 ++++++++-
>  lib/librte_mempool/rte_mempool.c           | 2 +-
>  4 files changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/lib/librte_eal/common/include/rte_malloc.h b/lib/librte_eal/common/include/rte_malloc.h
> index 61aac322b..3d37f79b8 100644
> --- a/lib/librte_eal/common/include/rte_malloc.h
> +++ b/lib/librte_eal/common/include/rte_malloc.h
> @@ -329,7 +329,7 @@ rte_malloc_set_limit(const char *type, size_t max);
>   * @param addr
>   *   Address obtained from a previous rte_malloc call
>   * @return
> - *   NULL on error
> + *   RTE_BAD_PHYS_ADDR on error
>   *   otherwise return physical address of the buffer
>   */
>  phys_addr_t
> diff --git a/lib/librte_eal/common/rte_malloc.c b/lib/librte_eal/common/rte_malloc.c
> index f4a883529..5c0627bf4 100644
> --- a/lib/librte_eal/common/rte_malloc.c
> +++ b/lib/librte_eal/common/rte_malloc.c
> @@ -253,6 +253,8 @@ rte_malloc_virt2phy(const void *addr)
>  {
>         const struct malloc_elem *elem = malloc_elem_from_data(addr);
>         if (elem == NULL)
> -               return 0;
> +               return RTE_BAD_PHYS_ADDR;
> +       if (elem->ms->phys_addr == RTE_BAD_PHYS_ADDR)
> +               return RTE_BAD_PHYS_ADDR;
>         return elem->ms->phys_addr + ((uintptr_t)addr - (uintptr_t)elem->ms->addr);
>  }
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index 647d89c58..040f24a43 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -112,6 +112,13 @@ test_phys_addrs_available(void)
>         if (rte_xen_dom0_supported())
>                 return;
>
> +       if (!rte_eal_has_hugepages()) {
> +               RTE_LOG(ERR, EAL,
> +                       "Started without hugepages support, physical addresses not available\n");
> +               phys_addrs_available = false;
> +               return;
> +       }
> +
>         physaddr = rte_mem_virt2phy(&tmp);
>         if (physaddr == RTE_BAD_PHYS_ADDR) {
>                 RTE_LOG(ERR, EAL,
> @@ -1054,7 +1061,7 @@ rte_eal_hugepage_init(void)
>                                         strerror(errno));
>                         return -1;
>                 }
> -               mcfg->memseg[0].phys_addr = (phys_addr_t)(uintptr_t)addr;
> +               mcfg->memseg[0].phys_addr = RTE_BAD_PHYS_ADDR;
>                 mcfg->memseg[0].addr = addr;
>                 mcfg->memseg[0].hugepage_sz = RTE_PGSIZE_4K;
>                 mcfg->memseg[0].len = internal_config.memory;
> diff --git a/lib/librte_mempool/rte_mempool.c b/lib/librte_mempool/rte_mempool.c
> index f65310f60..6fc3c9c7c 100644
> --- a/lib/librte_mempool/rte_mempool.c
> +++ b/lib/librte_mempool/rte_mempool.c
> @@ -476,7 +476,7 @@ rte_mempool_populate_virt(struct rte_mempool *mp, char *addr,
>                 /* required for xen_dom0 to get the machine address */
>                 paddr = rte_mem_phy2mch(-1, paddr);
>
> -               if (paddr == RTE_BAD_PHYS_ADDR) {
> +               if (paddr == RTE_BAD_PHYS_ADDR && rte_eal_has_hugepages()) {
>                         ret = -EINVAL;
>                         goto fail;
>                 }
> --
> 2.11.0
>

Re: [dpdk-dev] [dpdk-stable] [PATCH v2] eal: don't advertise a physical address when no hugepages

[Thomas Monjalon]

03/07/2017 12:17, Jan Blunck:
> On Mon, Jul 3, 2017 at 12:04 PM, Olivier Matz <olivier.matz@6wind.com> wrote:
> > When populating a mempool with a virtual memory area, the mempool
> > library expects to be able to get the physical address of each page.
> >
> > When started with --no-huge, the physical addresses may not be available
> > because the pages are not locked in memory. It sometimes returns
> > RTE_BAD_PHYS_ADDR, which makes the mempool_populate() function to fail.
> >
> > This was working before the commit cdc242f260e7 ("eal/linux: support
> > running as unprivileged user"), because rte_mem_virt2phy() was returning
> > 0 instead of RTE_BAD_PHYS_ADDR, which was seen as a valid physical
> > address.
> >
> > Since --no-huge is a debug function that breaks the support of physical
> > drivers, always set physical addresses to RTE_BAD_PHYS_ADDR in memzones
> > or in rte_mem_virt2phy(), and ensure that mempool won't complain in that
> > case.
> >
> > Fixes: cdc242f260e7 ("eal/linux: support running as unprivileged user")
> >
> > CC: stable@dpdk.org
> > Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
> 
> Reviewed-by: Jan Blunck <jblunck@infradead.org>

Applied, thanks

Re: [dpdk-dev] A (possible) problem with `--no-huge` option

[Ilya Matveychikov]

Hi Olivier,

The patch from you solves the problem for me.

Thank you.

> On Jun 9, 2017, at 12:27 PM, Olivier Matz <olivier.matz@6wind.com> wrote:
> 
> Hi Ilya,
> 
> On Sun, 14 May 2017 14:34:14 +0400, Ilya Matveychikov <matvejchikov@gmail.com> wrote:
>> Hi guys,
>> 
>> I have a problem while running DPDK with `--no-huge` option. It seems that the problem occurs since commit cdc242f260e766bd95a658b5e0686a62ec04f5b0 and that is the change that affects me:
>> 
>> +	if ((page & 0x7fffffffffffffULL) == 0)
>> +		return RTE_BAD_PHYS_ADDR;
>> +
>> 
>> What I did is to try to create memory pool using rte_pktmbuf_pool_create(). I dig into the issue and found that in my case “page" value is 0x0080000000000000 which means that the page is not present and “soft-dirty” (according to kernel’s documentation):
>> 
>>   * Bits 0-54  page frame number (PFN) if present
>>   * Bits 0-4   swap type if swapped
>>   * Bits 5-54  swap offset if swapped
>>   * Bit  55    pte is soft-dirty (see Documentation/vm/soft-dirty.txt)
>>   * Bit  56    page exclusively mapped (since 4.2)
>>   * Bits 57-60 zero
>>   * Bit  61    page is file-page or shared-anon (since 3.5)
>>   * Bit  62    page swapped
>>   * Bit  63    page present
>> 
>> So, before the change mentioned all “works” fine and such pages were not handled. But now the check causes rte_mempool_populate_default to fail with -EINVAL...
>> Can anyone familiar with the memory pool allocation helps with the issue?
>> 
>> Thanks in advice,
>> Ilya Matveychikov.
>> 
> 
> I can reproduce the issue:
> 
>  make config T=x86_64-native-linuxapp-gcc
>  make -j32 EXTRA_CFLAGS="-O0 -g"
>  mkdir -p /mnt/huge
>  mount -t hugetlbfs nodev /mnt/huge
>  echo 256 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
> 
>  # ok
>  ./build/app/testpmd -l 2,4 --log-level 8 --vdev=eth_null0 -- --no-numa --total-num-mbufs=4096 -i --port-topology=chained
> 
>  # fail
>  ./build/app/testpmd --no-huge -l 2,4 --log-level 8 --vdev=eth_null0 -- --no-numa --total-num-mbufs=4096 -i --port-topology=chained
> 
> 
> I confirm that rte_mem_virt2phy() returns RTE_BAD_PHYS_ADDR,
> which makes rte_mempool_populate_virt() to fail.
> 
> Reverting cdc242f260e7 ("eal/linux: support running as unprivileged user")
> fixes the problem. Actually, it makes rte_mem_virt2phy() return 0 instead
> of RTE_BAD_PHYS_ADDR, which is seen as a valid address.
> 
> I think querying the physical address when using --no-huge does not make
> sense because the memory is not locked, and could be swapped.
> 
> Another strange thing, when using --no-huge, the physical address returned
> when allocating a memzone is the virtual address.
> 
> I see several solutions to fix the issue:
> 
> 1/ Always set physical addresses to RTE_BAD_PHYS_ADDR when started
>   with --no-huge. We consider that the physical address is invalid
>   in that case and must not be used.
> 
>   This impacts rte_mem_virt2phy() and memzone_reserve*() functions.
> 
>   In rte_mempool_populate_virt(), don't expect a physical address
>   if the application is started with --no-huge.
> 
> 2/ Change rte_mem_virt2phy() to return the virtual address when we
>   ask for the physical address when started with --no-huge. This is
>   wrong, but consistent with what is done in memzones today.
> 
>   In rte_mem_virt2phy(), add at the beginning:
> 
>     if (!rte_eal_has_hugepages())
>         return (intptr_t)virtaddr;
> 
> 3/ lock pages in memory by reverting
>   729f17a932dd ("mem: revert page locking when not using hugepages")
> 
>   This would make the physical address available.
>   As explained in the commit log, this would also break the ability to
>   start dpdk with --no-huge for non-root users.
> 
> 
> I think 1/ is better. I'm sending a patch in reply to this mail.
> Ilya, please let me know if it fixes your issue.
> 
> Regards,
> Olivier