Re: [dpdk-dev] [PATCH v3 2/6] mem: use address hint for mapping hugepages

[Dariusz Stojaczyk <darek.stojaczyk@gmail.com>]

On Fri, Oct 5, 2018 at 2:47 PM Alejandro Lucero
<alejandro.lucero@netronome.com> wrote:
>
> Linux kernel uses a really high address as starting address for
> serving mmaps calls. If there exist addressing limitations and
> IOVA mode is VA, this starting address is likely too high for
> those devices. However, it is possible to use a lower address in
> the process virtual address space as with 64 bits there is a lot
> of available space.
>
> This patch adds an address hint as starting address for 64 bits
> systems and increments the hint for next invocations. If the mmap
> call does not use the hint address, repeat the mmap call using
> the hint address incremented by page size.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
>  lib/librte_eal/common/eal_common_memory.c | 34 ++++++++++++++++++++++++++++++-
>  1 file changed, 33 insertions(+), 1 deletion(-)
>
> diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/common/eal_common_memory.c
> index c482f0d..853c44c 100644
> --- a/lib/librte_eal/common/eal_common_memory.c
> +++ b/lib/librte_eal/common/eal_common_memory.c
> @@ -37,6 +37,23 @@
>  static void *next_baseaddr;
>  static uint64_t system_page_sz;
>
> +#ifdef RTE_ARCH_64
> +/*
> + * Linux kernel uses a really high address as starting address for serving
> + * mmaps calls. If there exists addressing limitations and IOVA mode is VA,
> + * this starting address is likely too high for those devices. However, it
> + * is possible to use a lower address in the process virtual address space
> + * as with 64 bits there is a lot of available space.
> + *
> + * Current known limitations are 39 or 40 bits. Setting the starting address
> + * at 4GB implies there are 508GB or 1020GB for mapping the available
> + * hugepages. This is likely enough for most systems, although a device with
> + * addressing limitations should call rte_eal_check_dma_mask for ensuring all
> + * memory is within supported range.
> + */
> +static uint64_t baseaddr = 0x100000000;
> +#endif

This breaks running with ASAN unless a custom --base-virtaddr option
is specified. The default base-virtaddr introduced by this patch falls
into an area that's already reserved by ASAN.

See here: https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm
The only available address space starts at 0x10007fff8000, which
unfortunately doesn't fit in 39 bits.

Right now the very first eal_get_virtual_area() in EAL initialization
is used with 4KB pagesize, meaning that DPDK will try to mmap at each
4KB-aligned offset all the way from 0x100000000 to 0x10007fff8000,
which takes quite a long, long time.

I'm not sure about the solution to this problem, but I verify that
starting DPDK 18.11-rc1 with `--base-virtaddr 0x200000000000` works
just fine under ASAN.

D.

>
> <snip>