Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in DPDK

[Shahaf Shuler <shahafs@mellanox.com>]

Hi Anatoly,

First thanks for the patchset, it is a great enhancement. 

See question below. 

Tuesday, September 4, 2018 4:12 PM, Anatoly Burakov:
> Subject: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in
> DPDK
> 
> This is a proposal to enable using externally allocated memory in DPDK.
> 
> In a nutshell, here is what is being done here:
> 
> - Index internal malloc heaps by NUMA node index, rather than NUMA
>   node itself (external heaps will have ID's in order of creation)
> - Add identifier string to malloc heap, to uniquely identify it
>   - Each new heap will receive a unique socket ID that will be used by
>     allocator to decide from which heap (internal or external) to
>     allocate requested amount of memory
> - Allow creating named heaps and add/remove memory to/from those
> heaps
> - Allocate memseg lists at runtime, to keep track of IOVA addresses
>   of externally allocated memory
>   - If IOVA addresses aren't provided, use RTE_BAD_IOVA
> - Allow malloc and memzones to allocate from external heaps
> - Allow other data structures to allocate from externall heaps
> 
> The responsibility to ensure memory is accessible before using it is on the
> shoulders of the user - there is no checking done with regards to validity of
> the memory (nor could there be...).

That makes sense. However who should be in-charge of mapping this memory for dma access?
The user or internally be the PMD when encounter the first packet or while traversing the existing mempools? 

> 
> The general approach is to create heap and add memory into it. For any other
> process wishing to use the same memory, said memory must first be
> attached (otherwise some things will not work).
> 
> A design decision was made to make multiprocess synchronization a manual
> process. Due to underlying issues with attaching to fbarrays in secondary
> processes, this design was deemed to be better because we don't want to
> fail to create external heap in the primary because something in the
> secondary has failed when in fact we may not eve have wanted this memory
> to be accessible in the secondary in the first place.
> 
> Using external memory in multiprocess is *hard*, because not only memory
> space needs to be preallocated, but it also needs to be attached in each
> process to allow other processes to access the page table. The attach API call
> may or may not succeed, depending on memory layout, for reasons similar to
> other multiprocess failures. This is treated as a "known issue" for this release.
> 
> RFC -> v1 changes:
> - Removed the "named heaps" API, allocate using fake socket ID instead
> - Added multiprocess support
> - Everything is now thread-safe
> - Numerous bugfixes and API improvements
> 
> Anatoly Burakov (16):
>   mem: add length to memseg list
>   mem: allow memseg lists to be marked as external
>   malloc: index heaps using heap ID rather than NUMA node
>   mem: do not check for invalid socket ID
>   flow_classify: do not check for invalid socket ID
>   pipeline: do not check for invalid socket ID
>   sched: do not check for invalid socket ID
>   malloc: add name to malloc heaps
>   malloc: add function to query socket ID of named heap
>   malloc: allow creating malloc heaps
>   malloc: allow destroying heaps
>   malloc: allow adding memory to named heaps
>   malloc: allow removing memory from named heaps
>   malloc: allow attaching to external memory chunks
>   malloc: allow detaching from external memory
>   test: add unit tests for external memory support
> 
>  config/common_base                            |   1 +
>  config/rte_config.h                           |   1 +
>  drivers/bus/fslmc/fslmc_vfio.c                |   7 +-
>  drivers/bus/pci/linux/pci.c                   |   2 +-
>  drivers/net/mlx4/mlx4_mr.c                    |   3 +
>  drivers/net/mlx5/mlx5.c                       |   5 +-
>  drivers/net/mlx5/mlx5_mr.c                    |   3 +
>  drivers/net/virtio/virtio_user/vhost_kernel.c |   5 +-
>  lib/librte_eal/bsdapp/eal/eal.c               |   3 +
>  lib/librte_eal/bsdapp/eal/eal_memory.c        |   9 +-
>  lib/librte_eal/common/eal_common_memory.c     |   9 +-
>  lib/librte_eal/common/eal_common_memzone.c    |   8 +-
>  .../common/include/rte_eal_memconfig.h        |   6 +-
>  lib/librte_eal/common/include/rte_malloc.h    | 181 +++++++++
>  .../common/include/rte_malloc_heap.h          |   3 +
>  lib/librte_eal/common/include/rte_memory.h    |   9 +
>  lib/librte_eal/common/malloc_heap.c           | 287 +++++++++++--
>  lib/librte_eal/common/malloc_heap.h           |  17 +
>  lib/librte_eal/common/rte_malloc.c            | 383 ++++++++++++++++-
>  lib/librte_eal/linuxapp/eal/eal.c             |   3 +
>  lib/librte_eal/linuxapp/eal/eal_memalloc.c    |  12 +-
>  lib/librte_eal/linuxapp/eal/eal_memory.c      |   4 +-
>  lib/librte_eal/linuxapp/eal/eal_vfio.c        |  17 +-
>  lib/librte_eal/rte_eal_version.map            |   7 +
>  lib/librte_flow_classify/rte_flow_classify.c  |   3 +-
>  lib/librte_mempool/rte_mempool.c              |  31 +-
>  lib/librte_pipeline/rte_pipeline.c            |   3 +-
>  lib/librte_sched/rte_sched.c                  |   2 +-
>  test/test/Makefile                            |   1 +
>  test/test/autotest_data.py                    |  14 +-
>  test/test/meson.build                         |   1 +
>  test/test/test_external_mem.c                 | 384 ++++++++++++++++++
>  test/test/test_malloc.c                       |   3 +
>  test/test/test_memzone.c                      |   3 +
>  34 files changed, 1346 insertions(+), 84 deletions(-)  create mode 100644
> test/test/test_external_mem.c
> 
> --
> 2.17.1