Re: [dpdk-dev] [PATCH 00/41] Memory Hotplug for DPDK

[gowrishankar muthukrishnan <gowrishankar.m@linux.vnet.ibm.com>]

On Saturday 03 March 2018 07:15 PM, Anatoly Burakov wrote:
> This patchset introduces dynamic memory allocation for DPDK (aka memory
> hotplug). Based upon RFC submitted in December [1].
Hi Anatoly,

I am able to bring up pmd with these patches in powerpc. I am continuing 
to validate more
on memory limits that this patch set has (eg pre-allocating anonymous 
mapping for largest
mem possible as per default mem and memseg values). I'll keep posting my 
observations further.

Thanks for the patches,
Gowrishankar

> Dependencies (to be applied in specified order):
> - IPC bugfixes patchset [2]
> - IPC improvements patchset [3]
> - IPC asynchronous request API patch [4]
> - Function to return number of sockets [5]
>
> Deprecation notices relevant to this patchset:
> - General outline of memory hotplug changes [6]
> - EAL NUMA node count changes [7]
>
> The vast majority of changes are in the EAL and malloc, the external API
> disruption is minimal: a new set of API's are added for contiguous memory
> allocation for rte_memzone, and a few API additions in rte_memory due to
> switch to memseg_lists as opposed to memsegs. Every other API change is
> internal to EAL, and all of the memory allocation/freeing is handled
> through rte_malloc, with no externally visible API changes.
>
> Quick outline of all changes done as part of this patchset:
>
>   * Malloc heap adjusted to handle holes in address space
>   * Single memseg list replaced by multiple memseg lists
>   * VA space for hugepages is preallocated in advance
>   * Added alloc/free for pages happening as needed on rte_malloc/rte_free
>   * Added contiguous memory allocation API's for rte_memzone
>   * Integrated Pawel Wodkowski's patch for registering/unregistering memory
>     with VFIO [8]
>   * Callbacks for registering memory allocations
>   * Multiprocess support done via DPDK IPC introduced in 18.02
>
> The biggest difference is a "memseg" now represents a single page (as opposed to
> being a big contiguous block of pages). As a consequence, both memzones and
> malloc elements are no longer guaranteed to be physically contiguous, unless
> the user asks for it at reserve time. To preserve whatever functionality that
> was dependent on previous behavior, a legacy memory option is also provided,
> however it is expected (or perhaps vainly hoped) to be temporary solution.
>
> Why multiple memseg lists instead of one? Since memseg is a single page now,
> the list of memsegs will get quite big, and we need to locate pages somehow
> when we allocate and free them. We could of course just walk the list and
> allocate one contiguous chunk of VA space for memsegs, but this
> implementation uses separate lists instead in order to speed up many
> operations with memseg lists.
>
> For v1, the following limitations are present:
> - FreeBSD does not even compile, let alone run
> - No 32-bit support
> - There are some minor quality-of-life improvements planned that aren't
>    ready yet and will be part of v2
> - VFIO support is only smoke-tested (but is expected to work), VFIO support
>    with secondary processes is not tested; work is ongoing to validate VFIO
>    for all use cases
> - Dynamic mapping/unmapping memory with VFIO is not supported in sPAPR
>    IOMMU mode - help from sPAPR maintainers requested
>
> Nevertheless, this patchset should be testable under 64-bit Linux, and
> should work for all use cases bar those mentioned above.
>
> [1] http://dpdk.org/dev/patchwork/bundle/aburakov/Memory_RFC/
> [2] http://dpdk.org/dev/patchwork/bundle/aburakov/IPC_Fixes/
> [3] http://dpdk.org/dev/patchwork/bundle/aburakov/IPC_Improvements/
> [4] http://dpdk.org/dev/patchwork/bundle/aburakov/IPC_Async_Request/
> [5] http://dpdk.org/dev/patchwork/bundle/aburakov/Num_Sockets/
> [6] http://dpdk.org/dev/patchwork/patch/34002/
> [7] http://dpdk.org/dev/patchwork/patch/33853/
> [8] http://dpdk.org/dev/patchwork/patch/24484/
>
> Anatoly Burakov (41):
>    eal: move get_virtual_area out of linuxapp eal_memory.c
>    eal: move all locking to heap
>    eal: make malloc heap a doubly-linked list
>    eal: add function to dump malloc heap contents
>    test: add command to dump malloc heap contents
>    eal: make malloc_elem_join_adjacent_free public
>    eal: make malloc free list remove public
>    eal: make malloc free return resulting malloc element
>    eal: add rte_fbarray
>    eal: add "single file segments" command-line option
>    eal: add "legacy memory" option
>    eal: read hugepage counts from node-specific sysfs path
>    eal: replace memseg with memseg lists
>    eal: add support for mapping hugepages at runtime
>    eal: add support for unmapping pages at runtime
>    eal: make use of memory hotplug for init
>    eal: enable memory hotplug support in rte_malloc
>    test: fix malloc autotest to support memory hotplug
>    eal: add API to check if memory is contiguous
>    eal: add backend support for contiguous allocation
>    eal: enable reserving physically contiguous memzones
>    eal: replace memzone array with fbarray
>    mempool: add support for the new allocation methods
>    vfio: allow to map other memory regions
>    eal: map/unmap memory with VFIO when alloc/free pages
>    eal: prepare memseg lists for multiprocess sync
>    eal: add multiprocess init with memory hotplug
>    eal: add support for multiprocess memory hotplug
>    eal: add support for callbacks on memory hotplug
>    eal: enable callbacks on malloc/free and mp sync
>    ethdev: use contiguous allocation for DMA memory
>    crypto/qat: use contiguous allocation for DMA memory
>    net/avf: use contiguous allocation for DMA memory
>    net/bnx2x: use contiguous allocation for DMA memory
>    net/cxgbe: use contiguous allocation for DMA memory
>    net/ena: use contiguous allocation for DMA memory
>    net/enic: use contiguous allocation for DMA memory
>    net/i40e: use contiguous allocation for DMA memory
>    net/qede: use contiguous allocation for DMA memory
>    net/virtio: use contiguous allocation for DMA memory
>    net/vmxnet3: use contiguous allocation for DMA memory
>
>   config/common_base                                |   15 +-
>   drivers/bus/pci/linux/pci.c                       |   29 +-
>   drivers/crypto/qat/qat_qp.c                       |    4 +-
>   drivers/net/avf/avf_ethdev.c                      |    2 +-
>   drivers/net/bnx2x/bnx2x.c                         |    2 +-
>   drivers/net/bnx2x/bnx2x_rxtx.c                    |    3 +-
>   drivers/net/cxgbe/sge.c                           |    3 +-
>   drivers/net/ena/base/ena_plat_dpdk.h              |    7 +-
>   drivers/net/ena/ena_ethdev.c                      |   10 +-
>   drivers/net/enic/enic_main.c                      |    4 +-
>   drivers/net/i40e/i40e_ethdev.c                    |    2 +-
>   drivers/net/i40e/i40e_rxtx.c                      |    2 +-
>   drivers/net/qede/base/bcm_osal.c                  |    5 +-
>   drivers/net/virtio/virtio_ethdev.c                |    8 +-
>   drivers/net/virtio/virtio_user/vhost_kernel.c     |  108 ++-
>   drivers/net/vmxnet3/vmxnet3_ethdev.c              |    7 +-
>   lib/librte_eal/bsdapp/eal/Makefile                |    4 +
>   lib/librte_eal/bsdapp/eal/eal.c                   |   25 +
>   lib/librte_eal/bsdapp/eal/eal_hugepage_info.c     |    7 +
>   lib/librte_eal/bsdapp/eal/eal_memalloc.c          |   33 +
>   lib/librte_eal/bsdapp/eal/meson.build             |    1 +
>   lib/librte_eal/common/Makefile                    |    2 +-
>   lib/librte_eal/common/eal_common_fbarray.c        |  859 +++++++++++++++++
>   lib/librte_eal/common/eal_common_memalloc.c       |  181 ++++
>   lib/librte_eal/common/eal_common_memory.c         |  512 +++++++++-
>   lib/librte_eal/common/eal_common_memzone.c        |  275 ++++--
>   lib/librte_eal/common/eal_common_options.c        |    8 +
>   lib/librte_eal/common/eal_filesystem.h            |   13 +
>   lib/librte_eal/common/eal_hugepages.h             |    7 +
>   lib/librte_eal/common/eal_internal_cfg.h          |   10 +-
>   lib/librte_eal/common/eal_memalloc.h              |   41 +
>   lib/librte_eal/common/eal_options.h               |    4 +
>   lib/librte_eal/common/eal_private.h               |   33 +
>   lib/librte_eal/common/include/rte_eal_memconfig.h |   29 +-
>   lib/librte_eal/common/include/rte_fbarray.h       |  352 +++++++
>   lib/librte_eal/common/include/rte_malloc.h        |    9 +
>   lib/librte_eal/common/include/rte_malloc_heap.h   |    6 +
>   lib/librte_eal/common/include/rte_memory.h        |   79 +-
>   lib/librte_eal/common/include/rte_memzone.h       |  155 ++-
>   lib/librte_eal/common/include/rte_vfio.h          |   39 +
>   lib/librte_eal/common/malloc_elem.c               |  436 +++++++--
>   lib/librte_eal/common/malloc_elem.h               |   41 +-
>   lib/librte_eal/common/malloc_heap.c               |  694 +++++++++++++-
>   lib/librte_eal/common/malloc_heap.h               |   15 +-
>   lib/librte_eal/common/malloc_mp.c                 |  723 ++++++++++++++
>   lib/librte_eal/common/malloc_mp.h                 |   86 ++
>   lib/librte_eal/common/meson.build                 |    4 +
>   lib/librte_eal/common/rte_malloc.c                |   75 +-
>   lib/librte_eal/linuxapp/eal/Makefile              |    5 +
>   lib/librte_eal/linuxapp/eal/eal.c                 |  102 +-
>   lib/librte_eal/linuxapp/eal/eal_hugepage_info.c   |  155 ++-
>   lib/librte_eal/linuxapp/eal/eal_memalloc.c        | 1049 +++++++++++++++++++++
>   lib/librte_eal/linuxapp/eal/eal_memory.c          |  516 ++++++----
>   lib/librte_eal/linuxapp/eal/eal_vfio.c            |  318 +++++--
>   lib/librte_eal/linuxapp/eal/eal_vfio.h            |   11 +
>   lib/librte_eal/linuxapp/eal/meson.build           |    1 +
>   lib/librte_eal/rte_eal_version.map                |   23 +-
>   lib/librte_ether/rte_ethdev.c                     |    3 +-
>   lib/librte_mempool/rte_mempool.c                  |   87 +-
>   test/test/commands.c                              |    3 +
>   test/test/test_malloc.c                           |   71 +-
>   test/test/test_memory.c                           |   43 +-
>   test/test/test_memzone.c                          |   26 +-
>   63 files changed, 6631 insertions(+), 751 deletions(-)
>   create mode 100644 lib/librte_eal/bsdapp/eal/eal_memalloc.c
>   create mode 100644 lib/librte_eal/common/eal_common_fbarray.c
>   create mode 100644 lib/librte_eal/common/eal_common_memalloc.c
>   create mode 100644 lib/librte_eal/common/eal_memalloc.h
>   create mode 100644 lib/librte_eal/common/include/rte_fbarray.h
>   create mode 100644 lib/librte_eal/common/malloc_mp.c
>   create mode 100644 lib/librte_eal/common/malloc_mp.h
>   create mode 100644 lib/librte_eal/linuxapp/eal/eal_memalloc.c
>