DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [RFC PATCH 0/2] dynamic memzones
@ 2015-05-08 16:37 Sergio Gonzalez Monroy
  2015-05-08 16:37 ` [dpdk-dev] [RFC PATCH 1/2] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
                   ` (3 more replies)
  0 siblings, 4 replies; 108+ messages in thread
From: Sergio Gonzalez Monroy @ 2015-05-08 16:37 UTC (permalink / raw)
  To: dev

Please NOTE that this series is meant to illustrate an idea/approach and start
discussion on the topic.

Current implemetation allows reserving/creating memzones but not the opposite
(unreserve/delete). This affects mempools and other memzone based objects.

>From my point of view, implementing unreserve functionality for memzones would
look like malloc over memsegs.
Thus, this approach moves malloc inside eal (which in turn removes a circular
dependency), where malloc heaps are composed of memsegs.
We keep both malloc and memzone APIs as they are, but memzones allocate its
memory by calling malloc_heap_alloc (there would be some ABI changes, see below).
Some extra functionality is required in malloc to allow for boundary constrained
memory requests.
In summary, currently malloc is based on memzones, and with this approach
memzones are based on malloc.

An alternative would be to move malloc internals (malloc_heap, malloc_elem)
to the eal, but keeping the malloc library as is, where malloc is based on
memzones. This way we could avoid ABI changes while keeping the existing
circular dependency between malloc and eal.

TODOs:
 - Implement memzone_unreserve, simply call rte_malloc_free.
 - Implement mempool_delete, simply call rte_memzone_unreserve.
 - Init heaps with all available memsegs at once.
 - Review symbols in version map.

ABI changes:
 - Removed support for rte_memzone_reserve_xxxx with len=0 (not needed?).
 - Removed librte_malloc as single library (linker script as work around?).

IDEAS FOR FUTURE WORK:
 - More control over requested memory, ie. shared/private, phys_contig, etc.
   One of the goals would be trying to reduce the need of physically contiguous
   memory when not required.
 - Attach/unattach hugepages at runtime (faster VM migration).
 - Improve malloc algorithm? ie. jemalloc (or any other).


Any comments/toughts and/or different approaches are welcome.


Sergio Gonzalez Monroy (2):
  eal: move librte_malloc to eal/common
  eal: memzone allocated by malloc

 config/common_bsdapp                            |   9 +-
 config/common_linuxapp                          |   9 +-
 lib/Makefile                                    |   1 -
 lib/librte_acl/Makefile                         |   2 +-
 lib/librte_eal/bsdapp/eal/Makefile              |   4 +-
 lib/librte_eal/bsdapp/eal/rte_eal_version.map   |  18 ++
 lib/librte_eal/common/Makefile                  |   1 +
 lib/librte_eal/common/eal_common_memzone.c      | 233 ++--------------
 lib/librte_eal/common/include/rte_malloc.h      | 342 ++++++++++++++++++++++++
 lib/librte_eal/common/include/rte_malloc_heap.h |   4 +-
 lib/librte_eal/common/include/rte_memory.h      |   1 +
 lib/librte_eal/common/malloc_elem.c             | 342 ++++++++++++++++++++++++
 lib/librte_eal/common/malloc_elem.h             | 192 +++++++++++++
 lib/librte_eal/common/malloc_heap.c             | 287 ++++++++++++++++++++
 lib/librte_eal/common/malloc_heap.h             |  70 +++++
 lib/librte_eal/common/rte_malloc.c              | 259 ++++++++++++++++++
 lib/librte_eal/linuxapp/eal/Makefile            |   4 +-
 lib/librte_eal/linuxapp/eal/rte_eal_version.map |  18 ++
 lib/librte_hash/Makefile                        |   2 +-
 lib/librte_lpm/Makefile                         |   2 +-
 lib/librte_malloc/Makefile                      |  52 ----
 lib/librte_malloc/malloc_elem.c                 | 320 ----------------------
 lib/librte_malloc/malloc_elem.h                 | 190 -------------
 lib/librte_malloc/malloc_heap.c                 | 209 ---------------
 lib/librte_malloc/malloc_heap.h                 |  70 -----
 lib/librte_malloc/rte_malloc.c                  | 260 ------------------
 lib/librte_malloc/rte_malloc.h                  | 342 ------------------------
 lib/librte_malloc/rte_malloc_version.map        |  19 --
 lib/librte_mempool/Makefile                     |   2 -
 lib/librte_pmd_af_packet/Makefile               |   1 -
 lib/librte_pmd_bond/Makefile                    |   1 -
 lib/librte_pmd_e1000/Makefile                   |   2 +-
 lib/librte_pmd_enic/Makefile                    |   2 +-
 lib/librte_pmd_fm10k/Makefile                   |   2 +-
 lib/librte_pmd_i40e/Makefile                    |   2 +-
 lib/librte_pmd_ixgbe/Makefile                   |   2 +-
 lib/librte_pmd_mlx4/Makefile                    |   1 -
 lib/librte_pmd_null/Makefile                    |   1 -
 lib/librte_pmd_pcap/Makefile                    |   1 -
 lib/librte_pmd_virtio/Makefile                  |   2 +-
 lib/librte_pmd_vmxnet3/Makefile                 |   2 +-
 lib/librte_pmd_xenvirt/Makefile                 |   2 +-
 lib/librte_port/Makefile                        |   1 -
 lib/librte_ring/Makefile                        |   3 +-
 lib/librte_table/Makefile                       |   1 -
 45 files changed, 1571 insertions(+), 1719 deletions(-)
 create mode 100644 lib/librte_eal/common/include/rte_malloc.h
 create mode 100644 lib/librte_eal/common/malloc_elem.c
 create mode 100644 lib/librte_eal/common/malloc_elem.h
 create mode 100644 lib/librte_eal/common/malloc_heap.c
 create mode 100644 lib/librte_eal/common/malloc_heap.h
 create mode 100644 lib/librte_eal/common/rte_malloc.c
 delete mode 100644 lib/librte_malloc/Makefile
 delete mode 100644 lib/librte_malloc/malloc_elem.c
 delete mode 100644 lib/librte_malloc/malloc_elem.h
 delete mode 100644 lib/librte_malloc/malloc_heap.c
 delete mode 100644 lib/librte_malloc/malloc_heap.h
 delete mode 100644 lib/librte_malloc/rte_malloc.c
 delete mode 100644 lib/librte_malloc/rte_malloc.h
 delete mode 100644 lib/librte_malloc/rte_malloc_version.map

-- 
1.9.3

^ permalink raw reply	[flat|nested] 108+ messages in thread

end of thread, other threads:[~2015-10-14 11:05 UTC | newest]

Thread overview: 108+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-05-08 16:37 [dpdk-dev] [RFC PATCH 0/2] dynamic memzones Sergio Gonzalez Monroy
2015-05-08 16:37 ` [dpdk-dev] [RFC PATCH 1/2] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-05-08 16:37 ` [dpdk-dev] [RFC PATCH 2/2] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-05-12 16:30 ` [dpdk-dev] [RFC PATCH 0/2] dynamic memzones Olivier MATZ
2015-06-06 10:32 ` [dpdk-dev] [PATCH v2 0/7] dynamic memzone Sergio Gonzalez Monroy
2015-06-06 10:32   ` [dpdk-dev] [PATCH v2 1/7] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-06-06 10:32   ` [dpdk-dev] [PATCH v2 2/7] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-06-06 10:32   ` [dpdk-dev] [PATCH v2 3/7] app/test: update malloc/memzone unit tests Sergio Gonzalez Monroy
2015-06-06 10:32   ` [dpdk-dev] [PATCH v2 4/7] config: remove CONFIG_RTE_MALLOC_MEMZONE_SIZE Sergio Gonzalez Monroy
2015-06-06 10:32   ` [dpdk-dev] [PATCH v2 5/7] eal: remove setup of free_memseg in ivshmem Sergio Gonzalez Monroy
2015-06-06 10:32   ` [dpdk-dev] [PATCH v2 6/7] eal: new rte_memzone_free Sergio Gonzalez Monroy
2015-06-06 10:32   ` [dpdk-dev] [PATCH v2 7/7] app/test: update unit test with rte_memzone_free Sergio Gonzalez Monroy
2015-06-19 17:21   ` [dpdk-dev] [PATCH v3 0/9] Dynamic memzone Sergio Gonzalez Monroy
2015-06-19 17:21     ` [dpdk-dev] [PATCH v3 1/9] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-06-19 17:21     ` [dpdk-dev] [PATCH v3 2/9] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-06-19 17:21     ` [dpdk-dev] [PATCH v3 3/9] app/test: update malloc/memzone unit tests Sergio Gonzalez Monroy
2015-06-19 17:21     ` [dpdk-dev] [PATCH v3 4/9] config: remove CONFIG_RTE_MALLOC_MEMZONE_SIZE Sergio Gonzalez Monroy
2015-06-19 17:21     ` [dpdk-dev] [PATCH v3 5/9] eal: remove free_memseg and references to it Sergio Gonzalez Monroy
2015-06-19 17:21     ` [dpdk-dev] [PATCH v3 6/9] eal: new rte_memzone_free Sergio Gonzalez Monroy
2015-06-19 17:21     ` [dpdk-dev] [PATCH v3 7/9] app/test: update unit test with rte_memzone_free Sergio Gonzalez Monroy
2015-06-19 17:21     ` [dpdk-dev] [PATCH v3 8/9] doc: announce ABI change of librte_malloc Sergio Gonzalez Monroy
2015-06-19 17:21     ` [dpdk-dev] [PATCH v3 9/9] doc: update malloc documentation Sergio Gonzalez Monroy
2015-06-25 14:05   ` [dpdk-dev] [PATCH v4 0/9] Dynamic memzone Sergio Gonzalez Monroy
2015-06-25 14:05     ` [dpdk-dev] [PATCH v4 1/9] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-06-25 14:05     ` [dpdk-dev] [PATCH v4 2/9] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-06-25 14:05     ` [dpdk-dev] [PATCH v4 3/9] app/test: update malloc/memzone unit tests Sergio Gonzalez Monroy
2015-06-25 14:05     ` [dpdk-dev] [PATCH v4 4/9] config: remove CONFIG_RTE_MALLOC_MEMZONE_SIZE Sergio Gonzalez Monroy
2015-06-25 14:05     ` [dpdk-dev] [PATCH v4 5/9] eal: remove free_memseg and references to it Sergio Gonzalez Monroy
2015-06-25 14:05     ` [dpdk-dev] [PATCH v4 6/9] eal: new rte_memzone_free Sergio Gonzalez Monroy
2015-06-25 14:05     ` [dpdk-dev] [PATCH v4 7/9] app/test: update unit test with rte_memzone_free Sergio Gonzalez Monroy
2015-06-25 14:05     ` [dpdk-dev] [PATCH v4 8/9] doc: announce ABI change of librte_malloc Sergio Gonzalez Monroy
2015-06-25 14:05     ` [dpdk-dev] [PATCH v4 9/9] doc: update malloc documentation Sergio Gonzalez Monroy
2015-06-26 11:32   ` [dpdk-dev] [PATCH v5 0/9] Dynamic memzones Sergio Gonzalez Monroy
2015-06-26 11:32     ` [dpdk-dev] [PATCH v5 1/9] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-06-26 11:32     ` [dpdk-dev] [PATCH v5 2/9] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-06-26 11:32     ` [dpdk-dev] [PATCH v5 3/9] app/test: update malloc/memzone unit tests Sergio Gonzalez Monroy
2015-06-26 11:32     ` [dpdk-dev] [PATCH v5 4/9] config: remove CONFIG_RTE_MALLOC_MEMZONE_SIZE Sergio Gonzalez Monroy
2015-06-26 11:32     ` [dpdk-dev] [PATCH v5 5/9] eal: remove free_memseg and references to it Sergio Gonzalez Monroy
2015-06-26 11:32     ` [dpdk-dev] [PATCH v5 6/9] eal: new rte_memzone_free Sergio Gonzalez Monroy
2015-06-26 11:32     ` [dpdk-dev] [PATCH v5 7/9] app/test: update unit test with rte_memzone_free Sergio Gonzalez Monroy
2015-06-26 11:32     ` [dpdk-dev] [PATCH v5 8/9] doc: announce ABI change of librte_malloc Sergio Gonzalez Monroy
2015-06-26 11:32     ` [dpdk-dev] [PATCH v5 9/9] doc: update malloc documentation Sergio Gonzalez Monroy
2015-06-26 15:29   ` [dpdk-dev] [PATCH v6 0/9] Dynamic memzones Sergio Gonzalez Monroy
2015-06-26 15:29     ` [dpdk-dev] [PATCH v6 1/9] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-07-02 12:14       ` Thomas Monjalon
2015-07-03  8:16         ` Gonzalez Monroy, Sergio
2015-07-03  9:02           ` Thomas Monjalon
2015-06-26 15:29     ` [dpdk-dev] [PATCH v6 2/9] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-06-26 15:29     ` [dpdk-dev] [PATCH v6 3/9] app/test: update malloc/memzone unit tests Sergio Gonzalez Monroy
2015-06-26 15:29     ` [dpdk-dev] [PATCH v6 4/9] config: remove CONFIG_RTE_MALLOC_MEMZONE_SIZE Sergio Gonzalez Monroy
2015-06-26 15:29     ` [dpdk-dev] [PATCH v6 5/9] eal: remove free_memseg and references to it Sergio Gonzalez Monroy
2015-06-26 15:29     ` [dpdk-dev] [PATCH v6 6/9] eal: new rte_memzone_free Sergio Gonzalez Monroy
2015-06-26 15:29     ` [dpdk-dev] [PATCH v6 7/9] app/test: rte_memzone_free unit test Sergio Gonzalez Monroy
2015-06-26 15:29     ` [dpdk-dev] [PATCH v6 8/9] doc: announce ABI change of librte_malloc Sergio Gonzalez Monroy
2015-06-26 15:29     ` [dpdk-dev] [PATCH v6 9/9] doc: update malloc documentation Sergio Gonzalez Monroy
2015-06-26 16:13     ` [dpdk-dev] [PATCH v6 0/9] Dynamic memzones Ananyev, Konstantin
2015-07-03  9:55     ` [dpdk-dev] [PATCH v7 " Sergio Gonzalez Monroy
2015-07-03  9:55       ` [dpdk-dev] [PATCH v7 1/9] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-07-03  9:55       ` [dpdk-dev] [PATCH v7 2/9] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-07-03  9:55       ` [dpdk-dev] [PATCH v7 3/9] app/test: update malloc/memzone unit tests Sergio Gonzalez Monroy
2015-07-03  9:55       ` [dpdk-dev] [PATCH v7 4/9] config: remove CONFIG_RTE_MALLOC_MEMZONE_SIZE Sergio Gonzalez Monroy
2015-07-03  9:55       ` [dpdk-dev] [PATCH v7 5/9] eal: remove free_memseg and references to it Sergio Gonzalez Monroy
2015-07-03  9:55       ` [dpdk-dev] [PATCH v7 6/9] eal: new rte_memzone_free Sergio Gonzalez Monroy
2015-07-03  9:55       ` [dpdk-dev] [PATCH v7 7/9] app/test: rte_memzone_free unit test Sergio Gonzalez Monroy
2015-07-03  9:55       ` [dpdk-dev] [PATCH v7 8/9] doc: announce ABI change of librte_malloc Sergio Gonzalez Monroy
2015-07-03  9:55       ` [dpdk-dev] [PATCH v7 9/9] doc: update malloc documentation Sergio Gonzalez Monroy
2015-07-13 13:15       ` [dpdk-dev] [PATCH v7 0/9] Dynamic memzones Thomas Monjalon
2015-07-14  8:57       ` [dpdk-dev] [PATCH v8 " Sergio Gonzalez Monroy
2015-07-14  8:57         ` [dpdk-dev] [PATCH v8 1/9] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-07-14  8:57         ` [dpdk-dev] [PATCH v8 2/9] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-07-14 20:41           ` Thomas Monjalon
2015-07-14  8:57         ` [dpdk-dev] [PATCH v8 3/9] app/test: update malloc/memzone unit tests Sergio Gonzalez Monroy
2015-07-14  8:57         ` [dpdk-dev] [PATCH v8 4/9] config: remove CONFIG_RTE_MALLOC_MEMZONE_SIZE Sergio Gonzalez Monroy
2015-07-14  8:57         ` [dpdk-dev] [PATCH v8 5/9] eal: remove free_memseg and references to it Sergio Gonzalez Monroy
2015-07-14  8:57         ` [dpdk-dev] [PATCH v8 6/9] eal: new rte_memzone_free Sergio Gonzalez Monroy
2015-07-14  8:57         ` [dpdk-dev] [PATCH v8 7/9] app/test: rte_memzone_free unit test Sergio Gonzalez Monroy
2015-07-14  8:57         ` [dpdk-dev] [PATCH v8 8/9] doc: announce ABI change of librte_malloc Sergio Gonzalez Monroy
2015-07-14  8:57         ` [dpdk-dev] [PATCH v8 9/9] doc: update malloc documentation Sergio Gonzalez Monroy
2015-07-15  8:26         ` [dpdk-dev] [PATCH v9 0/9] Dynamic memzones Sergio Gonzalez Monroy
2015-07-15  8:26           ` [dpdk-dev] [PATCH v9 1/9] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-07-15  8:26           ` [dpdk-dev] [PATCH v9 2/9] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-07-15  8:26           ` [dpdk-dev] [PATCH v9 3/9] app/test: update malloc/memzone unit tests Sergio Gonzalez Monroy
2015-07-15  8:26           ` [dpdk-dev] [PATCH v9 4/9] config: remove CONFIG_RTE_MALLOC_MEMZONE_SIZE Sergio Gonzalez Monroy
2015-07-15  8:26           ` [dpdk-dev] [PATCH v9 5/9] eal: remove free_memseg and references to it Sergio Gonzalez Monroy
2015-07-15  8:26           ` [dpdk-dev] [PATCH v9 6/9] eal: new rte_memzone_free Sergio Gonzalez Monroy
2015-07-15  8:26           ` [dpdk-dev] [PATCH v9 7/9] app/test: rte_memzone_free unit test Sergio Gonzalez Monroy
2015-07-15  8:26           ` [dpdk-dev] [PATCH v9 8/9] doc: announce ABI change of librte_malloc Sergio Gonzalez Monroy
2015-07-15  8:26           ` [dpdk-dev] [PATCH v9 9/9] doc: update malloc documentation Sergio Gonzalez Monroy
2015-07-15 14:07             ` Thomas Monjalon
2015-07-15 14:11               ` Gonzalez Monroy, Sergio
2015-07-15 14:48                 ` Thomas Monjalon
2015-07-15 16:32           ` [dpdk-dev] [PATCH v10 0/9] Dynamic memzones Sergio Gonzalez Monroy
2015-07-15 16:32             ` [dpdk-dev] [PATCH v10 1/9] eal: move librte_malloc to eal/common Sergio Gonzalez Monroy
2015-07-15 16:32             ` [dpdk-dev] [PATCH v10 2/9] eal: memzone allocated by malloc Sergio Gonzalez Monroy
2015-07-15 16:32             ` [dpdk-dev] [PATCH v10 3/9] app/test: update malloc/memzone unit tests Sergio Gonzalez Monroy
2015-07-15 16:32             ` [dpdk-dev] [PATCH v10 4/9] config: remove CONFIG_RTE_MALLOC_MEMZONE_SIZE Sergio Gonzalez Monroy
2015-07-15 16:32             ` [dpdk-dev] [PATCH v10 5/9] eal: remove free_memseg and references to it Sergio Gonzalez Monroy
2015-07-15 16:32             ` [dpdk-dev] [PATCH v10 6/9] eal: new rte_memzone_free Sergio Gonzalez Monroy
2015-07-15 16:32             ` [dpdk-dev] [PATCH v10 7/9] app/test: rte_memzone_free unit test Sergio Gonzalez Monroy
2015-07-15 16:32             ` [dpdk-dev] [PATCH v10 8/9] doc: announce ABI change of librte_malloc Sergio Gonzalez Monroy
2015-07-15 16:32             ` [dpdk-dev] [PATCH v10 9/9] doc: update malloc documentation Sergio Gonzalez Monroy
2015-07-16  7:37               ` [dpdk-dev] [PATCH v11] " Sergio Gonzalez Monroy
2015-07-16 10:07                 ` Thomas Monjalon
2015-07-16 10:12                 ` Thomas Monjalon
2015-07-16 10:39                   ` Gonzalez Monroy, Sergio
2015-07-16 12:05             ` [dpdk-dev] [PATCH v10 0/9] Dynamic memzones Thomas Monjalon
2015-10-14  0:12         ` [dpdk-dev] [PATCH v8 " Stephen Hemminger
2015-10-14 11:05           ` Gonzalez Monroy, Sergio

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).