From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by dpdk.org (Postfix) with ESMTP id C4F147CBD for ; Wed, 21 Mar 2018 10:11:00 +0100 (CET) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w2L9Ao1b124693 for ; Wed, 21 Mar 2018 05:11:00 -0400 Received: from e06smtp14.uk.ibm.com (e06smtp14.uk.ibm.com [195.75.94.110]) by mx0b-001b2d01.pphosted.com with ESMTP id 2gukyxseh0-1 (version=TLSv1.2 cipher=AES256-SHA256 bits=256 verify=NOT) for ; Wed, 21 Mar 2018 05:10:53 -0400 Received: from localhost by e06smtp14.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 21 Mar 2018 09:09:45 -0000 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp14.uk.ibm.com (192.168.101.144) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 21 Mar 2018 09:09:41 -0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w2L99eKx54722758; Wed, 21 Mar 2018 09:09:40 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3652411C06C; Wed, 21 Mar 2018 09:02:11 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9A2ED11C052; Wed, 21 Mar 2018 09:02:07 +0000 (GMT) Received: from [9.124.35.231] (unknown [9.124.35.231]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 21 Mar 2018 09:02:07 +0000 (GMT) To: Anatoly Burakov Cc: dev@dpdk.org, keith.wiles@intel.com, jianfeng.tan@intel.com, andras.kovacs@ericsson.com, laszlo.vadkeri@ericsson.com, benjamin.walker@intel.com, bruce.richardson@intel.com, thomas@monjalon.net, konstantin.ananyev@intel.com, kuralamudhan.ramakrishnan@intel.com, louise.m.daly@intel.com, nelio.laranjeiro@6wind.com, yskoh@mellanox.com, pepperjo@japf.ch, jerin.jacob@caviumnetworks.com, hemant.agrawal@nxp.com, olivier.matz@6wind.com, Pradeep Satyanarayana , Chao Zhu References: From: gowrishankar muthukrishnan Date: Wed, 21 Mar 2018 14:39:35 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-TM-AS-GCONF: 00 x-cbid: 18032109-0044-0000-0000-0000053E3B33 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18032109-0045-0000-0000-0000287D3D49 Message-Id: <2071dc94-9bea-97b1-7d8c-ef8b92bb9e13@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-03-21_02:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1803210113 Subject: Re: [dpdk-dev] [PATCH 00/41] Memory Hotplug for DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Mar 2018 09:11:01 -0000 On Saturday 03 March 2018 07:15 PM, Anatoly Burakov wrote: > This patchset introduces dynamic memory allocation for DPDK (aka memory > hotplug). Based upon RFC submitted in December [1]. Hi Anatoly, I am able to bring up pmd with these patches in powerpc. I am continuing to validate more on memory limits that this patch set has (eg pre-allocating anonymous mapping for largest mem possible as per default mem and memseg values). I'll keep posting my observations further. Thanks for the patches, Gowrishankar > Dependencies (to be applied in specified order): > - IPC bugfixes patchset [2] > - IPC improvements patchset [3] > - IPC asynchronous request API patch [4] > - Function to return number of sockets [5] > > Deprecation notices relevant to this patchset: > - General outline of memory hotplug changes [6] > - EAL NUMA node count changes [7] > > The vast majority of changes are in the EAL and malloc, the external API > disruption is minimal: a new set of API's are added for contiguous memory > allocation for rte_memzone, and a few API additions in rte_memory due to > switch to memseg_lists as opposed to memsegs. Every other API change is > internal to EAL, and all of the memory allocation/freeing is handled > through rte_malloc, with no externally visible API changes. > > Quick outline of all changes done as part of this patchset: > > * Malloc heap adjusted to handle holes in address space > * Single memseg list replaced by multiple memseg lists > * VA space for hugepages is preallocated in advance > * Added alloc/free for pages happening as needed on rte_malloc/rte_free > * Added contiguous memory allocation API's for rte_memzone > * Integrated Pawel Wodkowski's patch for registering/unregistering memory > with VFIO [8] > * Callbacks for registering memory allocations > * Multiprocess support done via DPDK IPC introduced in 18.02 > > The biggest difference is a "memseg" now represents a single page (as opposed to > being a big contiguous block of pages). As a consequence, both memzones and > malloc elements are no longer guaranteed to be physically contiguous, unless > the user asks for it at reserve time. To preserve whatever functionality that > was dependent on previous behavior, a legacy memory option is also provided, > however it is expected (or perhaps vainly hoped) to be temporary solution. > > Why multiple memseg lists instead of one? Since memseg is a single page now, > the list of memsegs will get quite big, and we need to locate pages somehow > when we allocate and free them. We could of course just walk the list and > allocate one contiguous chunk of VA space for memsegs, but this > implementation uses separate lists instead in order to speed up many > operations with memseg lists. > > For v1, the following limitations are present: > - FreeBSD does not even compile, let alone run > - No 32-bit support > - There are some minor quality-of-life improvements planned that aren't > ready yet and will be part of v2 > - VFIO support is only smoke-tested (but is expected to work), VFIO support > with secondary processes is not tested; work is ongoing to validate VFIO > for all use cases > - Dynamic mapping/unmapping memory with VFIO is not supported in sPAPR > IOMMU mode - help from sPAPR maintainers requested > > Nevertheless, this patchset should be testable under 64-bit Linux, and > should work for all use cases bar those mentioned above. > > [1] http://dpdk.org/dev/patchwork/bundle/aburakov/Memory_RFC/ > [2] http://dpdk.org/dev/patchwork/bundle/aburakov/IPC_Fixes/ > [3] http://dpdk.org/dev/patchwork/bundle/aburakov/IPC_Improvements/ > [4] http://dpdk.org/dev/patchwork/bundle/aburakov/IPC_Async_Request/ > [5] http://dpdk.org/dev/patchwork/bundle/aburakov/Num_Sockets/ > [6] http://dpdk.org/dev/patchwork/patch/34002/ > [7] http://dpdk.org/dev/patchwork/patch/33853/ > [8] http://dpdk.org/dev/patchwork/patch/24484/ > > Anatoly Burakov (41): > eal: move get_virtual_area out of linuxapp eal_memory.c > eal: move all locking to heap > eal: make malloc heap a doubly-linked list > eal: add function to dump malloc heap contents > test: add command to dump malloc heap contents > eal: make malloc_elem_join_adjacent_free public > eal: make malloc free list remove public > eal: make malloc free return resulting malloc element > eal: add rte_fbarray > eal: add "single file segments" command-line option > eal: add "legacy memory" option > eal: read hugepage counts from node-specific sysfs path > eal: replace memseg with memseg lists > eal: add support for mapping hugepages at runtime > eal: add support for unmapping pages at runtime > eal: make use of memory hotplug for init > eal: enable memory hotplug support in rte_malloc > test: fix malloc autotest to support memory hotplug > eal: add API to check if memory is contiguous > eal: add backend support for contiguous allocation > eal: enable reserving physically contiguous memzones > eal: replace memzone array with fbarray > mempool: add support for the new allocation methods > vfio: allow to map other memory regions > eal: map/unmap memory with VFIO when alloc/free pages > eal: prepare memseg lists for multiprocess sync > eal: add multiprocess init with memory hotplug > eal: add support for multiprocess memory hotplug > eal: add support for callbacks on memory hotplug > eal: enable callbacks on malloc/free and mp sync > ethdev: use contiguous allocation for DMA memory > crypto/qat: use contiguous allocation for DMA memory > net/avf: use contiguous allocation for DMA memory > net/bnx2x: use contiguous allocation for DMA memory > net/cxgbe: use contiguous allocation for DMA memory > net/ena: use contiguous allocation for DMA memory > net/enic: use contiguous allocation for DMA memory > net/i40e: use contiguous allocation for DMA memory > net/qede: use contiguous allocation for DMA memory > net/virtio: use contiguous allocation for DMA memory > net/vmxnet3: use contiguous allocation for DMA memory > > config/common_base | 15 +- > drivers/bus/pci/linux/pci.c | 29 +- > drivers/crypto/qat/qat_qp.c | 4 +- > drivers/net/avf/avf_ethdev.c | 2 +- > drivers/net/bnx2x/bnx2x.c | 2 +- > drivers/net/bnx2x/bnx2x_rxtx.c | 3 +- > drivers/net/cxgbe/sge.c | 3 +- > drivers/net/ena/base/ena_plat_dpdk.h | 7 +- > drivers/net/ena/ena_ethdev.c | 10 +- > drivers/net/enic/enic_main.c | 4 +- > drivers/net/i40e/i40e_ethdev.c | 2 +- > drivers/net/i40e/i40e_rxtx.c | 2 +- > drivers/net/qede/base/bcm_osal.c | 5 +- > drivers/net/virtio/virtio_ethdev.c | 8 +- > drivers/net/virtio/virtio_user/vhost_kernel.c | 108 ++- > drivers/net/vmxnet3/vmxnet3_ethdev.c | 7 +- > lib/librte_eal/bsdapp/eal/Makefile | 4 + > lib/librte_eal/bsdapp/eal/eal.c | 25 + > lib/librte_eal/bsdapp/eal/eal_hugepage_info.c | 7 + > lib/librte_eal/bsdapp/eal/eal_memalloc.c | 33 + > lib/librte_eal/bsdapp/eal/meson.build | 1 + > lib/librte_eal/common/Makefile | 2 +- > lib/librte_eal/common/eal_common_fbarray.c | 859 +++++++++++++++++ > lib/librte_eal/common/eal_common_memalloc.c | 181 ++++ > lib/librte_eal/common/eal_common_memory.c | 512 +++++++++- > lib/librte_eal/common/eal_common_memzone.c | 275 ++++-- > lib/librte_eal/common/eal_common_options.c | 8 + > lib/librte_eal/common/eal_filesystem.h | 13 + > lib/librte_eal/common/eal_hugepages.h | 7 + > lib/librte_eal/common/eal_internal_cfg.h | 10 +- > lib/librte_eal/common/eal_memalloc.h | 41 + > lib/librte_eal/common/eal_options.h | 4 + > lib/librte_eal/common/eal_private.h | 33 + > lib/librte_eal/common/include/rte_eal_memconfig.h | 29 +- > lib/librte_eal/common/include/rte_fbarray.h | 352 +++++++ > lib/librte_eal/common/include/rte_malloc.h | 9 + > lib/librte_eal/common/include/rte_malloc_heap.h | 6 + > lib/librte_eal/common/include/rte_memory.h | 79 +- > lib/librte_eal/common/include/rte_memzone.h | 155 ++- > lib/librte_eal/common/include/rte_vfio.h | 39 + > lib/librte_eal/common/malloc_elem.c | 436 +++++++-- > lib/librte_eal/common/malloc_elem.h | 41 +- > lib/librte_eal/common/malloc_heap.c | 694 +++++++++++++- > lib/librte_eal/common/malloc_heap.h | 15 +- > lib/librte_eal/common/malloc_mp.c | 723 ++++++++++++++ > lib/librte_eal/common/malloc_mp.h | 86 ++ > lib/librte_eal/common/meson.build | 4 + > lib/librte_eal/common/rte_malloc.c | 75 +- > lib/librte_eal/linuxapp/eal/Makefile | 5 + > lib/librte_eal/linuxapp/eal/eal.c | 102 +- > lib/librte_eal/linuxapp/eal/eal_hugepage_info.c | 155 ++- > lib/librte_eal/linuxapp/eal/eal_memalloc.c | 1049 +++++++++++++++++++++ > lib/librte_eal/linuxapp/eal/eal_memory.c | 516 ++++++---- > lib/librte_eal/linuxapp/eal/eal_vfio.c | 318 +++++-- > lib/librte_eal/linuxapp/eal/eal_vfio.h | 11 + > lib/librte_eal/linuxapp/eal/meson.build | 1 + > lib/librte_eal/rte_eal_version.map | 23 +- > lib/librte_ether/rte_ethdev.c | 3 +- > lib/librte_mempool/rte_mempool.c | 87 +- > test/test/commands.c | 3 + > test/test/test_malloc.c | 71 +- > test/test/test_memory.c | 43 +- > test/test/test_memzone.c | 26 +- > 63 files changed, 6631 insertions(+), 751 deletions(-) > create mode 100644 lib/librte_eal/bsdapp/eal/eal_memalloc.c > create mode 100644 lib/librte_eal/common/eal_common_fbarray.c > create mode 100644 lib/librte_eal/common/eal_common_memalloc.c > create mode 100644 lib/librte_eal/common/eal_memalloc.h > create mode 100644 lib/librte_eal/common/include/rte_fbarray.h > create mode 100644 lib/librte_eal/common/malloc_mp.c > create mode 100644 lib/librte_eal/common/malloc_mp.h > create mode 100644 lib/librte_eal/linuxapp/eal/eal_memalloc.c >