From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 1E6B61B728 for ; Mon, 17 Dec 2018 10:40:02 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Dec 2018 01:40:02 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,364,1539673200"; d="scan'208";a="130570746" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.251.86.46]) ([10.251.86.46]) by fmsmga001.fm.intel.com with ESMTP; 17 Dec 2018 01:40:00 -0800 To: Jim Harris , dev@dpdk.org Cc: dariusz.stojaczyk@intel.com, benjamin.walker@intel.com, seth.howell@intel.com References: <20181214171303.21817-1-james.r.harris@intel.com> From: "Burakov, Anatoly" Message-ID: <677d302e-6583-babe-edc7-3c462aa09a62@intel.com> Date: Mon, 17 Dec 2018 09:40:00 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: <20181214171303.21817-1-james.r.harris@intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v2] mem: add --match-allocations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 09:40:03 -0000 On 14-Dec-18 5:13 PM, Jim Harris wrote: > SPDK uses the rte_mem_event_callback_register API to > create RDMA memory regions (MRs) for newly allocated regions > of memory. This is used in both the SPDK NVMe-oF target > and the NVMe-oF host driver. > > DPDK creates internal malloc_elem structures for these > allocated regions. As users malloc and free memory, DPDK > will sometimes merge malloc_elems that originated from > different allocations that were notified through the > registered mem_event callback routine. This results > in subsequent allocations that can span across multiple > RDMA MRs. This requires SPDK to check each DPDK buffer to > see if it crosses an MR boundary, and if so, would have to > add considerable logic and complexity to describe that > buffer before it can be accessed by the RNIC. It is somewhat > analagous to rte_malloc returning a buffer that is not > IOVA-contiguous. > > As a malloc_elem gets split and some of these elements > get freed, it can also result in DPDK sending an > RTE_MEM_EVENT_FREE notification for a subset of the > original RTE_MEM_EVENT_ALLOC notification. This is also > problematic for RDMA memory regions, since unregistering > the memory region is all-or-nothing. It is not possible > to unregister part of a memory region. > > To support these types of applications, this patch adds > a new --match-allocations EAL init flag. When this > flag is specified, malloc elements from different > hugepage allocations will never be merged. Memory will > also only be freed back to the system (with the requisite > memory event callback) exactly as it was originally > allocated. > > Since part of this patch is extending the size of struct > malloc_elem, we also fix up the malloc autotests so they > do not assume its size exactly fits in one cacheline. > > Signed-off-by: Jim Harris > --- Reviewed-by: Anatoly Burakov -- Thanks, Anatoly