From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 2CA7B374 for ; Mon, 17 Dec 2018 10:45:12 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Dec 2018 01:45:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,364,1539673200"; d="scan'208";a="130571731" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.251.86.46]) ([10.251.86.46]) by fmsmga001.fm.intel.com with ESMTP; 17 Dec 2018 01:45:11 -0800 To: Jim Harris , dev@dpdk.org Cc: dariusz.stojaczyk@intel.com, benjamin.walker@intel.com, seth.howell@intel.com References: <20181214171303.21817-1-james.r.harris@intel.com> <677d302e-6583-babe-edc7-3c462aa09a62@intel.com> From: "Burakov, Anatoly" Message-ID: <44e46a5a-7edc-5146-9fa3-d3c3a62abc4e@intel.com> Date: Mon, 17 Dec 2018 09:45:09 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: <677d302e-6583-babe-edc7-3c462aa09a62@intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v2] mem: add --match-allocations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 09:45:13 -0000 On 17-Dec-18 9:40 AM, Burakov, Anatoly wrote: > On 14-Dec-18 5:13 PM, Jim Harris wrote: >> SPDK uses the rte_mem_event_callback_register API to >> create RDMA memory regions (MRs) for newly allocated regions >> of memory. This is used in both the SPDK NVMe-oF target >> and the NVMe-oF host driver. >> >> DPDK creates internal malloc_elem structures for these >> allocated regions. As users malloc and free memory, DPDK >> will sometimes merge malloc_elems that originated from >> different allocations that were notified through the >> registered mem_event callback routine. This results >> in subsequent allocations that can span across multiple >> RDMA MRs. This requires SPDK to check each DPDK buffer to >> see if it crosses an MR boundary, and if so, would have to >> add considerable logic and complexity to describe that >> buffer before it can be accessed by the RNIC. It is somewhat >> analagous to rte_malloc returning a buffer that is not >> IOVA-contiguous. >> >> As a malloc_elem gets split and some of these elements >> get freed, it can also result in DPDK sending an >> RTE_MEM_EVENT_FREE notification for a subset of the >> original RTE_MEM_EVENT_ALLOC notification. This is also >> problematic for RDMA memory regions, since unregistering >> the memory region is all-or-nothing. It is not possible >> to unregister part of a memory region. >> >> To support these types of applications, this patch adds >> a new --match-allocations EAL init flag. When this >> flag is specified, malloc elements from different >> hugepage allocations will never be merged. Memory will >> also only be freed back to the system (with the requisite >> memory event callback) exactly as it was originally >> allocated. >> >> Since part of this patch is extending the size of struct >> malloc_elem, we also fix up the malloc autotests so they >> do not assume its size exactly fits in one cacheline. >> >> Signed-off-by: Jim Harris >> --- > > Reviewed-by: Anatoly Burakov > To those reviewing, some relevant discussion: https://mails.dpdk.org/archives/dev/2018-December/120225.html -- Thanks, Anatoly