From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by dpdk.org (Postfix) with ESMTP id 0AE0E5F1C for ; Thu, 20 Dec 2018 13:00:31 +0100 (CET) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 6831522169; Thu, 20 Dec 2018 07:00:29 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Thu, 20 Dec 2018 07:00:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=mesmtp; bh=W9cdH1jGLqpIz4AF64eITooJ9Ou/lHkhXmwCto8geZs=; b=qY2ajEBwXNn/ htFhAKJek5Y/dVcIUQJVc2n1JvCWvFYimk/I/iUVa6Pro+8WX8u9dTYgxsEICah/ f6HGVzZ5YgtAbQVFh3Yp6jti2XcLPD4KxEz4KN8WfDqo5lnBjWmurx5Y0+vwLiHY PzelbBWAU7qNI8cgWSX2xLbbnpOrINE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=W9cdH1jGLqpIz4AF64eITooJ9Ou/lHkhXmwCto8ge Zs=; b=bswptmQI54RcLA8/45XIw/jOWvXi6h1B6FarsjaKP9krAXNcVeWrLtLn8 KZn0J1W6KOjPeRabsJpu6McJHdLLPhJK7+nQJsL6OzkydgnVE/9FevgRjyY7axPe v4joAOFq2aU2Qy4QziIvW1glXD/9mgONkHkDLXk8OS+mli+IpMyujnjM43P1TYMu erQbDd1QMgdSDri8XHBZvugZmstx+rMBR/XEIGAPc2xT83qH/lFb5XGrLP1igCVq Pz2EBttXNE+rQnKs2+uqx1F6SM7mCFx/4mLLPEyATiVgz9VR9IAH3CNXqE3Hd4/c NKDJFNHqxgu3uWUazy1Eg56RooJUw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedtkedrudejfedgfeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfquhhtnecuuegrihhlohhuthemucef tddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvffufffkjg hfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhmrghsucfoohhnjhgrlhhonhcu oehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucfkphepjeejrddufeegrddvtd efrddukeegnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrshesmhhonhhjrghl ohhnrdhnvghtnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id B2682E4598; Thu, 20 Dec 2018 07:00:27 -0500 (EST) From: Thomas Monjalon To: Jim Harris Cc: dev@dpdk.org, "Burakov, Anatoly" , dariusz.stojaczyk@intel.com, benjamin.walker@intel.com, seth.howell@intel.com, olivier.matz@6wind.com, david.marchand@redhat.com, bruce.richardson@intel.com, ferruh.yigit@intel.com, shahafs@mellanox.com Date: Thu, 20 Dec 2018 13:00:26 +0100 Message-ID: <2371021.HFLIh4q2uH@xps> In-Reply-To: <677d302e-6583-babe-edc7-3c462aa09a62@intel.com> References: <20181214171303.21817-1-james.r.harris@intel.com> <677d302e-6583-babe-edc7-3c462aa09a62@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-dev] [PATCH v2] mem: add --match-allocations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2018 12:00:31 -0000 17/12/2018 10:40, Burakov, Anatoly: > On 14-Dec-18 5:13 PM, Jim Harris wrote: > > SPDK uses the rte_mem_event_callback_register API to > > create RDMA memory regions (MRs) for newly allocated regions > > of memory. This is used in both the SPDK NVMe-oF target > > and the NVMe-oF host driver. > > > > DPDK creates internal malloc_elem structures for these > > allocated regions. As users malloc and free memory, DPDK > > will sometimes merge malloc_elems that originated from > > different allocations that were notified through the > > registered mem_event callback routine. This results > > in subsequent allocations that can span across multiple > > RDMA MRs. This requires SPDK to check each DPDK buffer to > > see if it crosses an MR boundary, and if so, would have to > > add considerable logic and complexity to describe that > > buffer before it can be accessed by the RNIC. It is somewhat > > analagous to rte_malloc returning a buffer that is not > > IOVA-contiguous. > > > > As a malloc_elem gets split and some of these elements > > get freed, it can also result in DPDK sending an > > RTE_MEM_EVENT_FREE notification for a subset of the > > original RTE_MEM_EVENT_ALLOC notification. This is also > > problematic for RDMA memory regions, since unregistering > > the memory region is all-or-nothing. It is not possible > > to unregister part of a memory region. > > > > To support these types of applications, this patch adds > > a new --match-allocations EAL init flag. When this > > flag is specified, malloc elements from different > > hugepage allocations will never be merged. Memory will > > also only be freed back to the system (with the requisite > > memory event callback) exactly as it was originally > > allocated. > > > > Since part of this patch is extending the size of struct > > malloc_elem, we also fix up the malloc autotests so they > > do not assume its size exactly fits in one cacheline. > > > > Signed-off-by: Jim Harris > > Reviewed-by: Anatoly Burakov Applied, thanks This is one more example of how bad is the DPDK initialization. This new option is fixing an application concern, so it should be an API through init functions, not a user option. I think we really need to refactor initialization APIs.