From: Thomas Monjalon <thomas@monjalon.net>
To: Jim Harris <james.r.harris@intel.com>
Cc: dev@dpdk.org, "Burakov, Anatoly" <anatoly.burakov@intel.com>,
dariusz.stojaczyk@intel.com, benjamin.walker@intel.com,
seth.howell@intel.com, olivier.matz@6wind.com,
david.marchand@redhat.com, bruce.richardson@intel.com,
ferruh.yigit@intel.com, shahafs@mellanox.com
Subject: Re: [dpdk-dev] [PATCH v2] mem: add --match-allocations
Date: Thu, 20 Dec 2018 13:00:26 +0100 [thread overview]
Message-ID: <2371021.HFLIh4q2uH@xps> (raw)
In-Reply-To: <677d302e-6583-babe-edc7-3c462aa09a62@intel.com>
17/12/2018 10:40, Burakov, Anatoly:
> On 14-Dec-18 5:13 PM, Jim Harris wrote:
> > SPDK uses the rte_mem_event_callback_register API to
> > create RDMA memory regions (MRs) for newly allocated regions
> > of memory. This is used in both the SPDK NVMe-oF target
> > and the NVMe-oF host driver.
> >
> > DPDK creates internal malloc_elem structures for these
> > allocated regions. As users malloc and free memory, DPDK
> > will sometimes merge malloc_elems that originated from
> > different allocations that were notified through the
> > registered mem_event callback routine. This results
> > in subsequent allocations that can span across multiple
> > RDMA MRs. This requires SPDK to check each DPDK buffer to
> > see if it crosses an MR boundary, and if so, would have to
> > add considerable logic and complexity to describe that
> > buffer before it can be accessed by the RNIC. It is somewhat
> > analagous to rte_malloc returning a buffer that is not
> > IOVA-contiguous.
> >
> > As a malloc_elem gets split and some of these elements
> > get freed, it can also result in DPDK sending an
> > RTE_MEM_EVENT_FREE notification for a subset of the
> > original RTE_MEM_EVENT_ALLOC notification. This is also
> > problematic for RDMA memory regions, since unregistering
> > the memory region is all-or-nothing. It is not possible
> > to unregister part of a memory region.
> >
> > To support these types of applications, this patch adds
> > a new --match-allocations EAL init flag. When this
> > flag is specified, malloc elements from different
> > hugepage allocations will never be merged. Memory will
> > also only be freed back to the system (with the requisite
> > memory event callback) exactly as it was originally
> > allocated.
> >
> > Since part of this patch is extending the size of struct
> > malloc_elem, we also fix up the malloc autotests so they
> > do not assume its size exactly fits in one cacheline.
> >
> > Signed-off-by: Jim Harris <james.r.harris@intel.com>
>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
Applied, thanks
This is one more example of how bad is the DPDK initialization.
This new option is fixing an application concern, so it should be
an API through init functions, not a user option.
I think we really need to refactor initialization APIs.
prev parent reply other threads:[~2018-12-20 12:00 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-14 17:13 Jim Harris
2018-12-17 9:40 ` Burakov, Anatoly
2018-12-17 9:45 ` Burakov, Anatoly
2018-12-20 12:00 ` Thomas Monjalon [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2371021.HFLIh4q2uH@xps \
--to=thomas@monjalon.net \
--cc=anatoly.burakov@intel.com \
--cc=benjamin.walker@intel.com \
--cc=bruce.richardson@intel.com \
--cc=dariusz.stojaczyk@intel.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=james.r.harris@intel.com \
--cc=olivier.matz@6wind.com \
--cc=seth.howell@intel.com \
--cc=shahafs@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).