From: Yongseok Koh <yskoh@mellanox.com>
To: Anatoly Burakov <anatoly.burakov@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
John McNamara <john.mcnamara@intel.com>,
Marko Kovacevic <marko.kovacevic@intel.com>,
Shahaf Shuler <shahafs@mellanox.com>,
Thomas Monjalon <thomas@monjalon.net>,
"shreyansh.jain@nxp.com" <shreyansh.jain@nxp.com>
Subject: Re: [dpdk-dev] [PATCH 4/4] mem: allow usage of non-heap external memory in multiprocess
Date: Fri, 14 Dec 2018 09:56:29 +0000 [thread overview]
Message-ID: <20181214095620.GD12221@mtidpdk.mti.labs.mlnx> (raw)
In-Reply-To: <b50a376ae05397a9f741c1291609b5540d64fe02.1543495935.git.anatoly.burakov@intel.com>
On Thu, Nov 29, 2018 at 01:48:35PM +0000, Anatoly Burakov wrote:
> Add multiprocess support for externally allocated memory areas that
> are not added to DPDK heap (and add relevant doc sections).
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
Acked-by: Yongseok Koh <yskoh@mellanox.com>
Thanks
> .../prog_guide/env_abstraction_layer.rst | 3 +
> lib/librte_eal/common/eal_common_memory.c | 42 +++++++++++++
> lib/librte_eal/common/include/rte_memory.h | 59 +++++++++++++++++++
> lib/librte_eal/rte_eal_version.map | 2 +
> 4 files changed, 106 insertions(+)
>
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index d7799b626..b0491bf2d 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -276,11 +276,14 @@ The expected workflow is as follows:
> * Register memory within DPDK
> - If IOVA table is not specified, IOVA addresses will be assumed to be
> unavailable
> + - Other processes must attach to the memory area before they can use it
> * Perform DMA mapping with ``rte_vfio_dma_map`` if needed
> * Use the memory area in your application
> * If memory area is no longer needed, it can be unregistered
> - If the area was mapped for DMA, unmapping must be performed before
> unregistering memory
> + - Other processes must detach from the memory area before it can be
> + unregistered
>
> Since these externally allocated memory areas will not be managed by DPDK, it is
> therefore up to the user application to decide how to use them and what to do
> diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/common/eal_common_memory.c
> index a2e085ae8..67b445c31 100644
> --- a/lib/librte_eal/common/eal_common_memory.c
> +++ b/lib/librte_eal/common/eal_common_memory.c
> @@ -849,6 +849,48 @@ rte_extmem_unregister(void *va_addr, size_t len)
> return ret;
> }
>
> +static int
> +sync_memory(void *va_addr, size_t len, bool attach)
> +{
> + struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
> + struct rte_memseg_list *msl;
> + int ret = 0;
> +
> + if (va_addr == NULL || len == 0) {
> + rte_errno = EINVAL;
> + return -1;
> + }
> + rte_rwlock_write_lock(&mcfg->memory_hotplug_lock);
> +
> + /* find our segment */
> + msl = malloc_heap_find_external_seg(va_addr, len);
> + if (msl == NULL) {
> + rte_errno = ENOENT;
> + ret = -1;
> + goto unlock;
> + }
> + if (attach)
> + ret = rte_fbarray_attach(&msl->memseg_arr);
> + else
> + ret = rte_fbarray_detach(&msl->memseg_arr);
> +
> +unlock:
> + rte_rwlock_write_unlock(&mcfg->memory_hotplug_lock);
> + return ret;
> +}
> +
> +int __rte_experimental
> +rte_extmem_attach(void *va_addr, size_t len)
> +{
> + return sync_memory(va_addr, len, true);
> +}
> +
> +int __rte_experimental
> +rte_extmem_detach(void *va_addr, size_t len)
> +{
> + return sync_memory(va_addr, len, false);
> +}
> +
> /* init memory subsystem */
> int
> rte_eal_memory_init(void)
> diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/common/include/rte_memory.h
> index 4a43c1a9e..050bb6d8e 100644
> --- a/lib/librte_eal/common/include/rte_memory.h
> +++ b/lib/librte_eal/common/include/rte_memory.h
> @@ -435,6 +435,10 @@ rte_memseg_get_fd_offset_thread_unsafe(const struct rte_memseg *ms,
> * @note This API will not perform any DMA mapping. It is expected that user
> * will do that themselves.
> *
> + * @note Before accessing this memory in other processes, it needs to be
> + * attached in each of those processes by calling ``rte_extmem_attach`` in
> + * each other process.
> + *
> * @param va_addr
> * Start of virtual area to register
> * @param len
> @@ -472,6 +476,9 @@ rte_extmem_register(void *va_addr, size_t len, rte_iova_t iova_addrs[],
> * @note This API will not perform any DMA unmapping. It is expected that user
> * will do that themselves.
> *
> + * @note Before calling this function, all other processes must call
> + * ``rte_extmem_detach`` to detach from the memory area.
> + *
> * @param va_addr
> * Start of virtual area to unregister
> * @param len
> @@ -486,6 +493,58 @@ rte_extmem_register(void *va_addr, size_t len, rte_iova_t iova_addrs[],
> int __rte_experimental
> rte_extmem_unregister(void *va_addr, size_t len);
>
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Attach to external memory chunk registered in another process.
> + *
> + * @note Using this API is mutually exclusive with ``rte_malloc`` family of
> + * API's.
> + *
> + * @note This API will not perform any DMA mapping. It is expected that user
> + * will do that themselves.
> + *
> + * @param va_addr
> + * Start of virtual area to register
> + * @param len
> + * Length of virtual area to register
> + *
> + * @return
> + * - 0 on success
> + * - -1 in case of error, with rte_errno set to one of the following:
> + * EINVAL - one of the parameters was invalid
> + * ENOENT - memory chunk was not found
> + */
> +int __rte_experimental
> +rte_extmem_attach(void *va_addr, size_t len);
> +
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Detach from external memory chunk registered in another process.
> + *
> + * @note Using this API is mutually exclusive with ``rte_malloc`` family of
> + * API's.
> + *
> + * @note This API will not perform any DMA unmapping. It is expected that user
> + * will do that themselves.
> + *
> + * @param va_addr
> + * Start of virtual area to unregister
> + * @param len
> + * Length of virtual area to unregister
> + *
> + * @return
> + * - 0 on success
> + * - -1 in case of error, with rte_errno set to one of the following:
> + * EINVAL - one of the parameters was invalid
> + * ENOENT - memory chunk was not found
> + */
> +int __rte_experimental
> +rte_extmem_detach(void *va_addr, size_t len);
> +
> /**
> * Dump the physical memory layout to a file.
> *
> diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> index 593691a14..eb5f7b9cb 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -296,6 +296,8 @@ EXPERIMENTAL {
> rte_devargs_remove;
> rte_devargs_type_count;
> rte_eal_cleanup;
> + rte_extmem_attach;
> + rte_extmem_detach;
> rte_extmem_register;
> rte_extmem_unregister;
> rte_fbarray_attach;
> --
> 2.17.1
next prev parent reply other threads:[~2018-12-14 9:56 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-29 13:48 [dpdk-dev] [PATCH 0/4] Allow using external memory without malloc Anatoly Burakov
2018-11-29 13:48 ` [dpdk-dev] [PATCH 1/4] malloc: separate creating memseg list and malloc heap Anatoly Burakov
2018-12-14 9:33 ` Yongseok Koh
2018-11-29 13:48 ` [dpdk-dev] [PATCH 2/4] malloc: separate destroying memseg list and heap data Anatoly Burakov
2018-12-14 9:34 ` Yongseok Koh
2018-11-29 13:48 ` [dpdk-dev] [PATCH 3/4] mem: allow registering external memory areas Anatoly Burakov
2018-12-14 9:55 ` Yongseok Koh
2018-12-14 11:03 ` Burakov, Anatoly
2018-11-29 13:48 ` [dpdk-dev] [PATCH 4/4] mem: allow usage of non-heap external memory in multiprocess Anatoly Burakov
2018-12-14 9:56 ` Yongseok Koh [this message]
2018-12-02 5:48 ` [dpdk-dev] [PATCH 0/4] Allow using external memory without malloc Shahaf Shuler
2018-12-02 23:28 ` Yongseok Koh
2018-12-03 10:23 ` Burakov, Anatoly
2018-12-12 12:55 ` Yongseok Koh
2018-12-12 13:17 ` Burakov, Anatoly
2018-12-14 11:50 ` [dpdk-dev] [PATCH v2 " Anatoly Burakov
2018-12-20 15:32 ` [dpdk-dev] [PATCH v3 " Anatoly Burakov
2018-12-20 16:16 ` Stephen Hemminger
2018-12-20 17:18 ` Thomas Monjalon
2018-12-21 9:17 ` Burakov, Anatoly
2018-12-20 17:17 ` Thomas Monjalon
2018-12-20 15:32 ` [dpdk-dev] [PATCH v3 1/4] malloc: separate creating memseg list and malloc heap Anatoly Burakov
2018-12-20 15:32 ` [dpdk-dev] [PATCH v3 2/4] malloc: separate destroying memseg list and heap data Anatoly Burakov
2018-12-20 15:32 ` [dpdk-dev] [PATCH v3 3/4] mem: allow registering external memory areas Anatoly Burakov
2018-12-20 15:32 ` [dpdk-dev] [PATCH v3 4/4] mem: allow usage of non-heap external memory in multiprocess Anatoly Burakov
2018-12-14 11:50 ` [dpdk-dev] [PATCH v2 1/4] malloc: separate creating memseg list and malloc heap Anatoly Burakov
2018-12-14 11:50 ` [dpdk-dev] [PATCH v2 2/4] malloc: separate destroying memseg list and heap data Anatoly Burakov
2018-12-14 11:50 ` [dpdk-dev] [PATCH v2 3/4] mem: allow registering external memory areas Anatoly Burakov
2018-12-14 11:50 ` [dpdk-dev] [PATCH v2 4/4] mem: allow usage of non-heap external memory in multiprocess Anatoly Burakov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181214095620.GD12221@mtidpdk.mti.labs.mlnx \
--to=yskoh@mellanox.com \
--cc=anatoly.burakov@intel.com \
--cc=dev@dpdk.org \
--cc=john.mcnamara@intel.com \
--cc=marko.kovacevic@intel.com \
--cc=shahafs@mellanox.com \
--cc=shreyansh.jain@nxp.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).