DPDK patches and discussions
 help / color / mirror / Atom feed
From: Tiwei Bie <tiwei.bie@intel.com>
To: Anatoly Burakov <anatoly.burakov@intel.com>
Cc: dev@dpdk.org, przemyslawx.lal@intel.com,
	kuralamudhan.ramakrishnan@intel.com, ivan.coughlan@intel.com,
	ray.kinsella@intel.com
Subject: Re: [dpdk-dev] [PATCH 19.02 2/2] mem: use memfd for no-huge mode
Date: Wed, 28 Nov 2018 12:57:53 +0800	[thread overview]
Message-ID: <20181128045753.GA26256@debian> (raw)
In-Reply-To: <6b5e1de44f42e18a6a9326a06eb800c6bb8ddfdf.1542130721.git.anatoly.burakov@intel.com>

On Tue, Nov 13, 2018 at 05:54:48PM +0000, Anatoly Burakov wrote:
> When running in no-huge mode, we anonymously allocate our memory.
> While this works for regular NICs and vdev's, it's not suitable
> for memory sharing scenarios such as virtio with vhost_user
> backend.
> 
> To fix this, allocate no-huge memory using memfd, and register
> it with memalloc just like any other memseg fd. This will enable
> using rte_memseg_get_fd() API with --no-huge EAL flag.
> 
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
>  lib/librte_eal/linuxapp/eal/eal_memory.c | 46 ++++++++++++++++++++++--
>  1 file changed, 44 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index 48b23ce19..8feac2c56 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -25,6 +25,7 @@
>  #include <sys/time.h>
>  #include <signal.h>
>  #include <setjmp.h>
> +#include <linux/memfd.h>
>  #ifdef RTE_EAL_NUMA_AWARE_HUGEPAGES
>  #include <numa.h>
>  #include <numaif.h>
> @@ -1345,12 +1346,15 @@ eal_legacy_hugepage_init(void)
>  	/* hugetlbfs can be disabled */
>  	if (internal_config.no_hugetlbfs) {
>  		struct rte_memseg_list *msl;
> +		int n_segs, cur_seg, fd, memfd, flags;
>  		uint64_t page_sz;
> -		int n_segs, cur_seg;
>  
>  		/* nohuge mode is legacy mode */
>  		internal_config.legacy_mem = 1;
>  
> +		/* nohuge mode is single-file segments mode */
> +		internal_config.single_file_segments = 1;
> +
>  		/* create a memseg list */
>  		msl = &mcfg->memsegs[0];
>  
> @@ -1363,8 +1367,36 @@ eal_legacy_hugepage_init(void)
>  			return -1;
>  		}
>  
> +		/* set up parameters for anonymous mmap */
> +		fd = -1;
> +		flags = MAP_PRIVATE | MAP_ANONYMOUS;
> +
> +		/* create a memfd and store it in the segment fd table */
> +		memfd = memfd_create("nohuge", 0);
> +		if (memfd < 0) {
> +			RTE_LOG(ERR, EAL, "Cannot create memfd: %s\n",
> +					strerror(errno));
> +			RTE_LOG(ERR, EAL, "Falling back to anonymous map\n");
> +		} else {
> +			/* we got an fd - now resize it */
> +			if (ftruncate(memfd, internal_config.memory) < 0) {
> +				RTE_LOG(ERR, EAL, "Cannot resize memfd: %s\n",
> +						strerror(errno));
> +				RTE_LOG(ERR, EAL, "Falling back to anonymous map\n");
> +				close(memfd);
> +			} else {
> +				/* creating memfd-backed file was successful.
> +				 * we want changes to memfd to be visible to
> +				 * other processes (such as vhost backend), so
> +				 * map it as shared memory.
> +				 */
> +				RTE_LOG(DEBUG, EAL, "Using memfd for anonymous memory\n");
> +				fd = memfd;
> +				flags = MAP_SHARED;
> +			}
> +		}
>  		addr = mmap(NULL, internal_config.memory, PROT_READ | PROT_WRITE,
> -				MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +				flags, fd, 0);
>  		if (addr == MAP_FAILED) {
>  			RTE_LOG(ERR, EAL, "%s: mmap() failed: %s\n", __func__,
>  					strerror(errno));
> @@ -1375,6 +1407,16 @@ eal_legacy_hugepage_init(void)
>  		msl->socket_id = 0;
>  		msl->len = internal_config.memory;
>  
> +		/* we're in single-file segments mode, so only the segment list
> +		 * fd needs to be set up.
> +		 */
> +		if (fd != -1) {
> +			if (eal_memalloc_set_seg_list_fd(0, fd) < 0) {
> +				RTE_LOG(ERR, EAL, "Cannot set up segment list fd\n");
> +				/* not a serious error, proceed */
> +			}
> +		}

Hi Anatoly,

Thanks for the work!

It seems the support for getting fd offset is missing in no-huge
mode. I got below error in virtio-user while trying this series
with --no-huge:

update_memory_region(): Failed to get offset, ms=0x10002e000 rte_errno=19

Thanks

> +
>  		/* populate memsegs. each memseg is one page long */
>  		for (cur_seg = 0; cur_seg < n_segs; cur_seg++) {
>  			arr = &msl->memseg_arr;
> -- 
> 2.17.1

  reply	other threads:[~2018-11-28  4:59 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-11-13 17:54 [dpdk-dev] [PATCH 19.02 0/2] Allow using virtio without hugepages Anatoly Burakov
2018-11-13 17:54 ` [dpdk-dev] [PATCH 19.02 1/2] memalloc: allow setting up segment list fd's Anatoly Burakov
2018-11-13 17:54 ` [dpdk-dev] [PATCH 19.02 2/2] mem: use memfd for no-huge mode Anatoly Burakov
2018-11-28  4:57   ` Tiwei Bie [this message]
2018-11-28  9:11     ` Burakov, Anatoly
2018-12-11 16:43 ` [dpdk-dev] [PATCH v2 0/5] Allow using virtio without hugepages Anatoly Burakov
2018-12-13  4:53   ` Tiwei Bie
2018-12-13 11:43   ` [dpdk-dev] [PATCH v3 0/5] Allow using virtio-user " Anatoly Burakov
2018-12-20 22:01     ` Thomas Monjalon
2018-12-13 11:43   ` [dpdk-dev] [PATCH v3 1/5] mem: fix error code for segment fd API for external segs Anatoly Burakov
2018-12-14  9:15     ` Maxime Coquelin
2018-12-13 11:43   ` [dpdk-dev] [PATCH v3 2/5] memalloc: check for memfd support in segment fd API Anatoly Burakov
2018-12-14  9:19     ` Maxime Coquelin
2018-12-13 11:43   ` [dpdk-dev] [PATCH v3 3/5] memalloc: allow setting up segment list fd's Anatoly Burakov
2018-12-14 10:03     ` Maxime Coquelin
2018-12-13 11:43   ` [dpdk-dev] [PATCH v3 4/5] mem: use memfd for no-huge mode Anatoly Burakov
2018-12-13 11:59     ` Burakov, Anatoly
2018-12-14 10:06     ` Maxime Coquelin
2018-12-13 11:43   ` [dpdk-dev] [PATCH v3 5/5] test: add segment fd API test Anatoly Burakov
2018-12-14 10:09     ` Maxime Coquelin
2018-12-11 16:43 ` [dpdk-dev] [PATCH v2 1/5] mem: fix error code for segment fd API for external segs Anatoly Burakov
2018-12-11 16:43 ` [dpdk-dev] [PATCH v2 2/5] memalloc: check for memfd support in segment fd API Anatoly Burakov
2018-12-11 16:43 ` [dpdk-dev] [PATCH v2 3/5] memalloc: allow setting up segment list fd's Anatoly Burakov
2018-12-11 16:43 ` [dpdk-dev] [PATCH v2 4/5] mem: use memfd for no-huge mode Anatoly Burakov
2018-12-13  4:59   ` Tiwei Bie
2018-12-13 11:36     ` Burakov, Anatoly
2018-12-11 16:43 ` [dpdk-dev] [PATCH v2 5/5] test: add segment fd API test Anatoly Burakov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20181128045753.GA26256@debian \
    --to=tiwei.bie@intel.com \
    --cc=anatoly.burakov@intel.com \
    --cc=dev@dpdk.org \
    --cc=ivan.coughlan@intel.com \
    --cc=kuralamudhan.ramakrishnan@intel.com \
    --cc=przemyslawx.lal@intel.com \
    --cc=ray.kinsella@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).