DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [PATCH] eal/mem: preallocate VA space in no-huge mode
@ 2020-01-24 17:01 Anatoly Burakov
  2020-01-24 17:05 ` [dpdk-dev] [PATCH v2] " Anatoly Burakov
  0 siblings, 1 reply; 10+ messages in thread
From: Anatoly Burakov @ 2020-01-24 17:01 UTC (permalink / raw)
  To: dev; +Cc: stable

When --no-huge mode is used, the memory is currently allocated with
mmap(NULL, ...). This is fine in most cases, but can fail in cases
where DPDK is run on a machine with an IOMMU that is of more limited
address width than that of a VA, because we're not specifying the
address hint for mmap() call.

Fix it by preallocating VA space before mapping it.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---

Notes:
    I couldn't figure out which specific commit has introduced
    the issue, so there's no fix tag. The most likely candidate
    is one that introduced the DMA mask thing in the first place
    but i'm not sure.

 lib/librte_eal/linux/eal/eal_memory.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/lib/librte_eal/linux/eal/eal_memory.c b/lib/librte_eal/linux/eal/eal_memory.c
index 43e4ffc757..672f8806dd 100644
--- a/lib/librte_eal/linux/eal/eal_memory.c
+++ b/lib/librte_eal/linux/eal/eal_memory.c
@@ -1340,6 +1340,8 @@ eal_legacy_hugepage_init(void)
 
 	/* hugetlbfs can be disabled */
 	if (internal_config.no_hugetlbfs) {
+		void *prealloc_addr;
+		size_t mem_sz;
 		struct rte_memseg_list *msl;
 		int n_segs, cur_seg, fd, flags;
 #ifdef MEMFD_SUPPORTED
@@ -1395,8 +1397,21 @@ eal_legacy_hugepage_init(void)
 			}
 		}
 #endif
-		addr = mmap(NULL, internal_config.memory, PROT_READ | PROT_WRITE,
-				flags, fd, 0);
+		/* preallocate address space for the memory, so that it can be
+		 * fit into the DMA mask.
+		 */
+		mem_sz = internal_config.memory;
+		prealloc_addr = eal_get_virtual_area(
+				NULL, &mem_sz, page_sz, 0, 0);
+		if (prealloc_addr == NULL) {
+			RTE_LOG(ERR, EAL,
+					"%s: reserving memory area failed: "
+					"%s\n",
+					__func__, strerror(errno));
+			return -1;
+		}
+		addr = mmap(prealloc_addr, internal_config.memory,
+				PROT_READ | PROT_WRITE, flags, fd, MAP_FIXED);
 		if (addr == MAP_FAILED) {
 			RTE_LOG(ERR, EAL, "%s: mmap() failed: %s\n", __func__,
 					strerror(errno));
-- 
2.17.1

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [dpdk-dev] [PATCH v2] eal/mem: preallocate VA space in no-huge mode
  2020-01-24 17:01 [dpdk-dev] [PATCH] eal/mem: preallocate VA space in no-huge mode Anatoly Burakov
@ 2020-01-24 17:05 ` Anatoly Burakov
  2020-02-06 15:39   ` Thomas Monjalon
  2020-02-07 11:11   ` [dpdk-dev] [PATCH v3] " Anatoly Burakov
  0 siblings, 2 replies; 10+ messages in thread
From: Anatoly Burakov @ 2020-01-24 17:05 UTC (permalink / raw)
  To: dev; +Cc: stable

When --no-huge mode is used, the memory is currently allocated with
mmap(NULL, ...). This is fine in most cases, but can fail in cases
where DPDK is run on a machine with an IOMMU that is of more limited
address width than that of a VA, because we're not specifying the
address hint for mmap() call.

Fix it by preallocating VA space before mapping it.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---

Notes:
    v2:
    - Add unmap on unsuccessful mmap
    
    I couldn't figure out which specific commit has introduced
    the issue, so there's no fix tag. The most likely candidate
    is one that introduced the DMA mask thing in the first place
    but i'm not sure.

 lib/librte_eal/linux/eal/eal_memory.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/lib/librte_eal/linux/eal/eal_memory.c b/lib/librte_eal/linux/eal/eal_memory.c
index 43e4ffc757..ce6326672f 100644
--- a/lib/librte_eal/linux/eal/eal_memory.c
+++ b/lib/librte_eal/linux/eal/eal_memory.c
@@ -1340,6 +1340,8 @@ eal_legacy_hugepage_init(void)
 
 	/* hugetlbfs can be disabled */
 	if (internal_config.no_hugetlbfs) {
+		void *prealloc_addr;
+		size_t mem_sz;
 		struct rte_memseg_list *msl;
 		int n_segs, cur_seg, fd, flags;
 #ifdef MEMFD_SUPPORTED
@@ -1395,11 +1397,25 @@ eal_legacy_hugepage_init(void)
 			}
 		}
 #endif
-		addr = mmap(NULL, internal_config.memory, PROT_READ | PROT_WRITE,
-				flags, fd, 0);
+		/* preallocate address space for the memory, so that it can be
+		 * fit into the DMA mask.
+		 */
+		mem_sz = internal_config.memory;
+		prealloc_addr = eal_get_virtual_area(
+				NULL, &mem_sz, page_sz, 0, 0);
+		if (prealloc_addr == NULL) {
+			RTE_LOG(ERR, EAL,
+					"%s: reserving memory area failed: "
+					"%s\n",
+					__func__, strerror(errno));
+			return -1;
+		}
+		addr = mmap(prealloc_addr, internal_config.memory,
+				PROT_READ | PROT_WRITE, flags, fd, MAP_FIXED);
 		if (addr == MAP_FAILED) {
 			RTE_LOG(ERR, EAL, "%s: mmap() failed: %s\n", __func__,
 					strerror(errno));
+			munmap(prealloc_addr, mem_sz);
 			return -1;
 		}
 		msl->base_va = addr;
-- 
2.17.1

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [PATCH v2] eal/mem: preallocate VA space in no-huge mode
  2020-01-24 17:05 ` [dpdk-dev] [PATCH v2] " Anatoly Burakov
@ 2020-02-06 15:39   ` Thomas Monjalon
  2020-02-06 21:07     ` Thomas Monjalon
  2020-02-07 11:11   ` [dpdk-dev] [PATCH v3] " Anatoly Burakov
  1 sibling, 1 reply; 10+ messages in thread
From: Thomas Monjalon @ 2020-02-06 15:39 UTC (permalink / raw)
  To: Anatoly Burakov; +Cc: dev, stable

24/01/2020 18:05, Anatoly Burakov:
> When --no-huge mode is used, the memory is currently allocated with
> mmap(NULL, ...). This is fine in most cases, but can fail in cases
> where DPDK is run on a machine with an IOMMU that is of more limited
> address width than that of a VA, because we're not specifying the
> address hint for mmap() call.
> 
> Fix it by preallocating VA space before mapping it.
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>

Applied, thanks




^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [PATCH v2] eal/mem: preallocate VA space in no-huge mode
  2020-02-06 15:39   ` Thomas Monjalon
@ 2020-02-06 21:07     ` Thomas Monjalon
  0 siblings, 0 replies; 10+ messages in thread
From: Thomas Monjalon @ 2020-02-06 21:07 UTC (permalink / raw)
  To: Anatoly Burakov; +Cc: dev, stable, david.marchand

06/02/2020 16:39, Thomas Monjalon:
> 24/01/2020 18:05, Anatoly Burakov:
> > When --no-huge mode is used, the memory is currently allocated with
> > mmap(NULL, ...). This is fine in most cases, but can fail in cases
> > where DPDK is run on a machine with an IOMMU that is of more limited
> > address width than that of a VA, because we're not specifying the
> > address hint for mmap() call.
> > 
> > Fix it by preallocating VA space before mapping it.
> > 
> > Cc: stable@dpdk.org
> > 
> > Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> 
> Applied, thanks

Eventually dropped from DPDK 20.02-rc2 because it is breaking no-huge mode.
Sorry



^ permalink raw reply	[flat|nested] 10+ messages in thread

* [dpdk-dev] [PATCH v3] eal/mem: preallocate VA space in no-huge mode
  2020-01-24 17:05 ` [dpdk-dev] [PATCH v2] " Anatoly Burakov
  2020-02-06 15:39   ` Thomas Monjalon
@ 2020-02-07 11:11   ` Anatoly Burakov
  2020-03-25 14:39     ` [dpdk-dev] [dpdk-stable] " David Marchand
  1 sibling, 1 reply; 10+ messages in thread
From: Anatoly Burakov @ 2020-02-07 11:11 UTC (permalink / raw)
  To: dev; +Cc: stable

When --no-huge mode is used, the memory is currently allocated with
mmap(NULL, ...). This is fine in most cases, but can fail in cases
where DPDK is run on a machine with an IOMMU that is of more limited
address width than that of a VA, because we're not specifying the
address hint for mmap() call.

Fix it by preallocating VA space before mapping it.

Cc: stable@dpdk.org

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---

Notes:
    v3:
    - Fix mmap flags used in place of offset
    - Fix using internal_config.memory in place of mem_sz
    - Add additional address sanity check
    
    v2:
    - Add unmap on unsuccessful mmap
    
    I couldn't figure out which specific commit has introduced
    the issue, so there's no fix tag. The most likely candidate
    is one that introduced the DMA mask thing in the first place
    but i'm not sure.

 lib/librte_eal/linux/eal/eal_memory.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/linux/eal/eal_memory.c b/lib/librte_eal/linux/eal/eal_memory.c
index 5604c2a7c0..7a9c97ff88 100644
--- a/lib/librte_eal/linux/eal/eal_memory.c
+++ b/lib/librte_eal/linux/eal/eal_memory.c
@@ -1340,6 +1340,8 @@ eal_legacy_hugepage_init(void)
 
 	/* hugetlbfs can be disabled */
 	if (internal_config.no_hugetlbfs) {
+		void *prealloc_addr;
+		size_t mem_sz;
 		struct rte_memseg_list *msl;
 		int n_segs, cur_seg, fd, flags;
 #ifdef MEMFD_SUPPORTED
@@ -1395,17 +1397,31 @@ eal_legacy_hugepage_init(void)
 			}
 		}
 #endif
-		addr = mmap(NULL, internal_config.memory, PROT_READ | PROT_WRITE,
-				flags, fd, 0);
-		if (addr == MAP_FAILED) {
+		/* preallocate address space for the memory, so that it can be
+		 * fit into the DMA mask.
+		 */
+		mem_sz = internal_config.memory;
+		prealloc_addr = eal_get_virtual_area(
+				NULL, &mem_sz, page_sz, 0, 0);
+		if (prealloc_addr == NULL) {
+			RTE_LOG(ERR, EAL,
+					"%s: reserving memory area failed: "
+					"%s\n",
+					__func__, strerror(errno));
+			return -1;
+		}
+		addr = mmap(prealloc_addr, mem_sz, PROT_READ | PROT_WRITE,
+				flags | MAP_FIXED, fd, 0);
+		if (addr == MAP_FAILED || addr != prealloc_addr) {
 			RTE_LOG(ERR, EAL, "%s: mmap() failed: %s\n", __func__,
 					strerror(errno));
+			munmap(prealloc_addr, mem_sz);
 			return -1;
 		}
 		msl->base_va = addr;
 		msl->page_sz = page_sz;
 		msl->socket_id = 0;
-		msl->len = internal_config.memory;
+		msl->len = mem_sz;
 		msl->heap = 1;
 
 		/* we're in single-file segments mode, so only the segment list
-- 
2.17.1

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [dpdk-stable] [PATCH v3] eal/mem: preallocate VA space in no-huge mode
  2020-02-07 11:11   ` [dpdk-dev] [PATCH v3] " Anatoly Burakov
@ 2020-03-25 14:39     ` David Marchand
  2020-03-26 17:06       ` Burakov, Anatoly
                         ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: David Marchand @ 2020-03-25 14:39 UTC (permalink / raw)
  To: Anatoly Burakov; +Cc: dev, dpdk stable

On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov
<anatoly.burakov@intel.com> wrote:
>
> When --no-huge mode is used, the memory is currently allocated with
> mmap(NULL, ...). This is fine in most cases, but can fail in cases
> where DPDK is run on a machine with an IOMMU that is of more limited
> address width than that of a VA, because we're not specifying the
> address hint for mmap() call.
>
> Fix it by preallocating VA space before mapping it.
>
> Cc: stable@dpdk.org
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>

Reproduced issue reported by Thomas on v2.
Works fine with v3.

Tested-by: David Marchand <david.marchand@redhat.com>


Does this issue affect FreeBSD too?

-- 
David Marchand


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [dpdk-stable] [PATCH v3] eal/mem: preallocate VA space in no-huge mode
  2020-03-25 14:39     ` [dpdk-dev] [dpdk-stable] " David Marchand
@ 2020-03-26 17:06       ` Burakov, Anatoly
  2020-03-27  6:33         ` David Marchand
       [not found]       ` <6473a4fe45d8437285fd8a1e931a1cb8@intel.com>
  2020-03-27 10:16       ` David Marchand
  2 siblings, 1 reply; 10+ messages in thread
From: Burakov, Anatoly @ 2020-03-26 17:06 UTC (permalink / raw)
  To: David Marchand; +Cc: dev, dpdk stable

On 25-Mar-20 2:39 PM, David Marchand wrote:
> On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov
> <anatoly.burakov@intel.com> wrote:
>>
>> When --no-huge mode is used, the memory is currently allocated with
>> mmap(NULL, ...). This is fine in most cases, but can fail in cases
>> where DPDK is run on a machine with an IOMMU that is of more limited
>> address width than that of a VA, because we're not specifying the
>> address hint for mmap() call.
>>
>> Fix it by preallocating VA space before mapping it.
>>
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> 
> Reproduced issue reported by Thomas on v2.
> Works fine with v3.
> 
> Tested-by: David Marchand <david.marchand@redhat.com>
> 
> 
> Does this issue affect FreeBSD too?
> 

I don't think we have support for IOMMU on FreeBSD so my guess is no :)

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [dpdk-stable] [PATCH v3] eal/mem: preallocate VA space in no-huge mode
       [not found]       ` <6473a4fe45d8437285fd8a1e931a1cb8@intel.com>
@ 2020-03-27  2:23         ` Zhou, JunX W
  0 siblings, 0 replies; 10+ messages in thread
From: Zhou, JunX W @ 2020-03-27  2:23 UTC (permalink / raw)
  To: David Marchand, Burakov, Anatoly; +Cc: dev, dpdk stable

Tested-by: Zhou, JunX W <junx.w.zhou@intel.com>
-----Original Message-----
From: Jiang, YuX 
Sent: Thursday, March 26, 2020 8:24 PM
To: David Marchand <david.marchand@redhat.com>; Burakov, Anatoly <anatoly.burakov@intel.com>
Cc: dev <dev@dpdk.org>; dpdk stable <stable@dpdk.org>; Zhou, JunX W <junx.w.zhou@intel.com>
Subject: RE: [dpdk-dev] [dpdk-stable] [PATCH v3] eal/mem: preallocate VA space in no-huge mode

+  Zhou, JunX W <junx.w.zhou@intel.com>

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of David Marchand
Sent: Wednesday, March 25, 2020 10:40 PM
To: Burakov, Anatoly <anatoly.burakov@intel.com>
Cc: dev <dev@dpdk.org>; dpdk stable <stable@dpdk.org>
Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH v3] eal/mem: preallocate VA space in no-huge mode

On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov <anatoly.burakov@intel.com> wrote:
>
> When --no-huge mode is used, the memory is currently allocated with 
> mmap(NULL, ...). This is fine in most cases, but can fail in cases 
> where DPDK is run on a machine with an IOMMU that is of more limited 
> address width than that of a VA, because we're not specifying the 
> address hint for mmap() call.
>
> Fix it by preallocating VA space before mapping it.
>
> Cc: stable@dpdk.org
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>

Reproduced issue reported by Thomas on v2.
Works fine with v3.

Tested-by: David Marchand <david.marchand@redhat.com>


Does this issue affect FreeBSD too?

--
David Marchand


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [dpdk-stable] [PATCH v3] eal/mem: preallocate VA space in no-huge mode
  2020-03-26 17:06       ` Burakov, Anatoly
@ 2020-03-27  6:33         ` David Marchand
  0 siblings, 0 replies; 10+ messages in thread
From: David Marchand @ 2020-03-27  6:33 UTC (permalink / raw)
  To: Burakov, Anatoly; +Cc: dev, dpdk stable

On Thu, Mar 26, 2020 at 6:07 PM Burakov, Anatoly
<anatoly.burakov@intel.com> wrote:
>
> On 25-Mar-20 2:39 PM, David Marchand wrote:
> > On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov
> > <anatoly.burakov@intel.com> wrote:
> >>
> >> When --no-huge mode is used, the memory is currently allocated with
> >> mmap(NULL, ...). This is fine in most cases, but can fail in cases
> >> where DPDK is run on a machine with an IOMMU that is of more limited
> >> address width than that of a VA, because we're not specifying the
> >> address hint for mmap() call.
> >>
> >> Fix it by preallocating VA space before mapping it.
> >>
> >> Cc: stable@dpdk.org
> >>
> >> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> >
> > Reproduced issue reported by Thomas on v2.
> > Works fine with v3.
> >
> > Tested-by: David Marchand <david.marchand@redhat.com>
> >
> >
> > Does this issue affect FreeBSD too?
> >
>
> I don't think we have support for IOMMU on FreeBSD so my guess is no :)

Fair enough, I will take it today.
Thanks.


--
David Marchand


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dpdk-dev] [dpdk-stable] [PATCH v3] eal/mem: preallocate VA space in no-huge mode
  2020-03-25 14:39     ` [dpdk-dev] [dpdk-stable] " David Marchand
  2020-03-26 17:06       ` Burakov, Anatoly
       [not found]       ` <6473a4fe45d8437285fd8a1e931a1cb8@intel.com>
@ 2020-03-27 10:16       ` David Marchand
  2 siblings, 0 replies; 10+ messages in thread
From: David Marchand @ 2020-03-27 10:16 UTC (permalink / raw)
  To: Anatoly Burakov; +Cc: dev, dpdk stable, junx.w.zhou

On Wed, Mar 25, 2020 at 3:39 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov
> <anatoly.burakov@intel.com> wrote:
> >
> > When --no-huge mode is used, the memory is currently allocated with
> > mmap(NULL, ...). This is fine in most cases, but can fail in cases
> > where DPDK is run on a machine with an IOMMU that is of more limited
> > address width than that of a VA, because we're not specifying the
> > address hint for mmap() call.
> >
> > Fix it by preallocating VA space before mapping it.
> >
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: David Marchand <david.marchand@redhat.com>
Tested-by: Jun W Zhou <junx.w.zhou@intel.com>

Applied, thanks.


-- 
David Marchand


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2020-03-27 10:16 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-24 17:01 [dpdk-dev] [PATCH] eal/mem: preallocate VA space in no-huge mode Anatoly Burakov
2020-01-24 17:05 ` [dpdk-dev] [PATCH v2] " Anatoly Burakov
2020-02-06 15:39   ` Thomas Monjalon
2020-02-06 21:07     ` Thomas Monjalon
2020-02-07 11:11   ` [dpdk-dev] [PATCH v3] " Anatoly Burakov
2020-03-25 14:39     ` [dpdk-dev] [dpdk-stable] " David Marchand
2020-03-26 17:06       ` Burakov, Anatoly
2020-03-27  6:33         ` David Marchand
     [not found]       ` <6473a4fe45d8437285fd8a1e931a1cb8@intel.com>
2020-03-27  2:23         ` Zhou, JunX W
2020-03-27 10:16       ` David Marchand

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).