[dpdk-stable] [PATH 0/6] Use IOVAs check based on DMA mask

[dpdk-stable] [PATH 0/6] Use IOVAs check based on DMA mask

[Alejandro Lucero]

This patchset adds, mainly, a check for ensuring IOVAs are within a
restricted range due to addressing limitations with some devices. There
are two known cases: NFP and IOMMU VT-d emulation.

With this check IOVAs out of range are detected and PMDs can abort
initialization. For the VT-d case, IOVA VA mode is allowed as long as
IOVAs are within the supported range, avoiding to forbid IOVA VA by
default.

For the addressing limitations known cases, there are just 40(NFP) or
39(VT-d) bits for handling IOVAs. When using IOVA PA, those limitations
imply 1TB(NFP) or 512M(VT-d) as upper limits, which is likely enough for
most systems. With machines using more memory, the added check will
ensure IOVAs within the range.

With IOVA VA, and because the way the Linux kernel serves mmap calls
in 64 bits systems, 39 or 40 bits are not enough. It is possible to
give an address hint with a lower starting address than the default one
used by the kernel, and then ensuring the mmap uses that hint or hint plus
some offset. With 64 bits systems, the process virtual address space is
large enoguh for doing the hugepages mmaping within the supported range
when those addressing limitations exist. This patchset also adds a change
for using such a hint making the use of IOVA VA a more than likely
possibility when there are those addressing limitations.

The check is not done by default but just when it is required. This
patchset adds the check for NFP initialization and for setting the IOVA
mode is an emulated VT-d is detected.

This patchset applies on 17.11.3.

Similar changes will be submitted to main DPDK branch soon.

[dpdk-stable] [PATCH 1/6] mem: add function for checking memsegs IOVAs addresses

[Alejandro Lucero]

A device can suffer addressing limitations. This functions checks
memsegs have iovas within the supported range based on dma mask.

PMD should use this during initialization if supported devices
suffer addressing limitations, returning an error if this function
returns memsegs out of range.

Another potential usage is for emulated IOMMU hardware with addressing
limitations.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 lib/librte_eal/common/eal_common_memory.c  | 37 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/include/rte_memory.h |  3 +++
 2 files changed, 40 insertions(+)

diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/common/eal_common_memory.c
index fc6c44d..ca49e5c 100644
--- a/lib/librte_eal/common/eal_common_memory.c
+++ b/lib/librte_eal/common/eal_common_memory.c
@@ -109,6 +109,43 @@
 	}
 }
 
+/* check memseg iovas are within the required range based on dma mask */
+int
+rte_eal_check_dma_mask(uint8_t maskbits)
+{
+
+	const struct rte_mem_config *mcfg;
+	uint64_t mask;
+	int i;
+	int ret = 0;
+
+	/* create dma mask */
+	mask = 1ULL << maskbits;
+	mask -= 1;
+
+	/* get pointer to global configuration */
+	mcfg = rte_eal_get_configuration()->mem_config;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++) {
+		if (mcfg->memseg[i].addr == NULL)
+			break;
+
+		if (mcfg->memseg[i].iova & ~mask) {
+			ret = -1;
+			break;
+		}
+	}
+
+	if (!ret)
+		return ret;
+
+	RTE_LOG(INFO, EAL, "memseg[%d] iova %"PRIx64" out of range:\n",
+			   i, mcfg->memseg[i].iova);
+	RTE_LOG(INFO, EAL, "\tusing dma mask %"PRIx64"\n", mask);
+
+	return -1;
+}
+
 /* return the number of memory channels */
 unsigned rte_memory_get_nchannel(void)
 {
diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/common/include/rte_memory.h
index 80a8fc0..b2a0168 100644
--- a/lib/librte_eal/common/include/rte_memory.h
+++ b/lib/librte_eal/common/include/rte_memory.h
@@ -209,6 +209,9 @@ struct rte_memseg {
  */
 unsigned rte_memory_get_nrank(void);
 
+/* check memsegs iovas are within a range based on dma mask */
+int rte_eal_check_dma_mask(uint8_t maskbits);
+
 /**
  * Drivers based on uio will not load unless physical
  * addresses are obtainable. It is only possible to get
-- 
1.9.1

[dpdk-stable] [PATCH 2/6] ethdev: add function for checking IOVAs by a device

[Alejandro Lucero]

A PMD should invoke this function for checking memsegs iovas are within
the supported range by the device.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 lib/librte_ether/rte_ethdev.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index eba11ca..e51a432 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -2799,6 +2799,19 @@ int rte_eth_dev_set_vlan_ether_type(uint16_t port_id,
 int rte_eth_dev_set_vlan_pvid(uint16_t port_id, uint16_t pvid, int on);
 
 /**
+ * check device dma mask within expected range based on dma mask.
+ *
+ * @param maskbits
+ *  mask length in bits
+ *
+ */
+static inline int
+rte_eth_dev_check_dma_mask(uint8_t maskbits)
+{
+	return rte_eal_check_dma_mask(maskbits);
+}
+
+/**
  *
  * Retrieve a burst of input packets from a receive queue of an Ethernet
  * device. The retrieved packets are stored in *rte_mbuf* structures whose
-- 
1.9.1

[dpdk-stable] [PATCH 3/6] bus/pci: use IOVAs check when setting IOVA mode

[Alejandro Lucero]

Although VT-d emulation currently only supports 39 bits, it could
be iovas being within that supported range. This patch allows
IOVA mode in such a case.

Indeed, memory initialization code can be modified for using lower
virtual addresses than those used by the kernel for 64 bits processes
by default, and therefore memsegs iovas can use 39 bits or less for
most system. And this is likely 100% true for VMs.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/bus/pci/linux/pci.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 74deef3..792c819 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -43,6 +43,7 @@
 #include <rte_devargs.h>
 #include <rte_memcpy.h>
 #include <rte_vfio.h>
+#include <rte_memory.h>
 
 #include "eal_private.h"
 #include "eal_filesystem.h"
@@ -613,10 +614,12 @@
 	fclose(fp);
 
 	mgaw = ((vtd_cap_reg & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT) + 1;
-	if (mgaw < X86_VA_WIDTH)
+
+	if (!rte_eal_check_dma_mask(mgaw))
+		return true;
+	else
 		return false;
 
-	return true;
 }
 #elif defined(RTE_ARCH_PPC_64)
 static bool
@@ -640,13 +643,17 @@
 {
 	struct rte_pci_device *dev = NULL;
 	struct rte_pci_driver *drv = NULL;
+	int iommu_dma_mask_check_done = 0;
 
 	FOREACH_DRIVER_ON_PCIBUS(drv) {
 		FOREACH_DEVICE_ON_PCIBUS(dev) {
 			if (!rte_pci_match(drv, dev))
 				continue;
-			if (!pci_one_device_iommu_support_va(dev))
-				return false;
+			if (!iommu_dma_mask_check_done) {
+				if (pci_one_device_iommu_support_va(dev) < 0)
+					return false;
+				iommu_dma_mask_check_done  = 1;
+			}
 		}
 	}
 	return true;
-- 
1.9.1

[dpdk-stable] [PATCH 4/6] mem: use address hint for mapping hugepages

[Alejandro Lucero]

Linux kernel uses a really high address as starting address for
serving mmaps calls. If there exists addressing limitations and
IOVA mode is VA, this starting address is likely too high for
those devices. However, it is possible to use a lower address in
the process virtual address space as with 64 bits there is a lot
of available space.

This patch adds an address hint as starting address for 64 bits
systems.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 lib/librte_eal/linuxapp/eal/eal_memory.c | 50 ++++++++++++++++++++++++++------
 1 file changed, 41 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 17c20d4..0a134c4 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -88,6 +88,23 @@
 
 static uint64_t baseaddr_offset;
 
+#ifdef RTE_ARCH_64
+/*
+ * Linux kernel uses a really high address as starting address for serving
+ * mmaps calls. If there exists addressing limitations and IOVA mode is VA,
+ * this starting address is likely too high for those devices. However, it
+ * is possible to use a lower address in the process virtual address space
+ * as with 64 bits there is a lot of available space.
+ *
+ * Current known limitations are 39 or 40 bits. Setting the starting address
+ * at 4GB implies there are 508GB or 1020GB for mapping the available
+ * hugepages. This is likely enough for most systems, although a device with
+ * addressing limitations should call rte_dev_check_dma_mask for ensuring all
+ * memory is within supported range.
+ */
+static uint64_t baseaddr = 0x100000000;
+#endif
+
 static bool phys_addrs_available = true;
 
 #define RANDOMIZE_VA_SPACE_FILE "/proc/sys/kernel/randomize_va_space"
@@ -260,16 +277,10 @@
 static void *
 get_virtual_area(size_t *size, size_t hugepage_sz)
 {
-	void *addr;
+	void *addr, *addr_hint;
 	int fd;
 	long aligned_addr;
 
-	if (internal_config.base_virtaddr != 0) {
-		addr = (void*) (uintptr_t) (internal_config.base_virtaddr +
-				baseaddr_offset);
-	}
-	else addr = NULL;
-
 	RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
 
 	fd = open("/dev/zero", O_RDONLY);
@@ -278,7 +289,22 @@
 		return NULL;
 	}
 	do {
-		addr = mmap(addr,
+		if (internal_config.base_virtaddr != 0) {
+			addr_hint = (void *) (uintptr_t)
+				    (internal_config.base_virtaddr +
+				     baseaddr_offset);
+		}
+#ifdef RTE_ARCH_64
+		else {
+			addr_hint = (void *) (uintptr_t) (baseaddr +
+					baseaddr_offset);
+		}
+#else
+		else {
+			addr_hint = NULL;
+		}
+#endif
+		addr = mmap(addr_hint,
 				(*size) + hugepage_sz, PROT_READ,
 #ifdef RTE_ARCH_PPC_64
 				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
@@ -286,8 +312,14 @@
 				MAP_PRIVATE,
 #endif
 				fd, 0);
-		if (addr == MAP_FAILED)
+		if (addr == MAP_FAILED) {
+			/* map failed. Let's try with less memory */
 			*size -= hugepage_sz;
+		} else if (addr_hint && addr != addr_hint) {
+			/* map not using hint. Let's try with another offset */
+			addr = MAP_FAILED;
+			baseaddr_offset += 0x100000000;
+		}
 	} while (addr == MAP_FAILED && *size > 0);
 
 	if (addr == MAP_FAILED) {
-- 
1.9.1

[dpdk-stable] [PATCH 5/6] net/nfp: check hugepages IOVAs based on DMA mask

[Alejandro Lucero]

NFP devices can not handle DMA addresses requiring more than
40 bits. This patch uses rte_dev_check_dma_mask with 40 bits
and avoids device initialization if memory out of NFP range.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index d9cd047..5976f37 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2649,6 +2649,14 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 
 	pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 
+	/* NFP can not handle DMA addresses requiring more than 40 bits */
+	if (rte_eth_dev_check_dma_mask(40) < 0) {
+		RTE_LOG(INFO, PMD, "device %s can not be used:",
+				   pci_dev->device.name);
+		RTE_LOG(INFO, PMD, "\trestricted dma mask to 40 bits!\n");
+		return -ENODEV;
+	};
+
 	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
 	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
 		port = get_pf_port_number(eth_dev->data->name);
-- 
1.9.1

[dpdk-stable] [PATCH 6/6] net/nfp: support IOVA VA mode

[Alejandro Lucero]

NFP can handle IOVA as VA. It requires to check those IOVAs
being in the supported range what is done during initialization.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 5976f37..354dec3 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -3053,14 +3053,16 @@ static int eth_nfp_pci_remove(struct rte_pci_device *pci_dev)
 
 static struct rte_pci_driver rte_nfp_net_pf_pmd = {
 	.id_table = pci_id_nfp_pf_net_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
 	.probe = nfp_pf_pci_probe,
 	.remove = eth_nfp_pci_remove,
 };
 
 static struct rte_pci_driver rte_nfp_net_vf_pmd = {
 	.id_table = pci_id_nfp_vf_net_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
 	.probe = eth_nfp_pci_probe,
 	.remove = eth_nfp_pci_remove,
 };
-- 
1.9.1

Re: [dpdk-stable] [PATCH 1/6] mem: add function for checking memsegs IOVAs addresses

[Burakov, Anatoly]

On 02-Jul-18 6:26 PM, Alejandro Lucero wrote:
> A device can suffer addressing limitations. This functions checks
> memsegs have iovas within the supported range based on dma mask.
> 
> PMD should use this during initialization if supported devices
> suffer addressing limitations, returning an error if this function
> returns memsegs out of range.
> 
> Another potential usage is for emulated IOMMU hardware with addressing
> limitations.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> ---

<snip>

> +	const struct rte_mem_config *mcfg;
> +	uint64_t mask;
> +	int i;
> +	int ret = 0;
> +
> +	/* create dma mask */
> +	mask = 1ULL << maskbits;
> +	mask -= 1;

mask = ~((1ULL << maskbits) - 1);

? IMO this makes it much more clear what you're trying to do.

> +
> +	/* get pointer to global configuration */
> +	mcfg = rte_eal_get_configuration()->mem_config;
> +
> +	for (i = 0; i < RTE_MAX_MEMSEG; i++) {
> +		if (mcfg->memseg[i].addr == NULL)
> +			break;
> +
> +		if (mcfg->memseg[i].iova & ~mask) {
> +			ret = -1;
> +			break;
> +		}
> +	}
> +
> +	if (!ret)
> +		return ret;
> +
> +	RTE_LOG(INFO, EAL, "memseg[%d] iova %"PRIx64" out of range:\n",
> +			   i, mcfg->memseg[i].iova);
> +	RTE_LOG(INFO, EAL, "\tusing dma mask %"PRIx64"\n", mask);
> +
> +	return -1;

The control flow looks weird to me. You break if iova has any bits that 
are in the mask, then you display log messages and return -1. How about 
just logging error and returning -1, and simply returning 0 after the loop?

-- 
Thanks,
Anatoly

Re: [dpdk-stable] [PATCH 3/6] bus/pci: use IOVAs check when setting IOVA mode

[Burakov, Anatoly]

On 02-Jul-18 6:27 PM, Alejandro Lucero wrote:
> Although VT-d emulation currently only supports 39 bits, it could
> be iovas being within that supported range. This patch allows
> IOVA mode in such a case.
> 
> Indeed, memory initialization code can be modified for using lower
> virtual addresses than those used by the kernel for 64 bits processes
> by default, and therefore memsegs iovas can use 39 bits or less for
> most system. And this is likely 100% true for VMs.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> ---

General question - is this issue only applicable to PCI? Do other buses 
need this?

-- 
Thanks,
Anatoly

Re: [dpdk-stable] [PATCH 4/6] mem: use address hint for mapping hugepages

[Burakov, Anatoly]

On 02-Jul-18 6:27 PM, Alejandro Lucero wrote:
> Linux kernel uses a really high address as starting address for
> serving mmaps calls. If there exists addressing limitations and
> IOVA mode is VA, this starting address is likely too high for
> those devices. However, it is possible to use a lower address in
> the process virtual address space as with 64 bits there is a lot
> of available space.
> 
> This patch adds an address hint as starting address for 64 bits
> systems.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> ---

<snip>

>   	long aligned_addr;
>   
> -	if (internal_config.base_virtaddr != 0) {
> -		addr = (void*) (uintptr_t) (internal_config.base_virtaddr +
> -				baseaddr_offset);
> -	}
> -	else addr = NULL;
> -
>   	RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
>   
>   	fd = open("/dev/zero", O_RDONLY);
> @@ -278,7 +289,22 @@
>   		return NULL;
>   	}
>   	do {
> -		addr = mmap(addr,
> +		if (internal_config.base_virtaddr != 0) {
> +			addr_hint = (void *) (uintptr_t)
> +				    (internal_config.base_virtaddr +
> +				     baseaddr_offset);
> +		}
> +#ifdef RTE_ARCH_64
> +		else {
> +			addr_hint = (void *) (uintptr_t) (baseaddr +
> +					baseaddr_offset);
> +		}
> +#else
> +		else {
> +			addr_hint = NULL;
> +		}
> +#endif

If my understanding is correct, calculations are all done on static 
variables, only the result is assigned to addr_hint which is local. Can 
we move this into a function and put these #ifdefs there, while keeping 
this code clean?

> +		addr = mmap(addr_hint,
>   				(*size) + hugepage_sz, PROT_READ,
>   #ifdef RTE_ARCH_PPC_64
>   				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
> @@ -286,8 +312,14 @@
>   				MAP_PRIVATE,
>   #endif
>   				fd, 0);
> -		if (addr == MAP_FAILED)
> +		if (addr == MAP_FAILED) {
> +			/* map failed. Let's try with less memory */
>   			*size -= hugepage_sz;
> +		} else if (addr_hint && addr != addr_hint) {
> +			/* map not using hint. Let's try with another offset */

Comment is slightly misleading - "map not using hint" implies we are 
about to map something without using hint. Suggested rewording:

suggested address hint was not used, try with another offset

> +			addr = MAP_FAILED;
> +			baseaddr_offset += 0x100000000;
> +		}
>   	} while (addr == MAP_FAILED && *size > 0);
>   
>   	if (addr == MAP_FAILED) {
> 


-- 
Thanks,
Anatoly

Re: [dpdk-stable] [PATCH 1/6] mem: add function for checking memsegs IOVAs addresses

[Alejandro Lucero]

On Tue, Jul 3, 2018 at 10:07 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 02-Jul-18 6:26 PM, Alejandro Lucero wrote:
>
>> A device can suffer addressing limitations. This functions checks
>> memsegs have iovas within the supported range based on dma mask.
>>
>> PMD should use this during initialization if supported devices
>> suffer addressing limitations, returning an error if this function
>> returns memsegs out of range.
>>
>> Another potential usage is for emulated IOMMU hardware with addressing
>> limitations.
>>
>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>> ---
>>
>
> <snip>
>
> +       const struct rte_mem_config *mcfg;
>> +       uint64_t mask;
>> +       int i;
>> +       int ret = 0;
>> +
>> +       /* create dma mask */
>> +       mask = 1ULL << maskbits;
>> +       mask -= 1;
>>
>
> mask = ~((1ULL << maskbits) - 1);
>
> ? IMO this makes it much more clear what you're trying to do.
>
>
Sure. I will change it.


> +
>> +       /* get pointer to global configuration */
>> +       mcfg = rte_eal_get_configuration()->mem_config;
>> +
>> +       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>> +               if (mcfg->memseg[i].addr == NULL)
>> +                       break;
>> +
>> +               if (mcfg->memseg[i].iova & ~mask) {
>> +                       ret = -1;
>> +                       break;
>> +               }
>> +       }
>> +
>> +       if (!ret)
>> +               return ret;
>> +
>> +       RTE_LOG(INFO, EAL, "memseg[%d] iova %"PRIx64" out of range:\n",
>> +                          i, mcfg->memseg[i].iova);
>> +       RTE_LOG(INFO, EAL, "\tusing dma mask %"PRIx64"\n", mask);
>> +
>> +       return -1;
>>
>
> The control flow looks weird to me. You break if iova has any bits that
> are in the mask, then you display log messages and return -1. How about
> just logging error and returning -1, and simply returning 0 after the loop?
>
>
I agree. The truth is I did that initially, but the log lines were too long
with the indent. I will go back to the original.

Thanks


> --
> Thanks,
> Anatoly
>

Re: [dpdk-stable] [PATCH 3/6] bus/pci: use IOVAs check when setting IOVA mode

[Alejandro Lucero]

On Tue, Jul 3, 2018 at 10:10 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 02-Jul-18 6:27 PM, Alejandro Lucero wrote:
>
>> Although VT-d emulation currently only supports 39 bits, it could
>> be iovas being within that supported range. This patch allows
>> IOVA mode in such a case.
>>
>> Indeed, memory initialization code can be modified for using lower
>> virtual addresses than those used by the kernel for 64 bits processes
>> by default, and therefore memsegs iovas can use 39 bits or less for
>> most system. And this is likely 100% true for VMs.
>>
>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>> ---
>>
>
> General question - is this issue only applicable to PCI? Do other buses
> need this?
>
>
I think there could be other buses or devices with those limitations.
Ideally, we could do more things like just discarding those memsegs out of
range, but that would imply other changes.

IMHO, this is good enough and just if it turns out this is causing
problems, other solution should be implemented. But with most current
systems out there, I do not think this is a priority.


> --
> Thanks,
> Anatoly
>

Re: [dpdk-stable] [PATCH 4/6] mem: use address hint for mapping hugepages

[Alejandro Lucero]

On Tue, Jul 3, 2018 at 10:17 AM, Burakov, Anatoly <anatoly.burakov@intel.com
> wrote:

> On 02-Jul-18 6:27 PM, Alejandro Lucero wrote:
>
>> Linux kernel uses a really high address as starting address for
>> serving mmaps calls. If there exists addressing limitations and
>> IOVA mode is VA, this starting address is likely too high for
>> those devices. However, it is possible to use a lower address in
>> the process virtual address space as with 64 bits there is a lot
>> of available space.
>>
>> This patch adds an address hint as starting address for 64 bits
>> systems.
>>
>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>> ---
>>
>
> <snip>
>
>
>         long aligned_addr;
>>   -     if (internal_config.base_virtaddr != 0) {
>> -               addr = (void*) (uintptr_t) (internal_config.base_virtaddr
>> +
>> -                               baseaddr_offset);
>> -       }
>> -       else addr = NULL;
>> -
>>         RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
>>         fd = open("/dev/zero", O_RDONLY);
>> @@ -278,7 +289,22 @@
>>                 return NULL;
>>         }
>>         do {
>> -               addr = mmap(addr,
>> +               if (internal_config.base_virtaddr != 0) {
>> +                       addr_hint = (void *) (uintptr_t)
>> +                                   (internal_config.base_virtaddr +
>> +                                    baseaddr_offset);
>> +               }
>> +#ifdef RTE_ARCH_64
>> +               else {
>> +                       addr_hint = (void *) (uintptr_t) (baseaddr +
>> +                                       baseaddr_offset);
>> +               }
>> +#else
>> +               else {
>> +                       addr_hint = NULL;
>> +               }
>> +#endif
>>
>
> If my understanding is correct, calculations are all done on static
> variables, only the result is assigned to addr_hint which is local. Can we
> move this into a function and put these #ifdefs there, while keeping this
> code clean?


Ok.


>
>
> +               addr = mmap(addr_hint,
>>                                 (*size) + hugepage_sz, PROT_READ,
>>   #ifdef RTE_ARCH_PPC_64
>>                                 MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
>> @@ -286,8 +312,14 @@
>>                                 MAP_PRIVATE,
>>   #endif
>>                                 fd, 0);
>> -               if (addr == MAP_FAILED)
>> +               if (addr == MAP_FAILED) {
>> +                       /* map failed. Let's try with less memory */
>>                         *size -= hugepage_sz;
>> +               } else if (addr_hint && addr != addr_hint) {
>> +                       /* map not using hint. Let's try with another
>> offset */
>>
>
> Comment is slightly misleading - "map not using hint" implies we are about
> to map something without using hint. Suggested rewording:
>
> suggested address hint was not used, try with another offset


What about "hint was not used. Try with another offset" ?

By the way, I forgot to unmap the memory in this case. I will fix this in
next version.


>
>
> +                       addr = MAP_FAILED;
>> +                       baseaddr_offset += 0x100000000;
>> +               }
>>         } while (addr == MAP_FAILED && *size > 0);
>>         if (addr == MAP_FAILED) {
>>
>>
>
> --
> Thanks,
> Anatoly
>

Re: [dpdk-stable] [PATCH 4/6] mem: use address hint for mapping hugepages

[Burakov, Anatoly]

On 03-Jul-18 11:44 AM, Alejandro Lucero wrote:
> 
>         +               addr = mmap(addr_hint,
>                                          (*size) + hugepage_sz, PROT_READ,
>            #ifdef RTE_ARCH_PPC_64
>                                          MAP_PRIVATE | MAP_ANONYMOUS |
>         MAP_HUGETLB,
>         @@ -286,8 +312,14 @@
>                                          MAP_PRIVATE,
>            #endif
>                                          fd, 0);
>         -               if (addr == MAP_FAILED)
>         +               if (addr == MAP_FAILED) {
>         +                       /* map failed. Let's try with less memory */
>                                  *size -= hugepage_sz;
>         +               } else if (addr_hint && addr != addr_hint) {
>         +                       /* map not using hint. Let's try with
>         another offset */
> 
> 
>     Comment is slightly misleading - "map not using hint" implies we are
>     about to map something without using hint. Suggested rewording:
> 
>     suggested address hint was not used, try with another offset
> 
> 
> What about "hint was not used. Try with another offset" ?

Works for me :)

-- 
Thanks,
Anatoly