[dpdk-dev] [PATCH v3 0/6] use IOVAs check based on DMA mask

[dpdk-dev] [PATCH v3 0/6] use IOVAs check based on DMA mask

[Alejandro Lucero]

This patchset adds, mainly, a check for ensuring IOVAs are within a
restricted range due to addressing limitations with some devices. There
are two known cases: NFP and IOMMU VT-d emulation.

With this check IOVAs out of range are detected and PMDs can abort
initialization. For the VT-d case, IOVA VA mode is allowed as long as
IOVAs are within the supported range, avoiding to forbid IOVA VA by
default.

For the addressing limitations known cases, there are just 40(NFP) or
39(VT-d) bits for handling IOVAs. When using IOVA PA, those limitations
imply 1TB(NFP) or 512M(VT-d) as upper limits, which is likely enough for
most systems. With machines using more memory, the added check will
ensure IOVAs within the range.

With IOVA VA, and because the way the Linux kernel serves mmap calls
in 64 bits systems, 39 or 40 bits are not enough. It is possible to
give an address hint with a lower starting address than the default one
used by the kernel, and then ensuring the mmap uses that hint or hint plus
some offset. With 64 bits systems, the process virtual address space is
large enoguh for doing the hugepages mmaping within the supported range
when those addressing limitations exist. This patchset also adds a change
for using such a hint making the use of IOVA VA a more than likely
possibility when there are those addressing limitations.

The check is not done by default but just when it is required. This
patchset adds the check for NFP initialization and for setting the IOVA
mode is an emulated VT-d is detected.

This patchset applies on 17.11.3.

Similar changes will be submitted to main DPDK branch soon.

v2:
 - add get_addr_hint function
 - call munmap when hint given and not used by mmap
 - create dma mask in one step
 - refactor logs

v3:
 - add new API functions to map files

[dpdk-dev] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Alejandro Lucero]

A device can suffer addressing limitations. This functions checks
memsegs have iovas within the supported range based on dma mask.

PMD should use this during initialization if supported devices
suffer addressing limitations, returning an error if this function
returns memsegs out of range.

Another potential usage is for emulated IOMMU hardware with addressing
limitations.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/eal_common_memory.c  | 33 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/include/rte_memory.h |  3 +++
 lib/librte_eal/rte_eal_version.map         |  1 +
 3 files changed, 37 insertions(+)

diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/common/eal_common_memory.c
index fc6c44d..f5efebe 100644
--- a/lib/librte_eal/common/eal_common_memory.c
+++ b/lib/librte_eal/common/eal_common_memory.c
@@ -109,6 +109,39 @@
 	}
 }
 
+/* check memseg iovas are within the required range based on dma mask */
+int
+rte_eal_check_dma_mask(uint8_t maskbits)
+{
+
+	const struct rte_mem_config *mcfg;
+	uint64_t mask;
+	int i;
+
+	/* create dma mask */
+	mask = ~((1ULL << maskbits) - 1);
+
+	/* get pointer to global configuration */
+	mcfg = rte_eal_get_configuration()->mem_config;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++) {
+		if (mcfg->memseg[i].addr == NULL)
+			break;
+
+		if (mcfg->memseg[i].iova & mask) {
+			RTE_LOG(INFO, EAL,
+				"memseg[%d] iova %"PRIx64" out of range:\n",
+				i, mcfg->memseg[i].iova);
+
+			RTE_LOG(INFO, EAL, "\tusing dma mask %"PRIx64"\n",
+				mask);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
 /* return the number of memory channels */
 unsigned rte_memory_get_nchannel(void)
 {
diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/common/include/rte_memory.h
index 80a8fc0..b2a0168 100644
--- a/lib/librte_eal/common/include/rte_memory.h
+++ b/lib/librte_eal/common/include/rte_memory.h
@@ -209,6 +209,9 @@ struct rte_memseg {
  */
 unsigned rte_memory_get_nrank(void);
 
+/* check memsegs iovas are within a range based on dma mask */
+int rte_eal_check_dma_mask(uint8_t maskbits);
+
 /**
  * Drivers based on uio will not load unless physical
  * addresses are obtainable. It is only possible to get
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index f4f46c1..aa6cf87 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -184,6 +184,7 @@ DPDK_17.11 {
 
 	rte_eal_create_uio_dev;
 	rte_bus_get_iommu_class;
+	rte_eal_check_dma_mask;
 	rte_eal_has_pci;
 	rte_eal_iova_mode;
 	rte_eal_mbuf_default_mempool_ops;
-- 
1.9.1

[dpdk-dev] [PATCH v3 2/6] ethdev: add function for checking IOVAs by a device

[Alejandro Lucero]

A PMD should invoke this function for checking memsegs iovas are within
the supported range by the device.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 lib/librte_ether/rte_ethdev.h           | 13 +++++++++++++
 lib/librte_ether/rte_ethdev_version.map |  1 +
 2 files changed, 14 insertions(+)

diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
index eba11ca..e51a432 100644
--- a/lib/librte_ether/rte_ethdev.h
+++ b/lib/librte_ether/rte_ethdev.h
@@ -2799,6 +2799,19 @@ int rte_eth_dev_set_vlan_ether_type(uint16_t port_id,
 int rte_eth_dev_set_vlan_pvid(uint16_t port_id, uint16_t pvid, int on);
 
 /**
+ * check device dma mask within expected range based on dma mask.
+ *
+ * @param maskbits
+ *  mask length in bits
+ *
+ */
+static inline int
+rte_eth_dev_check_dma_mask(uint8_t maskbits)
+{
+	return rte_eal_check_dma_mask(maskbits);
+}
+
+/**
  *
  * Retrieve a burst of input packets from a receive queue of an Ethernet
  * device. The retrieved packets are stored in *rte_mbuf* structures whose
diff --git a/lib/librte_ether/rte_ethdev_version.map b/lib/librte_ether/rte_ethdev_version.map
index e9681ac..0b11b8a 100644
--- a/lib/librte_ether/rte_ethdev_version.map
+++ b/lib/librte_ether/rte_ethdev_version.map
@@ -191,6 +191,7 @@ DPDK_17.08 {
 DPDK_17.11 {
 	global:
 
+	rte_eth_dev_check_dma_mask;
 	rte_eth_dev_get_sec_ctx;
 	rte_eth_dev_pool_ops_supported;
 	rte_eth_dev_reset;
-- 
1.9.1

[dpdk-dev] [PATCH v3 3/6] bus/pci: use IOVAs check when setting IOVA mode

[Alejandro Lucero]

Although VT-d emulation currently only supports 39 bits, it could
be iovas being within that supported range. This patch allows
IOVA mode in such a case.

Indeed, memory initialization code can be modified for using lower
virtual addresses than those used by the kernel for 64 bits processes
by default, and therefore memsegs iovas can use 39 bits or less for
most system. And this is likely 100% true for VMs.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/bus/pci/linux/pci.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 74deef3..792c819 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -43,6 +43,7 @@
 #include <rte_devargs.h>
 #include <rte_memcpy.h>
 #include <rte_vfio.h>
+#include <rte_memory.h>
 
 #include "eal_private.h"
 #include "eal_filesystem.h"
@@ -613,10 +614,12 @@
 	fclose(fp);
 
 	mgaw = ((vtd_cap_reg & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT) + 1;
-	if (mgaw < X86_VA_WIDTH)
+
+	if (!rte_eal_check_dma_mask(mgaw))
+		return true;
+	else
 		return false;
 
-	return true;
 }
 #elif defined(RTE_ARCH_PPC_64)
 static bool
@@ -640,13 +643,17 @@
 {
 	struct rte_pci_device *dev = NULL;
 	struct rte_pci_driver *drv = NULL;
+	int iommu_dma_mask_check_done = 0;
 
 	FOREACH_DRIVER_ON_PCIBUS(drv) {
 		FOREACH_DEVICE_ON_PCIBUS(dev) {
 			if (!rte_pci_match(drv, dev))
 				continue;
-			if (!pci_one_device_iommu_support_va(dev))
-				return false;
+			if (!iommu_dma_mask_check_done) {
+				if (pci_one_device_iommu_support_va(dev) < 0)
+					return false;
+				iommu_dma_mask_check_done  = 1;
+			}
 		}
 	}
 	return true;
-- 
1.9.1

[dpdk-dev] [PATCH v3 4/6] mem: use address hint for mapping hugepages

[Alejandro Lucero]

Linux kernel uses a really high address as starting address for
serving mmaps calls. If there exists addressing limitations and
IOVA mode is VA, this starting address is likely too high for
those devices. However, it is possible to use a lower address in
the process virtual address space as with 64 bits there is a lot
of available space.

This patch adds an address hint as starting address for 64 bits
systems.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_memory.c | 55 ++++++++++++++++++++++++++------
 1 file changed, 46 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 17c20d4..2ed4017 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -88,6 +88,23 @@
 
 static uint64_t baseaddr_offset;
 
+#ifdef RTE_ARCH_64
+/*
+ * Linux kernel uses a really high address as starting address for serving
+ * mmaps calls. If there exists addressing limitations and IOVA mode is VA,
+ * this starting address is likely too high for those devices. However, it
+ * is possible to use a lower address in the process virtual address space
+ * as with 64 bits there is a lot of available space.
+ *
+ * Current known limitations are 39 or 40 bits. Setting the starting address
+ * at 4GB implies there are 508GB or 1020GB for mapping the available
+ * hugepages. This is likely enough for most systems, although a device with
+ * addressing limitations should call rte_dev_check_dma_mask for ensuring all
+ * memory is within supported range.
+ */
+static uint64_t baseaddr = 0x100000000;
+#endif
+
 static bool phys_addrs_available = true;
 
 #define RANDOMIZE_VA_SPACE_FILE "/proc/sys/kernel/randomize_va_space"
@@ -250,6 +267,23 @@
 	}
 }
 
+static void *
+get_addr_hint(void)
+{
+	if (internal_config.base_virtaddr != 0) {
+		return (void *) (uintptr_t)
+			    (internal_config.base_virtaddr +
+			     baseaddr_offset);
+	} else {
+#ifdef RTE_ARCH_64
+		return (void *) (uintptr_t) (baseaddr +
+				baseaddr_offset);
+#else
+		return NULL;
+#endif
+	}
+}
+
 /*
  * Try to mmap *size bytes in /dev/zero. If it is successful, return the
  * pointer to the mmap'd area and keep *size unmodified. Else, retry
@@ -260,16 +294,10 @@
 static void *
 get_virtual_area(size_t *size, size_t hugepage_sz)
 {
-	void *addr;
+	void *addr, *addr_hint;
 	int fd;
 	long aligned_addr;
 
-	if (internal_config.base_virtaddr != 0) {
-		addr = (void*) (uintptr_t) (internal_config.base_virtaddr +
-				baseaddr_offset);
-	}
-	else addr = NULL;
-
 	RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
 
 	fd = open("/dev/zero", O_RDONLY);
@@ -278,7 +306,9 @@
 		return NULL;
 	}
 	do {
-		addr = mmap(addr,
+		addr_hint = get_addr_hint();
+
+		addr = mmap(addr_hint,
 				(*size) + hugepage_sz, PROT_READ,
 #ifdef RTE_ARCH_PPC_64
 				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
@@ -286,8 +316,15 @@
 				MAP_PRIVATE,
 #endif
 				fd, 0);
-		if (addr == MAP_FAILED)
+		if (addr == MAP_FAILED) {
+			/* map failed. Let's try with less memory */
 			*size -= hugepage_sz;
+		} else if (addr_hint && addr != addr_hint) {
+			/* hint was not used. Try with another offset */
+			munmap(addr, (*size) + hugepage_sz);
+			addr = MAP_FAILED;
+			baseaddr_offset += 0x100000000;
+		}
 	} while (addr == MAP_FAILED && *size > 0);
 
 	if (addr == MAP_FAILED) {
-- 
1.9.1

[dpdk-dev] [PATCH v3 5/6] net/nfp: check hugepages IOVAs based on DMA mask

[Alejandro Lucero]

NFP devices can not handle DMA addresses requiring more than
40 bits. This patch uses rte_dev_check_dma_mask with 40 bits
and avoids device initialization if memory out of NFP range.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index d9cd047..5976f37 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2649,6 +2649,14 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 
 	pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 
+	/* NFP can not handle DMA addresses requiring more than 40 bits */
+	if (rte_eth_dev_check_dma_mask(40) < 0) {
+		RTE_LOG(INFO, PMD, "device %s can not be used:",
+				   pci_dev->device.name);
+		RTE_LOG(INFO, PMD, "\trestricted dma mask to 40 bits!\n");
+		return -ENODEV;
+	};
+
 	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
 	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
 		port = get_pf_port_number(eth_dev->data->name);
-- 
1.9.1

[dpdk-dev] [PATCH v3 6/6] net/nfp: support IOVA VA mode

[Alejandro Lucero]

NFP can handle IOVA as VA. It requires to check those IOVAs
being in the supported range what is done during initialization.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 5976f37..354dec3 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -3053,14 +3053,16 @@ static int eth_nfp_pci_remove(struct rte_pci_device *pci_dev)
 
 static struct rte_pci_driver rte_nfp_net_pf_pmd = {
 	.id_table = pci_id_nfp_pf_net_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
 	.probe = nfp_pf_pci_probe,
 	.remove = eth_nfp_pci_remove,
 };
 
 static struct rte_pci_driver rte_nfp_net_vf_pmd = {
 	.id_table = pci_id_nfp_vf_net_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
 	.probe = eth_nfp_pci_probe,
 	.remove = eth_nfp_pci_remove,
 };
-- 
1.9.1

Re: [dpdk-dev] [PATCH v3 2/6] ethdev: add function for checking IOVAs by a device

[Andrew Rybchenko]

On 04.07.2018 15:53, Alejandro Lucero wrote:
> A PMD should invoke this function for checking memsegs iovas are within
> the supported range by the device.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> ---
>   lib/librte_ether/rte_ethdev.h           | 13 +++++++++++++
>   lib/librte_ether/rte_ethdev_version.map |  1 +
>   2 files changed, 14 insertions(+)
>
> diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h
> index eba11ca..e51a432 100644
> --- a/lib/librte_ether/rte_ethdev.h
> +++ b/lib/librte_ether/rte_ethdev.h
> @@ -2799,6 +2799,19 @@ int rte_eth_dev_set_vlan_ether_type(uint16_t port_id,
>   int rte_eth_dev_set_vlan_pvid(uint16_t port_id, uint16_t pvid, int on);
>   
>   /**
> + * check device dma mask within expected range based on dma mask.
> + *
> + * @param maskbits
> + *  mask length in bits
> + *
> + */
> +static inline int
> +rte_eth_dev_check_dma_mask(uint8_t maskbits)
> +{
> +	return rte_eal_check_dma_mask(maskbits);

I'm afraid I don't understand why do we need the wrapper.
May PMD use EAL function directly?

> +}
> +
> +/**
>    *
>    * Retrieve a burst of input packets from a receive queue of an Ethernet
>    * device. The retrieved packets are stored in *rte_mbuf* structures whose
> diff --git a/lib/librte_ether/rte_ethdev_version.map b/lib/librte_ether/rte_ethdev_version.map
> index e9681ac..0b11b8a 100644
> --- a/lib/librte_ether/rte_ethdev_version.map
> +++ b/lib/librte_ether/rte_ethdev_version.map
> @@ -191,6 +191,7 @@ DPDK_17.08 {
>   DPDK_17.11 {
>   	global:
>   
> +	rte_eth_dev_check_dma_mask;
>   	rte_eth_dev_get_sec_ctx;
>   	rte_eth_dev_pool_ops_supported;
>   	rte_eth_dev_reset;

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Eelco Chaudron]

On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:

> A device can suffer addressing limitations. This functions checks
> memsegs have iovas within the supported range based on dma mask.
>
> PMD should use this during initialization if supported devices
> suffer addressing limitations, returning an error if this function
> returns memsegs out of range.
>
> Another potential usage is for emulated IOMMU hardware with addressing
> limitations.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
>  lib/librte_eal/common/eal_common_memory.c  | 33 
> ++++++++++++++++++++++++++++++
>  lib/librte_eal/common/include/rte_memory.h |  3 +++
>  lib/librte_eal/rte_eal_version.map         |  1 +
>  3 files changed, 37 insertions(+)
>
> diff --git a/lib/librte_eal/common/eal_common_memory.c 
> b/lib/librte_eal/common/eal_common_memory.c
> index fc6c44d..f5efebe 100644
> --- a/lib/librte_eal/common/eal_common_memory.c
> +++ b/lib/librte_eal/common/eal_common_memory.c
> @@ -109,6 +109,39 @@
>  	}
>  }
>
> +/* check memseg iovas are within the required range based on dma mask 
> */
> +int
> +rte_eal_check_dma_mask(uint8_t maskbits)
> +{
> +
> +	const struct rte_mem_config *mcfg;
> +	uint64_t mask;
> +	int i;
> +

I think we should add some sanity check to the input maskbits, i.e. 
[64,0) or [64, 32]? What would be a reasonable lower bound.

> +	/* create dma mask */
> +	mask = ~((1ULL << maskbits) - 1);
> +
> +	/* get pointer to global configuration */
> +	mcfg = rte_eal_get_configuration()->mem_config;
> +
> +	for (i = 0; i < RTE_MAX_MEMSEG; i++) {
> +		if (mcfg->memseg[i].addr == NULL)
> +			break;
> +
> +		if (mcfg->memseg[i].iova & mask) {
> +			RTE_LOG(INFO, EAL,
> +				"memseg[%d] iova %"PRIx64" out of range:\n",
> +				i, mcfg->memseg[i].iova);
> +
> +			RTE_LOG(INFO, EAL, "\tusing dma mask %"PRIx64"\n",
> +				mask);
> +			return -1;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
>  /* return the number of memory channels */
>  unsigned rte_memory_get_nchannel(void)
>  {
> diff --git a/lib/librte_eal/common/include/rte_memory.h 
> b/lib/librte_eal/common/include/rte_memory.h
> index 80a8fc0..b2a0168 100644
> --- a/lib/librte_eal/common/include/rte_memory.h
> +++ b/lib/librte_eal/common/include/rte_memory.h
> @@ -209,6 +209,9 @@ struct rte_memseg {
>   */
>  unsigned rte_memory_get_nrank(void);
>
> +/* check memsegs iovas are within a range based on dma mask */
> +int rte_eal_check_dma_mask(uint8_t maskbits);
> +
>  /**
>   * Drivers based on uio will not load unless physical
>   * addresses are obtainable. It is only possible to get
> diff --git a/lib/librte_eal/rte_eal_version.map 
> b/lib/librte_eal/rte_eal_version.map
> index f4f46c1..aa6cf87 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -184,6 +184,7 @@ DPDK_17.11 {
>
>  	rte_eal_create_uio_dev;
>  	rte_bus_get_iommu_class;
> +	rte_eal_check_dma_mask;
>  	rte_eal_has_pci;
>  	rte_eal_iova_mode;
>  	rte_eal_mbuf_default_mempool_ops;
> -- 
> 1.9.1

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 2/6] ethdev: add function for checking IOVAs by a device

[Eelco Chaudron]

On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:

> A PMD should invoke this function for checking memsegs iovas are 
> within
> the supported range by the device.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

Agree with Andrew here, why not call rte_eal_check_dma_mask() directly 
in nfp_net_txq_full()?

> ---
>  lib/librte_ether/rte_ethdev.h           | 13 +++++++++++++
>  lib/librte_ether/rte_ethdev_version.map |  1 +
>  2 files changed, 14 insertions(+)
>
> diff --git a/lib/librte_ether/rte_ethdev.h 
> b/lib/librte_ether/rte_ethdev.h
> index eba11ca..e51a432 100644
> --- a/lib/librte_ether/rte_ethdev.h
> +++ b/lib/librte_ether/rte_ethdev.h
> @@ -2799,6 +2799,19 @@ int rte_eth_dev_set_vlan_ether_type(uint16_t 
> port_id,
>  int rte_eth_dev_set_vlan_pvid(uint16_t port_id, uint16_t pvid, int 
> on);
>
>  /**
> + * check device dma mask within expected range based on dma mask.
> + *
> + * @param maskbits
> + *  mask length in bits
> + *
> + */
> +static inline int
> +rte_eth_dev_check_dma_mask(uint8_t maskbits)
> +{
> +	return rte_eal_check_dma_mask(maskbits);
> +}
> +
> +/**
>   *
>   * Retrieve a burst of input packets from a receive queue of an 
> Ethernet
>   * device. The retrieved packets are stored in *rte_mbuf* structures 
> whose
> diff --git a/lib/librte_ether/rte_ethdev_version.map 
> b/lib/librte_ether/rte_ethdev_version.map
> index e9681ac..0b11b8a 100644
> --- a/lib/librte_ether/rte_ethdev_version.map
> +++ b/lib/librte_ether/rte_ethdev_version.map
> @@ -191,6 +191,7 @@ DPDK_17.08 {
>  DPDK_17.11 {
>  	global:
>
> +	rte_eth_dev_check_dma_mask;
>  	rte_eth_dev_get_sec_ctx;
>  	rte_eth_dev_pool_ops_supported;
>  	rte_eth_dev_reset;
> -- 
> 1.9.1

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Alejandro Lucero]

On Tue, Jul 10, 2018 at 9:56 AM, Eelco Chaudron <echaudro@redhat.com> wrote:

>
>
> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>
> A device can suffer addressing limitations. This functions checks
>> memsegs have iovas within the supported range based on dma mask.
>>
>> PMD should use this during initialization if supported devices
>> suffer addressing limitations, returning an error if this function
>> returns memsegs out of range.
>>
>> Another potential usage is for emulated IOMMU hardware with addressing
>> limitations.
>>
>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>> ---
>>  lib/librte_eal/common/eal_common_memory.c  | 33
>> ++++++++++++++++++++++++++++++
>>  lib/librte_eal/common/include/rte_memory.h |  3 +++
>>  lib/librte_eal/rte_eal_version.map         |  1 +
>>  3 files changed, 37 insertions(+)
>>
>> diff --git a/lib/librte_eal/common/eal_common_memory.c
>> b/lib/librte_eal/common/eal_common_memory.c
>> index fc6c44d..f5efebe 100644
>> --- a/lib/librte_eal/common/eal_common_memory.c
>> +++ b/lib/librte_eal/common/eal_common_memory.c
>> @@ -109,6 +109,39 @@
>>         }
>>  }
>>
>> +/* check memseg iovas are within the required range based on dma mask */
>> +int
>> +rte_eal_check_dma_mask(uint8_t maskbits)
>> +{
>> +
>> +       const struct rte_mem_config *mcfg;
>> +       uint64_t mask;
>> +       int i;
>> +
>>
>
> I think we should add some sanity check to the input maskbits, i.e. [64,0)
> or [64, 32]? What would be a reasonable lower bound.
>
>
This is not a user's API, so any invocation will be reviewed, but I guess
adding a sanity check here does not harm.

Not sure about lower bound but upper should 64, although it does not make
sense but it is safe. Lower bound is not so problematic.


>
> +       /* create dma mask */
>> +       mask = ~((1ULL << maskbits) - 1);
>> +
>> +       /* get pointer to global configuration */
>> +       mcfg = rte_eal_get_configuration()->mem_config;
>> +
>> +       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>> +               if (mcfg->memseg[i].addr == NULL)
>> +                       break;
>> +
>> +               if (mcfg->memseg[i].iova & mask) {
>> +                       RTE_LOG(INFO, EAL,
>> +                               "memseg[%d] iova %"PRIx64" out of
>> range:\n",
>> +                               i, mcfg->memseg[i].iova);
>> +
>> +                       RTE_LOG(INFO, EAL, "\tusing dma mask %"PRIx64"\n",
>> +                               mask);
>> +                       return -1;
>> +               }
>> +       }
>> +
>> +       return 0;
>> +}
>> +
>>  /* return the number of memory channels */
>>  unsigned rte_memory_get_nchannel(void)
>>  {
>> diff --git a/lib/librte_eal/common/include/rte_memory.h
>> b/lib/librte_eal/common/include/rte_memory.h
>> index 80a8fc0..b2a0168 100644
>> --- a/lib/librte_eal/common/include/rte_memory.h
>> +++ b/lib/librte_eal/common/include/rte_memory.h
>> @@ -209,6 +209,9 @@ struct rte_memseg {
>>   */
>>  unsigned rte_memory_get_nrank(void);
>>
>> +/* check memsegs iovas are within a range based on dma mask */
>> +int rte_eal_check_dma_mask(uint8_t maskbits);
>> +
>>  /**
>>   * Drivers based on uio will not load unless physical
>>   * addresses are obtainable. It is only possible to get
>> diff --git a/lib/librte_eal/rte_eal_version.map
>> b/lib/librte_eal/rte_eal_version.map
>> index f4f46c1..aa6cf87 100644
>> --- a/lib/librte_eal/rte_eal_version.map
>> +++ b/lib/librte_eal/rte_eal_version.map
>> @@ -184,6 +184,7 @@ DPDK_17.11 {
>>
>>         rte_eal_create_uio_dev;
>>         rte_bus_get_iommu_class;
>> +       rte_eal_check_dma_mask;
>>         rte_eal_has_pci;
>>         rte_eal_iova_mode;
>>         rte_eal_mbuf_default_mempool_ops;
>> --
>> 1.9.1
>>
>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 2/6] ethdev: add function for checking IOVAs by a device

[Alejandro Lucero]

On Tue, Jul 10, 2018 at 9:57 AM, Eelco Chaudron <echaudro@redhat.com> wrote:

>
>
> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>
> A PMD should invoke this function for checking memsegs iovas are within
>> the supported range by the device.
>>
>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>>
>
> Agree with Andrew here, why not call rte_eal_check_dma_mask() directly in
> nfp_net_txq_full()?
>
>
My idea was to add this indirection for handling dma mask when just part of
the IOVAs are not usable. Now, if the dma mask finds a problem, the PMD
does not make any port initialization.

Memory management is changing and ideally an app should just allocate
memory safe to be used by the PMD when that memory is going to be used for
sending or receiving data, what is not always the case.

It is true this indirection is not being used for any purpose by now, so
yes, I could use a direct call the the EAL one.


>
> ---
>>  lib/librte_ether/rte_ethdev.h           | 13 +++++++++++++
>>  lib/librte_ether/rte_ethdev_version.map |  1 +
>>  2 files changed, 14 insertions(+)
>>
>> diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.
>> h
>> index eba11ca..e51a432 100644
>> --- a/lib/librte_ether/rte_ethdev.h
>> +++ b/lib/librte_ether/rte_ethdev.h
>> @@ -2799,6 +2799,19 @@ int rte_eth_dev_set_vlan_ether_type(uint16_t
>> port_id,
>>  int rte_eth_dev_set_vlan_pvid(uint16_t port_id, uint16_t pvid, int on);
>>
>>  /**
>> + * check device dma mask within expected range based on dma mask.
>> + *
>> + * @param maskbits
>> + *  mask length in bits
>> + *
>> + */
>> +static inline int
>> +rte_eth_dev_check_dma_mask(uint8_t maskbits)
>> +{
>> +       return rte_eal_check_dma_mask(maskbits);
>> +}
>> +
>> +/**
>>   *
>>   * Retrieve a burst of input packets from a receive queue of an Ethernet
>>   * device. The retrieved packets are stored in *rte_mbuf* structures
>> whose
>> diff --git a/lib/librte_ether/rte_ethdev_version.map
>> b/lib/librte_ether/rte_ethdev_version.map
>> index e9681ac..0b11b8a 100644
>> --- a/lib/librte_ether/rte_ethdev_version.map
>> +++ b/lib/librte_ether/rte_ethdev_version.map
>> @@ -191,6 +191,7 @@ DPDK_17.08 {
>>  DPDK_17.11 {
>>         global:
>>
>> +       rte_eth_dev_check_dma_mask;
>>         rte_eth_dev_get_sec_ctx;
>>         rte_eth_dev_pool_ops_supported;
>>         rte_eth_dev_reset;
>> --
>> 1.9.1
>>
>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 2/6] ethdev: add function for checking IOVAs by a device

[Alejandro Lucero]

On Tue, Jul 10, 2018 at 10:42 AM, Alejandro Lucero <
alejandro.lucero@netronome.com> wrote:

>
>
> On Tue, Jul 10, 2018 at 9:57 AM, Eelco Chaudron <echaudro@redhat.com>
> wrote:
>
>>
>>
>> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>>
>> A PMD should invoke this function for checking memsegs iovas are within
>>> the supported range by the device.
>>>
>>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>>>
>>
>> Agree with Andrew here, why not call rte_eal_check_dma_mask() directly in
>> nfp_net_txq_full()?
>>
>>
BTW, IOVA checking can not be done inside PMD functions like that one
because that is in the fast path. Any IOVA checking needs to be done at
initialization time.


>
> My idea was to add this indirection for handling dma mask when just part
> of the IOVAs are not usable. Now, if the dma mask finds a problem, the PMD
> does not make any port initialization.
>
> Memory management is changing and ideally an app should just allocate
> memory safe to be used by the PMD when that memory is going to be used for
> sending or receiving data, what is not always the case.
>
> It is true this indirection is not being used for any purpose by now, so
> yes, I could use a direct call the the EAL one.
>
>
>>
>> ---
>>>  lib/librte_ether/rte_ethdev.h           | 13 +++++++++++++
>>>  lib/librte_ether/rte_ethdev_version.map |  1 +
>>>  2 files changed, 14 insertions(+)
>>>
>>> diff --git a/lib/librte_ether/rte_ethdev.h
>>> b/lib/librte_ether/rte_ethdev.h
>>> index eba11ca..e51a432 100644
>>> --- a/lib/librte_ether/rte_ethdev.h
>>> +++ b/lib/librte_ether/rte_ethdev.h
>>> @@ -2799,6 +2799,19 @@ int rte_eth_dev_set_vlan_ether_type(uint16_t
>>> port_id,
>>>  int rte_eth_dev_set_vlan_pvid(uint16_t port_id, uint16_t pvid, int on);
>>>
>>>  /**
>>> + * check device dma mask within expected range based on dma mask.
>>> + *
>>> + * @param maskbits
>>> + *  mask length in bits
>>> + *
>>> + */
>>> +static inline int
>>> +rte_eth_dev_check_dma_mask(uint8_t maskbits)
>>> +{
>>> +       return rte_eal_check_dma_mask(maskbits);
>>> +}
>>> +
>>> +/**
>>>   *
>>>   * Retrieve a burst of input packets from a receive queue of an Ethernet
>>>   * device. The retrieved packets are stored in *rte_mbuf* structures
>>> whose
>>> diff --git a/lib/librte_ether/rte_ethdev_version.map
>>> b/lib/librte_ether/rte_ethdev_version.map
>>> index e9681ac..0b11b8a 100644
>>> --- a/lib/librte_ether/rte_ethdev_version.map
>>> +++ b/lib/librte_ether/rte_ethdev_version.map
>>> @@ -191,6 +191,7 @@ DPDK_17.08 {
>>>  DPDK_17.11 {
>>>         global:
>>>
>>> +       rte_eth_dev_check_dma_mask;
>>>         rte_eth_dev_get_sec_ctx;
>>>         rte_eth_dev_pool_ops_supported;
>>>         rte_eth_dev_reset;
>>> --
>>> 1.9.1
>>>
>>
>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Eelco Chaudron]

On 10 Jul 2018, at 11:34, Alejandro Lucero wrote:

> On Tue, Jul 10, 2018 at 9:56 AM, Eelco Chaudron <echaudro@redhat.com> 
> wrote:
>
>>
>>
>> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>>
>> A device can suffer addressing limitations. This functions checks
>>> memsegs have iovas within the supported range based on dma mask.
>>>
>>> PMD should use this during initialization if supported devices
>>> suffer addressing limitations, returning an error if this function
>>> returns memsegs out of range.
>>>
>>> Another potential usage is for emulated IOMMU hardware with 
>>> addressing
>>> limitations.
>>>
>>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>>> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>> ---
>>>  lib/librte_eal/common/eal_common_memory.c  | 33
>>> ++++++++++++++++++++++++++++++
>>>  lib/librte_eal/common/include/rte_memory.h |  3 +++
>>>  lib/librte_eal/rte_eal_version.map         |  1 +
>>>  3 files changed, 37 insertions(+)
>>>
>>> diff --git a/lib/librte_eal/common/eal_common_memory.c
>>> b/lib/librte_eal/common/eal_common_memory.c
>>> index fc6c44d..f5efebe 100644
>>> --- a/lib/librte_eal/common/eal_common_memory.c
>>> +++ b/lib/librte_eal/common/eal_common_memory.c
>>> @@ -109,6 +109,39 @@
>>>         }
>>>  }
>>>
>>> +/* check memseg iovas are within the required range based on dma 
>>> mask */
>>> +int
>>> +rte_eal_check_dma_mask(uint8_t maskbits)
>>> +{
>>> +
>>> +       const struct rte_mem_config *mcfg;
>>> +       uint64_t mask;
>>> +       int i;
>>> +
>>>
>>
>> I think we should add some sanity check to the input maskbits, i.e. 
>> [64,0)
>> or [64, 32]? What would be a reasonable lower bound.
>>
>>
> This is not a user's API, so any invocation will be reviewed, but I 
> guess
> adding a sanity check here does not harm.
>
> Not sure about lower bound but upper should 64, although it does not 
> make
> sense but it is safe. Lower bound is not so problematic.
>
>
>>
>> +       /* create dma mask */
>>> +       mask = ~((1ULL << maskbits) - 1);
>>> +
>>> +       /* get pointer to global configuration */
>>> +       mcfg = rte_eal_get_configuration()->mem_config;
>>> +
>>> +       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>>> +               if (mcfg->memseg[i].addr == NULL)
>>> +                       break;

Looking at some other code, it looks like NULL entries might exists. So 
should a continue; rather than a break; be used here?

>>> +
>>> +               if (mcfg->memseg[i].iova & mask) {
>>> +                       RTE_LOG(INFO, EAL,
>>> +                               "memseg[%d] iova %"PRIx64" out of
>>> range:\n",
>>> +                               i, mcfg->memseg[i].iova);
>>> +
>>> +                       RTE_LOG(INFO, EAL, "\tusing dma mask 
>>> %"PRIx64"\n",
>>> +                               mask);
>>> +                       return -1;
>>> +               }
>>> +       }
>>> +
>>> +       return 0;
>>> +}
>>> +
>>>  /* return the number of memory channels */
>>>  unsigned rte_memory_get_nchannel(void)
>>>  {
>>> diff --git a/lib/librte_eal/common/include/rte_memory.h
>>> b/lib/librte_eal/common/include/rte_memory.h
>>> index 80a8fc0..b2a0168 100644
>>> --- a/lib/librte_eal/common/include/rte_memory.h
>>> +++ b/lib/librte_eal/common/include/rte_memory.h
>>> @@ -209,6 +209,9 @@ struct rte_memseg {
>>>   */
>>>  unsigned rte_memory_get_nrank(void);
>>>
>>> +/* check memsegs iovas are within a range based on dma mask */
>>> +int rte_eal_check_dma_mask(uint8_t maskbits);
>>> +
>>>  /**
>>>   * Drivers based on uio will not load unless physical
>>>   * addresses are obtainable. It is only possible to get
>>> diff --git a/lib/librte_eal/rte_eal_version.map
>>> b/lib/librte_eal/rte_eal_version.map
>>> index f4f46c1..aa6cf87 100644
>>> --- a/lib/librte_eal/rte_eal_version.map
>>> +++ b/lib/librte_eal/rte_eal_version.map
>>> @@ -184,6 +184,7 @@ DPDK_17.11 {
>>>
>>>         rte_eal_create_uio_dev;
>>>         rte_bus_get_iommu_class;
>>> +       rte_eal_check_dma_mask;
>>>         rte_eal_has_pci;
>>>         rte_eal_iova_mode;
>>>         rte_eal_mbuf_default_mempool_ops;
>>> --
>>> 1.9.1
>>>
>>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 3/6] bus/pci: use IOVAs check when setting IOVA mode

[Eelco Chaudron]

On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:

> Although VT-d emulation currently only supports 39 bits, it could
> be iovas being within that supported range. This patch allows
> IOVA mode in such a case.
>
> Indeed, memory initialization code can be modified for using lower
> virtual addresses than those used by the kernel for 64 bits processes
> by default, and therefore memsegs iovas can use 39 bits or less for
> most system. And this is likely 100% true for VMs.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> ---
>  drivers/bus/pci/linux/pci.c | 15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
> index 74deef3..792c819 100644
> --- a/drivers/bus/pci/linux/pci.c
> +++ b/drivers/bus/pci/linux/pci.c
> @@ -43,6 +43,7 @@
>  #include <rte_devargs.h>
>  #include <rte_memcpy.h>
>  #include <rte_vfio.h>
> +#include <rte_memory.h>
>
>  #include "eal_private.h"
>  #include "eal_filesystem.h"
> @@ -613,10 +614,12 @@
>  	fclose(fp);
>
>  	mgaw = ((vtd_cap_reg & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT) + 
> 1;
> -	if (mgaw < X86_VA_WIDTH)
> +
> +	if (!rte_eal_check_dma_mask(mgaw))

If think in this case we still need to check the X86_VA_WIDTH, i.e.
if (mgaw < X86_VA_WIDTH && !rte_eal_check_dma_mask(mgaw))


> +		return true;
> +	else
>  		return false;
>
> -	return true;
>  }
>  #elif defined(RTE_ARCH_PPC_64)
>  static bool
> @@ -640,13 +643,17 @@
>  {
>  	struct rte_pci_device *dev = NULL;
>  	struct rte_pci_driver *drv = NULL;
> +	int iommu_dma_mask_check_done = 0;
>
>  	FOREACH_DRIVER_ON_PCIBUS(drv) {
>  		FOREACH_DEVICE_ON_PCIBUS(dev) {
>  			if (!rte_pci_match(drv, dev))
>  				continue;
> -			if (!pci_one_device_iommu_support_va(dev))
> -				return false;
> +			if (!iommu_dma_mask_check_done) {
> +				if (pci_one_device_iommu_support_va(dev) < 0)
> +					return false;
> +				iommu_dma_mask_check_done  = 1;

Not sure why this change? Why do we only need to check one device on the 
bus?

In addition, if this is what was intended, rather than a variable you 
can return true in this case, or did you intended to clear the 
iommu_dma_mask_check_done on every PCI BUS iteration?

> +			}
>  		}
>  	}
>  	return true;
> -- 
> 1.9.1

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 5/6] net/nfp: check hugepages IOVAs based on DMA mask

[Eelco Chaudron]

On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:

> NFP devices can not handle DMA addresses requiring more than
> 40 bits. This patch uses rte_dev_check_dma_mask with 40 bits
> and avoids device initialization if memory out of NFP range.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

Changes look good to me!

Acked-by: Eelco Chaudron <echaudro@redhat.com>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 6/6] net/nfp: support IOVA VA mode

[Eelco Chaudron]

On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:

> NFP can handle IOVA as VA. It requires to check those IOVAs
> being in the supported range what is done during initialization.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

Changes look good to me!

Acked-by: Eelco Chaudron <echaudro@redhat.com>

<SNIP>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Alejandro Lucero]

On Tue, Jul 10, 2018 at 11:06 AM, Eelco Chaudron <echaudro@redhat.com>
wrote:

>
>
> On 10 Jul 2018, at 11:34, Alejandro Lucero wrote:
>
> On Tue, Jul 10, 2018 at 9:56 AM, Eelco Chaudron <echaudro@redhat.com>
>> wrote:
>>
>>
>>>
>>> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>>>
>>> A device can suffer addressing limitations. This functions checks
>>>
>>>> memsegs have iovas within the supported range based on dma mask.
>>>>
>>>> PMD should use this during initialization if supported devices
>>>> suffer addressing limitations, returning an error if this function
>>>> returns memsegs out of range.
>>>>
>>>> Another potential usage is for emulated IOMMU hardware with addressing
>>>> limitations.
>>>>
>>>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>>>> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>>> ---
>>>>  lib/librte_eal/common/eal_common_memory.c  | 33
>>>> ++++++++++++++++++++++++++++++
>>>>  lib/librte_eal/common/include/rte_memory.h |  3 +++
>>>>  lib/librte_eal/rte_eal_version.map         |  1 +
>>>>  3 files changed, 37 insertions(+)
>>>>
>>>> diff --git a/lib/librte_eal/common/eal_common_memory.c
>>>> b/lib/librte_eal/common/eal_common_memory.c
>>>> index fc6c44d..f5efebe 100644
>>>> --- a/lib/librte_eal/common/eal_common_memory.c
>>>> +++ b/lib/librte_eal/common/eal_common_memory.c
>>>> @@ -109,6 +109,39 @@
>>>>         }
>>>>  }
>>>>
>>>> +/* check memseg iovas are within the required range based on dma mask
>>>> */
>>>> +int
>>>> +rte_eal_check_dma_mask(uint8_t maskbits)
>>>> +{
>>>> +
>>>> +       const struct rte_mem_config *mcfg;
>>>> +       uint64_t mask;
>>>> +       int i;
>>>> +
>>>>
>>>>
>>> I think we should add some sanity check to the input maskbits, i.e.
>>> [64,0)
>>> or [64, 32]? What would be a reasonable lower bound.
>>>
>>>
>>> This is not a user's API, so any invocation will be reviewed, but I guess
>> adding a sanity check here does not harm.
>>
>> Not sure about lower bound but upper should 64, although it does not make
>> sense but it is safe. Lower bound is not so problematic.
>>
>>
>>
>>> +       /* create dma mask */
>>>
>>>> +       mask = ~((1ULL << maskbits) - 1);
>>>> +
>>>> +       /* get pointer to global configuration */
>>>> +       mcfg = rte_eal_get_configuration()->mem_config;
>>>> +
>>>> +       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>>>> +               if (mcfg->memseg[i].addr == NULL)
>>>> +                       break;
>>>>
>>>
> Looking at some other code, it looks like NULL entries might exists. So
> should a continue; rather than a break; be used here?
>
>
I do not think so. memsegs are allocated sequentially, so first with addr
as NULL implies no more memsegs.


>
> +
>>>> +               if (mcfg->memseg[i].iova & mask) {
>>>> +                       RTE_LOG(INFO, EAL,
>>>> +                               "memseg[%d] iova %"PRIx64" out of
>>>> range:\n",
>>>> +                               i, mcfg->memseg[i].iova);
>>>> +
>>>> +                       RTE_LOG(INFO, EAL, "\tusing dma mask
>>>> %"PRIx64"\n",
>>>> +                               mask);
>>>> +                       return -1;
>>>> +               }
>>>> +       }
>>>> +
>>>> +       return 0;
>>>> +}
>>>> +
>>>>  /* return the number of memory channels */
>>>>  unsigned rte_memory_get_nchannel(void)
>>>>  {
>>>> diff --git a/lib/librte_eal/common/include/rte_memory.h
>>>> b/lib/librte_eal/common/include/rte_memory.h
>>>> index 80a8fc0..b2a0168 100644
>>>> --- a/lib/librte_eal/common/include/rte_memory.h
>>>> +++ b/lib/librte_eal/common/include/rte_memory.h
>>>> @@ -209,6 +209,9 @@ struct rte_memseg {
>>>>   */
>>>>  unsigned rte_memory_get_nrank(void);
>>>>
>>>> +/* check memsegs iovas are within a range based on dma mask */
>>>> +int rte_eal_check_dma_mask(uint8_t maskbits);
>>>> +
>>>>  /**
>>>>   * Drivers based on uio will not load unless physical
>>>>   * addresses are obtainable. It is only possible to get
>>>> diff --git a/lib/librte_eal/rte_eal_version.map
>>>> b/lib/librte_eal/rte_eal_version.map
>>>> index f4f46c1..aa6cf87 100644
>>>> --- a/lib/librte_eal/rte_eal_version.map
>>>> +++ b/lib/librte_eal/rte_eal_version.map
>>>> @@ -184,6 +184,7 @@ DPDK_17.11 {
>>>>
>>>>         rte_eal_create_uio_dev;
>>>>         rte_bus_get_iommu_class;
>>>> +       rte_eal_check_dma_mask;
>>>>         rte_eal_has_pci;
>>>>         rte_eal_iova_mode;
>>>>         rte_eal_mbuf_default_mempool_ops;
>>>> --
>>>> 1.9.1
>>>>
>>>>
>>>
>
>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Eelco Chaudron]

On 10 Jul 2018, at 12:52, Alejandro Lucero wrote:

> On Tue, Jul 10, 2018 at 11:06 AM, Eelco Chaudron <echaudro@redhat.com>
> wrote:
>
>>
>>
>> On 10 Jul 2018, at 11:34, Alejandro Lucero wrote:
>>
>> On Tue, Jul 10, 2018 at 9:56 AM, Eelco Chaudron <echaudro@redhat.com>
>>> wrote:
>>>
>>>
>>>>
>>>> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>>>>
>>>> A device can suffer addressing limitations. This functions checks
>>>>
>>>>> memsegs have iovas within the supported range based on dma mask.
>>>>>
>>>>> PMD should use this during initialization if supported devices
>>>>> suffer addressing limitations, returning an error if this function
>>>>> returns memsegs out of range.
>>>>>
>>>>> Another potential usage is for emulated IOMMU hardware with 
>>>>> addressing
>>>>> limitations.
>>>>>
>>>>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>>>>> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>>>> ---
>>>>>  lib/librte_eal/common/eal_common_memory.c  | 33
>>>>> ++++++++++++++++++++++++++++++
>>>>>  lib/librte_eal/common/include/rte_memory.h |  3 +++
>>>>>  lib/librte_eal/rte_eal_version.map         |  1 +
>>>>>  3 files changed, 37 insertions(+)
>>>>>
>>>>> diff --git a/lib/librte_eal/common/eal_common_memory.c
>>>>> b/lib/librte_eal/common/eal_common_memory.c
>>>>> index fc6c44d..f5efebe 100644
>>>>> --- a/lib/librte_eal/common/eal_common_memory.c
>>>>> +++ b/lib/librte_eal/common/eal_common_memory.c
>>>>> @@ -109,6 +109,39 @@
>>>>>         }
>>>>>  }
>>>>>
>>>>> +/* check memseg iovas are within the required range based on dma 
>>>>> mask
>>>>> */
>>>>> +int
>>>>> +rte_eal_check_dma_mask(uint8_t maskbits)
>>>>> +{
>>>>> +
>>>>> +       const struct rte_mem_config *mcfg;
>>>>> +       uint64_t mask;
>>>>> +       int i;
>>>>> +
>>>>>
>>>>>
>>>> I think we should add some sanity check to the input maskbits, i.e.
>>>> [64,0)
>>>> or [64, 32]? What would be a reasonable lower bound.
>>>>
>>>>
>>>> This is not a user's API, so any invocation will be reviewed, but I 
>>>> guess
>>> adding a sanity check here does not harm.
>>>
>>> Not sure about lower bound but upper should 64, although it does not 
>>> make
>>> sense but it is safe. Lower bound is not so problematic.
>>>
>>>
>>>
>>>> +       /* create dma mask */
>>>>
>>>>> +       mask = ~((1ULL << maskbits) - 1);
>>>>> +
>>>>> +       /* get pointer to global configuration */
>>>>> +       mcfg = rte_eal_get_configuration()->mem_config;
>>>>> +
>>>>> +       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>>>>> +               if (mcfg->memseg[i].addr == NULL)
>>>>> +                       break;
>>>>>
>>>>
>> Looking at some other code, it looks like NULL entries might exists. 
>> So
>> should a continue; rather than a break; be used here?
>>
>>
> I do not think so. memsegs are allocated sequentially, so first with 
> addr
> as NULL implies no more memsegs.

I was referring to the mem walk functions, rte_memseg_list_walk(). Maybe 
some having more experience with this area can review/comment.

>
>
>>
>> +
>>>>> +               if (mcfg->memseg[i].iova & mask) {
>>>>> +                       RTE_LOG(INFO, EAL,
>>>>> +                               "memseg[%d] iova %"PRIx64" out of
>>>>> range:\n",
>>>>> +                               i, mcfg->memseg[i].iova);
>>>>> +
>>>>> +                       RTE_LOG(INFO, EAL, "\tusing dma mask
>>>>> %"PRIx64"\n",
>>>>> +                               mask);
>>>>> +                       return -1;
>>>>> +               }
>>>>> +       }
>>>>> +
>>>>> +       return 0;
>>>>> +}
>>>>> +
>>>>>  /* return the number of memory channels */
>>>>>  unsigned rte_memory_get_nchannel(void)
>>>>>  {
>>>>> diff --git a/lib/librte_eal/common/include/rte_memory.h
>>>>> b/lib/librte_eal/common/include/rte_memory.h
>>>>> index 80a8fc0..b2a0168 100644
>>>>> --- a/lib/librte_eal/common/include/rte_memory.h
>>>>> +++ b/lib/librte_eal/common/include/rte_memory.h
>>>>> @@ -209,6 +209,9 @@ struct rte_memseg {
>>>>>   */
>>>>>  unsigned rte_memory_get_nrank(void);
>>>>>
>>>>> +/* check memsegs iovas are within a range based on dma mask */
>>>>> +int rte_eal_check_dma_mask(uint8_t maskbits);
>>>>> +
>>>>>  /**
>>>>>   * Drivers based on uio will not load unless physical
>>>>>   * addresses are obtainable. It is only possible to get
>>>>> diff --git a/lib/librte_eal/rte_eal_version.map
>>>>> b/lib/librte_eal/rte_eal_version.map
>>>>> index f4f46c1..aa6cf87 100644
>>>>> --- a/lib/librte_eal/rte_eal_version.map
>>>>> +++ b/lib/librte_eal/rte_eal_version.map
>>>>> @@ -184,6 +184,7 @@ DPDK_17.11 {
>>>>>
>>>>>         rte_eal_create_uio_dev;
>>>>>         rte_bus_get_iommu_class;
>>>>> +       rte_eal_check_dma_mask;
>>>>>         rte_eal_has_pci;
>>>>>         rte_eal_iova_mode;
>>>>>         rte_eal_mbuf_default_mempool_ops;
>>>>> --
>>>>> 1.9.1
>>>>>
>>>>>
>>>>
>>
>>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 4/6] mem: use address hint for mapping hugepages

[Eelco Chaudron]

On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:

> Linux kernel uses a really high address as starting address for
> serving mmaps calls. If there exists addressing limitations and
> IOVA mode is VA, this starting address is likely too high for
> those devices. However, it is possible to use a lower address in
> the process virtual address space as with 64 bits there is a lot
> of available space.
>
> This patch adds an address hint as starting address for 64 bits
> systems.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

Looks good to me!

Cheers,

Eelco

Acked-by: Eelco Chaudron <echaudro@redhat.com>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Burakov, Anatoly]

On 10-Jul-18 12:14 PM, Eelco Chaudron wrote:
> 
> 
> On 10 Jul 2018, at 12:52, Alejandro Lucero wrote:
> 
>> On Tue, Jul 10, 2018 at 11:06 AM, Eelco Chaudron <echaudro@redhat.com>
>> wrote:
>>
>>>
>>>
>>> On 10 Jul 2018, at 11:34, Alejandro Lucero wrote:
>>>
>>> On Tue, Jul 10, 2018 at 9:56 AM, Eelco Chaudron <echaudro@redhat.com>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>>>>>
>>>>> A device can suffer addressing limitations. This functions checks
>>>>>
>>>>>> memsegs have iovas within the supported range based on dma mask.
>>>>>>
>>>>>> PMD should use this during initialization if supported devices
>>>>>> suffer addressing limitations, returning an error if this function
>>>>>> returns memsegs out of range.
>>>>>>
>>>>>> Another potential usage is for emulated IOMMU hardware with 
>>>>>> addressing
>>>>>> limitations.
>>>>>>
>>>>>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>>>>>> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>>>>> ---
>>>>>>  lib/librte_eal/common/eal_common_memory.c  | 33
>>>>>> ++++++++++++++++++++++++++++++
>>>>>>  lib/librte_eal/common/include/rte_memory.h |  3 +++
>>>>>>  lib/librte_eal/rte_eal_version.map         |  1 +
>>>>>>  3 files changed, 37 insertions(+)
>>>>>>
>>>>>> diff --git a/lib/librte_eal/common/eal_common_memory.c
>>>>>> b/lib/librte_eal/common/eal_common_memory.c
>>>>>> index fc6c44d..f5efebe 100644
>>>>>> --- a/lib/librte_eal/common/eal_common_memory.c
>>>>>> +++ b/lib/librte_eal/common/eal_common_memory.c
>>>>>> @@ -109,6 +109,39 @@
>>>>>>         }
>>>>>>  }
>>>>>>
>>>>>> +/* check memseg iovas are within the required range based on dma 
>>>>>> mask
>>>>>> */
>>>>>> +int
>>>>>> +rte_eal_check_dma_mask(uint8_t maskbits)
>>>>>> +{
>>>>>> +
>>>>>> +       const struct rte_mem_config *mcfg;
>>>>>> +       uint64_t mask;
>>>>>> +       int i;
>>>>>> +
>>>>>>
>>>>>>
>>>>> I think we should add some sanity check to the input maskbits, i.e.
>>>>> [64,0)
>>>>> or [64, 32]? What would be a reasonable lower bound.
>>>>>
>>>>>
>>>>> This is not a user's API, so any invocation will be reviewed, but I 
>>>>> guess
>>>> adding a sanity check here does not harm.
>>>>
>>>> Not sure about lower bound but upper should 64, although it does not 
>>>> make
>>>> sense but it is safe. Lower bound is not so problematic.
>>>>
>>>>
>>>>
>>>>> +       /* create dma mask */
>>>>>
>>>>>> +       mask = ~((1ULL << maskbits) - 1);
>>>>>> +
>>>>>> +       /* get pointer to global configuration */
>>>>>> +       mcfg = rte_eal_get_configuration()->mem_config;
>>>>>> +
>>>>>> +       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>>>>>> +               if (mcfg->memseg[i].addr == NULL)
>>>>>> +                       break;
>>>>>>
>>>>>
>>> Looking at some other code, it looks like NULL entries might exists. So
>>> should a continue; rather than a break; be used here?
>>>
>>>
>> I do not think so. memsegs are allocated sequentially, so first with addr
>> as NULL implies no more memsegs.
> 
> I was referring to the mem walk functions, rte_memseg_list_walk(). Maybe 
> some having more experience with this area can review/comment.

Pre-18.05, all memsegs are allocated continuously. Memseg lists and 
memseg list walk functions are 18.05+.

Alejandro, perhaps it would be worth it to tag your patchset with 
"pre-18.05" to avoid similar confusion in the future?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Alejandro Lucero]

On Tue, Jul 10, 2018 at 12:14 PM, Eelco Chaudron <echaudro@redhat.com>
wrote:

>
>
> On 10 Jul 2018, at 12:52, Alejandro Lucero wrote:
>
> On Tue, Jul 10, 2018 at 11:06 AM, Eelco Chaudron <echaudro@redhat.com>
>> wrote:
>>
>>
>>>
>>> On 10 Jul 2018, at 11:34, Alejandro Lucero wrote:
>>>
>>> On Tue, Jul 10, 2018 at 9:56 AM, Eelco Chaudron <echaudro@redhat.com>
>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>>>>>
>>>>> A device can suffer addressing limitations. This functions checks
>>>>>
>>>>> memsegs have iovas within the supported range based on dma mask.
>>>>>>
>>>>>> PMD should use this during initialization if supported devices
>>>>>> suffer addressing limitations, returning an error if this function
>>>>>> returns memsegs out of range.
>>>>>>
>>>>>> Another potential usage is for emulated IOMMU hardware with addressing
>>>>>> limitations.
>>>>>>
>>>>>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>>>>>> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>>>>> ---
>>>>>>  lib/librte_eal/common/eal_common_memory.c  | 33
>>>>>> ++++++++++++++++++++++++++++++
>>>>>>  lib/librte_eal/common/include/rte_memory.h |  3 +++
>>>>>>  lib/librte_eal/rte_eal_version.map         |  1 +
>>>>>>  3 files changed, 37 insertions(+)
>>>>>>
>>>>>> diff --git a/lib/librte_eal/common/eal_common_memory.c
>>>>>> b/lib/librte_eal/common/eal_common_memory.c
>>>>>> index fc6c44d..f5efebe 100644
>>>>>> --- a/lib/librte_eal/common/eal_common_memory.c
>>>>>> +++ b/lib/librte_eal/common/eal_common_memory.c
>>>>>> @@ -109,6 +109,39 @@
>>>>>>         }
>>>>>>  }
>>>>>>
>>>>>> +/* check memseg iovas are within the required range based on dma mask
>>>>>> */
>>>>>> +int
>>>>>> +rte_eal_check_dma_mask(uint8_t maskbits)
>>>>>> +{
>>>>>> +
>>>>>> +       const struct rte_mem_config *mcfg;
>>>>>> +       uint64_t mask;
>>>>>> +       int i;
>>>>>> +
>>>>>>
>>>>>>
>>>>>> I think we should add some sanity check to the input maskbits, i.e.
>>>>> [64,0)
>>>>> or [64, 32]? What would be a reasonable lower bound.
>>>>>
>>>>>
>>>>> This is not a user's API, so any invocation will be reviewed, but I
>>>>> guess
>>>>>
>>>> adding a sanity check here does not harm.
>>>>
>>>> Not sure about lower bound but upper should 64, although it does not
>>>> make
>>>> sense but it is safe. Lower bound is not so problematic.
>>>>
>>>>
>>>>
>>>> +       /* create dma mask */
>>>>>
>>>>> +       mask = ~((1ULL << maskbits) - 1);
>>>>>> +
>>>>>> +       /* get pointer to global configuration */
>>>>>> +       mcfg = rte_eal_get_configuration()->mem_config;
>>>>>> +
>>>>>> +       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>>>>>> +               if (mcfg->memseg[i].addr == NULL)
>>>>>> +                       break;
>>>>>>
>>>>>>
>>>>> Looking at some other code, it looks like NULL entries might exists. So
>>> should a continue; rather than a break; be used here?
>>>
>>>
>>> I do not think so. memsegs are allocated sequentially, so first with addr
>> as NULL implies no more memsegs.
>>
>
> I was referring to the mem walk functions, rte_memseg_list_walk(). Maybe
> some having more experience with this area can review/comment.
>
>
This patchset applies to 17.11.3 which has not that function implemented.

You can see what rte_eal_get_physmem_size and rte_dump_physmem_layout do in
lib/librte_eal/common/eal_common_memory.c file regarding memseg "walks"
when addr is NULL.




>
>
>>
>>
>>> +
>>>
>>>> +               if (mcfg->memseg[i].iova & mask) {
>>>>>> +                       RTE_LOG(INFO, EAL,
>>>>>> +                               "memseg[%d] iova %"PRIx64" out of
>>>>>> range:\n",
>>>>>> +                               i, mcfg->memseg[i].iova);
>>>>>> +
>>>>>> +                       RTE_LOG(INFO, EAL, "\tusing dma mask
>>>>>> %"PRIx64"\n",
>>>>>> +                               mask);
>>>>>> +                       return -1;
>>>>>> +               }
>>>>>> +       }
>>>>>> +
>>>>>> +       return 0;
>>>>>> +}
>>>>>> +
>>>>>>  /* return the number of memory channels */
>>>>>>  unsigned rte_memory_get_nchannel(void)
>>>>>>  {
>>>>>> diff --git a/lib/librte_eal/common/include/rte_memory.h
>>>>>> b/lib/librte_eal/common/include/rte_memory.h
>>>>>> index 80a8fc0..b2a0168 100644
>>>>>> --- a/lib/librte_eal/common/include/rte_memory.h
>>>>>> +++ b/lib/librte_eal/common/include/rte_memory.h
>>>>>> @@ -209,6 +209,9 @@ struct rte_memseg {
>>>>>>   */
>>>>>>  unsigned rte_memory_get_nrank(void);
>>>>>>
>>>>>> +/* check memsegs iovas are within a range based on dma mask */
>>>>>> +int rte_eal_check_dma_mask(uint8_t maskbits);
>>>>>> +
>>>>>>  /**
>>>>>>   * Drivers based on uio will not load unless physical
>>>>>>   * addresses are obtainable. It is only possible to get
>>>>>> diff --git a/lib/librte_eal/rte_eal_version.map
>>>>>> b/lib/librte_eal/rte_eal_version.map
>>>>>> index f4f46c1..aa6cf87 100644
>>>>>> --- a/lib/librte_eal/rte_eal_version.map
>>>>>> +++ b/lib/librte_eal/rte_eal_version.map
>>>>>> @@ -184,6 +184,7 @@ DPDK_17.11 {
>>>>>>
>>>>>>         rte_eal_create_uio_dev;
>>>>>>         rte_bus_get_iommu_class;
>>>>>> +       rte_eal_check_dma_mask;
>>>>>>         rte_eal_has_pci;
>>>>>>         rte_eal_iova_mode;
>>>>>>         rte_eal_mbuf_default_mempool_ops;
>>>>>> --
>>>>>> 1.9.1
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>
>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Alejandro Lucero]

On Tue, Jul 10, 2018 at 12:33 PM, Burakov, Anatoly <
anatoly.burakov@intel.com> wrote:

> On 10-Jul-18 12:14 PM, Eelco Chaudron wrote:
>
>>
>>
>> On 10 Jul 2018, at 12:52, Alejandro Lucero wrote:
>>
>> On Tue, Jul 10, 2018 at 11:06 AM, Eelco Chaudron <echaudro@redhat.com>
>>> wrote:
>>>
>>>
>>>>
>>>> On 10 Jul 2018, at 11:34, Alejandro Lucero wrote:
>>>>
>>>> On Tue, Jul 10, 2018 at 9:56 AM, Eelco Chaudron <echaudro@redhat.com>
>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>>>>>>
>>>>>> A device can suffer addressing limitations. This functions checks
>>>>>>
>>>>>> memsegs have iovas within the supported range based on dma mask.
>>>>>>>
>>>>>>> PMD should use this during initialization if supported devices
>>>>>>> suffer addressing limitations, returning an error if this function
>>>>>>> returns memsegs out of range.
>>>>>>>
>>>>>>> Another potential usage is for emulated IOMMU hardware with
>>>>>>> addressing
>>>>>>> limitations.
>>>>>>>
>>>>>>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>>>>>>> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>>>>>> ---
>>>>>>>  lib/librte_eal/common/eal_common_memory.c  | 33
>>>>>>> ++++++++++++++++++++++++++++++
>>>>>>>  lib/librte_eal/common/include/rte_memory.h |  3 +++
>>>>>>>  lib/librte_eal/rte_eal_version.map         |  1 +
>>>>>>>  3 files changed, 37 insertions(+)
>>>>>>>
>>>>>>> diff --git a/lib/librte_eal/common/eal_common_memory.c
>>>>>>> b/lib/librte_eal/common/eal_common_memory.c
>>>>>>> index fc6c44d..f5efebe 100644
>>>>>>> --- a/lib/librte_eal/common/eal_common_memory.c
>>>>>>> +++ b/lib/librte_eal/common/eal_common_memory.c
>>>>>>> @@ -109,6 +109,39 @@
>>>>>>>         }
>>>>>>>  }
>>>>>>>
>>>>>>> +/* check memseg iovas are within the required range based on dma
>>>>>>> mask
>>>>>>> */
>>>>>>> +int
>>>>>>> +rte_eal_check_dma_mask(uint8_t maskbits)
>>>>>>> +{
>>>>>>> +
>>>>>>> +       const struct rte_mem_config *mcfg;
>>>>>>> +       uint64_t mask;
>>>>>>> +       int i;
>>>>>>> +
>>>>>>>
>>>>>>>
>>>>>>> I think we should add some sanity check to the input maskbits, i.e.
>>>>>> [64,0)
>>>>>> or [64, 32]? What would be a reasonable lower bound.
>>>>>>
>>>>>>
>>>>>> This is not a user's API, so any invocation will be reviewed, but I
>>>>>> guess
>>>>>>
>>>>> adding a sanity check here does not harm.
>>>>>
>>>>> Not sure about lower bound but upper should 64, although it does not
>>>>> make
>>>>> sense but it is safe. Lower bound is not so problematic.
>>>>>
>>>>>
>>>>>
>>>>> +       /* create dma mask */
>>>>>>
>>>>>> +       mask = ~((1ULL << maskbits) - 1);
>>>>>>> +
>>>>>>> +       /* get pointer to global configuration */
>>>>>>> +       mcfg = rte_eal_get_configuration()->mem_config;
>>>>>>> +
>>>>>>> +       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>>>>>>> +               if (mcfg->memseg[i].addr == NULL)
>>>>>>> +                       break;
>>>>>>>
>>>>>>>
>>>>>> Looking at some other code, it looks like NULL entries might exists.
>>>> So
>>>> should a continue; rather than a break; be used here?
>>>>
>>>>
>>>> I do not think so. memsegs are allocated sequentially, so first with
>>> addr
>>> as NULL implies no more memsegs.
>>>
>>
>> I was referring to the mem walk functions, rte_memseg_list_walk(). Maybe
>> some having more experience with this area can review/comment.
>>
>
> Pre-18.05, all memsegs are allocated continuously. Memseg lists and memseg
> list walk functions are 18.05+.
>
> Alejandro, perhaps it would be worth it to tag your patchset with
> "pre-18.05" to avoid similar confusion in the future?
>
>
Yes, that will help. I'm sending a new version shortly and I'll make it
clear.



> --
> Thanks,
> Anatoly
>

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Eelco Chaudron]

On 10 Jul 2018, at 13:43, Alejandro Lucero wrote:

> On Tue, Jul 10, 2018 at 12:33 PM, Burakov, Anatoly <
> anatoly.burakov@intel.com> wrote:
>
>> On 10-Jul-18 12:14 PM, Eelco Chaudron wrote:
>>
>>>
>>>
>>> On 10 Jul 2018, at 12:52, Alejandro Lucero wrote:
>>>
>>> On Tue, Jul 10, 2018 at 11:06 AM, Eelco Chaudron 
>>> <echaudro@redhat.com>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> On 10 Jul 2018, at 11:34, Alejandro Lucero wrote:
>>>>>
>>>>> On Tue, Jul 10, 2018 at 9:56 AM, Eelco Chaudron 
>>>>> <echaudro@redhat.com>
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>>>>>>>
>>>>>>> A device can suffer addressing limitations. This functions 
>>>>>>> checks
>>>>>>>
>>>>>>> memsegs have iovas within the supported range based on dma mask.
>>>>>>>>
>>>>>>>> PMD should use this during initialization if supported devices
>>>>>>>> suffer addressing limitations, returning an error if this 
>>>>>>>> function
>>>>>>>> returns memsegs out of range.
>>>>>>>>
>>>>>>>> Another potential usage is for emulated IOMMU hardware with
>>>>>>>> addressing
>>>>>>>> limitations.
>>>>>>>>
>>>>>>>> Signed-off-by: Alejandro Lucero 
>>>>>>>> <alejandro.lucero@netronome.com>
>>>>>>>> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>>>>>>> ---
>>>>>>>>  lib/librte_eal/common/eal_common_memory.c  | 33
>>>>>>>> ++++++++++++++++++++++++++++++
>>>>>>>>  lib/librte_eal/common/include/rte_memory.h |  3 +++
>>>>>>>>  lib/librte_eal/rte_eal_version.map         |  1 +
>>>>>>>>  3 files changed, 37 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/lib/librte_eal/common/eal_common_memory.c
>>>>>>>> b/lib/librte_eal/common/eal_common_memory.c
>>>>>>>> index fc6c44d..f5efebe 100644
>>>>>>>> --- a/lib/librte_eal/common/eal_common_memory.c
>>>>>>>> +++ b/lib/librte_eal/common/eal_common_memory.c
>>>>>>>> @@ -109,6 +109,39 @@
>>>>>>>>         }
>>>>>>>>  }
>>>>>>>>
>>>>>>>> +/* check memseg iovas are within the required range based on 
>>>>>>>> dma
>>>>>>>> mask
>>>>>>>> */
>>>>>>>> +int
>>>>>>>> +rte_eal_check_dma_mask(uint8_t maskbits)
>>>>>>>> +{
>>>>>>>> +
>>>>>>>> +       const struct rte_mem_config *mcfg;
>>>>>>>> +       uint64_t mask;
>>>>>>>> +       int i;
>>>>>>>> +
>>>>>>>>
>>>>>>>>
>>>>>>>> I think we should add some sanity check to the input maskbits, 
>>>>>>>> i.e.
>>>>>>> [64,0)
>>>>>>> or [64, 32]? What would be a reasonable lower bound.
>>>>>>>
>>>>>>>
>>>>>>> This is not a user's API, so any invocation will be reviewed, 
>>>>>>> but I
>>>>>>> guess
>>>>>>>
>>>>>> adding a sanity check here does not harm.
>>>>>>
>>>>>> Not sure about lower bound but upper should 64, although it does 
>>>>>> not
>>>>>> make
>>>>>> sense but it is safe. Lower bound is not so problematic.
>>>>>>
>>>>>>
>>>>>>
>>>>>> +       /* create dma mask */
>>>>>>>
>>>>>>> +       mask = ~((1ULL << maskbits) - 1);
>>>>>>>> +
>>>>>>>> +       /* get pointer to global configuration */
>>>>>>>> +       mcfg = rte_eal_get_configuration()->mem_config;
>>>>>>>> +
>>>>>>>> +       for (i = 0; i < RTE_MAX_MEMSEG; i++) {
>>>>>>>> +               if (mcfg->memseg[i].addr == NULL)
>>>>>>>> +                       break;
>>>>>>>>
>>>>>>>>
>>>>>>> Looking at some other code, it looks like NULL entries might 
>>>>>>> exists.
>>>>> So
>>>>> should a continue; rather than a break; be used here?
>>>>>
>>>>>
>>>>> I do not think so. memsegs are allocated sequentially, so first 
>>>>> with
>>>> addr
>>>> as NULL implies no more memsegs.
>>>>
>>>
>>> I was referring to the mem walk functions, rte_memseg_list_walk(). 
>>> Maybe
>>> some having more experience with this area can review/comment.
>>>
>>
>> Pre-18.05, all memsegs are allocated continuously. Memseg lists and 
>> memseg
>> list walk functions are 18.05+.
>>
>> Alejandro, perhaps it would be worth it to tag your patchset with
>> "pre-18.05" to avoid similar confusion in the future?
>>
>>
> Yes, that will help. I'm sending a new version shortly and I'll make 
> it
> clear.

Thanks, I’ll review the new version if it’s ready before the end of 
tomorrow CET, as I will be on PTO.

Re: [dpdk-dev] [dpdk-stable] [PATCH v3 3/6] bus/pci: use IOVAs check when setting IOVA mode

[Alejandro Lucero]

On Tue, Jul 10, 2018 at 11:14 AM, Eelco Chaudron <echaudro@redhat.com>
wrote:

>
>
> On 4 Jul 2018, at 14:53, Alejandro Lucero wrote:
>
> Although VT-d emulation currently only supports 39 bits, it could
>> be iovas being within that supported range. This patch allows
>> IOVA mode in such a case.
>>
>> Indeed, memory initialization code can be modified for using lower
>> virtual addresses than those used by the kernel for 64 bits processes
>> by default, and therefore memsegs iovas can use 39 bits or less for
>> most system. And this is likely 100% true for VMs.
>>
>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>> ---
>>  drivers/bus/pci/linux/pci.c | 15 +++++++++++----
>>  1 file changed, 11 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
>> index 74deef3..792c819 100644
>> --- a/drivers/bus/pci/linux/pci.c
>> +++ b/drivers/bus/pci/linux/pci.c
>> @@ -43,6 +43,7 @@
>>  #include <rte_devargs.h>
>>  #include <rte_memcpy.h>
>>  #include <rte_vfio.h>
>> +#include <rte_memory.h>
>>
>>  #include "eal_private.h"
>>  #include "eal_filesystem.h"
>> @@ -613,10 +614,12 @@
>>         fclose(fp);
>>
>>         mgaw = ((vtd_cap_reg & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT)
>> + 1;
>> -       if (mgaw < X86_VA_WIDTH)
>> +
>> +       if (!rte_eal_check_dma_mask(mgaw))
>>
>
> If think in this case we still need to check the X86_VA_WIDTH, i.e.
> if (mgaw < X86_VA_WIDTH && !rte_eal_check_dma_mask(mgaw))
>
>
> +               return true;
>> +       else
>>                 return false;
>>
>> -       return true;
>>  }
>>  #elif defined(RTE_ARCH_PPC_64)
>>  static bool
>> @@ -640,13 +643,17 @@
>>  {
>>         struct rte_pci_device *dev = NULL;
>>         struct rte_pci_driver *drv = NULL;
>> +       int iommu_dma_mask_check_done = 0;
>>
>>         FOREACH_DRIVER_ON_PCIBUS(drv) {
>>                 FOREACH_DEVICE_ON_PCIBUS(dev) {
>>                         if (!rte_pci_match(drv, dev))
>>                                 continue;
>> -                       if (!pci_one_device_iommu_support_va(dev))
>> -                               return false;
>> +                       if (!iommu_dma_mask_check_done) {
>> +                               if (pci_one_device_iommu_support_va(dev)
>> < 0)
>> +                                       return false;
>> +                               iommu_dma_mask_check_done  = 1;
>>
>
> Not sure why this change? Why do we only need to check one device on the
> bus?
>
>
Because there is just one emulated IOMMU hardware. The limitation in this
case is not in a specific PCI device. And I do not think it is possible to
have two different (emulated or not) IOMMU hardware. Yes, you can have more
than one controller but being same IOMMU type.


> In addition, if this is what was intended, rather than a variable you can
> return true in this case, or did you intended to clear the
> iommu_dma_mask_check_done on every PCI BUS iteration?
>
>
If pci_one_device_iommu_support_va, because the dma check, finds out the
IOVAs are out of range, then the IOVA mode is PA and no further checks are
required. But there could be a PCI device precluding the IOVA VA, so all
the PCI devices need to be processed.


> +                       }
>>                 }
>>         }
>>         return true;
>> --
>> 1.9.1
>>
>

Re: [dpdk-dev] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Thomas Monjalon]

05/10/2018 14:45, Alejandro Lucero:
> A device can suffer addressing limitations. This function checks
> memsegs have iovas within the supported range based on dma mask.
> 
> PMDs should use this function during initialization if device
> suffers addressing limitations, returning an error if this function
> returns memsegs out of range.
> 
> Another usage is for emulated IOMMU hardware with addressing
> limitations.
> 
> It is necessary to save the most restricted dma mask for checking out
> memory allocated dynamically after initialization.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

I am really disappointed that no more people looked at this patch,
even after asking in several maintainers meeting:
	http://mails.dpdk.org/archives/dev/2018-October/116337.html
	http://mails.dpdk.org/archives/dev/2018-October/117092.html

This series will enter in 18.11-rc1 because no strong objection
(only 1 issue raised without any followup).

I had no time to review it recently, but after a quick look, I see
some details which can be improved. Please, Alejandro,
consider comments below for a separate patch in -rc2. Thanks

[...]
> +static int
> +check_iova(const struct rte_memseg_list *msl __rte_unused,
> +		const struct rte_memseg *ms, void *arg)
> +{
> +	uint64_t *mask = arg;
> +	rte_iova_t iova;
> +
> +	/* higher address within segment */
> +	iova = (ms->iova + ms->len) - 1;

This black magic needs to be better commented.

> +	if (!(iova & *mask))
> +		return 0;
> +
> +	RTE_LOG(DEBUG, EAL, "memseg iova %"PRIx64", len %zx, out of range\n",
> +			    ms->iova, ms->len);
> +
> +	RTE_LOG(DEBUG, EAL, "\tusing dma mask %"PRIx64"\n", *mask);
> +	return 1;
> +}
[...]
> --- a/lib/librte_eal/common/include/rte_memory.h
> +++ b/lib/librte_eal/common/include/rte_memory.h
> +/* check memsegs iovas are within a range based on dma mask */
> +int rte_eal_check_dma_mask(uint8_t maskbits);

You need to insert a doxygen comment, explaining the format
of the maskbits parameter, and the behaviour of the function.
The doxygen comment must also state the function is experimental.

About the name of the function, the trend is to use rte_mem_ prefix
instead of rte_eal_ in this case.

__rte_experimental tag must be added to the function prototype, I will do
when applying the patch.

Re: [dpdk-dev] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Alejandro Lucero]

On Wed, Oct 10, 2018 at 10:00 AM Tu, Lijuan <lijuan.tu@intel.com> wrote:

> Hi
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alejandro Lucero
> > Sent: Friday, October 5, 2018 8:45 PM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] [PATCH v3 1/6] mem: add function for checking
> > memsegs IOVAs addresses
> >
> > A device can suffer addressing limitations. This function checks memsegs
> > have iovas within the supported range based on dma mask.
> >
> > PMDs should use this function during initialization if device suffers
> > addressing limitations, returning an error if this function returns
> memsegs
> > out of range.
> >
> > Another usage is for emulated IOMMU hardware with addressing limitations.
> >
> > It is necessary to save the most restricted dma mask for checking out
> > memory allocated dynamically after initialization.
> >
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> > Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
> > ---
> >  doc/guides/rel_notes/release_18_11.rst            | 10 ++++
> >  lib/librte_eal/common/eal_common_memory.c         | 60
> > +++++++++++++++++++++++
> >  lib/librte_eal/common/include/rte_eal_memconfig.h |  3 ++
> >  lib/librte_eal/common/include/rte_memory.h        |  3 ++
> >  lib/librte_eal/common/malloc_heap.c               | 12 +++++
> >  lib/librte_eal/linuxapp/eal/eal.c                 |  2 +
> >  lib/librte_eal/rte_eal_version.map                |  1 +
> >  7 files changed, 91 insertions(+)
> >
> > diff --git a/doc/guides/rel_notes/release_18_11.rst
> > b/doc/guides/rel_notes/release_18_11.rst
> > index 2133a5b..c806dc6 100644
> > --- a/doc/guides/rel_notes/release_18_11.rst
> > +++ b/doc/guides/rel_notes/release_18_11.rst
> > @@ -104,6 +104,14 @@ New Features
> >    the specified port. The port must be stopped before the command call
> in
> > order
> >    to reconfigure queues.
> >
> > +* **Added check for ensuring allocated memory addressable by devices.**
> > +
> > +  Some devices can have addressing limitations so a new function,
> > + ``rte_eal_check_dma_mask``, has been added for checking allocated
> > + memory is  not out of the device range. Because now memory can be
> > + dynamically allocated  after initialization, a dma mask is kept and
> > + any new allocated memory will be  checked out against that dma mask
> > + and rejected if out of range. If more than  one device has addressing
> > limitations, the dma mask is the more restricted one.
> >
> >  API Changes
> >  -----------
> > @@ -156,6 +164,8 @@ ABI Changes
> >         ``rte_config`` structure on account of improving DPDK usability
> > when
> >         using either ``--legacy-mem`` or ``--single-file-segments``
> flags.
> >
> > +* eal: added ``dma_maskbits`` to ``rte_mem_config`` for keeping more
> > restricted
> > +       dma mask based on devices addressing limitations.
> >
> >  Removed Items
> >  -------------
> > diff --git a/lib/librte_eal/common/eal_common_memory.c
> > b/lib/librte_eal/common/eal_common_memory.c
> > index 0b69804..c482f0d 100644
> > --- a/lib/librte_eal/common/eal_common_memory.c
> > +++ b/lib/librte_eal/common/eal_common_memory.c
> > @@ -385,6 +385,66 @@ struct virtiova {
> >       rte_memseg_walk(dump_memseg, f);
> >  }
> >
> > +static int
> > +check_iova(const struct rte_memseg_list *msl __rte_unused,
> > +             const struct rte_memseg *ms, void *arg) {
> > +     uint64_t *mask = arg;
> > +     rte_iova_t iova;
> > +
> > +     /* higher address within segment */
> > +     iova = (ms->iova + ms->len) - 1;
> > +     if (!(iova & *mask))
> > +             return 0;
> > +
> > +     RTE_LOG(DEBUG, EAL, "memseg iova %"PRIx64", len %zx, out of
> > range\n",
> > +                         ms->iova, ms->len);
> > +
> > +     RTE_LOG(DEBUG, EAL, "\tusing dma mask %"PRIx64"\n", *mask);
> > +     return 1;
> > +}
> > +
> > +#if defined(RTE_ARCH_64)
> > +#define MAX_DMA_MASK_BITS 63
> > +#else
> > +#define MAX_DMA_MASK_BITS 31
> > +#endif
> > +
> > +/* check memseg iovas are within the required range based on dma mask
> > +*/ int __rte_experimental rte_eal_check_dma_mask(uint8_t maskbits) {
> > +     struct rte_mem_config *mcfg =
> > rte_eal_get_configuration()->mem_config;
> > +     uint64_t mask;
> > +
> > +     /* sanity check */
> > +     if (maskbits > MAX_DMA_MASK_BITS) {
> > +             RTE_LOG(ERR, EAL, "wrong dma mask size %u (Max: %u)\n",
> > +                                maskbits, MAX_DMA_MASK_BITS);
> > +             return -1;
> > +     }
> > +
> > +     /* create dma mask */
> > +     mask = ~((1ULL << maskbits) - 1);
> > +
> > +     if (rte_memseg_walk(check_iova, &mask))
>
> [Lijuan]In my environment, testpmd halts at rte_memseg_walk() when
> maskbits is 0.
>
>
Can you explain this further?
Who is calling rte_eal_check_dma_mask with mask 0? is this a X86_64 system?
The only explanation I can find is the IOMMU hardware reporting mgaw=0 what
I would say is something completely wrong.


> > +             /*
> > +              * Dma mask precludes hugepage usage.
> > +              * This device can not be used and we do not need to keep
> > +              * the dma mask.
> > +              */
> > +             return 1;
> > +
> > +     /*
> > +      * we need to keep the more restricted maskbit for checking
> > +      * potential dynamic memory allocation in the future.
> > +      */
> > +     mcfg->dma_maskbits = mcfg->dma_maskbits == 0 ? maskbits :
> > +                          RTE_MIN(mcfg->dma_maskbits, maskbits);
> > +
> > +     return 0;
> > +}
> > +
> >  /* return the number of memory channels */  unsigned
> > rte_memory_get_nchannel(void)  { diff --git
> > a/lib/librte_eal/common/include/rte_eal_memconfig.h
> > b/lib/librte_eal/common/include/rte_eal_memconfig.h
> > index 62a21c2..b5dff70 100644
> > --- a/lib/librte_eal/common/include/rte_eal_memconfig.h
> > +++ b/lib/librte_eal/common/include/rte_eal_memconfig.h
> > @@ -81,6 +81,9 @@ struct rte_mem_config {
> >       /* legacy mem and single file segments options are shared */
> >       uint32_t legacy_mem;
> >       uint32_t single_file_segments;
> > +
> > +     /* keeps the more restricted dma mask */
> > +     uint8_t dma_maskbits;
> >  } __attribute__((__packed__));
> >
> >
> > diff --git a/lib/librte_eal/common/include/rte_memory.h
> > b/lib/librte_eal/common/include/rte_memory.h
> > index 14bd277..c349d6c 100644
> > --- a/lib/librte_eal/common/include/rte_memory.h
> > +++ b/lib/librte_eal/common/include/rte_memory.h
> > @@ -454,6 +454,9 @@ typedef int (*rte_memseg_list_walk_t)(const struct
> > rte_memseg_list *msl,
> >   */
> >  unsigned rte_memory_get_nrank(void);
> >
> > +/* check memsegs iovas are within a range based on dma mask */ int
> > +rte_eal_check_dma_mask(uint8_t maskbits);
> > +
> >  /**
> >   * Drivers based on uio will not load unless physical
> >   * addresses are obtainable. It is only possible to get diff --git
> > a/lib/librte_eal/common/malloc_heap.c
> > b/lib/librte_eal/common/malloc_heap.c
> > index ac7bbb3..3b5b2b6 100644
> > --- a/lib/librte_eal/common/malloc_heap.c
> > +++ b/lib/librte_eal/common/malloc_heap.c
> > @@ -259,11 +259,13 @@ struct malloc_elem *
> >               int socket, unsigned int flags, size_t align, size_t bound,
> >               bool contig, struct rte_memseg **ms, int n_segs)  {
> > +     struct rte_mem_config *mcfg =
> > rte_eal_get_configuration()->mem_config;
> >       struct rte_memseg_list *msl;
> >       struct malloc_elem *elem = NULL;
> >       size_t alloc_sz;
> >       int allocd_pages;
> >       void *ret, *map_addr;
> > +     uint64_t mask;
> >
> >       alloc_sz = (size_t)pg_sz * n_segs;
> >
> > @@ -291,6 +293,16 @@ struct malloc_elem *
> >               goto fail;
> >       }
> >
> > +     if (mcfg->dma_maskbits) {
> > +             mask = ~((1ULL << mcfg->dma_maskbits) - 1);
> > +             if (rte_eal_check_dma_mask(mask)) {
> > +                     RTE_LOG(ERR, EAL,
> > +                             "%s(): couldn't allocate memory due to DMA
> mask\n",
> > +                             __func__);
> > +                     goto fail;
> > +             }
> > +     }
> > +
> >       /* add newly minted memsegs to malloc heap */
> >       elem = malloc_heap_add_memory(heap, msl, map_addr, alloc_sz);
> >
> > diff --git a/lib/librte_eal/linuxapp/eal/eal.c
> > b/lib/librte_eal/linuxapp/eal/eal.c
> > index 4a55d3b..dfe1b8c 100644
> > --- a/lib/librte_eal/linuxapp/eal/eal.c
> > +++ b/lib/librte_eal/linuxapp/eal/eal.c
> > @@ -263,6 +263,8 @@ enum rte_iova_mode
> >        * processes could later map the config into this exact location */
> >       rte_config.mem_config->mem_cfg_addr = (uintptr_t)
> > rte_mem_cfg_addr;
> >
> > +     rte_config.mem_config->dma_maskbits = 0;
> > +
> >  }
> >
> >  /* attach to an existing shared memory config */ diff --git
> > a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> > index 73282bb..2baefce 100644
> > --- a/lib/librte_eal/rte_eal_version.map
> > +++ b/lib/librte_eal/rte_eal_version.map
> > @@ -291,6 +291,7 @@ EXPERIMENTAL {
> >       rte_devargs_parsef;
> >       rte_devargs_remove;
> >       rte_devargs_type_count;
> > +     rte_eal_check_dma_mask;
> >       rte_eal_cleanup;
> >       rte_eal_hotplug_add;
> >       rte_eal_hotplug_remove;
> > --
> > 1.9.1
>
>

Re: [dpdk-dev] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Tu, Lijuan]

Hi 

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alejandro Lucero
> Sent: Friday, October 5, 2018 8:45 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v3 1/6] mem: add function for checking
> memsegs IOVAs addresses
> 
> A device can suffer addressing limitations. This function checks memsegs
> have iovas within the supported range based on dma mask.
> 
> PMDs should use this function during initialization if device suffers
> addressing limitations, returning an error if this function returns memsegs
> out of range.
> 
> Another usage is for emulated IOMMU hardware with addressing limitations.
> 
> It is necessary to save the most restricted dma mask for checking out
> memory allocated dynamically after initialization.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
>  doc/guides/rel_notes/release_18_11.rst            | 10 ++++
>  lib/librte_eal/common/eal_common_memory.c         | 60
> +++++++++++++++++++++++
>  lib/librte_eal/common/include/rte_eal_memconfig.h |  3 ++
>  lib/librte_eal/common/include/rte_memory.h        |  3 ++
>  lib/librte_eal/common/malloc_heap.c               | 12 +++++
>  lib/librte_eal/linuxapp/eal/eal.c                 |  2 +
>  lib/librte_eal/rte_eal_version.map                |  1 +
>  7 files changed, 91 insertions(+)
> 
> diff --git a/doc/guides/rel_notes/release_18_11.rst
> b/doc/guides/rel_notes/release_18_11.rst
> index 2133a5b..c806dc6 100644
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -104,6 +104,14 @@ New Features
>    the specified port. The port must be stopped before the command call in
> order
>    to reconfigure queues.
> 
> +* **Added check for ensuring allocated memory addressable by devices.**
> +
> +  Some devices can have addressing limitations so a new function,
> + ``rte_eal_check_dma_mask``, has been added for checking allocated
> + memory is  not out of the device range. Because now memory can be
> + dynamically allocated  after initialization, a dma mask is kept and
> + any new allocated memory will be  checked out against that dma mask
> + and rejected if out of range. If more than  one device has addressing
> limitations, the dma mask is the more restricted one.
> 
>  API Changes
>  -----------
> @@ -156,6 +164,8 @@ ABI Changes
>         ``rte_config`` structure on account of improving DPDK usability
> when
>         using either ``--legacy-mem`` or ``--single-file-segments`` flags.
> 
> +* eal: added ``dma_maskbits`` to ``rte_mem_config`` for keeping more
> restricted
> +       dma mask based on devices addressing limitations.
> 
>  Removed Items
>  -------------
> diff --git a/lib/librte_eal/common/eal_common_memory.c
> b/lib/librte_eal/common/eal_common_memory.c
> index 0b69804..c482f0d 100644
> --- a/lib/librte_eal/common/eal_common_memory.c
> +++ b/lib/librte_eal/common/eal_common_memory.c
> @@ -385,6 +385,66 @@ struct virtiova {
>  	rte_memseg_walk(dump_memseg, f);
>  }
> 
> +static int
> +check_iova(const struct rte_memseg_list *msl __rte_unused,
> +		const struct rte_memseg *ms, void *arg) {
> +	uint64_t *mask = arg;
> +	rte_iova_t iova;
> +
> +	/* higher address within segment */
> +	iova = (ms->iova + ms->len) - 1;
> +	if (!(iova & *mask))
> +		return 0;
> +
> +	RTE_LOG(DEBUG, EAL, "memseg iova %"PRIx64", len %zx, out of
> range\n",
> +			    ms->iova, ms->len);
> +
> +	RTE_LOG(DEBUG, EAL, "\tusing dma mask %"PRIx64"\n", *mask);
> +	return 1;
> +}
> +
> +#if defined(RTE_ARCH_64)
> +#define MAX_DMA_MASK_BITS 63
> +#else
> +#define MAX_DMA_MASK_BITS 31
> +#endif
> +
> +/* check memseg iovas are within the required range based on dma mask
> +*/ int __rte_experimental rte_eal_check_dma_mask(uint8_t maskbits) {
> +	struct rte_mem_config *mcfg =
> rte_eal_get_configuration()->mem_config;
> +	uint64_t mask;
> +
> +	/* sanity check */
> +	if (maskbits > MAX_DMA_MASK_BITS) {
> +		RTE_LOG(ERR, EAL, "wrong dma mask size %u (Max: %u)\n",
> +				   maskbits, MAX_DMA_MASK_BITS);
> +		return -1;
> +	}
> +
> +	/* create dma mask */
> +	mask = ~((1ULL << maskbits) - 1);
> +
> +	if (rte_memseg_walk(check_iova, &mask))

[Lijuan]In my environment, testpmd halts at rte_memseg_walk() when maskbits is 0.

> +		/*
> +		 * Dma mask precludes hugepage usage.
> +		 * This device can not be used and we do not need to keep
> +		 * the dma mask.
> +		 */
> +		return 1;
> +
> +	/*
> +	 * we need to keep the more restricted maskbit for checking
> +	 * potential dynamic memory allocation in the future.
> +	 */
> +	mcfg->dma_maskbits = mcfg->dma_maskbits == 0 ? maskbits :
> +			     RTE_MIN(mcfg->dma_maskbits, maskbits);
> +
> +	return 0;
> +}
> +
>  /* return the number of memory channels */  unsigned
> rte_memory_get_nchannel(void)  { diff --git
> a/lib/librte_eal/common/include/rte_eal_memconfig.h
> b/lib/librte_eal/common/include/rte_eal_memconfig.h
> index 62a21c2..b5dff70 100644
> --- a/lib/librte_eal/common/include/rte_eal_memconfig.h
> +++ b/lib/librte_eal/common/include/rte_eal_memconfig.h
> @@ -81,6 +81,9 @@ struct rte_mem_config {
>  	/* legacy mem and single file segments options are shared */
>  	uint32_t legacy_mem;
>  	uint32_t single_file_segments;
> +
> +	/* keeps the more restricted dma mask */
> +	uint8_t dma_maskbits;
>  } __attribute__((__packed__));
> 
> 
> diff --git a/lib/librte_eal/common/include/rte_memory.h
> b/lib/librte_eal/common/include/rte_memory.h
> index 14bd277..c349d6c 100644
> --- a/lib/librte_eal/common/include/rte_memory.h
> +++ b/lib/librte_eal/common/include/rte_memory.h
> @@ -454,6 +454,9 @@ typedef int (*rte_memseg_list_walk_t)(const struct
> rte_memseg_list *msl,
>   */
>  unsigned rte_memory_get_nrank(void);
> 
> +/* check memsegs iovas are within a range based on dma mask */ int
> +rte_eal_check_dma_mask(uint8_t maskbits);
> +
>  /**
>   * Drivers based on uio will not load unless physical
>   * addresses are obtainable. It is only possible to get diff --git
> a/lib/librte_eal/common/malloc_heap.c
> b/lib/librte_eal/common/malloc_heap.c
> index ac7bbb3..3b5b2b6 100644
> --- a/lib/librte_eal/common/malloc_heap.c
> +++ b/lib/librte_eal/common/malloc_heap.c
> @@ -259,11 +259,13 @@ struct malloc_elem *
>  		int socket, unsigned int flags, size_t align, size_t bound,
>  		bool contig, struct rte_memseg **ms, int n_segs)  {
> +	struct rte_mem_config *mcfg =
> rte_eal_get_configuration()->mem_config;
>  	struct rte_memseg_list *msl;
>  	struct malloc_elem *elem = NULL;
>  	size_t alloc_sz;
>  	int allocd_pages;
>  	void *ret, *map_addr;
> +	uint64_t mask;
> 
>  	alloc_sz = (size_t)pg_sz * n_segs;
> 
> @@ -291,6 +293,16 @@ struct malloc_elem *
>  		goto fail;
>  	}
> 
> +	if (mcfg->dma_maskbits) {
> +		mask = ~((1ULL << mcfg->dma_maskbits) - 1);
> +		if (rte_eal_check_dma_mask(mask)) {
> +			RTE_LOG(ERR, EAL,
> +				"%s(): couldn't allocate memory due to DMA mask\n",
> +				__func__);
> +			goto fail;
> +		}
> +	}
> +
>  	/* add newly minted memsegs to malloc heap */
>  	elem = malloc_heap_add_memory(heap, msl, map_addr, alloc_sz);
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal.c
> b/lib/librte_eal/linuxapp/eal/eal.c
> index 4a55d3b..dfe1b8c 100644
> --- a/lib/librte_eal/linuxapp/eal/eal.c
> +++ b/lib/librte_eal/linuxapp/eal/eal.c
> @@ -263,6 +263,8 @@ enum rte_iova_mode
>  	 * processes could later map the config into this exact location */
>  	rte_config.mem_config->mem_cfg_addr = (uintptr_t)
> rte_mem_cfg_addr;
> 
> +	rte_config.mem_config->dma_maskbits = 0;
> +
>  }
> 
>  /* attach to an existing shared memory config */ diff --git
> a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> index 73282bb..2baefce 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -291,6 +291,7 @@ EXPERIMENTAL {
>  	rte_devargs_parsef;
>  	rte_devargs_remove;
>  	rte_devargs_type_count;
> +	rte_eal_check_dma_mask;
>  	rte_eal_cleanup;
>  	rte_eal_hotplug_add;
>  	rte_eal_hotplug_remove;
> --
> 1.9.1

[dpdk-dev] [PATCH v3 1/6] mem: add function for checking memsegs IOVAs addresses

[Alejandro Lucero]

A device can suffer addressing limitations. This function checks
memsegs have iovas within the supported range based on dma mask.

PMDs should use this function during initialization if device
suffers addressing limitations, returning an error if this function
returns memsegs out of range.

Another usage is for emulated IOMMU hardware with addressing
limitations.

It is necessary to save the most restricted dma mask for checking out
memory allocated dynamically after initialization.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 doc/guides/rel_notes/release_18_11.rst            | 10 ++++
 lib/librte_eal/common/eal_common_memory.c         | 60 +++++++++++++++++++++++
 lib/librte_eal/common/include/rte_eal_memconfig.h |  3 ++
 lib/librte_eal/common/include/rte_memory.h        |  3 ++
 lib/librte_eal/common/malloc_heap.c               | 12 +++++
 lib/librte_eal/linuxapp/eal/eal.c                 |  2 +
 lib/librte_eal/rte_eal_version.map                |  1 +
 7 files changed, 91 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index 2133a5b..c806dc6 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -104,6 +104,14 @@ New Features
   the specified port. The port must be stopped before the command call in order
   to reconfigure queues.
 
+* **Added check for ensuring allocated memory addressable by devices.**
+
+  Some devices can have addressing limitations so a new function,
+  ``rte_eal_check_dma_mask``, has been added for checking allocated memory is
+  not out of the device range. Because now memory can be dynamically allocated
+  after initialization, a dma mask is kept and any new allocated memory will be
+  checked out against that dma mask and rejected if out of range. If more than
+  one device has addressing limitations, the dma mask is the more restricted one.
 
 API Changes
 -----------
@@ -156,6 +164,8 @@ ABI Changes
        ``rte_config`` structure on account of improving DPDK usability when
        using either ``--legacy-mem`` or ``--single-file-segments`` flags.
 
+* eal: added ``dma_maskbits`` to ``rte_mem_config`` for keeping more restricted
+       dma mask based on devices addressing limitations.
 
 Removed Items
 -------------
diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/common/eal_common_memory.c
index 0b69804..c482f0d 100644
--- a/lib/librte_eal/common/eal_common_memory.c
+++ b/lib/librte_eal/common/eal_common_memory.c
@@ -385,6 +385,66 @@ struct virtiova {
 	rte_memseg_walk(dump_memseg, f);
 }
 
+static int
+check_iova(const struct rte_memseg_list *msl __rte_unused,
+		const struct rte_memseg *ms, void *arg)
+{
+	uint64_t *mask = arg;
+	rte_iova_t iova;
+
+	/* higher address within segment */
+	iova = (ms->iova + ms->len) - 1;
+	if (!(iova & *mask))
+		return 0;
+
+	RTE_LOG(DEBUG, EAL, "memseg iova %"PRIx64", len %zx, out of range\n",
+			    ms->iova, ms->len);
+
+	RTE_LOG(DEBUG, EAL, "\tusing dma mask %"PRIx64"\n", *mask);
+	return 1;
+}
+
+#if defined(RTE_ARCH_64)
+#define MAX_DMA_MASK_BITS 63
+#else
+#define MAX_DMA_MASK_BITS 31
+#endif
+
+/* check memseg iovas are within the required range based on dma mask */
+int __rte_experimental
+rte_eal_check_dma_mask(uint8_t maskbits)
+{
+	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
+	uint64_t mask;
+
+	/* sanity check */
+	if (maskbits > MAX_DMA_MASK_BITS) {
+		RTE_LOG(ERR, EAL, "wrong dma mask size %u (Max: %u)\n",
+				   maskbits, MAX_DMA_MASK_BITS);
+		return -1;
+	}
+
+	/* create dma mask */
+	mask = ~((1ULL << maskbits) - 1);
+
+	if (rte_memseg_walk(check_iova, &mask))
+		/*
+		 * Dma mask precludes hugepage usage.
+		 * This device can not be used and we do not need to keep
+		 * the dma mask.
+		 */
+		return 1;
+
+	/*
+	 * we need to keep the more restricted maskbit for checking
+	 * potential dynamic memory allocation in the future.
+	 */
+	mcfg->dma_maskbits = mcfg->dma_maskbits == 0 ? maskbits :
+			     RTE_MIN(mcfg->dma_maskbits, maskbits);
+
+	return 0;
+}
+
 /* return the number of memory channels */
 unsigned rte_memory_get_nchannel(void)
 {
diff --git a/lib/librte_eal/common/include/rte_eal_memconfig.h b/lib/librte_eal/common/include/rte_eal_memconfig.h
index 62a21c2..b5dff70 100644
--- a/lib/librte_eal/common/include/rte_eal_memconfig.h
+++ b/lib/librte_eal/common/include/rte_eal_memconfig.h
@@ -81,6 +81,9 @@ struct rte_mem_config {
 	/* legacy mem and single file segments options are shared */
 	uint32_t legacy_mem;
 	uint32_t single_file_segments;
+
+	/* keeps the more restricted dma mask */
+	uint8_t dma_maskbits;
 } __attribute__((__packed__));
 
 
diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/common/include/rte_memory.h
index 14bd277..c349d6c 100644
--- a/lib/librte_eal/common/include/rte_memory.h
+++ b/lib/librte_eal/common/include/rte_memory.h
@@ -454,6 +454,9 @@ typedef int (*rte_memseg_list_walk_t)(const struct rte_memseg_list *msl,
  */
 unsigned rte_memory_get_nrank(void);
 
+/* check memsegs iovas are within a range based on dma mask */
+int rte_eal_check_dma_mask(uint8_t maskbits);
+
 /**
  * Drivers based on uio will not load unless physical
  * addresses are obtainable. It is only possible to get
diff --git a/lib/librte_eal/common/malloc_heap.c b/lib/librte_eal/common/malloc_heap.c
index ac7bbb3..3b5b2b6 100644
--- a/lib/librte_eal/common/malloc_heap.c
+++ b/lib/librte_eal/common/malloc_heap.c
@@ -259,11 +259,13 @@ struct malloc_elem *
 		int socket, unsigned int flags, size_t align, size_t bound,
 		bool contig, struct rte_memseg **ms, int n_segs)
 {
+	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
 	struct rte_memseg_list *msl;
 	struct malloc_elem *elem = NULL;
 	size_t alloc_sz;
 	int allocd_pages;
 	void *ret, *map_addr;
+	uint64_t mask;
 
 	alloc_sz = (size_t)pg_sz * n_segs;
 
@@ -291,6 +293,16 @@ struct malloc_elem *
 		goto fail;
 	}
 
+	if (mcfg->dma_maskbits) {
+		mask = ~((1ULL << mcfg->dma_maskbits) - 1);
+		if (rte_eal_check_dma_mask(mask)) {
+			RTE_LOG(ERR, EAL,
+				"%s(): couldn't allocate memory due to DMA mask\n",
+				__func__);
+			goto fail;
+		}
+	}
+
 	/* add newly minted memsegs to malloc heap */
 	elem = malloc_heap_add_memory(heap, msl, map_addr, alloc_sz);
 
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 4a55d3b..dfe1b8c 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -263,6 +263,8 @@ enum rte_iova_mode
 	 * processes could later map the config into this exact location */
 	rte_config.mem_config->mem_cfg_addr = (uintptr_t) rte_mem_cfg_addr;
 
+	rte_config.mem_config->dma_maskbits = 0;
+
 }
 
 /* attach to an existing shared memory config */
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index 73282bb..2baefce 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -291,6 +291,7 @@ EXPERIMENTAL {
 	rte_devargs_parsef;
 	rte_devargs_remove;
 	rte_devargs_type_count;
+	rte_eal_check_dma_mask;
 	rte_eal_cleanup;
 	rte_eal_hotplug_add;
 	rte_eal_hotplug_remove;
-- 
1.9.1