DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask
@ 2018-07-10 17:25 Alejandro Lucero
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 1/5] mem: add function for checking memsegs IOVAs addresses Alejandro Lucero
                   ` (5 more replies)
  0 siblings, 6 replies; 16+ messages in thread
From: Alejandro Lucero @ 2018-07-10 17:25 UTC (permalink / raw)
  To: dev; +Cc: stable, anatoly.burakov

This patchset adds, mainly, a check for ensuring IOVAs are within a
restricted range due to addressing limitations with some devices. There
are two known cases: NFP and IOMMU VT-d emulation.

With this check IOVAs out of range are detected and PMDs can abort
initialization. For the VT-d case, IOVA VA mode is allowed as long as
IOVAs are within the supported range, avoiding to forbid IOVA VA by
default.

For the addressing limitations known cases, there are just 40(NFP) or
39(VT-d) bits for handling IOVAs. When using IOVA PA, those limitations
imply 1TB(NFP) or 512M(VT-d) as upper limits, which is likely enough for
most systems. With machines using more memory, the added check will
ensure IOVAs within the range.

With IOVA VA, and because the way the Linux kernel serves mmap calls
in 64 bits systems, 39 or 40 bits are not enough. It is possible to
give an address hint with a lower starting address than the default one
used by the kernel, and then ensuring the mmap uses that hint or hint plus
some offset. With 64 bits systems, the process virtual address space is
large enoguh for doing the hugepages mmaping within the supported range
when those addressing limitations exist. This patchset also adds a change
for using such a hint making the use of IOVA VA a more than likely
possibility when there are those addressing limitations.

The check is not done by default but just when it is required. This
patchset adds the check for NFP initialization and for setting the IOVA
mode is an emulated VT-d is detected.

This patchset applies on 17.11.3.

Similar changes will be submitted to main DPDK branch soon.

v2:
 - add get_addr_hint function
 - call munmap when hint given and not used by mmap
 - create dma mask in one step
 - refactor logs

v3:
 - add new API functions to map files

v4:
 - add sanity check for dma mask bits
 - remove rte_eth_dev_check_dma_mask

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v4 1/5] mem: add function for checking memsegs IOVAs addresses
  2018-07-10 17:25 [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Alejandro Lucero
@ 2018-07-10 17:25 ` Alejandro Lucero
  2018-07-11 10:12   ` [dpdk-dev] [dpdk-stable] " Eelco Chaudron
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 2/5] bus/pci: use IOVAs check when setting IOVA mode Alejandro Lucero
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 16+ messages in thread
From: Alejandro Lucero @ 2018-07-10 17:25 UTC (permalink / raw)
  To: dev; +Cc: stable, anatoly.burakov

A device can suffer addressing limitations. This functions checks
memsegs have iovas within the supported range based on dma mask.

PMD should use this during initialization if supported devices
suffer addressing limitations, returning an error if this function
returns memsegs out of range.

Another potential usage is for emulated IOMMU hardware with addressing
limitations.

Applicable to v17.11.3 only.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/eal_common_memory.c  | 48 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/include/rte_memory.h |  3 ++
 lib/librte_eal/rte_eal_version.map         |  1 +
 3 files changed, 52 insertions(+)

diff --git a/lib/librte_eal/common/eal_common_memory.c b/lib/librte_eal/common/eal_common_memory.c
index fc6c44d..00ab393 100644
--- a/lib/librte_eal/common/eal_common_memory.c
+++ b/lib/librte_eal/common/eal_common_memory.c
@@ -109,6 +109,54 @@
 	}
 }
 
+#if defined(RTE_ARCH_X86)
+#define X86_VA_WIDTH 47 /* From Documentation/x86/x86_64/mm.txt */
+#define MAX_DMA_MASK_BITS X86_VA_WIDTH
+#else
+/* 63 bits is good enough for a sanity check */
+#define MAX_DMA_MASK_BITS 63
+#endif
+
+/* check memseg iovas are within the required range based on dma mask */
+int
+rte_eal_check_dma_mask(uint8_t maskbits)
+{
+
+	const struct rte_mem_config *mcfg;
+	uint64_t mask;
+	int i;
+
+	/* sanity check */
+	if (maskbits > MAX_DMA_MASK_BITS) {
+		RTE_LOG(INFO, EAL, "wrong dma mask size %u (Max: %u)\n",
+				   maskbits, MAX_DMA_MASK_BITS);
+		return -1;
+	}
+
+	/* create dma mask */
+	mask = ~((1ULL << maskbits) - 1);
+
+	/* get pointer to global configuration */
+	mcfg = rte_eal_get_configuration()->mem_config;
+
+	for (i = 0; i < RTE_MAX_MEMSEG; i++) {
+		if (mcfg->memseg[i].addr == NULL)
+			break;
+
+		if (mcfg->memseg[i].iova & mask) {
+			RTE_LOG(INFO, EAL,
+				"memseg[%d] iova %"PRIx64" out of range:\n",
+				i, mcfg->memseg[i].iova);
+
+			RTE_LOG(INFO, EAL, "\tusing dma mask %"PRIx64"\n",
+				mask);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
 /* return the number of memory channels */
 unsigned rte_memory_get_nchannel(void)
 {
diff --git a/lib/librte_eal/common/include/rte_memory.h b/lib/librte_eal/common/include/rte_memory.h
index 80a8fc0..b2a0168 100644
--- a/lib/librte_eal/common/include/rte_memory.h
+++ b/lib/librte_eal/common/include/rte_memory.h
@@ -209,6 +209,9 @@ struct rte_memseg {
  */
 unsigned rte_memory_get_nrank(void);
 
+/* check memsegs iovas are within a range based on dma mask */
+int rte_eal_check_dma_mask(uint8_t maskbits);
+
 /**
  * Drivers based on uio will not load unless physical
  * addresses are obtainable. It is only possible to get
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index f4f46c1..aa6cf87 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -184,6 +184,7 @@ DPDK_17.11 {
 
 	rte_eal_create_uio_dev;
 	rte_bus_get_iommu_class;
+	rte_eal_check_dma_mask;
 	rte_eal_has_pci;
 	rte_eal_iova_mode;
 	rte_eal_mbuf_default_mempool_ops;
-- 
1.9.1

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v4 2/5] bus/pci: use IOVAs check when setting IOVA mode
  2018-07-10 17:25 [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Alejandro Lucero
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 1/5] mem: add function for checking memsegs IOVAs addresses Alejandro Lucero
@ 2018-07-10 17:25 ` Alejandro Lucero
  2018-07-11 10:18   ` [dpdk-dev] [dpdk-stable] " Eelco Chaudron
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 3/5] mem: use address hint for mapping hugepages Alejandro Lucero
                   ` (3 subsequent siblings)
  5 siblings, 1 reply; 16+ messages in thread
From: Alejandro Lucero @ 2018-07-10 17:25 UTC (permalink / raw)
  To: dev; +Cc: stable, anatoly.burakov

Although VT-d emulation currently only supports 39 bits, it could
be iovas being within that supported range. This patch allows
IOVA mode in such a case.

Indeed, memory initialization code can be modified for using lower
virtual addresses than those used by the kernel for 64 bits processes
by default, and therefore memsegs iovas can use 39 bits or less for
most system. And this is likely 100% true for VMs.

Applicable to v17.11.3 only.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/bus/pci/linux/pci.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 74deef3..792c819 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -43,6 +43,7 @@
 #include <rte_devargs.h>
 #include <rte_memcpy.h>
 #include <rte_vfio.h>
+#include <rte_memory.h>
 
 #include "eal_private.h"
 #include "eal_filesystem.h"
@@ -613,10 +614,12 @@
 	fclose(fp);
 
 	mgaw = ((vtd_cap_reg & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT) + 1;
-	if (mgaw < X86_VA_WIDTH)
+
+	if (!rte_eal_check_dma_mask(mgaw))
+		return true;
+	else
 		return false;
 
-	return true;
 }
 #elif defined(RTE_ARCH_PPC_64)
 static bool
@@ -640,13 +643,17 @@
 {
 	struct rte_pci_device *dev = NULL;
 	struct rte_pci_driver *drv = NULL;
+	int iommu_dma_mask_check_done = 0;
 
 	FOREACH_DRIVER_ON_PCIBUS(drv) {
 		FOREACH_DEVICE_ON_PCIBUS(dev) {
 			if (!rte_pci_match(drv, dev))
 				continue;
-			if (!pci_one_device_iommu_support_va(dev))
-				return false;
+			if (!iommu_dma_mask_check_done) {
+				if (pci_one_device_iommu_support_va(dev) < 0)
+					return false;
+				iommu_dma_mask_check_done  = 1;
+			}
 		}
 	}
 	return true;
-- 
1.9.1

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v4 3/5] mem: use address hint for mapping hugepages
  2018-07-10 17:25 [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Alejandro Lucero
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 1/5] mem: add function for checking memsegs IOVAs addresses Alejandro Lucero
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 2/5] bus/pci: use IOVAs check when setting IOVA mode Alejandro Lucero
@ 2018-07-10 17:25 ` Alejandro Lucero
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 4/5] net/nfp: check hugepages IOVAs based on DMA mask Alejandro Lucero
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 16+ messages in thread
From: Alejandro Lucero @ 2018-07-10 17:25 UTC (permalink / raw)
  To: dev; +Cc: stable, anatoly.burakov

Linux kernel uses a really high address as starting address for
serving mmaps calls. If there exists addressing limitations and
IOVA mode is VA, this starting address is likely too high for
those devices. However, it is possible to use a lower address in
the process virtual address space as with 64 bits there is a lot
of available space.

This patch adds an address hint as starting address for 64 bits
systems.

Applicable to v17.11.3 only.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Eelco Chaudron <echaudro@redhat.com>
---
 lib/librte_eal/linuxapp/eal/eal_memory.c | 55 ++++++++++++++++++++++++++------
 1 file changed, 46 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 17c20d4..2ed4017 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -88,6 +88,23 @@
 
 static uint64_t baseaddr_offset;
 
+#ifdef RTE_ARCH_64
+/*
+ * Linux kernel uses a really high address as starting address for serving
+ * mmaps calls. If there exists addressing limitations and IOVA mode is VA,
+ * this starting address is likely too high for those devices. However, it
+ * is possible to use a lower address in the process virtual address space
+ * as with 64 bits there is a lot of available space.
+ *
+ * Current known limitations are 39 or 40 bits. Setting the starting address
+ * at 4GB implies there are 508GB or 1020GB for mapping the available
+ * hugepages. This is likely enough for most systems, although a device with
+ * addressing limitations should call rte_dev_check_dma_mask for ensuring all
+ * memory is within supported range.
+ */
+static uint64_t baseaddr = 0x100000000;
+#endif
+
 static bool phys_addrs_available = true;
 
 #define RANDOMIZE_VA_SPACE_FILE "/proc/sys/kernel/randomize_va_space"
@@ -250,6 +267,23 @@
 	}
 }
 
+static void *
+get_addr_hint(void)
+{
+	if (internal_config.base_virtaddr != 0) {
+		return (void *) (uintptr_t)
+			    (internal_config.base_virtaddr +
+			     baseaddr_offset);
+	} else {
+#ifdef RTE_ARCH_64
+		return (void *) (uintptr_t) (baseaddr +
+				baseaddr_offset);
+#else
+		return NULL;
+#endif
+	}
+}
+
 /*
  * Try to mmap *size bytes in /dev/zero. If it is successful, return the
  * pointer to the mmap'd area and keep *size unmodified. Else, retry
@@ -260,16 +294,10 @@
 static void *
 get_virtual_area(size_t *size, size_t hugepage_sz)
 {
-	void *addr;
+	void *addr, *addr_hint;
 	int fd;
 	long aligned_addr;
 
-	if (internal_config.base_virtaddr != 0) {
-		addr = (void*) (uintptr_t) (internal_config.base_virtaddr +
-				baseaddr_offset);
-	}
-	else addr = NULL;
-
 	RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
 
 	fd = open("/dev/zero", O_RDONLY);
@@ -278,7 +306,9 @@
 		return NULL;
 	}
 	do {
-		addr = mmap(addr,
+		addr_hint = get_addr_hint();
+
+		addr = mmap(addr_hint,
 				(*size) + hugepage_sz, PROT_READ,
 #ifdef RTE_ARCH_PPC_64
 				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
@@ -286,8 +316,15 @@
 				MAP_PRIVATE,
 #endif
 				fd, 0);
-		if (addr == MAP_FAILED)
+		if (addr == MAP_FAILED) {
+			/* map failed. Let's try with less memory */
 			*size -= hugepage_sz;
+		} else if (addr_hint && addr != addr_hint) {
+			/* hint was not used. Try with another offset */
+			munmap(addr, (*size) + hugepage_sz);
+			addr = MAP_FAILED;
+			baseaddr_offset += 0x100000000;
+		}
 	} while (addr == MAP_FAILED && *size > 0);
 
 	if (addr == MAP_FAILED) {
-- 
1.9.1

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v4 4/5] net/nfp: check hugepages IOVAs based on DMA mask
  2018-07-10 17:25 [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Alejandro Lucero
                   ` (2 preceding siblings ...)
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 3/5] mem: use address hint for mapping hugepages Alejandro Lucero
@ 2018-07-10 17:25 ` Alejandro Lucero
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 5/5] net/nfp: support IOVA VA mode Alejandro Lucero
  2018-07-26 15:41 ` [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Thomas Monjalon
  5 siblings, 0 replies; 16+ messages in thread
From: Alejandro Lucero @ 2018-07-10 17:25 UTC (permalink / raw)
  To: dev; +Cc: stable, anatoly.burakov

NFP devices can not handle DMA addresses requiring more than
40 bits. This patch uses rte_dev_check_dma_mask with 40 bits
and avoids device initialization if memory out of NFP range.

Applicable to v17.11.3 only.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
Acked-by: Eelco Chaudron <echaudro@redhat.com>
---
 drivers/net/nfp/nfp_net.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index d9cd047..8fc1b8f 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2649,6 +2649,14 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 
 	pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 
+	/* NFP can not handle DMA addresses requiring more than 40 bits */
+	if (rte_eal_check_dma_mask(40) < 0) {
+		RTE_LOG(INFO, PMD, "device %s can not be used:",
+				   pci_dev->device.name);
+		RTE_LOG(INFO, PMD, "\trestricted dma mask to 40 bits!\n");
+		return -ENODEV;
+	};
+
 	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
 	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
 		port = get_pf_port_number(eth_dev->data->name);
-- 
1.9.1

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v4 5/5] net/nfp: support IOVA VA mode
  2018-07-10 17:25 [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Alejandro Lucero
                   ` (3 preceding siblings ...)
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 4/5] net/nfp: check hugepages IOVAs based on DMA mask Alejandro Lucero
@ 2018-07-10 17:25 ` Alejandro Lucero
  2018-07-26 15:41 ` [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Thomas Monjalon
  5 siblings, 0 replies; 16+ messages in thread
From: Alejandro Lucero @ 2018-07-10 17:25 UTC (permalink / raw)
  To: dev; +Cc: stable, anatoly.burakov

NFP can handle IOVA as VA. It requires to check those IOVAs
being in the supported range what is done during initialization.

Applicable to v17.11.3 only.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
Acked-by: Eelco Chaudron <echaudro@redhat.com>
---
 drivers/net/nfp/nfp_net.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 8fc1b8f..98098f6 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -3053,14 +3053,16 @@ static int eth_nfp_pci_remove(struct rte_pci_device *pci_dev)
 
 static struct rte_pci_driver rte_nfp_net_pf_pmd = {
 	.id_table = pci_id_nfp_pf_net_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
 	.probe = nfp_pf_pci_probe,
 	.remove = eth_nfp_pci_remove,
 };
 
 static struct rte_pci_driver rte_nfp_net_vf_pmd = {
 	.id_table = pci_id_nfp_vf_net_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+		     RTE_PCI_DRV_IOVA_AS_VA,
 	.probe = eth_nfp_pci_probe,
 	.remove = eth_nfp_pci_remove,
 };
-- 
1.9.1

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [dpdk-stable] [PATCH v4 1/5] mem: add function for checking memsegs IOVAs addresses
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 1/5] mem: add function for checking memsegs IOVAs addresses Alejandro Lucero
@ 2018-07-11 10:12   ` Eelco Chaudron
  0 siblings, 0 replies; 16+ messages in thread
From: Eelco Chaudron @ 2018-07-11 10:12 UTC (permalink / raw)
  To: Alejandro Lucero; +Cc: dev, stable, anatoly.burakov



On 10 Jul 2018, at 19:25, Alejandro Lucero wrote:

> A device can suffer addressing limitations. This functions checks
> memsegs have iovas within the supported range based on dma mask.
>
> PMD should use this during initialization if supported devices
> suffer addressing limitations, returning an error if this function
> returns memsegs out of range.
>
> Another potential usage is for emulated IOMMU hardware with addressing
> limitations.
>
> Applicable to v17.11.3 only.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
Looks good to me.

Acked-by: Eelco Chaudron <echaudro@redhat.com>

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [dpdk-stable] [PATCH v4 2/5] bus/pci: use IOVAs check when setting IOVA mode
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 2/5] bus/pci: use IOVAs check when setting IOVA mode Alejandro Lucero
@ 2018-07-11 10:18   ` Eelco Chaudron
  2018-07-11 10:41     ` Burakov, Anatoly
  0 siblings, 1 reply; 16+ messages in thread
From: Eelco Chaudron @ 2018-07-11 10:18 UTC (permalink / raw)
  To: Alejandro Lucero; +Cc: dev, stable, anatoly.burakov



On 10 Jul 2018, at 19:25, Alejandro Lucero wrote:

> Although VT-d emulation currently only supports 39 bits, it could
> be iovas being within that supported range. This patch allows
> IOVA mode in such a case.
>
> Indeed, memory initialization code can be modified for using lower
> virtual addresses than those used by the kernel for 64 bits processes
> by default, and therefore memsegs iovas can use 39 bits or less for
> most system. And this is likely 100% true for VMs.
>
> Applicable to v17.11.3 only.
>
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> ---
>  drivers/bus/pci/linux/pci.c | 15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
> index 74deef3..792c819 100644
> --- a/drivers/bus/pci/linux/pci.c
> +++ b/drivers/bus/pci/linux/pci.c
> @@ -43,6 +43,7 @@
>  #include <rte_devargs.h>
>  #include <rte_memcpy.h>
>  #include <rte_vfio.h>
> +#include <rte_memory.h>
>
>  #include "eal_private.h"
>  #include "eal_filesystem.h"
> @@ -613,10 +614,12 @@
>  	fclose(fp);
>
>  	mgaw = ((vtd_cap_reg & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT) + 
> 1;
> -	if (mgaw < X86_VA_WIDTH)
> +
> +	if (!rte_eal_check_dma_mask(mgaw))
> +		return true;
> +	else
>  		return false;
>
> -	return true;
>  }
>  #elif defined(RTE_ARCH_PPC_64)
>  static bool
> @@ -640,13 +643,17 @@
>  {
>  	struct rte_pci_device *dev = NULL;
>  	struct rte_pci_driver *drv = NULL;
> +	int iommu_dma_mask_check_done = 0;
>
>  	FOREACH_DRIVER_ON_PCIBUS(drv) {
>  		FOREACH_DEVICE_ON_PCIBUS(dev) {
>  			if (!rte_pci_match(drv, dev))
>  				continue;
> -			if (!pci_one_device_iommu_support_va(dev))
> -				return false;
> +			if (!iommu_dma_mask_check_done) {
> +				if (pci_one_device_iommu_support_va(dev) < 0)
> +					return false;
> +				iommu_dma_mask_check_done  = 1;

As I do not know enough on what IOMMU hardware can coexist, I leave this 
patch for others to review.
Here is the previous question/answer:

>>> Not sure why this change? Why do we only need to check one device on 
>>> all the buses?

>> Because there is just one emulated IOMMU hardware. The limitation in 
>> this case is not in a specific PCI device. And I do not think it is 
>> possible to have two different (emulated or not) IOMMU hardware. Yes, 
>> you can have more than one controller but being same IOMMU type.

If the above is confirmed, you can consider this patch ack’ed.

> +			}
>  		}
>  	}
>  	return true;
> -- 
> 1.9.1

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [dpdk-stable] [PATCH v4 2/5] bus/pci: use IOVAs check when setting IOVA mode
  2018-07-11 10:18   ` [dpdk-dev] [dpdk-stable] " Eelco Chaudron
@ 2018-07-11 10:41     ` Burakov, Anatoly
  0 siblings, 0 replies; 16+ messages in thread
From: Burakov, Anatoly @ 2018-07-11 10:41 UTC (permalink / raw)
  To: Eelco Chaudron, Alejandro Lucero; +Cc: dev, stable

On 11-Jul-18 11:18 AM, Eelco Chaudron wrote:
> 
> 
> On 10 Jul 2018, at 19:25, Alejandro Lucero wrote:
> 
>> Although VT-d emulation currently only supports 39 bits, it could
>> be iovas being within that supported range. This patch allows
>> IOVA mode in such a case.
>>
>> Indeed, memory initialization code can be modified for using lower
>> virtual addresses than those used by the kernel for 64 bits processes
>> by default, and therefore memsegs iovas can use 39 bits or less for
>> most system. And this is likely 100% true for VMs.
>>
>> Applicable to v17.11.3 only.
>>
>> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>> ---
>>  drivers/bus/pci/linux/pci.c | 15 +++++++++++----
>>  1 file changed, 11 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
>> index 74deef3..792c819 100644
>> --- a/drivers/bus/pci/linux/pci.c
>> +++ b/drivers/bus/pci/linux/pci.c
>> @@ -43,6 +43,7 @@
>>  #include <rte_devargs.h>
>>  #include <rte_memcpy.h>
>>  #include <rte_vfio.h>
>> +#include <rte_memory.h>
>>
>>  #include "eal_private.h"
>>  #include "eal_filesystem.h"
>> @@ -613,10 +614,12 @@
>>      fclose(fp);
>>
>>      mgaw = ((vtd_cap_reg & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT) 
>> + 1;
>> -    if (mgaw < X86_VA_WIDTH)
>> +
>> +    if (!rte_eal_check_dma_mask(mgaw))
>> +        return true;
>> +    else
>>          return false;
>>
>> -    return true;
>>  }
>>  #elif defined(RTE_ARCH_PPC_64)
>>  static bool
>> @@ -640,13 +643,17 @@
>>  {
>>      struct rte_pci_device *dev = NULL;
>>      struct rte_pci_driver *drv = NULL;
>> +    int iommu_dma_mask_check_done = 0;
>>
>>      FOREACH_DRIVER_ON_PCIBUS(drv) {
>>          FOREACH_DEVICE_ON_PCIBUS(dev) {
>>              if (!rte_pci_match(drv, dev))
>>                  continue;
>> -            if (!pci_one_device_iommu_support_va(dev))
>> -                return false;
>> +            if (!iommu_dma_mask_check_done) {
>> +                if (pci_one_device_iommu_support_va(dev) < 0)
>> +                    return false;
>> +                iommu_dma_mask_check_done  = 1;
> 
> As I do not know enough on what IOMMU hardware can coexist, I leave this 
> patch for others to review.
> Here is the previous question/answer:
> 
>>>> Not sure why this change? Why do we only need to check one device on 
>>>> all the buses?
> 
>>> Because there is just one emulated IOMMU hardware. The limitation in 
>>> this case is not in a specific PCI device. And I do not think it is 
>>> possible to have two different (emulated or not) IOMMU hardware. Yes, 
>>> you can have more than one controller but being same IOMMU type.
> 
> If the above is confirmed, you can consider this patch ack’ed.

As far as my knowledge goes, the above is correct. There can't be more 
than one IOMMU type per platform, and limitations specific for IOMMU 
controller apply to all devices, not just one.

I'm not sure if it's possible to have two IOMMU controllers of the same 
type with different address width, but i think even if it is possible, 
we can safely consider this to be unlikely :)

> 
>> +            }
>>          }
>>      }
>>      return true;
>> -- 
>> 1.9.1
> 


-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask
  2018-07-10 17:25 [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Alejandro Lucero
                   ` (4 preceding siblings ...)
  2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 5/5] net/nfp: support IOVA VA mode Alejandro Lucero
@ 2018-07-26 15:41 ` Thomas Monjalon
  2018-07-27  7:03   ` Alejandro Lucero
  5 siblings, 1 reply; 16+ messages in thread
From: Thomas Monjalon @ 2018-07-26 15:41 UTC (permalink / raw)
  To: Alejandro Lucero; +Cc: dev, anatoly.burakov

10/07/2018 19:25, Alejandro Lucero:
> This patchset applies on 17.11.3.
> 
> Similar changes will be submitted to main DPDK branch soon.

In patchwork, I mark this patchset as Deferred,
waiting a new version for 18.11.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask
  2018-07-26 15:41 ` [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Thomas Monjalon
@ 2018-07-27  7:03   ` Alejandro Lucero
  2018-07-27  8:01     ` Thomas Monjalon
  0 siblings, 1 reply; 16+ messages in thread
From: Alejandro Lucero @ 2018-07-27  7:03 UTC (permalink / raw)
  To: Thomas Monjalon; +Cc: dev, Burakov, Anatoly

Hi Thomas,

Why deferred? This patch is only to be applied to 17.11.3. Because the
changes the memory code has gone through the last months, the same fix will
be different for newer versions.

I'm working on it but waiting all the current patches from Anatoly being
accepted for creating the final patchset.



On Thu, Jul 26, 2018 at 5:41 PM, Thomas Monjalon <thomas@monjalon.net>
wrote:

> 10/07/2018 19:25, Alejandro Lucero:
> > This patchset applies on 17.11.3.
> >
> > Similar changes will be submitted to main DPDK branch soon.
>
> In patchwork, I mark this patchset as Deferred,
> waiting a new version for 18.11.
>
>
>

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask
  2018-07-27  7:03   ` Alejandro Lucero
@ 2018-07-27  8:01     ` Thomas Monjalon
  2018-07-27  8:22       ` Alejandro Lucero
  0 siblings, 1 reply; 16+ messages in thread
From: Thomas Monjalon @ 2018-07-27  8:01 UTC (permalink / raw)
  To: Alejandro Lucero; +Cc: dev, Burakov, Anatoly

27/07/2018 09:03, Alejandro Lucero:
> Hi Thomas,
> 
> Why deferred? This patch is only to be applied to 17.11.3. Because the
> changes the memory code has gone through the last months, the same fix will
> be different for newer versions.

OK, I can change them to "Not Applicable".

> I'm working on it but waiting all the current patches from Anatoly being
> accepted for creating the final patchset.

Is there still some patches which are not applied?



> On Thu, Jul 26, 2018 at 5:41 PM, Thomas Monjalon <thomas@monjalon.net>
> wrote:
> 
> > 10/07/2018 19:25, Alejandro Lucero:
> > > This patchset applies on 17.11.3.
> > >
> > > Similar changes will be submitted to main DPDK branch soon.
> >
> > In patchwork, I mark this patchset as Deferred,
> > waiting a new version for 18.11.
> >
> >
> >
> 

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask
  2018-07-27  8:01     ` Thomas Monjalon
@ 2018-07-27  8:22       ` Alejandro Lucero
  2018-07-27  8:52         ` Thomas Monjalon
  2018-07-27  8:54         ` Burakov, Anatoly
  0 siblings, 2 replies; 16+ messages in thread
From: Alejandro Lucero @ 2018-07-27  8:22 UTC (permalink / raw)
  To: Thomas Monjalon, stable; +Cc: dev, Burakov, Anatoly

On Fri, Jul 27, 2018 at 10:01 AM, Thomas Monjalon <thomas@monjalon.net>
wrote:

> 27/07/2018 09:03, Alejandro Lucero:
> > Hi Thomas,
> >
> > Why deferred? This patch is only to be applied to 17.11.3. Because the
> > changes the memory code has gone through the last months, the same fix
> will
> > be different for newer versions.
>
> OK, I can change them to "Not Applicable".
>
>
It is not applicable to master but it should for 17.11.3. Adding
stable@dpdk.org to this thread.


> > I'm working on it but waiting all the current patches from Anatoly being
> > accepted for creating the final patchset.
>
> Is there still some patches which are not applied?
>
>
>
Uhmm, maybe a patch for 18.11 could be created now. I was thinking about
these ones:

Support externally allocated memory in DPDK


Support running DPDK without hugetlbfs mountpoint

But they are not going to be in 18.11, so I will submit the patch asap, but
I'm afraid it will be for stable 18.11.1
Next three weeks I'm on PTO, so hopefully I can submit the patch early
September.



> > On Thu, Jul 26, 2018 at 5:41 PM, Thomas Monjalon <thomas@monjalon.net>
> > wrote:
> >
> > > 10/07/2018 19:25, Alejandro Lucero:
> > > > This patchset applies on 17.11.3.
> > > >
> > > > Similar changes will be submitted to main DPDK branch soon.
> > >
> > > In patchwork, I mark this patchset as Deferred,
> > > waiting a new version for 18.11.
> > >
> > >
> > >
> >
>
>
>
>
>
>

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask
  2018-07-27  8:22       ` Alejandro Lucero
@ 2018-07-27  8:52         ` Thomas Monjalon
  2018-07-27  8:59           ` Alejandro Lucero
  2018-07-27  8:54         ` Burakov, Anatoly
  1 sibling, 1 reply; 16+ messages in thread
From: Thomas Monjalon @ 2018-07-27  8:52 UTC (permalink / raw)
  To: Alejandro Lucero; +Cc: stable, dev, Burakov, Anatoly, Yongseok Koh

27/07/2018 10:22, Alejandro Lucero:
> Thomas Monjalon <thomas@monjalon.net> wrote:
> > 27/07/2018 09:03, Alejandro Lucero:
> > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > 10/07/2018 19:25, Alejandro Lucero:
> > > > > This patchset applies on 17.11.3.
> > > > >
> > > > > Similar changes will be submitted to main DPDK branch soon.
> > > >
> > > > In patchwork, I mark this patchset as Deferred,
> > > > waiting a new version for 18.11.

> > > Hi Thomas,
> > >
> > > Why deferred? This patch is only to be applied to 17.11.3. Because the
> > > changes the memory code has gone through the last months, the same fix
> > will
> > > be different for newer versions.
> >
> > OK, I can change them to "Not Applicable".
> >
> It is not applicable to master but it should for 17.11.3. Adding
> stable@dpdk.org to this thread.

The process is to send the patches to stable@dpdk.org with
[PATCH 17.11] in the subject.
Then it must decided of accepting the exception or not.

> > > I'm working on it but waiting all the current patches from Anatoly being
> > > accepted for creating the final patchset.
> >
> > Is there still some patches which are not applied?
> >
> Uhmm, maybe a patch for 18.11 could be created now. I was thinking about
> these ones:
> 
> Support externally allocated memory in DPDK
> Support running DPDK without hugetlbfs mountpoint
> 
> But they are not going to be in 18.11, so I will submit the patch asap, but
> I'm afraid it will be for stable 18.11.1
> Next three weeks I'm on PTO, so hopefully I can submit the patch early
> September.

We are not going to apply them to 18.11.1. It should be in 18.11.0.
If you send them really early September, it can be fine.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask
  2018-07-27  8:22       ` Alejandro Lucero
  2018-07-27  8:52         ` Thomas Monjalon
@ 2018-07-27  8:54         ` Burakov, Anatoly
  1 sibling, 0 replies; 16+ messages in thread
From: Burakov, Anatoly @ 2018-07-27  8:54 UTC (permalink / raw)
  To: Alejandro Lucero, Thomas Monjalon, stable; +Cc: dev

On 27-Jul-18 9:22 AM, Alejandro Lucero wrote:
> 
> 
> On Fri, Jul 27, 2018 at 10:01 AM, Thomas Monjalon <thomas@monjalon.net 
> <mailto:thomas@monjalon.net>> wrote:
> 
>     27/07/2018 09:03, Alejandro Lucero:
>     > Hi Thomas,
>     > 
>     > Why deferred? This patch is only to be applied to 17.11.3. Because the
>     > changes the memory code has gone through the last months, the same fix will
>     > be different for newer versions.
> 
>     OK, I can change them to "Not Applicable".
> 
> 
> It is not applicable to master but it should for 17.11.3. Adding 
> stable@dpdk.org <mailto:stable@dpdk.org> to this thread.
> 
>     > I'm working on it but waiting all the current patches from Anatoly being
>     > accepted for creating the final patchset.
> 
>     Is there still some patches which are not applied?
> 
> 
> 
> Uhmm, maybe a patch for 18.11 could be created now. I was thinking about 
> these ones:
> 
> Support externally allocated memory in DPDK
> 
> 
> Support running DPDK without hugetlbfs mountpoint

External memory allocators is an RFC intended for 18.11. In-memory mode 
patches are already applied IIRC.

> 
> 
> But they are not going to be in 18.11, so I will submit the patch asap, 
> but I'm afraid it will be for stable 18.11.1
> Next three weeks I'm on PTO, so hopefully I can submit the patch early 
> September.
> 
> 
> 
>      > On Thu, Jul 26, 2018 at 5:41 PM, Thomas Monjalon
>     <thomas@monjalon.net <mailto:thomas@monjalon.net>>
>      > wrote:
>      >
>      > > 10/07/2018 19:25, Alejandro Lucero:
>      > > > This patchset applies on 17.11.3.
>      > > >
>      > > > Similar changes will be submitted to main DPDK branch soon.
>      > >
>      > > In patchwork, I mark this patchset as Deferred,
>      > > waiting a new version for 18.11.
>      > >
>      > >
>      > >
>      >
> 
> 
> 
> 
> 
> 


-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask
  2018-07-27  8:52         ` Thomas Monjalon
@ 2018-07-27  8:59           ` Alejandro Lucero
  0 siblings, 0 replies; 16+ messages in thread
From: Alejandro Lucero @ 2018-07-27  8:59 UTC (permalink / raw)
  To: Thomas Monjalon; +Cc: stable, dev, Burakov, Anatoly, Yongseok Koh

On Fri, Jul 27, 2018 at 10:52 AM, Thomas Monjalon <thomas@monjalon.net>
wrote:

> 27/07/2018 10:22, Alejandro Lucero:
> > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > 27/07/2018 09:03, Alejandro Lucero:
> > > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > 10/07/2018 19:25, Alejandro Lucero:
> > > > > > This patchset applies on 17.11.3.
> > > > > >
> > > > > > Similar changes will be submitted to main DPDK branch soon.
> > > > >
> > > > > In patchwork, I mark this patchset as Deferred,
> > > > > waiting a new version for 18.11.
>
> > > > Hi Thomas,
> > > >
> > > > Why deferred? This patch is only to be applied to 17.11.3. Because
> the
> > > > changes the memory code has gone through the last months, the same
> fix
> > > will
> > > > be different for newer versions.
> > >
> > > OK, I can change them to "Not Applicable".
> > >
> > It is not applicable to master but it should for 17.11.3. Adding
> > stable@dpdk.org to this thread.
>
> The process is to send the patches to stable@dpdk.org with
> [PATCH 17.11] in the subject.
> Then it must decided of accepting the exception or not.
>
>
OK. I'll do that then.


> > > > I'm working on it but waiting all the current patches from Anatoly
> being
> > > > accepted for creating the final patchset.
> > >
> > > Is there still some patches which are not applied?
> > >
> > Uhmm, maybe a patch for 18.11 could be created now. I was thinking about
> > these ones:
> >
> > Support externally allocated memory in DPDK
> > Support running DPDK without hugetlbfs mountpoint
> >
> > But they are not going to be in 18.11, so I will submit the patch asap,
> but
> > I'm afraid it will be for stable 18.11.1
> > Next three weeks I'm on PTO, so hopefully I can submit the patch early
> > September.
>
> We are not going to apply them to 18.11.1. It should be in 18.11.0.
> If you send them really early September, it can be fine.
>
>
Good. Thanks!

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2018-07-27  8:59 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-10 17:25 [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Alejandro Lucero
2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 1/5] mem: add function for checking memsegs IOVAs addresses Alejandro Lucero
2018-07-11 10:12   ` [dpdk-dev] [dpdk-stable] " Eelco Chaudron
2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 2/5] bus/pci: use IOVAs check when setting IOVA mode Alejandro Lucero
2018-07-11 10:18   ` [dpdk-dev] [dpdk-stable] " Eelco Chaudron
2018-07-11 10:41     ` Burakov, Anatoly
2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 3/5] mem: use address hint for mapping hugepages Alejandro Lucero
2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 4/5] net/nfp: check hugepages IOVAs based on DMA mask Alejandro Lucero
2018-07-10 17:25 ` [dpdk-dev] [PATCH v4 5/5] net/nfp: support IOVA VA mode Alejandro Lucero
2018-07-26 15:41 ` [dpdk-dev] [PATCH v4 0/5] use IOVAs check based on DMA mask Thomas Monjalon
2018-07-27  7:03   ` Alejandro Lucero
2018-07-27  8:01     ` Thomas Monjalon
2018-07-27  8:22       ` Alejandro Lucero
2018-07-27  8:52         ` Thomas Monjalon
2018-07-27  8:59           ` Alejandro Lucero
2018-07-27  8:54         ` Burakov, Anatoly

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).