[dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[Chas Williams]

From: Chas Williams <chas3@att.com>

The IOMMU in some machines report that they can only support
limited widths.  IOVA virtual addresses may exceed this width
making the use of IOVA virtual addresses difficult.  The option
CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical
address usage.

Signed-off-by: Chas Williams <chas3@att.com>
---
 config/common_base                | 5 +++++
 lib/librte_eal/bsdapp/eal/eal.c   | 4 ++++
 lib/librte_eal/linuxapp/eal/eal.c | 4 ++++
 3 files changed, 13 insertions(+)

diff --git a/config/common_base b/config/common_base
index e74febe..06b3ded 100644
--- a/config/common_base
+++ b/config/common_base
@@ -820,3 +820,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Force IOVA physical addresses in VFIO
+#
+CONFIG_RTE_LIBRTE_USE_PHYS_IOVA=n
diff --git a/lib/librte_eal/bsdapp/eal/eal.c b/lib/librte_eal/bsdapp/eal/eal.c
index 369a682..b396285 100644
--- a/lib/librte_eal/bsdapp/eal/eal.c
+++ b/lib/librte_eal/bsdapp/eal/eal.c
@@ -569,8 +569,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+#ifdef RTE_LIBRTE_USE_PHYS_IOVA
+	rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+#else
 	/* autodetect the iova mapping mode (default is iova_pa) */
 	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+#endif
 
 	if (internal_config.no_hugetlbfs == 0 &&
 			internal_config.process_type != RTE_PROC_SECONDARY &&
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 229eec9..13b02fd 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -804,8 +804,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+#ifdef RTE_LIBRTE_USE_PHYS_IOVA
+	rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+#else
 	/* autodetect the iova mapping mode (default is iova_pa) */
 	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+#endif
 
 	/* Workaround for KNI which requires physical address to work */
 	if (rte_eal_get_configuration()->iova_mode == RTE_IOVA_VA &&
-- 
2.9.5

Re: [dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[Thomas Monjalon]

30/11/2017 03:57, Chas Williams:
> From: Chas Williams <chas3@att.com>
> 
> The IOMMU in some machines report that they can only support
> limited widths.  IOVA virtual addresses may exceed this width
> making the use of IOVA virtual addresses difficult.  The option
> CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical
> address usage.

Which machines are you talking about?
A run-time option may be a better solution.

Re: [dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[Chas Williams]

The particular machine in my case is a Dell Optiplex 790 which is fairly
similar to another system that has basically the same problem.  The IOMMU
advertises that is can only handle 39 bits of addressing.  The DMAR address
tables have a width of 36 bits, so all is well when using IOVA physical
addresses.  With IOVA virtual addresses, they sometimes go beyond the 39
bit boundary.  I was under the impression that IOMMU widths are more
typically 48 bits, but I appear to have some low end systems with a simpler
IOMMU.

I could make it a runtime option.

On Thu, Nov 30, 2017 at 4:17 AM, Thomas Monjalon <thomas@monjalon.net>
wrote:

> 30/11/2017 03:57, Chas Williams:
> > From: Chas Williams <chas3@att.com>
> >
> > The IOMMU in some machines report that they can only support
> > limited widths.  IOVA virtual addresses may exceed this width
> > making the use of IOVA virtual addresses difficult.  The option
> > CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical
> > address usage.
>
> Which machines are you talking about?
> A run-time option may be a better solution.
>
>

Re: [dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[Thomas Monjalon]

03/12/2017 01:23, Chas Williams:
> The particular machine in my case is a Dell Optiplex 790 which is fairly
> similar to another system that has basically the same problem.  The IOMMU
> advertises that is can only handle 39 bits of addressing.  The DMAR address
> tables have a width of 36 bits, so all is well when using IOVA physical
> addresses.  With IOVA virtual addresses, they sometimes go beyond the 39
> bit boundary.  I was under the impression that IOMMU widths are more
> typically 48 bits, but I appear to have some low end systems with a simpler
> IOMMU.
> 
> I could make it a runtime option.
> 
> On Thu, Nov 30, 2017 at 4:17 AM, Thomas Monjalon <thomas@monjalon.net>
> wrote:
> 
> > 30/11/2017 03:57, Chas Williams:
> > > From: Chas Williams <chas3@att.com>
> > >
> > > The IOMMU in some machines report that they can only support
> > > limited widths.  IOVA virtual addresses may exceed this width
> > > making the use of IOVA virtual addresses difficult.  The option
> > > CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical
> > > address usage.
> >
> > Which machines are you talking about?
> > A run-time option may be a better solution.

Please check the patch from Maxime:
	https://dpdk.org/dev/patchwork/patch/33192/

Re: [dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[Chas Williams]

Thanks, and yes I already saw that.  That seems like a much better
solution.  I will give it a spin in my box with this problem.

On Thu, Jan 11, 2018 at 7:12 PM, Thomas Monjalon <thomas@monjalon.net>
wrote:

> 03/12/2017 01:23, Chas Williams:
> > The particular machine in my case is a Dell Optiplex 790 which is fairly
> > similar to another system that has basically the same problem.  The IOMMU
> > advertises that is can only handle 39 bits of addressing.  The DMAR
> address
> > tables have a width of 36 bits, so all is well when using IOVA physical
> > addresses.  With IOVA virtual addresses, they sometimes go beyond the 39
> > bit boundary.  I was under the impression that IOMMU widths are more
> > typically 48 bits, but I appear to have some low end systems with a
> simpler
> > IOMMU.
> >
> > I could make it a runtime option.
> >
> > On Thu, Nov 30, 2017 at 4:17 AM, Thomas Monjalon <thomas@monjalon.net>
> > wrote:
> >
> > > 30/11/2017 03:57, Chas Williams:
> > > > From: Chas Williams <chas3@att.com>
> > > >
> > > > The IOMMU in some machines report that they can only support
> > > > limited widths.  IOVA virtual addresses may exceed this width
> > > > making the use of IOVA virtual addresses difficult.  The option
> > > > CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical
> > > > address usage.
> > >
> > > Which machines are you talking about?
> > > A run-time option may be a better solution.
>
> Please check the patch from Maxime:
>         https://dpdk.org/dev/patchwork/patch/33192/
>
>
>

Re: [dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[Hemant Agrawal]

On 11/30/2017 8:27 AM, Chas Williams wrote:
> From: Chas Williams <chas3@att.com>
>
> The IOMMU in some machines report that they can only support
> limited widths.  IOVA virtual addresses may exceed this width
> making the use of IOVA virtual addresses difficult.  The option
> CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical
> address usage.
>

Is it possible to dynamically detect the machine limitation in bus iova 
code instead of compile time option?

Re: [dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[Chas Williams]

On Fri, Dec 1, 2017 at 12:48 AM, Hemant Agrawal <hemant.agrawal@nxp.com>
wrote:

> On 11/30/2017 8:27 AM, Chas Williams wrote:
>
>> From: Chas Williams <chas3@att.com>
>>
>> The IOMMU in some machines report that they can only support
>> limited widths.  IOVA virtual addresses may exceed this width
>> making the use of IOVA virtual addresses difficult.  The option
>> CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical
>> address usage.
>>
>>
> Is it possible to dynamically detect the machine limitation in bus iova
> code instead of compile time option?
>
>
Yes, I think you can examine the "cap" of the IOMMU group in /sys and
determine the bit width.  I will have to check on that.

[dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[Chas Williams]

From: Chas Williams <chas3@att.com>

The IOMMU in some machines report that they can only support
limited widths.  IOVA virtual addresses may exceed this width
making the use of IOVA virtual addresses difficult.  The option
CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical
address usage.

Signed-off-by: Chas Williams <chas3@att.com>
---
 config/common_base                | 5 +++++
 lib/librte_eal/bsdapp/eal/eal.c   | 4 ++++
 lib/librte_eal/linuxapp/eal/eal.c | 4 ++++
 3 files changed, 13 insertions(+)

diff --git a/config/common_base b/config/common_base
index e74febe..06b3ded 100644
--- a/config/common_base
+++ b/config/common_base
@@ -820,3 +820,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Force IOVA physical addresses in VFIO
+#
+CONFIG_RTE_LIBRTE_USE_PHYS_IOVA=n
diff --git a/lib/librte_eal/bsdapp/eal/eal.c b/lib/librte_eal/bsdapp/eal/eal.c
index 369a682..b396285 100644
--- a/lib/librte_eal/bsdapp/eal/eal.c
+++ b/lib/librte_eal/bsdapp/eal/eal.c
@@ -569,8 +569,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+#ifdef RTE_LIBRTE_USE_PHYS_IOVA
+	rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+#else
 	/* autodetect the iova mapping mode (default is iova_pa) */
 	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+#endif
 
 	if (internal_config.no_hugetlbfs == 0 &&
 			internal_config.process_type != RTE_PROC_SECONDARY &&
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 229eec9..13b02fd 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -804,8 +804,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+#ifdef RTE_LIBRTE_USE_PHYS_IOVA
+	rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+#else
 	/* autodetect the iova mapping mode (default is iova_pa) */
 	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+#endif
 
 	/* Workaround for KNI which requires physical address to work */
 	if (rte_eal_get_configuration()->iova_mode == RTE_IOVA_VA &&
-- 
2.9.5

[dpdk-dev] [PATCH] eal: add option to force IOVA as PA mode

[Chas Williams]

From: Chas Williams <chas3@att.com>

The IOMMU in some machines report that they can only support
limited widths.  IOVA virtual addresses may exceed this width
making the use of IOVA virtual addresses difficult.  The option
CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical
address usage.

Signed-off-by: Chas Williams <chas3@att.com>
---
 config/common_base                | 5 +++++
 lib/librte_eal/bsdapp/eal/eal.c   | 4 ++++
 lib/librte_eal/linuxapp/eal/eal.c | 4 ++++
 3 files changed, 13 insertions(+)

diff --git a/config/common_base b/config/common_base
index e74febe..06b3ded 100644
--- a/config/common_base
+++ b/config/common_base
@@ -820,3 +820,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Force IOVA physical addresses in VFIO
+#
+CONFIG_RTE_LIBRTE_USE_PHYS_IOVA=n
diff --git a/lib/librte_eal/bsdapp/eal/eal.c b/lib/librte_eal/bsdapp/eal/eal.c
index 369a682..b396285 100644
--- a/lib/librte_eal/bsdapp/eal/eal.c
+++ b/lib/librte_eal/bsdapp/eal/eal.c
@@ -569,8 +569,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+#ifdef RTE_LIBRTE_USE_PHYS_IOVA
+	rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+#else
 	/* autodetect the iova mapping mode (default is iova_pa) */
 	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+#endif
 
 	if (internal_config.no_hugetlbfs == 0 &&
 			internal_config.process_type != RTE_PROC_SECONDARY &&
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 229eec9..13b02fd 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -804,8 +804,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+#ifdef RTE_LIBRTE_USE_PHYS_IOVA
+	rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+#else
 	/* autodetect the iova mapping mode (default is iova_pa) */
 	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+#endif
 
 	/* Workaround for KNI which requires physical address to work */
 	if (rte_eal_get_configuration()->iova_mode == RTE_IOVA_VA &&
-- 
2.9.5