[PATCH] eal: choose IOVA mode according to compilation flags

[PATCH] eal: choose IOVA mode according to compilation flags

[Viacheslav Ovsiienko]

The DPDK can be compiled to be run in IOVA VA mode with
'enable_iova_as_pa=false' meson option. If there is no
explicit EAL --iova-mode parameter specified in the command
line the rte_eal_init() tried to deduce  VA or PA mode without
taking into account the above mentioned compile time option,
resulting into initialization failure.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 lib/eal/linux/eal.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index c37868b7f0..4481bc4ad8 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -1080,7 +1080,10 @@ rte_eal_init(int argc, char **argv)
 		if (iova_mode == RTE_IOVA_DC) {
 			RTE_LOG(DEBUG, EAL, "Buses did not request a specific IOVA mode.\n");
 
-			if (!phys_addrs) {
+			if (!RTE_IOVA_IN_MBUF) {
+				iova_mode = RTE_IOVA_VA;
+				RTE_LOG(DEBUG, EAL, "IOVA VA mode is forced by build option.\n");
+			} else if (!phys_addrs) {
 				/* if we have no access to physical addresses,
 				 * pick IOVA as VA mode.
 				 */
-- 
2.18.1

RE: [PATCH] eal: choose IOVA mode according to compilation flags

[Morten Brørup]

> From: Viacheslav Ovsiienko [mailto:viacheslavo@nvidia.com]
> Sent: Wednesday, 12 April 2023 19.20
> 
> The DPDK can be compiled to be run in IOVA VA mode with
> 'enable_iova_as_pa=false' meson option. If there is no
> explicit EAL --iova-mode parameter specified in the command
> line the rte_eal_init() tried to deduce  VA or PA mode without
> taking into account the above mentioned compile time option,
> resulting into initialization failure.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---
>  lib/eal/linux/eal.c | 5 ++++-

This patch is close to being a bugfix. Good catch!

You could also consider another patch, logging warnings in rte_bus_get_iommu_class() [1] for the busses that want IOVA as PA when !RTE_IOVA_IN_MBUF.

[1]: https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/common/eal_common_bus.c#L224

>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
> index c37868b7f0..4481bc4ad8 100644
> --- a/lib/eal/linux/eal.c
> +++ b/lib/eal/linux/eal.c
> @@ -1080,7 +1080,10 @@ rte_eal_init(int argc, char **argv)
>  		if (iova_mode == RTE_IOVA_DC) {
>  			RTE_LOG(DEBUG, EAL, "Buses did not request a specific
> IOVA mode.\n");
> 
> -			if (!phys_addrs) {
> +			if (!RTE_IOVA_IN_MBUF) {
> +				iova_mode = RTE_IOVA_VA;
> +				RTE_LOG(DEBUG, EAL, "IOVA VA mode is forced by
> build option.\n");

Minor detail regarding conventions: "IOVA VA " -> "IOVA as VA"

> +			} else if (!phys_addrs) {
>  				/* if we have no access to physical addresses,
>  				 * pick IOVA as VA mode.
>  				 */
> --
> 2.18.1

Reviewed-by: Morten Brørup <mb@smartsharesystems.com>

RE: [PATCH] eal: choose IOVA mode according to compilation flags

[Slava Ovsiienko]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: среда, 12 апреля 2023 г. 22:12
> To: Slava Ovsiienko <viacheslavo@nvidia.com>; dev@dpdk.org
> Cc: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> david.marchand@redhat.com
> Subject: RE: [PATCH] eal: choose IOVA mode according to compilation flags
> 
> > From: Viacheslav Ovsiienko [mailto:viacheslavo@nvidia.com]
> > Sent: Wednesday, 12 April 2023 19.20
> >
> > The DPDK can be compiled to be run in IOVA VA mode with
> > 'enable_iova_as_pa=false' meson option. If there is no explicit EAL
> > --iova-mode parameter specified in the command line the rte_eal_init()
> > tried to deduce  VA or PA mode without taking into account the above
> > mentioned compile time option, resulting into initialization failure.
> >
> > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> > ---
> >  lib/eal/linux/eal.c | 5 ++++-
> 
> This patch is close to being a bugfix. Good catch!
> 
> You could also consider another patch, logging warnings in
> rte_bus_get_iommu_class() [1] for the busses that want IOVA as PA when
> !RTE_IOVA_IN_MBUF.

Yes, looks reasonable, thank you.
Let me gather the comments first, then will update the patch.

> 
> [1]:
> https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/common/eal_common_
> bus.c#L224
> 
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c index
> > c37868b7f0..4481bc4ad8 100644
> > --- a/lib/eal/linux/eal.c
> > +++ b/lib/eal/linux/eal.c
> > @@ -1080,7 +1080,10 @@ rte_eal_init(int argc, char **argv)
> >  		if (iova_mode == RTE_IOVA_DC) {
> >  			RTE_LOG(DEBUG, EAL, "Buses did not request a
> specific IOVA
> > mode.\n");
> >
> > -			if (!phys_addrs) {
> > +			if (!RTE_IOVA_IN_MBUF) {
> > +				iova_mode = RTE_IOVA_VA;
> > +				RTE_LOG(DEBUG, EAL, "IOVA VA mode is
> forced by
> > build option.\n");
> 
> Minor detail regarding conventions: "IOVA VA " -> "IOVA as VA"
> 
> > +			} else if (!phys_addrs) {
> >  				/* if we have no access to physical addresses,
> >  				 * pick IOVA as VA mode.
> >  				 */
> > --
> > 2.18.1
> 
> Reviewed-by: Morten Brørup <mb@smartsharesystems.com>

[PATCH v2] eal: choose IOVA mode according to compilation flags

[Viacheslav Ovsiienko]

The DPDK can be compiled to be run in IOVA VA mode with
'enable_iova_as_pa=false' meson option. If there is no
explicit EAL --iova-mode parameter specified in the command
line the rte_eal_init() tried to deduce VA or PA mode without
taking into account the above mentioned compile time option,
resulting into initialization failure.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 lib/eal/linux/eal.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index ae323cd492..9856ec9d12 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -1081,7 +1081,10 @@ rte_eal_init(int argc, char **argv)
 		if (iova_mode == RTE_IOVA_DC) {
 			RTE_LOG(DEBUG, EAL, "Buses did not request a specific IOVA mode.\n");
 
-			if (!phys_addrs) {
+			if (!RTE_IOVA_IN_MBUF) {
+				iova_mode = RTE_IOVA_VA;
+				RTE_LOG(DEBUG, EAL, "IOVA as VA mode is forced by build option.\n");
+			} else if (!phys_addrs) {
 				/* if we have no access to physical addresses,
 				 * pick IOVA as VA mode.
 				 */
-- 
2.18.1

Re: [PATCH v2] eal: choose IOVA mode according to compilation flags

[Bruce Richardson]

On Tue, Jun 06, 2023 at 06:15:11PM +0300, Viacheslav Ovsiienko wrote:
> The DPDK can be compiled to be run in IOVA VA mode with
> 'enable_iova_as_pa=false' meson option. If there is no
> explicit EAL --iova-mode parameter specified in the command
> line the rte_eal_init() tried to deduce VA or PA mode without
> taking into account the above mentioned compile time option,
> resulting into initialization failure.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---
>  lib/eal/linux/eal.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
> index ae323cd492..9856ec9d12 100644
> --- a/lib/eal/linux/eal.c
> +++ b/lib/eal/linux/eal.c
> @@ -1081,7 +1081,10 @@ rte_eal_init(int argc, char **argv)
>  		if (iova_mode == RTE_IOVA_DC) {
>  			RTE_LOG(DEBUG, EAL, "Buses did not request a specific IOVA mode.\n");
>  
> -			if (!phys_addrs) {
> +			if (!RTE_IOVA_IN_MBUF) {
> +				iova_mode = RTE_IOVA_VA;
> +				RTE_LOG(DEBUG, EAL, "IOVA as VA mode is forced by build option.\n");
> +			} else if (!phys_addrs) {
>  				/* if we have no access to physical addresses,
>  				 * pick IOVA as VA mode.
>  				 */

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [PATCH v2] eal: choose IOVA mode according to compilation flags

[Thomas Monjalon]

06/06/2023 17:23, Bruce Richardson:
> On Tue, Jun 06, 2023 at 06:15:11PM +0300, Viacheslav Ovsiienko wrote:
> > The DPDK can be compiled to be run in IOVA VA mode with
> > 'enable_iova_as_pa=false' meson option. If there is no
> > explicit EAL --iova-mode parameter specified in the command
> > line the rte_eal_init() tried to deduce VA or PA mode without
> > taking into account the above mentioned compile time option,
> > resulting into initialization failure.

It looks like a bug fix to backport. Do you agree?
Could you add the fix tags?

> > 
> > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

You forgot previous tag:
Reviewed-by: Morten Brørup <mb@smartsharesystems.com>

> > ---
> >  lib/eal/linux/eal.c | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)

What about FreeBSD and Windows?
I suspect there is the same issue.

> > @@ -1081,7 +1081,10 @@ rte_eal_init(int argc, char **argv)
> >  		if (iova_mode == RTE_IOVA_DC) {
> >  			RTE_LOG(DEBUG, EAL, "Buses did not request a specific IOVA mode.\n");
> >  
> > -			if (!phys_addrs) {
> > +			if (!RTE_IOVA_IN_MBUF) {
> > +				iova_mode = RTE_IOVA_VA;
> > +				RTE_LOG(DEBUG, EAL, "IOVA as VA mode is forced by build option.\n");
> > +			} else if (!phys_addrs) {
> >  				/* if we have no access to physical addresses,
> >  				 * pick IOVA as VA mode.
> >  				 */
> 
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

[PATCH v3] eal: choose IOVA mode according to compilation flags

[Viacheslav Ovsiienko]

The DPDK can be compiled to be run in IOVA VA mode with
'enable_iova_as_pa=false' meson option. If there is no
explicit EAL --iova-mode parameter specified in the command
line the rte_eal_init() tried to deduce VA or PA mode without
taking into account the above mentioned compile time option,
resulting into initialization failure.

Also, for FreeBSD and Windows EALs the IOVE mode checks were
added or moved to be after the mode selection code block for
final checking.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Reviewed-by: Morten Brørup <mb@smartsharesystems.com>
---
 lib/eal/freebsd/eal.c | 28 +++++++++++++++++++++-------
 lib/eal/linux/eal.c   |  5 ++++-
 lib/eal/windows/eal.c | 28 +++++++++++++++++++++-------
 3 files changed, 46 insertions(+), 15 deletions(-)

---
v2: - minor test message update
v3: - added the same change to FreeBSD and Windwos EALs
    - added/mode chosen IOVA mode checks in FreeBSD/Windows

diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 7daf22e314..ff7da2473a 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -694,22 +694,36 @@ rte_eal_init(int argc, char **argv)
 	 */
 	has_phys_addr = internal_conf->no_hugetlbfs == 0;
 	iova_mode = internal_conf->iova_mode;
-	if (iova_mode == RTE_IOVA_PA && !has_phys_addr) {
-		rte_eal_init_alert("Cannot use IOVA as 'PA' since physical addresses are not available");
-		rte_errno = EINVAL;
-		return -1;
-	}
 	if (iova_mode == RTE_IOVA_DC) {
 		RTE_LOG(DEBUG, EAL, "Specific IOVA mode is not requested, autodetecting\n");
 		if (has_phys_addr) {
 			RTE_LOG(DEBUG, EAL, "Selecting IOVA mode according to bus requests\n");
 			iova_mode = rte_bus_get_iommu_class();
-			if (iova_mode == RTE_IOVA_DC)
-				iova_mode = RTE_IOVA_PA;
+			if (iova_mode == RTE_IOVA_DC) {
+				if (!RTE_IOVA_IN_MBUF) {
+					iova_mode = RTE_IOVA_VA;
+					RTE_LOG(DEBUG, EAL, "IOVA as VA mode is forced by build option.\n");
+				} else	{
+					iova_mode = RTE_IOVA_PA;
+				}
+			}
 		} else {
 			iova_mode = RTE_IOVA_VA;
 		}
 	}
+
+	if (iova_mode == RTE_IOVA_PA && !has_phys_addr) {
+		rte_eal_init_alert("Cannot use IOVA as 'PA' since physical addresses are not available");
+		rte_errno = EINVAL;
+		return -1;
+	}
+
+	if (iova_mode == RTE_IOVA_PA && !RTE_IOVA_IN_MBUF) {
+		rte_eal_init_alert("Cannot use IOVA as 'PA' as it is disabled during build");
+		rte_errno = EINVAL;
+		return -1;
+	}
+
 	rte_eal_get_configuration()->iova_mode = iova_mode;
 	RTE_LOG(INFO, EAL, "Selected IOVA mode '%s'\n",
 		rte_eal_iova_mode() == RTE_IOVA_PA ? "PA" : "VA");
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index ae323cd492..9856ec9d12 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -1081,7 +1081,10 @@ rte_eal_init(int argc, char **argv)
 		if (iova_mode == RTE_IOVA_DC) {
 			RTE_LOG(DEBUG, EAL, "Buses did not request a specific IOVA mode.\n");
 
-			if (!phys_addrs) {
+			if (!RTE_IOVA_IN_MBUF) {
+				iova_mode = RTE_IOVA_VA;
+				RTE_LOG(DEBUG, EAL, "IOVA as VA mode is forced by build option.\n");
+			} else if (!phys_addrs) {
 				/* if we have no access to physical addresses,
 				 * pick IOVA as VA mode.
 				 */
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index 033825c14c..9c9a49fda0 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -362,22 +362,36 @@ rte_eal_init(int argc, char **argv)
 	}
 
 	iova_mode = internal_conf->iova_mode;
-	if (iova_mode == RTE_IOVA_PA && !has_phys_addr) {
-		rte_eal_init_alert("Cannot use IOVA as 'PA' since physical addresses are not available");
-		rte_errno = EINVAL;
-		return -1;
-	}
 	if (iova_mode == RTE_IOVA_DC) {
 		RTE_LOG(DEBUG, EAL, "Specific IOVA mode is not requested, autodetecting\n");
 		if (has_phys_addr) {
 			RTE_LOG(DEBUG, EAL, "Selecting IOVA mode according to bus requests\n");
 			iova_mode = rte_bus_get_iommu_class();
-			if (iova_mode == RTE_IOVA_DC)
-				iova_mode = RTE_IOVA_PA;
+			if (iova_mode == RTE_IOVA_DC) {
+				if (!RTE_IOVA_IN_MBUF) {
+					iova_mode = RTE_IOVA_VA;
+					RTE_LOG(DEBUG, EAL, "IOVA as VA mode is forced by build option.\n");
+				} else	{
+					iova_mode = RTE_IOVA_PA;
+				}
+			}
 		} else {
 			iova_mode = RTE_IOVA_VA;
 		}
 	}
+
+	if (iova_mode == RTE_IOVA_PA && !has_phys_addr) {
+		rte_eal_init_alert("Cannot use IOVA as 'PA' since physical addresses are not available");
+		rte_errno = EINVAL;
+		return -1;
+	}
+
+	if (iova_mode == RTE_IOVA_PA && !RTE_IOVA_IN_MBUF) {
+		rte_eal_init_alert("Cannot use IOVA as 'PA' as it is disabled during build");
+		rte_errno = EINVAL;
+		return -1;
+	}
+
 	RTE_LOG(DEBUG, EAL, "Selected IOVA mode '%s'\n",
 		iova_mode == RTE_IOVA_PA ? "PA" : "VA");
 	rte_eal_get_configuration()->iova_mode = iova_mode;
-- 
2.18.1

Re: [PATCH v3] eal: choose IOVA mode according to compilation flags

[Thomas Monjalon]

08/06/2023 17:45, Viacheslav Ovsiienko:
> The DPDK can be compiled to be run in IOVA VA mode with
> 'enable_iova_as_pa=false' meson option. If there is no
> explicit EAL --iova-mode parameter specified in the command
> line the rte_eal_init() tried to deduce VA or PA mode without
> taking into account the above mentioned compile time option,
> resulting into initialization failure.
> 
> Also, for FreeBSD and Windows EALs the IOVE mode checks were
> added or moved to be after the mode selection code block for
> final checking.
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> Reviewed-by: Morten Brørup <mb@smartsharesystems.com>

Applied, thanks.