[dpdk-dev] [PATCH v3 0/3] force IOVA to a particular mode

[dpdk-dev] [PATCH v3 0/3] force IOVA to a particular mode

[eric zhang]

This patchset introduces an EAL command line option "--iova-mode"
to give the user a facility to force IOVA mode to a special value.

Auto detection of the IOVA mode, based on probing the bus and IOMMU
configuration, may not report the desired addressing mode when virtual
devices that are not directly attached to the bus are present.
The EAL command line option "--iova-mode" can be used to select either
physical addressing('pa') or virtual addressing('va'). 

------
v3:
* document --iova-mode EAL option
* change default iova mode to RTE_IOVA_DC

v2:
* use eal option instead of compilation option to configure IOVA
* apply http://patchwork.dpdk.org/patch/25192/


Santosh Shukla (1):
  eal: add eal option to configure iova mode

eric zhang (2):
  eal: force IOVA to particular mode
  doc:document --iova-mode EAL flag

 doc/guides/prog_guide/env_abstraction_layer.rst |  8 +++++++
 doc/guides/testpmd_app_ug/run_app.rst           |  4 ++++
 lib/librte_eal/bsdapp/eal/eal.c                 | 11 +++++++--
 lib/librte_eal/common/eal_common_options.c      | 30 +++++++++++++++++++++++++
 lib/librte_eal/common/eal_internal_cfg.h        |  1 +
 lib/librte_eal/common/eal_options.h             |  2 ++
 lib/librte_eal/linuxapp/eal/eal.c               | 27 +++++++++++++---------
 7 files changed, 71 insertions(+), 12 deletions(-)

-- 
1.8.3.1

[dpdk-dev] [PATCH v3 1/3] eal: add eal option to configure iova mode

[eric zhang]

From: Santosh Shukla <santosh.shukla@caviumnetworks.com>

In the case of user don't want to use bus iova scheme and want
to override.

For that, Adding eal option --iova-mode=<string> where valid input
string is 'pa' or 'va'.

Signed-off-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Signed-off-by: eric zhang <eric.zhang@windriver.com>
---
 lib/librte_eal/common/eal_common_options.c | 30 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 3 files changed, 33 insertions(+)

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 996a034..70c0fc8 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -82,6 +82,7 @@
 	{OPT_HELP,              0, NULL, OPT_HELP_NUM             },
 	{OPT_HUGE_DIR,          1, NULL, OPT_HUGE_DIR_NUM         },
 	{OPT_HUGE_UNLINK,       0, NULL, OPT_HUGE_UNLINK_NUM      },
+	{OPT_IOVA_MODE,	        1, NULL, OPT_IOVA_MODE_NUM        },
 	{OPT_LCORES,            1, NULL, OPT_LCORES_NUM           },
 	{OPT_LOG_LEVEL,         1, NULL, OPT_LOG_LEVEL_NUM        },
 	{OPT_MASTER_LCORE,      1, NULL, OPT_MASTER_LCORE_NUM     },
@@ -218,6 +219,7 @@ struct device_option {
 #endif
 	internal_cfg->vmware_tsc_map = 0;
 	internal_cfg->create_uio_dev = 0;
+	internal_cfg->iova_mode = RTE_IOVA_DC;
 	internal_cfg->mbuf_pool_ops_name = RTE_MBUF_DEFAULT_MEMPOOL_OPS;
 }
 
@@ -994,6 +996,25 @@ static int xdigit2val(unsigned char c)
 	return RTE_PROC_INVALID;
 }
 
+static int
+eal_parse_iova_mode(const char *name)
+{
+	int mode;
+
+	if (name == NULL)
+		return -1;
+
+	if (!strcmp("pa", name))
+		mode = RTE_IOVA_PA;
+	else if (!strcmp("va", name))
+		mode = RTE_IOVA_VA;
+	else
+		return -1;
+
+	internal_config.iova_mode = mode;
+	return 0;
+}
+
 int
 eal_parse_common_option(int opt, const char *optarg,
 			struct internal_config *conf)
@@ -1158,6 +1179,13 @@ static int xdigit2val(unsigned char c)
 		}
 		core_parsed = 1;
 		break;
+	case OPT_IOVA_MODE_NUM:
+		if (eal_parse_iova_mode(optarg) < 0) {
+			RTE_LOG(ERR, EAL, "invalid parameters for --"
+				OPT_IOVA_MODE "\n");
+			return -1;
+		}
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1306,6 +1334,8 @@ static int xdigit2val(unsigned char c)
 	       "  -h, --help          This help\n"
 	       "\nEAL options for DEBUG use only:\n"
 	       "  --"OPT_HUGE_UNLINK"       Unlink hugepage files after init\n"
+	       "  --"OPT_IOVA_MODE"         Set iova mode. 'pa' for IOVA_PA\n"
+	       "                            'va' for IOVA_VA\n"
 	       "  --"OPT_NO_HUGE"           Use malloc instead of hugetlbfs\n"
 	       "  --"OPT_NO_PCI"            Disable PCI\n"
 	       "  --"OPT_NO_HPET"           Disable HPET\n"
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index fa6ccbe..29bf53f 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -83,6 +83,7 @@ struct internal_config {
 	const char *hugepage_dir;         /**< specific hugetlbfs directory to use */
 	const char *mbuf_pool_ops_name;   /**< mbuf pool ops name */
 	unsigned num_hugepage_sizes;      /**< how many sizes on this system */
+	enum rte_iova_mode iova_mode ;    /**< Set iova mode on this system  */
 	struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES];
 };
 extern struct internal_config internal_config; /**< Global EAL configuration. */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index 30e6bb4..7786189 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -83,6 +83,8 @@ enum {
 	OPT_VFIO_INTR_NUM,
 #define OPT_VMWARE_TSC_MAP    "vmware-tsc-map"
 	OPT_VMWARE_TSC_MAP_NUM,
+#define OPT_IOVA_MODE          "iova-mode"
+	OPT_IOVA_MODE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
-- 
1.8.3.1

[dpdk-dev] [PATCH v3 2/3] eal: force IOVA to a particular mode

[eric zhang]

This patch uses EAL option "--iova-mode" to force the IOVA mode to a
particular value. There exists virtual devices that are not directly
attached to the PCI bus, and therefore the auto detectioni of the IOVA
mode based on probing the PCI bus and IOMMU configuration may not
report the required addressing mode. Using the EAL option permits the
mode to be explicitly configured in this scenario.

Signed-off-by: eric zhang <eric.zhang@windriver.com>
---
 lib/librte_eal/bsdapp/eal/eal.c   | 11 +++++++++--
 lib/librte_eal/linuxapp/eal/eal.c | 27 +++++++++++++++++----------
 2 files changed, 26 insertions(+), 12 deletions(-)

diff --git a/lib/librte_eal/bsdapp/eal/eal.c b/lib/librte_eal/bsdapp/eal/eal.c
index 369a682..5392f06 100644
--- a/lib/librte_eal/bsdapp/eal/eal.c
+++ b/lib/librte_eal/bsdapp/eal/eal.c
@@ -569,8 +569,15 @@ static void rte_eal_init_alert(const char *msg)
 		return -1;
 	}
 
-	/* autodetect the iova mapping mode (default is iova_pa) */
-	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+	/* if no eal option "--iova-mode=<pa|va>", use bus iova scheme */
+	if (internal_config.iova_mode == RTE_IOVA_DC) {
+		/* autodetect the iova mapping mode (default is iova_pa) */
+		rte_eal_get_configuration()->iova_mode =
+			rte_bus_get_iommu_class();
+	} else {
+		rte_eal_get_configuration()->iova_mode =
+			internal_config.iova_mode;
+	}
 
 	if (internal_config.no_hugetlbfs == 0 &&
 			internal_config.process_type != RTE_PROC_SECONDARY &&
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index e0b5ae1..1af9458 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -805,16 +805,23 @@ static void rte_eal_init_alert(const char *msg)
 		return -1;
 	}
 
-	/* autodetect the iova mapping mode (default is iova_pa) */
-	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
-
-	/* Workaround for KNI which requires physical address to work */
-	if (rte_eal_get_configuration()->iova_mode == RTE_IOVA_VA &&
-			rte_eal_check_module("rte_kni") == 1) {
-		rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
-		RTE_LOG(WARNING, EAL,
-			"Some devices want IOVA as VA but PA will be used because.. "
-			"KNI module inserted\n");
+	/* if no eal option "--iova-mode=<pa|va>", use bus iova scheme */
+	if (internal_config.iova_mode == RTE_IOVA_DC) {
+		/* autodetect the iova mapping mode (default is iova_pa) */
+		rte_eal_get_configuration()->iova_mode =
+			rte_bus_get_iommu_class();
+
+		/* Workaround for KNI which requires physical address to work */
+		if (rte_eal_get_configuration()->iova_mode == RTE_IOVA_VA &&
+				rte_eal_check_module("rte_kni") == 1) {
+			rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+			RTE_LOG(WARNING, EAL,
+				"Some devices want IOVA as VA but PA will be used because.. "
+				"KNI module inserted\n");
+		}
+	} else {
+		rte_eal_get_configuration()->iova_mode =
+			internal_config.iova_mode;
 	}
 
 	if (internal_config.no_hugetlbfs == 0 &&
-- 
1.8.3.1

[dpdk-dev] [PATCH v3 3/3] doc: document --iova-mode EAL option

[eric zhang]

This patch updates Programmer's Guide and EAL parameter guides
to show EAL option "--iova-mode" support.

Signed-off-by: eric zhang <eric.zhang@windriver.com>
---
 doc/guides/prog_guide/env_abstraction_layer.rst | 8 ++++++++
 doc/guides/testpmd_app_ug/run_app.rst           | 4 ++++
 2 files changed, 12 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 34d871c..208c454 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -225,6 +225,14 @@ Misc Functions
 
 Locks and atomic operations are per-architecture (i686 and x86_64).
 
+IOVA Mode Configuration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Auto detection of the IOVA mode, based on probing the bus and IOMMU configuration, may not report
+the desired addressing mode when virtual devices that are not directly attached to the bus are present.
+To facilitate forcing the IOVA mode to a specific value the EAL command line option ``--iova-mode`` can
+be used to select either physical addressing('pa') or virtual addressing('va').
+
 Memory Segments and Memory Zones (memzone)
 ------------------------------------------
 
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index 4c0d2ce..94fd0cc 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -156,6 +156,10 @@ See the DPDK Getting Started Guides for more information on these options.
 
     Use malloc instead of hugetlbfs.
 
+*   ``--iova-mode <pa|va>``
+
+    Force IOVA mode to a specific value.
+
 
 Testpmd Command-line Options
 ----------------------------
-- 
1.8.3.1

Re: [dpdk-dev] [PATCH v3 0/3] force IOVA to a particular mode

[Burakov, Anatoly]

On 03-Oct-18 9:53 PM, eric zhang wrote:
> This patchset introduces an EAL command line option "--iova-mode"
> to give the user a facility to force IOVA mode to a special value.
> 
> Auto detection of the IOVA mode, based on probing the bus and IOMMU
> configuration, may not report the desired addressing mode when virtual
> devices that are not directly attached to the bus are present.
> The EAL command line option "--iova-mode" can be used to select either
> physical addressing('pa') or virtual addressing('va').
> 
> ------
> v3:
> * document --iova-mode EAL option
> * change default iova mode to RTE_IOVA_DC
> 
> v2:
> * use eal option instead of compilation option to configure IOVA
> * apply http://patchwork.dpdk.org/patch/25192/
> 
> 
> Santosh Shukla (1):
>    eal: add eal option to configure iova mode
> 
> eric zhang (2):
>    eal: force IOVA to particular mode
>    doc:document --iova-mode EAL flag
> 
>   doc/guides/prog_guide/env_abstraction_layer.rst |  8 +++++++
>   doc/guides/testpmd_app_ug/run_app.rst           |  4 ++++
>   lib/librte_eal/bsdapp/eal/eal.c                 | 11 +++++++--
>   lib/librte_eal/common/eal_common_options.c      | 30 +++++++++++++++++++++++++
>   lib/librte_eal/common/eal_internal_cfg.h        |  1 +
>   lib/librte_eal/common/eal_options.h             |  2 ++
>   lib/librte_eal/linuxapp/eal/eal.c               | 27 +++++++++++++---------
>   7 files changed, 71 insertions(+), 12 deletions(-)
> 

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 0/3] force IOVA to a particular mode

[Thomas Monjalon]

+Cc more maintainers in order to collect more reviews

04/10/2018 11:19, Burakov, Anatoly:
> On 03-Oct-18 9:53 PM, eric zhang wrote:
> > This patchset introduces an EAL command line option "--iova-mode"
> > to give the user a facility to force IOVA mode to a special value.
> > 
> > Auto detection of the IOVA mode, based on probing the bus and IOMMU
> > configuration, may not report the desired addressing mode when virtual
> > devices that are not directly attached to the bus are present.
> > The EAL command line option "--iova-mode" can be used to select either
> > physical addressing('pa') or virtual addressing('va').
> 
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

Re: [dpdk-dev] [PATCH v3 3/3] doc: document --iova-mode EAL option

[Kovacevic, Marko]

Only one thing I noticed:

Running git log check: 45999
Wrong headline lowercase:
        doc: document --iova-mode EAL option

All else seem ok.

Reviewed-by: Marko Kovacevic <marko.kovacevic@intel.com>

Re: [dpdk-dev] [PATCH v3 0/3] force IOVA to a particular mode

[Thomas Monjalon]

11/10/2018 12:08, Thomas Monjalon:
> +Cc more maintainers in order to collect more reviews
> 
> 04/10/2018 11:19, Burakov, Anatoly:
> > On 03-Oct-18 9:53 PM, eric zhang wrote:
> > > This patchset introduces an EAL command line option "--iova-mode"
> > > to give the user a facility to force IOVA mode to a special value.
> > > 
> > > Auto detection of the IOVA mode, based on probing the bus and IOMMU
> > > configuration, may not report the desired addressing mode when virtual
> > > devices that are not directly attached to the bus are present.
> > > The EAL command line option "--iova-mode" can be used to select either
> > > physical addressing('pa') or virtual addressing('va').
> > 
> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

Rebased and applied, thanks

Re: [dpdk-dev] [PATCH v3 0/3] force IOVA to a particular mode

[Alejandro Lucero]

On Sun, Oct 28, 2018 at 11:04 PM Thomas Monjalon <thomas@monjalon.net>
wrote:

> 11/10/2018 12:08, Thomas Monjalon:
> > +Cc more maintainers in order to collect more reviews
> >
> > 04/10/2018 11:19, Burakov, Anatoly:
> > > On 03-Oct-18 9:53 PM, eric zhang wrote:
> > > > This patchset introduces an EAL command line option "--iova-mode"
> > > > to give the user a facility to force IOVA mode to a special value.
> > > >
> > > > Auto detection of the IOVA mode, based on probing the bus and IOMMU
> > > > configuration, may not report the desired addressing mode when
> virtual
> > > > devices that are not directly attached to the bus are present.
> > > > The EAL command line option "--iova-mode" can be used to select
> either
> > > > physical addressing('pa') or virtual addressing('va').
> > >
> > > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
>
> Rebased and applied, thanks
>
>
>
Could not this lead to a problem if a device can not wok with the mode set?
For example, IOVA mode set to VA and IOMMU hw with less bits than those
required for the virtual addresses?

IMO any device should be attached to a bus, and a bus should have a
function for setting IOVA mode and the --iova-mode option just allowed with
supported IOVA modes within the bus.

Re: [dpdk-dev] [PATCH v3 0/3] force IOVA to a particular mode

[Burakov, Anatoly]

On 30-Oct-18 12:02 PM, Alejandro Lucero wrote:
> 
> 
> On Sun, Oct 28, 2018 at 11:04 PM Thomas Monjalon <thomas@monjalon.net 
> <mailto:thomas@monjalon.net>> wrote:
> 
>     11/10/2018 12:08, Thomas Monjalon:
>      > +Cc more maintainers in order to collect more reviews
>      >
>      > 04/10/2018 11:19, Burakov, Anatoly:
>      > > On 03-Oct-18 9:53 PM, eric zhang wrote:
>      > > > This patchset introduces an EAL command line option "--iova-mode"
>      > > > to give the user a facility to force IOVA mode to a special
>     value.
>      > > >
>      > > > Auto detection of the IOVA mode, based on probing the bus and
>     IOMMU
>      > > > configuration, may not report the desired addressing mode
>     when virtual
>      > > > devices that are not directly attached to the bus are present.
>      > > > The EAL command line option "--iova-mode" can be used to
>     select either
>      > > > physical addressing('pa') or virtual addressing('va').
>      > >
>      > > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com
>     <mailto:anatoly.burakov@intel.com>>
> 
>     Rebased and applied, thanks
> 
> 
> 
> Could not this lead to a problem if a device can not wok with the mode set?
> For example, IOVA mode set to VA and IOMMU hw with less bits than those 
> required for the virtual addresses?
> 
> IMO any device should be attached to a bus, and a bus should have a 
> function for setting IOVA mode and the --iova-mode option just allowed 
> with supported IOVA modes within the bus.

I don't think it should work that way. It should warn the user that an 
incompatible IOVA mode was selected, but the user has specified an IOVA 
mode for a reason - it probably implies he really means it, so let him :)

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 0/3] force IOVA to a particular mode

[Alejandro Lucero]

On Tue, Oct 30, 2018 at 1:48 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 30-Oct-18 12:02 PM, Alejandro Lucero wrote:
> >
> >
> > On Sun, Oct 28, 2018 at 11:04 PM Thomas Monjalon <thomas@monjalon.net
> > <mailto:thomas@monjalon.net>> wrote:
> >
> >     11/10/2018 12:08, Thomas Monjalon:
> >      > +Cc more maintainers in order to collect more reviews
> >      >
> >      > 04/10/2018 11:19, Burakov, Anatoly:
> >      > > On 03-Oct-18 9:53 PM, eric zhang wrote:
> >      > > > This patchset introduces an EAL command line option
> "--iova-mode"
> >      > > > to give the user a facility to force IOVA mode to a special
> >     value.
> >      > > >
> >      > > > Auto detection of the IOVA mode, based on probing the bus and
> >     IOMMU
> >      > > > configuration, may not report the desired addressing mode
> >     when virtual
> >      > > > devices that are not directly attached to the bus are present.
> >      > > > The EAL command line option "--iova-mode" can be used to
> >     select either
> >      > > > physical addressing('pa') or virtual addressing('va').
> >      > >
> >      > > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com
> >     <mailto:anatoly.burakov@intel.com>>
> >
> >     Rebased and applied, thanks
> >
> >
> >
> > Could not this lead to a problem if a device can not wok with the mode
> set?
> > For example, IOVA mode set to VA and IOMMU hw with less bits than those
> > required for the virtual addresses?
> >
> > IMO any device should be attached to a bus, and a bus should have a
> > function for setting IOVA mode and the --iova-mode option just allowed
> > with supported IOVA modes within the bus.
>
> I don't think it should work that way. It should warn the user that an
> incompatible IOVA mode was selected, but the user has specified an IOVA
> mode for a reason - it probably implies he really means it, so let him :)
>
>
Yes, that's true, but setting IOVA VA when a device has problems with it,
it is a bad idea and it could lead to a system crash.
My concern is not with the user knowing what he is doing but with the user
that "uhmm, what is this for, let's try this option".


> --
> Thanks,
> Anatoly
>