* Re: [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities
2019-08-02 10:14 [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities David Marchand
@ 2019-08-02 10:20 ` David Marchand
2019-08-02 17:13 ` David Christensen
2019-08-02 17:10 ` David Christensen
` (3 subsequent siblings)
4 siblings, 1 reply; 9+ messages in thread
From: David Marchand @ 2019-08-02 10:20 UTC (permalink / raw)
To: Takeshi Yoshimura, David Christensen; +Cc: Burakov, Anatoly, dev
On Fri, Aug 2, 2019 at 12:14 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> IOMMU capabilities won't change and must be checked even if no PCI device
> seem to be supported yet when EAL initialised.
>
> This is to accommodate with SPDK that registers its drivers after
> rte_eal_init(), especially on PPC platform where the IOMMU does not
> support VA.
David, Takeshi,
Can you test your setups with (only) this patch?
This should address both of your issues.
>
> Fixes: 703458e19c16 ("bus/pci: consider only usable devices for IOVA mode")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> drivers/bus/pci/bsd/pci.c | 6 ++++++
> drivers/bus/pci/linux/pci.c | 25 ++++++-------------------
> drivers/bus/pci/pci_common.c | 16 +++++++++++++++-
> drivers/bus/pci/private.h | 5 ++++-
> 4 files changed, 31 insertions(+), 21 deletions(-)
>
[snip]
> diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
> index 9794552..8d1d6ab 100644
> --- a/drivers/bus/pci/pci_common.c
> +++ b/drivers/bus/pci/pci_common.c
[snip]
> @@ -643,7 +651,13 @@ rte_pci_get_iommu_class(void)
> devices_want_va = true;
> }
> }
> - if (devices_want_va && !devices_want_pa) {
> + if (iommu_no_va == 1) {
> + iova_mode = RTE_IOVA_PA;
> + if (devices_want_va) {
> + RTE_LOG(WARNING, EAL, "Some devices want 'VA' but because IOMMU does not support 'VA'.\n");
Note to self, s/but because/but/
> + RTE_LOG(WARNING, EAL, "The devices that want 'VA' won't initialize.\n");
> + }
> + } else if (devices_want_va && !devices_want_pa) {
> iova_mode = RTE_IOVA_VA;
> } else if (devices_want_pa && !devices_want_va) {
> iova_mode = RTE_IOVA_PA;
--
David Marchand
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities
2019-08-02 10:20 ` David Marchand
@ 2019-08-02 17:13 ` David Christensen
0 siblings, 0 replies; 9+ messages in thread
From: David Christensen @ 2019-08-02 17:13 UTC (permalink / raw)
To: David Marchand, Takeshi Yoshimura; +Cc: Burakov, Anatoly, dev
> David, Takeshi,
>
> Can you test your setups with (only) this patch?
> This should address both of your issues.
Correctly identifies my system as requiring PA mode:
sudo LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib
/home/davec/src/dpdk/build/app/testpmd --log-level="lib.*:debug" -w
0000:01:00.0,txq_inline_min=0 -w 0000:01:00.1,txq_inline_min=0 -l 4-59
-n 4 -- --rxq=8 --rxd=1024 --txq=8 --txd=4096 --nb-cores=16 -i -a --numa
--forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or
directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'PA'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Selected IOVA mode 'PA'
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities
2019-08-02 10:14 [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities David Marchand
2019-08-02 10:20 ` David Marchand
@ 2019-08-02 17:10 ` David Christensen
2019-08-03 18:22 ` Jerin Jacob Kollanukkaran
` (2 subsequent siblings)
4 siblings, 0 replies; 9+ messages in thread
From: David Christensen @ 2019-08-02 17:10 UTC (permalink / raw)
To: David Marchand, dev; +Cc: anatoly.burakov, tyos
> IOMMU capabilities won't change and must be checked even if no PCI device
> seem to be supported yet when EAL initialised.
>
> This is to accommodate with SPDK that registers its drivers after
> rte_eal_init(), especially on PPC platform where the IOMMU does not
> support VA.
>
> Fixes: 703458e19c16 ("bus/pci: consider only usable devices for IOVA mode")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities
2019-08-02 10:14 [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities David Marchand
2019-08-02 10:20 ` David Marchand
2019-08-02 17:10 ` David Christensen
@ 2019-08-03 18:22 ` Jerin Jacob Kollanukkaran
2019-08-05 3:57 ` Takeshi T Yoshimura
2019-08-05 6:23 ` [dpdk-dev] [PATCH v2] " David Marchand
4 siblings, 0 replies; 9+ messages in thread
From: Jerin Jacob Kollanukkaran @ 2019-08-03 18:22 UTC (permalink / raw)
To: David Marchand, dev; +Cc: anatoly.burakov, tyos, drc
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of David Marchand
> Sent: Friday, August 2, 2019 3:45 PM
> To: dev@dpdk.org
> Cc: anatoly.burakov@intel.com; tyos@jp.ibm.com; drc@linux.vnet.ibm.com
> Subject: [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities
>
> IOMMU capabilities won't change and must be checked even if no PCI device
> seem to be supported yet when EAL initialised.
>
> This is to accommodate with SPDK that registers its drivers after
> rte_eal_init(), especially on PPC platform where the IOMMU does not
> support VA.
>
> Fixes: 703458e19c16 ("bus/pci: consider only usable devices for IOVA mode")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> drivers/bus/pci/bsd/pci.c | 6 ++++++
> drivers/bus/pci/linux/pci.c | 25 ++++++-------------------
> drivers/bus/pci/pci_common.c | 16 +++++++++++++++-
> drivers/bus/pci/private.h | 5 ++++-
> 4 files changed, 31 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
> index 9794552..8d1d6ab 100644
> --- a/drivers/bus/pci/pci_common.c
> +++ b/drivers/bus/pci/pci_common.c
> @@ -616,8 +616,16 @@ rte_pci_get_iommu_class(void)
> const struct rte_pci_driver *drv;
> bool devices_want_va = false;
> bool devices_want_pa = false;
> + static int iommu_no_va = -1;
I think, "static" is not required in this context of rte_pci_get_iommu_class()
function. Not a big deal, Trying to avoid unnecessary global variable here.
Either way,
Acked-by: Jerin Jacob <jerinj@marvell.com>
Tested-by: Jerin Jacob <jerinj@marvell.com>
> FOREACH_DEVICE_ON_PCIBUS(dev) {
> + /*
> + * We can check this only once, because the IOMMU
> hardware is
> + * the same for all of them.
> + */
> + if (iommu_no_va == -1)
> + iommu_no_va =
> pci_device_iommu_support_va(dev)
> + ? 0 : 1;
> if (pci_ignore_device(dev))
> continue;
> if (dev->kdrv == RTE_KDRV_UNKNOWN ||
> @@ -643,7 +651,13 @@ rte_pci_get_iommu_class(void)
> devices_want_va = true;
> }
> }
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities
2019-08-02 10:14 [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities David Marchand
` (2 preceding siblings ...)
2019-08-03 18:22 ` Jerin Jacob Kollanukkaran
@ 2019-08-05 3:57 ` Takeshi T Yoshimura
2019-08-05 6:16 ` David Marchand
2019-08-05 6:23 ` [dpdk-dev] [PATCH v2] " David Marchand
4 siblings, 1 reply; 9+ messages in thread
From: Takeshi T Yoshimura @ 2019-08-05 3:57 UTC (permalink / raw)
To: David Marchand; +Cc: dev, anatoly.burakov, drc
-----David Marchand <david.marchand@redhat.com> wrote: -----
>To: dev@dpdk.org
>From: David Marchand <david.marchand@redhat.com>
>Date: 08/02/2019 07:14PM
>Cc: anatoly.burakov@intel.com, tyos@jp.ibm.com,
>drc@linux.vnet.ibm.com
>Subject: [PATCH] bus/pci: always check IOMMU capabilities
>
>IOMMU capabilities won't change and must be checked even if no PCI
>device
>seem to be supported yet when EAL initialised.
>
>This is to accommodate with SPDK that registers its drivers after
>rte_eal_init(), especially on PPC platform where the IOMMU does not
>support VA.
>
>Fixes: 703458e19c16 ("bus/pci: consider only usable devices for IOVA
>mode")
>
>Signed-off-by: David Marchand <david.marchand@redhat.com>
>---
> drivers/bus/pci/bsd/pci.c | 6 ++++++
> drivers/bus/pci/linux/pci.c | 25 ++++++-------------------
> drivers/bus/pci/pci_common.c | 16 +++++++++++++++-
> drivers/bus/pci/private.h | 5 ++++-
> 4 files changed, 31 insertions(+), 21 deletions(-)
>
>diff --git a/drivers/bus/pci/bsd/pci.c b/drivers/bus/pci/bsd/pci.c
>index a2de709..8f07ed9 100644
>--- a/drivers/bus/pci/bsd/pci.c
>+++ b/drivers/bus/pci/bsd/pci.c
>@@ -376,6 +376,12 @@ error:
> return -1;
> }
>
>+bool
>+pci_device_iommu_support_va(__rte_unused const struct rte_pci_device
>*dev)
>+{
>+ return false;
>+}
>+
> enum rte_iova_mode
> pci_device_iova_mode(const struct rte_pci_driver *pdrv __rte_unused,
> const struct rte_pci_device *pdev)
>diff --git a/drivers/bus/pci/linux/pci.c
>b/drivers/bus/pci/linux/pci.c
>index f4fb742..43debaa 100644
>--- a/drivers/bus/pci/linux/pci.c
>+++ b/drivers/bus/pci/linux/pci.c
>@@ -498,8 +498,8 @@ error:
> }
>
> #if defined(RTE_ARCH_X86)
>-static bool
>-pci_one_device_iommu_support_va(const struct rte_pci_device *dev)
>+bool
>+pci_device_iommu_support_va(const struct rte_pci_device *dev)
> {
> #define VTD_CAP_MGAW_SHIFT 16
> #define VTD_CAP_MGAW_MASK (0x3fULL << VTD_CAP_MGAW_SHIFT)
>@@ -546,14 +546,14 @@ pci_one_device_iommu_support_va(const struct
>rte_pci_device *dev)
> return true;
> }
> #elif defined(RTE_ARCH_PPC_64)
>-static bool
>-pci_one_device_iommu_support_va(__rte_unused const struct
>rte_pci_device *dev)
>+bool
>+pci_device_iommu_support_va(__rte_unused const struct rte_pci_device
>*dev)
> {
> return false;
> }
> #else
>-static bool
>-pci_one_device_iommu_support_va(__rte_unused const struct
>rte_pci_device *dev)
>+bool
>+pci_device_iommu_support_va(__rte_unused const struct rte_pci_device
>*dev)
> {
> return true;
> }
>@@ -564,7 +564,6 @@ pci_device_iova_mode(const struct rte_pci_driver
>*pdrv,
> const struct rte_pci_device *pdev)
> {
> enum rte_iova_mode iova_mode = RTE_IOVA_DC;
>- static int iommu_no_va = -1;
>
> switch (pdev->kdrv) {
> case RTE_KDRV_VFIO: {
>@@ -595,18 +594,6 @@ pci_device_iova_mode(const struct rte_pci_driver
>*pdrv,
> iova_mode = RTE_IOVA_VA;
> break;
> }
>-
>- if (iova_mode != RTE_IOVA_PA) {
>- /*
>- * We can check this only once, because the IOMMU hardware is
>- * the same for all of them.
>- */
>- if (iommu_no_va == -1)
>- iommu_no_va = pci_one_device_iommu_support_va(pdev)
>- ? 0 : 1;
>- if (iommu_no_va != 0)
>- iova_mode = RTE_IOVA_PA;
>- }
> return iova_mode;
> }
>
>diff --git a/drivers/bus/pci/pci_common.c
>b/drivers/bus/pci/pci_common.c
>index 9794552..8d1d6ab 100644
>--- a/drivers/bus/pci/pci_common.c
>+++ b/drivers/bus/pci/pci_common.c
>@@ -616,8 +616,16 @@ rte_pci_get_iommu_class(void)
> const struct rte_pci_driver *drv;
> bool devices_want_va = false;
> bool devices_want_pa = false;
>+ static int iommu_no_va = -1;
>
> FOREACH_DEVICE_ON_PCIBUS(dev) {
>+ /*
>+ * We can check this only once, because the IOMMU hardware is
>+ * the same for all of them.
>+ */
>+ if (iommu_no_va == -1)
>+ iommu_no_va = pci_device_iommu_support_va(dev)
>+ ? 0 : 1;
> if (pci_ignore_device(dev))
> continue;
> if (dev->kdrv == RTE_KDRV_UNKNOWN ||
>@@ -643,7 +651,13 @@ rte_pci_get_iommu_class(void)
> devices_want_va = true;
> }
> }
>- if (devices_want_va && !devices_want_pa) {
>+ if (iommu_no_va == 1) {
>+ iova_mode = RTE_IOVA_PA;
>+ if (devices_want_va) {
>+ RTE_LOG(WARNING, EAL, "Some devices want 'VA' but because IOMMU
>does not support 'VA'.\n");
>+ RTE_LOG(WARNING, EAL, "The devices that want 'VA' won't
>initialize.\n");
>+ }
>+ } else if (devices_want_va && !devices_want_pa) {
> iova_mode = RTE_IOVA_VA;
> } else if (devices_want_pa && !devices_want_va) {
> iova_mode = RTE_IOVA_PA;
>diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
>index 8a55240..a205d4d 100644
>--- a/drivers/bus/pci/private.h
>+++ b/drivers/bus/pci/private.h
>@@ -173,9 +173,12 @@ rte_pci_match(const struct rte_pci_driver
>*pci_drv,
> const struct rte_pci_device *pci_dev);
>
> /**
>- * OS specific callback for rte_pci_get_iommu_class
>+ * OS specific callbacks for rte_pci_get_iommu_class
> *
> */
>+bool
>+pci_device_iommu_support_va(const struct rte_pci_device *dev);
>+
> enum rte_iova_mode
> pci_device_iova_mode(const struct rte_pci_driver *pci_drv,
> const struct rte_pci_device *pci_dev);
>--
>1.8.3.1
>
>
Tested-by: Takeshi Yoshimura <tyos@jp.ibm.com>
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities
2019-08-05 3:57 ` Takeshi T Yoshimura
@ 2019-08-05 6:16 ` David Marchand
0 siblings, 0 replies; 9+ messages in thread
From: David Marchand @ 2019-08-05 6:16 UTC (permalink / raw)
To: Takeshi T Yoshimura
Cc: dev, Burakov, Anatoly, David Christensen, Jerin Jacob Kollanukkaran
On Mon, Aug 5, 2019 at 5:57 AM Takeshi T Yoshimura <TYOS@jp.ibm.com> wrote:
>
> -----David Marchand <david.marchand@redhat.com> wrote: -----
>
> >To: dev@dpdk.org
> >From: David Marchand <david.marchand@redhat.com>
> >Date: 08/02/2019 07:14PM
> >Cc: anatoly.burakov@intel.com, tyos@jp.ibm.com,
> >drc@linux.vnet.ibm.com
> >Subject: [PATCH] bus/pci: always check IOMMU capabilities
> >
> >IOMMU capabilities won't change and must be checked even if no PCI
> >device
> >seem to be supported yet when EAL initialised.
> >
> >This is to accommodate with SPDK that registers its drivers after
> >rte_eal_init(), especially on PPC platform where the IOMMU does not
> >support VA.
> >
> >Fixes: 703458e19c16 ("bus/pci: consider only usable devices for IOVA
> >mode")
> >
> >Signed-off-by: David Marchand <david.marchand@redhat.com>
> >---
> > drivers/bus/pci/bsd/pci.c | 6 ++++++
> > drivers/bus/pci/linux/pci.c | 25 ++++++-------------------
> > drivers/bus/pci/pci_common.c | 16 +++++++++++++++-
> > drivers/bus/pci/private.h | 5 ++++-
> > 4 files changed, 31 insertions(+), 21 deletions(-)
> >
> >diff --git a/drivers/bus/pci/bsd/pci.c b/drivers/bus/pci/bsd/pci.c
> >index a2de709..8f07ed9 100644
> >--- a/drivers/bus/pci/bsd/pci.c
> >+++ b/drivers/bus/pci/bsd/pci.c
> >@@ -376,6 +376,12 @@ error:
> > return -1;
> > }
> >
> >+bool
> >+pci_device_iommu_support_va(__rte_unused const struct rte_pci_device
> >*dev)
> >+{
> >+ return false;
> >+}
> >+
> > enum rte_iova_mode
> > pci_device_iova_mode(const struct rte_pci_driver *pdrv __rte_unused,
> > const struct rte_pci_device *pdev)
> >diff --git a/drivers/bus/pci/linux/pci.c
> >b/drivers/bus/pci/linux/pci.c
> >index f4fb742..43debaa 100644
> >--- a/drivers/bus/pci/linux/pci.c
> >+++ b/drivers/bus/pci/linux/pci.c
> >@@ -498,8 +498,8 @@ error:
> > }
> >
> > #if defined(RTE_ARCH_X86)
> >-static bool
> >-pci_one_device_iommu_support_va(const struct rte_pci_device *dev)
> >+bool
> >+pci_device_iommu_support_va(const struct rte_pci_device *dev)
> > {
> > #define VTD_CAP_MGAW_SHIFT 16
> > #define VTD_CAP_MGAW_MASK (0x3fULL << VTD_CAP_MGAW_SHIFT)
> >@@ -546,14 +546,14 @@ pci_one_device_iommu_support_va(const struct
> >rte_pci_device *dev)
> > return true;
> > }
> > #elif defined(RTE_ARCH_PPC_64)
> >-static bool
> >-pci_one_device_iommu_support_va(__rte_unused const struct
> >rte_pci_device *dev)
> >+bool
> >+pci_device_iommu_support_va(__rte_unused const struct rte_pci_device
> >*dev)
> > {
> > return false;
> > }
> > #else
> >-static bool
> >-pci_one_device_iommu_support_va(__rte_unused const struct
> >rte_pci_device *dev)
> >+bool
> >+pci_device_iommu_support_va(__rte_unused const struct rte_pci_device
> >*dev)
> > {
> > return true;
> > }
> >@@ -564,7 +564,6 @@ pci_device_iova_mode(const struct rte_pci_driver
> >*pdrv,
> > const struct rte_pci_device *pdev)
> > {
> > enum rte_iova_mode iova_mode = RTE_IOVA_DC;
> >- static int iommu_no_va = -1;
> >
> > switch (pdev->kdrv) {
> > case RTE_KDRV_VFIO: {
> >@@ -595,18 +594,6 @@ pci_device_iova_mode(const struct rte_pci_driver
> >*pdrv,
> > iova_mode = RTE_IOVA_VA;
> > break;
> > }
> >-
> >- if (iova_mode != RTE_IOVA_PA) {
> >- /*
> >- * We can check this only once, because the IOMMU hardware is
> >- * the same for all of them.
> >- */
> >- if (iommu_no_va == -1)
> >- iommu_no_va = pci_one_device_iommu_support_va(pdev)
> >- ? 0 : 1;
> >- if (iommu_no_va != 0)
> >- iova_mode = RTE_IOVA_PA;
> >- }
> > return iova_mode;
> > }
> >
> >diff --git a/drivers/bus/pci/pci_common.c
> >b/drivers/bus/pci/pci_common.c
> >index 9794552..8d1d6ab 100644
> >--- a/drivers/bus/pci/pci_common.c
> >+++ b/drivers/bus/pci/pci_common.c
> >@@ -616,8 +616,16 @@ rte_pci_get_iommu_class(void)
> > const struct rte_pci_driver *drv;
> > bool devices_want_va = false;
> > bool devices_want_pa = false;
> >+ static int iommu_no_va = -1;
> >
> > FOREACH_DEVICE_ON_PCIBUS(dev) {
> >+ /*
> >+ * We can check this only once, because the IOMMU hardware is
> >+ * the same for all of them.
> >+ */
> >+ if (iommu_no_va == -1)
> >+ iommu_no_va = pci_device_iommu_support_va(dev)
> >+ ? 0 : 1;
> > if (pci_ignore_device(dev))
> > continue;
> > if (dev->kdrv == RTE_KDRV_UNKNOWN ||
> >@@ -643,7 +651,13 @@ rte_pci_get_iommu_class(void)
> > devices_want_va = true;
> > }
> > }
> >- if (devices_want_va && !devices_want_pa) {
> >+ if (iommu_no_va == 1) {
> >+ iova_mode = RTE_IOVA_PA;
> >+ if (devices_want_va) {
> >+ RTE_LOG(WARNING, EAL, "Some devices want 'VA' but because IOMMU
> >does not support 'VA'.\n");
> >+ RTE_LOG(WARNING, EAL, "The devices that want 'VA' won't
> >initialize.\n");
> >+ }
> >+ } else if (devices_want_va && !devices_want_pa) {
> > iova_mode = RTE_IOVA_VA;
> > } else if (devices_want_pa && !devices_want_va) {
> > iova_mode = RTE_IOVA_PA;
> >diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
> >index 8a55240..a205d4d 100644
> >--- a/drivers/bus/pci/private.h
> >+++ b/drivers/bus/pci/private.h
> >@@ -173,9 +173,12 @@ rte_pci_match(const struct rte_pci_driver
> >*pci_drv,
> > const struct rte_pci_device *pci_dev);
> >
> > /**
> >- * OS specific callback for rte_pci_get_iommu_class
> >+ * OS specific callbacks for rte_pci_get_iommu_class
> > *
> > */
> >+bool
> >+pci_device_iommu_support_va(const struct rte_pci_device *dev);
> >+
> > enum rte_iova_mode
> > pci_device_iova_mode(const struct rte_pci_driver *pci_drv,
> > const struct rte_pci_device *pci_dev);
> >--
> >1.8.3.1
> >
> >
>
> Tested-by: Takeshi Yoshimura <tyos@jp.ibm.com>
Thank you all.
I'll send a v2 with the comment from Jerin and the log message fix.
--
David Marchand
^ permalink raw reply [flat|nested] 9+ messages in thread
* [dpdk-dev] [PATCH v2] bus/pci: always check IOMMU capabilities
2019-08-02 10:14 [dpdk-dev] [PATCH] bus/pci: always check IOMMU capabilities David Marchand
` (3 preceding siblings ...)
2019-08-05 3:57 ` Takeshi T Yoshimura
@ 2019-08-05 6:23 ` David Marchand
2019-08-05 10:09 ` Thomas Monjalon
4 siblings, 1 reply; 9+ messages in thread
From: David Marchand @ 2019-08-05 6:23 UTC (permalink / raw)
To: dev; +Cc: anatoly.burakov, tyos, drc, jerinj
IOMMU capabilities won't change and must be checked even if no PCI device
seem to be supported yet when EAL initialised.
This is to accommodate with SPDK that registers its drivers after
rte_eal_init(), especially on PPC platform where the IOMMU does not
support VA.
Fixes: 703458e19c16 ("bus/pci: consider only usable devices for IOVA mode")
Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>
Tested-by: Jerin Jacob <jerinj@marvell.com>
Tested-by: Takeshi Yoshimura <tyos@jp.ibm.com>
---
Changelog since v1:
- fixed log message
- changed static global variable iommu_no_va as a local variable
---
drivers/bus/pci/bsd/pci.c | 6 ++++++
drivers/bus/pci/linux/pci.c | 25 ++++++-------------------
drivers/bus/pci/pci_common.c | 16 +++++++++++++++-
drivers/bus/pci/private.h | 5 ++++-
4 files changed, 31 insertions(+), 21 deletions(-)
diff --git a/drivers/bus/pci/bsd/pci.c b/drivers/bus/pci/bsd/pci.c
index a2de709..8f07ed9 100644
--- a/drivers/bus/pci/bsd/pci.c
+++ b/drivers/bus/pci/bsd/pci.c
@@ -376,6 +376,12 @@ error:
return -1;
}
+bool
+pci_device_iommu_support_va(__rte_unused const struct rte_pci_device *dev)
+{
+ return false;
+}
+
enum rte_iova_mode
pci_device_iova_mode(const struct rte_pci_driver *pdrv __rte_unused,
const struct rte_pci_device *pdev)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index f4fb742..43debaa 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -498,8 +498,8 @@ error:
}
#if defined(RTE_ARCH_X86)
-static bool
-pci_one_device_iommu_support_va(const struct rte_pci_device *dev)
+bool
+pci_device_iommu_support_va(const struct rte_pci_device *dev)
{
#define VTD_CAP_MGAW_SHIFT 16
#define VTD_CAP_MGAW_MASK (0x3fULL << VTD_CAP_MGAW_SHIFT)
@@ -546,14 +546,14 @@ pci_one_device_iommu_support_va(const struct rte_pci_device *dev)
return true;
}
#elif defined(RTE_ARCH_PPC_64)
-static bool
-pci_one_device_iommu_support_va(__rte_unused const struct rte_pci_device *dev)
+bool
+pci_device_iommu_support_va(__rte_unused const struct rte_pci_device *dev)
{
return false;
}
#else
-static bool
-pci_one_device_iommu_support_va(__rte_unused const struct rte_pci_device *dev)
+bool
+pci_device_iommu_support_va(__rte_unused const struct rte_pci_device *dev)
{
return true;
}
@@ -564,7 +564,6 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
const struct rte_pci_device *pdev)
{
enum rte_iova_mode iova_mode = RTE_IOVA_DC;
- static int iommu_no_va = -1;
switch (pdev->kdrv) {
case RTE_KDRV_VFIO: {
@@ -595,18 +594,6 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
iova_mode = RTE_IOVA_VA;
break;
}
-
- if (iova_mode != RTE_IOVA_PA) {
- /*
- * We can check this only once, because the IOMMU hardware is
- * the same for all of them.
- */
- if (iommu_no_va == -1)
- iommu_no_va = pci_one_device_iommu_support_va(pdev)
- ? 0 : 1;
- if (iommu_no_va != 0)
- iova_mode = RTE_IOVA_PA;
- }
return iova_mode;
}
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 9794552..6b46b4f 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -616,8 +616,16 @@ rte_pci_get_iommu_class(void)
const struct rte_pci_driver *drv;
bool devices_want_va = false;
bool devices_want_pa = false;
+ int iommu_no_va = -1;
FOREACH_DEVICE_ON_PCIBUS(dev) {
+ /*
+ * We can check this only once, because the IOMMU hardware is
+ * the same for all of them.
+ */
+ if (iommu_no_va == -1)
+ iommu_no_va = pci_device_iommu_support_va(dev)
+ ? 0 : 1;
if (pci_ignore_device(dev))
continue;
if (dev->kdrv == RTE_KDRV_UNKNOWN ||
@@ -643,7 +651,13 @@ rte_pci_get_iommu_class(void)
devices_want_va = true;
}
}
- if (devices_want_va && !devices_want_pa) {
+ if (iommu_no_va == 1) {
+ iova_mode = RTE_IOVA_PA;
+ if (devices_want_va) {
+ RTE_LOG(WARNING, EAL, "Some devices want 'VA' but IOMMU does not support 'VA'.\n");
+ RTE_LOG(WARNING, EAL, "The devices that want 'VA' won't initialize.\n");
+ }
+ } else if (devices_want_va && !devices_want_pa) {
iova_mode = RTE_IOVA_VA;
} else if (devices_want_pa && !devices_want_va) {
iova_mode = RTE_IOVA_PA;
diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h
index 8a55240..a205d4d 100644
--- a/drivers/bus/pci/private.h
+++ b/drivers/bus/pci/private.h
@@ -173,9 +173,12 @@ rte_pci_match(const struct rte_pci_driver *pci_drv,
const struct rte_pci_device *pci_dev);
/**
- * OS specific callback for rte_pci_get_iommu_class
+ * OS specific callbacks for rte_pci_get_iommu_class
*
*/
+bool
+pci_device_iommu_support_va(const struct rte_pci_device *dev);
+
enum rte_iova_mode
pci_device_iova_mode(const struct rte_pci_driver *pci_drv,
const struct rte_pci_device *pci_dev);
--
1.8.3.1
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [dpdk-dev] [PATCH v2] bus/pci: always check IOMMU capabilities
2019-08-05 6:23 ` [dpdk-dev] [PATCH v2] " David Marchand
@ 2019-08-05 10:09 ` Thomas Monjalon
0 siblings, 0 replies; 9+ messages in thread
From: Thomas Monjalon @ 2019-08-05 10:09 UTC (permalink / raw)
To: David Marchand; +Cc: dev, anatoly.burakov, tyos, drc, jerinj
05/08/2019 08:23, David Marchand:
> IOMMU capabilities won't change and must be checked even if no PCI device
> seem to be supported yet when EAL initialised.
>
> This is to accommodate with SPDK that registers its drivers after
> rte_eal_init(), especially on PPC platform where the IOMMU does not
> support VA.
>
> Fixes: 703458e19c16 ("bus/pci: consider only usable devices for IOVA mode")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>
> Acked-by: Jerin Jacob <jerinj@marvell.com>
> Tested-by: Jerin Jacob <jerinj@marvell.com>
> Tested-by: Takeshi Yoshimura <tyos@jp.ibm.com>
Applied, thanks
^ permalink raw reply [flat|nested] 9+ messages in thread