[dpdk-dev] [PATCH] vfio: make container open error non-fatal

[dpdk-dev] [PATCH] vfio: make container open error non-fatal

[Bruce Richardson]

When setting up an app to run using the uio driver, errors caused by
VFIO failures should not abruptly cause the app to fail.

Example: on a board with 8 ports bound to igb_uio module, and no VFIO
configuration, a testpmd run currently fails with:

EAL:   cannot open VFIO container!
EAL:   0000:04:00.0 cannot open VFIO container!
EAL: Error - exiting with code: 1
  Cause: Requested device 0000:04:00.0 cannot be used

With this patch applied, the problem with VFIO is ignored and testpmd
successfully starts up - with ignored errors with vfio - as below:

EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   unknown IOMMU driver!
EAL:   0000:04:00.0 cannot open VFIO container!
EAL:   0000:04:00.0 not managed by UIO driver, skipping
<...scan results for other ports skipped...>
EAL: PCI device 0000:8e:00.0 on NUMA socket 1
EAL:   probe driver: 8086:154a rte_ixgbe_pmd
EAL:   unknown IOMMU driver!
EAL:   0000:8e:00.0 cannot open VFIO container!
EAL:   PCI memory mapped at 0x7ff4ff5fa000
EAL:   PCI memory mapped at 0x7ff4ff5f6000
EAL: PCI device 0000:8e:00.1 on NUMA socket 1
EAL:   probe driver: 8086:154a rte_ixgbe_pmd
EAL:   unknown IOMMU driver!
EAL:   0000:8e:00.1 cannot open VFIO container!
EAL:   PCI memory mapped at 0x7ff4ff4f6000
EAL:   PCI memory mapped at 0x7ff4ff4f2000
Interactive-mode selected
Configuring Port 0 (socket 0)
<...other 7 ports ...>
Checking link statuses...
Port 0 Link Up - speed 10000 Mbps - full-duplex
Port 1 Link Down
Port 2 Link Up - speed 10000 Mbps - full-duplex
Port 3 Link Down
Port 4 Link Up - speed 10000 Mbps - full-duplex
Port 5 Link Down
Port 6 Link Up - speed 10000 Mbps - full-duplex
Port 7 Link Down
Done
testpmd>

This issue is introduced by the VFIO patch set addition, specifically
commit ff0b67d1.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index 4de6061..4af38f6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -528,7 +528,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		int vfio_container_fd = pci_vfio_get_container_fd();
 		if (vfio_container_fd < 0) {
 			RTE_LOG(ERR, EAL, "  %s cannot open VFIO container!\n", pci_addr);
-			return -1;
+			return 1;
 		}
 
 		vfio_cfg.vfio_container_fd = vfio_container_fd;
-- 
1.9.3

Re: [dpdk-dev] [PATCH] vfio: make container open error non-fatal

[Richardson, Bruce]

The below patch is the quickest fix I found to make my applications work again, but I'm not sure it's the best solution. Can anyone else offer other suggestions to improve this?

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Monday, June 16, 2014 3:29 PM
> To: dev@dpdk.org
> Cc: Richardson, Bruce
> Subject: [PATCH] vfio: make container open error non-fatal
> 
> When setting up an app to run using the uio driver, errors caused by
> VFIO failures should not abruptly cause the app to fail.
> 
> Example: on a board with 8 ports bound to igb_uio module, and no VFIO
> configuration, a testpmd run currently fails with:
> 
> EAL:   cannot open VFIO container!
> EAL:   0000:04:00.0 cannot open VFIO container!
> EAL: Error - exiting with code: 1
>   Cause: Requested device 0000:04:00.0 cannot be used
> 
> With this patch applied, the problem with VFIO is ignored and testpmd
> successfully starts up - with ignored errors with vfio - as below:
> 
> EAL: PCI device 0000:04:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 rte_igb_pmd
> EAL:   unknown IOMMU driver!
> EAL:   0000:04:00.0 cannot open VFIO container!
> EAL:   0000:04:00.0 not managed by UIO driver, skipping
> <...scan results for other ports skipped...>
> EAL: PCI device 0000:8e:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:154a rte_ixgbe_pmd
> EAL:   unknown IOMMU driver!
> EAL:   0000:8e:00.0 cannot open VFIO container!
> EAL:   PCI memory mapped at 0x7ff4ff5fa000
> EAL:   PCI memory mapped at 0x7ff4ff5f6000
> EAL: PCI device 0000:8e:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:154a rte_ixgbe_pmd
> EAL:   unknown IOMMU driver!
> EAL:   0000:8e:00.1 cannot open VFIO container!
> EAL:   PCI memory mapped at 0x7ff4ff4f6000
> EAL:   PCI memory mapped at 0x7ff4ff4f2000
> Interactive-mode selected
> Configuring Port 0 (socket 0)
> <...other 7 ports ...>
> Checking link statuses...
> Port 0 Link Up - speed 10000 Mbps - full-duplex
> Port 1 Link Down
> Port 2 Link Up - speed 10000 Mbps - full-duplex
> Port 3 Link Down
> Port 4 Link Up - speed 10000 Mbps - full-duplex
> Port 5 Link Down
> Port 6 Link Up - speed 10000 Mbps - full-duplex
> Port 7 Link Down
> Done
> testpmd>
> 
> This issue is introduced by the VFIO patch set addition, specifically
> commit ff0b67d1.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> index 4de6061..4af38f6 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> @@ -528,7 +528,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
>  		int vfio_container_fd = pci_vfio_get_container_fd();
>  		if (vfio_container_fd < 0) {
>  			RTE_LOG(ERR, EAL, "  %s cannot open VFIO
> container!\n", pci_addr);
> -			return -1;
> +			return 1;
>  		}
> 
>  		vfio_cfg.vfio_container_fd = vfio_container_fd;
> --
> 1.9.3

Re: [dpdk-dev] [PATCH] vfio: make container open error non-fatal

[Burakov, Anatoly]

Hi Bruce,

> The below patch is the quickest fix I found to make my applications work
> again, but I'm not sure it's the best solution. Can anyone else offer other
> suggestions to improve this?

Are you running things as root? If not, I suggest to try and use the setup.sh script to correct permissions on the VFIO container and see if it works.

The inability of opening a container is likely a problem with permissions on the container, and thus should be considered fatal as far as VFIO is concerned. However, given that we try to use VFIO unconditionally, I think your suggestion is a good solution to the problem, however I would also close the group fd's that were already opened before returning 1.

Best regards,
Anatoly Burakov
DPDK SW Engineer

Re: [dpdk-dev] [PATCH] vfio: make container open error non-fatal

[Burakov, Anatoly]

Hi Bruce,

> Hi Bruce,
> 
> > The below patch is the quickest fix I found to make my applications
> > work again, but I'm not sure it's the best solution. Can anyone else
> > offer other suggestions to improve this?
> 
> Are you running things as root? If not, I suggest to try and use the setup.sh
> script to correct permissions on the VFIO container and see if it works.
> 
> The inability of opening a container is likely a problem with permissions on
> the container, and thus should be considered fatal as far as VFIO is
> concerned. However, given that we try to use VFIO unconditionally, I think
> your suggestion is a good solution to the problem, however I would also
> close the group fd's that were already opened before returning 1.

On a second thought, I think this may be better solved by checking access() on the container. Right now I think PCI init checks for access on /dev/vfio (the VFIO dir) but not /dev/vfio/vfio (the container). I will prepare a patch shortly, so I would appreciate if you self-NAKed yours :-)

Best regards,
Anatoly Burakov
DPDK SW Engineer

Re: [dpdk-dev] [PATCH] vfio: make container open error non-fatal

[Neil Horman]

On Mon, Jun 16, 2014 at 10:30:54PM +0000, Richardson, Bruce wrote:
> The below patch is the quickest fix I found to make my applications work again, but I'm not sure it's the best solution. Can anyone else offer other suggestions to improve this?
> 
> > -----Original Message-----
> > From: Richardson, Bruce
> > Sent: Monday, June 16, 2014 3:29 PM
> > To: dev@dpdk.org
> > Cc: Richardson, Bruce
> > Subject: [PATCH] vfio: make container open error non-fatal
> > 
> > When setting up an app to run using the uio driver, errors caused by
> > VFIO failures should not abruptly cause the app to fail.
> > 
> > Example: on a board with 8 ports bound to igb_uio module, and no VFIO
> > configuration, a testpmd run currently fails with:
> > 
> > EAL:   cannot open VFIO container!
> > EAL:   0000:04:00.0 cannot open VFIO container!
> > EAL: Error - exiting with code: 1
> >   Cause: Requested device 0000:04:00.0 cannot be used
> > 
> > With this patch applied, the problem with VFIO is ignored and testpmd
> > successfully starts up - with ignored errors with vfio - as below:
> > 
> > EAL: PCI device 0000:04:00.0 on NUMA socket 0
> > EAL:   probe driver: 8086:1521 rte_igb_pmd
> > EAL:   unknown IOMMU driver!
> > EAL:   0000:04:00.0 cannot open VFIO container!
> > EAL:   0000:04:00.0 not managed by UIO driver, skipping
> > <...scan results for other ports skipped...>
> > EAL: PCI device 0000:8e:00.0 on NUMA socket 1
> > EAL:   probe driver: 8086:154a rte_ixgbe_pmd
> > EAL:   unknown IOMMU driver!
> > EAL:   0000:8e:00.0 cannot open VFIO container!
> > EAL:   PCI memory mapped at 0x7ff4ff5fa000
> > EAL:   PCI memory mapped at 0x7ff4ff5f6000
> > EAL: PCI device 0000:8e:00.1 on NUMA socket 1
> > EAL:   probe driver: 8086:154a rte_ixgbe_pmd
> > EAL:   unknown IOMMU driver!
> > EAL:   0000:8e:00.1 cannot open VFIO container!
> > EAL:   PCI memory mapped at 0x7ff4ff4f6000
> > EAL:   PCI memory mapped at 0x7ff4ff4f2000
> > Interactive-mode selected
> > Configuring Port 0 (socket 0)
> > <...other 7 ports ...>
> > Checking link statuses...
> > Port 0 Link Up - speed 10000 Mbps - full-duplex
> > Port 1 Link Down
> > Port 2 Link Up - speed 10000 Mbps - full-duplex
> > Port 3 Link Down
> > Port 4 Link Up - speed 10000 Mbps - full-duplex
> > Port 5 Link Down
> > Port 6 Link Up - speed 10000 Mbps - full-duplex
> > Port 7 Link Down
> > Done
> > testpmd>
> > 
> > This issue is introduced by the VFIO patch set addition, specifically
> > commit ff0b67d1.
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> >  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> > b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> > index 4de6061..4af38f6 100644
> > --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> > +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> > @@ -528,7 +528,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
> >  		int vfio_container_fd = pci_vfio_get_container_fd();
> >  		if (vfio_container_fd < 0) {
> >  			RTE_LOG(ERR, EAL, "  %s cannot open VFIO
> > container!\n", pci_addr);
> > -			return -1;
> > +			return 1;
> >  		}
> > 
> >  		vfio_cfg.vfio_container_fd = vfio_container_fd;
> > --
> > 1.9.3
> 
> 

I think it would be preferable to convert the pci_vfio_get_container_fd function
to return not -1, but some -ERRNO value, so that the caller can differentiate
between fatal and non-fatal errors (for instance, not having any vfio container
seems non-fatal, but having one with an incompatible api version may be
unworkable.

Neil

Re: [dpdk-dev] [PATCH] vfio: make container open error non-fatal

[Richardson, Bruce]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, June 17, 2014 1:52 AM
> To: Richardson, Bruce; dev@dpdk.org
> Subject: RE: [PATCH] vfio: make container open error non-fatal
> 
> Hi Bruce,
> 
> > The below patch is the quickest fix I found to make my applications work
> > again, but I'm not sure it's the best solution. Can anyone else offer other
> > suggestions to improve this?
> 
> Are you running things as root? If not, I suggest to try and use the setup.sh script
> to correct permissions on the VFIO container and see if it works.
> 
Same error running with or without sudo. I'm not trying to get vfio to work, I'm trying to find out how to make dpdk ignore vfio since I didn't do any setting up of vfio. :-)

Re: [dpdk-dev] [PATCH] vfio: make container open error non-fatal

[Richardson, Bruce]

> -----Original Message-----
> From: Richardson, Bruce
> Sent: Monday, June 16, 2014 3:29 PM
> To: dev@dpdk.org
> Cc: Richardson, Bruce
> Subject: [PATCH] vfio: make container open error non-fatal
> 
> When setting up an app to run using the uio driver, errors caused by
> VFIO failures should not abruptly cause the app to fail.
> 
> Example: on a board with 8 ports bound to igb_uio module, and no VFIO
> configuration, a testpmd run currently fails with:
> 
> EAL:   cannot open VFIO container!
> EAL:   0000:04:00.0 cannot open VFIO container!
> EAL: Error - exiting with code: 1
>   Cause: Requested device 0000:04:00.0 cannot be used
> 
> With this patch applied, the problem with VFIO is ignored and testpmd
> successfully starts up - with ignored errors with vfio - as below:
> 
> EAL: PCI device 0000:04:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 rte_igb_pmd
> EAL:   unknown IOMMU driver!
> EAL:   0000:04:00.0 cannot open VFIO container!
> EAL:   0000:04:00.0 not managed by UIO driver, skipping
> <...scan results for other ports skipped...>
> EAL: PCI device 0000:8e:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:154a rte_ixgbe_pmd
> EAL:   unknown IOMMU driver!
> EAL:   0000:8e:00.0 cannot open VFIO container!
> EAL:   PCI memory mapped at 0x7ff4ff5fa000
> EAL:   PCI memory mapped at 0x7ff4ff5f6000
> EAL: PCI device 0000:8e:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:154a rte_ixgbe_pmd
> EAL:   unknown IOMMU driver!
> EAL:   0000:8e:00.1 cannot open VFIO container!
> EAL:   PCI memory mapped at 0x7ff4ff4f6000
> EAL:   PCI memory mapped at 0x7ff4ff4f2000
> Interactive-mode selected
> Configuring Port 0 (socket 0)
> <...other 7 ports ...>
> Checking link statuses...
> Port 0 Link Up - speed 10000 Mbps - full-duplex
> Port 1 Link Down
> Port 2 Link Up - speed 10000 Mbps - full-duplex
> Port 3 Link Down
> Port 4 Link Up - speed 10000 Mbps - full-duplex
> Port 5 Link Down
> Port 6 Link Up - speed 10000 Mbps - full-duplex
> Port 7 Link Down
> Done
> testpmd>
> 
> This issue is introduced by the VFIO patch set addition, specifically
> commit ff0b67d1.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---


Self-NAK. Anatoly's patch is a better fix.