[dpdk-dev] VFIO in setup.sh

[dpdk-dev] VFIO in setup.sh

[Stephen Hemminger]

This code around vfio in setup script looks incorrect, is anyone using it:

1. Why set the execute bit, when you want read-write?
   Looks like a bug (or worse a security hole).
	# make sure regular users can read /dev/vfio
	echo "chmod /dev/vfio"
	sudo chmod a+x /dev/vfio

3. Why depend on location of vfio module in kernel tree?
   modprobe does the right thing and finds it.

	VFIO_PATH="kernel/drivers/vfio/pci/vfio-pci.ko"

	echo "Loading VFIO module"
	/sbin/lsmod | grep -s vfio_pci > /dev/null
	if [ $? -ne 0 ] ; then
		if [ -f /lib/modules/$(uname -r)/$VFIO_PATH ] ; then
			sudo /sbin/modprobe vfio-pci
		fi
	fi

Re: [dpdk-dev] VFIO in setup.sh

[Stephen Hemminger]

On Mon, 30 Mar 2015 13:35:07 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:

> This code around vfio in setup script looks incorrect, is anyone using it:
> 
> 1. Why set the execute bit, when you want read-write?
>    Looks like a bug (or worse a security hole).
> 	# make sure regular users can read /dev/vfio
> 	echo "chmod /dev/vfio"
> 	sudo chmod a+x /dev/vfio
> 
> 3. Why depend on location of vfio module in kernel tree?
>    modprobe does the right thing and finds it.
> 
> 	VFIO_PATH="kernel/drivers/vfio/pci/vfio-pci.ko"
> 
> 	echo "Loading VFIO module"
> 	/sbin/lsmod | grep -s vfio_pci > /dev/null
> 	if [ $? -ne 0 ] ; then
> 		if [ -f /lib/modules/$(uname -r)/$VFIO_PATH ] ; then
> 			sudo /sbin/modprobe vfio-pci
> 		fi
> 	fi
> 
> 

Never mind, it is fine. /dev/vfio is a directory and loading
the module creates the directory.

Re: [dpdk-dev] VFIO in setup.sh

[Burakov, Anatoly]

> > 3. Why depend on location of vfio module in kernel tree?
> >    modprobe does the right thing and finds it.
> >
> > 	VFIO_PATH="kernel/drivers/vfio/pci/vfio-pci.ko"
> >
> > 	echo "Loading VFIO module"
> > 	/sbin/lsmod | grep -s vfio_pci > /dev/null
> > 	if [ $? -ne 0 ] ; then
> > 		if [ -f /lib/modules/$(uname -r)/$VFIO_PATH ] ; then
> > 			sudo /sbin/modprobe vfio-pci
> > 		fi
> > 	fi
> >

Here I agree. Needs to be fixed.

Thanks,
Anatoly

Re: [dpdk-dev] VFIO in setup.sh

[Andre Richter]

I think the whole process of VFIO binding maybe needs at least a second
thought regarding corner cases and security.

1) in the setup process, there currently is no mechanism that checks if the
Device to be used has other devices in the same iommu group that need to be
bound to VFIO too. Otherwise using VFIO will fail.
I think currently, it only works if the network device is the only one in
its iommu group.

2) Right now everything inside /dev/vfio/ is granted to the all users,
right? Maybe this leads to (security) issues if VFIO is in active use by
other non-dpdk processes for other PCIe devices.

Cheers,
Andre
Burakov, Anatoly <anatoly.burakov@intel.com> schrieb am Di., 31. März 2015
um 11:05:

> > > 3. Why depend on location of vfio module in kernel tree?
> > >    modprobe does the right thing and finds it.
> > >
> > >     VFIO_PATH="kernel/drivers/vfio/pci/vfio-pci.ko"
> > >
> > >     echo "Loading VFIO module"
> > >     /sbin/lsmod | grep -s vfio_pci > /dev/null
> > >     if [ $? -ne 0 ] ; then
> > >             if [ -f /lib/modules/$(uname -r)/$VFIO_PATH ] ; then
> > >                     sudo /sbin/modprobe vfio-pci
> > >             fi
> > >     fi
> > >
>
> Here I agree. Needs to be fixed.
>
> Thanks,
> Anatoly
>

Re: [dpdk-dev] VFIO in setup.sh

[Burakov, Anatoly]

> I think the whole process of VFIO binding maybe needs at least a second thought regarding corner cases and security.
> 
> 1) in the setup process, there currently is no mechanism that checks if the Device to be used has other devices in the
> same iommu group that need to be bound to VFIO too. Otherwise using VFIO will fail.
> I think currently, it only works if the network device is the only one in its iommu group.
> 
> 2) Right now everything inside /dev/vfio/ is granted to the all users, right? Maybe this leads to (security) issues if VFIO
> is in active use by other non-dpdk processes for other PCIe devices.

I believe that's how VFIO is meant to be used. At least according to VFIO documentation, located here: https://www.kernel.org/doc/Documentation/vfio.txt

Regarding 1), this can only be done by unbinding the device from the host driver and binding it to vfio-pci, which can't be done by the user. If the device is not bound to vfio-pci, we have no way of knowing if it belongs to this or that IOMMU group. Regarding 2), as noted above, the administrator should set up VFIO devices. While I agree that the way setup.sh sets up VFIO security permissions is not ideal, it's really there to cover the most common use case. An administrator can always set up VFIO on his own, granting permissions as needed. 

Thanks,
Anatoly

Re: [dpdk-dev] VFIO in setup.sh

[Andre Richter]

2015-03-31 11:35 GMT+02:00 Burakov, Anatoly <anatoly.burakov@intel.com>:
>
> > I think the whole process of VFIO binding maybe needs at least a second thought regarding corner cases and security.
> >
> > 1) in the setup process, there currently is no mechanism that checks if the Device to be used has other devices in the
> > same iommu group that need to be bound to VFIO too. Otherwise using VFIO will fail.
> > I think currently, it only works if the network device is the only one in its iommu group.
> >
> > 2) Right now everything inside /dev/vfio/ is granted to the all users, right? Maybe this leads to (security) issues if VFIO
> > is in active use by other non-dpdk processes for other PCIe devices.
>
> I believe that's how VFIO is meant to be used. At least according to VFIO documentation, located here: https://www.kernel.org/doc/Documentation/vfio.txt
>
> Regarding 1), this can only be done by unbinding the device from the host driver and binding it to vfio-pci, which can't be done by the user. If the device is not bound to vfio-pci, we have no way of knowing if it belongs to this or that IOMMU group.

iommu groups already exist before vfio-pci is loaded.
The whole setup process as described in the VFIO documentation, where
a PCIe device shares an iommu group with other devices, can therefore
be automated. Some time ago I wrote a ruby script that does exactly
that (https://github.com/andre-richter/rVFIO/blob/master/example/pci-bind.rb).
Porting it to bash should be no problem.

However, the question is if dpdk needs to cover this (corner?) case.
I would assume that the bulk of dpdk use-cases deals with scenarios
where the targeted NIC is the only one in its iommu group, because it
is a high-speed NIC that is connected to a CPU-integrated PCIe-Port.
A NIC sharing an iommu group with other devices would be most likely
if it is behind a bridge, aka a chipset-integrated NIC or a NIC that
resides in a PCIe-Slot that connects to the chipset. This is more
common in desktop machines than in server grade machines.

Cheers,
Andre

>
> Regarding 2), as noted above, the administrator should set up VFIO devices. While I agree that the way setup.sh sets up VFIO security permissions is not ideal, it's really there to cover the most common use case. An administrator can always set up VFIO on his own, granting permissions as needed.
>
> Thanks,
> Anatoly

Re: [dpdk-dev] VFIO in setup.sh

[Burakov, Anatoly]

> iommu groups already exist before vfio-pci is loaded.
> The whole setup process as described in the VFIO documentation, where a
> PCIe device shares an iommu group with other devices, can therefore be
> automated. Some time ago I wrote a ruby script that does exactly that
> (https://github.com/andre-richter/rVFIO/blob/master/example/pci-
> bind.rb).
> Porting it to bash should be no problem.

Makes sense to me. Maybe we could output a warning if not all devices in a group were bound to vfio-pci (and possibly list all devices that aren't bound)? IMO, the right place for such a warning would probably be dpdk_nic_bind.py though, not setup.sh.

> However, the question is if dpdk needs to cover this (corner?) case.
> I would assume that the bulk of dpdk use-cases deals with scenarios where
> the targeted NIC is the only one in its iommu group, because it is a high-
> speed NIC that is connected to a CPU-integrated PCIe-Port.
> A NIC sharing an iommu group with other devices would be most likely if it is
> behind a bridge, aka a chipset-integrated NIC or a NIC that resides in a PCIe-
> Slot that connects to the chipset. This is more common in desktop machines
> than in server grade machines.

If no other devices need to be unbound, no warnings, and no harm done :)

Thanks,
Anatoly