I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py but am unable to do so. The reason behind this is that I am unable to write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions I tried a couple of things such as setting iommu=pt and intel_iommu=on and ensured vt-d is enabled. Along with this I have made sure that the vfio-pci module is correctly loaded. I have also tried chmod 666 /sys/bus/pci/drivers/vfio-pci/bind So that I have permissions to write in this file. The error I get when I use usertools/dpdk-devbind.py to bind is this: Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci The details of 0000:b7:00.1 are as follows: Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e I have also unbinded The pci bridge to which 0000:b7:00.1 was connected. What more can be done to resolve this? Regards, Sarosh Arif
Hi. > -----Original Message----- > From: dev <dev-bounces@dpdk.org> On Behalf Of Sarosh Arif > Sent: Thursday, 17 September, 2020 08:53 > To: dev@dpdk.org > Subject: [dpdk-dev] unable to bind to vfio-pci > > I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py but > am unable to do so. The reason behind this is that I am unable to write in > /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions I tried a couple > of things such as setting iommu=pt and intel_iommu=on and ensured vt-d is > enabled. [Adam] First thing: to check if VT-d is enabled in Linux, you can read MSR 0x3a register. Its lowest bit set will show VT-d works: (optionally on Ubuntu: apt-get install msr-tools) modprobe msr rdmsr 0x3a 3 or 5 -> OK Other possible way to check for VT-d (IOMMU) working is to view the booting log (dmesg): dmesg | grep DMAR "DMAR: IOMMU enabled" -> OK Both of these should be met in order to use vfio-pci in the next step. > Along with this I have made sure that the vfio-pci module is correctly loaded. > I have also tried > > chmod 666 /sys/bus/pci/drivers/vfio-pci/bind > > So that I have permissions to write in this file. > > The error I get when I use usertools/dpdk-devbind.py to bind is this: > Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci > > The details of 0000:b7:00.1 are as follows: > Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e > > I have also unbinded The pci bridge to which 0000:b7:00.1 was connected. > > What more can be done to resolve this? > > Regards, > Sarosh Arif Adam Dybkowski
Hello, On Thu, Sep 17, 2020 at 12:32 PM Dybkowski, AdamX <adamx.dybkowski@intel.com> wrote: > > Hi. > > > -----Original Message----- > > From: dev <dev-bounces@dpdk.org> On Behalf Of Sarosh Arif > > Sent: Thursday, 17 September, 2020 08:53 > > To: dev@dpdk.org > > Subject: [dpdk-dev] unable to bind to vfio-pci > > > > I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py but > > am unable to do so. The reason behind this is that I am unable to write in > > /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions I tried a couple > > of things such as setting iommu=pt and intel_iommu=on and ensured vt-d is > > enabled. > > [Adam] First thing: to check if VT-d is enabled in Linux, you can read MSR 0x3a register. Its lowest bit set will show VT-d works: > > (optionally on Ubuntu: apt-get install msr-tools) > modprobe msr > rdmsr 0x3a > 3 or 5 -> OK > > Other possible way to check for VT-d (IOMMU) working is to view the booting log (dmesg): > > dmesg | grep DMAR > "DMAR: IOMMU enabled" -> OK > I used this method to check and vt-d is enabled. Is there anything else that I can do? > Both of these should be met in order to use vfio-pci in the next step. > > > Along with this I have made sure that the vfio-pci module is correctly loaded. > > I have also tried > > > > chmod 666 /sys/bus/pci/drivers/vfio-pci/bind > > > > So that I have permissions to write in this file. > > > > The error I get when I use usertools/dpdk-devbind.py to bind is this: > > Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci > > > > The details of 0000:b7:00.1 are as follows: > > Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e > > > > I have also unbinded The pci bridge to which 0000:b7:00.1 was connected. > > > > What more can be done to resolve this? > > > > Regards, > > Sarosh Arif > > Adam Dybkowski >
On Thu, Sep 17, 2020 at 11:52:58AM +0500, Sarosh Arif wrote:
> I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py
> but am unable to do so. The reason behind this is that I am unable to
> write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions
> I tried a couple of things such as setting iommu=pt and intel_iommu=on
> and ensured vt-d is enabled.
> Along with this I have made sure that the vfio-pci module is correctly
> loaded. I have also tried
>
> chmod 666 /sys/bus/pci/drivers/vfio-pci/bind
>
> So that I have permissions to write in this file.
>
> The error I get when I use usertools/dpdk-devbind.py to bind is this:
> Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci
>
> The details of 0000:b7:00.1 are as follows:
> Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e
>
> I have also unbinded The pci bridge to which 0000:b7:00.1 was connected.
>
> What more can be done to resolve this?
>
Since you describe changing permissions on the "bind" file, are you trying
to run dpdk-devbind.py as a non-root user? Does it work as root?
On Thu, Sep 17, 2020 at 2:17 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Thu, Sep 17, 2020 at 11:52:58AM +0500, Sarosh Arif wrote:
> > I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py
> > but am unable to do so. The reason behind this is that I am unable to
> > write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions
> > I tried a couple of things such as setting iommu=pt and intel_iommu=on
> > and ensured vt-d is enabled.
> > Along with this I have made sure that the vfio-pci module is correctly
> > loaded. I have also tried
> >
> > chmod 666 /sys/bus/pci/drivers/vfio-pci/bind
> >
> > So that I have permissions to write in this file.
> >
> > The error I get when I use usertools/dpdk-devbind.py to bind is this:
> > Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci
> >
> > The details of 0000:b7:00.1 are as follows:
> > Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e
> >
> > I have also unbinded The pci bridge to which 0000:b7:00.1 was connected.
> >
> > What more can be done to resolve this?
> >
> Since you describe changing permissions on the "bind" file, are you trying
> to run dpdk-devbind.py as a non-root user? Does it work as root?
I am running it as a root user. It does not work as a root user.
On 17-Sep-20 10:21 AM, Sarosh Arif wrote:
> On Thu, Sep 17, 2020 at 2:17 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
>>
>> On Thu, Sep 17, 2020 at 11:52:58AM +0500, Sarosh Arif wrote:
>>> I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py
>>> but am unable to do so. The reason behind this is that I am unable to
>>> write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions
>>> I tried a couple of things such as setting iommu=pt and intel_iommu=on
>>> and ensured vt-d is enabled.
>>> Along with this I have made sure that the vfio-pci module is correctly
>>> loaded. I have also tried
>>>
>>> chmod 666 /sys/bus/pci/drivers/vfio-pci/bind
>>>
>>> So that I have permissions to write in this file.
>>>
>>> The error I get when I use usertools/dpdk-devbind.py to bind is this:
>>> Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci
>>>
>>> The details of 0000:b7:00.1 are as follows:
>>> Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e
>>>
>>> I have also unbinded The pci bridge to which 0000:b7:00.1 was connected.
>>>
>>> What more can be done to resolve this?
>>>
>> Since you describe changing permissions on the "bind" file, are you trying
>> to run dpdk-devbind.py as a non-root user? Does it work as root?
> I am running it as a root user. It does not work as a root user.
>
Does "dmesg | tail" say anything of interest?
--
Thanks,
Anatoly
On Thu, Sep 17, 2020 at 02:21:27PM +0500, Sarosh Arif wrote:
> On Thu, Sep 17, 2020 at 2:17 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Thu, Sep 17, 2020 at 11:52:58AM +0500, Sarosh Arif wrote:
> > > I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py
> > > but am unable to do so. The reason behind this is that I am unable to
> > > write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions
> > > I tried a couple of things such as setting iommu=pt and intel_iommu=on
> > > and ensured vt-d is enabled.
> > > Along with this I have made sure that the vfio-pci module is correctly
> > > loaded. I have also tried
> > >
> > > chmod 666 /sys/bus/pci/drivers/vfio-pci/bind
> > >
> > > So that I have permissions to write in this file.
> > >
> > > The error I get when I use usertools/dpdk-devbind.py to bind is this:
> > > Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci
> > >
> > > The details of 0000:b7:00.1 are as follows:
> > > Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e
> > >
> > > I have also unbinded The pci bridge to which 0000:b7:00.1 was connected.
> > >
> > > What more can be done to resolve this?
> > >
> > Since you describe changing permissions on the "bind" file, are you trying
> > to run dpdk-devbind.py as a non-root user? Does it work as root?
> I am running it as a root user. It does not work as a root user.
One possible problem that it could be, is that you will need to ensure that
any other ports on the same device are either similarly bound to vfio-pci
or not bound to any driver. You can't have e.g. a 2-port X722 NIC where one
port is bound to the kernel driver, while another is bound to vfio in
userspace.
On 17-Sep-20 11:22 AM, Bruce Richardson wrote:
> On Thu, Sep 17, 2020 at 02:21:27PM +0500, Sarosh Arif wrote:
>> On Thu, Sep 17, 2020 at 2:17 PM Bruce Richardson
>> <bruce.richardson@intel.com> wrote:
>>>
>>> On Thu, Sep 17, 2020 at 11:52:58AM +0500, Sarosh Arif wrote:
>>>> I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py
>>>> but am unable to do so. The reason behind this is that I am unable to
>>>> write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions
>>>> I tried a couple of things such as setting iommu=pt and intel_iommu=on
>>>> and ensured vt-d is enabled.
>>>> Along with this I have made sure that the vfio-pci module is correctly
>>>> loaded. I have also tried
>>>>
>>>> chmod 666 /sys/bus/pci/drivers/vfio-pci/bind
>>>>
>>>> So that I have permissions to write in this file.
>>>>
>>>> The error I get when I use usertools/dpdk-devbind.py to bind is this:
>>>> Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci
>>>>
>>>> The details of 0000:b7:00.1 are as follows:
>>>> Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e
>>>>
>>>> I have also unbinded The pci bridge to which 0000:b7:00.1 was connected.
>>>>
>>>> What more can be done to resolve this?
>>>>
>>> Since you describe changing permissions on the "bind" file, are you trying
>>> to run dpdk-devbind.py as a non-root user? Does it work as root?
>> I am running it as a root user. It does not work as a root user.
>
> One possible problem that it could be, is that you will need to ensure that
> any other ports on the same device are either similarly bound to vfio-pci
> or not bound to any driver. You can't have e.g. a 2-port X722 NIC where one
> port is bound to the kernel driver, while another is bound to vfio in
> userspace.
>
I don't think this would result in a bind error; rather, you'd see the
IOMMU group as non-viable when attempting to use it with VFIO, but the
binding itself should succeed.
--
Thanks,
Anatoly
On Thu, Sep 17, 2020 at 4:50 PM Burakov, Anatoly <anatoly.burakov@intel.com> wrote: > > On 17-Sep-20 11:22 AM, Bruce Richardson wrote: > > On Thu, Sep 17, 2020 at 02:21:27PM +0500, Sarosh Arif wrote: > >> On Thu, Sep 17, 2020 at 2:17 PM Bruce Richardson > >> <bruce.richardson@intel.com> wrote: > >>> > >>> On Thu, Sep 17, 2020 at 11:52:58AM +0500, Sarosh Arif wrote: > >>>> I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py > >>>> but am unable to do so. The reason behind this is that I am unable to > >>>> write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions > >>>> I tried a couple of things such as setting iommu=pt and intel_iommu=on > >>>> and ensured vt-d is enabled. > >>>> Along with this I have made sure that the vfio-pci module is correctly > >>>> loaded. I have also tried > >>>> > >>>> chmod 666 /sys/bus/pci/drivers/vfio-pci/bind > >>>> > >>>> So that I have permissions to write in this file. > >>>> > >>>> The error I get when I use usertools/dpdk-devbind.py to bind is this: > >>>> Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci > >>>> > >>>> The details of 0000:b7:00.1 are as follows: > >>>> Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e > >>>> > >>>> I have also unbinded The pci bridge to which 0000:b7:00.1 was connected. > >>>> > >>>> What more can be done to resolve this? > >>>> > >>> Since you describe changing permissions on the "bind" file, are you trying > >>> to run dpdk-devbind.py as a non-root user? Does it work as root? > >> I am running it as a root user. It does not work as a root user. > > > > One possible problem that it could be, is that you will need to ensure that > > any other ports on the same device are either similarly bound to vfio-pci > > or not bound to any driver. You can't have e.g. a 2-port X722 NIC where one > > port is bound to the kernel driver, while another is bound to vfio in > > userspace. > > I unbinded all other ports from the kernel driver yet I could not bind to vfio. The problem seems to be the inability to write in /sys/bus/pci/drivers/vfio-pci/bind and /sys/bus/pci/drivers/vfio-pci/new_id. When I manually try to echo in these files I get "Permission denied". > > I don't think this would result in a bind error; rather, you'd see the > IOMMU group as non-viable when attempting to use it with VFIO, but the > binding itself should succeed. > > -- > Thanks, > Anatoly
On 21-Sep-20 12:10 PM, Sarosh Arif wrote: > On Thu, Sep 17, 2020 at 4:50 PM Burakov, Anatoly > <anatoly.burakov@intel.com> wrote: >> >> On 17-Sep-20 11:22 AM, Bruce Richardson wrote: >>> On Thu, Sep 17, 2020 at 02:21:27PM +0500, Sarosh Arif wrote: >>>> On Thu, Sep 17, 2020 at 2:17 PM Bruce Richardson >>>> <bruce.richardson@intel.com> wrote: >>>>> >>>>> On Thu, Sep 17, 2020 at 11:52:58AM +0500, Sarosh Arif wrote: >>>>>> I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py >>>>>> but am unable to do so. The reason behind this is that I am unable to >>>>>> write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions >>>>>> I tried a couple of things such as setting iommu=pt and intel_iommu=on >>>>>> and ensured vt-d is enabled. >>>>>> Along with this I have made sure that the vfio-pci module is correctly >>>>>> loaded. I have also tried >>>>>> >>>>>> chmod 666 /sys/bus/pci/drivers/vfio-pci/bind >>>>>> >>>>>> So that I have permissions to write in this file. >>>>>> >>>>>> The error I get when I use usertools/dpdk-devbind.py to bind is this: >>>>>> Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci >>>>>> >>>>>> The details of 0000:b7:00.1 are as follows: >>>>>> Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e >>>>>> >>>>>> I have also unbinded The pci bridge to which 0000:b7:00.1 was connected. >>>>>> >>>>>> What more can be done to resolve this? >>>>>> >>>>> Since you describe changing permissions on the "bind" file, are you trying >>>>> to run dpdk-devbind.py as a non-root user? Does it work as root? >>>> I am running it as a root user. It does not work as a root user. >>> >>> One possible problem that it could be, is that you will need to ensure that >>> any other ports on the same device are either similarly bound to vfio-pci >>> or not bound to any driver. You can't have e.g. a 2-port X722 NIC where one >>> port is bound to the kernel driver, while another is bound to vfio in >>> userspace. >>> > I unbinded all other ports from the kernel driver yet I could not bind > to vfio. The problem seems to be the inability to write in > /sys/bus/pci/drivers/vfio-pci/bind and > /sys/bus/pci/drivers/vfio-pci/new_id. When I manually try to echo in > these files I get "Permission denied". Does dmesg | tail indicate anything? >> >> I don't think this would result in a bind error; rather, you'd see the >> IOMMU group as non-viable when attempting to use it with VFIO, but the >> binding itself should succeed. >> >> -- >> Thanks, >> Anatoly -- Thanks, Anatoly
On Thu, Sep 17, 2020 at 2:35 PM Burakov, Anatoly <anatoly.burakov@intel.com> wrote: > > On 17-Sep-20 10:21 AM, Sarosh Arif wrote: > > On Thu, Sep 17, 2020 at 2:17 PM Bruce Richardson > > <bruce.richardson@intel.com> wrote: > >> > >> On Thu, Sep 17, 2020 at 11:52:58AM +0500, Sarosh Arif wrote: > >>> I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py > >>> but am unable to do so. The reason behind this is that I am unable to > >>> write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions > >>> I tried a couple of things such as setting iommu=pt and intel_iommu=on > >>> and ensured vt-d is enabled. > >>> Along with this I have made sure that the vfio-pci module is correctly > >>> loaded. I have also tried > >>> > >>> chmod 666 /sys/bus/pci/drivers/vfio-pci/bind > >>> > >>> So that I have permissions to write in this file. > >>> > >>> The error I get when I use usertools/dpdk-devbind.py to bind is this: > >>> Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci > >>> > >>> The details of 0000:b7:00.1 are as follows: > >>> Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e > >>> > >>> I have also unbinded The pci bridge to which 0000:b7:00.1 was connected. > >>> > >>> What more can be done to resolve this? > >>> > >> Since you describe changing permissions on the "bind" file, are you trying > >> to run dpdk-devbind.py as a non-root user? Does it work as root? > > I am running it as a root user. It does not work as a root user. > > > > Does "dmesg | tail" say anything of interest? This is the output of dmesg | tail: [136286.136271] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Down [221230.023654] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX [221230.024134] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Down [249273.956525] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX [249273.957003] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Down [314864.386303] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem [314867.734973] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) [332584.888223] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX [332584.888700] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Down [358429.954026] VFIO - User Level meta-driver version: 0.3 > > -- > Thanks, > Anatoly
On 21-Sep-20 12:54 PM, Sarosh Arif wrote: > On Thu, Sep 17, 2020 at 2:35 PM Burakov, Anatoly > <anatoly.burakov@intel.com> wrote: >> >> On 17-Sep-20 10:21 AM, Sarosh Arif wrote: >>> On Thu, Sep 17, 2020 at 2:17 PM Bruce Richardson >>> <bruce.richardson@intel.com> wrote: >>>> >>>> On Thu, Sep 17, 2020 at 11:52:58AM +0500, Sarosh Arif wrote: >>>>> I have been trying to bind to vfio-pci using usertools/dpdk-devbind.py >>>>> but am unable to do so. The reason behind this is that I am unable to >>>>> write in /sys/bus/pci/drivers/vfio-pci/bind. Upon searching solutions >>>>> I tried a couple of things such as setting iommu=pt and intel_iommu=on >>>>> and ensured vt-d is enabled. >>>>> Along with this I have made sure that the vfio-pci module is correctly >>>>> loaded. I have also tried >>>>> >>>>> chmod 666 /sys/bus/pci/drivers/vfio-pci/bind >>>>> >>>>> So that I have permissions to write in this file. >>>>> >>>>> The error I get when I use usertools/dpdk-devbind.py to bind is this: >>>>> Error: bind failed for 0000:b7:00.1 - Cannot bind to driver vfio-pci >>>>> >>>>> The details of 0000:b7:00.1 are as follows: >>>>> Ethernet Connection X722 for 10GBASE-T 37d2' if=eno6 drv=i40e >>>>> >>>>> I have also unbinded The pci bridge to which 0000:b7:00.1 was connected. >>>>> >>>>> What more can be done to resolve this? >>>>> >>>> Since you describe changing permissions on the "bind" file, are you trying >>>> to run dpdk-devbind.py as a non-root user? Does it work as root? >>> I am running it as a root user. It does not work as a root user. >>> >> >> Does "dmesg | tail" say anything of interest? > This is the output of dmesg | tail: > [136286.136271] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Down > [221230.023654] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Up 10 Gbps, > Flow Control: RX/TX > [221230.024134] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Down > [249273.956525] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Up 10 Gbps, > Flow Control: RX/TX > [249273.957003] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Down > [314864.386303] EXT4-fs (sda1): mounting ext3 file system using the > ext4 subsystem > [314867.734973] EXT4-fs (sda1): mounted filesystem with ordered data > mode. Opts: (null) > [332584.888223] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Up 10 Gbps, > Flow Control: RX/TX > [332584.888700] ixgbe 0000:65:00.0 enp101s0f0: NIC Link is Down > [358429.954026] VFIO - User Level meta-driver version: 0.3 So, nothing useful :) To me, it starts to sound more and more like some kind of security measure. Either it's something like AppArmor/SELinux preventing you from binding the drivers, or maybe it's something like Secure Boot, or some other security-related feature. A "permission denied" error is usually indicative of such things. I'm not really an expert on this so i can't tell you off the top of my head what to check, but my first stop would have been either Secure Boot-related settings, or SELinux/AppArmor logs. >> >> -- >> Thanks, >> Anatoly -- Thanks, Anatoly