unable to bind to vfio-pci

unable to bind to vfio-pci

[Nagendra Prabhu A T]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

Hi All,

I am trying to bind vfio-pci using dpdk-devbind.py, But i am seeing teh
following error :

root@nat-kvm:/usr/src/dpdk-22.11/usertools# dpdk-devbind.py --bind=vfio-pci
0000:51:00.0 Error: bind failed for 0000:51:00.0 - Cannot bind to driver
vfio-pci: [Errno 22] Invalid argument
root@nat-kvm:/usr/src/dpdk-22.11/usertools#


I see that Sarosh has faced the same issue earlier. I saw the mail thread
and followed all the steps mentioned there. But it dint help.

Can you please help.

Thanks
Nagendra

[-- Attachment #2: Type: text/html, Size: 1447 bytes --]

Re: unable to bind to vfio-pci

[Bruce Richardson]

On Tue, Feb 21, 2023 at 08:05:20PM +0530, Nagendra Prabhu A T wrote:
>    Hi All,
>    I am trying to bind vfio-pci using dpdk-devbind.py, But i am seeing teh
>    following error :
>    root@nat-kvm:/usr/src/dpdk-22.11/usertools# dpdk-devbind.py
>    --bind=vfio-pci 0000:51:00.0 Error: bind failed for 0000:51:00.0 -
>    Cannot bind to driver vfio-pci: [Errno 22] Invalid argument
>    root@nat-kvm:/usr/src/dpdk-22.11/usertools#
>    I see that Sarosh has faced the same issue earlier. I saw the mail
>    thread and followed all the steps mentioned there. But it dint help.
>    Can you please help.

An error like that I would attribute to one of two possible likely causes.
[There may be other causes, but these two come to mind immediately].

1. No IOMMU available for use. In this case you need to enable no-iommu
   mode for VFIO and see if that works.
2. You have other ports on the same card bound to a kernel driver. You
   cannot use one port bound to the kernel (and therefore using IOMMU
   mappings for the kernel), while another port on the same card is bound
   to userspace vfio (and therefore looking for userspace IOMMU mappings).
   If this is the case, you need to unbind the other port from the kernel,
   and leave it unbound or else also bound to vfio. It's possible that
   no-iommu mode may help here too, but I have never tested that.

Regards,
/Bruce

Re: unable to bind to vfio-pci

[Nagendra Prabhu A T]

[-- Attachment #1: Type: text/plain, Size: 2274 bytes --]

HI Bruce,

Thanks for your reply.

I have removed the binding of all other ports. So , i can confirm that #2
is taken care of.

I tried using a different driver "uio_pci_generic". I still see the same
issue for that driver as well.

The problem seems to be because the write operation is not happening in
"bind" file.

Infact tried doing the following step manually. That also failed

echo sh -c 0000:51:00.0 > /sys/bus/pci/drivers/uio_pci_generic/bind bash:
echo: write error: No such device

I have required permission for the file.
I tried doing with "sudo"
I dont have selinux running

So not sure whats blocking me. Infact , even "cat
/sys/bus/pci/drivers/uio_pci_generic/bind"
is failing

Can you please help.

Thanks
Nagendra

Thanks
Nagendra

On Tue, Feb 21, 2023 at 8:15 PM Bruce Richardson <bruce.richardson@intel.com>
wrote:

> On Tue, Feb 21, 2023 at 08:05:20PM +0530, Nagendra Prabhu A T wrote:
> >    Hi All,
> >    I am trying to bind vfio-pci using dpdk-devbind.py, But i am seeing
> teh
> >    following error :
> >    root@nat-kvm:/usr/src/dpdk-22.11/usertools# dpdk-devbind.py
> >    --bind=vfio-pci 0000:51:00.0 Error: bind failed for 0000:51:00.0 -
> >    Cannot bind to driver vfio-pci: [Errno 22] Invalid argument
> >    root@nat-kvm:/usr/src/dpdk-22.11/usertools#
> >    I see that Sarosh has faced the same issue earlier. I saw the mail
> >    thread and followed all the steps mentioned there. But it dint help.
> >    Can you please help.
>
> An error like that I would attribute to one of two possible likely causes.
> [There may be other causes, but these two come to mind immediately].
>
> 1. No IOMMU available for use. In this case you need to enable no-iommu
>    mode for VFIO and see if that works.
> 2. You have other ports on the same card bound to a kernel driver. You
>    cannot use one port bound to the kernel (and therefore using IOMMU
>    mappings for the kernel), while another port on the same card is bound
>    to userspace vfio (and therefore looking for userspace IOMMU mappings).
>    If this is the case, you need to unbind the other port from the kernel,
>    and leave it unbound or else also bound to vfio. It's possible that
>    no-iommu mode may help here too, but I have never tested that.
>
> Regards,
> /Bruce
>

[-- Attachment #2: Type: text/html, Size: 7520 bytes --]

Re: unable to bind to vfio-pci

[Bruce Richardson]

On Tue, Feb 21, 2023 at 09:30:15PM +0530, Nagendra Prabhu A T wrote:
>    HI Bruce,
>    Thanks for your reply.
>    I have removed the binding of all other ports. So , i can confirm that
>    #2 is taken care of.
>    I tried using a different driver "uio_pci_generic". I still see the
>    same issue for that driver as well.
>    The problem seems to be because the write operation is not happening in
>    "bind" file.
>    Infact tried doing the following step manually. That also failed
>    echo sh -c 0000:51:00.0 > /sys/bus/pci/drivers/uio_pci_generic/bind
>    bash: echo: write error: No such device

The "sh -c" doesn't look right in that command. It should just echo the
device id to the bind file. However, it's generally recommended just to use
the devbind script, as more writes to sysfs may be necessary first, e.g. to
register the device ids first.

>    I have required permission for the file.
>    I tried doing with "sudo"
>    I dont have selinux running
>    So not sure whats blocking me. Infact , even
>    "cat /sys/bus/pci/drivers/uio_pci_generic/bind" is failing
>    Can you please help.

To track down issues other than the obvious, I think we need more
information. For example, what NIC card is this? Are there any useful error
messages in dmesg?

/Bruce

Re: unable to bind to vfio-pci

[Nagendra Prabhu A T]

[-- Attachment #1: Type: text/plain, Size: 3001 bytes --]

These are the messages i am seeing dmesg

[16588.298660] vfio-pci: probe of 0000:8a:00.2 failed with error -22
[16591.574630] i40e 0000:8a:00.3: i40e_ptp_stop: removed PHC on enp138s0f3
[16593.138613] vfio-pci: probe of 0000:8a:00.3 failed with error -22
[16593.138674] vfio-pci: probe of 0000:8a:00.3 failed with error -22
[16627.551866] vfio-pci: probe of 0000:51:00.0 failed with error -22
[16627.551922] vfio-pci: probe of 0000:51:00.0 failed with error -22
[16716.344530] vfio-pci: probe of 0000:51:00.0 failed with error -22
[16716.344585] vfio-pci: probe of 0000:51:00.0 failed with error -22
[17859.876566] vfio-pci: probe of 0000:51:00.0 failed with error -22
[17859.876622] vfio-pci: probe of 0000:51:00.0 failed with error -22
root@nat-kvm:/home/nat# cat /boot/config-$(uname -r) | grep NOIOMMU
CONFIG_VFIO_NOIOMMU=y
root@nat-kvm:/home/nat#



Interface type :

Other Network devices
=====================
0000:51:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
unused=i40e,vfio-pci
0000:51:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
unused=i40e,vfio-pci
0000:51:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
unused=i40e,vfio-pci
0000:51:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
unused=i40e,vfio-pci
0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
unused=i40e,vfio-pci
0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
unused=i40e,vfio-pci
0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
unused=i40e,vfio-pci
0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
unused=i40e,vfio-pci


On Tue, Feb 21, 2023 at 9:37 PM Bruce Richardson <bruce.richardson@intel.com>
wrote:

> On Tue, Feb 21, 2023 at 09:30:15PM +0530, Nagendra Prabhu A T wrote:
> >    HI Bruce,
> >    Thanks for your reply.
> >    I have removed the binding of all other ports. So , i can confirm that
> >    #2 is taken care of.
> >    I tried using a different driver "uio_pci_generic". I still see the
> >    same issue for that driver as well.
> >    The problem seems to be because the write operation is not happening
> in
> >    "bind" file.
> >    Infact tried doing the following step manually. That also failed
> >    echo sh -c 0000:51:00.0 > /sys/bus/pci/drivers/uio_pci_generic/bind
> >    bash: echo: write error: No such device
>
> The "sh -c" doesn't look right in that command. It should just echo the
> device id to the bind file. However, it's generally recommended just to use
> the devbind script, as more writes to sysfs may be necessary first, e.g. to
> register the device ids first.
>
> >    I have required permission for the file.
> >    I tried doing with "sudo"
> >    I dont have selinux running
> >    So not sure whats blocking me. Infact , even
> >    "cat /sys/bus/pci/drivers/uio_pci_generic/bind" is failing
> >    Can you please help.
>
> To track down issues other than the obvious, I think we need more
> information. For example, what NIC card is this? Are there any useful error
> messages in dmesg?
>
> /Bruce
>

[-- Attachment #2: Type: text/html, Size: 3975 bytes --]

Re: unable to bind to vfio-pci

[Bruce Richardson]

On Tue, Feb 21, 2023 at 09:46:30PM +0530, Nagendra Prabhu A T wrote:
>    These are the messages i am seeing dmesg
>    [16588.298660] vfio-pci: probe of 0000:8a:00.2 failed with error -22
>    [16591.574630] i40e 0000:8a:00.3: i40e_ptp_stop: removed PHC on
>    enp138s0f3
>    [16593.138613] vfio-pci: probe of 0000:8a:00.3 failed with error -22
>    [16593.138674] vfio-pci: probe of 0000:8a:00.3 failed with error -22
>    [16627.551866] vfio-pci: probe of 0000:51:00.0 failed with error -22
>    [16627.551922] vfio-pci: probe of 0000:51:00.0 failed with error -22
>    [16716.344530] vfio-pci: probe of 0000:51:00.0 failed with error -22
>    [16716.344585] vfio-pci: probe of 0000:51:00.0 failed with error -22
>    [17859.876566] vfio-pci: probe of 0000:51:00.0 failed with error -22
>    [17859.876622] vfio-pci: probe of 0000:51:00.0 failed with error -22
>    root@nat-kvm:/home/nat# cat /boot/config-$(uname -r) | grep NOIOMMU
>    CONFIG_VFIO_NOIOMMU=y
>    root@nat-kvm:/home/nat#
>    Interface type :
>    Other Network devices
>    =====================
>    0000:51:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
>    unused=i40e,vfio-pci
>    0000:51:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
>    unused=i40e,vfio-pci
>    0000:51:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
>    unused=i40e,vfio-pci
>    0000:51:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
>    unused=i40e,vfio-pci
>    0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
>    unused=i40e,vfio-pci
>    0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
>    unused=i40e,vfio-pci
>    0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
>    unused=i40e,vfio-pci
>    0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
>    unused=i40e,vfio-pci
> 

Please try the following:

	echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
	dpdk-devbind.py -b vfio-pci 8a:00.*

Does anything change then?

/Bruce

PS: For mailing list discussions, please don't top-post. Replies are easier
to follow if posted immediately below the relevant content.

>    On Tue, Feb 21, 2023 at 9:37 PM Bruce Richardson
>    <[1]bruce.richardson@intel.com> wrote:
> 
>      On Tue, Feb 21, 2023 at 09:30:15PM +0530, Nagendra Prabhu A T wrote:
>      >    HI Bruce,
>      >    Thanks for your reply.
>      >    I have removed the binding of all other ports. So , i can
>      confirm that
>      >    #2 is taken care of.
>      >    I tried using a different driver "uio_pci_generic". I still see
>      the
>      >    same issue for that driver as well.
>      >    The problem seems to be because the write operation is not
>      happening in
>      >    "bind" file.
>      >    Infact tried doing the following step manually. That also
>      failed
>      >    echo sh -c 0000:51:00.0 >
>      /sys/bus/pci/drivers/uio_pci_generic/bind
>      >    bash: echo: write error: No such device
>      The "sh -c" doesn't look right in that command. It should just echo
>      the
>      device id to the bind file. However, it's generally recommended just
>      to use
>      the devbind script, as more writes to sysfs may be necessary first,
>      e.g. to
>      register the device ids first.
>      >    I have required permission for the file.
>      >    I tried doing with "sudo"
>      >    I dont have selinux running
>      >    So not sure whats blocking me. Infact , even
>      >    "cat /sys/bus/pci/drivers/uio_pci_generic/bind" is failing
>      >    Can you please help.
>      To track down issues other than the obvious, I think we need more
>      information. For example, what NIC card is this? Are there any
>      useful error
>      messages in dmesg?
>      /Bruce
> 
> References
> 
>    1. mailto:bruce.richardson@intel.com

Re: unable to bind to vfio-pci

[Stephen Hemminger]

On Tue, 21 Feb 2023 16:27:31 +0000
Bruce Richardson <bruce.richardson@intel.com> wrote:

> On Tue, Feb 21, 2023 at 09:46:30PM +0530, Nagendra Prabhu A T wrote:
> >    These are the messages i am seeing dmesg
> >    [16588.298660] vfio-pci: probe of 0000:8a:00.2 failed with error -22
> >    [16591.574630] i40e 0000:8a:00.3: i40e_ptp_stop: removed PHC on
> >    enp138s0f3
> >    [16593.138613] vfio-pci: probe of 0000:8a:00.3 failed with error -22
> >    [16593.138674] vfio-pci: probe of 0000:8a:00.3 failed with error -22
> >    [16627.551866] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [16627.551922] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [16716.344530] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [16716.344585] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [17859.876566] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [17859.876622] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    root@nat-kvm:/home/nat# cat /boot/config-$(uname -r) | grep NOIOMMU
> >    CONFIG_VFIO_NOIOMMU=y
> >    root@nat-kvm:/home/nat#
> >    Interface type :
> >    Other Network devices
> >    =====================
> >    0000:51:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:51:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:51:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:51:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >   
> 
> Please try the following:
> 
> 	echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
> 	dpdk-devbind.py -b vfio-pci 8a:00.*
> 
> Does anything change then?
> 
> /Bruce
> 
> PS: For mailing list discussions, please don't top-post. Replies are easier
> to follow if posted immediately below the relevant content.
> 
> >    On Tue, Feb 21, 2023 at 9:37 PM Bruce Richardson
> >    <[1]bruce.richardson@intel.com> wrote:
> > 
> >      On Tue, Feb 21, 2023 at 09:30:15PM +0530, Nagendra Prabhu A T wrote:  
> >      >    HI Bruce,
> >      >    Thanks for your reply.
> >      >    I have removed the binding of all other ports. So , i can  
> >      confirm that  
> >      >    #2 is taken care of.
> >      >    I tried using a different driver "uio_pci_generic". I still see  
> >      the  
> >      >    same issue for that driver as well.
> >      >    The problem seems to be because the write operation is not  
> >      happening in  
> >      >    "bind" file.
> >      >    Infact tried doing the following step manually. That also  
> >      failed  
> >      >    echo sh -c 0000:51:00.0 >  
> >      /sys/bus/pci/drivers/uio_pci_generic/bind  
> >      >    bash: echo: write error: No such device  
> >      The "sh -c" doesn't look right in that command. It should just echo
> >      the
> >      device id to the bind file. However, it's generally recommended just
> >      to use
> >      the devbind script, as more writes to sysfs may be necessary first,
> >      e.g. to
> >      register the device ids first.  
> >      >    I have required permission for the file.
> >      >    I tried doing with "sudo"
> >      >    I dont have selinux running
> >      >    So not sure whats blocking me. Infact , even
> >      >    "cat /sys/bus/pci/drivers/uio_pci_generic/bind" is failing
> >      >    Can you please help.  
> >      To track down issues other than the obvious, I think we need more
> >      information. For example, what NIC card is this? Are there any
> >      useful error
> >      messages in dmesg?
> >      /Bruce
> > 
> > References
> > 
> >    1. mailto:bruce.richardson@intel.com  

Another possibility is that the hardware shares IOMMU channels.
One machine I was using had multiple PCI devices sharing same IOMMU channel and kernel
would not let it work unless all of them were unbound from kernel.

Re: unable to bind to vfio-pci

[Nagendra Prabhu A T]

[-- Attachment #1: Type: text/plain, Size: 4816 bytes --]

On Tue, Feb 21, 2023 at 9:57 PM Bruce Richardson <bruce.richardson@intel.com>
wrote:

> On Tue, Feb 21, 2023 at 09:46:30PM +0530, Nagendra Prabhu A T wrote:
> >    These are the messages i am seeing dmesg
> >    [16588.298660] vfio-pci: probe of 0000:8a:00.2 failed with error -22
> >    [16591.574630] i40e 0000:8a:00.3: i40e_ptp_stop: removed PHC on
> >    enp138s0f3
> >    [16593.138613] vfio-pci: probe of 0000:8a:00.3 failed with error -22
> >    [16593.138674] vfio-pci: probe of 0000:8a:00.3 failed with error -22
> >    [16627.551866] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [16627.551922] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [16716.344530] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [16716.344585] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [17859.876566] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    [17859.876622] vfio-pci: probe of 0000:51:00.0 failed with error -22
> >    root@nat-kvm:/home/nat# cat /boot/config-$(uname -r) | grep NOIOMMU
> >    CONFIG_VFIO_NOIOMMU=y
> >    root@nat-kvm:/home/nat#
> >    Interface type :
> >    Other Network devices
> >    =====================
> >    0000:51:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:51:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:51:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:51:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >    0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> >    unused=i40e,vfio-pci
> >
>
> Please try the following:
>
>         echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
>         dpdk-devbind.py -b vfio-pci 8a:00.*
>
> Does anything change then?
>
> /Bruce
>
> PS: For mailing list discussions, please don't top-post. Replies are easier
> to follow if posted immediately below the relevant content.
>

Thanks Bruce, the above solution worked.

root@nat-kvm:/usr/src/dpdk-22.11/usertools#  echo 1 >
/sys/module/vfio/parameters/enable_unsafe_noiommu_mode
root@nat-kvm:/usr/src/dpdk-22.11/usertools#  dpdk-devbind.py -b vfio-pci
8a:00.*
root@nat-kvm:/usr/src/dpdk-22.11/usertools# dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:8a:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci
unused=i40e
0000:8a:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci
unused=i40e
0000:8a:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci
unused=i40e
0000:8a:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci
unused=i40e


Thanks

>
> >    On Tue, Feb 21, 2023 at 9:37 PM Bruce Richardson
> >    <[1]bruce.richardson@intel.com> wrote:
> >
> >      On Tue, Feb 21, 2023 at 09:30:15PM +0530, Nagendra Prabhu A T wrote:
> >      >    HI Bruce,
> >      >    Thanks for your reply.
> >      >    I have removed the binding of all other ports. So , i can
> >      confirm that
> >      >    #2 is taken care of.
> >      >    I tried using a different driver "uio_pci_generic". I still see
> >      the
> >      >    same issue for that driver as well.
> >      >    The problem seems to be because the write operation is not
> >      happening in
> >      >    "bind" file.
> >      >    Infact tried doing the following step manually. That also
> >      failed
> >      >    echo sh -c 0000:51:00.0 >
> >      /sys/bus/pci/drivers/uio_pci_generic/bind
> >      >    bash: echo: write error: No such device
> >      The "sh -c" doesn't look right in that command. It should just echo
> >      the
> >      device id to the bind file. However, it's generally recommended just
> >      to use
> >      the devbind script, as more writes to sysfs may be necessary first,
> >      e.g. to
> >      register the device ids first.
> >      >    I have required permission for the file.
> >      >    I tried doing with "sudo"
> >      >    I dont have selinux running
> >      >    So not sure whats blocking me. Infact , even
> >      >    "cat /sys/bus/pci/drivers/uio_pci_generic/bind" is failing
> >      >    Can you please help.
> >      To track down issues other than the obvious, I think we need more
> >      information. For example, what NIC card is this? Are there any
> >      useful error
> >      messages in dmesg?
> >      /Bruce
> >
> > References
> >
> >    1. mailto:bruce.richardson@intel.com
>

[-- Attachment #2: Type: text/html, Size: 6550 bytes --]