How to resolve "Not all devices in IOMMU group bound to VFIO or unbound" ?

How to resolve "Not all devices in IOMMU group bound to VFIO or unbound" ?

[sysman]

Hello!

  I have built and deployed latest DPDK from the git, so:

dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:04.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
drv=vfio-pci unused=e1000
0000:02:05.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
drv=vfio-pci unused=e1000
0000:02:06.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
drv=vfio-pci unused=e1000
0000:02:07.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
drv=vfio-pci unused=e1000

Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=eth0 
drv=e1000 unused=vfio-pci *Active*



./ipv4_multicast
...
EAL: 0000:02:04.0 VFIO group is not viable! Not all devices in IOMMU 
group bound to VFIO or unbound
EAL: Requested device 0000:02:04.0 cannot be used
EAL: 0000:02:05.0 VFIO group is not viable! Not all devices in IOMMU 
group bound to VFIO or unbound
EAL: Requested device 0000:02:05.0 cannot be used
EAL: 0000:02:06.0 VFIO group is not viable! Not all devices in IOMMU 
group bound to VFIO or unbound
EAL: Requested device 0000:02:06.0 cannot be used
EAL: 0000:02:07.0 VFIO group is not viable! Not all devices in IOMMU 
group bound to VFIO or unbound
EAL: Requested device 0000:02:07.0 cannot be used
...


root@devuan4-sysman:~# readlink 
"/sys/bus/pci/devices/0000:02:04.0/iommu_group"
../../../../kernel/iommu_groups/5
root@devuan4-sysman:~# readlink 
"/sys/bus/pci/devices/0000:02:05.0/iommu_group"
../../../../kernel/iommu_groups/5
root@devuan4-sysman:~# readlink 
"/sys/bus/pci/devices/0000:02:06.0/iommu_group"
../../../../kernel/iommu_groups/5
root@devuan4-sysman:~# readlink 
"/sys/bus/pci/devices/0000:02:07.0/iommu_group"
../../../../kernel/iommu_groups/5



Can someone, please, help me resolve this problem?

Re: How to resolve "Not all devices in IOMMU group bound to VFIO or unbound" ?

[Stephen Hemminger]

On Wed, 10 Aug 2022 16:52:39 +0300
sysman@scaled.cloud wrote:

> Hello!
> 
>   I have built and deployed latest DPDK from the git, so:
> 
> dpdk-devbind.py -s
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:02:04.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
> drv=vfio-pci unused=e1000
> 0000:02:05.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
> drv=vfio-pci unused=e1000
> 0000:02:06.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
> drv=vfio-pci unused=e1000
> 0000:02:07.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
> drv=vfio-pci unused=e1000
> 
> Network devices using kernel driver
> ===================================
> 0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' if=eth0 
> drv=e1000 unused=vfio-pci *Active*
> 
> 
> 
> ./ipv4_multicast
> ...
> EAL: 0000:02:04.0 VFIO group is not viable! Not all devices in IOMMU 
> group bound to VFIO or unbound
> EAL: Requested device 0000:02:04.0 cannot be used
> EAL: 0000:02:05.0 VFIO group is not viable! Not all devices in IOMMU 
> group bound to VFIO or unbound
> EAL: Requested device 0000:02:05.0 cannot be used
> EAL: 0000:02:06.0 VFIO group is not viable! Not all devices in IOMMU 
> group bound to VFIO or unbound
> EAL: Requested device 0000:02:06.0 cannot be used
> EAL: 0000:02:07.0 VFIO group is not viable! Not all devices in IOMMU 
> group bound to VFIO or unbound
> EAL: Requested device 0000:02:07.0 cannot be used
> ...
> 
> 
> root@devuan4-sysman:~# readlink 
> "/sys/bus/pci/devices/0000:02:04.0/iommu_group"
> ../../../../kernel/iommu_groups/5
> root@devuan4-sysman:~# readlink 
> "/sys/bus/pci/devices/0000:02:05.0/iommu_group"
> ../../../../kernel/iommu_groups/5
> root@devuan4-sysman:~# readlink 
> "/sys/bus/pci/devices/0000:02:06.0/iommu_group"
> ../../../../kernel/iommu_groups/5
> root@devuan4-sysman:~# readlink 
> "/sys/bus/pci/devices/0000:02:07.0/iommu_group"
> ../../../../kernel/iommu_groups/5
> 
> 
> 
> Can someone, please, help me resolve this problem?

Can't really be fixed in software, the IOMMU groups come from the hardware.
The best workaround if possible is to use a different NIC for eth0.

Re: How to resolve "Not all devices in IOMMU group bound to VFIO or unbound" ?

[sysman]

Hello!

    "The best workaround if possible is to use a different NIC for eth0."

  Can I ask u, please, explain with more words what I'm need to do ?
  Thanks!



Stephen Hemminger писал 2022-08-10 16:55:
> On Wed, 10 Aug 2022 16:52:39 +0300
> sysman@scaled.cloud wrote:
> 
>> Hello!
>> 
>>   I have built and deployed latest DPDK from the git, so:
>> 
>> dpdk-devbind.py -s
>> 
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:02:04.0 '82545EM Gigabit Ethernet Controller (Copper) 100f'
>> drv=vfio-pci unused=e1000
>> 0000:02:05.0 '82545EM Gigabit Ethernet Controller (Copper) 100f'
>> drv=vfio-pci unused=e1000
>> 0000:02:06.0 '82545EM Gigabit Ethernet Controller (Copper) 100f'
>> drv=vfio-pci unused=e1000
>> 0000:02:07.0 '82545EM Gigabit Ethernet Controller (Copper) 100f'
>> drv=vfio-pci unused=e1000
>> 
>> Network devices using kernel driver
>> ===================================
>> 0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
>> if=eth0
>> drv=e1000 unused=vfio-pci *Active*
>> 
>> 
>> 
>> ./ipv4_multicast
>> ...
>> EAL: 0000:02:04.0 VFIO group is not viable! Not all devices in IOMMU
>> group bound to VFIO or unbound
>> EAL: Requested device 0000:02:04.0 cannot be used
>> EAL: 0000:02:05.0 VFIO group is not viable! Not all devices in IOMMU
>> group bound to VFIO or unbound
>> EAL: Requested device 0000:02:05.0 cannot be used
>> EAL: 0000:02:06.0 VFIO group is not viable! Not all devices in IOMMU
>> group bound to VFIO or unbound
>> EAL: Requested device 0000:02:06.0 cannot be used
>> EAL: 0000:02:07.0 VFIO group is not viable! Not all devices in IOMMU
>> group bound to VFIO or unbound
>> EAL: Requested device 0000:02:07.0 cannot be used
>> ...
>> 
>> 
>> root@devuan4-sysman:~# readlink
>> "/sys/bus/pci/devices/0000:02:04.0/iommu_group"
>> ../../../../kernel/iommu_groups/5
>> root@devuan4-sysman:~# readlink
>> "/sys/bus/pci/devices/0000:02:05.0/iommu_group"
>> ../../../../kernel/iommu_groups/5
>> root@devuan4-sysman:~# readlink
>> "/sys/bus/pci/devices/0000:02:06.0/iommu_group"
>> ../../../../kernel/iommu_groups/5
>> root@devuan4-sysman:~# readlink
>> "/sys/bus/pci/devices/0000:02:07.0/iommu_group"
>> ../../../../kernel/iommu_groups/5
>> 
>> 
>> 
>> Can someone, please, help me resolve this problem?
> 
> Can't really be fixed in software, the IOMMU groups come from the 
> hardware.
> The best workaround if possible is to use a different NIC for eth0.

Re: How to resolve "Not all devices in IOMMU group bound to VFIO or unbound" ?

[Stephen Hemminger]

On Wed, 10 Aug 2022 17:11:04 +0300
sysman@scaled.cloud wrote:

> Hello!
> 
>     "The best workaround if possible is to use a different NIC for eth0."
> 
>   Can I ask u, please, explain with more words what I'm need to do ?

Buy a USB network and plug it into the system and use that for management
connection.

Error: bind failed for 0000:02:04.0 - Cannot bind to driver uio_pci_generic: [Errno 19] No such device

[sysman]

Hello!

  I tried to use uio_pci_generic instead of VFIO :

root@devuan4-sysman:~# lsmod | grep uio
uio_pci_generic        16384  0
igb_uio                20480  0
uio                    20480  2 igb_uio,uio_pci_generic
root@devuan4-sysman:~#


root@devuan4-sysman:~# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:02:05.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
drv=igb_uio unused=e1000,vfio-pci,uio_pci_generic
0000:02:06.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
drv=igb_uio unused=e1000,vfio-pci,uio_pci_generic
0000:02:07.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
drv=igb_uio unused=e1000,vfio-pci,uio_pci_generic

Other Network devices
=====================
0000:02:04.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
unused=e1000,igb_uio,vfio-pci,uio_pci_generic


root@devuan4-sysman:~# dpdk-devbind.py -b uio_pci_generic  0000:02:04.0
Error: bind failed for 0000:02:04.0 - Cannot bind to driver 
uio_pci_generic: [Errno 19] No such device


  Please, help me to resolve this problem.

  Thanks!

Re: Error: bind failed for 0000:02:04.0 - Cannot bind to driver uio_pci_generic: [Errno 19] No such device

[Stephen Hemminger]

On Thu, 11 Aug 2022 12:00:22 +0300
sysman@scaled.cloud wrote:

> Hello!
> 
>   I tried to use uio_pci_generic instead of VFIO :
> 
> root@devuan4-sysman:~# lsmod | grep uio
> uio_pci_generic        16384  0
> igb_uio                20480  0
> uio                    20480  2 igb_uio,uio_pci_generic
> root@devuan4-sysman:~#
> 
> 
> root@devuan4-sysman:~# dpdk-devbind.py -s
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:02:05.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
> drv=igb_uio unused=e1000,vfio-pci,uio_pci_generic
> 0000:02:06.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
> drv=igb_uio unused=e1000,vfio-pci,uio_pci_generic
> 0000:02:07.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
> drv=igb_uio unused=e1000,vfio-pci,uio_pci_generic
> 
> Other Network devices
> =====================
> 0000:02:04.0 '82545EM Gigabit Ethernet Controller (Copper) 100f' 
> unused=e1000,igb_uio,vfio-pci,uio_pci_generic
> 
> 
> root@devuan4-sysman:~# dpdk-devbind.py -b uio_pci_generic  0000:02:04.0
> Error: bind failed for 0000:02:04.0 - Cannot bind to driver 
> uio_pci_generic: [Errno 19] No such device
> 
> 
>   Please, help me to resolve this problem.

Check kernel dmesg log, there maybe a better message waiting there.
Note: igb_uio and uio_pci_generic may not be compatiable with each other.

Since this looks like a 4 port NIC card, you may have issues.
Internally the for card has a PCI bridge. Looking at PCI topology
with lspci may give some insight.

DPD-22.07 bind error=-19

[sysman]

Hello All!

  Can someone help me to resolved situation:

ubuntu 20 lts
dpdk 22-0


root@noserve1:/home/user/Works/tootoo2# dpdk-devbind.py -s

Network devices using kernel driver
===================================
0000:05:00.0 'I211 Gigabit Network Connection 1539' if=enp5s0 drv=igb 
unused=vfio-pci,uio_pci_generic *Active*
0000:06:00.0 'I211 Gigabit Network Connection 1539' if=enp6s0 drv=igb 
unused=vfio-pci,uio_pci_generic
0000:07:00.0 'I211 Gigabit Network Connection 1539' if=enp7s0 drv=igb 
unused=vfio-pci,uio_pci_generic
0000:08:00.0 'I211 Gigabit Network Connection 1539' if=enp8s0 drv=igb 
unused=vfio-pci,uio_pci_generic
0000:09:00.0 'I211 Gigabit Network Connection 1539' if=enp9s0 drv=igb 
unused=vfio-pci,uio_pci_generic
0000:0a:00.0 'I211 Gigabit Network Connection 1539' if=enp10s0 drv=igb 
unused=vfio-pci,uio_pci_generic

Other Network devices
=====================
0000:01:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
unused=i40e,vfio-pci,uio_pci_generic
0000:01:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
unused=i40e,vfio-pci,uio_pci_generic
0000:01:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
unused=i40e,vfio-pci,uio_pci_generic
0000:01:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' 
unused=i40e,vfio-pci,uio_pci_generic


When i try to bind:


root@noserve1:/home/user/Works/tootoo2# dpdk-devbind.py -b i40e 
0000:01:00.0
Error: bind failed for 0000:01:00.0 - Cannot bind to driver i40e: [Errno 
19] No such device
root@noserve1:/home/user/Works/tootoo2# dpdk-devbind.py -b 
uio_pci_generic 0000:01:00.0
Error: bind failed for 0000:01:00.0 - Cannot bind to driver 
uio_pci_generic: [Errno 19] No such device



Thanks in advance!

IP/UDP packet reordering

[sysman]

Hello there!

  I have tried to get experience with the DPDK, so, I wrote a small app 
based on the L2FWD.
  Simple schema: VM/DPDK -----10Gb DAC ---- VM/DPDK. When I performs a 
testing of traffic pass with the IPERF3 I getting:

...
[SUM]  0.0-10.0 sec  40844 datagrams received out-of-order
...

So, what I'm need to check additionally to understand how control of 
order is performed in the DPDK/EAL engine ?

TIA.