* l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
@ 2022-09-14 3:18 F32
2022-09-16 15:10 ` Stephen Hemminger
0 siblings, 1 reply; 6+ messages in thread
From: F32 @ 2022-09-14 3:18 UTC (permalink / raw)
To: users
[-- Attachment #1: Type: text/plain, Size: 6904 bytes --]
Hi,
Recently I'm testing dpdk on a mini PC with N5095 CPU and 5 I225-V ethernet adapters under Ubuntu Server 22.04.1.
$ uname -r
5.15.0-47-generic
$ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt"
$ update-grub
$ reboot
$ dmesg | grep DMAR
[ 0.016399] ACPI: DMAR 0x00000000787D6000 000088 (v02 INTEL EDK2 00000002 01000013)
[ 0.016443] ACPI: Reserving DMAR table memory at [mem 0x787d6000-0x787d6087]
[ 0.029273] DMAR: IOMMU enabled
[ 0.093267] DMAR: Host address width 39
[ 0.093269] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.093276] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 49e2ff0505e
[ 0.093281] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.093287] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.093291] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fbfffff
[ 0.093296] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.093299] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.093301] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.095158] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.288306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[ 0.372113] DMAR: No ATSR found
[ 0.372115] DMAR: No SATC found
[ 0.372118] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.372120] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.372122] DMAR: IOMMU feature nwfs inconsistent
[ 0.372124] DMAR: IOMMU feature pds inconsistent
[ 0.372125] DMAR: IOMMU feature eafs inconsistent
[ 0.372127] DMAR: IOMMU feature prs inconsistent
[ 0.372129] DMAR: IOMMU feature nest inconsistent
[ 0.372130] DMAR: IOMMU feature mts inconsistent
[ 0.372132] DMAR: IOMMU feature sc_support inconsistent
[ 0.372134] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.372136] DMAR: dmar0: Using Queued invalidation
[ 0.372141] DMAR: dmar1: Using Queued invalidation
[ 0.373075] DMAR: Intel(R) Virtualization Technology for Directed I/O
$ lspci
lspci
00:00.0 Host bridge: Intel Corporation Device 4e24
00:02.0 VGA compatible controller: Intel Corporation JasperLake [UHD Graphics] (rev 01)
00:04.0 Signal processing controller: Intel Corporation Dynamic Tuning service
00:08.0 System peripheral: Intel Corporation Device 4e11
00:14.0 USB controller: Intel Corporation Device 4ded (rev 01)
00:14.2 RAM memory: Intel Corporation Device 4def (rev 01)
00:15.0 Serial bus controller: Intel Corporation Serial IO I2C Host Controller (rev 01)
00:15.2 Serial bus controller: Intel Corporation Device 4dea (rev 01)
00:16.0 Communication controller: Intel Corporation Management Engine Interface (rev 01)
00:17.0 SATA controller: Intel Corporation Device 4dd3 (rev 01)
00:19.0 Serial bus controller: Intel Corporation Device 4dc5 (rev 01)
00:19.1 Serial bus controller: Intel Corporation Device 4dc6 (rev 01)
00:1c.0 PCI bridge: Intel Corporation Device 4db8 (rev 01)
00:1c.4 PCI bridge: Intel Corporation Device 4dbc (rev 01)
00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01)
00:1c.6 PCI bridge: Intel Corporation Device 4dbe (rev 01)
00:1c.7 PCI bridge: Intel Corporation Device 4dbf (rev 01)
00:1e.0 Communication controller: Intel Corporation Device 4da8 (rev 01)
00:1e.3 Serial bus controller: Intel Corporation Device 4dab (rev 01)
00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)
00:1f.3 Audio device: Intel Corporation Device 4dc8 (rev 01)
00:1f.4 SMBus: Intel Corporation JaserLake SMBus (rev 01)
00:1f.5 Serial bus controller: Intel Corporation JaserLake SPI (flash) Controller (rev 01)
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
03:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
04:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
(after huge page setup)
$ python3 dpdk-hugepages.py -s
Node Pages Size Total
0 1607 2Mb 3Gb
Hugepages mounted on /dev/hugepages /mnt/huge
$ ifconfig enp2s0 down
$ ifconfig enp3s0 down
$ python3 dpdk-devbind.py --bind=vfio-pci 0000:02:00.0
$ python3 dpdk-devbind.py --bind=vfio-pci 0000:03:00.0
$ python3 dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
0000:03:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
Network devices using kernel driver
===================================
0000:01:00.0 'Ethernet Controller I225-V 15f3' if=enp1s0 drv=igc unused=vfio-pci *Active*
0000:04:00.0 'Ethernet Controller I225-V 15f3' if=enp4s0 drv=igc unused=vfio-pci
0000:05:00.0 'Ethernet Controller I225-V 15f3' if=enp5s0 drv=igc unused=vfio-pci
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'DMA' devices detected
=========================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
==============================
No 'Misc (rawdev)' devices detected
===================================
No 'Regex' devices detected
===========================
After the above configuration, dpck setup should be done. Then I tried to run l2fwd. It generates the following errror messages:
$ ./l2fwd
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:02:00.0 (socket 0)
EAL: Releasing PCI mapped resource for 0000:02:00.0
EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100800000
EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100900000
EAL: Requested device 0000:02:00.0 cannot be used
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:03:00.0 (socket 0)
EAL: Releasing PCI mapped resource for 0000:03:00.0
EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100904000
EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100a04000
EAL: Requested device 0000:03:00.0 cannot be used
TELEMETRY: No legacy callbacks, legacy socket not created
MAC updating enabled
EAL: Error - exiting with code: 1
Cause: No Ethernet ports - bye
It seems that the I225 adapters are not recognized by DPDK. I can not see any detailed reasons why the adapters cannot be used.
Any suggestions to fix this issue?
Regards,
F32
[-- Attachment #2: Type: text/html, Size: 11852 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
2022-09-14 3:18 l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V) F32
@ 2022-09-16 15:10 ` Stephen Hemminger
2022-09-20 11:04 ` F32
0 siblings, 1 reply; 6+ messages in thread
From: Stephen Hemminger @ 2022-09-16 15:10 UTC (permalink / raw)
To: F32; +Cc: users
On Wed, 14 Sep 2022 11:18:11 +0800 (CST)
F32 <feng32@163.com> wrote:
> Hi,
>
>
> Recently I'm testing dpdk on a mini PC with N5095 CPU and 5 I225-V ethernet adapters under Ubuntu Server 22.04.1.
>
>
> $ uname -r
> 5.15.0-47-generic
>
>
> $ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
> GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt"
> $ update-grub
> $ reboot
>
>
> $ dmesg | grep DMAR
> [ 0.016399] ACPI: DMAR 0x00000000787D6000 000088 (v02 INTEL EDK2 00000002 01000013)
> [ 0.016443] ACPI: Reserving DMAR table memory at [mem 0x787d6000-0x787d6087]
> [ 0.029273] DMAR: IOMMU enabled
> [ 0.093267] DMAR: Host address width 39
> [ 0.093269] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
> [ 0.093276] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 49e2ff0505e
> [ 0.093281] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
> [ 0.093287] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
> [ 0.093291] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fbfffff
> [ 0.093296] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
> [ 0.093299] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
> [ 0.093301] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
> [ 0.095158] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [ 0.288306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
> [ 0.372113] DMAR: No ATSR found
> [ 0.372115] DMAR: No SATC found
> [ 0.372118] DMAR: IOMMU feature fl1gp_support inconsistent
> [ 0.372120] DMAR: IOMMU feature pgsel_inv inconsistent
> [ 0.372122] DMAR: IOMMU feature nwfs inconsistent
> [ 0.372124] DMAR: IOMMU feature pds inconsistent
> [ 0.372125] DMAR: IOMMU feature eafs inconsistent
> [ 0.372127] DMAR: IOMMU feature prs inconsistent
> [ 0.372129] DMAR: IOMMU feature nest inconsistent
> [ 0.372130] DMAR: IOMMU feature mts inconsistent
> [ 0.372132] DMAR: IOMMU feature sc_support inconsistent
> [ 0.372134] DMAR: IOMMU feature dev_iotlb_support inconsistent
> [ 0.372136] DMAR: dmar0: Using Queued invalidation
> [ 0.372141] DMAR: dmar1: Using Queued invalidation
> [ 0.373075] DMAR: Intel(R) Virtualization Technology for Directed I/O
>
>
> $ lspci
> lspci
> 00:00.0 Host bridge: Intel Corporation Device 4e24
> 00:02.0 VGA compatible controller: Intel Corporation JasperLake [UHD Graphics] (rev 01)
> 00:04.0 Signal processing controller: Intel Corporation Dynamic Tuning service
> 00:08.0 System peripheral: Intel Corporation Device 4e11
> 00:14.0 USB controller: Intel Corporation Device 4ded (rev 01)
> 00:14.2 RAM memory: Intel Corporation Device 4def (rev 01)
> 00:15.0 Serial bus controller: Intel Corporation Serial IO I2C Host Controller (rev 01)
> 00:15.2 Serial bus controller: Intel Corporation Device 4dea (rev 01)
> 00:16.0 Communication controller: Intel Corporation Management Engine Interface (rev 01)
> 00:17.0 SATA controller: Intel Corporation Device 4dd3 (rev 01)
> 00:19.0 Serial bus controller: Intel Corporation Device 4dc5 (rev 01)
> 00:19.1 Serial bus controller: Intel Corporation Device 4dc6 (rev 01)
> 00:1c.0 PCI bridge: Intel Corporation Device 4db8 (rev 01)
> 00:1c.4 PCI bridge: Intel Corporation Device 4dbc (rev 01)
> 00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01)
> 00:1c.6 PCI bridge: Intel Corporation Device 4dbe (rev 01)
> 00:1c.7 PCI bridge: Intel Corporation Device 4dbf (rev 01)
> 00:1e.0 Communication controller: Intel Corporation Device 4da8 (rev 01)
> 00:1e.3 Serial bus controller: Intel Corporation Device 4dab (rev 01)
> 00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)
> 00:1f.3 Audio device: Intel Corporation Device 4dc8 (rev 01)
> 00:1f.4 SMBus: Intel Corporation JaserLake SMBus (rev 01)
> 00:1f.5 Serial bus controller: Intel Corporation JaserLake SPI (flash) Controller (rev 01)
> 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
> 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
> 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
> 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
> 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>
>
> (after huge page setup)
> $ python3 dpdk-hugepages.py -s
> Node Pages Size Total
> 0 1607 2Mb 3Gb
>
>
> Hugepages mounted on /dev/hugepages /mnt/huge
>
>
> $ ifconfig enp2s0 down
> $ ifconfig enp3s0 down
> $ python3 dpdk-devbind.py --bind=vfio-pci 0000:02:00.0
> $ python3 dpdk-devbind.py --bind=vfio-pci 0000:03:00.0
> $ python3 dpdk-devbind.py -s
>
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:02:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
> 0000:03:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
>
>
> Network devices using kernel driver
> ===================================
> 0000:01:00.0 'Ethernet Controller I225-V 15f3' if=enp1s0 drv=igc unused=vfio-pci *Active*
> 0000:04:00.0 'Ethernet Controller I225-V 15f3' if=enp4s0 drv=igc unused=vfio-pci
> 0000:05:00.0 'Ethernet Controller I225-V 15f3' if=enp5s0 drv=igc unused=vfio-pci
>
>
> No 'Baseband' devices detected
> ==============================
>
>
> No 'Crypto' devices detected
> ============================
>
>
> No 'DMA' devices detected
> =========================
>
>
> No 'Eventdev' devices detected
> ==============================
>
>
> No 'Mempool' devices detected
> =============================
>
>
> No 'Compress' devices detected
> ==============================
>
>
> No 'Misc (rawdev)' devices detected
> ===================================
>
>
> No 'Regex' devices detected
> ===========================
>
>
> After the above configuration, dpck setup should be done. Then I tried to run l2fwd. It generates the following errror messages:
>
>
> $ ./l2fwd
> EAL: Detected CPU lcores: 4
> EAL: Detected NUMA nodes: 1
> EAL: Detected shared linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> EAL: Using IOMMU type 1 (Type 1)
> EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:02:00.0 (socket 0)
> EAL: Releasing PCI mapped resource for 0000:02:00.0
> EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100800000
> EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100900000
> EAL: Requested device 0000:02:00.0 cannot be used
> EAL: Using IOMMU type 1 (Type 1)
> EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:03:00.0 (socket 0)
> EAL: Releasing PCI mapped resource for 0000:03:00.0
> EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100904000
> EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100a04000
> EAL: Requested device 0000:03:00.0 cannot be used
> TELEMETRY: No legacy callbacks, legacy socket not created
> MAC updating enabled
> EAL: Error - exiting with code: 1
> Cause: No Ethernet ports - bye
>
>
> It seems that the I225 adapters are not recognized by DPDK. I can not see any detailed reasons why the adapters cannot be used.
> Any suggestions to fix this issue?
>
>
> Regards,
> F32
Check the IOMMU groups on that system.
Often multiple PCI devices (especially multi-port NICs) share a single IOMMU
group and VFIO does not allow using these devices unless all subdevices
in the group are unbound from the kernel driver.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re:Re: l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
2022-09-16 15:10 ` Stephen Hemminger
@ 2022-09-20 11:04 ` F32
2022-09-20 15:16 ` Stephen Hemminger
0 siblings, 1 reply; 6+ messages in thread
From: F32 @ 2022-09-20 11:04 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: users
[-- Attachment #1: Type: text/plain, Size: 8320 bytes --]
Hi Stephen,
I have checked related iommu_groups:
$ readlink 0000\:01\:00.0/iommu_group
../../../../kernel/iommu_groups/16
$ readlink 0000\:02\:00.0/iommu_group
../../../../kernel/iommu_groups/17
$ readlink 0000\:03\:00.0/iommu_group
../../../../kernel/iommu_groups/18
$ readlink 0000\:04\:00.0/iommu_group
../../../../kernel/iommu_groups/19
$ readlink 0000\:05\:00.0/iommu_group
../../../../kernel/iommu_groups/20
It seems that the adapters belongs to different iommu_groups.
At 2022-09-16 23:10:59, "Stephen Hemminger" <stephen@networkplumber.org> wrote:
>On Wed, 14 Sep 2022 11:18:11 +0800 (CST)
>F32 <feng32@163.com> wrote:
>
>> Hi,
>>
>>
>> Recently I'm testing dpdk on a mini PC with N5095 CPU and 5 I225-V ethernet adapters under Ubuntu Server 22.04.1.
>>
>>
>> $ uname -r
>> 5.15.0-47-generic
>>
>>
>> $ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
>> GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt"
>> $ update-grub
>> $ reboot
>>
>>
>> $ dmesg | grep DMAR
>> [ 0.016399] ACPI: DMAR 0x00000000787D6000 000088 (v02 INTEL EDK2 00000002 01000013)
>> [ 0.016443] ACPI: Reserving DMAR table memory at [mem 0x787d6000-0x787d6087]
>> [ 0.029273] DMAR: IOMMU enabled
>> [ 0.093267] DMAR: Host address width 39
>> [ 0.093269] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
>> [ 0.093276] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 49e2ff0505e
>> [ 0.093281] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
>> [ 0.093287] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
>> [ 0.093291] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fbfffff
>> [ 0.093296] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
>> [ 0.093299] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
>> [ 0.093301] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
>> [ 0.095158] DMAR-IR: Enabled IRQ remapping in x2apic mode
>> [ 0.288306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
>> [ 0.372113] DMAR: No ATSR found
>> [ 0.372115] DMAR: No SATC found
>> [ 0.372118] DMAR: IOMMU feature fl1gp_support inconsistent
>> [ 0.372120] DMAR: IOMMU feature pgsel_inv inconsistent
>> [ 0.372122] DMAR: IOMMU feature nwfs inconsistent
>> [ 0.372124] DMAR: IOMMU feature pds inconsistent
>> [ 0.372125] DMAR: IOMMU feature eafs inconsistent
>> [ 0.372127] DMAR: IOMMU feature prs inconsistent
>> [ 0.372129] DMAR: IOMMU feature nest inconsistent
>> [ 0.372130] DMAR: IOMMU feature mts inconsistent
>> [ 0.372132] DMAR: IOMMU feature sc_support inconsistent
>> [ 0.372134] DMAR: IOMMU feature dev_iotlb_support inconsistent
>> [ 0.372136] DMAR: dmar0: Using Queued invalidation
>> [ 0.372141] DMAR: dmar1: Using Queued invalidation
>> [ 0.373075] DMAR: Intel(R) Virtualization Technology for Directed I/O
>>
>>
>> $ lspci
>> lspci
>> 00:00.0 Host bridge: Intel Corporation Device 4e24
>> 00:02.0 VGA compatible controller: Intel Corporation JasperLake [UHD Graphics] (rev 01)
>> 00:04.0 Signal processing controller: Intel Corporation Dynamic Tuning service
>> 00:08.0 System peripheral: Intel Corporation Device 4e11
>> 00:14.0 USB controller: Intel Corporation Device 4ded (rev 01)
>> 00:14.2 RAM memory: Intel Corporation Device 4def (rev 01)
>> 00:15.0 Serial bus controller: Intel Corporation Serial IO I2C Host Controller (rev 01)
>> 00:15.2 Serial bus controller: Intel Corporation Device 4dea (rev 01)
>> 00:16.0 Communication controller: Intel Corporation Management Engine Interface (rev 01)
>> 00:17.0 SATA controller: Intel Corporation Device 4dd3 (rev 01)
>> 00:19.0 Serial bus controller: Intel Corporation Device 4dc5 (rev 01)
>> 00:19.1 Serial bus controller: Intel Corporation Device 4dc6 (rev 01)
>> 00:1c.0 PCI bridge: Intel Corporation Device 4db8 (rev 01)
>> 00:1c.4 PCI bridge: Intel Corporation Device 4dbc (rev 01)
>> 00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01)
>> 00:1c.6 PCI bridge: Intel Corporation Device 4dbe (rev 01)
>> 00:1c.7 PCI bridge: Intel Corporation Device 4dbf (rev 01)
>> 00:1e.0 Communication controller: Intel Corporation Device 4da8 (rev 01)
>> 00:1e.3 Serial bus controller: Intel Corporation Device 4dab (rev 01)
>> 00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)
>> 00:1f.3 Audio device: Intel Corporation Device 4dc8 (rev 01)
>> 00:1f.4 SMBus: Intel Corporation JaserLake SMBus (rev 01)
>> 00:1f.5 Serial bus controller: Intel Corporation JaserLake SPI (flash) Controller (rev 01)
>> 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>> 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>> 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>> 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>> 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>>
>>
>> (after huge page setup)
>> $ python3 dpdk-hugepages.py -s
>> Node Pages Size Total
>> 0 1607 2Mb 3Gb
>>
>>
>> Hugepages mounted on /dev/hugepages /mnt/huge
>>
>>
>> $ ifconfig enp2s0 down
>> $ ifconfig enp3s0 down
>> $ python3 dpdk-devbind.py --bind=vfio-pci 0000:02:00.0
>> $ python3 dpdk-devbind.py --bind=vfio-pci 0000:03:00.0
>> $ python3 dpdk-devbind.py -s
>>
>>
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:02:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
>> 0000:03:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
>>
>>
>> Network devices using kernel driver
>> ===================================
>> 0000:01:00.0 'Ethernet Controller I225-V 15f3' if=enp1s0 drv=igc unused=vfio-pci *Active*
>> 0000:04:00.0 'Ethernet Controller I225-V 15f3' if=enp4s0 drv=igc unused=vfio-pci
>> 0000:05:00.0 'Ethernet Controller I225-V 15f3' if=enp5s0 drv=igc unused=vfio-pci
>>
>>
>> No 'Baseband' devices detected
>> ==============================
>>
>>
>> No 'Crypto' devices detected
>> ============================
>>
>>
>> No 'DMA' devices detected
>> =========================
>>
>>
>> No 'Eventdev' devices detected
>> ==============================
>>
>>
>> No 'Mempool' devices detected
>> =============================
>>
>>
>> No 'Compress' devices detected
>> ==============================
>>
>>
>> No 'Misc (rawdev)' devices detected
>> ===================================
>>
>>
>> No 'Regex' devices detected
>> ===========================
>>
>>
>> After the above configuration, dpck setup should be done. Then I tried to run l2fwd. It generates the following errror messages:
>>
>>
>> $ ./l2fwd
>> EAL: Detected CPU lcores: 4
>> EAL: Detected NUMA nodes: 1
>> EAL: Detected shared linkage of DPDK
>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> EAL: Selected IOVA mode 'VA'
>> EAL: VFIO support initialized
>> EAL: Using IOMMU type 1 (Type 1)
>> EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:02:00.0 (socket 0)
>> EAL: Releasing PCI mapped resource for 0000:02:00.0
>> EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100800000
>> EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100900000
>> EAL: Requested device 0000:02:00.0 cannot be used
>> EAL: Using IOMMU type 1 (Type 1)
>> EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:03:00.0 (socket 0)
>> EAL: Releasing PCI mapped resource for 0000:03:00.0
>> EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100904000
>> EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100a04000
>> EAL: Requested device 0000:03:00.0 cannot be used
>> TELEMETRY: No legacy callbacks, legacy socket not created
>> MAC updating enabled
>> EAL: Error - exiting with code: 1
>> Cause: No Ethernet ports - bye
>>
>>
>> It seems that the I225 adapters are not recognized by DPDK. I can not see any detailed reasons why the adapters cannot be used.
>> Any suggestions to fix this issue?
>>
>>
>> Regards,
>> F32
>
>Check the IOMMU groups on that system.
>Often multiple PCI devices (especially multi-port NICs) share a single IOMMU
>group and VFIO does not allow using these devices unless all subdevices
>in the group are unbound from the kernel driver.
[-- Attachment #2: Type: text/html, Size: 9901 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
2022-09-20 11:04 ` F32
@ 2022-09-20 15:16 ` Stephen Hemminger
2022-09-21 11:51 ` F32
0 siblings, 1 reply; 6+ messages in thread
From: Stephen Hemminger @ 2022-09-20 15:16 UTC (permalink / raw)
To: F32; +Cc: users
On Tue, 20 Sep 2022 19:04:57 +0800 (CST)
F32 <feng32@163.com> wrote:
> Hi Stephen,
>
>
> I have checked related iommu_groups:
>
>
> $ readlink 0000\:01\:00.0/iommu_group
>
> ../../../../kernel/iommu_groups/16
>
> $ readlink 0000\:02\:00.0/iommu_group
>
> ../../../../kernel/iommu_groups/17
>
> $ readlink 0000\:03\:00.0/iommu_group
>
> ../../../../kernel/iommu_groups/18
>
> $ readlink 0000\:04\:00.0/iommu_group
>
> ../../../../kernel/iommu_groups/19
>
> $ readlink 0000\:05\:00.0/iommu_group
>
> ../../../../kernel/iommu_groups/20
>
>
>
> It seems that the adapters belongs to different iommu_groups.
The way I would fix it is add instrumentation (ie printf) in EAL
code where the can not be used message is and work backwards from there.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re:Re: l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
2022-09-20 15:16 ` Stephen Hemminger
@ 2022-09-21 11:51 ` F32
2022-10-04 6:49 ` David Marchand
0 siblings, 1 reply; 6+ messages in thread
From: F32 @ 2022-09-21 11:51 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: users
[-- Attachment #1: Type: text/plain, Size: 1946 bytes --]
Thanks for the suggestion.
I took half an hour to add debug logs layer by layer and finally I came to igc_init_phy_params_i225 in drivers/net/igc/base/igc_i225.c.
There is a piece of code that looks like this in the function:
> /* Verify phy id and set remaining function pointers */
> switch (phy->id) {
> case I225_I_PHY_ID:
> case I226_LM_PHY_ID:
> phy->type= igc_phy_i225;
> phy->ops.set_d0_lplu_state = igc_set_d0_lplu_state_i225;
> phy->ops.set_d3_lplu_state = igc_set_d3_lplu_state_i225;
> /* TODO - complete with GPY PHY information */
> break;
> default:
> ret_val = -IGC_ERR_PHY;
> goto out;
> }
The supported I225 model is I225_I with PHY_ID 0x67C9DC00, but the adapters on my Mini-PC are I225_V with PHY_ID 0x67C9DCC0.
After adding the case l2fwd is now working.
BTW, I noticed that there are so many different cases (in different source files) which might generate the "Requested device xxx cannot be used" error. Introducing a standard error code may be hellpful for ordinary dpdk users, but it requires lots of work.
At 2022-09-20 23:16:25, "Stephen Hemminger" <stephen@networkplumber.org> wrote:
>On Tue, 20 Sep 2022 19:04:57 +0800 (CST)
>F32 <feng32@163.com> wrote:
>
>> Hi Stephen,
>>
>>
>> I have checked related iommu_groups:
>>
>>
>> $ readlink 0000\:01\:00.0/iommu_group
>>
>> ../../../../kernel/iommu_groups/16
>>
>> $ readlink 0000\:02\:00.0/iommu_group
>>
>> ../../../../kernel/iommu_groups/17
>>
>> $ readlink 0000\:03\:00.0/iommu_group
>>
>> ../../../../kernel/iommu_groups/18
>>
>> $ readlink 0000\:04\:00.0/iommu_group
>>
>> ../../../../kernel/iommu_groups/19
>>
>> $ readlink 0000\:05\:00.0/iommu_group
>>
>> ../../../../kernel/iommu_groups/20
>>
>>
>>
>> It seems that the adapters belongs to different iommu_groups.
>
>The way I would fix it is add instrumentation (ie printf) in EAL
>code where the can not be used message is and work backwards from there.
[-- Attachment #2: Type: text/html, Size: 3252 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Re: l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
2022-09-21 11:51 ` F32
@ 2022-10-04 6:49 ` David Marchand
0 siblings, 0 replies; 6+ messages in thread
From: David Marchand @ 2022-10-04 6:49 UTC (permalink / raw)
To: F32
Cc: Stephen Hemminger, users, Guo, Junfeng, simei, Mah, Yock Gen,
samuel.taripin
On Tue, Oct 4, 2022 at 8:33 AM F32 <feng32@163.com> wrote:
>
> Thanks for the suggestion.
>
> I took half an hour to add debug logs layer by layer and finally I came to igc_init_phy_params_i225 in drivers/net/igc/base/igc_i225.c.
>
> There is a piece of code that looks like this in the function:
>
> > /* Verify phy id and set remaining function pointers */
> > switch (phy->id) {
> > case I225_I_PHY_ID:
> > case I226_LM_PHY_ID:
> > phy->type = igc_phy_i225;
> > phy->ops.set_d0_lplu_state = igc_set_d0_lplu_state_i225;
> > phy->ops.set_d3_lplu_state = igc_set_d3_lplu_state_i225;
> > /* TODO - complete with GPY PHY information */
> > break;
> > default:
> > ret_val = -IGC_ERR_PHY;
> > goto out;
> > }
>
> The supported I225 model is I225_I with PHY_ID 0x67C9DC00, but the adapters on my Mini-PC are I225_V with PHY_ID 0x67C9DCC0.
> After adding the case l2fwd is now working.
A recent change in this area of the driver got merged in next-net-intel.
c493e5ee25fe ("net/igc: remove unncessary PHY ID checking")
Cc: igc maintainers and authors of the change
>
> BTW, I noticed that there are so many different cases (in different source files) which might generate the "Requested device xxx cannot be used" error. Introducing a standard error code may be hellpful for ordinary dpdk users, but it requires lots of work.
I don't see how this is feasible to standardize error codes that would
help here.
The less awful is to get debug log messages, which is the driver responsibility.
--
David Marchand
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-10-04 6:50 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-09-14 3:18 l2fwd: Requested device 0000:02:00.0 cannot be used (Ubuntu 22.04, DPDK 22.07, N5095, I225-V) F32
2022-09-16 15:10 ` Stephen Hemminger
2022-09-20 11:04 ` F32
2022-09-20 15:16 ` Stephen Hemminger
2022-09-21 11:51 ` F32
2022-10-04 6:49 ` David Marchand
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).