binding i226 NIC to dpdk causes link/activity led blinking with no link or activty

[Paul Ugolini <paul@meter.com>]

Hi all, I'm noticing strange behavior when binding an i226-LM to dpdk: the link/activity light will blink until the system is fully rebooted, regardless of physical link state or if there is actually activity on the link.

Using dpdk 22.07.0

Steps to reproduce:
1. remove ethernet from nic 03:00.0
2. reboot
3. boot linux 6.1.12-intel-pk-standard (03:00.0 registers through igc)
	igc 0000:03:00.0: enabling device (0140 -> 0142)
	igc 0000:03:00.0: PCIe PTM not supported by PCIe bus/controller
	igc 0000:03:00.0 (unnamed net_device) (uninitialized): PHC added
	igc 0000:03:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
	igc 0000:03:00.0 eth0: MAC: XX:XX:XX:XX:XX:XX
	igc 0000:03:00.0 enp3s0: renamed from eth0
	igc 0000:03:00.0 enp3s0: PHC removed
4. observe: link/activity led is off
5. bind 03:00.0 to dpdk: dpdk-devbind.py -s vfio-pci 03:00.0
6. observe: link/activity led is flashing with no link
7. attach ethernet cable, bring up dpdk link, pass traffic
8. observe: link/activity led blinks regardless of if traffic is passing or not
9. stop dpdk app on 03:00.0
10. unbind 03:00.0 from dpdk, to igc: dpdk-devbind.py -s igc 03:00.0
	igc 0000:03:00.0: PCIe PTM not supported by PCIe bus/controller
	pps pps0: new PPS source ptp0
	igc 0000:03:00.0 (unnamed net_device) (uninitialized): PHC added
	igc 0000:03:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
	igc 0000:03:00.0 enp3s0: renamed from eth0
	igc 0000:03:00.0 enp3s0: MAC: XX:XX:XX:XX:XX:XX
11. remove ethernet cable from 03:00.0
12. observe: link/activity led is still flashing, despite enp3s0 link being fully down

Resetting the nic with `echo 1 > /sys/bus/pci/devices/0000\:03\:00.0/remove` or removing the nic with `echo 1 > /sys/bus/pci/devices/0000\:03\:00.0/remove; sleep 1; echo 1 > /sys/bus/pci/rescan` does not stop the blinking, but since a soft reboot does

This device has several igxbe nics as well, which don't exhibit this behavior with dpdk. If I replace 03:00.0 with 08:00.0 the link/activity led will behave as expected: light up with a link and only blink on activity.

~# dpdk-devbind -s
Network devices using DPDK-compatible driver
============================================
0000:03:00.0 'Ethernet Controller I226-LM 125b' drv=vfio-pci unused=igc,uio_pci_generic
...
0000:08:00.0 'Ethernet Connection X553 1GbE 15e4' drv=vfio-pci unused=uio_pci_generic


~# lspci -nn
...
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-LM [8086:125b] (rev 04)
...
08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X553 1GbE [8086:15e4] (rev 11)
...


Thanks,
Paul