Re: [PATCH v2] PCI: Explicitly put devices into D0 when initializing

Re: [PATCH v2] PCI: Explicitly put devices into D0 when initializing

[Nicolas Dichtel]

Le 06/05/2025 à 01:06, Bjorn Helgaas a écrit :
> On Wed, Apr 23, 2025 at 11:31:32PM -0500, Mario Limonciello wrote:
>> From: Mario Limonciello <mario.limonciello@amd.com>
>>
>> AMD BIOS team has root caused an issue that NVME storage failed to come
>> back from suspend to a lack of a call to _REG when NVME device was probed.
>>
>> commit 112a7f9c8edbf ("PCI/ACPI: Call _REG when transitioning D-states")
>> added support for calling _REG when transitioning D-states, but this only
>> works if the device actually "transitions" D-states.
>>
>> commit 967577b062417 ("PCI/PM: Keep runtime PM enabled for unbound PCI
>> devices") added support for runtime PM on PCI devices, but never actually
>> 'explicitly' sets the device to D0.
>>
>> To make sure that devices are in D0 and that platform methods such as
>> _REG are called, explicitly set all devices into D0 during initialization.
>>
>> Fixes: 967577b062417 ("PCI/PM: Keep runtime PM enabled for unbound PCI devices")
>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> 
> Applied to pci/pm for v6.16, thanks!
> 

I've a regression after this commit.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4d4c10f763d7

I've started a QEMU with "-cpu host" on an AMD (AMD Ryzen 5 3600 6-Core
Processor) machine + virtio-net interfaces. When I try to start a testpmd (a
DPDK app), it cannot find the virtio port. The ioctl VFIO_GROUP_GET_DEVICE_FD fails.

To reproduce the issue:
qemu-system-x86_64 --enable-kvm -m 5G -cpu host \
	-smp sockets=1,cores=2,threads=2 \
	-snapshot -vga none -display none -nographic \
	-drive if=none,file=/opt/vm/ubuntu-24.04-with-linux-net.qcow2,id=hda \
	-device virtio-blk,drive=hda \
	-device virtio-net,netdev=eth0,addr=03 -netdev user,id=eth0 \
	-device virtio-net,netdev=eth1,addr=04 -netdev socket,id=eth1,mcast=230.0.0.1:1234

git clone git://dpdk.org/dpdk
cd dpdk/
meson build-static --werror --default-library=static --debug
ninja -C build-static
echo 3 > /proc/sys/vm/drop_caches
echo 256 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
modprobe vfio-pci
lspci
python3 ./usertools/dpdk-devbind.py --noiommu-mode -b vfio-pci 0000:00:04.0
./build-static/app/dpdk-testpmd -l 1,2 --socket-mem 512,0 -a 0000:00:04.0 -- -i

Here is the output:
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Getting a vfio_dev_fd for 0000:00:04.0 failed
PCI_BUS: Cannot get offset of region 0.
PCI_BUS: fail to disable req notifier.
PCI_BUS: fail to disable req notifier.
VIRTIO_INIT: eth_virtio_pci_init(): Failed to init PCI device
PCI_BUS: Requested device 0000:00:04.0 cannot be used
EAL: Bus (pci) probe failed.
testpmd: No probed ethernet devices
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
testpmd>

=> the problem starts at the line "Getting a vfio_dev_fd for 0000:00:04.0 failed"
https://git.dpdk.org/dpdk/tree/lib/eal/linux/eal_vfio.c#n966

FWIW, here is the output when it starts correctly:
EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
Interactive-mode selected
Warning: NUMA should be configured manually by using --port-numa-config and
--ring-numa-config parameters along with --numa.
testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will
pair with itself.

Configuring Port 0 (socket 0)
EAL: Error disabling MSI-X interrupts for fd 277
Port 0: DE:ED:01:E0:1B:75
Checking link statuses...
Done
testpmd>

Any help would be appreciated.

Regards,
Nicolas

Re: [PATCH v2] PCI: Explicitly put devices into D0 when initializing

[Mario Limonciello]

On 6/11/2025 7:14 AM, Nicolas Dichtel wrote:
> Le 06/05/2025 à 01:06, Bjorn Helgaas a écrit :
>> On Wed, Apr 23, 2025 at 11:31:32PM -0500, Mario Limonciello wrote:
>>> From: Mario Limonciello <mario.limonciello@amd.com>
>>>
>>> AMD BIOS team has root caused an issue that NVME storage failed to come
>>> back from suspend to a lack of a call to _REG when NVME device was probed.
>>>
>>> commit 112a7f9c8edbf ("PCI/ACPI: Call _REG when transitioning D-states")
>>> added support for calling _REG when transitioning D-states, but this only
>>> works if the device actually "transitions" D-states.
>>>
>>> commit 967577b062417 ("PCI/PM: Keep runtime PM enabled for unbound PCI
>>> devices") added support for runtime PM on PCI devices, but never actually
>>> 'explicitly' sets the device to D0.
>>>
>>> To make sure that devices are in D0 and that platform methods such as
>>> _REG are called, explicitly set all devices into D0 during initialization.
>>>
>>> Fixes: 967577b062417 ("PCI/PM: Keep runtime PM enabled for unbound PCI devices")
>>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>>
>> Applied to pci/pm for v6.16, thanks!
>>
> 
> I've a regression after this commit.
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4d4c10f763d7
> 
> I've started a QEMU with "-cpu host" on an AMD (AMD Ryzen 5 3600 6-Core
> Processor) machine + virtio-net interfaces. When I try to start a testpmd (a
> DPDK app), it cannot find the virtio port. The ioctl VFIO_GROUP_GET_DEVICE_FD fails.
> 
> To reproduce the issue:
> qemu-system-x86_64 --enable-kvm -m 5G -cpu host \
> 	-smp sockets=1,cores=2,threads=2 \
> 	-snapshot -vga none -display none -nographic \
> 	-drive if=none,file=/opt/vm/ubuntu-24.04-with-linux-net.qcow2,id=hda \
> 	-device virtio-blk,drive=hda \
> 	-device virtio-net,netdev=eth0,addr=03 -netdev user,id=eth0 \
> 	-device virtio-net,netdev=eth1,addr=04 -netdev socket,id=eth1,mcast=230.0.0.1:1234
> 
> git clone git://dpdk.org/dpdk
> cd dpdk/
> meson build-static --werror --default-library=static --debug
> ninja -C build-static
> echo 3 > /proc/sys/vm/drop_caches
> echo 256 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
> modprobe vfio-pci
> lspci
> python3 ./usertools/dpdk-devbind.py --noiommu-mode -b vfio-pci 0000:00:04.0
> ./build-static/app/dpdk-testpmd -l 1,2 --socket-mem 512,0 -a 0000:00:04.0 -- -i
> 
> Here is the output:
> EAL: Detected CPU lcores: 4
> EAL: Detected NUMA nodes: 1
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: VFIO support initialized
> EAL: Using IOMMU type 8 (No-IOMMU)
> EAL: Getting a vfio_dev_fd for 0000:00:04.0 failed
> PCI_BUS: Cannot get offset of region 0.
> PCI_BUS: fail to disable req notifier.
> PCI_BUS: fail to disable req notifier.
> VIRTIO_INIT: eth_virtio_pci_init(): Failed to init PCI device
> PCI_BUS: Requested device 0000:00:04.0 cannot be used
> EAL: Bus (pci) probe failed.
> testpmd: No probed ethernet devices
> Interactive-mode selected
> testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> Done
> testpmd>
> 
> => the problem starts at the line "Getting a vfio_dev_fd for 0000:00:04.0 failed"
> https://git.dpdk.org/dpdk/tree/lib/eal/linux/eal_vfio.c#n966
> 
> FWIW, here is the output when it starts correctly:
> EAL: Detected CPU lcores: 4
> EAL: Detected NUMA nodes: 1
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: VFIO support initialized
> EAL: Using IOMMU type 8 (No-IOMMU)
> Interactive-mode selected
> Warning: NUMA should be configured manually by using --port-numa-config and
> --ring-numa-config parameters along with --numa.
> testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> 
> Warning! port-topology=paired and odd forward ports number, the last port will
> pair with itself.
> 
> Configuring Port 0 (socket 0)
> EAL: Error disabling MSI-X interrupts for fd 277
> Port 0: DE:ED:01:E0:1B:75
> Checking link statuses...
> Done
> testpmd>
> 
> Any help would be appreciated.
> 
> Regards,
> Nicolas

+AlexW

Thanks for the report and especially for the repro steps.  This sounds 
just like the one reported for the QAT regression also in this thread.

https://lore.kernel.org/linux-pci/aEmS+OQL7IbjdwKs@gcabiddu-mobl.ger.corp.intel.com/T/#m7e8929d6421690dc8bd6dc639d86c2b4db27cbc4

I'm traveling this week, but as your report doesn't have a dependency on 
QAT hardware I will try to reproduce next week to understand what's 
going on.

Alex - if you have any ideas please let me know.

Re: [PATCH v2] PCI: Explicitly put devices into D0 when initializing

[Alex Williamson]

On Wed, 11 Jun 2025 07:56:21 -0700
Mario Limonciello <superm1@kernel.org> wrote:

> On 6/11/2025 7:14 AM, Nicolas Dichtel wrote:
> > Le 06/05/2025 à 01:06, Bjorn Helgaas a écrit :  
> >> On Wed, Apr 23, 2025 at 11:31:32PM -0500, Mario Limonciello wrote:  
> >>> From: Mario Limonciello <mario.limonciello@amd.com>
> >>>
> >>> AMD BIOS team has root caused an issue that NVME storage failed to come
> >>> back from suspend to a lack of a call to _REG when NVME device was probed.
> >>>
> >>> commit 112a7f9c8edbf ("PCI/ACPI: Call _REG when transitioning D-states")
> >>> added support for calling _REG when transitioning D-states, but this only
> >>> works if the device actually "transitions" D-states.
> >>>
> >>> commit 967577b062417 ("PCI/PM: Keep runtime PM enabled for unbound PCI
> >>> devices") added support for runtime PM on PCI devices, but never actually
> >>> 'explicitly' sets the device to D0.
> >>>
> >>> To make sure that devices are in D0 and that platform methods such as
> >>> _REG are called, explicitly set all devices into D0 during initialization.
> >>>
> >>> Fixes: 967577b062417 ("PCI/PM: Keep runtime PM enabled for unbound PCI devices")
> >>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>  
> >>
> >> Applied to pci/pm for v6.16, thanks!
> >>  
> > 
> > I've a regression after this commit.
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4d4c10f763d7
> > 
> > I've started a QEMU with "-cpu host" on an AMD (AMD Ryzen 5 3600 6-Core
> > Processor) machine + virtio-net interfaces. When I try to start a testpmd (a
> > DPDK app), it cannot find the virtio port. The ioctl VFIO_GROUP_GET_DEVICE_FD fails.
> > 
> > To reproduce the issue:
> > qemu-system-x86_64 --enable-kvm -m 5G -cpu host \
> > 	-smp sockets=1,cores=2,threads=2 \
> > 	-snapshot -vga none -display none -nographic \
> > 	-drive if=none,file=/opt/vm/ubuntu-24.04-with-linux-net.qcow2,id=hda \
> > 	-device virtio-blk,drive=hda \
> > 	-device virtio-net,netdev=eth0,addr=03 -netdev user,id=eth0 \
> > 	-device virtio-net,netdev=eth1,addr=04 -netdev socket,id=eth1,mcast=230.0.0.1:1234
> > 
> > git clone git://dpdk.org/dpdk
> > cd dpdk/
> > meson build-static --werror --default-library=static --debug
> > ninja -C build-static
> > echo 3 > /proc/sys/vm/drop_caches
> > echo 256 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
> > modprobe vfio-pci
> > lspci
> > python3 ./usertools/dpdk-devbind.py --noiommu-mode -b vfio-pci 0000:00:04.0
> > ./build-static/app/dpdk-testpmd -l 1,2 --socket-mem 512,0 -a 0000:00:04.0 -- -i
> > 
> > Here is the output:
> > EAL: Detected CPU lcores: 4
> > EAL: Detected NUMA nodes: 1
> > EAL: Detected static linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Selected IOVA mode 'PA'
> > EAL: VFIO support initialized
> > EAL: Using IOMMU type 8 (No-IOMMU)
> > EAL: Getting a vfio_dev_fd for 0000:00:04.0 failed
> > PCI_BUS: Cannot get offset of region 0.
> > PCI_BUS: fail to disable req notifier.
> > PCI_BUS: fail to disable req notifier.
> > VIRTIO_INIT: eth_virtio_pci_init(): Failed to init PCI device
> > PCI_BUS: Requested device 0000:00:04.0 cannot be used
> > EAL: Bus (pci) probe failed.
> > testpmd: No probed ethernet devices
> > Interactive-mode selected
> > testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
> > testpmd: preferred mempool ops selected: ring_mp_mc
> > Done  
> > testpmd>  
> >   
> > => the problem starts at the line "Getting a vfio_dev_fd for 0000:00:04.0 failed"  
> > https://git.dpdk.org/dpdk/tree/lib/eal/linux/eal_vfio.c#n966
> > 
> > FWIW, here is the output when it starts correctly:
> > EAL: Detected CPU lcores: 4
> > EAL: Detected NUMA nodes: 1
> > EAL: Detected static linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Selected IOVA mode 'PA'
> > EAL: VFIO support initialized
> > EAL: Using IOMMU type 8 (No-IOMMU)
> > Interactive-mode selected
> > Warning: NUMA should be configured manually by using --port-numa-config and
> > --ring-numa-config parameters along with --numa.
> > testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
> > testpmd: preferred mempool ops selected: ring_mp_mc
> > 
> > Warning! port-topology=paired and odd forward ports number, the last port will
> > pair with itself.
> > 
> > Configuring Port 0 (socket 0)
> > EAL: Error disabling MSI-X interrupts for fd 277
> > Port 0: DE:ED:01:E0:1B:75
> > Checking link statuses...
> > Done  
> > testpmd>  
> > 
> > Any help would be appreciated.
> > 
> > Regards,
> > Nicolas  
> 
> +AlexW
> 
> Thanks for the report and especially for the repro steps.  This sounds 
> just like the one reported for the QAT regression also in this thread.
> 
> https://lore.kernel.org/linux-pci/aEmS+OQL7IbjdwKs@gcabiddu-mobl.ger.corp.intel.com/T/#m7e8929d6421690dc8bd6dc639d86c2b4db27cbc4
> 
> I'm traveling this week, but as your report doesn't have a dependency on 
> QAT hardware I will try to reproduce next week to understand what's 
> going on.
> 
> Alex - if you have any ideas please let me know.

Note that this instantiation of the virtio-net device creates it as a
non-PCIe device, where QEMU only seems to create a PM capability when
the device is exposed as PCIe.  Therefore this could also be a
manifestation that we've made pm_runtime initialization dependent on
the device having a PM capability.  Thanks,

Alex