[dpdk-dev] ixgbevf does not recover from pf reset

[dpdk-dev] ixgbevf does not recover from pf reset

[David Marchand]

Hello Helin, Konstantin,

I noticed this issue quite some time ago (maybe around dpdk-1.3.0, not
sure) but had no time to investigate/report.
I hit it again with dpdk-2.2.0-rc2, so maybe all dpdk versions are impacted.

The test is quite simple :
- send icmp packets from a system (tn) to itself using two 10G interfaces
connected to two VF ports of a dual port 82599 nic,
- run testpmd in a vm owning those two VF ports,
- reset a pf interface in the host,
- then no connectivity on the associated VF port unless it is completely
restarted.

In my previous test, I think I also unplugged the fiber then replugged
(without touching the pf interface status on the host) and observed the
same issue.

I looked at ixgbevf pmd, but I can't find a place where the PF asking for
reset would be handled except at devinit.



Here is the full description of my test :

The system owning the two VF ports is a ubuntu 14.04 virtual machine
(3.13.0-71-generic kernel).
The host running this virtual machine is running ubuntu 14.04 as well (but
kernel is newer 3.19.0-39-generic).


- host setup :
echo 3072
>/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 3072
>/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
mkdir -p
/mnt/huge
mount -t hugetlbfs none /mnt/huge

echo 1 >
"/sys/bus/pci/devices/0000:83:00.0/sriov_numvfs"
bridge link set dev ixgbe0 hwmode veb
ip link set dev ixgbe0 vf 0 mac 00:09:c0:0e:4e:64
ip link set dev ixgbe0 vf 0 spoofchk off
ip link set dev ixgbe0 mtu 9000
ip link set dev ixgbe0 up
ip link set dev ixgbe0 address 00:09:c0:0e:4e:64

echo 1 >
"/sys/bus/pci/devices/0000:83:00.1/sriov_numvfs"
bridge link set dev ixgbe1 hwmode veb
ip link set dev ixgbe1 vf 0 mac 00:09:c0:0e:4e:65
ip link set dev ixgbe1 vf 0 spoofchk off
ip link set dev ixgbe1 mtu 9000
ip link set dev ixgbe1 up
ip link set dev ixgbe1 address 00:09:c0:0e:4e:65

qemu-system-x86_64 -k fr -daemonize -S --enable-kvm -m 3G -mem-path
/mnt/huge -cpu host -smp cores=6,threads=2,sockets=1 -serial
telnet::53714,server,nowait -serial null -monitor
telnet::36576,server,nowait -device
virtio-net,mac=DE:AD:DE:01:02:03,netdev=user.0,addr=03 -netdev
user,id=user.0 -device pci-assign,host=0000:83:10.0,addr=04 -device
pci-assign,host=0000:83:10.1,addr=05 -hda ubuntu-14.04-template.qcow2
-snapshot -vga none -display none


- guest setup :
echo 1024 >
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
mkdir -p
/mnt/huge
mount -t hugetlbfs none /mnt/huge

modprobe uio
modprobe igb_uio
echo 0000:00:04.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind
echo 0000:00:05.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind
echo 8086 10ed > /sys/bus/pci/drivers/igb_uio/new_id

testpmd -w0000:00:04.0 -w0000:00:05.0 --huge-dir=/mnt/huge -n 4 -c 0xe -- -i
[snip]
Port 0:
00:09:C0:0E:4E:64
[snip]
Port 1:
00:09:C0:0E:4E:65
Checking link
statuses...
Port 0 Link Up - speed 10000 Mbps -
full-duplex
Port 1 Link Up - speed 10000 Mbps -
full-duplex
Done

testpmd>
start


- tn setup :
ip link set dev xaui0 up
ip link set dev xaui1 up
ip addr add 1.1.1.1/24 dev xaui0
arp -i xaui0 -s 1.1.1.2 00:09:c0:0e:4e:64



- From here, on tn, send icmp packets with a tcpdump running background on
the other iface :
ping 1.1.1.2
PING 1.1.1.2 (1.1.1.2): 56 data
bytes
03:10:15.003047 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
(0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 1
03:10:16.003151 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
(0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 2
03:10:17.003246 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
(0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 3


- Ok, now, kick ixgbe0 pf interface down / up on host system.
ip link set dev ixgbe0 down
ip link set dev ixgbe0 up


- On tn, no packet is received anymore, the only workaround I found is to
stop and restart the port 0 :

testpmd>
stop
[snip]
Done.

testpmd> port stop
all
Stopping
ports...
Checking link
statuses...
Port 0 Link
Down
Port 1 Link Up - speed 10000 Mbps -
full-duplex
Done


Funny thing here, the link is reported down, while the pf is already back
up.

testpmd> port start all
PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
size no less than 4 (port=0).
Port 0:
00:09:C0:0E:4E:64
PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
size no less than 4 (port=1).
Port 1:
00:09:C0:0E:4E:65
Checking link
statuses...
Port 0 Link Up - speed 10000 Mbps -
full-duplex
Port 1 Link Up - speed 10000 Mbps -
full-duplex
Done



Thanks.

-- 
David Marchand

Re: [dpdk-dev] ixgbevf does not recover from pf reset

[Wu, Jingjing]

Hi, David

We also noticed this issue before. And we are planning to fix this issue.
And the patch for i40e is ready:
http://dpdk.org/dev/patchwork/patch/9832/
http://dpdk.org/dev/patchwork/patch/9833/

The solution is that: PF reset interrupt will be captured by DPDK VF, then DPDK
Call the application's callback function to reset the VF device. For i40e device,
application need to detach and re-attach the device.

For ixgbe, I think the solution would be similar.

Thanks
Jingjing

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of David Marchand
> Sent: Friday, December 11, 2015 10:47 PM
> To: Zhang, Helin; Ananyev, Konstantin
> Cc: dev@dpdk.org
> Subject: [dpdk-dev] ixgbevf does not recover from pf reset
> 
> Hello Helin, Konstantin,
> 
> I noticed this issue quite some time ago (maybe around dpdk-1.3.0, not
> sure) but had no time to investigate/report.
> I hit it again with dpdk-2.2.0-rc2, so maybe all dpdk versions are impacted.
> 
> The test is quite simple :
> - send icmp packets from a system (tn) to itself using two 10G interfaces
> connected to two VF ports of a dual port 82599 nic,
> - run testpmd in a vm owning those two VF ports,
> - reset a pf interface in the host,
> - then no connectivity on the associated VF port unless it is completely
> restarted.
> 
> In my previous test, I think I also unplugged the fiber then replugged
> (without touching the pf interface status on the host) and observed the
> same issue.
> 
> I looked at ixgbevf pmd, but I can't find a place where the PF asking for reset
> would be handled except at devinit.
> 
> 
> 
> Here is the full description of my test :
> 
> The system owning the two VF ports is a ubuntu 14.04 virtual machine
> (3.13.0-71-generic kernel).
> The host running this virtual machine is running ubuntu 14.04 as well (but
> kernel is newer 3.19.0-39-generic).
> 
> 
> - host setup :
> echo 3072
> >/sys/devices/system/node/node0/hugepages/hugepages-
> 2048kB/nr_hugepages
> echo 3072
> >/sys/devices/system/node/node1/hugepages/hugepages-
> 2048kB/nr_hugepages
> mkdir -p
> /mnt/huge
> mount -t hugetlbfs none /mnt/huge
> 
> echo 1 >
> "/sys/bus/pci/devices/0000:83:00.0/sriov_numvfs"
> bridge link set dev ixgbe0 hwmode veb
> ip link set dev ixgbe0 vf 0 mac 00:09:c0:0e:4e:64 ip link set dev ixgbe0 vf 0
> spoofchk off ip link set dev ixgbe0 mtu 9000 ip link set dev ixgbe0 up ip link
> set dev ixgbe0 address 00:09:c0:0e:4e:64
> 
> echo 1 >
> "/sys/bus/pci/devices/0000:83:00.1/sriov_numvfs"
> bridge link set dev ixgbe1 hwmode veb
> ip link set dev ixgbe1 vf 0 mac 00:09:c0:0e:4e:65 ip link set dev ixgbe1 vf 0
> spoofchk off ip link set dev ixgbe1 mtu 9000 ip link set dev ixgbe1 up ip link
> set dev ixgbe1 address 00:09:c0:0e:4e:65
> 
> qemu-system-x86_64 -k fr -daemonize -S --enable-kvm -m 3G -mem-path
> /mnt/huge -cpu host -smp cores=6,threads=2,sockets=1 -serial
> telnet::53714,server,nowait -serial null -monitor telnet::36576,server,nowait
> -device
> virtio-net,mac=DE:AD:DE:01:02:03,netdev=user.0,addr=03 -netdev
> user,id=user.0 -device pci-assign,host=0000:83:10.0,addr=04 -device
> pci-assign,host=0000:83:10.1,addr=05 -hda ubuntu-14.04-template.qcow2 -
> snapshot -vga none -display none
> 
> 
> - guest setup :
> echo 1024 >
> /sys/devices/system/node/node0/hugepages/hugepages-
> 2048kB/nr_hugepages
> mkdir -p
> /mnt/huge
> mount -t hugetlbfs none /mnt/huge
> 
> modprobe uio
> modprobe igb_uio
> echo 0000:00:04.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind
> echo 0000:00:05.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind
> echo 8086 10ed > /sys/bus/pci/drivers/igb_uio/new_id
> 
> testpmd -w0000:00:04.0 -w0000:00:05.0 --huge-dir=/mnt/huge -n 4 -c 0xe -- -i
> [snip] Port 0:
> 00:09:C0:0E:4E:64
> [snip]
> Port 1:
> 00:09:C0:0E:4E:65
> Checking link
> statuses...
> Port 0 Link Up - speed 10000 Mbps -
> full-duplex
> Port 1 Link Up - speed 10000 Mbps -
> full-duplex
> Done
> 
> testpmd>
> start
> 
> 
> - tn setup :
> ip link set dev xaui0 up
> ip link set dev xaui1 up
> ip addr add 1.1.1.1/24 dev xaui0
> arp -i xaui0 -s 1.1.1.2 00:09:c0:0e:4e:64
> 
> 
> 
> - From here, on tn, send icmp packets with a tcpdump running background
> on the other iface :
> ping 1.1.1.2
> PING 1.1.1.2 (1.1.1.2): 56 data
> bytes
> 03:10:15.003047 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
> (0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 1
> 03:10:16.003151 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
> (0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 2
> 03:10:17.003246 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4
> (0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 3
> 
> 
> - Ok, now, kick ixgbe0 pf interface down / up on host system.
> ip link set dev ixgbe0 down
> ip link set dev ixgbe0 up
> 
> 
> - On tn, no packet is received anymore, the only workaround I found is to
> stop and restart the port 0 :
> 
> testpmd>
> stop
> [snip]
> Done.
> 
> testpmd> port stop
> all
> Stopping
> ports...
> Checking link
> statuses...
> Port 0 Link
> Down
> Port 1 Link Up - speed 10000 Mbps -
> full-duplex
> Done
> 
> 
> Funny thing here, the link is reported down, while the pf is already back up.
> 
> testpmd> port start all
> PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
> size no less than 4 (port=0).
> Port 0:
> 00:09:C0:0E:4E:64
> PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
> size no less than 4 (port=1).
> Port 1:
> 00:09:C0:0E:4E:65
> Checking link
> statuses...
> Port 0 Link Up - speed 10000 Mbps -
> full-duplex
> Port 1 Link Up - speed 10000 Mbps -
> full-duplex
> Done
> 
> 
> 
> Thanks.
> 
> --
> David Marchand

Re: [dpdk-dev] ixgbevf does not recover from pf reset

[David Marchand]

Hello,

On Fri, Jan 22, 2016 at 3:05 AM, Wu, Jingjing <jingjing.wu@intel.com> wrote:
> Hi, David
>
> We also noticed this issue before. And we are planning to fix this issue.
> And the patch for i40e is ready:
> http://dpdk.org/dev/patchwork/patch/9832/
> http://dpdk.org/dev/patchwork/patch/9833/
>
> The solution is that: PF reset interrupt will be captured by DPDK VF, then DPDK
> Call the application's callback function to reset the VF device. For i40e device,
> application need to detach and re-attach the device.
>
> For ixgbe, I think the solution would be similar.

Ok, so to handle a problem in hardware, the application must implement
a new mechanism to recover the port ?
I don't find this solution really elegant ...

This should be handled by the pmd itself.


Regards,
-- 
David Marchand

Re: [dpdk-dev] ixgbevf does not recover from pf reset

[Wu, Jingjing]

Hi

Yes. But just consider about how to use DPDK on eth device. During bring up we need:
eth_dev_init
rte_pktmbuf_pool_create
rte_eth_dev_configure
rte_eth_rx_queue_setup
rte_eth_tx_queue_setup
rte_eth_dev_start

All above behavior need to be called by application, pmd driver cannot did all the
Recovery without application's help.
By the simple notification mechanism, application can simple call following step to recovery:

Stop forwarding on this device (rx, tx)
rte_eth_dev_close
rte_eth_dev_detach
rte_eth_dev_attach
rte_eth_dev_configure
rte_eth_rx_queue_setup
rte_eth_tx_queue_setup
rte_eth_dev_start

Any idea is welcome. We are eager to hear better solution on that.
Anyway, no matter whether the recovery is done by application, the patches mentioned
Is also useful to notice user about the even. Correct?

Thanks
Jingjing

> -----Original Message-----
> From: David Marchand [mailto:david.marchand@6wind.com]
> Sent: Friday, January 22, 2016 4:54 PM
> To: Wu, Jingjing
> Cc: Zhang, Helin; Ananyev, Konstantin; Lu, Wenzhuo; dev@dpdk.org
> Subject: Re: [dpdk-dev] ixgbevf does not recover from pf reset
> 
> Hello,
> 
> On Fri, Jan 22, 2016 at 3:05 AM, Wu, Jingjing <jingjing.wu@intel.com> wrote:
> > Hi, David
> >
> > We also noticed this issue before. And we are planning to fix this issue.
> > And the patch for i40e is ready:
> > http://dpdk.org/dev/patchwork/patch/9832/
> > http://dpdk.org/dev/patchwork/patch/9833/
> >
> > The solution is that: PF reset interrupt will be captured by DPDK VF,
> > then DPDK Call the application's callback function to reset the VF
> > device. For i40e device, application need to detach and re-attach the device.
> >
> > For ixgbe, I think the solution would be similar.
> 
> Ok, so to handle a problem in hardware, the application must implement a
> new mechanism to recover the port ?
> I don't find this solution really elegant ...
> 
> This should be handled by the pmd itself.
> 
> 
> Regards,
> --
> David Marchand

Re: [dpdk-dev] ixgbevf does not recover from pf reset

[David Marchand]

Hello Jingjing,

On Mon, Jan 25, 2016 at 2:58 AM, Wu, Jingjing <jingjing.wu@intel.com> wrote:
> Hi
>
> Yes. But just consider about how to use DPDK on eth device. During bring up we need:
> eth_dev_init
> rte_pktmbuf_pool_create
> rte_eth_dev_configure
> rte_eth_rx_queue_setup
> rte_eth_tx_queue_setup
> rte_eth_dev_start
>
> All above behavior need to be called by application, pmd driver cannot did all the
> Recovery without application's help.

Well, that is if the only solution is to restart the port from the application.
The linux i40e vf driver handles this kind of event itself.
Could we have something similar ?


Regards,
-- 
David Marchand