[dpdk-dev] [Bug 372] ixgbevf: cannot probe or start port if PF link is unstable

[dpdk-dev] [Bug 372] ixgbevf: cannot probe or start port if PF link is unstable

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=372

            Bug ID: 372
           Summary: ixgbevf: cannot probe or start port if PF link is
                    unstable
           Product: DPDK
           Version: unspecified
          Hardware: All
                OS: All
            Status: UNCONFIRMED
          Severity: major
          Priority: Normal
         Component: ethdev
          Assignee: dev@dpdk.org
          Reporter: julien.meunier@nokia.com
  Target Milestone: ---

Created attachment 76
  --> https://bugs.dpdk.org/attachment.cgi?id=76&action=edit
ixgbevf: debug mbx during init

Configuration
=============
Kernel: 4.19
NIC: Intel Niantic X550
PF driver: Kernel ixgbe v.5.1.0
SRIO-V: enabled
DPDK version: v18.05 - v19.11-rc4

Detail Test Steps
=================
- Set PF up::
ip link set dev eth0 up

- Export 16 VF::
echo 16 > /sys/class/net/eth0/device/sriov_numvfs 

- Configure MAC of the VF::
for i in {0..16}; do ip link set dev eth0 vf $i mac 00:11:22:33:44:${i}; done

- Bind it to igb_uio (example: 04:10.0 ...)::
dpdk-devbind -b igb_uio 04:10.0 04:10.2 04:10.4 04:10.6 04:11.0 \
04:11.2 04:11.4 04:11.6 04:12.0 04:12.2 04:12.4 04:12.6 04:13.0 \
04:13.2 04:13.4 04:13.6 

- PF is UP, but physical link changes status each seconds (UP / DOWN)

- Start a DPDK application (example: testpmd) when link is unstable::
testpmd -w 04:10.0 -w 04:10.2 -w 04:10.4 -w 04:10.6 \
-w 04:11.0 -w 04:11.2 -w 04:11.4 -w 04:11.6 \
-w 04:12.0 -w 04:12.2 -w 04:12.4 -w 04:12.6 \
-w 04:13.0 -w 04:13.2 -w 04:13.4 -w 04:13.6 -- -i --no-mlockall

Expected Result
===============
No issue during EAL init or during rte_eth_dev_start when link changes its
status.

Actual Result
=============
Issue during the EAL init::
        EAL: PCI device 0000:04:10.0 on NUMA socket 0
        EAL:   probe driver: 8086:15a8 net_ixgbe_vf
        EAL: Requested device 0000:04:10.0 cannot be used

Issue during the rte_eth_dev_start::
        ixgbevf_dev_start(): Unable to reset vf hardware (-10)

No issue if link status is stable during the init

Workaround
==========
- Application has to probe again the same device
- Application has to start again the same port

Analysis
========
I added some traces in ixgbe_reset_hw_vf in order to dump the content of the
mailbox. (see attachment)

When VF is correctly initialized: 
        ixgbe_reset_hw_vf: jmeunier - msgbuf[0] = 80000001
                => (IXGBE_VF_RESET | IXGBE_VT_MSGTYPE_ACK)

When VF is NOT correctly initialized:
        ixgbe_reset_hw_vf: jmeunier - msgbuf[0] = 20000100
                => (IXGBE_VT_MSGTYPE_CTS | IXGBE_PF_CONTROL_MSG)

An invalid message is received in the mailbox during the initialization.
This message seems to be sent by the PF to notify that the link has changed its
status.

According to the current implementation (on Kernel side), when link status
changes, PF notifies
all associated VFs through the mailbox::
        ixgbe_watchdog_link_is_up => ixgbe_ping_all_vfs => ixgbe_write_mbx(hw,
&ping, 1, i)

So, if VF waits for one specific message, if link changes at the same time, VF
will read an invalid message.
In this PR, it's IXGBE_VF_RESET, but I can see others cases like that in the
current PF/VF implementation.

BTW, I suspect that Kernel overwrites the mailbox message (to be confirmed)

-- 
You are receiving this mail because:
You are the assignee for the bug.