IXGBE LSC IRQ issue

["Bly, Mike" <mbly@ciena.com>]

[-- Attachment #1: Type: text/plain, Size: 6948 bytes --]

Hello,

We recently ran into an issue with DPDK 20.11 for the IXGBE driver operating in 10G BASE-T mode. We have been able to replicate this behavior using dpdk-testpmd and do not see any recent/pertinent updates, so we are hopeful someone may be able to advise based on the information provided below. On the surface, based on our investigation, it would appear the current link-down transition logic does not correctly preserve IRQ mask configurations, specifically LSC, when a link partner causes some sort of slow or bounced link down event.

Background:
We recently started using a new 3rd party traffic generator card for testing our application. We found when using this card in 10G BASE-T mode and toggling link up/down, it would correctly cause our application to detect the port to be down in our DPDK design. However, the link down event handling by the DPDK IXGBE driver appears to permanently disable its LSC IRQ detection on the first port down event such that any subsequent link up or down events from the external test card on this port would no longer be detected. The only way to restore link up was to restart the DPDK port in our design (stop/start). Having looked at this a bit, we switched over to the classic testpmd application and observed the exact same behavior.

Here is the data we believe you would find interesting:

NIC in question:

# lspci -D -nn | grep -F [0200] | grep 552
0000:03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
0000:03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
# dpdk-devbind.py -s | grep 552
0000:03:00.0 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' drv=vfio-pci unused=uio_pci_generic
0000:03:00.1 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' drv=vfio-pci unused=uio_pci_generic

We made the following debug logging changes to try an capture interesting data to share:

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 5a30c39593..75a9f9163b 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -4497,7 +4497,7 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)
     /* read-on-clear nic registers here */
    eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
-    PMD_DRV_LOG(DEBUG, "eicr %x", eicr);
+    PMD_DRV_LOG(ERR, "eicr %x", eicr);
     intr->flags = 0;
@@ -4614,7 +4613,7 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
          }
    }
-    PMD_DRV_LOG(DEBUG, "enable intr immediately");
+    PMD_DRV_LOG(ERR, "enable intr immediately, mask: 0x%08x, orig: 0x%08x, flags: 0x%08x", intr->mask, intr->mask_original, intr->flags);
    ixgbe_enable_intr(dev);
     return 0;
@@ -4648,7 +4647,9 @@ ixgbe_dev_interrupt_delayed_handler(void *param)
     ixgbe_disable_intr(hw);
-    eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
+   eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
+   PMD_DRV_LOG(ERR, "in delay func: eicr 0x%08x", eicr);
+   PMD_DRV_LOG(ERR, "enable intr delayed, mask: 0x%08x, orig: 0x%08x, flags: 0x%08x", intr->mask, intr->mask_original, intr->flags);
    if (eicr & IXGBE_EICR_MAILBOX)
          ixgbe_pf_mbx_process(dev);

With the above "log-err" additions, we have provided the following results. The first set of data below was generated using an older 3rd party traffic generator card to provide "good" results that show the IXGBE driver working correctly. Following that are the non-working (bad) logging results for the new traffic generator card. Both 3rd party cards correctly transition between down and up states.


######################################################################
# good sequence, both down detection and then up detection
######################################################################
# port transition from up to down
<27>1 2022-03-05T00:12:11.415436+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 100000
<27>1 2022-03-05T00:12:11.415489+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x00000001
<27>1 2022-03-05T00:12:11.425448+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 2000000
<27>1 2022-03-05T00:12:11.446191+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x00000000
<27>1 2022-03-05T00:12:15.415600+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): in delay func: eicr 0x00000000
<27>1 2022-03-05T00:12:15.415655+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x02300000, flags: 0x00000000

# port transition from down to up
<27>1 2022-03-05T00:12:33.856734+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 2000000
<27>1 2022-03-05T00:12:33.877463+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02300000, orig: 0x00000000, flags: 0x00000000
<27>1 2022-03-05T00:12:34.203274+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 100000
<27>1 2022-03-05T00:12:34.207905+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x00000001
<27>1 2022-03-05T00:12:35.207994+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): in delay func: eicr 0x00100000
<27>1 2022-03-05T00:12:35.208027+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x02300000, flags: 0x00000001

######################################################################
# bad sequence, detects down event, but does not see the up event
######################################################################
# port transition from up to down
<27>1 2022-03-05T00:13:00.377072+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 100000
<27>1 2022-03-05T00:13:00.377127+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x00000001
<27>1 2022-03-05T00:13:00.643788+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 2100000
<27>1 2022-03-05T00:13:00.664603+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02200000, flags: 0x00000001
<27>1 2022-03-05T00:13:01.664703+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): in delay func: eicr 0x00000000
<27>1 2022-03-05T00:13:01.664738+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x02200000, flags: 0x00000001
<27>1 2022-03-05T00:13:04.377237+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): in delay func: eicr 0x00000000
<27>1 2022-03-05T00:13:04.377269+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x00000000, flags: 0x00000000

# port transition from down to up
<nothing happens as LSC IRQ is not enabled due to above link-down sequence>

Let me know what additional data can be provided to help root cause this.

-Mike


[-- Attachment #2: Type: text/html, Size: 17071 bytes --]