DPDK usage discussions
 help / color / mirror / Atom feed
From: "Zhang, Qi Z" <qi.z.zhang@intel.com>
To: "Bly, Mike" <mbly@ciena.com>, "dev@dpdk.org" <dev@dpdk.org>,
	"users@dpdk.org" <users@dpdk.org>
Subject: RE: IXGBE LSC IRQ issue
Date: Sun, 13 Mar 2022 12:22:31 +0000
Message-ID: <b42a7220432b4e19af6971ea38b0af09@intel.com> (raw)
In-Reply-To: <BYAPR04MB4325C5B6EE71A0562EBA365CCF069@BYAPR04MB4325.namprd04.prod.outlook.com>

[-- Attachment #1: Type: text/plain, Size: 7206 bytes --]

this is not a patch, but an issue report, suggest to fire a ticket on Bugzilla https://bugs.dpdk.org/


From: Bly, Mike <mbly@ciena.com>
Sent: Saturday, March 5, 2022 9:42 AM
To: dev@dpdk.org; users@dpdk.org
Subject: IXGBE LSC IRQ issue

Hello,

We recently ran into an issue with DPDK 20.11 for the IXGBE driver operating in 10G BASE-T mode. We have been able to replicate this behavior using dpdk-testpmd and do not see any recent/pertinent updates, so we are hopeful someone may be able to advise based on the information provided below. On the surface, based on our investigation, it would appear the current link-down transition logic does not correctly preserve IRQ mask configurations, specifically LSC, when a link partner causes some sort of slow or bounced link down event.
Background:
We recently started using a new 3rd party traffic generator card for testing our application. We found when using this card in 10G BASE-T mode and toggling link up/down, it would correctly cause our application to detect the port to be down in our DPDK design. However, the link down event handling by the DPDK IXGBE driver appears to permanently disable its LSC IRQ detection on the first port down event such that any subsequent link up or down events from the external test card on this port would no longer be detected. The only way to restore link up was to restart the DPDK port in our design (stop/start). Having looked at this a bit, we switched over to the classic testpmd application and observed the exact same behavior.

Here is the data we believe you would find interesting:

NIC in question:

# lspci -D -nn | grep -F [0200] | grep 552
0000:03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
0000:03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
# dpdk-devbind.py -s | grep 552
0000:03:00.0 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' drv=vfio-pci unused=uio_pci_generic
0000:03:00.1 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' drv=vfio-pci unused=uio_pci_generic

We made the following debug logging changes to try an capture interesting data to share:

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 5a30c39593..75a9f9163b 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -4497,7 +4497,7 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *dev)

     /* read-on-clear nic registers here */
    eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
-    PMD_DRV_LOG(DEBUG, "eicr %x", eicr);
+    PMD_DRV_LOG(ERR, "eicr %x", eicr);

     intr->flags = 0;

@@ -4614,7 +4613,7 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev)
          }
    }

-    PMD_DRV_LOG(DEBUG, "enable intr immediately");
+    PMD_DRV_LOG(ERR, "enable intr immediately, mask: 0x%08x, orig: 0x%08x, flags: 0x%08x", intr->mask, intr->mask_original, intr->flags);
    ixgbe_enable_intr(dev);

     return 0;
@@ -4648,7 +4647,9 @@ ixgbe_dev_interrupt_delayed_handler(void *param)

     ixgbe_disable_intr(hw);

-    eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
+   eicr = IXGBE_READ_REG(hw, IXGBE_EICR);
+   PMD_DRV_LOG(ERR, "in delay func: eicr 0x%08x", eicr);
+   PMD_DRV_LOG(ERR, "enable intr delayed, mask: 0x%08x, orig: 0x%08x, flags: 0x%08x", intr->mask, intr->mask_original, intr->flags);
    if (eicr & IXGBE_EICR_MAILBOX)
          ixgbe_pf_mbx_process(dev);

With the above "log-err" additions, we have provided the following results. The first set of data below was generated using an older 3rd party traffic generator card to provide "good" results that show the IXGBE driver working correctly. Following that are the non-working (bad) logging results for the new traffic generator card. Both 3rd party cards correctly transition between down and up states.


######################################################################
# good sequence, both down detection and then up detection
######################################################################
# port transition from up to down
<27>1 2022-03-05T00:12:11.415436+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 100000
<27>1 2022-03-05T00:12:11.415489+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x00000001
<27>1 2022-03-05T00:12:11.425448+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 2000000
<27>1 2022-03-05T00:12:11.446191+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x00000000
<27>1 2022-03-05T00:12:15.415600+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): in delay func: eicr 0x00000000
<27>1 2022-03-05T00:12:15.415655+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x02300000, flags: 0x00000000

# port transition from down to up
<27>1 2022-03-05T00:12:33.856734+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 2000000
<27>1 2022-03-05T00:12:33.877463+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02300000, orig: 0x00000000, flags: 0x00000000
<27>1 2022-03-05T00:12:34.203274+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 100000
<27>1 2022-03-05T00:12:34.207905+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x00000001
<27>1 2022-03-05T00:12:35.207994+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): in delay func: eicr 0x00100000
<27>1 2022-03-05T00:12:35.208027+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x02300000, flags: 0x00000001

######################################################################
# bad sequence, detects down event, but does not see the up event
######################################################################
# port transition from up to down
<27>1 2022-03-05T00:13:00.377072+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 100000
<27>1 2022-03-05T00:13:00.377127+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x00000001
<27>1 2022-03-05T00:13:00.643788+00:00 - -  ixgbe_dev_interrupt_get_status(): eicr 2100000
<27>1 2022-03-05T00:13:00.664603+00:00 - -  ixgbe_dev_interrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02200000, flags: 0x00000001
<27>1 2022-03-05T00:13:01.664703+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): in delay func: eicr 0x00000000
<27>1 2022-03-05T00:13:01.664738+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x02200000, flags: 0x00000001
<27>1 2022-03-05T00:13:04.377237+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): in delay func: eicr 0x00000000
<27>1 2022-03-05T00:13:04.377269+00:00 - -  ixgbe_dev_interrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x00000000, flags: 0x00000000

# port transition from down to up
<nothing happens as LSC IRQ is not enabled due to above link-down sequence>

Let me know what additional data can be provided to help root cause this.

-Mike


[-- Attachment #2: Type: text/html, Size: 19832 bytes --]

      reply	other threads:[~2022-03-13 12:22 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-05  1:42 Bly, Mike
2022-03-13 12:22 ` Zhang, Qi Z [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=b42a7220432b4e19af6971ea38b0af09@intel.com \
    --to=qi.z.zhang@intel.com \
    --cc=dev@dpdk.org \
    --cc=mbly@ciena.com \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK usage discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git