From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E58A6A00C4 for ; Sat, 5 Mar 2022 02:42:24 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8A19942791; Sat, 5 Mar 2022 02:42:18 +0100 (CET) Received: from mx0b-00103a01.pphosted.com (mx0b-00103a01.pphosted.com [67.231.152.227]) by mails.dpdk.org (Postfix) with ESMTP id 8E5BF40042; Sat, 5 Mar 2022 02:42:15 +0100 (CET) Received: from pps.filterd (m0222748.ppops.net [127.0.0.1]) by mx0a-00103a01.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 224Np6mr002557; Fri, 4 Mar 2022 20:42:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ciena.com; h=from : to : subject : date : message-id : content-type : mime-version; s=06252019; bh=9BBX2cDLWvX4t9w7C4NiQm9J4d71kEOQxuHY9NZQLjs=; b=sdXVHDVgHyBRyki8sNYNlOO27XGGtZ1Sx0vINC08YI4mGyzGOXFiJQ5CJ76b3ccy+51W jV4xxOUCzg0zqLvU+85GPAkRFMSJCYEyL2QcB5IWD8JObeg9uF27C6Wm1XABZTMyT1AP TFSSvqNpT5U+L1Ikpp822G+00/78CXMnRD6Vlnxcm5N0r5elEQ6wr7T3rwnoeVbK8FQl IBTdsKiwRzE0yXjSnawAoNy8uEt5LPRBK0hM1LwOmcmDwf7HsYTFT+8XLdd9vqfgLuyR 7A9zUAwVlkhK1e8yJ4CEwzrgUtbZD0TXWj1/7g/8rKm/hswvo9zqVV7iJO0M9fa80HYm iw== Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2172.outbound.protection.outlook.com [104.47.58.172]) by mx0a-00103a01.pphosted.com (PPS) with ESMTPS id 3ek4jmupyk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 04 Mar 2022 20:42:14 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=k5AIPfML07YeSiDjiRVYRKoNyEgZwMw2JrtN1hTZonyh/8vaguE5onpy/qrShTvevuMT3HdxpGXKvGbTePZTIDvmehGboPR4AV5qC1ieNF/n9QUQPFAL4KMt+YuaSpb/v7b7qA6o/w8+0fxDSVGk2PBeNgFzTKMwt1Gjsvu7pDuDsDG7SGlZQKblWGXQTaIIvJt7cfxi8bGgVulwwZ2esDvub+uV0TqNdr92mMQkg6tDHcMv33O2JuAZbPVBNJ7v26bN1esVlBeduILe4AgT46KkOmvnRHVDyGPyPc5AlIMCeS4nw1Z1FWiXu6hqs1GYWykBtvd/1dgUxSymVjxvJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9BBX2cDLWvX4t9w7C4NiQm9J4d71kEOQxuHY9NZQLjs=; b=V/WV8j4VkV2ozSGEVL5uHnmpoXM3lbUDEaore1BGKSixsLJHu2VQLvN129v5AmhjK12gUWdV56M7tyDYU6i5myqwqB3voBoVLdfdQAfVYxyJnGsFvz+/PmIhBThmFxRLySgwFyljGj1VJvRDtRj8Xod3aEvGJPkE9qQTDqiL4bKQK+SkWwaLxXevtjquzOojwYO2GYvQhUXYoIB1QcBXRftEuIeq3n/s0Oa3h6Jvt1w3Q2uzLCq1j41M7qqiGjjg2kmnrQlF+I43Pr3FNzZGRmmK24Z5aI2GWZ+U5KSCwpwScDUUN+ejAAIU5UUHXCxZ73Exa+K9vcTc1cs/ePYwvQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ciena.com; dmarc=pass action=none header.from=ciena.com; dkim=pass header.d=ciena.com; arc=none Received: from BYAPR04MB4325.namprd04.prod.outlook.com (2603:10b6:a02:fe::33) by DM8PR04MB7927.namprd04.prod.outlook.com (2603:10b6:8:3::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Sat, 5 Mar 2022 01:42:12 +0000 Received: from BYAPR04MB4325.namprd04.prod.outlook.com ([fe80::3d12:4680:48d6:9145]) by BYAPR04MB4325.namprd04.prod.outlook.com ([fe80::3d12:4680:48d6:9145%6]) with mapi id 15.20.5017.031; Sat, 5 Mar 2022 01:42:12 +0000 From: "Bly, Mike" To: "dev@dpdk.org" , "users@dpdk.org" Subject: IXGBE LSC IRQ issue Thread-Topic: IXGBE LSC IRQ issue Thread-Index: AdgwMjt3Aw43DWuZSvGGTkLNbT/dKA== Date: Sat, 5 Mar 2022 01:42:12 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 35168ef2-ad0b-4874-b8d7-08d9fe495eb8 x-ms-traffictypediagnostic: DM8PR04MB7927:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 46HhxsGWBRXpEy9Vgz+OY0ZH/qnXoUbU3zc5Q/k6/DJznKZG3KcZoVBLnCIXrdSKxputyX0Qs2uvGITNSl9mVzvAIM/XyTcY0mNW60lC/X+pdn0IjOW1rJtp+x3KMUXqqRqoQBg9TUfeep38GUlIQoSMlYey1Q3jP75xinwM4sYqIl5FgB/dLm0AEvwmzs6UHUleV7GL+3oCPCrAnosG55qUVkAPF4YKWINyDQmRysC188d9OZ3I/iLIX8JeWoPj+pzCaILplb/nRZ4yjiWYM2O8c/4CL1+NVW7AaqRhPWXi9bIedUuz5qUqqjQegknpMrjx4aQD1kYbP20UKvjgtT++9jX2rM1UgNo7KhEYaeoYf0u8M1cgVuNCs5KfM8ICKsxQzxPheJS2rPdAfA3by0KDcOL8fY7nj1WqiPV7DV0WxRoyByNmDx4BhiB9qTQn+onRYV+L9O5+Tu3Ymtl4vs36/6HNpd2T3d35JiouCZmx7iJBiYXx79tkpN20uDC660MSB9AS9UD6ZOlowbi2qS+e1rbrEJ0x6W/YhTuwkOEmBkmZDMe0SHSFDyH3AlHBTLD76sn62ux9SGiDNDR6fvBRSEFMfmQlAZjuU8NQJs7/EMlxa9l48ryjCm+2abbam7DJ5y6WhbfPak0yFi2RJKbNA5SiwHhJqE4JdoXlmQqcLP4AXWMH+PmPG/j1yoc2V8QMFXt0tWl20ZrnV/2YcQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR04MB4325.namprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(83380400001)(38100700002)(122000001)(9686003)(71200400001)(450100002)(52536014)(9326002)(66556008)(64756008)(8936002)(3480700007)(8676002)(66446008)(66946007)(66476007)(316002)(508600001)(5660300002)(2906002)(76116006)(33656002)(186003)(110136005)(7696005)(6506007)(26005)(38070700005)(55016003)(86362001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?/2FATf9iTjGsTdkCV+7h1WJzudxCNtZxao/B3HOf437+hllvoY/1UtjDWfL+?= =?us-ascii?Q?I8tBkibvGZNBUFbsayy+mL6xZvrhfpKAeI1ZcuJrCwAqfDY410qRcUXAtWal?= =?us-ascii?Q?1GPu8DzJPDcTfdT/o85vgKOP45qxshwT15guPMmYQEZLXXlpse15msTHGn55?= =?us-ascii?Q?xuYxxeUJmejyORxwBu1ILxsw9A4ZGn8+XoOC6zXWR4YzG0xflwnetvl6EdBj?= =?us-ascii?Q?KooRag+8G6Gih+wwPzoS9iwzo/7swu4Nr0N/5wOJHWZLjpLfaHPLuQ1FSAmd?= =?us-ascii?Q?odie0Zj6FhIMEB9iZDk1M4Jg9kCS9A0lVBQ3IcPJWCsbbmk4RSnfR7yQLkzK?= =?us-ascii?Q?l12UrgW73QuIG3vBDPS9O8jSgxLsbLksNgikWe9F5DCKi/o4VWe8Uc/rd/lJ?= =?us-ascii?Q?fDvc5zlkh9AHgllFzEvgwlBPra6qhvzz7RHKgNvbhvQ1LCLcr13/fifg/9h6?= =?us-ascii?Q?S/ZK0lkcboxoXqlS3/hVDSNbuCcjH2wJGfHdC0HKN0QVTdN5GaRCtLpKuDYC?= =?us-ascii?Q?vDDT99B4s0CMt6InZIYl55hX0ZXRqAbz+8dAvIdVMCfuwYmpRKUOHk3eEpsu?= =?us-ascii?Q?y9jwSzX7bJ9Nd8t9vvcrnjrjscUo/9xzTXzG4HaiCaO/D/iWku32HA7MAcgP?= =?us-ascii?Q?no+FSkNinRjFeYBkIMcsdoT+t6Jsdyz1EyTWiw35MpbuegFEdYfqMPRoyzIX?= =?us-ascii?Q?kZP8B3K31r7jWiJxGOBADUgjW/uYHHz/brHPVqKUMZWvWZst4yiIFkiyksXP?= =?us-ascii?Q?StXhgf8o5hGpsSk+v38VNN5aOWzYJ0ZY5/sKiZPodZAQ3V45e/yJcaWNnROI?= =?us-ascii?Q?5Q3POs8s/kG3aysV28z8158yBBUgEDfoAtyLnsRBItfvgj6jzfOhvrGnGGs6?= =?us-ascii?Q?DaprZBIuuqgjLKVq935+dBp7uwrcNrbuU9E2GxQw6ypB7afGLhGrzGisfvwo?= =?us-ascii?Q?WzhGBX20mAEjhTP4Bfk8v1Uwwj3EseTnSwOJgziLu8gVH5CXZdtqH9CMo7mi?= =?us-ascii?Q?xbtApQWZnwZTLsU7fn6cQQmB4gcgyWZodrdwR60mWjAG+R8tGw3jnA/0HLui?= =?us-ascii?Q?CBoxG++O/6hU8cwOaEGtdE4qiY+iKXX7irSJyKyXVsmGoVy0Yyl4XB5MXq3a?= =?us-ascii?Q?uFjK2xBqyFJouMqzwV5vDzim2ZzzClRpV2o0VB5Ob28jgf6symy2kJZ+CBVl?= =?us-ascii?Q?pn7gQfEGgvjR4ZY8/51rNhN1hp0Z4UNuVBaNfnPHV7VSGtZJoz++vvwyZWwN?= =?us-ascii?Q?mP2jY9ylGnEI8MmZHiUjS4M1XqfoknQ+1rok4Dt4YRZcUh/63XF5MH1apr2M?= =?us-ascii?Q?ekJVZjhsjO2qFb4JxzhlZhSRmg8hr6veqRh90REmI9Xw23UI2yZXMKpXblL7?= =?us-ascii?Q?CP4zA04NaxZjwWnZ1MIhxul2Ta0YDv/N1I64PAjSw/n/eBCQD2xOwrO2Uqx2?= =?us-ascii?Q?wUnVqoZwW5xieoeKo72pBXA9KCowKe8c80eCqpGuDxU79r4S7cygvLgZyjYc?= =?us-ascii?Q?GyEDPgsVbwm0N9g9SBuH3RYQxSsUnbqnl3RWunm5D2rpwosdvnHalXt8jGLv?= =?us-ascii?Q?xEDUL1SxybyQQNznmSo=3D?= Content-Type: multipart/alternative; boundary="_000_BYAPR04MB4325C5B6EE71A0562EBA365CCF069BYAPR04MB4325namp_" MIME-Version: 1.0 X-OriginatorOrg: ciena.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR04MB4325.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 35168ef2-ad0b-4874-b8d7-08d9fe495eb8 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Mar 2022 01:42:12.4242 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 457a2b01-0019-42ba-a449-45f99e96b60a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: M6bUNb9+wUQA5UeZAUlUueuR5HCB8LqvQbM62sm/OYChwX08NxM80PsZlM6xQJSa X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR04MB7927 X-Proofpoint-GUID: oDEKuM-z1RL6Z9y8dnyubdiVn2Rak7Sn X-Proofpoint-ORIG-GUID: oDEKuM-z1RL6Z9y8dnyubdiVn2Rak7Sn X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-04_09,2022-03-04_01,2022-02-23_01 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_000_BYAPR04MB4325C5B6EE71A0562EBA365CCF069BYAPR04MB4325namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, We recently ran into an issue with DPDK 20.11 for the IXGBE driver operatin= g in 10G BASE-T mode. We have been able to replicate this behavior using dp= dk-testpmd and do not see any recent/pertinent updates, so we are hopeful s= omeone may be able to advise based on the information provided below. On th= e surface, based on our investigation, it would appear the current link-dow= n transition logic does not correctly preserve IRQ mask configurations, spe= cifically LSC, when a link partner causes some sort of slow or bounced link= down event. Background: We recently started using a new 3rd party traffic generator card for testin= g our application. We found when using this card in 10G BASE-T mode and tog= gling link up/down, it would correctly cause our application to detect the = port to be down in our DPDK design. However, the link down event handling b= y the DPDK IXGBE driver appears to permanently disable its LSC IRQ detectio= n on the first port down event such that any subsequent link up or down eve= nts from the external test card on this port would no longer be detected. T= he only way to restore link up was to restart the DPDK port in our design (= stop/start). Having looked at this a bit, we switched over to the classic t= estpmd application and observed the exact same behavior. Here is the data we believe you would find interesting: NIC in question: # lspci -D -nn | grep -F [0200] | grep 552 0000:03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connect= ion X552/X557-AT 10GBASE-T [8086:15ad] 0000:03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connect= ion X552/X557-AT 10GBASE-T [8086:15ad] # dpdk-devbind.py -s | grep 552 0000:03:00.0 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' drv=3Dvfio-p= ci unused=3Duio_pci_generic 0000:03:00.1 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' drv=3Dvfio-p= ci unused=3Duio_pci_generic We made the following debug logging changes to try an capture interesting d= ata to share: diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_eth= dev.c index 5a30c39593..75a9f9163b 100644 --- a/drivers/net/ixgbe/ixgbe_ethdev.c +++ b/drivers/net/ixgbe/ixgbe_ethdev.c @@ -4497,7 +4497,7 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_dev *de= v) /* read-on-clear nic registers here */ eicr =3D IXGBE_READ_REG(hw, IXGBE_EICR); - PMD_DRV_LOG(DEBUG, "eicr %x", eicr); + PMD_DRV_LOG(ERR, "eicr %x", eicr); intr->flags =3D 0; @@ -4614,7 +4613,7 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *dev) } } - PMD_DRV_LOG(DEBUG, "enable intr immediately"); + PMD_DRV_LOG(ERR, "enable intr immediately, mask: 0x%08x, orig: 0x%08x,= flags: 0x%08x", intr->mask, intr->mask_original, intr->flags); ixgbe_enable_intr(dev); return 0; @@ -4648,7 +4647,9 @@ ixgbe_dev_interrupt_delayed_handler(void *param) ixgbe_disable_intr(hw); - eicr =3D IXGBE_READ_REG(hw, IXGBE_EICR); + eicr =3D IXGBE_READ_REG(hw, IXGBE_EICR); + PMD_DRV_LOG(ERR, "in delay func: eicr 0x%08x", eicr); + PMD_DRV_LOG(ERR, "enable intr delayed, mask: 0x%08x, orig: 0x%08x, flag= s: 0x%08x", intr->mask, intr->mask_original, intr->flags); if (eicr & IXGBE_EICR_MAILBOX) ixgbe_pf_mbx_process(dev); With the above "log-err" additions, we have provided the following results.= The first set of data below was generated using an older 3rd party traffic= generator card to provide "good" results that show the IXGBE driver workin= g correctly. Following that are the non-working (bad) logging results for t= he new traffic generator card. Both 3rd party cards correctly transition be= tween down and up states. ###################################################################### # good sequence, both down detection and then up detection ###################################################################### # port transition from up to down <27>1 2022-03-05T00:12:11.415436+00:00 - - ixgbe_dev_interrupt_get_status(= ): eicr 100000 <27>1 2022-03-05T00:12:11.415489+00:00 - - ixgbe_dev_interrupt_action(): e= nable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x000000= 01 <27>1 2022-03-05T00:12:11.425448+00:00 - - ixgbe_dev_interrupt_get_status(= ): eicr 2000000 <27>1 2022-03-05T00:12:11.446191+00:00 - - ixgbe_dev_interrupt_action(): e= nable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x000000= 00 <27>1 2022-03-05T00:12:15.415600+00:00 - - ixgbe_dev_interrupt_delayed_han= dler(): in delay func: eicr 0x00000000 <27>1 2022-03-05T00:12:15.415655+00:00 - - ixgbe_dev_interrupt_delayed_han= dler(): enable intr delayed, mask: 0x02200000, orig: 0x02300000, flags: 0x0= 0000000 # port transition from down to up <27>1 2022-03-05T00:12:33.856734+00:00 - - ixgbe_dev_interrupt_get_status(= ): eicr 2000000 <27>1 2022-03-05T00:12:33.877463+00:00 - - ixgbe_dev_interrupt_action(): e= nable intr immediately, mask: 0x02300000, orig: 0x00000000, flags: 0x000000= 00 <27>1 2022-03-05T00:12:34.203274+00:00 - - ixgbe_dev_interrupt_get_status(= ): eicr 100000 <27>1 2022-03-05T00:12:34.207905+00:00 - - ixgbe_dev_interrupt_action(): e= nable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x000000= 01 <27>1 2022-03-05T00:12:35.207994+00:00 - - ixgbe_dev_interrupt_delayed_han= dler(): in delay func: eicr 0x00100000 <27>1 2022-03-05T00:12:35.208027+00:00 - - ixgbe_dev_interrupt_delayed_han= dler(): enable intr delayed, mask: 0x02200000, orig: 0x02300000, flags: 0x0= 0000001 ###################################################################### # bad sequence, detects down event, but does not see the up event ###################################################################### # port transition from up to down <27>1 2022-03-05T00:13:00.377072+00:00 - - ixgbe_dev_interrupt_get_status(= ): eicr 100000 <27>1 2022-03-05T00:13:00.377127+00:00 - - ixgbe_dev_interrupt_action(): e= nable intr immediately, mask: 0x02200000, orig: 0x02300000, flags: 0x000000= 01 <27>1 2022-03-05T00:13:00.643788+00:00 - - ixgbe_dev_interrupt_get_status(= ): eicr 2100000 <27>1 2022-03-05T00:13:00.664603+00:00 - - ixgbe_dev_interrupt_action(): e= nable intr immediately, mask: 0x02200000, orig: 0x02200000, flags: 0x000000= 01 <27>1 2022-03-05T00:13:01.664703+00:00 - - ixgbe_dev_interrupt_delayed_han= dler(): in delay func: eicr 0x00000000 <27>1 2022-03-05T00:13:01.664738+00:00 - - ixgbe_dev_interrupt_delayed_han= dler(): enable intr delayed, mask: 0x02200000, orig: 0x02200000, flags: 0x0= 0000001 <27>1 2022-03-05T00:13:04.377237+00:00 - - ixgbe_dev_interrupt_delayed_han= dler(): in delay func: eicr 0x00000000 <27>1 2022-03-05T00:13:04.377269+00:00 - - ixgbe_dev_interrupt_delayed_han= dler(): enable intr delayed, mask: 0x02200000, orig: 0x00000000, flags: 0x0= 0000000 # port transition from down to up Let me know what additional data can be provided to help root cause this. -Mike --_000_BYAPR04MB4325C5B6EE71A0562EBA365CCF069BYAPR04MB4325namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,

 

We recently ran into an issue with DPDK 20.11 for th= e IXGBE driver operating in 10G BASE-T mode. We have been able to replicate= this behavior using dpdk-testpmd and do not see any recent/pertinent updat= es, so we are hopeful someone may be able to advise based on the information provided below. On the surface,= based on our investigation, it would appear the current link-down transiti= on logic does not correctly preserve IRQ mask configurations, specifically = LSC, when a link partner causes some sort of slow or bounced link down event.

Background:
We recently started using a new 3rd party traffic generator card= for testing our application. We found when using this card in 10G BASE-T m= ode and toggling link up/down, it would correctly cause our application to = detect the port to be down in our DPDK design. However, the link down event handling by the DPDK IXGBE drive= r appears to permanently disable its LSC IRQ detection on the first port do= wn event such that any subsequent link up or down events from the external = test card on this port would no longer be detected. The only way to restore link up was to restart the DPD= K port in our design (stop/start). Having looked at this a bit, we switched= over to the classic testpmd application and observed the exact same behavi= or.

 

Here is the data we believe you would find interesti= ng:

 

NIC in question:

 

# lspci -D -nn | grep -F [0200] | grep 552

0000:03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet C= onnection X552/X557-AT 10GBASE-T [8086:15ad]

0000:03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet C= onnection X552/X557-AT 10GBASE-T [8086:15ad]

# dpdk-devbind.py -s | grep 552

0000:03:00.0 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' drv=3D= vfio-pci unused=3Duio_pci_generic

0000:03:00.1 'Ethernet Connection X552/X557-AT 10GBASE-T 15ad' drv=3D= vfio-pci unused=3Duio_pci_generic

 

We made the following debug logging changes to try a= n capture interesting data to share:

 

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixg= be_ethdev.c

index 5a30c39593..75a9f9163b 100644

--- a/drivers/net/ixgbe/ixgbe_ethdev.c

+++ b/drivers/net/ixgbe/ixgbe_ethdev.c

@@ -4497,7 +4497,7 @@ ixgbe_dev_interrupt_get_status(struct rte_eth_d= ev *dev)

     /* read-on-clear nic registers here */<= /o:p>

    eicr =3D IXGBE_READ_REG(hw, IXGBE_EICR);

-    PMD_DRV_LOG(DEBUG, "eicr %x", eicr);

+    PMD_DRV_LOG(ERR, "eicr %x", eicr);=

     intr->flags =3D 0;

@@ -4614,7 +4613,7 @@ ixgbe_dev_interrupt_action(struct rte_eth_dev *= dev)

          }

    }

-    PMD_DRV_LOG(DEBUG, "enable intr immediately&= quot;);

+    PMD_DRV_LOG(ERR, "enable intr immediately, m= ask: 0x%08x, orig: 0x%08x, flags: 0x%08x", intr->mask, intr->mas= k_original, intr->flags);

    ixgbe_enable_intr(dev);

     return 0;

@@ -4648,7 +4647,9 @@ ixgbe_dev_interrupt_delayed_handler(void *param= )

     ixgbe_disable_intr(hw);

-    eicr =3D IXGBE_READ_REG(hw, IXGBE_EICR);

+   eicr =3D IXGBE_READ_REG(hw, IXGBE_EICR);

+   PMD_DRV_LOG(ERR, "in delay func: eicr 0x%08x"= , eicr);

+   PMD_DRV_LOG(ERR, "enable intr delayed, mask: 0x%08= x, orig: 0x%08x, flags: 0x%08x", intr->mask, intr->mask_original= , intr->flags);

    if (eicr & IXGBE_EICR_MAILBOX)

          ixgbe_pf_mbx_p= rocess(dev);

 

With the above “log-err” additions, we h= ave provided the following results. The first set of data below was generat= ed using an older 3rd party traffic generator card to provide &#= 8220;good” results that show the IXGBE driver working correctly. Following that are the non-working (bad) logging results for th= e new traffic generator card. Both 3rd party cards correctly tra= nsition between down and up states.

 

 

#####################################################################= #

# good sequence, both down detection and then up detection=

#####################################################################= #

# port transition from up to down

<27>1 2022-03-05T00:12:11.415436+00:00 - -  ixgbe_dev_inte= rrupt_get_status(): eicr 100000

<27>1 2022-03-05T00:12:11.415489+00:00 - -  ixgbe_dev_inte= rrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000= , flags: 0x00000001

<27>1 2022-03-05T00:12:11.425448+00:00 - -  ixgbe_dev_inte= rrupt_get_status(): eicr 2000000

<27>1 2022-03-05T00:12:11.446191+00:00 - -  ixgbe_dev_inte= rrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000= , flags: 0x00000000

<27>1 2022-03-05T00:12:15.415600+00:00 - -  ixgbe_dev_inte= rrupt_delayed_handler(): in delay func: eicr 0x00000000

<27>1 2022-03-05T00:12:15.415655+00:00 - -  ixgbe_dev_inte= rrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x023= 00000, flags: 0x00000000

 

# port transition from down to up

<27>1 2022-03-05T00:12:33.856734+00:00 - -  ixgbe_dev_inte= rrupt_get_status(): eicr 2000000

<27>1 2022-03-05T00:12:33.877463+00:00 - -  ixgbe_dev_inte= rrupt_action(): enable intr immediately, mask: 0x02300000, orig: 0x00000000= , flags: 0x00000000

<27>1 2022-03-05T00:12:34.203274+00:00 - -  ixgbe_dev_inte= rrupt_get_status(): eicr 100000

<27>1 2022-03-05T00:12:34.207905+00:00 - -  ixgbe_dev_inte= rrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000= , flags: 0x00000001

<27>1 2022-03-05T00:12:35.207994+00:00 - -  ixgbe_dev_inte= rrupt_delayed_handler(): in delay func: eicr 0x00100000

<27>1 2022-03-05T00:12:35.208027+00:00 - -  ixgbe_dev_inte= rrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x023= 00000, flags: 0x00000001

 

#####################################################################= #

# bad sequence, detects down event, but does not see the up event

#####################################################################= #

# port transition from up to down

<27>1 2022-03-05T00:13:00.377072+00:00 - -  ixgbe_dev_inte= rrupt_get_status(): eicr 100000

<27>1 2022-03-05T00:13:00.377127+00:00 - -  ixgbe_dev_inte= rrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02300000= , flags: 0x00000001

<27>1 2022-03-05T00:13:00.643788+00:00 - -  ixgbe_dev_inte= rrupt_get_status(): eicr 2100000

<27>1 2022-03-05T00:13:00.664603+00:00 - -  ixgbe_dev_inte= rrupt_action(): enable intr immediately, mask: 0x02200000, orig: 0x02200000= , flags: 0x00000001

<27>1 2022-03-05T00:13:01.664703+00:00 - -  ixgbe_dev_inte= rrupt_delayed_handler(): in delay func: eicr 0x00000000

<27>1 2022-03-05T00:13:01.664738+00:00 - -  ixgbe_dev_inte= rrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x022= 00000, flags: 0x00000001

<27>1 2022-03-05T00:13:04.377237+00:00 - -  ixgbe_dev_inte= rrupt_delayed_handler(): in delay func: eicr 0x00000000

<27>1 2022-03-05T00:13:04.377269+00:00 - -  ixgbe_dev_inte= rrupt_delayed_handler(): enable intr delayed, mask: 0x02200000, orig: 0x000= 00000, flags: 0x00000000

 

# port transition from down to up

<nothing happens as LSC IRQ is not enabled due to above link-down = sequence>

 

Let me know what additional data can be provided to = help root cause this.

 

-Mike

 

--_000_BYAPR04MB4325C5B6EE71A0562EBA365CCF069BYAPR04MB4325namp_--