From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id AC9B7A046B for ; Thu, 25 Jul 2019 18:28:36 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 773BF1C395; Thu, 25 Jul 2019 18:28:35 +0200 (CEST) Received: from mx0a-00103a01.pphosted.com (mx0b-00103a01.pphosted.com [67.231.152.227]) by dpdk.org (Postfix) with ESMTP id D03CE1C38F for ; Thu, 25 Jul 2019 18:28:32 +0200 (CEST) Received: from pps.filterd (m0002317.ppops.net [127.0.0.1]) by mx0b-00103a01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x6PFubQC031341; Thu, 25 Jul 2019 12:28:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ciena.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=06252019; bh=7cReFWyarbgl5+piPsOmQNXjFG8YIAI4AMWlC/I07LY=; b=D0oXFrNywNMiS770563YIqw+kXyQRszQMkAx/HVu/LX+CzpSySCpKItFCfkN7cJNJiOz 1nmopLQCHj5JGuStdODJP26sbj8v0Ih3PhYAhzvBtelbCmW+S1KmPsntRaRDTzB0gJau ynhORY5HIMxdcEBuDrFb0MgrSfAZTce0FQDxy482qIEedtCLh60pDBi3EGGS+K3cLc1v WrRcZwWytiF+UlCsEwZkdWFtf/5qPlhKluHVoUFY6LMIaFWdkbt+O/2X2l4rB+0WmG8V E8R9rftBwsANhaEX5ZglCOuyAZrcQiH2Te2Eu7CT4tVRWmHms1jIwUkc9lB4rEp1e3PZ 6A== Received: from nam03-dm3-obe.outbound.protection.outlook.com (mail-dm3nam03lp2056.outbound.protection.outlook.com [104.47.41.56]) by mx0b-00103a01.pphosted.com with ESMTP id 2tx618aw1m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 25 Jul 2019 12:28:31 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=W0O5yhJVdVuNIC0VY2ZG3oTQYWxHCZcrcTVjVS1c1Yt8BWeDQ1VGLQa5XcLa1py0AqFardtsAwNqo2ND4wFLDPOSUT65W1bfdj9SHITZ7CtNMOIKMpDq6jJpK6sCTFduEdJ4zqwgfFuKdIlXyhNYe4w+OiscihonXs8bXJdwjarUvNOcv3F8USFqbhhIusYJVSYDMHw6qnTCiWAIw5iRknQ7zcbWgXms+M9idFIS5KN+FaLyz6MVfskg6N228mY7QUNTLeWjdvolWLTx6azVt32imuObisLh85fAgJ5YZMPTPD9g+gbeOMKxSuDYvCmh3mLYW9bumdXYi/mlEdHgnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7cReFWyarbgl5+piPsOmQNXjFG8YIAI4AMWlC/I07LY=; b=Cx2nIS4E3lWTdFuW1ndbKd8dspmy15JhZlpPUO0brBVXDqY3dreI6Z8kS4EjmxUNGoDyHTfeZUie97FjSOOfy8SzgU0b3xdr3faRZNqquMehrFt7t6aWvO4/xK5uU0xBXRygKULexrdQ3KrRHb3cgJuk5voqdFoCEDFeXD/uZ5sB8BFveOhw4hyxfkqYKWPwZ/7cfoNnvlA+gWu0Jq5nS/iT9z0gEKi2Xbk2Z7T6ySvm4HSdT5q+iEXzLOUC5KGJyJeB3b/7YgWnGiotiim4tJrqqzu5VsL8a2VZN0IGWoIvf5kg9ArTJrwQXvAXtUt9Wq9p99UYea4w0uCCzKwr/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=ciena.com;dmarc=pass action=none header.from=ciena.com;dkim=pass header.d=ciena.com;arc=none Received: from MWHPR04MB0592.namprd04.prod.outlook.com (10.173.49.145) by MWHPR04MB1247.namprd04.prod.outlook.com (10.174.175.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2094.16; Thu, 25 Jul 2019 16:28:28 +0000 Received: from MWHPR04MB0592.namprd04.prod.outlook.com ([fe80::a45f:7fde:28e4:8519]) by MWHPR04MB0592.namprd04.prod.outlook.com ([fe80::a45f:7fde:28e4:8519%7]) with mapi id 15.20.2094.013; Thu, 25 Jul 2019 16:28:28 +0000 From: "Bly, Mike" To: "Ananyev, Konstantin" , "'dev@dpdk.org'" CC: "Zhang, Qi Z" , "Lu, Wenzhuo" Thread-Topic: [dpdk-dev] x552 transmit issue and rte_ethtool - rte_ethtool_get_regs() Thread-Index: AQHVQXAg9vjPmvftnkCp1nm+mPlQ5KbYbdCAgABgZMCAAJhzgIACIXwA Date: Thu, 25 Jul 2019 16:28:28 +0000 Message-ID: References: <2601191342CEEE43887BDE71AB9772580168A5A018@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772580168A5A51F@irsmsx105.ger.corp.intel.com> In-Reply-To: <2601191342CEEE43887BDE71AB9772580168A5A51F@irsmsx105.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [165.225.50.165] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8f03d199-4818-4add-a879-08d7111d2042 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7167020)(7193020); SRVR:MWHPR04MB1247; x-ms-traffictypediagnostic: MWHPR04MB1247: x-ms-exchange-purlcount: 2 x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 0109D382B0 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(136003)(346002)(396003)(39860400002)(366004)(376002)(199004)(189003)(13464003)(4326008)(7736002)(9686003)(6436002)(478600001)(25786009)(110136005)(99286004)(68736007)(86362001)(3846002)(316002)(7696005)(2906002)(76176011)(6116002)(305945005)(81156014)(486006)(5660300002)(54906003)(66066001)(55016002)(14454004)(5024004)(6246003)(102836004)(6306002)(71190400001)(26005)(52536014)(256004)(966005)(14444005)(229853002)(71200400001)(476003)(33656002)(55236004)(64756008)(66446008)(66556008)(66476007)(74316002)(66946007)(186003)(76116006)(11346002)(446003)(53546011)(81166006)(6506007)(8936002)(53936002)(491001); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR04MB1247; H:MWHPR04MB0592.namprd04.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: ciena.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: FY5kwyq6eGXaWunTVGpJuogBuq1kxjY7AynyGPqQ0H4aySALwIO3WqJdbnl2785HOcWpFhBsGiSO45/TJ7jBihilvUNd684XpUUhWtz/pHWLDvY7NH1j0S0MCTmLnWDXuENuD7p+vaDjjB+2htEgT40+xzbg58R/GSAqkdAUe0E/XLxLIbuJ56rRNfLOODL8nygw0V6O/HNKm/Lr+anMeaSOpaUP5DtowFWBlwUNbeDiRPBpQ4mNjSEMxscKsBW+dZo406aN6AfNTE9nepK4tiL8z8uhpCBV0TfojH7SF20dbTKBrQY8TT65aKsCEemBbi02OLeTAJ+wQakmRVOKX+aVw1gEhbEYiLViIn3/ns6WAbKIhe5nVOXDmLgTB/REXYLaGzYbuuqe7gjmEoaKSg2nakmGpt5GeDxGj54mc8E= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: ciena.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8f03d199-4818-4add-a879-08d7111d2042 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Jul 2019 16:28:28.7836 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 457a2b01-0019-42ba-a449-45f99e96b60a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: mbly@ciena.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR04MB1247 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:5.22.84,1.0.8 definitions=2019-07-25_06:2019-07-25,2019-07-25 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 adultscore=0 clxscore=1015 priorityscore=1501 impostorscore=0 lowpriorityscore=0 phishscore=0 malwarescore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1907250188 Subject: Re: [dpdk-dev] x552 transmit issue and rte_ethtool - rte_ethtool_get_regs() X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Konstantin, After digging a bit further, we discovered our custom test setup was inadve= rtently running two threads calling the same transmit sequence, thus wedgin= g the NIC. While this reproduced the DD not ready symptoms we are chasing, = it is not a valid reproduction of what our application is capable of doing.= We will continue looking and update when we have more to share. -Mike -----Original Message----- From: Ananyev, Konstantin =20 Sent: Wednesday, July 24, 2019 12:53 AM To: Bly, Mike ; 'dev@dpdk.org' Cc: Zhang, Qi Z ; Lu, Wenzhuo Subject: [**EXTERNAL**] RE: [dpdk-dev] x552 transmit issue and rte_ethtool = - rte_ethtool_get_regs() Hi Mike, > Konstantin, >=20 > The recommended use of rte_eth_tx_prepare() had no effect, which after=20 > looking at it, makes sense. We are using "large" mbufs to support=20 > Jumbo frames, so nb-seg will always =3D=3D 1. Additionally, we are not=20 > currently leveraging any HW offload capabilities. As such, > rte_eth_tx_prepare() always returns "num-frames". >=20 > Taking this a step further, I have reproduced the problem using a=20 > simple c-unit test that builds bursts of frames, where each burst=20 > contains a max-burst of frames (32 in our application), where the=20 > interleaved frames have either a legal frame length (124-bytes) or=20 > intentionally a runt frame length (20-bytes++). These are dumb-simple=20 > L2 frames, i.e. NOT ip-frames. The NIC is setup to pad and append, so=20 > it should just do that without issue as needed. The test repeats this bur= st sequence a 100 times, resulting in 3200 frames attempting to be transmit= ted. Run to run, I am seeing anywhere from 750 to 3000 frames get transmitt= ed. Thereafter, the NIC will no longer accept frames for transmit. Using GD= B, we have confirmed the same DD status problem is present and preventing i= xgbe_tx_free_bufs() from doing any actual freeing of resources. >=20 > Is there a minimum runt size officially supported by DPDK and/or Intel=20 > on the x550 NIC family? We could certainly do a simple frame-length=20 > check and discard accordingly. However, we have seen 3rd party applicatio= ns send us runts, e.g. 40-byte ARP requests, over vhost-user and tap interf= aces, so we are a bit hesitant to blindly enforce this at 60 bytes (min ETH= minus CRC). AFAIK, sending frames smaller then 64B shouldn't cause a problem. At least I never hit such limitation. Qi, Wenzhuo - did you ever see such issue? My suggestion would be to open a new Bugzilla ticket for investigation and = attach pcap file (or some scapy script to generate it) so it could be repro= duced with test-pmd. Thanks Konstantin =20 >=20 > -Mike >=20 > -----Original Message----- > From: Bly, Mike > Sent: Tuesday, July 23, 2019 10:08 AM > To: Ananyev, Konstantin ; 'dev@dpdk.org' > Subject: RE: [dpdk-dev] x552 transmit issue and rte_ethtool - rte_ethtool= _get_regs() >=20 > Konstantin, >=20 > Thank you for the prompt reply on this posting. In looking at the single = use-case in test-pmd's csumonly.c, it would seem prepare + > retry_enabled may have some shortcomings as currently coded when nb_prep = < nb_rx. Has anyone looked at this? I happened to notice this > when looking for a reference for how it is expected to be used. It would = seem nb_rx should be replaced with nb_prep in the retry code. I > think the rest of the code should "just work" from there. Thoughts? >=20 > Regards, > Mike >=20 > -----Original Message----- > From: Ananyev, Konstantin > Sent: Tuesday, July 23, 2019 9:03 AM > To: Bly, Mike ; 'dev@dpdk.org' > Subject: [**EXTERNAL**] RE: [dpdk-dev] x552 transmit issue and rte_ethtoo= l - rte_ethtool_get_regs() >=20 >=20 >=20 >=20 > > > > Hello, > > > > We are chasing an interesting NIC transmit issue where after some > > period of time with normal operation the NIC enters a state where it > > refuses to transmit frames from our DPDK application via rte_eth_tx_bur= st(). All indications are the port is up and otherwise operational > and is still receiving traffic. It simply refuses to transmit anymore. > > > > Our application is running DPDK 17.05.1. In digging through the email > > archives, this appears to be related to the following posts, as we see = the same nb_free =3D 0 and IXGBE_ADVTXD_STAT_DD not set > problems they describe: > > http://mails.dpdk.org/archives/dev/2017-August/073240.html > > http://inbox.dpdk.org/dev/b704af91-dcc6-4481-a54c-3e174b744d17.h.liu@a > > libaba-inc.com/ > > > > Having not seen any resolution on the above DPDK posts and after a > > number of other investigative steps, we incorporated the rte_ethtool > > lib to provide the ability to dump the NIC register set via > > rte_ethtool_get_regs() in the hopes that perhaps there would be somethi= ng there in a status register to point us in the right direction. The > question now is what is the best way to check the register contents dumpe= d to the binary output file this API creates, for the x552 NIC? > Does anyone know if a decoder script exists? > > > > Other ideas to pursue? >=20 > It is hard to tell without any other information, but sometimes that happ= ens when user tries to TX malformed packet. > Might be worth to try using rte_eth_tx_prepare() inside your app. > It does some sanity checks to prevent such situations, especially when RT= E_LIBRTE_ETHDEV_DEBUG is on. > Konstantin >=20 >=20 >=20