From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A536842B51 for ; Sat, 20 May 2023 09:06:31 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1DCD342BFE; Sat, 20 May 2023 09:06:31 +0200 (CEST) Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by mails.dpdk.org (Postfix) with ESMTP id B6D6E42BC9 for ; Sat, 20 May 2023 09:06:29 +0200 (CEST) Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-61b40562054so18567396d6.2 for ; Sat, 20 May 2023 00:06:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684566389; x=1687158389; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Q77YngXd38hKolY6xF4UBOLdI/mtmyBzNFLrVMg3Jmk=; b=R/3873EMhq87kafGycxoqa8C4NfOyfu1+uzq7Kmi5yAlt7db2OYPdclqjXqGwnTpnI EKHR1k5U1EnAsaatBLzAxW53s4ZuMs/4qcAFB0m2z8skLyks1cgrRe+1mjbABysQOrjJ f4/YwazGmw/Tgw0Z8ui/BcQDjqHuRDJDxAZoG4xQRG8kuQ4LIl7TUs3bHYKSwBYZYDIL 6E7lZExysuhxFomXY+AaVGEUSFYNOcG2lqKIkPjcGKUtBGe35SBAyRA27w5UKAF0GXKr TREGfFNTOo2P359WkAaht/vlr4xQYReRONcGfCKxYTd7fm9eu/mYNRus8NJ0cPTdJkdj hZdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684566389; x=1687158389; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Q77YngXd38hKolY6xF4UBOLdI/mtmyBzNFLrVMg3Jmk=; b=SHucTdpSac7bi+qb33/VMZvDrx6utSvC7t2NqPc67RbdRgWP/L9K3+aDrqpMRu5RZq MW/LM5TsRowCR+Rmigj0FEufqxb4JQrgLW2cU6QivORvs1vmqU1JGJS97+cEDv+OyZXk rVwhv7Cg5BWgyR3JUm1jkC09WnFzAf1fhbAInyNmdsDa3nc+uQY7bwx/IXOHY1fggVrd Oe9Bqlm7J34BJBaRyIG5pjjNUkg/JQWu7dAZKI/PawkBbr7Ni1BfCdty/Qr0aDLkz2po xp/fMHmueHuna8tz23oI+IEiNQZQhMPmIIegiGKq8ZjBCNXx3VxEaLk2qxe5OcNF8HG/ oawA== X-Gm-Message-State: AC+VfDxihRxLfrM1AItBheWn83J+cZZEbSbL34sNX/mduQ4J87f5Gjiq yfQACR+eFzlavJIXE+D8RDDsDfSCCizLuo++uQSXGFXgkqI= X-Google-Smtp-Source: ACHHUZ71gji52xI3NKyVRuUYSuTuO2o8hoH0CxdoXU1YQ6MoLUOEaXinsl8kAmwKE8e3WtcmEcatsD5XNvLMMwUm7A0= X-Received: by 2002:a05:6214:1309:b0:621:712c:fa03 with SMTP id pn9-20020a056214130900b00621712cfa03mr8981052qvb.44.1684566388825; Sat, 20 May 2023 00:06:28 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: madhukar mythri Date: Sat, 20 May 2023 12:36:20 +0530 Message-ID: Subject: Re: i40e PMD: Tx hang issue To: Nishant Verma Cc: users Content-Type: multipart/alternative; boundary="00000000000034e28c05fc1aad5a" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --00000000000034e28c05fc1aad5a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Nishant, Our application runs on top of DPDK-21.11 with net_i40e PMD(X710 NIC). After running traffic for some days(8 to 10 days), intermittently we could see Tx-shall on Queue-0(we had enabled only a single queue) and this was identified by looking into the Rx/Tx eth-stats of NIC, where we could see Tx-packets stats was not incremented and only Rx-stats were incremented and the 'rte_eth_tx_burst()', this function returns '0'. Work-arounds: Monitor this Rx/Tx stats for every second in an other thread and do the following steps: 1) Restart the PMD. using rte_eth_dev_stop(port) , usleep(10) and rte_eth_dev_start(). 2) Restart the respective Tx Queue-0 only, which got stalled, by using rte_eth_dev_tx_queue_stop() and rte_eth_dev_tx_queue_start(). First approach takes a long time(around 10 seconds) to recover the NIC. whereas the second approach recovers within < 3 seconds. Regards, Madhukar. On Thu, May 18, 2023 at 5:34=E2=80=AFPM Nishant Verma w= rote: > Hi Madhukar, > > Can you please elaborate what issue you found in TX-side? Any solution fo= r > that? > For me it seems to be both rx and tx. > > > Thanks. > > Regards, > Nishant Verma > > > On Mon, May 8, 2023 at 2:53=E2=80=AFAM madhukar mythri > wrote: > >> Hi, >> >> We are facing an issue at the Transmit side randomly after 8 to 10 days >> of network traffic flow on the Intel X710 10G NIC with i40e PMD. >> We found the issue is at Transmit side Tx-queue, as the Tx packets were >> not going out and also observed that the Tx-queue stats were not >> incrementing, even though we are sending packets out through the >> 'rte_eth_tx_burst()', this function returns '0'. Whereas the Rx-packets >> were received well at the same-time and Rx-stats were incremented well, >> even the NIC Link is also stable at 1G speed. >> >> So, has anyone faced such issue on this DPDK i40e PMD at Transmit >> side(Tx-queue), if so, please let us know, if any known-issue 'Or' >> fix/work-around on this. >> >> DPDK-version: DPDK-21.11 >> X710 NIC speed: Connected with 1G speed switch-port. so, as per >> auto-negotiation the link-speed is 1G only. >> X710 NIC firmware-version: "fw 6.0.48442 api 1.7 nvm 6.01 0x80003b43 >> 1.1824.0 [8086:1589] [108e:7b1c]". >> >> Thanks, >> Madhukar. >> > --00000000000034e28c05fc1aad5a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Nishant,

Our application runs on top= of DPDK-21.11 with net_i40e PMD(X710 NIC).
After running traffic= for some days(8 to 10 days), intermittently we could see Tx-shall on Queue= -0(we had enabled only a single queue) and this was identified by looking i= nto the Rx/Tx eth-stats of NIC, where we could see Tx-packets stats was not= incremented and only Rx-stats were incremented and the 'rte_eth_tx_bur= st()', this function returns '0'.

Work= -arounds: Monitor this Rx/Tx stats for every second in an other thread and = do the following steps:
1) Restart the PMD. using rte_eth_dev_sto= p(port) , usleep(10) and=C2=A0 rte_eth_dev_start().=C2=A0=C2=A0
2= ) Restart the respective Tx Queue-0 only, which got stalled, by using=C2=A0= rte_eth_dev_tx_queue_stop() and=C2=A0rte_eth_dev_tx_queue_start().

First approach takes a long time(around 10 seconds) to rec= over the NIC. whereas the second approach recovers within < 3 seconds.

Regards,
Madhukar.


On= Thu, May 18, 2023 at 5:34=E2=80=AFPM Nishant Verma <vnish11@gmail.com> wrote:
Hi=C2=A0M= adhukar,

Can you please elaborat= e what issue you found in TX-side? Any solution for that?
For me it seems to be both rx and tx.


Thanks.

Regards,
Nishant Verma

<= br>
On Mon,= May 8, 2023 at 2:53=E2=80=AFAM madhukar mythri <madhukar.mythri@gmail.com> w= rote:
Hi,

We are facing an issue at t= he Transmit side randomly after 8 to 10 days of network traffic flow on the= Intel X710 10G NIC with i40e PMD.
We found the issue is at Transmit si= de Tx-queue, as the Tx packets were not going out and also observed that th= e Tx-queue stats were not incrementing, even though we are sending packets = out through the 'rte_eth_tx_burst()', this function returns '0&= #39;. Whereas the Rx-packets were received well at the same-time and Rx-sta= ts were incremented well, even the NIC Link is also stable at 1G speed.
=
So, has anyone faced such issue on this DPDK i40e PMD at Transmit sid= e(Tx-queue), if so, please let us know, if any known-issue 'Or' fix= /work-around on this.

DPDK-version: DPDK-21.11
X710 NIC speed:= Connected with 1G speed switch-port. so, as per auto-negotiation the link-= speed is 1G only.
X710 NIC firmware-version: "fw 6.0.48442 api 1.7= nvm 6.01 0x80003b43 1.1824.0 [8086:1589] [108e:7b1c]".
=
Thanks,
Madhukar.
--00000000000034e28c05fc1aad5a--