i40e PMD: Tx hang issue

i40e PMD: Tx hang issue

[madhukar mythri]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

Hi,

We are facing an issue at the Transmit side randomly after 8 to 10 days of
network traffic flow on the Intel X710 10G NIC with i40e PMD.
We found the issue is at Transmit side Tx-queue, as the Tx packets were not
going out and also observed that the Tx-queue stats were not incrementing,
even though we are sending packets out through the 'rte_eth_tx_burst()',
this function returns '0'. Whereas the Rx-packets were received well at the
same-time and Rx-stats were incremented well, even the NIC Link is also
stable at 1G speed.

So, has anyone faced such issue on this DPDK i40e PMD at Transmit
side(Tx-queue), if so, please let us know, if any known-issue 'Or'
fix/work-around on this.

DPDK-version: DPDK-21.11
X710 NIC speed: Connected with 1G speed switch-port. so, as per
auto-negotiation the link-speed is 1G only.
X710 NIC firmware-version: "fw 6.0.48442 api 1.7 nvm 6.01 0x80003b43
1.1824.0 [8086:1589] [108e:7b1c]".

Thanks,
Madhukar.

[-- Attachment #2: Type: text/html, Size: 1112 bytes --]

Re: i40e PMD: Tx hang issue

[Nishant Verma]

[-- Attachment #1: Type: text/plain, Size: 1288 bytes --]

Hi Madhukar,

Can you please elaborate what issue you found in TX-side? Any solution for
that?
For me it seems to be both rx and tx.


Thanks.

Regards,
Nishant Verma


On Mon, May 8, 2023 at 2:53 AM madhukar mythri <madhukar.mythri@gmail.com>
wrote:

> Hi,
>
> We are facing an issue at the Transmit side randomly after 8 to 10 days of
> network traffic flow on the Intel X710 10G NIC with i40e PMD.
> We found the issue is at Transmit side Tx-queue, as the Tx packets were
> not going out and also observed that the Tx-queue stats were not
> incrementing, even though we are sending packets out through the
> 'rte_eth_tx_burst()', this function returns '0'. Whereas the Rx-packets
> were received well at the same-time and Rx-stats were incremented well,
> even the NIC Link is also stable at 1G speed.
>
> So, has anyone faced such issue on this DPDK i40e PMD at Transmit
> side(Tx-queue), if so, please let us know, if any known-issue 'Or'
> fix/work-around on this.
>
> DPDK-version: DPDK-21.11
> X710 NIC speed: Connected with 1G speed switch-port. so, as per
> auto-negotiation the link-speed is 1G only.
> X710 NIC firmware-version: "fw 6.0.48442 api 1.7 nvm 6.01 0x80003b43
> 1.1824.0 [8086:1589] [108e:7b1c]".
>
> Thanks,
> Madhukar.
>

[-- Attachment #2: Type: text/html, Size: 2525 bytes --]

Re: i40e PMD: Tx hang issue

[madhukar mythri]

[-- Attachment #1: Type: text/plain, Size: 2352 bytes --]

Hi Nishant,

Our application runs on top of DPDK-21.11 with net_i40e PMD(X710 NIC).
After running traffic for some days(8 to 10 days), intermittently we could
see Tx-shall on Queue-0(we had enabled only a single queue) and this was
identified by looking into the Rx/Tx eth-stats of NIC, where we could see
Tx-packets stats was not incremented and only Rx-stats were incremented and
the 'rte_eth_tx_burst()', this function returns '0'.

Work-arounds: Monitor this Rx/Tx stats for every second in an other thread
and do the following steps:
1) Restart the PMD. using rte_eth_dev_stop(port) , usleep(10) and
rte_eth_dev_start().
2) Restart the respective Tx Queue-0 only, which got stalled, by
using rte_eth_dev_tx_queue_stop() and rte_eth_dev_tx_queue_start().

First approach takes a long time(around 10 seconds) to recover the NIC.
whereas the second approach recovers within < 3 seconds.

Regards,
Madhukar.


On Thu, May 18, 2023 at 5:34 PM Nishant Verma <vnish11@gmail.com> wrote:

> Hi Madhukar,
>
> Can you please elaborate what issue you found in TX-side? Any solution for
> that?
> For me it seems to be both rx and tx.
>
>
> Thanks.
>
> Regards,
> Nishant Verma
>
>
> On Mon, May 8, 2023 at 2:53 AM madhukar mythri <madhukar.mythri@gmail.com>
> wrote:
>
>> Hi,
>>
>> We are facing an issue at the Transmit side randomly after 8 to 10 days
>> of network traffic flow on the Intel X710 10G NIC with i40e PMD.
>> We found the issue is at Transmit side Tx-queue, as the Tx packets were
>> not going out and also observed that the Tx-queue stats were not
>> incrementing, even though we are sending packets out through the
>> 'rte_eth_tx_burst()', this function returns '0'. Whereas the Rx-packets
>> were received well at the same-time and Rx-stats were incremented well,
>> even the NIC Link is also stable at 1G speed.
>>
>> So, has anyone faced such issue on this DPDK i40e PMD at Transmit
>> side(Tx-queue), if so, please let us know, if any known-issue 'Or'
>> fix/work-around on this.
>>
>> DPDK-version: DPDK-21.11
>> X710 NIC speed: Connected with 1G speed switch-port. so, as per
>> auto-negotiation the link-speed is 1G only.
>> X710 NIC firmware-version: "fw 6.0.48442 api 1.7 nvm 6.01 0x80003b43
>> 1.1824.0 [8086:1589] [108e:7b1c]".
>>
>> Thanks,
>> Madhukar.
>>
>

[-- Attachment #2: Type: text/html, Size: 3955 bytes --]