[dpdk-dev] net/ena: traffic lock

[dpdk-dev] net/ena: traffic lock

[Rajesh Kumar]

Dpdk: 19.11

Driver: ena



During longevity(after 24+ hrs) testing at 10Gbps, one of tx-queue is
getting into unrecoverable state ( its not able to find enough
tx-descriptor nor its freeing up).



So for every tx-burst, eth_ena_xmit_pkts() neither finds free tx-descriptor
nor able to free txd (ena_com_tx_comp_req_id_get() is always returning
ENA_COM_TRY_AGAIN).



We see eth_ena_xmit_pkts() has been refactored in latest LTS version, is
there any related issue got fixed ? Can you help





(gdb) p *(struct ena_ring *) rte_eth_devices[2].data->tx_queues[5]

$14 = {

  next_to_use = 4979,

  next_to_clean = 3958,

  type = ENA_RING_TYPE_TX,

  tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_DEV,

  {

    empty_tx_reqs = 0x11e406b00,

    empty_rx_reqs = 0x11e406b00

  },

  {

    tx_buffer_info = 0x11d2dfc80,

    rx_buffer_info = 0x11d2dfc80

  },

 rx_refill_buffer = 0x0,

  ring_size = 1024,

  ena_com_io_cq = 0x11e40e640,

  ena_com_io_sq = 0x11e4168c0,

  ena_bufs = {{

      len = 0,

      req_id = 0

    } <repeats 17 times>},

  mb_pool = 0x0,

  port_id = 2,

  id = 5,

  tx_max_header_size = 96 '`',

 configured = 1,

  push_buf_intermediate_buf = 0x11e406a00 "",

  adapter = 0x11e40e040,

  offloads = 2,

  sgl_size = 17,

  {

    rx_stats = {

      cnt = 4979,

      bytes = 417580,

      refill_partial = 35426,

      bad_csum = 0,

      mbuf_alloc_fail = 0,

      bad_desc_num = 38603,

---Type <return> to continue, or q <return> to quit---

      bad_req_id = 3178

    },

    tx_stats = {

      cnt = 4979,

      bytes = 417580,

      prepare_ctx_err = 35426, ß Errors

      linearize = 0,

      linearize_failed = 0,

      tx_poll = 38603,

      doorbells = 3178,

      bad_req_id = 0,

      available_desc = 2 <<-- No free descriptor

    }

  },

  numa_socket_id = 0

}



Thanks,

*-Rajesh*

[dpdk-dev] net/ena: traffic lock

[RajeshKumar Kalidass]

Dpdk: 19.11
Driver: ena

During longevity(after 24+ hrs) testing at 10Gbps, one of tx-queue is getting into unrecoverable state ( its not able to find enough tx-descriptor nor its freeing up).

So for every tx-burst, eth_ena_xmit_pkts() neither finds free tx-descriptor nor able to free txd (ena_com_tx_comp_req_id_get() is always returning ENA_COM_TRY_AGAIN).

We see eth_ena_xmit_pkts() has been refactored in latest LTS version, is there any related issue got fixed ? Can you help


(gdb) p *(struct ena_ring *) rte_eth_devices[2].data->tx_queues[5]
$14 = {
  next_to_use = 4979,
  next_to_clean = 3958,
  type = ENA_RING_TYPE_TX,
  tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_DEV,
  {
    empty_tx_reqs = 0x11e406b00,
    empty_rx_reqs = 0x11e406b00
  },
  {
    tx_buffer_info = 0x11d2dfc80,
    rx_buffer_info = 0x11d2dfc80
  },
 rx_refill_buffer = 0x0,
  ring_size = 1024,
  ena_com_io_cq = 0x11e40e640,
  ena_com_io_sq = 0x11e4168c0,
  ena_bufs = {{
      len = 0,
      req_id = 0
    } <repeats 17 times>},
  mb_pool = 0x0,
  port_id = 2,
  id = 5,
  tx_max_header_size = 96 '`',
 configured = 1,
  push_buf_intermediate_buf = 0x11e406a00 "",
  adapter = 0x11e40e040,
  offloads = 2,
  sgl_size = 17,
  {
    rx_stats = {
      cnt = 4979,
      bytes = 417580,
      refill_partial = 35426,
      bad_csum = 0,
      mbuf_alloc_fail = 0,
      bad_desc_num = 38603,
---Type <return> to continue, or q <return> to quit---
      bad_req_id = 3178
    },
    tx_stats = {
      cnt = 4979,
      bytes = 417580,
      prepare_ctx_err = 35426, <-- Errors
      linearize = 0,
      linearize_failed = 0,
      tx_poll = 38603,
      doorbells = 3178,
      bad_req_id = 0,
      available_desc = 2
    }
  },
  numa_socket_id = 0
}

Thanks,
-Rajesh

This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Gigamon email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails ("spam").

Re: [dpdk-dev] net/ena: traffic lock

[Chauskin, Igor]

Hi Rajesh,

 

For the tx queue getting into unrecoverable state - can you please share the
details of your usecase? (instance types, traffic type, number of queues in
use, etc.). Also, please share your instances ids, if possible.

 

Regarding prepare_ctx_err counter - under certain circumstances this counter
indeed can be incorrectly incremented and we're working on a fix to this.
However, this shouldn't have effect beyond statistics (unless your
application explicitly relies on this counter in its business logic). 

 

Thanks,

Igor

 

From: RajeshKumar Kalidass <rajesh.kalidass@gigamon.com> 
Sent: Tuesday, December 22, 2020 10:03
To: mk@semihalf.com; Chauskin, Igor <igorch@amazon.com>; Tzalik, Guy
<gtzalik@amazon.com>; ar@semihalf.com; dev@dpdk.org
Cc: Tanmay Kishore <tanmay.kishore@gigamon.com>; Rakesh Jagota
<rakesh.jagota@gigamon.com>
Subject: [EXTERNAL] net/ena: traffic lock 

 


CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you can confirm the sender and know
the content is safe.

 

Dpdk: 19.11

Driver: ena

 

During longevity(after 24+ hrs) testing at 10Gbps, one of tx-queue is
getting into unrecoverable state ( its not able to find enough tx-descriptor
nor its freeing up).

 

So for every tx-burst, eth_ena_xmit_pkts() neither finds free tx-descriptor
nor able to free txd (ena_com_tx_comp_req_id_get() is always returning
ENA_COM_TRY_AGAIN).

 

We see eth_ena_xmit_pkts() has been refactored in latest LTS version, is
there any related issue got fixed ? Can you help

 

 

(gdb) p *(struct ena_ring *) rte_eth_devices[2].data->tx_queues[5]

$14 = {

  next_to_use = 4979,

  next_to_clean = 3958,

  type = ENA_RING_TYPE_TX,

  tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_DEV,

  {

    empty_tx_reqs = 0x11e406b00,

    empty_rx_reqs = 0x11e406b00

  },

  {

    tx_buffer_info = 0x11d2dfc80,

    rx_buffer_info = 0x11d2dfc80

  },

 rx_refill_buffer = 0x0,

  ring_size = 1024,

  ena_com_io_cq = 0x11e40e640,

  ena_com_io_sq = 0x11e4168c0,

  ena_bufs = {{

      len = 0,

      req_id = 0

    } <repeats 17 times>},

  mb_pool = 0x0,

  port_id = 2,

  id = 5,

  tx_max_header_size = 96 '`',

 configured = 1,

  push_buf_intermediate_buf = 0x11e406a00 "",

  adapter = 0x11e40e040,

  offloads = 2,

  sgl_size = 17,

  {

    rx_stats = {

      cnt = 4979,

      bytes = 417580,

      refill_partial = 35426,

      bad_csum = 0,

      mbuf_alloc_fail = 0,

      bad_desc_num = 38603,

---Type <return> to continue, or q <return> to quit---

      bad_req_id = 3178

    },

    tx_stats = {

      cnt = 4979,

      bytes = 417580,

      prepare_ctx_err = 35426, <-- Errors

      linearize = 0,

      linearize_failed = 0,

      tx_poll = 38603,

      doorbells = 3178,

      bad_req_id = 0,

      available_desc = 2

    }

  },

  numa_socket_id = 0

}

 

Thanks,

-Rajesh

 

This message may contain confidential and privileged information. If it has
been sent to you in error, please reply to advise the sender of the error
and then immediately delete it. If you are not the intended recipient, do
not read, copy, disclose or otherwise use this message. The sender disclaims
any liability for such unauthorized use. NOTE that all incoming emails sent
to Gigamon email accounts will be archived and may be scanned by us and/or
by external service providers to detect and prevent threats to our systems,
investigate illegal or inappropriate behavior, and/or eliminate unsolicited
promotional emails ("spam").