DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] Questions about TX descriptors run out occasionally
@ 2018-07-27  2:13 Hui Liu
  2018-08-09  9:31 ` Bruce Richardson
  0 siblings, 1 reply; 5+ messages in thread
From: Hui Liu @ 2018-07-27  2:13 UTC (permalink / raw)
  To: dev

Hi Experts,

I'm developing my own dpdk-based application via Intel 82599ES port. My
Application is doing a job to send ICMP requests (packet size varies from
64 bytes to 1472 bytes, 200,000 pps, 1.1Gbps) and receive responses, with
ARP request/response and ICMP response handling when necessary. It was
working pretty fine in 5 hours to 10 days  randomly and then TX descriptors
run out and cannot be freed by ixgbe_tx_free_bufs() due to DD bit is not
set:

        /* check DD bit on threshold descriptor */
        status = txq->tx_ring[txq->tx_next_dd].wb.status;
        if (!(status & IXGBE_ADVTXD_STAT_DD))
                return 0;

My tx queue setup is:
        tx_conf->tx_thresh.pthresh = 64;
        tx_conf->tx_thresh.hthresh = 0;
        tx_conf->tx_thresh.wthresh = 0;
        tx_conf->tx_free_thresh = 256;
        tx_conf->tx_rs_thresh = 32;
        tx_conf->txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
ETH_TXQ_FLAGS_NOOFFLOADS;


I tried to read code to see if there is any case to take these descriptors
and never set IXGBE_ADVTXD_STAT_DD back but no luck yet. And I have not
even found the related code when IXGBE_ADVTXD_STAT_DD is set/unset when
descriptor is taken/released other than reset queues... So may I ask:
1. where do we set/unset IXGBE_ADVTXD_STAT_DD when descriptor is
taken/released?
2. any suggestion or information I should focus on to debug this issue? Is
this typically because of my upper application not alloc/free correctly or
any other problem?
3. another friend in dpdk-user list raised same issue in fm10k driver, but
later he mentioned his problem was because of overheating of NIC
(temperature was close to 85 degree Celsius). After setting system FAN to
full speed, he made it work perfectly. Since in my system I don't have
fan/temp sensors so I could not check this. Might this problem be caused by
high temperature in case?

Regards,
Hui

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [dpdk-dev] Questions about TX descriptors run out occasionally
  2018-07-27  2:13 [dpdk-dev] Questions about TX descriptors run out occasionally Hui Liu
@ 2018-08-09  9:31 ` Bruce Richardson
  2018-08-10 18:41   ` [dpdk-dev] 回复: " 刘辉
  0 siblings, 1 reply; 5+ messages in thread
From: Bruce Richardson @ 2018-08-09  9:31 UTC (permalink / raw)
  To: Hui Liu; +Cc: dev

On Thu, Jul 26, 2018 at 07:13:50PM -0700, Hui Liu wrote:
> Hi Experts,
> 
> I'm developing my own dpdk-based application via Intel 82599ES port. My
> Application is doing a job to send ICMP requests (packet size varies from
> 64 bytes to 1472 bytes, 200,000 pps, 1.1Gbps) and receive responses, with
> ARP request/response and ICMP response handling when necessary. It was
> working pretty fine in 5 hours to 10 days  randomly and then TX descriptors
> run out and cannot be freed by ixgbe_tx_free_bufs() due to DD bit is not
> set:
> 
>         /* check DD bit on threshold descriptor */
>         status = txq->tx_ring[txq->tx_next_dd].wb.status;
>         if (!(status & IXGBE_ADVTXD_STAT_DD))
>                 return 0;
> 
> My tx queue setup is:
>         tx_conf->tx_thresh.pthresh = 64;
>         tx_conf->tx_thresh.hthresh = 0;
>         tx_conf->tx_thresh.wthresh = 0;
>         tx_conf->tx_free_thresh = 256;
>         tx_conf->tx_rs_thresh = 32;
>         tx_conf->txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
> ETH_TXQ_FLAGS_NOOFFLOADS;
> 
> 
> I tried to read code to see if there is any case to take these descriptors
> and never set IXGBE_ADVTXD_STAT_DD back but no luck yet. And I have not
> even found the related code when IXGBE_ADVTXD_STAT_DD is set/unset when
> descriptor is taken/released other than reset queues... So may I ask:
> 1. where do we set/unset IXGBE_ADVTXD_STAT_DD when descriptor is
> taken/released?

For RX and TX, the DD bit is never set by software, only by hardware. When
writing a descriptor to memory for the NIC to read, the DD bit is cleared.
Software knows the NIC has finished with that descriptor by checking for
the DD bit being set by the NIC. If the DD bit is not being set, then the
problem is likely on the NIC side. [Potential software issues that could
cause this might be buffer e.g. overflows where we overwrite a DD bit set
by the NIC, or where we are polling an incorrect address, etc. etc.]

Regards,
/Bruce

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [dpdk-dev] 回复: Questions about TX descriptors run out occasionally
  2018-08-09  9:31 ` Bruce Richardson
@ 2018-08-10 18:41   ` 刘辉
  0 siblings, 0 replies; 5+ messages in thread
From: 刘辉 @ 2018-08-10 18:41 UTC (permalink / raw)
  To: Hui Liu, dev; +Cc: dev

Hi Bruce,

Thank you for your information! I will try different firmware version and also double check our application code and see if there is any potential incorrect memory access. Will update if I see anything.

Best Regards,
Hui



------------------------------------------------------------------
发件人:Bruce Richardson <bruce.richardson@intel.com>
发送时间:2018年8月9日(星期四) 02:31
收件人:Hui Liu <huiliu0213@gmail.com>
抄 送:dev <dev@dpdk.org>
主 题:Re: [dpdk-dev] Questions about TX descriptors run out occasionally

On Thu, Jul 26, 2018 at 07:13:50PM -0700, Hui Liu wrote:
> Hi Experts,
> 
> I'm developing my own dpdk-based application via Intel 82599ES port. My
> Application is doing a job to send ICMP requests (packet size varies from
> 64 bytes to 1472 bytes, 200,000 pps, 1.1Gbps) and receive responses, with
> ARP request/response and ICMP response handling when necessary. It was
> working pretty fine in 5 hours to 10 days  randomly and then TX descriptors
> run out and cannot be freed by ixgbe_tx_free_bufs() due to DD bit is not
> set:
> 
>         /* check DD bit on threshold descriptor */
>         status = txq->tx_ring[txq->tx_next_dd].wb.status;
>         if (!(status & IXGBE_ADVTXD_STAT_DD))
>                 return 0;
> 
> My tx queue setup is:
>         tx_conf->tx_thresh.pthresh = 64;
>         tx_conf->tx_thresh.hthresh = 0;
>         tx_conf->tx_thresh.wthresh = 0;
>         tx_conf->tx_free_thresh = 256;
>         tx_conf->tx_rs_thresh = 32;
>         tx_conf->txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
> ETH_TXQ_FLAGS_NOOFFLOADS;
> 
> 
> I tried to read code to see if there is any case to take these descriptors
> and never set IXGBE_ADVTXD_STAT_DD back but no luck yet. And I have not
> even found the related code when IXGBE_ADVTXD_STAT_DD is set/unset when
> descriptor is taken/released other than reset queues... So may I ask:
> 1. where do we set/unset IXGBE_ADVTXD_STAT_DD when descriptor is
> taken/released?

For RX and TX, the DD bit is never set by software, only by hardware. When
writing a descriptor to memory for the NIC to read, the DD bit is cleared.
Software knows the NIC has finished with that descriptor by checking for
the DD bit being set by the NIC. If the DD bit is not being set, then the
problem is likely on the NIC side. [Potential software issues that could
cause this might be buffer e.g. overflows where we overwrite a DD bit set
by the NIC, or where we are polling an incorrect address, etc. etc.]

Regards,
/Bruce


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [dpdk-dev]  Questions about TX descriptors run out occasionally
@ 2018-07-30 19:01 刘辉
  0 siblings, 0 replies; 5+ messages in thread
From: 刘辉 @ 2018-07-30 19:01 UTC (permalink / raw)
  To: dev

Hi Experts,

I'm developing my own dpdk-based application via Intel 82599ES port. My Application is doing a job to send ICMP requests (packet size varies from 64 bytes to 1472 bytes, 200,000 pps, 1.1Gbps) and receive responses, with ARP request/response and ICMP response handling when necessary. It was working pretty fine in 5 hours to 10 days  randomly and then TX descriptors run out and cannot be freed by ixgbe_tx_free_bufs() due to DD bit is not set:

        /* check DD bit on threshold descriptor */
        status = txq->tx_ring[txq->tx_next_dd].wb.status;
        if (!(status & IXGBE_ADVTXD_STAT_DD))
                return 0;

My tx queue setup is:
        tx_conf->tx_thresh.pthresh = 64;
        tx_conf->tx_thresh.hthresh = 0;
        tx_conf->tx_thresh.wthresh = 0;
        tx_conf->tx_free_thresh = 256;
        tx_conf->tx_rs_thresh = 32;
        tx_conf->txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS | ETH_TXQ_FLAGS_NOOFFLOADS;


I tried to read code to see if there is any case to take these descriptors and never set IXGBE_ADVTXD_STAT_DD back but no luck yet. And I have not even found the related code when IXGBE_ADVTXD_STAT_DD is set/unset when descriptor is taken/released other than reset queues... So may I ask:
1. where do we set/unset IXGBE_ADVTXD_STAT_DD when descriptor is taken/released?
2. any suggestion or information I should focus on to debug this issue? Is this typically because of my upper application not alloc/free correctly or any other problem?
3. another friend in dpdk-user list raised same issue in fm10k driver, but later he mentioned his problem was because of overheating of NIC (temperature was close to 85 degree Celsius). After setting system FAN to full speed, he made it work perfectly. Since in my system I don't have fan/temp sensors so I could not check this. Might this problem be caused by high temperature in case?

Regards,
Hui 

from Alimail macOS

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [dpdk-dev]  Questions about TX descriptors run out occasionally
@ 2018-07-27 18:15 Hui Liu
  0 siblings, 0 replies; 5+ messages in thread
From: Hui Liu @ 2018-07-27 18:15 UTC (permalink / raw)
  To: dev

Hi Experts,

I'm developing my own dpdk-based application via Intel 82599ES port. My
Application is doing a job to send ICMP requests (packet size varies from
64 bytes to 1472 bytes, 200,000 pps, 1.1Gbps) and receive responses, with
ARP request/response and ICMP response handling when necessary. It was
working pretty fine in 5 hours to 10 days  randomly and then TX descriptors
run out and cannot be freed by ixgbe_tx_free_bufs() due to DD bit is not
set:

        /* check DD bit on threshold descriptor */
        status = txq->tx_ring[txq->tx_next_dd].wb.status;
        if (!(status & IXGBE_ADVTXD_STAT_DD))
                return 0;

My tx queue setup is:
        tx_conf->tx_thresh.pthresh = 64;
        tx_conf->tx_thresh.hthresh = 0;
        tx_conf->tx_thresh.wthresh = 0;
        tx_conf->tx_free_thresh = 256;
        tx_conf->tx_rs_thresh = 32;
        tx_conf->txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
ETH_TXQ_FLAGS_NOOFFLOADS;


I tried to read code to see if there is any case to take these descriptors
and never set IXGBE_ADVTXD_STAT_DD back but no luck yet. And I have not
even found the related code when IXGBE_ADVTXD_STAT_DD is set/unset when
descriptor is taken/released other than reset queues... So may I ask:
1. where do we set/unset IXGBE_ADVTXD_STAT_DD when descriptor is
taken/released?
2. any suggestion or information I should focus on to debug this issue? Is
this typically because of my upper application not alloc/free correctly or
any other problem?
3. another friend in dpdk-user list raised same issue in fm10k driver, but
later he mentioned his problem was because of overheating of NIC
(temperature was close to 85 degree Celsius). After setting system FAN to
full speed, he made it work perfectly. Since in my system I don't have
fan/temp sensors so I could not check this. Might this problem be caused by
high temperature in case?

Regards,
Hui

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2018-08-10 18:42 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-27  2:13 [dpdk-dev] Questions about TX descriptors run out occasionally Hui Liu
2018-08-09  9:31 ` Bruce Richardson
2018-08-10 18:41   ` [dpdk-dev] 回复: " 刘辉
2018-07-27 18:15 [dpdk-dev] " Hui Liu
2018-07-30 19:01 刘辉

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).