[dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02

[dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02

[Jiang Huiyou]

Hi,
     My user-space TCP/IP stack  works fine on DPDK 17.02, now I upgrade it to 18.02. And then I use  wrk http benchmark to test my stack, it work well fine, but after a while, it couldn't send packet.  I debug with gdb and find the rte_eth_tx_burst always return 0, which means there is no available transmit descriptor. And then I enable DPDK debug mode, the ixgbe_xmit_pkts has been called to send packets (My NIC is Intel 82599ES 10G) and it called ixgbe_xmit_cleanup to recycle  used transmit descriptor, but ixgbe_xmit_cleanup always return -1. The relevant code is blew.

/* Check to make sure the last descriptor to clean is done */
desc_to_clean_to = sw_ring[desc_to_clean_to].last_id;
status = txr[desc_to_clean_to].wb.status;
if (!(status & rte_cpu_to_le_32(IXGBE_TXD_STAT_DD))) {
 PMD_TX_FREE_LOG(DEBUG,
   "TX descriptor %4u is not done"
   "(port=%d queue=%d)",
   desc_to_clean_to,
   txq->port_id, txq->queue_id);
 /* Failed to clean any descriptors, better luck next time */
 return -(1);
}

I'm stuck here, and wonder how DPDK upgrade brings up this issue. Btw, there was a similar mail which encountered the same issue, http://mails.dpdk.org/archives/dev/2017-August/073240.html, but I find no final solution for it.

Thanks a lot for your time, and dying for your reply!

Huiyou

Re: [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02

[Wiles, Keith]

Sent from my iPhone

> On Nov 7, 2018, at 2:36 AM, Jiang Huiyou <jhy19941008@hotmail.com> wrote:
> 
> Hi,
>     My user-space TCP/IP stack  works fine on DPDK 17.02, now I upgrade it to 18.02. And then I use  wrk http benchmark to test my stack, it work well fine, but after a while, it couldn't send packet.  I debug with gdb and find the rte_eth_tx_burst always return 0, which means there is no available transmit descriptor. And then I enable DPDK debug mode, the ixgbe_xmit_pkts has been called to send packets (My NIC is Intel 82599ES 10G) and it called ixgbe_xmit_cleanup to recycle  used transmit descriptor, but ixgbe_xmit_cleanup always return -1. The relevant code is blew.
> 

Can you make sure the tx flush threshold?

Some of the rx/tx rings were increased and if you do not have enough transmit traffic it can seem like it never does a cleanup. This is just guess try to make sure you have enough tx buffers allocated and rx as well. 
Hope that helps
> /* Check to make sure the last descriptor to clean is done */
> desc_to_clean_to = sw_ring[desc_to_clean_to].last_id;
> status = txr[desc_to_clean_to].wb.status;
> if (!(status & rte_cpu_to_le_32(IXGBE_TXD_STAT_DD))) {
> PMD_TX_FREE_LOG(DEBUG,
>   "TX descriptor %4u is not done"
>   "(port=%d queue=%d)",
>   desc_to_clean_to,
>   txq->port_id, txq->queue_id);
> /* Failed to clean any descriptors, better luck next time */
> return -(1);
> }
> 
> I'm stuck here, and wonder how DPDK upgrade brings up this issue. Btw, there was a similar mail which encountered the same issue, http://mails.dpdk.org/archives/dev/2017-August/073240.html, but I find no final solution for it.
> 
> Thanks a lot for your time, and dying for your reply!
> 
> Huiyou
> 

[dpdk-dev] 答复: rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02

[Jiang Huiyou]

Hi Wiles,

    Thanks a lot for your reply, I'm not sure what you said "tx flush threshold" exactly is. Is it in parameter of rte_eth_tx_queue_setup function to init tx queue? If so,  it's configured as

static struct rte_eth_txconf tx_conf =
{
.tx_thresh = {
 .pthresh = TX_PTHRESH,  // 36
 .hthresh = TX_HTHRESH,  // 0
 .wthresh = TX_WTHRESH,  // 0
},
.tx_free_thresh = MAX_PKT_BURST, // Use PMD default values, supposed to be 32
.tx_rs_thresh = MAX_PKT_BURST, // Use PMD default values
.txq_flags = (uint32_t)ETH_TXQ_FLAGS_NOOFFLOADS,
};

And the transmit ring size is 4096.

As for network traffic, I use the http benchmark tool called wrk to perform stress testing for nginx, which running on my user-space TCP/IP stacks. I think the traffic is absolutely enough, and the issue occurs during the stress testing.

Thanks again for your reply.
Best wishes
________________________________
发件人: Wiles, Keith <keith.wiles@intel.com>
发送时间: 2018年11月9日 15:37
收件人: Jiang Huiyou
抄送: dev@dpdk.org; Zhang, Qi Z
主题: Re: [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02



Sent from my iPhone

> On Nov 7, 2018, at 2:36 AM, Jiang Huiyou <jhy19941008@hotmail.com> wrote:
>
> Hi,
>     My user-space TCP/IP stack  works fine on DPDK 17.02, now I upgrade it to 18.02. And then I use  wrk http benchmark to test my stack, it work well fine, but after a while, it couldn't send packet.  I debug with gdb and find the rte_eth_tx_burst always return 0, which means there is no available transmit descriptor. And then I enable DPDK debug mode, the ixgbe_xmit_pkts has been called to send packets (My NIC is Intel 82599ES 10G) and it called ixgbe_xmit_cleanup to recycle  used transmit descriptor, but ixgbe_xmit_cleanup always return -1. The relevant code is blew.
>

Can you make sure the tx flush threshold?

Some of the rx/tx rings were increased and if you do not have enough transmit traffic it can seem like it never does a cleanup. This is just guess try to make sure you have enough tx buffers allocated and rx as well.
Hope that helps
> /* Check to make sure the last descriptor to clean is done */
> desc_to_clean_to = sw_ring[desc_to_clean_to].last_id;
> status = txr[desc_to_clean_to].wb.status;
> if (!(status & rte_cpu_to_le_32(IXGBE_TXD_STAT_DD))) {
> PMD_TX_FREE_LOG(DEBUG,
>   "TX descriptor %4u is not done"
>   "(port=%d queue=%d)",
>   desc_to_clean_to,
>   txq->port_id, txq->queue_id);
> /* Failed to clean any descriptors, better luck next time */
> return -(1);
> }
>
> I'm stuck here, and wonder how DPDK upgrade brings up this issue. Btw, there was a similar mail which encountered the same issue, http://mails.dpdk.org/archives/dev/2017-August/073240.html, but I find no final solution for it.
>
> Thanks a lot for your time, and dying for your reply!
>
> Huiyou
>

Re: [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02

[Wiles, Keith]

On Nov 9, 2018, at 3:18 AM, Jiang Huiyou <jhy19941008@hotmail.com<mailto:jhy19941008@hotmail.com>> wrote:

Hi Wiles,

    Thanks a lot for your reply, I'm not sure what you said "tx flush threshold" exactly is. Is it in parameter of rte_eth_tx_queue_setup function to init tx queue? If so,  it's configured as

Sorry, for the late reply.

static struct rte_eth_txconf tx_conf =
{
.tx_thresh = {
 .pthresh = TX_PTHRESH,  // 36
 .hthresh = TX_HTHRESH,  // 0
 .wthresh = TX_WTHRESH,  // 0
},
.tx_free_thresh = MAX_PKT_BURST, // Use PMD default values, supposed to be 32
.tx_rs_thresh = MAX_PKT_BURST, // Use PMD default values
.txq_flags = (uint32_t)ETH_TXQ_FLAGS_NOOFFLOADS,
};

And the transmit ring size is 4096.

As for network traffic, I use the http benchmark tool called wrk to perform stress testing for nginx, which running on my user-space TCP/IP stacks. I think the traffic is absolutely enough, and the issue occurs during the stress testing.

Some how the TX buffers are not getting freed is my, using the rte_mempool_dump() routine to see if you have any TX buffers, plus it will show you were in the mempool any free buffers are located.

For pktmbuf pool is just a mempool with a wrapper on top for the mbufs. If the mempool is basically empty then you have a memory leak some place for the TX mbufs. Another place to check in when you do the rte_eth_tx_burst() call is to check the return value and see if some of the TX mbufs are not being sent. If that is case then you  have to free the non-sent mbufs or retry until they are sent.


Thanks again for your reply.
Best wishes
________________________________
发件人: Wiles, Keith <keith.wiles@intel.com<mailto:keith.wiles@intel.com>>
发送时间: 2018年11月9日 15:37
收件人: Jiang Huiyou
抄送: dev@dpdk.org<mailto:dev@dpdk.org>; Zhang, Qi Z
主题: Re: [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02



Sent from my iPhone

> On Nov 7, 2018, at 2:36 AM, Jiang Huiyou <jhy19941008@hotmail.com<mailto:jhy19941008@hotmail.com>> wrote:
>
> Hi,
>     My user-space TCP/IP stack  works fine on DPDK 17.02, now I upgrade it to 18.02. And then I use  wrk http benchmark to test my stack, it work well fine, but after a while, it couldn't send packet.  I debug with gdb and find the rte_eth_tx_burst always return 0, which means there is no available transmit descriptor. And then I enable DPDK debug mode, the ixgbe_xmit_pkts has been called to send packets (My NIC is Intel 82599ES 10G) and it called ixgbe_xmit_cleanup to recycle  used transmit descriptor, but ixgbe_xmit_cleanup always return -1. The relevant code is blew.
>

Can you make sure the tx flush threshold?

Some of the rx/tx rings were increased and if you do not have enough transmit traffic it can seem like it never does a cleanup. This is just guess try to make sure you have enough tx buffers allocated and rx as well.
Hope that helps
> /* Check to make sure the last descriptor to clean is done */
> desc_to_clean_to = sw_ring[desc_to_clean_to].last_id;
> status = txr[desc_to_clean_to].wb.status;
> if (!(status & rte_cpu_to_le_32(IXGBE_TXD_STAT_DD))) {
> PMD_TX_FREE_LOG(DEBUG,
>   "TX descriptor %4u is not done"
>   "(port=%d queue=%d)",
>   desc_to_clean_to,
>   txq->port_id, txq->queue_id);
> /* Failed to clean any descriptors, better luck next time */
> return -(1);
> }
>
> I'm stuck here, and wonder how DPDK upgrade brings up this issue. Btw, there was a similar mail which encountered the same issue,http://mails.dpdk.org/archives/dev/2017-August/073240.html, but I find no final solution for it.
>
> Thanks a lot for your time, and dying for your reply!
>
> Huiyou
>

Regards,
Keith