* [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02 @ 2018-11-07 2:36 Jiang Huiyou 2018-11-09 7:37 ` Wiles, Keith 0 siblings, 1 reply; 4+ messages in thread From: Jiang Huiyou @ 2018-11-07 2:36 UTC (permalink / raw) To: dev; +Cc: qi.z.zhang Hi, My user-space TCP/IP stack works fine on DPDK 17.02, now I upgrade it to 18.02. And then I use wrk http benchmark to test my stack, it work well fine, but after a while, it couldn't send packet. I debug with gdb and find the rte_eth_tx_burst always return 0, which means there is no available transmit descriptor. And then I enable DPDK debug mode, the ixgbe_xmit_pkts has been called to send packets (My NIC is Intel 82599ES 10G) and it called ixgbe_xmit_cleanup to recycle used transmit descriptor, but ixgbe_xmit_cleanup always return -1. The relevant code is blew. /* Check to make sure the last descriptor to clean is done */ desc_to_clean_to = sw_ring[desc_to_clean_to].last_id; status = txr[desc_to_clean_to].wb.status; if (!(status & rte_cpu_to_le_32(IXGBE_TXD_STAT_DD))) { PMD_TX_FREE_LOG(DEBUG, "TX descriptor %4u is not done" "(port=%d queue=%d)", desc_to_clean_to, txq->port_id, txq->queue_id); /* Failed to clean any descriptors, better luck next time */ return -(1); } I'm stuck here, and wonder how DPDK upgrade brings up this issue. Btw, there was a similar mail which encountered the same issue, http://mails.dpdk.org/archives/dev/2017-August/073240.html, but I find no final solution for it. Thanks a lot for your time, and dying for your reply! Huiyou ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02 2018-11-07 2:36 [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02 Jiang Huiyou @ 2018-11-09 7:37 ` Wiles, Keith 2018-11-09 9:18 ` [dpdk-dev] 答复: " Jiang Huiyou 0 siblings, 1 reply; 4+ messages in thread From: Wiles, Keith @ 2018-11-09 7:37 UTC (permalink / raw) To: Jiang Huiyou; +Cc: dev, Zhang, Qi Z Sent from my iPhone > On Nov 7, 2018, at 2:36 AM, Jiang Huiyou <jhy19941008@hotmail.com> wrote: > > Hi, > My user-space TCP/IP stack works fine on DPDK 17.02, now I upgrade it to 18.02. And then I use wrk http benchmark to test my stack, it work well fine, but after a while, it couldn't send packet. I debug with gdb and find the rte_eth_tx_burst always return 0, which means there is no available transmit descriptor. And then I enable DPDK debug mode, the ixgbe_xmit_pkts has been called to send packets (My NIC is Intel 82599ES 10G) and it called ixgbe_xmit_cleanup to recycle used transmit descriptor, but ixgbe_xmit_cleanup always return -1. The relevant code is blew. > Can you make sure the tx flush threshold? Some of the rx/tx rings were increased and if you do not have enough transmit traffic it can seem like it never does a cleanup. This is just guess try to make sure you have enough tx buffers allocated and rx as well. Hope that helps > /* Check to make sure the last descriptor to clean is done */ > desc_to_clean_to = sw_ring[desc_to_clean_to].last_id; > status = txr[desc_to_clean_to].wb.status; > if (!(status & rte_cpu_to_le_32(IXGBE_TXD_STAT_DD))) { > PMD_TX_FREE_LOG(DEBUG, > "TX descriptor %4u is not done" > "(port=%d queue=%d)", > desc_to_clean_to, > txq->port_id, txq->queue_id); > /* Failed to clean any descriptors, better luck next time */ > return -(1); > } > > I'm stuck here, and wonder how DPDK upgrade brings up this issue. Btw, there was a similar mail which encountered the same issue, http://mails.dpdk.org/archives/dev/2017-August/073240.html, but I find no final solution for it. > > Thanks a lot for your time, and dying for your reply! > > Huiyou > ^ permalink raw reply [flat|nested] 4+ messages in thread
* [dpdk-dev] 答复: rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02 2018-11-09 7:37 ` Wiles, Keith @ 2018-11-09 9:18 ` Jiang Huiyou 2018-11-30 20:28 ` [dpdk-dev] " Wiles, Keith 0 siblings, 1 reply; 4+ messages in thread From: Jiang Huiyou @ 2018-11-09 9:18 UTC (permalink / raw) To: Wiles, Keith; +Cc: dev, Zhang, Qi Z Hi Wiles, Thanks a lot for your reply, I'm not sure what you said "tx flush threshold" exactly is. Is it in parameter of rte_eth_tx_queue_setup function to init tx queue? If so, it's configured as static struct rte_eth_txconf tx_conf = { .tx_thresh = { .pthresh = TX_PTHRESH, // 36 .hthresh = TX_HTHRESH, // 0 .wthresh = TX_WTHRESH, // 0 }, .tx_free_thresh = MAX_PKT_BURST, // Use PMD default values, supposed to be 32 .tx_rs_thresh = MAX_PKT_BURST, // Use PMD default values .txq_flags = (uint32_t)ETH_TXQ_FLAGS_NOOFFLOADS, }; And the transmit ring size is 4096. As for network traffic, I use the http benchmark tool called wrk to perform stress testing for nginx, which running on my user-space TCP/IP stacks. I think the traffic is absolutely enough, and the issue occurs during the stress testing. Thanks again for your reply. Best wishes ________________________________ 发件人: Wiles, Keith <keith.wiles@intel.com> 发送时间: 2018年11月9日 15:37 收件人: Jiang Huiyou 抄送: dev@dpdk.org; Zhang, Qi Z 主题: Re: [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02 Sent from my iPhone > On Nov 7, 2018, at 2:36 AM, Jiang Huiyou <jhy19941008@hotmail.com> wrote: > > Hi, > My user-space TCP/IP stack works fine on DPDK 17.02, now I upgrade it to 18.02. And then I use wrk http benchmark to test my stack, it work well fine, but after a while, it couldn't send packet. I debug with gdb and find the rte_eth_tx_burst always return 0, which means there is no available transmit descriptor. And then I enable DPDK debug mode, the ixgbe_xmit_pkts has been called to send packets (My NIC is Intel 82599ES 10G) and it called ixgbe_xmit_cleanup to recycle used transmit descriptor, but ixgbe_xmit_cleanup always return -1. The relevant code is blew. > Can you make sure the tx flush threshold? Some of the rx/tx rings were increased and if you do not have enough transmit traffic it can seem like it never does a cleanup. This is just guess try to make sure you have enough tx buffers allocated and rx as well. Hope that helps > /* Check to make sure the last descriptor to clean is done */ > desc_to_clean_to = sw_ring[desc_to_clean_to].last_id; > status = txr[desc_to_clean_to].wb.status; > if (!(status & rte_cpu_to_le_32(IXGBE_TXD_STAT_DD))) { > PMD_TX_FREE_LOG(DEBUG, > "TX descriptor %4u is not done" > "(port=%d queue=%d)", > desc_to_clean_to, > txq->port_id, txq->queue_id); > /* Failed to clean any descriptors, better luck next time */ > return -(1); > } > > I'm stuck here, and wonder how DPDK upgrade brings up this issue. Btw, there was a similar mail which encountered the same issue, http://mails.dpdk.org/archives/dev/2017-August/073240.html, but I find no final solution for it. > > Thanks a lot for your time, and dying for your reply! > > Huiyou > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02 2018-11-09 9:18 ` [dpdk-dev] 答复: " Jiang Huiyou @ 2018-11-30 20:28 ` Wiles, Keith 0 siblings, 0 replies; 4+ messages in thread From: Wiles, Keith @ 2018-11-30 20:28 UTC (permalink / raw) To: Jiang Huiyou; +Cc: dev, Zhang, Qi Z On Nov 9, 2018, at 3:18 AM, Jiang Huiyou <jhy19941008@hotmail.com<mailto:jhy19941008@hotmail.com>> wrote: Hi Wiles, Thanks a lot for your reply, I'm not sure what you said "tx flush threshold" exactly is. Is it in parameter of rte_eth_tx_queue_setup function to init tx queue? If so, it's configured as Sorry, for the late reply. static struct rte_eth_txconf tx_conf = { .tx_thresh = { .pthresh = TX_PTHRESH, // 36 .hthresh = TX_HTHRESH, // 0 .wthresh = TX_WTHRESH, // 0 }, .tx_free_thresh = MAX_PKT_BURST, // Use PMD default values, supposed to be 32 .tx_rs_thresh = MAX_PKT_BURST, // Use PMD default values .txq_flags = (uint32_t)ETH_TXQ_FLAGS_NOOFFLOADS, }; And the transmit ring size is 4096. As for network traffic, I use the http benchmark tool called wrk to perform stress testing for nginx, which running on my user-space TCP/IP stacks. I think the traffic is absolutely enough, and the issue occurs during the stress testing. Some how the TX buffers are not getting freed is my, using the rte_mempool_dump() routine to see if you have any TX buffers, plus it will show you were in the mempool any free buffers are located. For pktmbuf pool is just a mempool with a wrapper on top for the mbufs. If the mempool is basically empty then you have a memory leak some place for the TX mbufs. Another place to check in when you do the rte_eth_tx_burst() call is to check the return value and see if some of the TX mbufs are not being sent. If that is case then you have to free the non-sent mbufs or retry until they are sent. Thanks again for your reply. Best wishes ________________________________ 发件人: Wiles, Keith <keith.wiles@intel.com<mailto:keith.wiles@intel.com>> 发送时间: 2018年11月9日 15:37 收件人: Jiang Huiyou 抄送: dev@dpdk.org<mailto:dev@dpdk.org>; Zhang, Qi Z 主题: Re: [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02 Sent from my iPhone > On Nov 7, 2018, at 2:36 AM, Jiang Huiyou <jhy19941008@hotmail.com<mailto:jhy19941008@hotmail.com>> wrote: > > Hi, > My user-space TCP/IP stack works fine on DPDK 17.02, now I upgrade it to 18.02. And then I use wrk http benchmark to test my stack, it work well fine, but after a while, it couldn't send packet. I debug with gdb and find the rte_eth_tx_burst always return 0, which means there is no available transmit descriptor. And then I enable DPDK debug mode, the ixgbe_xmit_pkts has been called to send packets (My NIC is Intel 82599ES 10G) and it called ixgbe_xmit_cleanup to recycle used transmit descriptor, but ixgbe_xmit_cleanup always return -1. The relevant code is blew. > Can you make sure the tx flush threshold? Some of the rx/tx rings were increased and if you do not have enough transmit traffic it can seem like it never does a cleanup. This is just guess try to make sure you have enough tx buffers allocated and rx as well. Hope that helps > /* Check to make sure the last descriptor to clean is done */ > desc_to_clean_to = sw_ring[desc_to_clean_to].last_id; > status = txr[desc_to_clean_to].wb.status; > if (!(status & rte_cpu_to_le_32(IXGBE_TXD_STAT_DD))) { > PMD_TX_FREE_LOG(DEBUG, > "TX descriptor %4u is not done" > "(port=%d queue=%d)", > desc_to_clean_to, > txq->port_id, txq->queue_id); > /* Failed to clean any descriptors, better luck next time */ > return -(1); > } > > I'm stuck here, and wonder how DPDK upgrade brings up this issue. Btw, there was a similar mail which encountered the same issue,http://mails.dpdk.org/archives/dev/2017-August/073240.html, but I find no final solution for it. > > Thanks a lot for your time, and dying for your reply! > > Huiyou > Regards, Keith ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2018-11-30 20:28 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-11-07 2:36 [dpdk-dev] rte_eth_tx_burst send packet fail when upgrade dpdk from 17.02 to 18.02 Jiang Huiyou 2018-11-09 7:37 ` Wiles, Keith 2018-11-09 9:18 ` [dpdk-dev] 答复: " Jiang Huiyou 2018-11-30 20:28 ` [dpdk-dev] " Wiles, Keith
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).