* [dpdk-dev] NIC Stops Transmitting @ 2013-07-26 19:39 Scott Talbert 2013-07-26 19:53 ` Stephen Hemminger 0 siblings, 1 reply; 10+ messages in thread From: Scott Talbert @ 2013-07-26 19:39 UTC (permalink / raw) To: dev Hi, I'm writing an application using DPDK that transmits a large number of packets (it doesn't receive any). When I transmit at 2 Gb/sec, everything will run fine for several seconds (receiver is receiving at correct rate), but then the NIC appears to get 'stuck' and doesn't transmit any more packets. In this state, rte_eth_tx_burst() is returning zero (suggesting that there are no available transmit descriptors), but even if I sleep() for a second and try again, rte_eth_tx_burst() still returns 0. It almost appears as if a packet gets stuck in the transmit ring and keeps everything from flowing. I'm using an Intel 82599EB NIC. Does anyone have any ideas of what might be going on? Thanks, Scott ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] NIC Stops Transmitting 2013-07-26 19:39 [dpdk-dev] NIC Stops Transmitting Scott Talbert @ 2013-07-26 19:53 ` Stephen Hemminger 2013-07-26 20:04 ` Scott Talbert 0 siblings, 1 reply; 10+ messages in thread From: Stephen Hemminger @ 2013-07-26 19:53 UTC (permalink / raw) To: Scott Talbert; +Cc: dev On Fri, 26 Jul 2013 15:39:18 -0400 (EDT) Scott Talbert <swt@techie.net> wrote: > Hi, > > I'm writing an application using DPDK that transmits a large number of > packets (it doesn't receive any). When I transmit at 2 Gb/sec, everything > will run fine for several seconds (receiver is receiving at correct rate), > but then the NIC appears to get 'stuck' and doesn't transmit any more > packets. In this state, rte_eth_tx_burst() is returning zero (suggesting > that there are no available transmit descriptors), but even if I sleep() > for a second and try again, rte_eth_tx_burst() still returns 0. It almost > appears as if a packet gets stuck in the transmit ring and keeps > everything from flowing. I'm using an Intel 82599EB NIC. > > Does anyone have any ideas of what might be going on? > > Thanks, > Scott Make sure there is enough memory for mbufs. Also what is your ring size and transmit free threshold? It is easy to instrument the driver to see where it is saying "no space left" Also be careful with threshold values, many values of pthresh/hthresh/wthresh don't work. I would check the Intel reference manual for your hardware. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] NIC Stops Transmitting 2013-07-26 19:53 ` Stephen Hemminger @ 2013-07-26 20:04 ` Scott Talbert 2013-07-26 22:31 ` jinho hwang 0 siblings, 1 reply; 10+ messages in thread From: Scott Talbert @ 2013-07-26 20:04 UTC (permalink / raw) To: Stephen Hemminger; +Cc: dev On Fri, 26 Jul 2013, Stephen Hemminger wrote: >> I'm writing an application using DPDK that transmits a large number of >> packets (it doesn't receive any). When I transmit at 2 Gb/sec, everything >> will run fine for several seconds (receiver is receiving at correct rate), >> but then the NIC appears to get 'stuck' and doesn't transmit any more >> packets. In this state, rte_eth_tx_burst() is returning zero (suggesting >> that there are no available transmit descriptors), but even if I sleep() >> for a second and try again, rte_eth_tx_burst() still returns 0. It almost >> appears as if a packet gets stuck in the transmit ring and keeps >> everything from flowing. I'm using an Intel 82599EB NIC. >> > Make sure there is enough memory for mbufs. > Also what is your ring size and transmit free threshold? > It is easy to instrument the driver to see where it is saying "no space left" > Also be careful with threshold values, many values of pthresh/hthresh/wthresh > don't work. I would check the Intel reference manual for your hardware. Thanks for the tips. I don't think I'm running out of mbufs, but I'll check that again. I am using these values from one of the examples - which claim to be correct for the 82599EB. /* * These default values are optimized for use with the Intel(R) 82599 10 GbE * Controller and the DPDK ixgbe PMD. Consider using other values for other * network controllers and/or network drivers. */ #define TX_PTHRESH 36 /**< Default values of TX prefetch threshold reg. */ #define TX_HTHRESH 0 /**< Default values of TX host threshold reg. */ #define TX_WTHRESH 0 /**< Default values of TX write-back threshold reg. */ static const struct rte_eth_txconf tx_conf = { .tx_thresh = { .pthresh = TX_PTHRESH, .hthresh = TX_HTHRESH, .wthresh = TX_WTHRESH, }, .tx_free_thresh = 0, /* Use PMD default values */ .tx_rs_thresh = 0, /* Use PMD default values */ }; /* * Configurable number of RX/TX ring descriptors */ #define RTE_TEST_TX_DESC_DEFAULT 512 static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT; ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] NIC Stops Transmitting 2013-07-26 20:04 ` Scott Talbert @ 2013-07-26 22:31 ` jinho hwang 2013-07-26 23:26 ` Scott Talbert 0 siblings, 1 reply; 10+ messages in thread From: jinho hwang @ 2013-07-26 22:31 UTC (permalink / raw) To: Scott Talbert; +Cc: dev On Fri, Jul 26, 2013 at 4:04 PM, Scott Talbert <swt@techie.net> wrote: > On Fri, 26 Jul 2013, Stephen Hemminger wrote: > >>> I'm writing an application using DPDK that transmits a large number of >>> packets (it doesn't receive any). When I transmit at 2 Gb/sec, >>> everything >>> will run fine for several seconds (receiver is receiving at correct >>> rate), >>> but then the NIC appears to get 'stuck' and doesn't transmit any more >>> packets. In this state, rte_eth_tx_burst() is returning zero (suggesting >>> that there are no available transmit descriptors), but even if I sleep() >>> for a second and try again, rte_eth_tx_burst() still returns 0. It >>> almost >>> appears as if a packet gets stuck in the transmit ring and keeps >>> everything from flowing. I'm using an Intel 82599EB NIC. >>> >> Make sure there is enough memory for mbufs. >> Also what is your ring size and transmit free threshold? >> It is easy to instrument the driver to see where it is saying "no space >> left" >> Also be careful with threshold values, many values of >> pthresh/hthresh/wthresh >> don't work. I would check the Intel reference manual for your hardware. > > > Thanks for the tips. I don't think I'm running out of mbufs, but I'll check > that again. I am using these values from one of the examples - which claim > to be correct for the 82599EB. > > /* > * These default values are optimized for use with the Intel(R) 82599 10 GbE > * Controller and the DPDK ixgbe PMD. Consider using other values for other > * network controllers and/or network drivers. > */ > #define TX_PTHRESH 36 /**< Default values of TX prefetch threshold reg. */ > #define TX_HTHRESH 0 /**< Default values of TX host threshold reg. */ > #define TX_WTHRESH 0 /**< Default values of TX write-back threshold reg. */ > > static const struct rte_eth_txconf tx_conf = { > .tx_thresh = { > .pthresh = TX_PTHRESH, > .hthresh = TX_HTHRESH, > .wthresh = TX_WTHRESH, > }, > .tx_free_thresh = 0, /* Use PMD default values */ > .tx_rs_thresh = 0, /* Use PMD default values */ > }; > > /* > * Configurable number of RX/TX ring descriptors > */ > #define RTE_TEST_TX_DESC_DEFAULT 512 > static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT; > Scott, I am wondering whether you use multiple cores accessing the same receive queue. I had this problem before, but after I make the same number of receiving queues as the number of receiving cores, the problem disappeared. I did not dig more since I did not care how many receive queues I have did not matter. Jinho ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] NIC Stops Transmitting 2013-07-26 22:31 ` jinho hwang @ 2013-07-26 23:26 ` Scott Talbert 2013-07-29 17:02 ` Scott Talbert 0 siblings, 1 reply; 10+ messages in thread From: Scott Talbert @ 2013-07-26 23:26 UTC (permalink / raw) To: jinho hwang; +Cc: dev On Fri, 26 Jul 2013, jinho hwang wrote: >> Thanks for the tips. I don't think I'm running out of mbufs, but I'll check >> that again. I am using these values from one of the examples - which claim >> to be correct for the 82599EB. >> >> /* >> * These default values are optimized for use with the Intel(R) 82599 10 GbE >> * Controller and the DPDK ixgbe PMD. Consider using other values for other >> * network controllers and/or network drivers. >> */ >> #define TX_PTHRESH 36 /**< Default values of TX prefetch threshold reg. */ >> #define TX_HTHRESH 0 /**< Default values of TX host threshold reg. */ >> #define TX_WTHRESH 0 /**< Default values of TX write-back threshold reg. */ >> >> static const struct rte_eth_txconf tx_conf = { >> .tx_thresh = { >> .pthresh = TX_PTHRESH, >> .hthresh = TX_HTHRESH, >> .wthresh = TX_WTHRESH, >> }, >> .tx_free_thresh = 0, /* Use PMD default values */ >> .tx_rs_thresh = 0, /* Use PMD default values */ >> }; >> >> /* >> * Configurable number of RX/TX ring descriptors >> */ >> #define RTE_TEST_TX_DESC_DEFAULT 512 >> static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT; >> > I am wondering whether you use multiple cores accessing the same > receive queue. I had this problem before, but after I make the same > number of receiving queues as the number of receiving cores, the > problem disappeared. I did not dig more since I did not care how many > receive queues I have did not matter. Jinho, Thanks. I have only one queue (should I be using more?) but as far as I know, I'm only using one core to transmit as well. Scott ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] NIC Stops Transmitting 2013-07-26 23:26 ` Scott Talbert @ 2013-07-29 17:02 ` Scott Talbert 2013-07-31 2:43 ` [dpdk-dev] miss include file in DPDK 1.3.1r2 Jia.Sui 0 siblings, 1 reply; 10+ messages in thread From: Scott Talbert @ 2013-07-29 17:02 UTC (permalink / raw) To: jinho hwang; +Cc: dev On Fri, 26 Jul 2013, Scott Talbert wrote: >> I am wondering whether you use multiple cores accessing the same >> receive queue. I had this problem before, but after I make the same >> number of receiving queues as the number of receiving cores, the >> problem disappeared. I did not dig more since I did not care how many >> receive queues I have did not matter. > > Thanks. I have only one queue (should I be using more?) but as far as I > know, I'm only using one core to transmit as well. All, It turns out that the problem appears to have been caused by me using a signal handler to control my data rates. After switching to use RTE timers instead, I am able to transmit close to 10 Gb/sec reliably. Thanks, Scott ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] miss include file in DPDK 1.3.1r2 2013-07-29 17:02 ` Scott Talbert @ 2013-07-31 2:43 ` Jia.Sui 2013-07-31 9:18 ` [dpdk-dev] [PATCH] mem: fix include in rte_malloc Thomas Monjalon 0 siblings, 1 reply; 10+ messages in thread From: Jia.Sui(贾睢) @ 2013-07-31 2:43 UTC (permalink / raw) To: dev Hi All, In DPDK 1.3.1r2, When I include 'rte_malloc.h' in my app, and then compile it Will cause this issue: /root/dpdk/x86_64-default-linuxapp-gcc/include/rte_malloc.h:333: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘rte_malloc_virt2phy’ It looks it miss the definition of 'phys_addr_t' which is defined in 'rte_memory.h' Thanks BRs Jia Sui ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH] mem: fix include in rte_malloc 2013-07-31 2:43 ` [dpdk-dev] miss include file in DPDK 1.3.1r2 Jia.Sui @ 2013-07-31 9:18 ` Thomas Monjalon 2013-07-31 11:36 ` didier.pallard 0 siblings, 1 reply; 10+ messages in thread From: Thomas Monjalon @ 2013-07-31 9:18 UTC (permalink / raw) To: dev The function rte_malloc_virt2phy has a dependency on rte_memory.h: phys_addr_t must be defined. The dependency handling for apps was wrong in the commit 8c86825cbf. Let's revert this part. Reported-by: Jia Sui <jia.sui@advantech.com.cn> Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com> --- app/test/test_hash.c | 2 +- app/test/test_hash_perf.c | 2 +- app/test/test_memcpy.c | 1 - app/test/test_memcpy_perf.c | 1 - lib/librte_malloc/rte_malloc.h | 1 + 5 files changed, 3 insertions(+), 4 deletions(-) diff --git a/app/test/test_hash.c b/app/test/test_hash.c index 4de8cb1..5437e05 100644 --- a/app/test/test_hash.c +++ b/app/test/test_hash.c @@ -41,10 +41,10 @@ #include <sys/queue.h> #include <rte_common.h> +#include <rte_malloc.h> #include <rte_cycles.h> #include <rte_random.h> #include <rte_memory.h> -#include <rte_malloc.h> #include <rte_memzone.h> #include <rte_tailq.h> #include <rte_eal.h> diff --git a/app/test/test_hash_perf.c b/app/test/test_hash_perf.c index 311c2bd..8a0feb3 100644 --- a/app/test/test_hash_perf.c +++ b/app/test/test_hash_perf.c @@ -42,10 +42,10 @@ #include <rte_common.h> #include <rte_lcore.h> +#include <rte_malloc.h> #include <rte_cycles.h> #include <rte_random.h> #include <rte_memory.h> -#include <rte_malloc.h> #include <rte_memzone.h> #include <rte_tailq.h> #include <rte_eal.h> diff --git a/app/test/test_memcpy.c b/app/test/test_memcpy.c index 729a2ff..d7777a8 100644 --- a/app/test/test_memcpy.c +++ b/app/test/test_memcpy.c @@ -41,7 +41,6 @@ #include <cmdline_parse.h> #include <rte_cycles.h> #include <rte_random.h> -#include <rte_memory.h> #include <rte_malloc.h> #include <rte_memcpy.h> diff --git a/app/test/test_memcpy_perf.c b/app/test/test_memcpy_perf.c index 1e75f53..236b295 100644 --- a/app/test/test_memcpy_perf.c +++ b/app/test/test_memcpy_perf.c @@ -41,7 +41,6 @@ #include <cmdline_parse.h> #include <rte_cycles.h> #include <rte_random.h> -#include <rte_memory.h> #include <rte_malloc.h> #include <rte_memcpy.h> diff --git a/lib/librte_malloc/rte_malloc.h b/lib/librte_malloc/rte_malloc.h index e68187a..fe85689 100644 --- a/lib/librte_malloc/rte_malloc.h +++ b/lib/librte_malloc/rte_malloc.h @@ -42,6 +42,7 @@ */ #include <stddef.h> +#include <rte_memory.h> #ifdef __cplusplus extern "C" { -- 1.7.10.4 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] mem: fix include in rte_malloc 2013-07-31 9:18 ` [dpdk-dev] [PATCH] mem: fix include in rte_malloc Thomas Monjalon @ 2013-07-31 11:36 ` didier.pallard 2013-07-31 11:48 ` Thomas Monjalon 0 siblings, 1 reply; 10+ messages in thread From: didier.pallard @ 2013-07-31 11:36 UTC (permalink / raw) To: Thomas Monjalon; +Cc: dev On 07/31/2013 11:18 AM, Thomas Monjalon wrote: > The function rte_malloc_virt2phy has a dependency on rte_memory.h: > phys_addr_t must be defined. > > The dependency handling for apps was wrong in the commit 8c86825cbf. > Let's revert this part. > > Reported-by: Jia Sui <jia.sui@advantech.com.cn> > Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com> Acked-by: Didier Pallard <didier.pallard@6wind.com> ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] mem: fix include in rte_malloc 2013-07-31 11:36 ` didier.pallard @ 2013-07-31 11:48 ` Thomas Monjalon 0 siblings, 0 replies; 10+ messages in thread From: Thomas Monjalon @ 2013-07-31 11:48 UTC (permalink / raw) To: Didier Pallard, Jia Sui; +Cc: dev 31/07/2013 13:36, didier.pallard : > On 07/31/2013 11:18 AM, Thomas Monjalon wrote: > > The function rte_malloc_virt2phy has a dependency on rte_memory.h: > > phys_addr_t must be defined. > > > > The dependency handling for apps was wrong in the commit 8c86825cbf. > > Let's revert this part. > > > > Reported-by: Jia Sui <jia.sui@advantech.com.cn> > > Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com> > > Acked-by: Didier Pallard <didier.pallard@6wind.com> pushed Thanks to Jia Sui for reporting. Side note: please do not reply to an old email when creating new thread. (see http://dpdk.org/ml/archives/dev/2013-July/thread.html) -- Thomas ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2013-07-31 11:47 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-07-26 19:39 [dpdk-dev] NIC Stops Transmitting Scott Talbert 2013-07-26 19:53 ` Stephen Hemminger 2013-07-26 20:04 ` Scott Talbert 2013-07-26 22:31 ` jinho hwang 2013-07-26 23:26 ` Scott Talbert 2013-07-29 17:02 ` Scott Talbert 2013-07-31 2:43 ` [dpdk-dev] miss include file in DPDK 1.3.1r2 Jia.Sui 2013-07-31 9:18 ` [dpdk-dev] [PATCH] mem: fix include in rte_malloc Thomas Monjalon 2013-07-31 11:36 ` didier.pallard 2013-07-31 11:48 ` Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).