[dpdk-dev] NIC Stops Transmitting

[dpdk-dev] NIC Stops Transmitting

[Scott Talbert]

Hi,

I'm writing an application using DPDK that transmits a large number of 
packets (it doesn't receive any).  When I transmit at 2 Gb/sec, everything 
will run fine for several seconds (receiver is receiving at correct rate), 
but then the NIC appears to get 'stuck' and doesn't transmit any more 
packets.  In this state, rte_eth_tx_burst() is returning zero (suggesting 
that there are no available transmit descriptors), but even if I sleep() 
for a second and try again, rte_eth_tx_burst() still returns 0.  It almost 
appears as if a packet gets stuck in the transmit ring and keeps 
everything from flowing.  I'm using an Intel 82599EB NIC.

Does anyone have any ideas of what might be going on?

Thanks,
Scott

Re: [dpdk-dev] NIC Stops Transmitting

[Stephen Hemminger]

On Fri, 26 Jul 2013 15:39:18 -0400 (EDT)
Scott Talbert <swt@techie.net> wrote:

> Hi,
> 
> I'm writing an application using DPDK that transmits a large number of 
> packets (it doesn't receive any).  When I transmit at 2 Gb/sec, everything 
> will run fine for several seconds (receiver is receiving at correct rate), 
> but then the NIC appears to get 'stuck' and doesn't transmit any more 
> packets.  In this state, rte_eth_tx_burst() is returning zero (suggesting 
> that there are no available transmit descriptors), but even if I sleep() 
> for a second and try again, rte_eth_tx_burst() still returns 0.  It almost 
> appears as if a packet gets stuck in the transmit ring and keeps 
> everything from flowing.  I'm using an Intel 82599EB NIC.
> 
> Does anyone have any ideas of what might be going on?
> 
> Thanks,
> Scott

Make sure there is enough memory for mbufs.
Also what is your ring size and transmit free threshold?
It is easy to instrument the driver to see where it is saying "no space left"
Also be careful with threshold values, many values of pthresh/hthresh/wthresh
don't work. I would check the Intel reference manual for your hardware.

Re: [dpdk-dev] NIC Stops Transmitting

[Scott Talbert]

On Fri, 26 Jul 2013, Stephen Hemminger wrote:

>> I'm writing an application using DPDK that transmits a large number of
>> packets (it doesn't receive any).  When I transmit at 2 Gb/sec, everything
>> will run fine for several seconds (receiver is receiving at correct rate),
>> but then the NIC appears to get 'stuck' and doesn't transmit any more
>> packets.  In this state, rte_eth_tx_burst() is returning zero (suggesting
>> that there are no available transmit descriptors), but even if I sleep()
>> for a second and try again, rte_eth_tx_burst() still returns 0.  It almost
>> appears as if a packet gets stuck in the transmit ring and keeps
>> everything from flowing.  I'm using an Intel 82599EB NIC.
>>
> Make sure there is enough memory for mbufs.
> Also what is your ring size and transmit free threshold?
> It is easy to instrument the driver to see where it is saying "no space left"
> Also be careful with threshold values, many values of pthresh/hthresh/wthresh
> don't work. I would check the Intel reference manual for your hardware.

Thanks for the tips.  I don't think I'm running out of mbufs, but I'll 
check that again.  I am using these values from one of the examples - 
which claim to be correct for the 82599EB.

/*
  * These default values are optimized for use with the Intel(R) 82599 10 
GbE
  * Controller and the DPDK ixgbe PMD. Consider using other values for 
other
  * network controllers and/or network drivers.
  */
#define TX_PTHRESH 36 /**< Default values of TX prefetch threshold reg. */
#define TX_HTHRESH 0  /**< Default values of TX host threshold reg. */
#define TX_WTHRESH 0  /**< Default values of TX write-back threshold reg. 
*/

static const struct rte_eth_txconf tx_conf = {
     .tx_thresh = {
         .pthresh = TX_PTHRESH,
         .hthresh = TX_HTHRESH,
         .wthresh = TX_WTHRESH,
     },
     .tx_free_thresh = 0, /* Use PMD default values */
     .tx_rs_thresh = 0, /* Use PMD default values */
};

/*
  * Configurable number of RX/TX ring descriptors
  */
#define RTE_TEST_TX_DESC_DEFAULT 512
static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT;

Re: [dpdk-dev] NIC Stops Transmitting

[jinho hwang]

On Fri, Jul 26, 2013 at 4:04 PM, Scott Talbert <swt@techie.net> wrote:
> On Fri, 26 Jul 2013, Stephen Hemminger wrote:
>
>>> I'm writing an application using DPDK that transmits a large number of
>>> packets (it doesn't receive any).  When I transmit at 2 Gb/sec,
>>> everything
>>> will run fine for several seconds (receiver is receiving at correct
>>> rate),
>>> but then the NIC appears to get 'stuck' and doesn't transmit any more
>>> packets.  In this state, rte_eth_tx_burst() is returning zero (suggesting
>>> that there are no available transmit descriptors), but even if I sleep()
>>> for a second and try again, rte_eth_tx_burst() still returns 0.  It
>>> almost
>>> appears as if a packet gets stuck in the transmit ring and keeps
>>> everything from flowing.  I'm using an Intel 82599EB NIC.
>>>
>> Make sure there is enough memory for mbufs.
>> Also what is your ring size and transmit free threshold?
>> It is easy to instrument the driver to see where it is saying "no space
>> left"
>> Also be careful with threshold values, many values of
>> pthresh/hthresh/wthresh
>> don't work. I would check the Intel reference manual for your hardware.
>
>
> Thanks for the tips.  I don't think I'm running out of mbufs, but I'll check
> that again.  I am using these values from one of the examples - which claim
> to be correct for the 82599EB.
>
> /*
>  * These default values are optimized for use with the Intel(R) 82599 10 GbE
>  * Controller and the DPDK ixgbe PMD. Consider using other values for other
>  * network controllers and/or network drivers.
>  */
> #define TX_PTHRESH 36 /**< Default values of TX prefetch threshold reg. */
> #define TX_HTHRESH 0  /**< Default values of TX host threshold reg. */
> #define TX_WTHRESH 0  /**< Default values of TX write-back threshold reg. */
>
> static const struct rte_eth_txconf tx_conf = {
>     .tx_thresh = {
>         .pthresh = TX_PTHRESH,
>         .hthresh = TX_HTHRESH,
>         .wthresh = TX_WTHRESH,
>     },
>     .tx_free_thresh = 0, /* Use PMD default values */
>     .tx_rs_thresh = 0, /* Use PMD default values */
> };
>
> /*
>  * Configurable number of RX/TX ring descriptors
>  */
> #define RTE_TEST_TX_DESC_DEFAULT 512
> static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT;
>

Scott,

I am wondering whether you use multiple cores accessing the same
receive queue. I had this problem before, but after I make the same
number of receiving queues as the number of receiving cores, the
problem disappeared. I did not dig more since I did not care how many
receive queues I have did not matter.

Jinho

Re: [dpdk-dev] NIC Stops Transmitting

[Scott Talbert]

On Fri, 26 Jul 2013, jinho hwang wrote:

>> Thanks for the tips.  I don't think I'm running out of mbufs, but I'll check
>> that again.  I am using these values from one of the examples - which claim
>> to be correct for the 82599EB.
>>
>> /*
>>  * These default values are optimized for use with the Intel(R) 82599 10 GbE
>>  * Controller and the DPDK ixgbe PMD. Consider using other values for other
>>  * network controllers and/or network drivers.
>>  */
>> #define TX_PTHRESH 36 /**< Default values of TX prefetch threshold reg. */
>> #define TX_HTHRESH 0  /**< Default values of TX host threshold reg. */
>> #define TX_WTHRESH 0  /**< Default values of TX write-back threshold reg. */
>>
>> static const struct rte_eth_txconf tx_conf = {
>>     .tx_thresh = {
>>         .pthresh = TX_PTHRESH,
>>         .hthresh = TX_HTHRESH,
>>         .wthresh = TX_WTHRESH,
>>     },
>>     .tx_free_thresh = 0, /* Use PMD default values */
>>     .tx_rs_thresh = 0, /* Use PMD default values */
>> };
>>
>> /*
>>  * Configurable number of RX/TX ring descriptors
>>  */
>> #define RTE_TEST_TX_DESC_DEFAULT 512
>> static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT;
>>
> I am wondering whether you use multiple cores accessing the same
> receive queue. I had this problem before, but after I make the same
> number of receiving queues as the number of receiving cores, the
> problem disappeared. I did not dig more since I did not care how many
> receive queues I have did not matter.

Jinho,

Thanks.  I have only one queue (should I be using more?) but as far as I 
know, I'm only using one core to transmit as well.

Scott

Re: [dpdk-dev] NIC Stops Transmitting

[Scott Talbert]

On Fri, 26 Jul 2013, Scott Talbert wrote:

>> I am wondering whether you use multiple cores accessing the same
>> receive queue. I had this problem before, but after I make the same
>> number of receiving queues as the number of receiving cores, the
>> problem disappeared. I did not dig more since I did not care how many
>> receive queues I have did not matter.
>
> Thanks.  I have only one queue (should I be using more?) but as far as I 
> know, I'm only using one core to transmit as well.

All,

It turns out that the problem appears to have been caused by me using a 
signal handler to control my data rates.  After switching to use RTE 
timers instead, I am able to transmit close to 10 Gb/sec reliably.

Thanks,
Scott

[dpdk-dev] miss include file in DPDK 1.3.1r2

[Jia.Sui(贾睢)]

Hi All,

In DPDK 1.3.1r2, When I include 'rte_malloc.h' in my app, and then compile it
Will cause this issue:
/root/dpdk/x86_64-default-linuxapp-gcc/include/rte_malloc.h:333: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘rte_malloc_virt2phy’

It looks it miss the definition of 'phys_addr_t' which is defined in 'rte_memory.h'

Thanks

BRs
Jia Sui

[dpdk-dev] [PATCH] mem: fix include in rte_malloc

[Thomas Monjalon]

The function rte_malloc_virt2phy has a dependency on rte_memory.h:
phys_addr_t must be defined.

The dependency handling for apps was wrong in the commit 8c86825cbf.
Let's revert this part.

Reported-by: Jia Sui <jia.sui@advantech.com.cn>
Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 app/test/test_hash.c           |    2 +-
 app/test/test_hash_perf.c      |    2 +-
 app/test/test_memcpy.c         |    1 -
 app/test/test_memcpy_perf.c    |    1 -
 lib/librte_malloc/rte_malloc.h |    1 +
 5 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/app/test/test_hash.c b/app/test/test_hash.c
index 4de8cb1..5437e05 100644
--- a/app/test/test_hash.c
+++ b/app/test/test_hash.c
@@ -41,10 +41,10 @@
 #include <sys/queue.h>
 
 #include <rte_common.h>
+#include <rte_malloc.h>
 #include <rte_cycles.h>
 #include <rte_random.h>
 #include <rte_memory.h>
-#include <rte_malloc.h>
 #include <rte_memzone.h>
 #include <rte_tailq.h>
 #include <rte_eal.h>
diff --git a/app/test/test_hash_perf.c b/app/test/test_hash_perf.c
index 311c2bd..8a0feb3 100644
--- a/app/test/test_hash_perf.c
+++ b/app/test/test_hash_perf.c
@@ -42,10 +42,10 @@
 
 #include <rte_common.h>
 #include <rte_lcore.h>
+#include <rte_malloc.h>
 #include <rte_cycles.h>
 #include <rte_random.h>
 #include <rte_memory.h>
-#include <rte_malloc.h>
 #include <rte_memzone.h>
 #include <rte_tailq.h>
 #include <rte_eal.h>
diff --git a/app/test/test_memcpy.c b/app/test/test_memcpy.c
index 729a2ff..d7777a8 100644
--- a/app/test/test_memcpy.c
+++ b/app/test/test_memcpy.c
@@ -41,7 +41,6 @@
 #include <cmdline_parse.h>
 #include <rte_cycles.h>
 #include <rte_random.h>
-#include <rte_memory.h>
 #include <rte_malloc.h>
 
 #include <rte_memcpy.h>
diff --git a/app/test/test_memcpy_perf.c b/app/test/test_memcpy_perf.c
index 1e75f53..236b295 100644
--- a/app/test/test_memcpy_perf.c
+++ b/app/test/test_memcpy_perf.c
@@ -41,7 +41,6 @@
 #include <cmdline_parse.h>
 #include <rte_cycles.h>
 #include <rte_random.h>
-#include <rte_memory.h>
 #include <rte_malloc.h>
 
 #include <rte_memcpy.h>
diff --git a/lib/librte_malloc/rte_malloc.h b/lib/librte_malloc/rte_malloc.h
index e68187a..fe85689 100644
--- a/lib/librte_malloc/rte_malloc.h
+++ b/lib/librte_malloc/rte_malloc.h
@@ -42,6 +42,7 @@
  */
 
 #include <stddef.h>
+#include <rte_memory.h>
 
 #ifdef __cplusplus
 extern "C" {
-- 
1.7.10.4

Re: [dpdk-dev] [PATCH] mem: fix include in rte_malloc

[didier.pallard]

On 07/31/2013 11:18 AM, Thomas Monjalon wrote:
> The function rte_malloc_virt2phy has a dependency on rte_memory.h:
> phys_addr_t must be defined.
>
> The dependency handling for apps was wrong in the commit 8c86825cbf.
> Let's revert this part.
>
> Reported-by: Jia Sui <jia.sui@advantech.com.cn>
> Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>

Acked-by: Didier Pallard <didier.pallard@6wind.com>

Re: [dpdk-dev] [PATCH] mem: fix include in rte_malloc

[Thomas Monjalon]

31/07/2013 13:36, didier.pallard :
> On 07/31/2013 11:18 AM, Thomas Monjalon wrote:
> > The function rte_malloc_virt2phy has a dependency on rte_memory.h:
> > phys_addr_t must be defined.
> > 
> > The dependency handling for apps was wrong in the commit 8c86825cbf.
> > Let's revert this part.
> > 
> > Reported-by: Jia Sui <jia.sui@advantech.com.cn>
> > Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
> 
> Acked-by: Didier Pallard <didier.pallard@6wind.com>

pushed

Thanks to Jia Sui for reporting.

Side note: please do not reply to an old email when creating new thread.
(see http://dpdk.org/ml/archives/dev/2013-July/thread.html)

-- 
Thomas