[dpdk-dev] Calling rte_eth_rx_burst() multiple times

[dpdk-dev] Calling rte_eth_rx_burst() multiple times

[Younghwan Go]

Hi,

I'm pretty new to playing with DPDK. I was trying to see if I can always 
receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times 
on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2 
dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel 
3.13.0-63-generic).

Since packet processing is slower (~10 Gbps) than pure RX speed (~40 
Gbps), I assumed rte_eth_rx_burst() would always receive some number of 
packets, eventually filling up MAX_BURST. But for multi-core case (4 
CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some 
time, causing all cores to be blocked forever. Analyzing the DPDK code 
(drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside 
ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always 
returns 0 (where is this value set by the way?).

I didn't see this problem for single-core case, in which it returned 
MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out 
of while loop when I receive 0, I keep receiving packets in next <port, 
queue> pairs. Does anyone know why this block might happen? Or am I not 
allowed to call rte_eth_rx_burst() multiple times on same <port, queue> 
pair if I get 0? Any help will be great! Thank you!

------------------------------------------------------------------------
int cnt = MAX_BURST; // MAX_BURST = 32
int off = 0;
do {
     ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);
     if (ret == 0) {
         // don't break out but continue
     } else if (ret > 0) {
         off += ret;
         cnt -= ret;
     }
} while (cnt > 0);
------------------------------------------------------------------------

Best,
Younghwan

Re: [dpdk-dev] Calling rte_eth_rx_burst() multiple times

[Zoltan Kiss]

On 15/10/15 09:32, Younghwan Go wrote:
> Hi,
>
> I'm pretty new to playing with DPDK. I was trying to see if I can always
> receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times
> on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2
> dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel
> 3.13.0-63-generic).
>
> Since packet processing is slower (~10 Gbps) than pure RX speed (~40
> Gbps), I assumed rte_eth_rx_burst() would always receive some number of
> packets, eventually filling up MAX_BURST. But for multi-core case (4
> CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some
> time, causing all cores to be blocked forever. Analyzing the DPDK code
> (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside
> ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always
> returns 0 (where is this value set by the way?).

I think it is set by the hardware.

>
> I didn't see this problem for single-core case, in which it returned
> MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out
> of while loop when I receive 0, I keep receiving packets in next <port,
> queue> pairs. Does anyone know why this block might happen? Or am I not
> allowed to call rte_eth_rx_burst() multiple times on same <port, queue>
> pair if I get 0? Any help will be great! Thank you!

Although not mentioned in the documentation itself, as far as I know 
rte_eth_rx_burst() is not thread-safe. If you look in to receive 
functions, there are no locking anywhere. You should call it on separate 
queues from different threads, and configure e.g RSS to distribute the 
traffic by the hardware.

>
> ------------------------------------------------------------------------
> int cnt = MAX_BURST; // MAX_BURST = 32
> int off = 0;
> do {
>      ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);
>      if (ret == 0) {
>          // don't break out but continue
>      } else if (ret > 0) {
>          off += ret;
>          cnt -= ret;
>      }
> } while (cnt > 0);
> ------------------------------------------------------------------------
>
> Best,
> Younghwan

Re: [dpdk-dev] Calling rte_eth_rx_burst() multiple times

[Younghwan Go]

Hi Zoltan,

Thanks for the email.

2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글:
>
>
> On 15/10/15 09:32, Younghwan Go wrote:
>> Hi,
>>
>> I'm pretty new to playing with DPDK. I was trying to see if I can always
>> receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times
>> on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2
>> dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel
>> 3.13.0-63-generic).
>>
>> Since packet processing is slower (~10 Gbps) than pure RX speed (~40
>> Gbps), I assumed rte_eth_rx_burst() would always receive some number of
>> packets, eventually filling up MAX_BURST. But for multi-core case (4
>> CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some
>> time, causing all cores to be blocked forever. Analyzing the DPDK code
>> (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside
>> ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always
>> returns 0 (where is this value set by the way?).
>
> I think it is set by the hardware.
>
>>
>> I didn't see this problem for single-core case, in which it returned
>> MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out
>> of while loop when I receive 0, I keep receiving packets in next <port,
>> queue> pairs. Does anyone know why this block might happen? Or am I not
>> allowed to call rte_eth_rx_burst() multiple times on same <port, queue>
>> pair if I get 0? Any help will be great! Thank you!
>
> Although not mentioned in the documentation itself, as far as I know 
> rte_eth_rx_burst() is not thread-safe. If you look in to receive 
> functions, there are no locking anywhere. You should call it on 
> separate queues from different threads, and configure e.g RSS to 
> distribute the traffic by the hardware.

I'm calling rte_eth_rx_burst() on separate queue ids for each thread. 
I'm actually using lcore_id (= 0, 1, 2, 3 for 4 threads pinned to each 
separate CPU core) as queue_id. I also made sure that this problem is 
not caused by threads conflicting by locking before calling 
rte_eth_rx_burst().

For RSS, I configured with ETH_RSS_IP for load balancing traffic to each 
port and queue. But even if RSS wasn't set, shouldn't at least one core 
be receiving packets? What I'm seeing is all threads getting stuck at 
rte_eth_rx_burst() with return value of 0s indefinitely.

>>
>> ------------------------------------------------------------------------
>> int cnt = MAX_BURST; // MAX_BURST = 32
>> int off = 0;
>> do {
>>      ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);
>>      if (ret == 0) {
>>          // don't break out but continue
>>      } else if (ret > 0) {
>>          off += ret;
>>          cnt -= ret;
>>      }
>> } while (cnt > 0);
>> ------------------------------------------------------------------------
>>
>> Best,
>> Younghwan

Thanks,
Younghwan

Re: [dpdk-dev] Calling rte_eth_rx_burst() multiple times

[Zoltan Kiss]

On 15/10/15 11:43, Younghwan Go wrote:
> Hi Zoltan,
>
> Thanks for the email.
>
> 2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글:
>>
>>
>> On 15/10/15 09:32, Younghwan Go wrote:
>>> Hi,
>>>
>>> I'm pretty new to playing with DPDK. I was trying to see if I can always
>>> receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times
>>> on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2
>>> dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel
>>> 3.13.0-63-generic).
>>>
>>> Since packet processing is slower (~10 Gbps) than pure RX speed (~40
>>> Gbps), I assumed rte_eth_rx_burst() would always receive some number of
>>> packets, eventually filling up MAX_BURST. But for multi-core case (4
>>> CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some
>>> time, causing all cores to be blocked forever. Analyzing the DPDK code
>>> (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside
>>> ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always
>>> returns 0 (where is this value set by the way?).
>>
>> I think it is set by the hardware.
>>
>>>
>>> I didn't see this problem for single-core case, in which it returned
>>> MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out
>>> of while loop when I receive 0, I keep receiving packets in next <port,
>>> queue> pairs. Does anyone know why this block might happen? Or am I not
>>> allowed to call rte_eth_rx_burst() multiple times on same <port, queue>
>>> pair if I get 0? Any help will be great! Thank you!
>>
>> Although not mentioned in the documentation itself, as far as I know
>> rte_eth_rx_burst() is not thread-safe. If you look in to receive
>> functions, there are no locking anywhere. You should call it on
>> separate queues from different threads, and configure e.g RSS to
>> distribute the traffic by the hardware.
>
> I'm calling rte_eth_rx_burst() on separate queue ids for each thread.
> I'm actually using lcore_id (= 0, 1, 2, 3 for 4 threads pinned to each
> separate CPU core) as queue_id. I also made sure that this problem is
> not caused by threads conflicting by locking before calling
> rte_eth_rx_burst().
>
> For RSS, I configured with ETH_RSS_IP for load balancing traffic to each
> port and queue. But even if RSS wasn't set, shouldn't at least one core
> be receiving packets? What I'm seeing is all threads getting stuck at
> rte_eth_rx_burst() with return value of 0s indefinitely.
>
>>>
>>> ------------------------------------------------------------------------
>>> int cnt = MAX_BURST; // MAX_BURST = 32
>>> int off = 0;
>>> do {
>>>      ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);

Another thing which might cause your problem is that I don't see where 
do you release the buffers after received into m_table. You need to call 
rte_pktmbuf_free() on them at some point, otherwise your pool can get 
depleted, and the receive function can't refill the descriptor rings.


>>>      if (ret == 0) {
>>>          // don't break out but continue
>>>      } else if (ret > 0) {
>>>          off += ret;
>>>          cnt -= ret;
>>>      }
>>> } while (cnt > 0);
>>> ------------------------------------------------------------------------
>>>
>>> Best,
>>> Younghwan
>
> Thanks,
> Younghwan