Re: [dpdk-dev] Calling rte_eth_rx_burst() multiple times

[Younghwan Go <yhwan@ndsl.kaist.edu>]

Hi Zoltan,

Thanks for the email.

2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글:
>
>
> On 15/10/15 09:32, Younghwan Go wrote:
>> Hi,
>>
>> I'm pretty new to playing with DPDK. I was trying to see if I can always
>> receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times
>> on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2
>> dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel
>> 3.13.0-63-generic).
>>
>> Since packet processing is slower (~10 Gbps) than pure RX speed (~40
>> Gbps), I assumed rte_eth_rx_burst() would always receive some number of
>> packets, eventually filling up MAX_BURST. But for multi-core case (4
>> CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some
>> time, causing all cores to be blocked forever. Analyzing the DPDK code
>> (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside
>> ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always
>> returns 0 (where is this value set by the way?).
>
> I think it is set by the hardware.
>
>>
>> I didn't see this problem for single-core case, in which it returned
>> MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out
>> of while loop when I receive 0, I keep receiving packets in next <port,
>> queue> pairs. Does anyone know why this block might happen? Or am I not
>> allowed to call rte_eth_rx_burst() multiple times on same <port, queue>
>> pair if I get 0? Any help will be great! Thank you!
>
> Although not mentioned in the documentation itself, as far as I know 
> rte_eth_rx_burst() is not thread-safe. If you look in to receive 
> functions, there are no locking anywhere. You should call it on 
> separate queues from different threads, and configure e.g RSS to 
> distribute the traffic by the hardware.

I'm calling rte_eth_rx_burst() on separate queue ids for each thread. 
I'm actually using lcore_id (= 0, 1, 2, 3 for 4 threads pinned to each 
separate CPU core) as queue_id. I also made sure that this problem is 
not caused by threads conflicting by locking before calling 
rte_eth_rx_burst().

For RSS, I configured with ETH_RSS_IP for load balancing traffic to each 
port and queue. But even if RSS wasn't set, shouldn't at least one core 
be receiving packets? What I'm seeing is all threads getting stuck at 
rte_eth_rx_burst() with return value of 0s indefinitely.

>>
>> ------------------------------------------------------------------------
>> int cnt = MAX_BURST; // MAX_BURST = 32
>> int off = 0;
>> do {
>>      ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);
>>      if (ret == 0) {
>>          // don't break out but continue
>>      } else if (ret > 0) {
>>          off += ret;
>>          cnt -= ret;
>>      }
>> } while (cnt > 0);
>> ------------------------------------------------------------------------
>>
>> Best,
>> Younghwan

Thanks,
Younghwan