From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.ndsl.kaist.edu (www.ndsl.kaist.ac.kr [143.248.57.3]) by dpdk.org (Postfix) with ESMTP id E21245693 for ; Thu, 15 Oct 2015 12:43:34 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mail.ndsl.kaist.edu (Postfix) with ESMTP id 4BE2AC09A4 for ; Thu, 15 Oct 2015 19:43:35 +0900 (KST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ndsl.kaist.edu; h=content-transfer-encoding:content-type:content-type :in-reply-to:mime-version:user-agent:date:date:message-id:from :from:references:to:subject:subject; s=dkim; t=1444905814; x= 1445769814; bh=lgrx8o8V+aeTu3hJIioLjRJOPa9WTWT3qkgJTE8u/po=; b=J vc3HFZG3nA1RI9JX6sK/nUO702CoYse6X+kHx6grB8Sbk5HZThu5Nb7z8f+ARu1P MrVXpArxm4GPQzDwwJ2hrYmng6K0AC5CJaVyU1LoKekY7qdg7Jlgv//MJqDtfGhx FCOMZ2moglTp+6COq6wGnZb7OSAGBJd8Rgr9xc2xTs= X-Virus-Scanned: Debian amavisd-new at ndsl.kaist.ac.kr Received: from mail.ndsl.kaist.edu ([127.0.0.1]) by localhost (ndsl.kaist.ac.kr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qQpvbPqWxWBb for ; Thu, 15 Oct 2015 19:43:34 +0900 (KST) Received: from [10.0.2.15] (ndsl-pc2.kaist.ac.kr [143.248.129.22]) by mail.ndsl.kaist.edu (Postfix) with ESMTPSA id DA66AC01CC; Thu, 15 Oct 2015 19:43:30 +0900 (KST) To: Zoltan Kiss , dev@dpdk.org References: <561F64BA.2000502@ndsl.kaist.edu> <561F7E8C.1080508@linaro.org> From: Younghwan Go Message-ID: <561F835F.3020300@ndsl.kaist.edu> Date: Thu, 15 Oct 2015 19:43:43 +0900 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <561F7E8C.1080508@linaro.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] Calling rte_eth_rx_burst() multiple times X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2015 10:43:35 -0000 Hi Zoltan, Thanks for the email. 2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글: > > > On 15/10/15 09:32, Younghwan Go wrote: >> Hi, >> >> I'm pretty new to playing with DPDK. I was trying to see if I can always >> receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times >> on same pair (code shown below). I'm using DPDK-2.1.0 on 2 >> dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel >> 3.13.0-63-generic). >> >> Since packet processing is slower (~10 Gbps) than pure RX speed (~40 >> Gbps), I assumed rte_eth_rx_burst() would always receive some number of >> packets, eventually filling up MAX_BURST. But for multi-core case (4 >> CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some >> time, causing all cores to be blocked forever. Analyzing the DPDK code >> (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside >> ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always >> returns 0 (where is this value set by the way?). > > I think it is set by the hardware. > >> >> I didn't see this problem for single-core case, in which it returned >> MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out >> of while loop when I receive 0, I keep receiving packets in next > queue> pairs. Does anyone know why this block might happen? Or am I not >> allowed to call rte_eth_rx_burst() multiple times on same >> pair if I get 0? Any help will be great! Thank you! > > Although not mentioned in the documentation itself, as far as I know > rte_eth_rx_burst() is not thread-safe. If you look in to receive > functions, there are no locking anywhere. You should call it on > separate queues from different threads, and configure e.g RSS to > distribute the traffic by the hardware. I'm calling rte_eth_rx_burst() on separate queue ids for each thread. I'm actually using lcore_id (= 0, 1, 2, 3 for 4 threads pinned to each separate CPU core) as queue_id. I also made sure that this problem is not caused by threads conflicting by locking before calling rte_eth_rx_burst(). For RSS, I configured with ETH_RSS_IP for load balancing traffic to each port and queue. But even if RSS wasn't set, shouldn't at least one core be receiving packets? What I'm seeing is all threads getting stuck at rte_eth_rx_burst() with return value of 0s indefinitely. >> >> ------------------------------------------------------------------------ >> int cnt = MAX_BURST; // MAX_BURST = 32 >> int off = 0; >> do { >> ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt); >> if (ret == 0) { >> // don't break out but continue >> } else if (ret > 0) { >> off += ret; >> cnt -= ret; >> } >> } while (cnt > 0); >> ------------------------------------------------------------------------ >> >> Best, >> Younghwan Thanks, Younghwan