From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by dpdk.org (Postfix) with ESMTP id 774298E8D for ; Thu, 15 Oct 2015 13:51:47 +0200 (CEST) Received: by wicgb1 with SMTP id gb1so268770595wic.1 for ; Thu, 15 Oct 2015 04:51:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=fUTxsbZLBZaQP/etg4m0vT7WzkjxL2XUUTfI9W6c06U=; b=eAshT0mhf76PmqVQ6agP4IRDQOtFiTpWcK2a9LU5TQ9BwZr/8xv0g1k7ma++4POvRA RkssHRWFCXcayu3c69mOCb3GDNH9qBsnMqmJOzhgp/mhje1cvI+x7sevBmLB0VhLcCbB 3YOy38GEVmhtU92mjuetl6UcXb4egv4BTB4IbO8BpUbt3XtkJANSWDkEBURg8/694T3w iAw0JPWvFDEz/uAB01kIIU7fYO+7toU5LKpVTRel1+tApACOK217k/YgPwnzPUYGEq8H m+o3JSURNYRR8lCWMxpX9ciVbUvf+OQSo1u8PC6/OMAWk2R+HCeLb8MKiyjhJcR8a7tl /eCw== X-Gm-Message-State: ALoCoQkQllgwH1RK2By9b98jITwFMS9m+82VSb8fYHffhpSFfmWPv4wbb1J+vQbWrYohueAcexvp X-Received: by 10.180.182.107 with SMTP id ed11mr10246864wic.52.1444909907245; Thu, 15 Oct 2015 04:51:47 -0700 (PDT) Received: from [192.168.0.101] ([90.152.119.35]) by smtp.googlemail.com with ESMTPSA id xa5sm16024396wjc.20.2015.10.15.04.51.46 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Oct 2015 04:51:46 -0700 (PDT) To: Younghwan Go , dev@dpdk.org References: <561F64BA.2000502@ndsl.kaist.edu> <561F7E8C.1080508@linaro.org> <561F835F.3020300@ndsl.kaist.edu> From: Zoltan Kiss Message-ID: <561F9353.8080306@linaro.org> Date: Thu, 15 Oct 2015 12:51:47 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <561F835F.3020300@ndsl.kaist.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] Calling rte_eth_rx_burst() multiple times X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2015 11:51:47 -0000 On 15/10/15 11:43, Younghwan Go wrote: > Hi Zoltan, > > Thanks for the email. > > 2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글: >> >> >> On 15/10/15 09:32, Younghwan Go wrote: >>> Hi, >>> >>> I'm pretty new to playing with DPDK. I was trying to see if I can always >>> receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times >>> on same pair (code shown below). I'm using DPDK-2.1.0 on 2 >>> dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel >>> 3.13.0-63-generic). >>> >>> Since packet processing is slower (~10 Gbps) than pure RX speed (~40 >>> Gbps), I assumed rte_eth_rx_burst() would always receive some number of >>> packets, eventually filling up MAX_BURST. But for multi-core case (4 >>> CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some >>> time, causing all cores to be blocked forever. Analyzing the DPDK code >>> (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside >>> ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always >>> returns 0 (where is this value set by the way?). >> >> I think it is set by the hardware. >> >>> >>> I didn't see this problem for single-core case, in which it returned >>> MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out >>> of while loop when I receive 0, I keep receiving packets in next >> queue> pairs. Does anyone know why this block might happen? Or am I not >>> allowed to call rte_eth_rx_burst() multiple times on same >>> pair if I get 0? Any help will be great! Thank you! >> >> Although not mentioned in the documentation itself, as far as I know >> rte_eth_rx_burst() is not thread-safe. If you look in to receive >> functions, there are no locking anywhere. You should call it on >> separate queues from different threads, and configure e.g RSS to >> distribute the traffic by the hardware. > > I'm calling rte_eth_rx_burst() on separate queue ids for each thread. > I'm actually using lcore_id (= 0, 1, 2, 3 for 4 threads pinned to each > separate CPU core) as queue_id. I also made sure that this problem is > not caused by threads conflicting by locking before calling > rte_eth_rx_burst(). > > For RSS, I configured with ETH_RSS_IP for load balancing traffic to each > port and queue. But even if RSS wasn't set, shouldn't at least one core > be receiving packets? What I'm seeing is all threads getting stuck at > rte_eth_rx_burst() with return value of 0s indefinitely. > >>> >>> ------------------------------------------------------------------------ >>> int cnt = MAX_BURST; // MAX_BURST = 32 >>> int off = 0; >>> do { >>> ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt); Another thing which might cause your problem is that I don't see where do you release the buffers after received into m_table. You need to call rte_pktmbuf_free() on them at some point, otherwise your pool can get depleted, and the receive function can't refill the descriptor rings. >>> if (ret == 0) { >>> // don't break out but continue >>> } else if (ret > 0) { >>> off += ret; >>> cnt -= ret; >>> } >>> } while (cnt > 0); >>> ------------------------------------------------------------------------ >>> >>> Best, >>> Younghwan > > Thanks, > Younghwan