From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f45.google.com (mail-pb0-f45.google.com [209.85.160.45]) by dpdk.org (Postfix) with ESMTP id 1A0EE5907 for ; Thu, 13 Mar 2014 23:05:54 +0100 (CET) Received: by mail-pb0-f45.google.com with SMTP id uo5so1716134pbc.4 for ; Thu, 13 Mar 2014 15:07:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=eRjO3VdNWy7A9nvdnREXFuZE92KbhS1ndUZ3kbYMBco=; b=LSGrhrYRBUr2IL8aReshgX206WCUEQddk+Ob0qeFKO3MAG/lc75LJKhT4U3ODhxfkp VLbD8oJmFILM8bPNzL6xRBF9F/e1jzM0FBKw5imPE/pFZ+25bpKFAkDHDnxMwNdF6Jb3 hcUsSaxT7ZsuQpKYisPHMlYiPGgGAS8qlJPTsnUARTQ+BcYCRzXJRoQo8mpluyK1l0qt 30mj53Jzv4gpyNhbGy+8/Jqwoyrc6gEmSUXT4BMuujz+FknH192tCTYiEG352BoYAFJJ AL57xbdADQKyttQlpf7M1rFUbk/BBJXN+MaoGynG9d/rzT6c6NDhf+Ha2BNA1C1h2AcJ SGGA== X-Gm-Message-State: ALoCoQn1xdOX56IS9p9ROHFwNUwh77SjGdHjlgwlnrYSSPWQh0LNqKDLOZUbct2RE0x0Hs2MVttI X-Received: by 10.68.201.67 with SMTP id jy3mr5242593pbc.20.1394748444645; Thu, 13 Mar 2014 15:07:24 -0700 (PDT) Received: from [192.168.128.242] ([50.242.95.29]) by mx.google.com with ESMTPSA id iv2sm10198142pbc.19.2014.03.13.15.07.22 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 13 Mar 2014 15:07:23 -0700 (PDT) From: Daniel Kan Message-Id: Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Date: Thu, 13 Mar 2014 15:07:20 -0700 References: <3274259C-EE9A-461C-9586-D7B0820CDE2B@nyansa.com> To: dev@dpdk.org In-Reply-To: <3274259C-EE9A-461C-9586-D7B0820CDE2B@nyansa.com> X-Mailer: Apple Mail (2.1874) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] Vmxnet3-pmd not receiving packets after random time under high load X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Mar 2014 22:05:55 -0000 I dug deeper into the problem vmxnet-pmd not capturing packets. Whenever = pmd stops capturing, it does not respect the number of rxd. For example, = I set rxd to 512. The pkt mbuf mempool I allocate is 4482, which is much = bigger than rxd. I would expect ~512 mbufs to be removed from mempool at = any moment in time. When not able to capture new packets is imminent, = pmd would use much more than 512 to the point of exhausting all the = mbufs in mempool. After this point, pmd stops capturing packet and does = not recover; it never enters the while loop below.=20 vmxnet3_rxtx.c 457 while (rcd->gen =3D=3D rxq->comp_ring.gen) { This issue occurs quite randomly, which leads me to believe there is = some race condition. It occurs when packets are arriving faster than the = application can process.=20 I understand that vmxnet3 pmd preallocates rxq descriptors during = initialization. Yet, even when it=92s receiving packets correctly, it = uses a little more than rxq descriptors. I find the number of mbufs in = mempool very interesting.=20 Before vmxnet3 init 4482 After vmxnet3 init 3970 When receiving packets properly ranges between 3920 - 3970 Prior to stop receiving packets ranges between 0 and 3970 After the last packet is received 4481 I appreciate any comment/help on this. Thanks.=20 Dan On Mar 11, 2014, at 12:55 PM, Daniel Kan wrote: > I=92m unable to get RSS to work properly with vmxnet3-pmd. The first = issue is that the number of rxqs must be power of 2. Otherwise, = rte_eth_dev_start() fails due to inability to activate vmxnet3 NIC. This = is not too big of a deal, but physical NICs don=92t have this = requirement.=20 >=20 > The second issue is that RSS is just not working at all for me. The = rxmode is set to ETH_MQ_RX_RSS and rss_hf =3D ETH_RSS_IPV4_TCP | = ETH_RSS_IPV4_UDP | ETH_RSS_IPV4 | ETH_RSS_IPV6. The same configuration = works for a real NIC. When I checked mb->pkt.hash, the value is all = zeroed out.=20 >=20 > Even if I disabled RSS, I found the performance of vmxnet3-pmd to be = quite poor, peaking out at 600k pps with 64 byte packet, while libpcap = can do 650k pps.=20 >=20 > Lastly, there is a stability issue. On a number of occasions, = vmxnet3-pmd stops receiving packets after some random time and several = million packets.=20 >=20 > I=92m not sure if anyone else is having as much issue as I=92m, I will = give vmxnet3-usermap a try.=20 >=20 > Finally, does either vmxnet3-usermap or vmxnet3-pmd work well for = non-Intel-based underlying physical NIC?=20 >=20 > Thanks.=20 >=20 > Dan >=20 >=20