From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f174.google.com (mail-qk0-f174.google.com [209.85.220.174]) by dpdk.org (Postfix) with ESMTP id 2403C3237 for ; Wed, 8 Apr 2015 18:35:36 +0200 (CEST) Received: by qku63 with SMTP id 63so89405715qku.3 for ; Wed, 08 Apr 2015 09:35:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arbor.net; s=m0; h=from:content-type:content-transfer-encoding:date:subject:to :message-id:mime-version; bh=kipVR4T3nr/0hR6m9rcz2QaWSlgUgDaKoxMMX7bvNDQ=; b=HxTXBzVl6SyVAlpZYX8eZ51pMGg6+0M1R+zaAKvyRKVlXTd8OST08Lqt3Rd1zbsonU N+TtOlIuyFttvtWCrpM4Vuh2Tqn47r1v/Dt2bZ7p0Kx6UwxIxL6t/Njwp9jPjhYNuiVu lVZ/18SfKnC2toRufbno7qat84kNmpzDgyYYw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:content-transfer-encoding:date :subject:to:message-id:mime-version; bh=kipVR4T3nr/0hR6m9rcz2QaWSlgUgDaKoxMMX7bvNDQ=; b=V1IMtjlzo0v3XrsAxhDfxvxUTBKowcoou5mwsjD63NIJthuyhjAyO1sAO3mfyG+BAg O966aBCai4dTpR/vcTb5DmtrWjmH7xCP3++GK7L+l81bOCyOvaWSqREUa0gLn4p+OCSo jQ8QgPHB8zZkZfyQiU6fmCuPxfPGhoCLF4RoD5cBhdP8qS1nFPAQv3zB8cQJdarqltdx 593r+JTCRAkkPAncDapmnpNJVp0XxHG6Hxs2r5/VxflZr7l1a6NL0nIfgc8QcdpsqY4p p6ep3HXPhs1hnhP5PvRQlHCGVipX4Abtynh+XRu0j+DeFRFG5sM6PO153+I7IbtpO9JT oBTQ== X-Gm-Message-State: ALoCoQmJvogdgakM0s8bvymp7IuE9+sWyAje4mkIkikBpNbNomxQ43mcR635sPaXoADt9z0D4Tiu X-Received: by 10.55.52.77 with SMTP id b74mr51252334qka.78.1428510935673; Wed, 08 Apr 2015 09:35:35 -0700 (PDT) Received: from [172.30.1.4] ([76.11.68.116]) by mx.google.com with ESMTPSA id z67sm7896781qgz.10.2015.04.08.09.35.34 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 08 Apr 2015 09:35:34 -0700 (PDT) From: Aaron Campbell Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Wed, 8 Apr 2015 13:35:34 -0300 To: dev@dpdk.org Message-Id: <5D6C8629-393C-4195-8063-8168E206335B@arbor.net> Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) X-Mailer: Apple Mail (2.2070.6) Subject: [dpdk-dev] Polling too often at lower packet rates? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Apr 2015 16:35:36 -0000 Hi, I have a machine with 6 DPDK ports (4 igb, 2 ixgbe), with 1.23Mpps = traffic offered to only one of the 10G ports (the other 5 are unused). = I also have a program with a pretty standard looking DPDK receive loop, = where it calls rte_eth_rx_burst() for each configured port. If I = configure the loop to read from all 6 ports, it can read the 1.23Mpps = rate with no drops. If I configure the loop to poll only 1 port (the = ixgbe receiving the traffic), I lose about 1/3rd of the packets (i.e., = the NIC drops ~400Kpps). Another data point is that if I configure the loop to read from 3 out of = the 6 ports, the drop rate is reduced to less than half (i.e., the NIC = is only dropping ~190Kpps now). So it seems that in this test, = throughput improves by adding NICs, not removing them, which is = counter-intuitive. Again, I get no drops when polling all 6 ports. = Note, the burst size is 32. I did find a reference to a similar issue in a recent paper = (http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ICN2015.pdf= ), Section III, which states: "The DPDK L2FWD application initially only managed to forward 13.8 Mpps = in the single direction test at the maximum CPU frequency, a similar = result can be found in [11]. Reducing the CPU frequency increased the = throughput to the expected value of 14.88 Mpps. Our investigation of = this anomaly revealed that the lack of any processing combined with the = fast CPU caused DPDK to poll the NIC too often. DPDK does not use = interrupts, it utilizes a busy wait loop that polls the NIC until at = least one packet is returned. This resulted in a high poll rate which = affected the throughput. We limited the poll rate to 500,000 poll = operations per second (i.e., a batch size of about 30 packets) and = achieved line rate in the unidirectional test with all frequencies. This = effect was only observed with the X520 NIC, tests with X540 NICs did not = show this anomaly.=E2=80=9D Another reference, from this mailing list last year = (http://wiki.dpdk.org/ml/archives/dev/2014-January/001169.html): "I suggest you to check average burst sizes on receive queues. Looks = like I stumbled upon a similar issue several times. If you are calling = rte_eth_rx_burst too frequently, NIC begins losing packets no matter how = many CPU horse power you have (more you have, more it loses, actually). = In my case this situation occured when average burst size is less than = 20 packets or so. I'm not sure what's the reason for this behavior, but = I observed it on several applications on Intel 82599 10Gb cards.=E2=80=9D So I=E2=80=99m wondering if anyone can explain at a lower level what = happens when you poll =E2=80=9Ctoo often=E2=80=9D, and if there are any = practical workarounds. We=E2=80=99re using this same program and DPDK = version to process 10G line-rate in other scenarios, so I=E2=80=99m = confident that the overall packet capture architecture is sound. -Aaron=