From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f169.google.com (mail-ig0-f169.google.com [209.85.213.169]) by dpdk.org (Postfix) with ESMTP id 7D97B5A41 for ; Wed, 8 Apr 2015 20:00:59 +0200 (CEST) Received: by igblo3 with SMTP id lo3so45391860igb.0 for ; Wed, 08 Apr 2015 11:00:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=q0QVRM4EEctLfAuV13/uZy2HhNmr4A4k1CZJWVR7LNY=; b=D5Um5B/W83rbrlrL6lWzqbTjW00q4dbIDs4q7MeWelN5uxV8zsAG7rxuEtnWqUe+TG LKlbAmarjVuVru52mi1Ze4fHCub7IPLoasYni0R1AGr1jFfkyPPZUrMvkLXSOFnULWV5 NJm6l/oNuuudDXhUTyPnCzpgfubANkP1itG9Tpj/LIW2wObMVkKJf2++IofKDR3MQOrm 76RUeYw+t9iLfPYNt91vB2xIvlvM9futH+QJdlFpiTLNxIwNzVtsZPhzyknBH1K4VuIE OfetMNII9vm56ySP1pcM8e+HiRR/OJhFlUroIOUW3OjDPaucHu/NmFkFwQxTetilTmGF QihA== X-Gm-Message-State: ALoCoQmhx4/5xOyy4NC0jiqxDKkgBozWOnC+gHUpM0HAuwpwf68Ng6BSUfhi3ln2sBnOBNrhLI2h MIME-Version: 1.0 X-Received: by 10.50.178.133 with SMTP id cy5mr13358586igc.5.1428516058893; Wed, 08 Apr 2015 11:00:58 -0700 (PDT) Received: by 10.64.58.129 with HTTP; Wed, 8 Apr 2015 11:00:58 -0700 (PDT) In-Reply-To: <5D6C8629-393C-4195-8063-8168E206335B@arbor.net> References: <5D6C8629-393C-4195-8063-8168E206335B@arbor.net> Date: Wed, 8 Apr 2015 11:00:58 -0700 Message-ID: From: Stephen Hemminger To: Aaron Campbell Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] Polling too often at lower packet rates? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Apr 2015 18:00:59 -0000 We use adaptive polling loop similar to l3fwd-power example. See: http://video.fosdem.org/2015/devroom-network_management_and_sdn/ On Wed, Apr 8, 2015 at 9:35 AM, Aaron Campbell wrote: > Hi, > > I have a machine with 6 DPDK ports (4 igb, 2 ixgbe), with 1.23Mpps traffi= c > offered to only one of the 10G ports (the other 5 are unused). I also ha= ve > a program with a pretty standard looking DPDK receive loop, where it call= s > rte_eth_rx_burst() for each configured port. If I configure the loop to > read from all 6 ports, it can read the 1.23Mpps rate with no drops. If I > configure the loop to poll only 1 port (the ixgbe receiving the traffic),= I > lose about 1/3rd of the packets (i.e., the NIC drops ~400Kpps). > > Another data point is that if I configure the loop to read from 3 out of > the 6 ports, the drop rate is reduced to less than half (i.e., the NIC is > only dropping ~190Kpps now). So it seems that in this test, throughput > improves by adding NICs, not removing them, which is counter-intuitive. > Again, I get no drops when polling all 6 ports. Note, the burst size is = 32. > > I did find a reference to a similar issue in a recent paper ( > http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ICN2015.pdf= ), > Section III, which states: > > "The DPDK L2FWD application initially only managed to forward 13.8 Mpps i= n > the single direction test at the maximum CPU frequency, a similar result > can be found in [11]. Reducing the CPU frequency increased the throughput > to the expected value of 14.88 Mpps. Our investigation of this anomaly > revealed that the lack of any processing combined with the fast CPU cause= d > DPDK to poll the NIC too often. DPDK does not use interrupts, it utilizes= a > busy wait loop that polls the NIC until at least one packet is returned. > This resulted in a high poll rate which affected the throughput. We limit= ed > the poll rate to 500,000 poll operations per second (i.e., a batch size o= f > about 30 packets) and achieved line rate in the unidirectional test with > all frequencies. This effect was only observed with the X520 NIC, tests > with X540 NICs did not show this anomaly.=E2=80=9D > > Another reference, from this mailing list last year ( > http://wiki.dpdk.org/ml/archives/dev/2014-January/001169.html): > > "I suggest you to check average burst sizes on receive queues. Looks like > I stumbled upon a similar issue several times. If you are calling > rte_eth_rx_burst too frequently, NIC begins losing packets no matter how > many CPU horse power you have (more you have, more it loses, actually). I= n > my case this situation occured when average burst size is less than 20 > packets or so. I'm not sure what's the reason for this behavior, but I > observed it on several applications on Intel 82599 10Gb cards.=E2=80=9D > > So I=E2=80=99m wondering if anyone can explain at a lower level what happ= ens when > you poll =E2=80=9Ctoo often=E2=80=9D, and if there are any practical work= arounds. We=E2=80=99re > using this same program and DPDK version to process 10G line-rate in othe= r > scenarios, so I=E2=80=99m confident that the overall packet capture archi= tecture is > sound. > > -Aaron