From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-f171.google.com (mail-ie0-f171.google.com [209.85.223.171]) by dpdk.org (Postfix) with ESMTP id 3DCEE688E for ; Thu, 9 Apr 2015 20:26:26 +0200 (CEST) Received: by iebmp1 with SMTP id mp1so108128495ieb.0 for ; Thu, 09 Apr 2015 11:26:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arbor.net; s=m0; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=xBuz/S1kg9F+rHMpSRExUYEo0KWOAXv0XJlamXyvxM4=; b=ZetPqslT3corosGUiTGWYFuADvRMn8yCF3zPTylGKRb+OSUu4lBZJZ5a2TtIhpBTaC ijxRg9DbNUL2Rlowqb2dyftgHPlK5yopNAuroP4uQfXy1xr4kdeXym/NsZfvvf386BmV OPwYOxpYJ4577byV077f1OcyRDKs8sMg4Vf9U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=xBuz/S1kg9F+rHMpSRExUYEo0KWOAXv0XJlamXyvxM4=; b=hYVqiX4XmzF1Keq3xneCVxfLsYEhuUY1mzB2+6Ot4Tg5hqEvOIFA7WHuXKFwJ47bse VcJxuRihUDr3asxClIvV7wg3XVb7caR5ZE+iSta9eHTsUzNuLBEmfwyAlNv8cbeF9qjp 7nsKe+gUJnwlj224s5gCz5VI7WTbJATRt+DHdAbj+UO0yKlLdUOi7pUoTv4dONuJzLR4 2wxrC9zNmPt1CeIyy9ABtT7o16lJH6kfmyNahpQuHu/ynOORAm7QNBmrubYE0bOd7uDu MImam3vHQOfkHbeuJ56ZvPWfKWTSS+RV/Lau54KSPW4+TXYY0EEud7ZI+yStm1JR4KcO BW5g== X-Gm-Message-State: ALoCoQnU8oG9dwdbN2A0ab6f3KR38G774rEExDZaeAKWtCpeQ6MLPNBF5Zlfw3+nVgG1K3Qv9nYs X-Received: by 10.42.44.132 with SMTP id b4mr40740118icf.73.1428603985628; Thu, 09 Apr 2015 11:26:25 -0700 (PDT) Received: from [172.30.1.2] ([76.11.68.116]) by mx.google.com with ESMTPSA id m136sm9139694iom.21.2015.04.09.11.26.24 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 09 Apr 2015 11:26:24 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) From: Aaron Campbell In-Reply-To: Date: Thu, 9 Apr 2015 15:26:23 -0300 Message-Id: <16D8F22D-333F-499F-8B5A-4839E582054D@arbor.net> References: <5D6C8629-393C-4195-8063-8168E206335B@arbor.net> To: Stephen Hemminger X-Mailer: Apple Mail (2.2098) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] Polling too often at lower packet rates? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Apr 2015 18:26:26 -0000 Hi Stephen, Thanks, that was an informative talk. In this case, are you referring = to your comments about the thermal budget? That=E2=80=99s definitely interesting, but there must be more to it than = that. Again, if I loop over all 6 ports (i.e., continue to keep the CPU = busy), it works around the problem. I agree that adaptive polling makes sense and will look into it. But = will still take any further ideas on what is going on here. -Aaron > On Apr 8, 2015, at 3:00 PM, Stephen Hemminger = wrote: >=20 > We use adaptive polling loop similar to l3fwd-power example. > See: > =20 >=20 > http://video.fosdem.org/2015/devroom-network_management_and_sdn/ = >=20 > On Wed, Apr 8, 2015 at 9:35 AM, Aaron Campbell > wrote: > Hi, >=20 > I have a machine with 6 DPDK ports (4 igb, 2 ixgbe), with 1.23Mpps = traffic offered to only one of the 10G ports (the other 5 are unused). = I also have a program with a pretty standard looking DPDK receive loop, = where it calls rte_eth_rx_burst() for each configured port. If I = configure the loop to read from all 6 ports, it can read the 1.23Mpps = rate with no drops. If I configure the loop to poll only 1 port (the = ixgbe receiving the traffic), I lose about 1/3rd of the packets (i.e., = the NIC drops ~400Kpps). >=20 > Another data point is that if I configure the loop to read from 3 out = of the 6 ports, the drop rate is reduced to less than half (i.e., the = NIC is only dropping ~190Kpps now). So it seems that in this test, = throughput improves by adding NICs, not removing them, which is = counter-intuitive. Again, I get no drops when polling all 6 ports. = Note, the burst size is 32. >=20 > I did find a reference to a similar issue in a recent paper = (http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ICN2015.pdf= = ), Section III, which states: >=20 > "The DPDK L2FWD application initially only managed to forward 13.8 = Mpps in the single direction test at the maximum CPU frequency, a = similar result can be found in [11]. Reducing the CPU frequency = increased the throughput to the expected value of 14.88 Mpps. Our = investigation of this anomaly revealed that the lack of any processing = combined with the fast CPU caused DPDK to poll the NIC too often. DPDK = does not use interrupts, it utilizes a busy wait loop that polls the NIC = until at least one packet is returned. This resulted in a high poll rate = which affected the throughput. We limited the poll rate to 500,000 poll = operations per second (i.e., a batch size of about 30 packets) and = achieved line rate in the unidirectional test with all frequencies. This = effect was only observed with the X520 NIC, tests with X540 NICs did not = show this anomaly.=E2=80=9D >=20 > Another reference, from this mailing list last year = (http://wiki.dpdk.org/ml/archives/dev/2014-January/001169.html = ): >=20 > "I suggest you to check average burst sizes on receive queues. Looks = like I stumbled upon a similar issue several times. If you are calling = rte_eth_rx_burst too frequently, NIC begins losing packets no matter how = many CPU horse power you have (more you have, more it loses, actually). = In my case this situation occured when average burst size is less than = 20 packets or so. I'm not sure what's the reason for this behavior, but = I observed it on several applications on Intel 82599 10Gb cards.=E2=80=9D >=20 > So I=E2=80=99m wondering if anyone can explain at a lower level what = happens when you poll =E2=80=9Ctoo often=E2=80=9D, and if there are any = practical workarounds. We=E2=80=99re using this same program and DPDK = version to process 10G line-rate in other scenarios, so I=E2=80=99m = confident that the overall packet capture architecture is sound. >=20 > -Aaron >=20