From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f169.google.com (mail-ig0-f169.google.com [209.85.213.169]) by dpdk.org (Postfix) with ESMTP id 00AF95A71 for ; Tue, 17 Nov 2015 16:04:50 +0100 (CET) Received: by igl9 with SMTP id 9so80832237igl.0 for ; Tue, 17 Nov 2015 07:04:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infiniteio-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=BjRW9SxCGUB6DOlZH7FDrhCeT0T5ZM303AFf2/yMdq4=; b=B+en9i8BnAXcGlq1csbcNxEdzftvReLUlCPKTmxhq+ijURKUnHVJPeDe/bHqqhh588 7Kp3W6jugc8sthLaimIgN9ShKzxqFK5ppuIqqzfTeQ4RtmKwscsrtgbjFKSseTyAexBz TNWvXfy2mGGyGwbIxTIeDRwZ0nsN38ZlvOALqFPl/XX2SbROdvyVFSb+pb56QjlBl2O7 sM96OHFJDe5VYIFJuSWxfT8Sd/YYbui2riNld+0kcos82sshqGAM7vbQ7jY8Cml9JSUV Jzyjq5zVSGxoxg86K6NxE6bte6A7Sre4D0+fzYNEl7/SN8JlIQCsnlKJRG65qVNhvP4g ClXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=BjRW9SxCGUB6DOlZH7FDrhCeT0T5ZM303AFf2/yMdq4=; b=QtNFvwszWFDtQZWV4WuctzWco4pXko5q/BBrxQrkhaJ4GtGFeJNJRRhXvr3kCAPrXG TWfv7UsJX1vUu84EfbnOcFlCCeQW/9VG69j+d52iMil0VZHBHE6qUwLt8ZXBgLdsP/CL 5G0aRLcOhqFRUL7nScjOJ2/pGspNT9SOMZl7xT8Zy1M1EJtiljlZT6zrpagBMCOlbzuw bOUhK7pJVNrtgwJ90Cd80VapeZ2Gw+S86alajuMVJolBBUHFzlsIFZrvlpL4cy3E5Z9K 1js5hns2xOZ1TMZK8PzWUBhcfNFIzD7gapHUVOiQJLG68O4U0nj094waBeGg4ahJ3ziN k7KA== X-Gm-Message-State: ALoCoQkTtxbU98+YqpZtuowdLDAvjtDKjp+eD/htMbxVsE6+uixgHo/EbP+eBhmq6SMBcSBs47DS MIME-Version: 1.0 X-Received: by 10.50.50.102 with SMTP id b6mr2559219igo.0.1447772690261; Tue, 17 Nov 2015 07:04:50 -0800 (PST) Received: by 10.107.140.15 with HTTP; Tue, 17 Nov 2015 07:04:50 -0800 (PST) In-Reply-To: <2601191342CEEE43887BDE71AB97725836AC9A40@irsmsx105.ger.corp.intel.com> References: <20151116161201.7e951097@samsung9> <20151116173129.2a429930@samsung9> <2601191342CEEE43887BDE71AB97725836AC9A40@irsmsx105.ger.corp.intel.com> Date: Tue, 17 Nov 2015 09:04:50 -0600 Message-ID: From: Matt Laswell To: "Ananyev, Konstantin" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] How to approach packet TX lockups X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 15:04:51 -0000 Hey Konstantin, Moving from 1.6r2 to 2.2 is going to be a pretty significant change due to things like changes in the MBuf format, API differences, etc. Even as an experiment, that's an awfully large change to absorb. Is there a subset that you're referring to that could be more readily included without modifying so many touch points into DPDK? For reference, my transmit function is rte_eth_tx_burst(). It seems to reliably tell me that it has enqueued all of the packets that I gave it, however the stats from rte_eth_stats_get() indicate that no packets are actually being sent. Thanks, - Matt On Tue, Nov 17, 2015 at 8:44 AM, Ananyev, Konstantin < konstantin.ananyev@intel.com> wrote: > > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matt Laswell > > Sent: Tuesday, November 17, 2015 2:24 PM > > To: Stephen Hemminger > > Cc: dev@dpdk.org > > Subject: Re: [dpdk-dev] How to approach packet TX lockups > > > > Yes, we're on 1.6r2. That said, I've tried a number of different values > > for the thresholds without a lot of luck. Setting > wthresh/hthresh/pthresh > > to 0/0/32 or 0/0/0 doesn't appear to fix things. And, as Matthew > > suggested, I'm pretty sure using 0 for the thresholds leads to > auto-config > > by the driver. I also tried 1/1/32, which required that I also change > the > > rs_thresh value from 0 to 1 to work around a panic in PMD initialization > > ("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1"). > > > > Any other suggestions? > > That's not only DPDK code changed since 1.6. > I am pretty sure that we also have a new update of shared code since then > (and as I remember probably more than one). > One suggestion would be at least try to upgrade the shared code up to the > latest. > Another one - even if you can't upgrade to 2.2 in you production > environment, > it probably worth to do that in some test environment and then check does > the problem persist. > If yes, then we'll need some guidance how to reproduce it. > > Another question it is not clear what TX function do you use? > Konstantin > > > > > On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger < > > stephen@networkplumber.org> wrote: > > > > > On Mon, 16 Nov 2015 18:49:15 -0600 > > > Matt Laswell wrote: > > > > > > > Hey Stephen, > > > > > > > > Thanks a lot; that's really useful information. Unfortunately, I'm > at a > > > > stage in our release cycle where upgrading to a new version of DPDK > isn't > > > > feasible. Any chance you (or others reading this) has a pointer to > the > > > > relevant changes? While I can't afford to upgrade DPDK entirely, > > > > backporting targeted fixes is more doable. > > > > > > > > Again, thanks. > > > > > > > > - Matt > > > > > > > > > > > > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger < > > > > stephen@networkplumber.org> wrote: > > > > > > > > > On Mon, 16 Nov 2015 17:48:35 -0600 > > > > > Matt Laswell wrote: > > > > > > > > > > > Hey Folks, > > > > > > > > > > > > I sent this to the users email list, but I'm not sure how many > > > people are > > > > > > actively reading that list at this point. I'm dealing with a > > > situation > > > > > in > > > > > > which my application loses the ability to transmit packets out > of a > > > port > > > > > > during times of moderate stress. I'd love to hear suggestions > for > > > how to > > > > > > approach this problem, as I'm a bit at a loss at the moment. > > > > > > > > > > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on > > > Haswell > > > > > > processors. I'm using the 82599 controller, configured to spread > > > packets > > > > > > across multiple queues. Each queue is accessed by a different > lcore > > > in > > > > > my > > > > > > application; there is therefore concurrent access to the > controller, > > > but > > > > > > not to any of the queues. We're binding the ports to the igb_uio > > > driver. > > > > > > The symptoms I see are these: > > > > > > > > > > > > > > > > > > - All transmit out of a particular port stops > > > > > > - rte_eth_tx_burst() indicates that it is sending all of the > > > packets > > > > > > that I give to it > > > > > > - rte_eth_stats_get() gives me stats indicating that no > packets > > > are > > > > > > being sent on the affected port. Also, no tx errors, and no > pause > > > > > frames > > > > > > sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.) > > > > > > - All other ports continue to work normally > > > > > > - The affected port continues to receive packets without > problems; > > > > > only > > > > > > TX is affected > > > > > > - Resetting the port via rte_eth_dev_stop() and > > > rte_eth_dev_start() > > > > > > restores things and packets can flow again > > > > > > - The problem is replicable on multiple devices, and doesn't > > > follow > > > > > one > > > > > > particular port > > > > > > > > > > > > I've tried calling rte_mbuf_sanity_check() on all packets before > > > sending > > > > > > them. I've also instrumented my code to look for packets that > have > > > > > already > > > > > > been sent or freed, as well as cycles in chained packets being > > > sent. I > > > > > > also put a lock around all accesses to rte_eth* calls to > synchronize > > > > > access > > > > > > to the NIC. Given some recent discussion here, I also tried > > > changing the > > > > > > TX RS threshold from 0 to 32, 16, and 1. None of these > strategies > > > proved > > > > > > effective. > > > > > > > > > > > > Like I said at the top, I'm a little at a loss at this point. > If you > > > > > were > > > > > > dealing with this set of symptoms, how would you proceed? > > > > > > > > > > > > > > > > I remember some issues with old DPDK 1.6 with some of the prefetch > > > > > thresholds on 82599. You would be better off going to a later DPDK > > > > > version. > > > > > > > > > > > I hope you are on 1.6.0r2 at least?? > > > > > > With older DPDK there was no way to get driver to tell you what the > > > preferred settings were for pthresh/hthresh/wthresh. And the values > > > in Intel sample applications were broken on some hardware. > > > > > > I remember reverse engineering the safe values from reading the Linux > > > driver. > > > > > > The Linux driver is much better tested than the DPDK one... > > > In the Linux driver, the Transmit Descriptor Controller (txdctl) > > > is fixed at (for transmit) > > > wthresh = 1 > > > hthresh = 1 > > > pthresh = 32 > > > > > > The DPDK 2.2 driver uses: > > > wthresh = 0 > > > hthresh = 0 > > > pthresh = 32 > > > > > > > > > > > > > > > > > > > > > >