Re: [dpdk-dev] How to approach packet TX lockups

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matt Laswell
> Sent: Tuesday, November 17, 2015 2:24 PM
> To: Stephen Hemminger
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] How to approach packet TX lockups
> 
> Yes, we're on 1.6r2.  That said, I've tried a number of different values
> for the thresholds without a lot of luck.  Setting wthresh/hthresh/pthresh
> to 0/0/32 or 0/0/0 doesn't appear to fix things.  And, as Matthew
> suggested, I'm pretty sure using 0 for the thresholds leads to auto-config
> by the driver.  I also tried 1/1/32, which required that I also change the
> rs_thresh value from 0 to 1 to work around a panic in PMD initialization
> ("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1").
> 
> Any other suggestions?

That's not only DPDK code changed since 1.6.
I am pretty sure that we also have a new update of shared code since then
(and as I remember probably more than one).
One suggestion would be at least try to upgrade the shared code up to the latest.
Another one - even if you can't upgrade to 2.2 in you production environment,
it probably worth to do that in some test environment and then check does the problem persist.
If yes,  then we'll need some guidance how to reproduce it.

Another question it is not clear what TX function do you use?
Konstantin

> 
> On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger <
> stephen@networkplumber.org> wrote:
> 
> > On Mon, 16 Nov 2015 18:49:15 -0600
> > Matt Laswell <laswell@infiniteio.com> wrote:
> >
> > > Hey Stephen,
> > >
> > > Thanks a lot; that's really useful information.  Unfortunately, I'm at a
> > > stage in our release cycle where upgrading to a new version of DPDK isn't
> > > feasible.  Any chance you (or others reading this) has a pointer to the
> > > relevant changes?  While I can't afford to upgrade DPDK entirely,
> > > backporting targeted fixes is more doable.
> > >
> > > Again, thanks.
> > >
> > > - Matt
> > >
> > >
> > > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> > > stephen@networkplumber.org> wrote:
> > >
> > > > On Mon, 16 Nov 2015 17:48:35 -0600
> > > > Matt Laswell <laswell@infiniteio.com> wrote:
> > > >
> > > > > Hey Folks,
> > > > >
> > > > > I sent this to the users email list, but I'm not sure how many
> > people are
> > > > > actively reading that list at this point.  I'm dealing with a
> > situation
> > > > in
> > > > > which my application loses the ability to transmit packets out of a
> > port
> > > > > during times of moderate stress.  I'd love to hear suggestions for
> > how to
> > > > > approach this problem, as I'm a bit at a loss at the moment.
> > > > >
> > > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on
> > Haswell
> > > > > processors.  I'm using the 82599 controller, configured to spread
> > packets
> > > > > across multiple queues.  Each queue is accessed by a different lcore
> > in
> > > > my
> > > > > application; there is therefore concurrent access to the controller,
> > but
> > > > > not to any of the queues.  We're binding the ports to the igb_uio
> > driver.
> > > > > The symptoms I see are these:
> > > > >
> > > > >
> > > > >    - All transmit out of a particular port stops
> > > > >    - rte_eth_tx_burst() indicates that it is sending all of the
> > packets
> > > > >    that I give to it
> > > > >    - rte_eth_stats_get() gives me stats indicating that no packets
> > are
> > > > >    being sent on the affected port.  Also, no tx errors, and no pause
> > > > frames
> > > > >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> > > > >    - All other ports continue to work normally
> > > > >    - The affected port continues to receive packets without problems;
> > > > only
> > > > >    TX is affected
> > > > >    - Resetting the port via rte_eth_dev_stop() and
> > rte_eth_dev_start()
> > > > >    restores things and packets can flow again
> > > > >    - The problem is replicable on multiple devices, and doesn't
> > follow
> > > > one
> > > > >    particular port
> > > > >
> > > > > I've tried calling rte_mbuf_sanity_check() on all packets before
> > sending
> > > > > them.  I've also instrumented my code to look for packets that have
> > > > already
> > > > > been sent or freed, as well as cycles in chained packets being
> > sent.  I
> > > > > also put a lock around all accesses to rte_eth* calls to synchronize
> > > > access
> > > > > to the NIC.  Given some recent discussion here, I also tried
> > changing the
> > > > > TX RS threshold from 0 to 32, 16, and 1.  None of these strategies
> > proved
> > > > > effective.
> > > > >
> > > > > Like I said at the top, I'm a little at a loss at this point.  If you
> > > > were
> > > > > dealing with this set of symptoms, how would you proceed?
> > > > >
> > > >
> > > > I remember some issues with old DPDK 1.6 with some of the prefetch
> > > > thresholds on 82599. You would be better off going to a later DPDK
> > > > version.
> > > >
> >
> > I hope you are on 1.6.0r2 at least??
> >
> > With older DPDK there was no way to get driver to tell you what the
> > preferred settings were for pthresh/hthresh/wthresh. And the values
> > in Intel sample applications were broken on some hardware.
> >
> > I remember reverse engineering the safe values from reading the Linux
> > driver.
> >
> > The Linux driver is much better tested than the DPDK one...
> > In the Linux driver, the Transmit Descriptor Controller (txdctl)
> > is fixed at (for transmit)
> >    wthresh = 1
> >    hthresh = 1
> >    pthresh = 32
> >
> > The DPDK 2.2 driver uses:
> >     wthresh = 0
> >     hthresh = 0
> >     pthresh = 32
> >
> >
> >
> >
> >
> >
> >