Re: [dpdk-dev] How to approach packet TX lockups

[Matt Laswell <laswell@infiniteio.com>]

Hey Konstantin,

Moving from 1.6r2 to 2.2 is going to be a pretty significant change due to
things like changes in the MBuf format, API differences, etc.  Even as an
experiment, that's an awfully large change to absorb.  Is there a subset
that you're referring to that could be more readily included without
modifying so many touch points into DPDK?

For reference, my transmit function is  rte_eth_tx_burst().  It seems to
reliably tell me that it has enqueued all of the packets that I gave it,
however the stats from rte_eth_stats_get() indicate that no packets are
actually being sent.

Thanks,

- Matt

On Tue, Nov 17, 2015 at 8:44 AM, Ananyev, Konstantin <
konstantin.ananyev@intel.com> wrote:

>
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matt Laswell
> > Sent: Tuesday, November 17, 2015 2:24 PM
> > To: Stephen Hemminger
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] How to approach packet TX lockups
> >
> > Yes, we're on 1.6r2.  That said, I've tried a number of different values
> > for the thresholds without a lot of luck.  Setting
> wthresh/hthresh/pthresh
> > to 0/0/32 or 0/0/0 doesn't appear to fix things.  And, as Matthew
> > suggested, I'm pretty sure using 0 for the thresholds leads to
> auto-config
> > by the driver.  I also tried 1/1/32, which required that I also change
> the
> > rs_thresh value from 0 to 1 to work around a panic in PMD initialization
> > ("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1").
> >
> > Any other suggestions?
>
> That's not only DPDK code changed since 1.6.
> I am pretty sure that we also have a new update of shared code since then
> (and as I remember probably more than one).
> One suggestion would be at least try to upgrade the shared code up to the
> latest.
> Another one - even if you can't upgrade to 2.2 in you production
> environment,
> it probably worth to do that in some test environment and then check does
> the problem persist.
> If yes,  then we'll need some guidance how to reproduce it.
>
> Another question it is not clear what TX function do you use?
> Konstantin
>
> >
> > On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger <
> > stephen@networkplumber.org> wrote:
> >
> > > On Mon, 16 Nov 2015 18:49:15 -0600
> > > Matt Laswell <laswell@infiniteio.com> wrote:
> > >
> > > > Hey Stephen,
> > > >
> > > > Thanks a lot; that's really useful information.  Unfortunately, I'm
> at a
> > > > stage in our release cycle where upgrading to a new version of DPDK
> isn't
> > > > feasible.  Any chance you (or others reading this) has a pointer to
> the
> > > > relevant changes?  While I can't afford to upgrade DPDK entirely,
> > > > backporting targeted fixes is more doable.
> > > >
> > > > Again, thanks.
> > > >
> > > > - Matt
> > > >
> > > >
> > > > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> > > > stephen@networkplumber.org> wrote:
> > > >
> > > > > On Mon, 16 Nov 2015 17:48:35 -0600
> > > > > Matt Laswell <laswell@infiniteio.com> wrote:
> > > > >
> > > > > > Hey Folks,
> > > > > >
> > > > > > I sent this to the users email list, but I'm not sure how many
> > > people are
> > > > > > actively reading that list at this point.  I'm dealing with a
> > > situation
> > > > > in
> > > > > > which my application loses the ability to transmit packets out
> of a
> > > port
> > > > > > during times of moderate stress.  I'd love to hear suggestions
> for
> > > how to
> > > > > > approach this problem, as I'm a bit at a loss at the moment.
> > > > > >
> > > > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on
> > > Haswell
> > > > > > processors.  I'm using the 82599 controller, configured to spread
> > > packets
> > > > > > across multiple queues.  Each queue is accessed by a different
> lcore
> > > in
> > > > > my
> > > > > > application; there is therefore concurrent access to the
> controller,
> > > but
> > > > > > not to any of the queues.  We're binding the ports to the igb_uio
> > > driver.
> > > > > > The symptoms I see are these:
> > > > > >
> > > > > >
> > > > > >    - All transmit out of a particular port stops
> > > > > >    - rte_eth_tx_burst() indicates that it is sending all of the
> > > packets
> > > > > >    that I give to it
> > > > > >    - rte_eth_stats_get() gives me stats indicating that no
> packets
> > > are
> > > > > >    being sent on the affected port.  Also, no tx errors, and no
> pause
> > > > > frames
> > > > > >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> > > > > >    - All other ports continue to work normally
> > > > > >    - The affected port continues to receive packets without
> problems;
> > > > > only
> > > > > >    TX is affected
> > > > > >    - Resetting the port via rte_eth_dev_stop() and
> > > rte_eth_dev_start()
> > > > > >    restores things and packets can flow again
> > > > > >    - The problem is replicable on multiple devices, and doesn't
> > > follow
> > > > > one
> > > > > >    particular port
> > > > > >
> > > > > > I've tried calling rte_mbuf_sanity_check() on all packets before
> > > sending
> > > > > > them.  I've also instrumented my code to look for packets that
> have
> > > > > already
> > > > > > been sent or freed, as well as cycles in chained packets being
> > > sent.  I
> > > > > > also put a lock around all accesses to rte_eth* calls to
> synchronize
> > > > > access
> > > > > > to the NIC.  Given some recent discussion here, I also tried
> > > changing the
> > > > > > TX RS threshold from 0 to 32, 16, and 1.  None of these
> strategies
> > > proved
> > > > > > effective.
> > > > > >
> > > > > > Like I said at the top, I'm a little at a loss at this point.
> If you
> > > > > were
> > > > > > dealing with this set of symptoms, how would you proceed?
> > > > > >
> > > > >
> > > > > I remember some issues with old DPDK 1.6 with some of the prefetch
> > > > > thresholds on 82599. You would be better off going to a later DPDK
> > > > > version.
> > > > >
> > >
> > > I hope you are on 1.6.0r2 at least??
> > >
> > > With older DPDK there was no way to get driver to tell you what the
> > > preferred settings were for pthresh/hthresh/wthresh. And the values
> > > in Intel sample applications were broken on some hardware.
> > >
> > > I remember reverse engineering the safe values from reading the Linux
> > > driver.
> > >
> > > The Linux driver is much better tested than the DPDK one...
> > > In the Linux driver, the Transmit Descriptor Controller (txdctl)
> > > is fixed at (for transmit)
> > >    wthresh = 1
> > >    hthresh = 1
> > >    pthresh = 32
> > >
> > > The DPDK 2.2 driver uses:
> > >     wthresh = 0
> > >     hthresh = 0
> > >     pthresh = 32
> > >
> > >
> > >
> > >
> > >
> > >
> > >
>