[dpdk-dev] How to approach packet TX lockups

[dpdk-dev] How to approach packet TX lockups

[Matt Laswell]

Hey Folks,

I sent this to the users email list, but I'm not sure how many people are
actively reading that list at this point.  I'm dealing with a situation in
which my application loses the ability to transmit packets out of a port
during times of moderate stress.  I'd love to hear suggestions for how to
approach this problem, as I'm a bit at a loss at the moment.

Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on Haswell
processors.  I'm using the 82599 controller, configured to spread packets
across multiple queues.  Each queue is accessed by a different lcore in my
application; there is therefore concurrent access to the controller, but
not to any of the queues.  We're binding the ports to the igb_uio driver.
The symptoms I see are these:


   - All transmit out of a particular port stops
   - rte_eth_tx_burst() indicates that it is sending all of the packets
   that I give to it
   - rte_eth_stats_get() gives me stats indicating that no packets are
   being sent on the affected port.  Also, no tx errors, and no pause frames
   sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
   - All other ports continue to work normally
   - The affected port continues to receive packets without problems; only
   TX is affected
   - Resetting the port via rte_eth_dev_stop() and rte_eth_dev_start()
   restores things and packets can flow again
   - The problem is replicable on multiple devices, and doesn't follow one
   particular port

I've tried calling rte_mbuf_sanity_check() on all packets before sending
them.  I've also instrumented my code to look for packets that have already
been sent or freed, as well as cycles in chained packets being sent.  I
also put a lock around all accesses to rte_eth* calls to synchronize access
to the NIC.  Given some recent discussion here, I also tried changing the
TX RS threshold from 0 to 32, 16, and 1.  None of these strategies proved
effective.

Like I said at the top, I'm a little at a loss at this point.  If you were
dealing with this set of symptoms, how would you proceed?

Thanks in advance.

--
Matt Laswell
infinite io, inc.
laswell@infiniteio.com

Re: [dpdk-dev] How to approach packet TX lockups

[Stephen Hemminger]

On Mon, 16 Nov 2015 17:48:35 -0600
Matt Laswell <laswell@infiniteio.com> wrote:

> Hey Folks,
> 
> I sent this to the users email list, but I'm not sure how many people are
> actively reading that list at this point.  I'm dealing with a situation in
> which my application loses the ability to transmit packets out of a port
> during times of moderate stress.  I'd love to hear suggestions for how to
> approach this problem, as I'm a bit at a loss at the moment.
> 
> Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on Haswell
> processors.  I'm using the 82599 controller, configured to spread packets
> across multiple queues.  Each queue is accessed by a different lcore in my
> application; there is therefore concurrent access to the controller, but
> not to any of the queues.  We're binding the ports to the igb_uio driver.
> The symptoms I see are these:
> 
> 
>    - All transmit out of a particular port stops
>    - rte_eth_tx_burst() indicates that it is sending all of the packets
>    that I give to it
>    - rte_eth_stats_get() gives me stats indicating that no packets are
>    being sent on the affected port.  Also, no tx errors, and no pause frames
>    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
>    - All other ports continue to work normally
>    - The affected port continues to receive packets without problems; only
>    TX is affected
>    - Resetting the port via rte_eth_dev_stop() and rte_eth_dev_start()
>    restores things and packets can flow again
>    - The problem is replicable on multiple devices, and doesn't follow one
>    particular port
> 
> I've tried calling rte_mbuf_sanity_check() on all packets before sending
> them.  I've also instrumented my code to look for packets that have already
> been sent or freed, as well as cycles in chained packets being sent.  I
> also put a lock around all accesses to rte_eth* calls to synchronize access
> to the NIC.  Given some recent discussion here, I also tried changing the
> TX RS threshold from 0 to 32, 16, and 1.  None of these strategies proved
> effective.
> 
> Like I said at the top, I'm a little at a loss at this point.  If you were
> dealing with this set of symptoms, how would you proceed?
> 

I remember some issues with old DPDK 1.6 with some of the prefetch
thresholds on 82599. You would be better off going to a later DPDK
version.

Re: [dpdk-dev] How to approach packet TX lockups

[Matt Laswell]

Hey Stephen,

Thanks a lot; that's really useful information.  Unfortunately, I'm at a
stage in our release cycle where upgrading to a new version of DPDK isn't
feasible.  Any chance you (or others reading this) has a pointer to the
relevant changes?  While I can't afford to upgrade DPDK entirely,
backporting targeted fixes is more doable.

Again, thanks.

- Matt


On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Mon, 16 Nov 2015 17:48:35 -0600
> Matt Laswell <laswell@infiniteio.com> wrote:
>
> > Hey Folks,
> >
> > I sent this to the users email list, but I'm not sure how many people are
> > actively reading that list at this point.  I'm dealing with a situation
> in
> > which my application loses the ability to transmit packets out of a port
> > during times of moderate stress.  I'd love to hear suggestions for how to
> > approach this problem, as I'm a bit at a loss at the moment.
> >
> > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on Haswell
> > processors.  I'm using the 82599 controller, configured to spread packets
> > across multiple queues.  Each queue is accessed by a different lcore in
> my
> > application; there is therefore concurrent access to the controller, but
> > not to any of the queues.  We're binding the ports to the igb_uio driver.
> > The symptoms I see are these:
> >
> >
> >    - All transmit out of a particular port stops
> >    - rte_eth_tx_burst() indicates that it is sending all of the packets
> >    that I give to it
> >    - rte_eth_stats_get() gives me stats indicating that no packets are
> >    being sent on the affected port.  Also, no tx errors, and no pause
> frames
> >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> >    - All other ports continue to work normally
> >    - The affected port continues to receive packets without problems;
> only
> >    TX is affected
> >    - Resetting the port via rte_eth_dev_stop() and rte_eth_dev_start()
> >    restores things and packets can flow again
> >    - The problem is replicable on multiple devices, and doesn't follow
> one
> >    particular port
> >
> > I've tried calling rte_mbuf_sanity_check() on all packets before sending
> > them.  I've also instrumented my code to look for packets that have
> already
> > been sent or freed, as well as cycles in chained packets being sent.  I
> > also put a lock around all accesses to rte_eth* calls to synchronize
> access
> > to the NIC.  Given some recent discussion here, I also tried changing the
> > TX RS threshold from 0 to 32, 16, and 1.  None of these strategies proved
> > effective.
> >
> > Like I said at the top, I'm a little at a loss at this point.  If you
> were
> > dealing with this set of symptoms, how would you proceed?
> >
>
> I remember some issues with old DPDK 1.6 with some of the prefetch
> thresholds on 82599. You would be better off going to a later DPDK
> version.
>

Re: [dpdk-dev] How to approach packet TX lockups

[Stephen Hemminger]

On Mon, 16 Nov 2015 18:49:15 -0600
Matt Laswell <laswell@infiniteio.com> wrote:

> Hey Stephen,
> 
> Thanks a lot; that's really useful information.  Unfortunately, I'm at a
> stage in our release cycle where upgrading to a new version of DPDK isn't
> feasible.  Any chance you (or others reading this) has a pointer to the
> relevant changes?  While I can't afford to upgrade DPDK entirely,
> backporting targeted fixes is more doable.
> 
> Again, thanks.
> 
> - Matt
> 
> 
> On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> stephen@networkplumber.org> wrote:
> 
> > On Mon, 16 Nov 2015 17:48:35 -0600
> > Matt Laswell <laswell@infiniteio.com> wrote:
> >
> > > Hey Folks,
> > >
> > > I sent this to the users email list, but I'm not sure how many people are
> > > actively reading that list at this point.  I'm dealing with a situation
> > in
> > > which my application loses the ability to transmit packets out of a port
> > > during times of moderate stress.  I'd love to hear suggestions for how to
> > > approach this problem, as I'm a bit at a loss at the moment.
> > >
> > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on Haswell
> > > processors.  I'm using the 82599 controller, configured to spread packets
> > > across multiple queues.  Each queue is accessed by a different lcore in
> > my
> > > application; there is therefore concurrent access to the controller, but
> > > not to any of the queues.  We're binding the ports to the igb_uio driver.
> > > The symptoms I see are these:
> > >
> > >
> > >    - All transmit out of a particular port stops
> > >    - rte_eth_tx_burst() indicates that it is sending all of the packets
> > >    that I give to it
> > >    - rte_eth_stats_get() gives me stats indicating that no packets are
> > >    being sent on the affected port.  Also, no tx errors, and no pause
> > frames
> > >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> > >    - All other ports continue to work normally
> > >    - The affected port continues to receive packets without problems;
> > only
> > >    TX is affected
> > >    - Resetting the port via rte_eth_dev_stop() and rte_eth_dev_start()
> > >    restores things and packets can flow again
> > >    - The problem is replicable on multiple devices, and doesn't follow
> > one
> > >    particular port
> > >
> > > I've tried calling rte_mbuf_sanity_check() on all packets before sending
> > > them.  I've also instrumented my code to look for packets that have
> > already
> > > been sent or freed, as well as cycles in chained packets being sent.  I
> > > also put a lock around all accesses to rte_eth* calls to synchronize
> > access
> > > to the NIC.  Given some recent discussion here, I also tried changing the
> > > TX RS threshold from 0 to 32, 16, and 1.  None of these strategies proved
> > > effective.
> > >
> > > Like I said at the top, I'm a little at a loss at this point.  If you
> > were
> > > dealing with this set of symptoms, how would you proceed?
> > >
> >
> > I remember some issues with old DPDK 1.6 with some of the prefetch
> > thresholds on 82599. You would be better off going to a later DPDK
> > version.
> >

I hope you are on 1.6.0r2 at least??

With older DPDK there was no way to get driver to tell you what the
preferred settings were for pthresh/hthresh/wthresh. And the values
in Intel sample applications were broken on some hardware.

I remember reverse engineering the safe values from reading the Linux driver.

The Linux driver is much better tested than the DPDK one...
In the Linux driver, the Transmit Descriptor Controller (txdctl)
is fixed at (for transmit)
   wthresh = 1
   hthresh = 1
   pthresh = 32

The DPDK 2.2 driver uses:
    wthresh = 0
    hthresh = 0
    pthresh = 32

Re: [dpdk-dev] How to approach packet TX lockups

[Matthew Hall]

On Mon, Nov 16, 2015 at 05:31:29PM -0800, Stephen Hemminger wrote:
> The DPDK 2.2 driver uses:
>     wthresh = 0
>     hthresh = 0
>     pthresh = 32

Stephen,

I thought the zero values lead to doing the auto-config by the driver itself?

Matthew.

Re: [dpdk-dev] How to approach packet TX lockups

[Matt Laswell]

Yes, we're on 1.6r2.  That said, I've tried a number of different values
for the thresholds without a lot of luck.  Setting wthresh/hthresh/pthresh
to 0/0/32 or 0/0/0 doesn't appear to fix things.  And, as Matthew
suggested, I'm pretty sure using 0 for the thresholds leads to auto-config
by the driver.  I also tried 1/1/32, which required that I also change the
rs_thresh value from 0 to 1 to work around a panic in PMD initialization
("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1").

Any other suggestions?

On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Mon, 16 Nov 2015 18:49:15 -0600
> Matt Laswell <laswell@infiniteio.com> wrote:
>
> > Hey Stephen,
> >
> > Thanks a lot; that's really useful information.  Unfortunately, I'm at a
> > stage in our release cycle where upgrading to a new version of DPDK isn't
> > feasible.  Any chance you (or others reading this) has a pointer to the
> > relevant changes?  While I can't afford to upgrade DPDK entirely,
> > backporting targeted fixes is more doable.
> >
> > Again, thanks.
> >
> > - Matt
> >
> >
> > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> > stephen@networkplumber.org> wrote:
> >
> > > On Mon, 16 Nov 2015 17:48:35 -0600
> > > Matt Laswell <laswell@infiniteio.com> wrote:
> > >
> > > > Hey Folks,
> > > >
> > > > I sent this to the users email list, but I'm not sure how many
> people are
> > > > actively reading that list at this point.  I'm dealing with a
> situation
> > > in
> > > > which my application loses the ability to transmit packets out of a
> port
> > > > during times of moderate stress.  I'd love to hear suggestions for
> how to
> > > > approach this problem, as I'm a bit at a loss at the moment.
> > > >
> > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on
> Haswell
> > > > processors.  I'm using the 82599 controller, configured to spread
> packets
> > > > across multiple queues.  Each queue is accessed by a different lcore
> in
> > > my
> > > > application; there is therefore concurrent access to the controller,
> but
> > > > not to any of the queues.  We're binding the ports to the igb_uio
> driver.
> > > > The symptoms I see are these:
> > > >
> > > >
> > > >    - All transmit out of a particular port stops
> > > >    - rte_eth_tx_burst() indicates that it is sending all of the
> packets
> > > >    that I give to it
> > > >    - rte_eth_stats_get() gives me stats indicating that no packets
> are
> > > >    being sent on the affected port.  Also, no tx errors, and no pause
> > > frames
> > > >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> > > >    - All other ports continue to work normally
> > > >    - The affected port continues to receive packets without problems;
> > > only
> > > >    TX is affected
> > > >    - Resetting the port via rte_eth_dev_stop() and
> rte_eth_dev_start()
> > > >    restores things and packets can flow again
> > > >    - The problem is replicable on multiple devices, and doesn't
> follow
> > > one
> > > >    particular port
> > > >
> > > > I've tried calling rte_mbuf_sanity_check() on all packets before
> sending
> > > > them.  I've also instrumented my code to look for packets that have
> > > already
> > > > been sent or freed, as well as cycles in chained packets being
> sent.  I
> > > > also put a lock around all accesses to rte_eth* calls to synchronize
> > > access
> > > > to the NIC.  Given some recent discussion here, I also tried
> changing the
> > > > TX RS threshold from 0 to 32, 16, and 1.  None of these strategies
> proved
> > > > effective.
> > > >
> > > > Like I said at the top, I'm a little at a loss at this point.  If you
> > > were
> > > > dealing with this set of symptoms, how would you proceed?
> > > >
> > >
> > > I remember some issues with old DPDK 1.6 with some of the prefetch
> > > thresholds on 82599. You would be better off going to a later DPDK
> > > version.
> > >
>
> I hope you are on 1.6.0r2 at least??
>
> With older DPDK there was no way to get driver to tell you what the
> preferred settings were for pthresh/hthresh/wthresh. And the values
> in Intel sample applications were broken on some hardware.
>
> I remember reverse engineering the safe values from reading the Linux
> driver.
>
> The Linux driver is much better tested than the DPDK one...
> In the Linux driver, the Transmit Descriptor Controller (txdctl)
> is fixed at (for transmit)
>    wthresh = 1
>    hthresh = 1
>    pthresh = 32
>
> The DPDK 2.2 driver uses:
>     wthresh = 0
>     hthresh = 0
>     pthresh = 32
>
>
>
>
>
>
>

Re: [dpdk-dev] How to approach packet TX lockups

[Ananyev, Konstantin]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matt Laswell
> Sent: Tuesday, November 17, 2015 2:24 PM
> To: Stephen Hemminger
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] How to approach packet TX lockups
> 
> Yes, we're on 1.6r2.  That said, I've tried a number of different values
> for the thresholds without a lot of luck.  Setting wthresh/hthresh/pthresh
> to 0/0/32 or 0/0/0 doesn't appear to fix things.  And, as Matthew
> suggested, I'm pretty sure using 0 for the thresholds leads to auto-config
> by the driver.  I also tried 1/1/32, which required that I also change the
> rs_thresh value from 0 to 1 to work around a panic in PMD initialization
> ("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1").
> 
> Any other suggestions?

That's not only DPDK code changed since 1.6.
I am pretty sure that we also have a new update of shared code since then
(and as I remember probably more than one).
One suggestion would be at least try to upgrade the shared code up to the latest.
Another one - even if you can't upgrade to 2.2 in you production environment,
it probably worth to do that in some test environment and then check does the problem persist.
If yes,  then we'll need some guidance how to reproduce it.

Another question it is not clear what TX function do you use?
Konstantin

> 
> On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger <
> stephen@networkplumber.org> wrote:
> 
> > On Mon, 16 Nov 2015 18:49:15 -0600
> > Matt Laswell <laswell@infiniteio.com> wrote:
> >
> > > Hey Stephen,
> > >
> > > Thanks a lot; that's really useful information.  Unfortunately, I'm at a
> > > stage in our release cycle where upgrading to a new version of DPDK isn't
> > > feasible.  Any chance you (or others reading this) has a pointer to the
> > > relevant changes?  While I can't afford to upgrade DPDK entirely,
> > > backporting targeted fixes is more doable.
> > >
> > > Again, thanks.
> > >
> > > - Matt
> > >
> > >
> > > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> > > stephen@networkplumber.org> wrote:
> > >
> > > > On Mon, 16 Nov 2015 17:48:35 -0600
> > > > Matt Laswell <laswell@infiniteio.com> wrote:
> > > >
> > > > > Hey Folks,
> > > > >
> > > > > I sent this to the users email list, but I'm not sure how many
> > people are
> > > > > actively reading that list at this point.  I'm dealing with a
> > situation
> > > > in
> > > > > which my application loses the ability to transmit packets out of a
> > port
> > > > > during times of moderate stress.  I'd love to hear suggestions for
> > how to
> > > > > approach this problem, as I'm a bit at a loss at the moment.
> > > > >
> > > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on
> > Haswell
> > > > > processors.  I'm using the 82599 controller, configured to spread
> > packets
> > > > > across multiple queues.  Each queue is accessed by a different lcore
> > in
> > > > my
> > > > > application; there is therefore concurrent access to the controller,
> > but
> > > > > not to any of the queues.  We're binding the ports to the igb_uio
> > driver.
> > > > > The symptoms I see are these:
> > > > >
> > > > >
> > > > >    - All transmit out of a particular port stops
> > > > >    - rte_eth_tx_burst() indicates that it is sending all of the
> > packets
> > > > >    that I give to it
> > > > >    - rte_eth_stats_get() gives me stats indicating that no packets
> > are
> > > > >    being sent on the affected port.  Also, no tx errors, and no pause
> > > > frames
> > > > >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> > > > >    - All other ports continue to work normally
> > > > >    - The affected port continues to receive packets without problems;
> > > > only
> > > > >    TX is affected
> > > > >    - Resetting the port via rte_eth_dev_stop() and
> > rte_eth_dev_start()
> > > > >    restores things and packets can flow again
> > > > >    - The problem is replicable on multiple devices, and doesn't
> > follow
> > > > one
> > > > >    particular port
> > > > >
> > > > > I've tried calling rte_mbuf_sanity_check() on all packets before
> > sending
> > > > > them.  I've also instrumented my code to look for packets that have
> > > > already
> > > > > been sent or freed, as well as cycles in chained packets being
> > sent.  I
> > > > > also put a lock around all accesses to rte_eth* calls to synchronize
> > > > access
> > > > > to the NIC.  Given some recent discussion here, I also tried
> > changing the
> > > > > TX RS threshold from 0 to 32, 16, and 1.  None of these strategies
> > proved
> > > > > effective.
> > > > >
> > > > > Like I said at the top, I'm a little at a loss at this point.  If you
> > > > were
> > > > > dealing with this set of symptoms, how would you proceed?
> > > > >
> > > >
> > > > I remember some issues with old DPDK 1.6 with some of the prefetch
> > > > thresholds on 82599. You would be better off going to a later DPDK
> > > > version.
> > > >
> >
> > I hope you are on 1.6.0r2 at least??
> >
> > With older DPDK there was no way to get driver to tell you what the
> > preferred settings were for pthresh/hthresh/wthresh. And the values
> > in Intel sample applications were broken on some hardware.
> >
> > I remember reverse engineering the safe values from reading the Linux
> > driver.
> >
> > The Linux driver is much better tested than the DPDK one...
> > In the Linux driver, the Transmit Descriptor Controller (txdctl)
> > is fixed at (for transmit)
> >    wthresh = 1
> >    hthresh = 1
> >    pthresh = 32
> >
> > The DPDK 2.2 driver uses:
> >     wthresh = 0
> >     hthresh = 0
> >     pthresh = 32
> >
> >
> >
> >
> >
> >
> >

Re: [dpdk-dev] How to approach packet TX lockups

[Matt Laswell]

Hey Konstantin,

Moving from 1.6r2 to 2.2 is going to be a pretty significant change due to
things like changes in the MBuf format, API differences, etc.  Even as an
experiment, that's an awfully large change to absorb.  Is there a subset
that you're referring to that could be more readily included without
modifying so many touch points into DPDK?

For reference, my transmit function is  rte_eth_tx_burst().  It seems to
reliably tell me that it has enqueued all of the packets that I gave it,
however the stats from rte_eth_stats_get() indicate that no packets are
actually being sent.

Thanks,

- Matt

On Tue, Nov 17, 2015 at 8:44 AM, Ananyev, Konstantin <
konstantin.ananyev@intel.com> wrote:

>
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matt Laswell
> > Sent: Tuesday, November 17, 2015 2:24 PM
> > To: Stephen Hemminger
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] How to approach packet TX lockups
> >
> > Yes, we're on 1.6r2.  That said, I've tried a number of different values
> > for the thresholds without a lot of luck.  Setting
> wthresh/hthresh/pthresh
> > to 0/0/32 or 0/0/0 doesn't appear to fix things.  And, as Matthew
> > suggested, I'm pretty sure using 0 for the thresholds leads to
> auto-config
> > by the driver.  I also tried 1/1/32, which required that I also change
> the
> > rs_thresh value from 0 to 1 to work around a panic in PMD initialization
> > ("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1").
> >
> > Any other suggestions?
>
> That's not only DPDK code changed since 1.6.
> I am pretty sure that we also have a new update of shared code since then
> (and as I remember probably more than one).
> One suggestion would be at least try to upgrade the shared code up to the
> latest.
> Another one - even if you can't upgrade to 2.2 in you production
> environment,
> it probably worth to do that in some test environment and then check does
> the problem persist.
> If yes,  then we'll need some guidance how to reproduce it.
>
> Another question it is not clear what TX function do you use?
> Konstantin
>
> >
> > On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger <
> > stephen@networkplumber.org> wrote:
> >
> > > On Mon, 16 Nov 2015 18:49:15 -0600
> > > Matt Laswell <laswell@infiniteio.com> wrote:
> > >
> > > > Hey Stephen,
> > > >
> > > > Thanks a lot; that's really useful information.  Unfortunately, I'm
> at a
> > > > stage in our release cycle where upgrading to a new version of DPDK
> isn't
> > > > feasible.  Any chance you (or others reading this) has a pointer to
> the
> > > > relevant changes?  While I can't afford to upgrade DPDK entirely,
> > > > backporting targeted fixes is more doable.
> > > >
> > > > Again, thanks.
> > > >
> > > > - Matt
> > > >
> > > >
> > > > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> > > > stephen@networkplumber.org> wrote:
> > > >
> > > > > On Mon, 16 Nov 2015 17:48:35 -0600
> > > > > Matt Laswell <laswell@infiniteio.com> wrote:
> > > > >
> > > > > > Hey Folks,
> > > > > >
> > > > > > I sent this to the users email list, but I'm not sure how many
> > > people are
> > > > > > actively reading that list at this point.  I'm dealing with a
> > > situation
> > > > > in
> > > > > > which my application loses the ability to transmit packets out
> of a
> > > port
> > > > > > during times of moderate stress.  I'd love to hear suggestions
> for
> > > how to
> > > > > > approach this problem, as I'm a bit at a loss at the moment.
> > > > > >
> > > > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on
> > > Haswell
> > > > > > processors.  I'm using the 82599 controller, configured to spread
> > > packets
> > > > > > across multiple queues.  Each queue is accessed by a different
> lcore
> > > in
> > > > > my
> > > > > > application; there is therefore concurrent access to the
> controller,
> > > but
> > > > > > not to any of the queues.  We're binding the ports to the igb_uio
> > > driver.
> > > > > > The symptoms I see are these:
> > > > > >
> > > > > >
> > > > > >    - All transmit out of a particular port stops
> > > > > >    - rte_eth_tx_burst() indicates that it is sending all of the
> > > packets
> > > > > >    that I give to it
> > > > > >    - rte_eth_stats_get() gives me stats indicating that no
> packets
> > > are
> > > > > >    being sent on the affected port.  Also, no tx errors, and no
> pause
> > > > > frames
> > > > > >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> > > > > >    - All other ports continue to work normally
> > > > > >    - The affected port continues to receive packets without
> problems;
> > > > > only
> > > > > >    TX is affected
> > > > > >    - Resetting the port via rte_eth_dev_stop() and
> > > rte_eth_dev_start()
> > > > > >    restores things and packets can flow again
> > > > > >    - The problem is replicable on multiple devices, and doesn't
> > > follow
> > > > > one
> > > > > >    particular port
> > > > > >
> > > > > > I've tried calling rte_mbuf_sanity_check() on all packets before
> > > sending
> > > > > > them.  I've also instrumented my code to look for packets that
> have
> > > > > already
> > > > > > been sent or freed, as well as cycles in chained packets being
> > > sent.  I
> > > > > > also put a lock around all accesses to rte_eth* calls to
> synchronize
> > > > > access
> > > > > > to the NIC.  Given some recent discussion here, I also tried
> > > changing the
> > > > > > TX RS threshold from 0 to 32, 16, and 1.  None of these
> strategies
> > > proved
> > > > > > effective.
> > > > > >
> > > > > > Like I said at the top, I'm a little at a loss at this point.
> If you
> > > > > were
> > > > > > dealing with this set of symptoms, how would you proceed?
> > > > > >
> > > > >
> > > > > I remember some issues with old DPDK 1.6 with some of the prefetch
> > > > > thresholds on 82599. You would be better off going to a later DPDK
> > > > > version.
> > > > >
> > >
> > > I hope you are on 1.6.0r2 at least??
> > >
> > > With older DPDK there was no way to get driver to tell you what the
> > > preferred settings were for pthresh/hthresh/wthresh. And the values
> > > in Intel sample applications were broken on some hardware.
> > >
> > > I remember reverse engineering the safe values from reading the Linux
> > > driver.
> > >
> > > The Linux driver is much better tested than the DPDK one...
> > > In the Linux driver, the Transmit Descriptor Controller (txdctl)
> > > is fixed at (for transmit)
> > >    wthresh = 1
> > >    hthresh = 1
> > >    pthresh = 32
> > >
> > > The DPDK 2.2 driver uses:
> > >     wthresh = 0
> > >     hthresh = 0
> > >     pthresh = 32
> > >
> > >
> > >
> > >
> > >
> > >
> > >
>

Re: [dpdk-dev] How to approach packet TX lockups

[Ananyev, Konstantin]

Hi Matt,

As I said, at least  try to upgrade contents of shared code to the latest one.
In previous releases: lib/librte_pmd_ixgbe/ixgbe, now located at: drivers/net/ixgbe/.

> For reference, my transmit function is  rte_eth_tx_burst().
I meant what ixgbe TX function it points to: ixgbe_xmit_pkts or ixgbe_xmit_pkts_simple()?
For ixgbe_xmit_pkts_simple() don’t set tx_rs_thresh > 32,
for ixgbe_xmit_pkts() the safest way is to set  tx_rs_thresh=1.
Though as I understand from your previous mails, you already did that, and it didn’t help.
Konstantin


From: Matt Laswell [mailto:laswell@infiniteio.com]
Sent: Tuesday, November 17, 2015 3:05 PM
To: Ananyev, Konstantin
Cc: Stephen Hemminger; dev@dpdk.org
Subject: Re: [dpdk-dev] How to approach packet TX lockups

Hey Konstantin,

Moving from 1.6r2 to 2.2 is going to be a pretty significant change due to things like changes in the MBuf format, API differences, etc.  Even as an experiment, that's an awfully large change to absorb.  Is there a subset that you're referring to that could be more readily included without modifying so many touch points into DPDK?

For reference, my transmit function is  rte_eth_tx_burst().  It seems to reliably tell me that it has enqueued all of the packets that I gave it, however the stats from rte_eth_stats_get() indicate that no packets are actually being sent.

Thanks,

- Matt

On Tue, Nov 17, 2015 at 8:44 AM, Ananyev, Konstantin <konstantin.ananyev@intel.com<mailto:konstantin.ananyev@intel.com>> wrote:


> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>] On Behalf Of Matt Laswell
> Sent: Tuesday, November 17, 2015 2:24 PM
> To: Stephen Hemminger
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: Re: [dpdk-dev] How to approach packet TX lockups
>
> Yes, we're on 1.6r2.  That said, I've tried a number of different values
> for the thresholds without a lot of luck.  Setting wthresh/hthresh/pthresh
> to 0/0/32 or 0/0/0 doesn't appear to fix things.  And, as Matthew
> suggested, I'm pretty sure using 0 for the thresholds leads to auto-config
> by the driver.  I also tried 1/1/32, which required that I also change the
> rs_thresh value from 0 to 1 to work around a panic in PMD initialization
> ("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1").
>
> Any other suggestions?

That's not only DPDK code changed since 1.6.
I am pretty sure that we also have a new update of shared code since then
(and as I remember probably more than one).
One suggestion would be at least try to upgrade the shared code up to the latest.
Another one - even if you can't upgrade to 2.2 in you production environment,
it probably worth to do that in some test environment and then check does the problem persist.
If yes,  then we'll need some guidance how to reproduce it.

Another question it is not clear what TX function do you use?
Konstantin

>
> On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger <
> stephen@networkplumber.org<mailto:stephen@networkplumber.org>> wrote:
>
> > On Mon, 16 Nov 2015 18:49:15 -0600
> > Matt Laswell <laswell@infiniteio.com<mailto:laswell@infiniteio.com>> wrote:
> >
> > > Hey Stephen,
> > >
> > > Thanks a lot; that's really useful information.  Unfortunately, I'm at a
> > > stage in our release cycle where upgrading to a new version of DPDK isn't
> > > feasible.  Any chance you (or others reading this) has a pointer to the
> > > relevant changes?  While I can't afford to upgrade DPDK entirely,
> > > backporting targeted fixes is more doable.
> > >
> > > Again, thanks.
> > >
> > > - Matt
> > >
> > >
> > > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> > > stephen@networkplumber.org<mailto:stephen@networkplumber.org>> wrote:
> > >
> > > > On Mon, 16 Nov 2015 17:48:35 -0600
> > > > Matt Laswell <laswell@infiniteio.com<mailto:laswell@infiniteio.com>> wrote:
> > > >
> > > > > Hey Folks,
> > > > >
> > > > > I sent this to the users email list, but I'm not sure how many
> > people are
> > > > > actively reading that list at this point.  I'm dealing with a
> > situation
> > > > in
> > > > > which my application loses the ability to transmit packets out of a
> > port
> > > > > during times of moderate stress.  I'd love to hear suggestions for
> > how to
> > > > > approach this problem, as I'm a bit at a loss at the moment.
> > > > >
> > > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on
> > Haswell
> > > > > processors.  I'm using the 82599 controller, configured to spread
> > packets
> > > > > across multiple queues.  Each queue is accessed by a different lcore
> > in
> > > > my
> > > > > application; there is therefore concurrent access to the controller,
> > but
> > > > > not to any of the queues.  We're binding the ports to the igb_uio
> > driver.
> > > > > The symptoms I see are these:
> > > > >
> > > > >
> > > > >    - All transmit out of a particular port stops
> > > > >    - rte_eth_tx_burst() indicates that it is sending all of the
> > packets
> > > > >    that I give to it
> > > > >    - rte_eth_stats_get() gives me stats indicating that no packets
> > are
> > > > >    being sent on the affected port.  Also, no tx errors, and no pause
> > > > frames
> > > > >    sent or received (opackets = 0, obytes = 0, oerrors = 0, etc.)
> > > > >    - All other ports continue to work normally
> > > > >    - The affected port continues to receive packets without problems;
> > > > only
> > > > >    TX is affected
> > > > >    - Resetting the port via rte_eth_dev_stop() and
> > rte_eth_dev_start()
> > > > >    restores things and packets can flow again
> > > > >    - The problem is replicable on multiple devices, and doesn't
> > follow
> > > > one
> > > > >    particular port
> > > > >
> > > > > I've tried calling rte_mbuf_sanity_check() on all packets before
> > sending
> > > > > them.  I've also instrumented my code to look for packets that have
> > > > already
> > > > > been sent or freed, as well as cycles in chained packets being
> > sent.  I
> > > > > also put a lock around all accesses to rte_eth* calls to synchronize
> > > > access
> > > > > to the NIC.  Given some recent discussion here, I also tried
> > changing the
> > > > > TX RS threshold from 0 to 32, 16, and 1.  None of these strategies
> > proved
> > > > > effective.
> > > > >
> > > > > Like I said at the top, I'm a little at a loss at this point.  If you
> > > > were
> > > > > dealing with this set of symptoms, how would you proceed?
> > > > >
> > > >
> > > > I remember some issues with old DPDK 1.6 with some of the prefetch
> > > > thresholds on 82599. You would be better off going to a later DPDK
> > > > version.
> > > >
> >
> > I hope you are on 1.6.0r2 at least??
> >
> > With older DPDK there was no way to get driver to tell you what the
> > preferred settings were for pthresh/hthresh/wthresh. And the values
> > in Intel sample applications were broken on some hardware.
> >
> > I remember reverse engineering the safe values from reading the Linux
> > driver.
> >
> > The Linux driver is much better tested than the DPDK one...
> > In the Linux driver, the Transmit Descriptor Controller (txdctl)
> > is fixed at (for transmit)
> >    wthresh = 1
> >    hthresh = 1
> >    pthresh = 32
> >
> > The DPDK 2.2 driver uses:
> >     wthresh = 0
> >     hthresh = 0
> >     pthresh = 32
> >
> >
> >
> >
> >
> >
> >

Re: [dpdk-dev] How to approach packet TX lockups

[Matt Laswell]

Thanks, I'll give that a try.

In my environment, I'm pretty sure we're using the fully-featured
ixgbe_xmit_pkts() and not _simple().   If setting rs_thresh=1 is safer,
I'll stick with that.

Again, thanks to all for the assistance.

- Matt

On Tue, Nov 17, 2015 at 10:20 AM, Ananyev, Konstantin <
konstantin.ananyev@intel.com> wrote:

> Hi Matt,
>
>
>
> As I said, at least  try to upgrade contents of shared code to the latest
> one.
>
> In previous releases: lib/librte_pmd_ixgbe/ixgbe, now located at:
> drivers/net/ixgbe/.
>
>
>
> > For reference, my transmit function is  rte_eth_tx_burst().
>
> I meant what ixgbe TX function it points to: ixgbe_xmit_pkts or
> ixgbe_xmit_pkts_simple()?
>
> For ixgbe_xmit_pkts_simple() don’t set tx_rs_thresh > 32,
>
> for ixgbe_xmit_pkts() the safest way is to set  tx_rs_thresh=1.
>
> Though as I understand from your previous mails, you already did that, and
> it didn’t help.
>
> Konstantin
>
>
>
>
>
> *From:* Matt Laswell [mailto:laswell@infiniteio.com]
> *Sent:* Tuesday, November 17, 2015 3:05 PM
> *To:* Ananyev, Konstantin
> *Cc:* Stephen Hemminger; dev@dpdk.org
>
> *Subject:* Re: [dpdk-dev] How to approach packet TX lockups
>
>
>
> Hey Konstantin,
>
>
>
> Moving from 1.6r2 to 2.2 is going to be a pretty significant change due to
> things like changes in the MBuf format, API differences, etc.  Even as an
> experiment, that's an awfully large change to absorb.  Is there a subset
> that you're referring to that could be more readily included without
> modifying so many touch points into DPDK?
>
>
>
> For reference, my transmit function is  rte_eth_tx_burst().  It seems to
> reliably tell me that it has enqueued all of the packets that I gave it,
> however the stats from rte_eth_stats_get() indicate that no packets are
> actually being sent.
>
>
>
> Thanks,
>
>
>
> - Matt
>
>
>
> On Tue, Nov 17, 2015 at 8:44 AM, Ananyev, Konstantin <
> konstantin.ananyev@intel.com> wrote:
>
>
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Matt Laswell
> > Sent: Tuesday, November 17, 2015 2:24 PM
> > To: Stephen Hemminger
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] How to approach packet TX lockups
> >
> > Yes, we're on 1.6r2.  That said, I've tried a number of different values
> > for the thresholds without a lot of luck.  Setting wthresh/hthresh/
> pthresh
> > to 0/0/32 or 0/0/0 doesn't appear to fix things.  And, as Matthew
> > suggested, I'm pretty sure using 0 for the thresholds leads to auto-
> config
> > by the driver.  I also tried 1/1/32, which required that I also change
> the
> > rs_thresh value from 0 to 1 to work around a panic in PMD initialization
> > ("TX WTHRESH must be set to 0 if tx_rs_thresh is greater than 1").
> >
> > Any other suggestions?
>
> That's not only DPDK code changed since 1.6.
> I am pretty sure that we also have a new update of shared code since then
> (and as I remember probably more than one).
> One suggestion would be at least try to upgrade the shared code up to the
> latest.
> Another one - even if you can't upgrade to 2.2 in you production
> environment,
> it probably worth to do that in some test environment and then check does
> the problem persist.
> If yes,  then we'll need some guidance how to reproduce it.
>
> Another question it is not clear what TX function do you use?
> Konstantin
>
>
> >
> > On Mon, Nov 16, 2015 at 7:31 PM, Stephen Hemminger <
> > stephen@networkplumber.org> wrote:
> >
> > > On Mon, 16 Nov 2015 18:49:15 -0600
> > > Matt Laswell <laswell@infiniteio.com> wrote:
> > >
> > > > Hey Stephen,
> > > >
> > > > Thanks a lot; that's really useful information.  Unfortunately, I'm
> at a
> > > > stage in our release cycle where upgrading to a new version of DPDK
> isn't
> > > > feasible.  Any chance you (or others reading this) has a pointer to
> the
> > > > relevant changes?  While I can't afford to upgrade DPDK entirely,
> > > > backporting targeted fixes is more doable.
> > > >
> > > > Again, thanks.
> > > >
> > > > - Matt
> > > >
> > > >
> > > > On Mon, Nov 16, 2015 at 6:12 PM, Stephen Hemminger <
> > > > stephen@networkplumber.org> wrote:
> > > >
> > > > > On Mon, 16 Nov 2015 17:48:35 -0600
> > > > > Matt Laswell <laswell@infiniteio.com> wrote:
> > > > >
> > > > > > Hey Folks,
> > > > > >
> > > > > > I sent this to the users email list, but I'm not sure how many
> > > people are
> > > > > > actively reading that list at this point.  I'm dealing with a
> > > situation
> > > > > in
> > > > > > which my application loses the ability to transmit packets out
> of a
> > > port
> > > > > > during times of moderate stress.  I'd love to hear suggestions
> for
> > > how to
> > > > > > approach this problem, as I'm a bit at a loss at the moment.
> > > > > >
> > > > > > Specifically, I'm using DPDK 1.6r2 running on Ubuntu 14.04LTS on
> > > Haswell
> > > > > > processors.  I'm using the 82599 controller, configured to spread
> > > packets
> > > > > > across multiple queues.  Each queue is accessed by a different
> lcore
> > > in
> > > > > my
> > > > > > application; there is therefore concurrent access to the
> controller,
> > > but
> > > > > > not to any of the queues.  We're binding the ports to the
> igb_uio
> > > driver.
> > > > > > The symptoms I see are these:
> > > > > >
> > > > > >
> > > > > >    - All transmit out of a particular port stops
> > > > > >    - rte_eth_tx_burst() indicates that it is sending all of the
> > > packets
> > > > > >    that I give to it
> > > > > >    - rte_eth_stats_get() gives me stats indicating that no
> packets
> > > are
> > > > > >    being sent on the affected port.  Also, no tx errors, and no
> pause
> > > > > frames
> > > > > >    sent or received (opackets = 0, obytes = 0, oerrors = 0,
> etc.)
> > > > > >    - All other ports continue to work normally
> > > > > >    - The affected port continues to receive packets without
> problems;
> > > > > only
> > > > > >    TX is affected
> > > > > >    - Resetting the port via rte_eth_dev_stop() and
> > > rte_eth_dev_start()
> > > > > >    restores things and packets can flow again
> > > > > >    - The problem is replicable on multiple devices, and doesn't
> > > follow
> > > > > one
> > > > > >    particular port
> > > > > >
> > > > > > I've tried calling rte_mbuf_sanity_check() on all packets before
> > > sending
> > > > > > them.  I've also instrumented my code to look for packets that
> have
> > > > > already
> > > > > > been sent or freed, as well as cycles in chained packets being
> > > sent.  I
> > > > > > also put a lock around all accesses to rte_eth* calls to
> synchronize
> > > > > access
> > > > > > to the NIC.  Given some recent discussion here, I also tried
> > > changing the
> > > > > > TX RS threshold from 0 to 32, 16, and 1.  None of these
> strategies
> > > proved
> > > > > > effective.
> > > > > >
> > > > > > Like I said at the top, I'm a little at a loss at this point.
> If you
> > > > > were
> > > > > > dealing with this set of symptoms, how would you proceed?
> > > > > >
> > > > >
> > > > > I remember some issues with old DPDK 1.6 with some of the prefetch
> > > > > thresholds on 82599. You would be better off going to a later DPDK
> > > > > version.
> > > > >
> > >
> > > I hope you are on 1.6.0r2 at least??
> > >
> > > With older DPDK there was no way to get driver to tell you what the
> > > preferred settings were for pthresh/hthresh/wthresh. And the values
> > > in Intel sample applications were broken on some hardware.
> > >
> > > I remember reverse engineering the safe values from reading the Linux
> > > driver.
> > >
> > > The Linux driver is much better tested than the DPDK one...
> > > In the Linux driver, the Transmit Descriptor Controller (txdctl)
> > > is fixed at (for transmit)
> > >    wthresh = 1
> > >    hthresh = 1
> > >    pthresh = 32
> > >
> > > The DPDK 2.2 driver uses:
> > >     wthresh = 0
> > >     hthresh = 0
> > >     pthresh = 32
> > >
> > >
> > >
> > >
> > >
> > >
> > >
>
>
>