From: "Lu, Wenzhuo" <wenzhuo.lu@intel.com>
To: "Zhang, Helin" <helin.zhang@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v1] ixgbe_pmd: forbid tx_rs_thresh above 1 for all NICs but 82598
Date: Wed, 19 Aug 2015 00:42:15 +0000 [thread overview]
Message-ID: <6A0DE07E22DDAD4C9103DF62FEBC0909D3E116@shsmsx102.ccr.corp.intel.com> (raw)
In-Reply-To: <55CD7EA5.6060100@cloudius-systems.com>
Hi Helin,
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Vlad Zolotarov
> Sent: Friday, August 14, 2015 1:38 PM
> To: Zhang, Helin; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v1] ixgbe_pmd: forbid tx_rs_thresh above 1 for
> all NICs but 82598
>
>
>
> On 08/13/15 23:28, Zhang, Helin wrote:
> > Hi Vlad
> >
> > I don't think the changes are needed. It says in datasheet that the RS
> > bit should be set on the last descriptor of every packet, ONLY WHEN
> TXDCTL.WTHRESH equals to ZERO.
>
> Of course it's needed! See below.
> Exactly the same spec a few lines above the place u've just quoted states:
>
> "Software should not set the RS bit when TXDCTL.WTHRESH is greater than
> zero."
>
> And since all three (3) ixgbe xmit callbacks are utilizing RS bit notation ixgbe PMD
> is actually not supporting any value of WTHRESH different from zero.
I think Vlad is right. We need to fix this issue. Any suggestion? If not, I'd like to ack this patch.
>
> >
> > Regards,
> > Helin
> >
> >> -----Original Message-----
> >> From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com]
> >> Sent: Thursday, August 13, 2015 11:07 AM
> >> To: dev@dpdk.org
> >> Cc: Zhang, Helin; Ananyev, Konstantin; avi@cloudius-systems.com; Vlad
> >> Zolotarov
> >> Subject: [dpdk-dev] [PATCH v1] ixgbe_pmd: forbid tx_rs_thresh above 1 for
> all
> >> NICs but 82598
> >>
> >> According to 82599 and x540 HW specifications RS bit *must* be set in the
> last
> >> descriptor of *every* packet.
> > There is a condition that if TXDCTL.WTHRESH equal to zero.
>
> Right and ixgbe PMD requires this condition to be fulfilled in order to
> function. See above.
>
> >
> >> This patch fixes the Tx hang we were constantly hitting with a seastar-based
> >> application on x540 NIC.
> > Could you help to share with us how to reproduce the tx hang issue, with using
> > typical DPDK examples?
>
> Sorry. I'm not very familiar with the typical DPDK examples to help u
> here. However this is quite irrelevant since without this this patch
> ixgbe PMD obviously abuses the HW spec as has been explained above.
>
> We saw the issue when u stressed the xmit path with a lot of highly
> fragmented TCP frames (packets with up to 33 fragments with non-headers
> fragments as small as 4 bytes) with all offload features enabled.
>
> Thanks,
> vlad
> >
> >> Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
> >> ---
> >> drivers/net/ixgbe/ixgbe_ethdev.c | 9 +++++++++
> >> drivers/net/ixgbe/ixgbe_rxtx.c | 23 ++++++++++++++++++++++-
> >> 2 files changed, 31 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> >> b/drivers/net/ixgbe/ixgbe_ethdev.c
> >> index b8ee1e9..6714fd9 100644
> >> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> >> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> >> @@ -2414,6 +2414,15 @@ ixgbe_dev_info_get(struct rte_eth_dev *dev,
> struct
> >> rte_eth_dev_info *dev_info)
> >> .txq_flags = ETH_TXQ_FLAGS_NOMULTSEGS |
> >> ETH_TXQ_FLAGS_NOOFFLOADS,
> >> };
> >> +
> >> + /*
> >> + * According to 82599 and x540 specifications RS bit *must* be set on
> the
> >> + * last descriptor of *every* packet. Therefore we will not allow the
> >> + * tx_rs_thresh above 1 for all NICs newer than 82598.
> >> + */
> >> + if (hw->mac.type > ixgbe_mac_82598EB)
> >> + dev_info->default_txconf.tx_rs_thresh = 1;
> >> +
> >> dev_info->hash_key_size = IXGBE_HKEY_MAX_INDEX * sizeof(uint32_t);
> >> dev_info->reta_size = ETH_RSS_RETA_SIZE_128;
> >> dev_info->flow_type_rss_offloads = IXGBE_RSS_OFFLOAD_ALL; diff --
> git
> >> a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c index
> >> 91023b9..8dbdffc 100644
> >> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
> >> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
> >> @@ -2085,11 +2085,19 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev
> >> *dev,
> >> struct ixgbe_tx_queue *txq;
> >> struct ixgbe_hw *hw;
> >> uint16_t tx_rs_thresh, tx_free_thresh;
> >> + bool rs_deferring_allowed;
> >>
> >> PMD_INIT_FUNC_TRACE();
> >> hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
> >>
> >> /*
> >> + * According to 82599 and x540 specifications RS bit *must* be set on
> the
> >> + * last descriptor of *every* packet. Therefore we will not allow the
> >> + * tx_rs_thresh above 1 for all NICs newer than 82598.
> >> + */
> >> + rs_deferring_allowed = (hw->mac.type <= ixgbe_mac_82598EB);
> >> +
> >> + /*
> >> * Validate number of transmit descriptors.
> >> * It must not exceed hardware maximum, and must be multiple
> >> * of IXGBE_ALIGN.
> >> @@ -2110,6 +2118,8 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev
> *dev,
> >> * to transmit a packet is greater than the number of free TX
> >> * descriptors.
> >> * The following constraints must be satisfied:
> >> + * tx_rs_thresh must be less than 2 for NICs for which RS deferring is
> >> + * forbidden (all but 82598).
> >> * tx_rs_thresh must be greater than 0.
> >> * tx_rs_thresh must be less than the size of the ring minus 2.
> >> * tx_rs_thresh must be less than or equal to tx_free_thresh.
> >> @@ -2121,9 +2131,20 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev
> *dev,
> >> * When set to zero use default values.
> >> */
> >> tx_rs_thresh = (uint16_t)((tx_conf->tx_rs_thresh) ?
> >> - tx_conf->tx_rs_thresh : DEFAULT_TX_RS_THRESH);
> >> + tx_conf->tx_rs_thresh :
> >> + (rs_deferring_allowed ? DEFAULT_TX_RS_THRESH : 1));
> >> tx_free_thresh = (uint16_t)((tx_conf->tx_free_thresh) ?
> >> tx_conf->tx_free_thresh : DEFAULT_TX_FREE_THRESH);
> >> +
> >> + if (!rs_deferring_allowed && tx_rs_thresh > 1) {
> >> + PMD_INIT_LOG(ERR, "tx_rs_thresh must be less than 2 since RS
> "
> >> + "must be set for every packet for this HW. "
> >> + "(tx_rs_thresh=%u port=%d queue=%d)",
> >> + (unsigned int)tx_rs_thresh,
> >> + (int)dev->data->port_id, (int)queue_idx);
> >> + return -(EINVAL);
> >> + }
> >> +
> >> if (tx_rs_thresh >= (nb_desc - 2)) {
> >> PMD_INIT_LOG(ERR, "tx_rs_thresh must be less than the
> number "
> >> "of TX descriptors minus 2. (tx_rs_thresh=%u "
> >> --
> >> 2.1.0
next prev parent reply other threads:[~2015-08-19 0:42 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-08-13 18:06 Vlad Zolotarov
2015-08-13 20:28 ` Zhang, Helin
2015-08-14 5:37 ` Vlad Zolotarov
2015-08-19 0:42 ` Lu, Wenzhuo [this message]
2015-08-19 4:55 ` Vladislav Zolotarov
2015-08-19 7:43 ` Ananyev, Konstantin
2015-08-19 10:02 ` Vlad Zolotarov
2015-08-20 8:41 ` Ananyev, Konstantin
2015-08-20 8:56 ` Vlad Zolotarov
2015-08-20 9:05 ` Vlad Zolotarov
2015-08-20 9:06 ` Vlad Zolotarov
2015-08-25 17:33 ` Ananyev, Konstantin
2015-08-25 17:39 ` Avi Kivity
2015-08-19 17:29 ` Zhang, Helin
2015-08-25 18:13 ` Zhang, Helin
2015-08-25 18:33 ` Vladislav Zolotarov
2015-08-25 18:43 ` Zhang, Helin
2015-08-25 18:52 ` Vlad Zolotarov
2015-08-25 19:16 ` Zhang, Helin
2015-08-25 19:23 ` Avi Kivity
2015-08-25 19:30 ` Vladislav Zolotarov
2015-08-25 20:07 ` Vlad Zolotarov
2015-08-25 20:13 ` Zhang, Helin
2015-09-09 12:18 ` Thomas Monjalon
2015-09-09 13:19 ` Ananyev, Konstantin
2015-09-11 15:17 ` Vladislav Zolotarov
2015-09-11 14:25 ` didier.pallard
2015-09-11 14:47 ` Avi Kivity
2015-09-11 14:55 ` Thomas Monjalon
2015-09-11 15:12 ` Vladislav Zolotarov
2015-09-11 15:43 ` Avi Kivity
2015-09-11 16:04 ` Vladislav Zolotarov
2015-09-11 16:07 ` Richardson, Bruce
2015-09-11 16:14 ` Vladislav Zolotarov
2015-09-11 17:44 ` Avi Kivity
2015-09-11 16:08 ` Thomas Monjalon
2015-09-11 16:18 ` Vladislav Zolotarov
2015-09-11 17:17 ` Matthew Hall
2015-09-11 17:42 ` Ananyev, Konstantin
2015-09-11 17:58 ` Matthew Hall
2015-09-11 17:48 ` Avi Kivity
2015-09-13 11:47 ` Ananyev, Konstantin
2015-09-13 12:24 ` Vlad Zolotarov
2015-09-13 12:32 ` Avi Kivity
2015-09-13 15:54 ` Ananyev, Konstantin
2015-09-13 16:01 ` Avi Kivity
2015-09-11 16:00 ` Richardson, Bruce
2015-09-11 16:13 ` Vladislav Zolotarov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6A0DE07E22DDAD4C9103DF62FEBC0909D3E116@shsmsx102.ccr.corp.intel.com \
--to=wenzhuo.lu@intel.com \
--cc=dev@dpdk.org \
--cc=helin.zhang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).