DPDK usage discussions
 help / color / mirror / Atom feed
From: Eric Christian <erclists@gmail.com>
To: Antti-Pekka Liedes <apl@iki.fi>
Cc: users@dpdk.org
Subject: Re: RSS queue problem with i40e on DPDK 20.11.3
Date: Mon, 24 Jan 2022 10:16:35 -0500	[thread overview]
Message-ID: <CAFbSC54S0cNQJJ0f1xCQ9VOJRnbEarrmL=q905MX+fcMEWQc-A@mail.gmail.com> (raw)
In-Reply-To: <90321313-B075-42B4-A16A-87A2A79490B9@iki.fi>

[-- Attachment #1: Type: text/plain, Size: 3983 bytes --]

Hi,

I am curious if you resolved this?

Eric

On Fri, Nov 19, 2021 at 3:49 AM Antti-Pekka Liedes <apl@iki.fi> wrote:

> Hi DPDK experts,
>
> I have a problem with upgrading our software from DPDK 20.11.1 to DPDK
> 20.11.3: the RSS we use for i40e now delivers all the packets to the first
> queue 0 only. I'm using the rte_flow API to configure the queues first and
> then all the flow types one by one to distribute the incoming packets using
> symmetric Toeplitz hash to 8 queues.
>
> Note that this is C++ code, and the m_ prefixed variables are members of
> the Port object, ie., port specific parameters.
>
> The queue region setup is:
>
> const struct rte_flow_attr attr = {
>  .group = 0,
>  .priority = 0,
>  .ingress = 1,
>  .egress = 0,
>  .transfer = 0,
>  .reserved = 0
> };
> uint16_t rss_queue[m_num_rx_queues];
> for (int i = 0; i < m_num_rx_queues; i++)
>  {
>    rss_queue[i] = i;
>  }
>
> {
>  const struct rte_flow_item pattern[] = {
>    {
>      .type = RTE_FLOW_ITEM_TYPE_END
>    }
>  };
>
>  const struct rte_flow_action_rss action_rss = {
>    .level = 0,
>    .types = 0,
>    .key_len = rss_key_len,
>    .queue_num = m_num_rx_queues,
>    .key = rss_key,
>    .queue = rss_queue
>  };
>  const struct rte_flow_action action[] = {
>    {
>      .type = RTE_FLOW_ACTION_TYPE_RSS,
>      .conf = &action_rss
>    },
>    {
>      .type = RTE_FLOW_ACTION_TYPE_END
>    }
>  };
>  struct rte_flow_error flow_error;
>
>  struct rte_flow* flow = rte_flow_create(
>    m_portid,
>    &attr,
>    pattern,
>    action,
>    &flow_error);
>
> where m_num_rx_queues = 8, and rss_key and rss_key_len are our enforced
> RSS key originally pulled from an X710. The rss_key_len = 52.
>
> After this I configure all the flow types:
>
> uint64_t rss_types[] = {
>  ETH_RSS_FRAG_IPV4,
>  ETH_RSS_NONFRAG_IPV4_TCP,
>  ETH_RSS_NONFRAG_IPV4_UDP,
>  ETH_RSS_NONFRAG_IPV4_SCTP,
>  ETH_RSS_NONFRAG_IPV4_OTHER,
>
>  ETH_RSS_FRAG_IPV6,
>  ETH_RSS_NONFRAG_IPV6_TCP,
>  ETH_RSS_NONFRAG_IPV6_UDP,
>  ETH_RSS_NONFRAG_IPV6_SCTP,
>  ETH_RSS_NONFRAG_IPV6_OTHER
> };
>
> and for each type:
>
> const struct rte_flow_attr attr = {
>  .group = 0,
>  .priority = 0,
>  .ingress = 1,
>  .egress = 0,
>  .transfer = 0,
>  .reserved = 0
> };
>
> // Room for L2 to L4.
> struct rte_flow_item pattern[] = {
>  {
>    .type = RTE_FLOW_ITEM_TYPE_ETH
>  },
>  {
>    .type = RTE_FLOW_ITEM_TYPE_END
>  },
>  {
>    .type = RTE_FLOW_ITEM_TYPE_END
>  },
>  {
>    .type = RTE_FLOW_ITEM_TYPE_END
>  }
> };
>
> // Add L2/L3/L4 to pattern according to rss_type.
>
> const struct rte_flow_action_rss action_rss = {
>  .func = RTE_ETH_HASH_FUNCTION_SYMMETRIC_TOEPLITZ,
>  .level = 0,
>  .types = rss_type,
>  .key_len = rss_key_len,
>  .queue_num = 0,
>  .key = rss_key,
>  .queue = NULL
> };
> const struct rte_flow_action action[] = {
>  {
>    .type = RTE_FLOW_ACTION_TYPE_RSS,
>    .conf = &action_rss
>  },
>  {
>    .type = RTE_FLOW_ACTION_TYPE_END
>  }
> };
> struct rte_flow_error flow_error;
>
> struct rte_flow* flow = rte_flow_create(
>  m_portid,
>  &attr,
>  pattern,
>  action,
>  &flow_error);
>
> We also have a software Toeplitz calculator that agrees with the HW hash
> value for both DPDK 20.11.1 and 20.11.3, so the hash calculation by the HW
> seems to be ok. AFAICT, the above is similar to the RSS setup instructions
> in https://doc.dpdk.org/guides-20.11/nics/i40e.html for test-pmd, except
> that we give our own key as well.
>
> Some random facts:
> - Changing rss_queues to all 3's doesn't affect the distribution; all the
> packets still go to queue 0.
> - I use Intel X710 for debugging and the observed behavior is from there,
> but according to performance testing X722 also exhibits the same problem.
> - My X710 fw versions are: i40e 0000:01:00.0: fw 8.4.66032 api 1.14 nvm
> 8.40 0x8000aba4 1.2992.0.
> - A quick test on DPDK 20.11.2 indicates correct spread among all 8 RX
> queues, so the problem is probably introduced in 20.11.3.
>
> Thanks,
>
> Antti-Pekka Liedes
>
>

[-- Attachment #2: Type: text/html, Size: 4969 bytes --]

  reply	other threads:[~2022-01-24 15:16 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-19  8:45 Antti-Pekka Liedes
2022-01-24 15:16 ` Eric Christian [this message]
2022-01-30  9:00   ` Antti-Pekka Liedes

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAFbSC54S0cNQJJ0f1xCQ9VOJRnbEarrmL=q905MX+fcMEWQc-A@mail.gmail.com' \
    --to=erclists@gmail.com \
    --cc=apl@iki.fi \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).