From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 37D1CA04A8 for ; Mon, 24 Jan 2022 16:16:49 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CCBE841152; Mon, 24 Jan 2022 16:16:48 +0100 (CET) Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by mails.dpdk.org (Postfix) with ESMTP id 8A76F40040 for ; Mon, 24 Jan 2022 16:16:47 +0100 (CET) Received: by mail-pj1-f44.google.com with SMTP id pf13so16815007pjb.0 for ; Mon, 24 Jan 2022 07:16:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=jGR/5WOGeHAoUBDVJWYyJ845FTRJBqdCrKFqrlI/VKE=; b=OiRhomaIuLdPwcIYeSxR9zqvEfXqrjAJLQaxBFQbBII1OizJje5v1qXAXZtoEUDgpm LPYrOyslTlySsKkXoRAtVElcaXE2rr8HUJ4/YmlPNBVncrnz9VaAHlZnHKOAGe8XaNjB wK3Z4oO9h5feYon6MDchc9gIgkI6TcTn6vjHX87Z/LTS2/mjwm59+wt8iEN4oYS0yFbI 5+kcUaQdWFXKHdSOJHU8EXIZu3anuTL9xPpSiF6fuo1IRotYIeNFkqOpKqobM1u0OgVX aHcJP3Ht7sfVzob7e7UXVQ0mJoscB7TQVe5okU9r7YG9Prdjgfyx06I1859ir1cIp2ZR nIWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=jGR/5WOGeHAoUBDVJWYyJ845FTRJBqdCrKFqrlI/VKE=; b=xWGJch+nQ2hU5ugQpPN4sEtw1QGR51J7CKqFAiJM7HCQqYFH8yYrvXWIybI6vtYPU7 8b02QGBjP1GfIMpS144eBH5Er8eUTaKyKoHsAR49Lytag92BGKVmwJWRL1kUDH1UeBBl MHySEWcMwdSQjA7Q1Erpd8wEIWP+TaQ83G/8goHd41R8nT4kZclVYscBW/yveBs4nCe3 C3/flhEHD2ylbicpSTWkFwhGRU7OCqkeZNi0cpJuqGWfmYG6l9Zz6j33b9PUxuQWztKd RAFB2uoFD1rbH5mJAGi/MURnnwC4cRnbg2pzQxeaADN3jPDSVMjl+JLZLHS5B+XzvB1Q LPjg== X-Gm-Message-State: AOAM530FU+0/gBhqHuDtBdGz90jTqeSJ+VB+Yx84S+d5UpklSco7ZC6A q5SUSmrdxUNEHE/t1c8GMehQuqcvoHaFjhyacEw= X-Google-Smtp-Source: ABdhPJw0/TBgJteynwdONh6K8IE2BsaaCES2OvdyYNPBEIRnJm6sPOKOS3ozcqrSutZQFSZUo3EcP+aKQyrvSeAfTUQ= X-Received: by 2002:a17:90b:3b88:: with SMTP id pc8mr2342815pjb.245.1643037406666; Mon, 24 Jan 2022 07:16:46 -0800 (PST) MIME-Version: 1.0 References: <90321313-B075-42B4-A16A-87A2A79490B9@iki.fi> In-Reply-To: <90321313-B075-42B4-A16A-87A2A79490B9@iki.fi> From: Eric Christian Date: Mon, 24 Jan 2022 10:16:35 -0500 Message-ID: Subject: Re: RSS queue problem with i40e on DPDK 20.11.3 To: Antti-Pekka Liedes Cc: users@dpdk.org Content-Type: multipart/alternative; boundary="000000000000fa24b205d655748a" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000fa24b205d655748a Content-Type: text/plain; charset="UTF-8" Hi, I am curious if you resolved this? Eric On Fri, Nov 19, 2021 at 3:49 AM Antti-Pekka Liedes wrote: > Hi DPDK experts, > > I have a problem with upgrading our software from DPDK 20.11.1 to DPDK > 20.11.3: the RSS we use for i40e now delivers all the packets to the first > queue 0 only. I'm using the rte_flow API to configure the queues first and > then all the flow types one by one to distribute the incoming packets using > symmetric Toeplitz hash to 8 queues. > > Note that this is C++ code, and the m_ prefixed variables are members of > the Port object, ie., port specific parameters. > > The queue region setup is: > > const struct rte_flow_attr attr = { > .group = 0, > .priority = 0, > .ingress = 1, > .egress = 0, > .transfer = 0, > .reserved = 0 > }; > uint16_t rss_queue[m_num_rx_queues]; > for (int i = 0; i < m_num_rx_queues; i++) > { > rss_queue[i] = i; > } > > { > const struct rte_flow_item pattern[] = { > { > .type = RTE_FLOW_ITEM_TYPE_END > } > }; > > const struct rte_flow_action_rss action_rss = { > .level = 0, > .types = 0, > .key_len = rss_key_len, > .queue_num = m_num_rx_queues, > .key = rss_key, > .queue = rss_queue > }; > const struct rte_flow_action action[] = { > { > .type = RTE_FLOW_ACTION_TYPE_RSS, > .conf = &action_rss > }, > { > .type = RTE_FLOW_ACTION_TYPE_END > } > }; > struct rte_flow_error flow_error; > > struct rte_flow* flow = rte_flow_create( > m_portid, > &attr, > pattern, > action, > &flow_error); > > where m_num_rx_queues = 8, and rss_key and rss_key_len are our enforced > RSS key originally pulled from an X710. The rss_key_len = 52. > > After this I configure all the flow types: > > uint64_t rss_types[] = { > ETH_RSS_FRAG_IPV4, > ETH_RSS_NONFRAG_IPV4_TCP, > ETH_RSS_NONFRAG_IPV4_UDP, > ETH_RSS_NONFRAG_IPV4_SCTP, > ETH_RSS_NONFRAG_IPV4_OTHER, > > ETH_RSS_FRAG_IPV6, > ETH_RSS_NONFRAG_IPV6_TCP, > ETH_RSS_NONFRAG_IPV6_UDP, > ETH_RSS_NONFRAG_IPV6_SCTP, > ETH_RSS_NONFRAG_IPV6_OTHER > }; > > and for each type: > > const struct rte_flow_attr attr = { > .group = 0, > .priority = 0, > .ingress = 1, > .egress = 0, > .transfer = 0, > .reserved = 0 > }; > > // Room for L2 to L4. > struct rte_flow_item pattern[] = { > { > .type = RTE_FLOW_ITEM_TYPE_ETH > }, > { > .type = RTE_FLOW_ITEM_TYPE_END > }, > { > .type = RTE_FLOW_ITEM_TYPE_END > }, > { > .type = RTE_FLOW_ITEM_TYPE_END > } > }; > > // Add L2/L3/L4 to pattern according to rss_type. > > const struct rte_flow_action_rss action_rss = { > .func = RTE_ETH_HASH_FUNCTION_SYMMETRIC_TOEPLITZ, > .level = 0, > .types = rss_type, > .key_len = rss_key_len, > .queue_num = 0, > .key = rss_key, > .queue = NULL > }; > const struct rte_flow_action action[] = { > { > .type = RTE_FLOW_ACTION_TYPE_RSS, > .conf = &action_rss > }, > { > .type = RTE_FLOW_ACTION_TYPE_END > } > }; > struct rte_flow_error flow_error; > > struct rte_flow* flow = rte_flow_create( > m_portid, > &attr, > pattern, > action, > &flow_error); > > We also have a software Toeplitz calculator that agrees with the HW hash > value for both DPDK 20.11.1 and 20.11.3, so the hash calculation by the HW > seems to be ok. AFAICT, the above is similar to the RSS setup instructions > in https://doc.dpdk.org/guides-20.11/nics/i40e.html for test-pmd, except > that we give our own key as well. > > Some random facts: > - Changing rss_queues to all 3's doesn't affect the distribution; all the > packets still go to queue 0. > - I use Intel X710 for debugging and the observed behavior is from there, > but according to performance testing X722 also exhibits the same problem. > - My X710 fw versions are: i40e 0000:01:00.0: fw 8.4.66032 api 1.14 nvm > 8.40 0x8000aba4 1.2992.0. > - A quick test on DPDK 20.11.2 indicates correct spread among all 8 RX > queues, so the problem is probably introduced in 20.11.3. > > Thanks, > > Antti-Pekka Liedes > > --000000000000fa24b205d655748a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I am curious if you resolved this?<= /div>

Eric

On Fri, Nov 19, 2021 at 3:49 AM Antti-Pe= kka Liedes <apl@iki.fi> wrote:
<= /div>
Hi DPDK experts,

I have a problem with upgrading our software from DPDK 20.11.1 to DPDK 20.1= 1.3: the RSS we use for i40e now delivers all the packets to the first queu= e 0 only. I'm using the rte_flow API to configure the queues first and = then all the flow types one by one to distribute the incoming packets using= symmetric Toeplitz hash to 8 queues.

Note that this is C++ code, and the m_ prefixed variables are members of th= e Port object, ie., port specific parameters.

The queue region setup is:

const struct rte_flow_attr attr =3D {
=C2=A0.group =3D 0,
=C2=A0.priority =3D 0,
=C2=A0.ingress =3D 1,
=C2=A0.egress =3D 0,
=C2=A0.transfer =3D 0,
=C2=A0.reserved =3D 0
};
uint16_t rss_queue[m_num_rx_queues];
for (int i =3D 0; i < m_num_rx_queues; i++)
=C2=A0{
=C2=A0 =C2=A0rss_queue[i] =3D i;
=C2=A0}

{
=C2=A0const struct rte_flow_item pattern[] =3D {
=C2=A0 =C2=A0{
=C2=A0 =C2=A0 =C2=A0.type =3D RTE_FLOW_ITEM_TYPE_END
=C2=A0 =C2=A0}
=C2=A0};

=C2=A0const struct rte_flow_action_rss action_rss =3D {
=C2=A0 =C2=A0.level =3D 0,
=C2=A0 =C2=A0.types =3D 0,
=C2=A0 =C2=A0.key_len =3D rss_key_len,
=C2=A0 =C2=A0.queue_num =3D m_num_rx_queues,
=C2=A0 =C2=A0.key =3D rss_key,
=C2=A0 =C2=A0.queue =3D rss_queue
=C2=A0};
=C2=A0const struct rte_flow_action action[] =3D {
=C2=A0 =C2=A0{
=C2=A0 =C2=A0 =C2=A0.type =3D RTE_FLOW_ACTION_TYPE_RSS,
=C2=A0 =C2=A0 =C2=A0.conf =3D &action_rss
=C2=A0 =C2=A0},
=C2=A0 =C2=A0{
=C2=A0 =C2=A0 =C2=A0.type =3D RTE_FLOW_ACTION_TYPE_END
=C2=A0 =C2=A0}
=C2=A0};
=C2=A0struct rte_flow_error flow_error;

=C2=A0struct rte_flow* flow =3D rte_flow_create(
=C2=A0 =C2=A0m_portid,
=C2=A0 =C2=A0&attr,
=C2=A0 =C2=A0pattern,
=C2=A0 =C2=A0action,
=C2=A0 =C2=A0&flow_error);

where m_num_rx_queues =3D 8, and rss_key and rss_key_len are our enforced R= SS key originally pulled from an X710. The rss_key_len =3D 52.

After this I configure all the flow types:

uint64_t rss_types[] =3D {
=C2=A0ETH_RSS_FRAG_IPV4,
=C2=A0ETH_RSS_NONFRAG_IPV4_TCP,
=C2=A0ETH_RSS_NONFRAG_IPV4_UDP,
=C2=A0ETH_RSS_NONFRAG_IPV4_SCTP,
=C2=A0ETH_RSS_NONFRAG_IPV4_OTHER,

=C2=A0ETH_RSS_FRAG_IPV6,
=C2=A0ETH_RSS_NONFRAG_IPV6_TCP,
=C2=A0ETH_RSS_NONFRAG_IPV6_UDP,
=C2=A0ETH_RSS_NONFRAG_IPV6_SCTP,
=C2=A0ETH_RSS_NONFRAG_IPV6_OTHER
};

and for each type:

const struct rte_flow_attr attr =3D {
=C2=A0.group =3D 0,
=C2=A0.priority =3D 0,
=C2=A0.ingress =3D 1,
=C2=A0.egress =3D 0,
=C2=A0.transfer =3D 0,
=C2=A0.reserved =3D 0
};

// Room for L2 to L4.
struct rte_flow_item pattern[] =3D {
=C2=A0{
=C2=A0 =C2=A0.type =3D RTE_FLOW_ITEM_TYPE_ETH
=C2=A0},
=C2=A0{
=C2=A0 =C2=A0.type =3D RTE_FLOW_ITEM_TYPE_END
=C2=A0},
=C2=A0{
=C2=A0 =C2=A0.type =3D RTE_FLOW_ITEM_TYPE_END
=C2=A0},
=C2=A0{
=C2=A0 =C2=A0.type =3D RTE_FLOW_ITEM_TYPE_END
=C2=A0}
};

// Add L2/L3/L4 to pattern according to rss_type.

const struct rte_flow_action_rss action_rss =3D {
=C2=A0.func =3D RTE_ETH_HASH_FUNCTION_SYMMETRIC_TOEPLITZ,
=C2=A0.level =3D 0,
=C2=A0.types =3D rss_type,
=C2=A0.key_len =3D rss_key_len,
=C2=A0.queue_num =3D 0,
=C2=A0.key =3D rss_key,
=C2=A0.queue =3D NULL
};
const struct rte_flow_action action[] =3D {
=C2=A0{
=C2=A0 =C2=A0.type =3D RTE_FLOW_ACTION_TYPE_RSS,
=C2=A0 =C2=A0.conf =3D &action_rss
=C2=A0},
=C2=A0{
=C2=A0 =C2=A0.type =3D RTE_FLOW_ACTION_TYPE_END
=C2=A0}
};
struct rte_flow_error flow_error;

struct rte_flow* flow =3D rte_flow_create(
=C2=A0m_portid,
=C2=A0&attr,
=C2=A0pattern,
=C2=A0action,
=C2=A0&flow_error);

We also have a software Toeplitz calculator that agrees with the HW hash va= lue for both DPDK 20.11.1 and 20.11.3, so the hash calculation by the HW se= ems to be ok. AFAICT, the above is similar to the RSS setup instructions in= https://doc.dpdk.org/guides-20.11/nics/i40e.html for test-pmd, except that we give our own key as well.

Some random facts:
- Changing rss_queues to all 3's doesn't affect the distribution; a= ll the packets still go to queue 0.
- I use Intel X710 for debugging and the observed behavior is from there, b= ut according to performance testing X722 also exhibits the same problem. - My X710 fw versions are: i40e 0000:01:00.0: fw 8.4.66032 api 1.14 nvm 8.4= 0 0x8000aba4 1.2992.0.
- A quick test on DPDK 20.11.2 indicates correct spread among all 8 RX queu= es, so the problem is probably introduced in 20.11.3.

Thanks,

Antti-Pekka Liedes

--000000000000fa24b205d655748a--