From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2075BA00C5 for ; Sun, 30 Jan 2022 10:00:52 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A979740E09; Sun, 30 Jan 2022 10:00:51 +0100 (CET) Received: from apl.kvy.fi (apl.kvy.fi [83.143.59.162]) by mails.dpdk.org (Postfix) with ESMTP id 7B1FF40041 for ; Sun, 30 Jan 2022 10:00:50 +0100 (CET) X-Virus-Scanned: Yes Received: from smtpclient.apple (unknown [10.10.10.3]) by apl.kvy.fi (Postfix) with ESMTP id 0C1731E040F; Sun, 30 Jan 2022 11:00:47 +0200 (EET) Authentication-Results: apl.kvy.fi; dmarc=none (p=none dis=none) header.from=iki.fi Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: RSS queue problem with i40e on DPDK 20.11.3 From: Antti-Pekka Liedes In-Reply-To: Date: Sun, 30 Jan 2022 11:00:47 +0200 Cc: users@dpdk.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <90321313-B075-42B4-A16A-87A2A79490B9@iki.fi> To: Eric Christian X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, it seems that with another round of firmware updates the problem just = went away. We had some other moving parts as well, so I=E2=80=99m not = 100% sure as to what exactly fixed this, but all is well now. We also = upgraded to DPDK 20.11.4 and still going fine. =E2=80=94=20 Antti-Pekka Liedes > On 24. Jan 2022, at 17.16, Eric Christian wrote: >=20 > Hi, >=20 > I am curious if you resolved this? >=20 > Eric >=20 > On Fri, Nov 19, 2021 at 3:49 AM Antti-Pekka Liedes wrote: > Hi DPDK experts, >=20 > I have a problem with upgrading our software from DPDK 20.11.1 to DPDK = 20.11.3: the RSS we use for i40e now delivers all the packets to the = first queue 0 only. I'm using the rte_flow API to configure the queues = first and then all the flow types one by one to distribute the incoming = packets using symmetric Toeplitz hash to 8 queues. >=20 > Note that this is C++ code, and the m_ prefixed variables are members = of the Port object, ie., port specific parameters. >=20 > The queue region setup is: >=20 > const struct rte_flow_attr attr =3D { > .group =3D 0, > .priority =3D 0, > .ingress =3D 1, > .egress =3D 0, > .transfer =3D 0, > .reserved =3D 0 > }; > uint16_t rss_queue[m_num_rx_queues]; > for (int i =3D 0; i < m_num_rx_queues; i++) > { > rss_queue[i] =3D i; > } >=20 > { > const struct rte_flow_item pattern[] =3D { > { > .type =3D RTE_FLOW_ITEM_TYPE_END > } > }; >=20 > const struct rte_flow_action_rss action_rss =3D { > .level =3D 0, > .types =3D 0, > .key_len =3D rss_key_len, > .queue_num =3D m_num_rx_queues, > .key =3D rss_key, > .queue =3D rss_queue > }; > const struct rte_flow_action action[] =3D { > { > .type =3D RTE_FLOW_ACTION_TYPE_RSS, > .conf =3D &action_rss > }, > { > .type =3D RTE_FLOW_ACTION_TYPE_END > } > }; > struct rte_flow_error flow_error; >=20 > struct rte_flow* flow =3D rte_flow_create( > m_portid, > &attr, > pattern, > action, > &flow_error); >=20 > where m_num_rx_queues =3D 8, and rss_key and rss_key_len are our = enforced RSS key originally pulled from an X710. The rss_key_len =3D 52. >=20 > After this I configure all the flow types: >=20 > uint64_t rss_types[] =3D { > ETH_RSS_FRAG_IPV4, > ETH_RSS_NONFRAG_IPV4_TCP, > ETH_RSS_NONFRAG_IPV4_UDP, > ETH_RSS_NONFRAG_IPV4_SCTP, > ETH_RSS_NONFRAG_IPV4_OTHER, >=20 > ETH_RSS_FRAG_IPV6, > ETH_RSS_NONFRAG_IPV6_TCP, > ETH_RSS_NONFRAG_IPV6_UDP, > ETH_RSS_NONFRAG_IPV6_SCTP, > ETH_RSS_NONFRAG_IPV6_OTHER > }; >=20 > and for each type: >=20 > const struct rte_flow_attr attr =3D { > .group =3D 0, > .priority =3D 0, > .ingress =3D 1, > .egress =3D 0, > .transfer =3D 0, > .reserved =3D 0 > }; >=20 > // Room for L2 to L4. > struct rte_flow_item pattern[] =3D { > { > .type =3D RTE_FLOW_ITEM_TYPE_ETH > }, > { > .type =3D RTE_FLOW_ITEM_TYPE_END > }, > { > .type =3D RTE_FLOW_ITEM_TYPE_END > }, > { > .type =3D RTE_FLOW_ITEM_TYPE_END > } > }; >=20 > // Add L2/L3/L4 to pattern according to rss_type. >=20 > const struct rte_flow_action_rss action_rss =3D { > .func =3D RTE_ETH_HASH_FUNCTION_SYMMETRIC_TOEPLITZ, > .level =3D 0, > .types =3D rss_type, > .key_len =3D rss_key_len, > .queue_num =3D 0, > .key =3D rss_key, > .queue =3D NULL > }; > const struct rte_flow_action action[] =3D { > { > .type =3D RTE_FLOW_ACTION_TYPE_RSS, > .conf =3D &action_rss > }, > { > .type =3D RTE_FLOW_ACTION_TYPE_END > } > }; > struct rte_flow_error flow_error; >=20 > struct rte_flow* flow =3D rte_flow_create( > m_portid, > &attr, > pattern, > action, > &flow_error); >=20 > We also have a software Toeplitz calculator that agrees with the HW = hash value for both DPDK 20.11.1 and 20.11.3, so the hash calculation by = the HW seems to be ok. AFAICT, the above is similar to the RSS setup = instructions in https://doc.dpdk.org/guides-20.11/nics/i40e.html for = test-pmd, except that we give our own key as well. >=20 > Some random facts: > - Changing rss_queues to all 3's doesn't affect the distribution; all = the packets still go to queue 0. > - I use Intel X710 for debugging and the observed behavior is from = there, but according to performance testing X722 also exhibits the same = problem. > - My X710 fw versions are: i40e 0000:01:00.0: fw 8.4.66032 api 1.14 = nvm 8.40 0x8000aba4 1.2992.0. > - A quick test on DPDK 20.11.2 indicates correct spread among all 8 RX = queues, so the problem is probably introduced in 20.11.3. >=20 > Thanks, >=20 > Antti-Pekka Liedes >=20