From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 65471A0C4B for ; Fri, 19 Nov 2021 09:49:27 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 30E1440143; Fri, 19 Nov 2021 09:49:27 +0100 (CET) Received: from apl.kvy.fi (apl.kvy.fi [83.143.59.162]) by mails.dpdk.org (Postfix) with ESMTP id B7E1540140 for ; Fri, 19 Nov 2021 09:49:25 +0100 (CET) X-Virus-Scanned: Yes Received: from smtpclient.apple (unknown [172.21.0.1]) by apl.kvy.fi (Postfix) with ESMTP id 6180E1E055F for ; Fri, 19 Nov 2021 10:49:23 +0200 (EET) Authentication-Results: 'apl.kvy.fi'; dmarc=none (p=none dis=none) header.from=iki.fi From: Antti-Pekka Liedes Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: RSS queue problem with i40e on DPDK 20.11.3 Message-Id: <90321313-B075-42B4-A16A-87A2A79490B9@iki.fi> Date: Fri, 19 Nov 2021 10:45:03 +0200 To: users@dpdk.org X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi DPDK experts, I have a problem with upgrading our software from DPDK 20.11.1 to DPDK = 20.11.3: the RSS we use for i40e now delivers all the packets to the = first queue 0 only. I'm using the rte_flow API to configure the queues = first and then all the flow types one by one to distribute the incoming = packets using symmetric Toeplitz hash to 8 queues. Note that this is C++ code, and the m_ prefixed variables are members of = the Port object, ie., port specific parameters. The queue region setup is: const struct rte_flow_attr attr =3D { .group =3D 0, .priority =3D 0, .ingress =3D 1, .egress =3D 0, .transfer =3D 0, .reserved =3D 0 }; uint16_t rss_queue[m_num_rx_queues]; for (int i =3D 0; i < m_num_rx_queues; i++) { rss_queue[i] =3D i; } { const struct rte_flow_item pattern[] =3D { { .type =3D RTE_FLOW_ITEM_TYPE_END } }; const struct rte_flow_action_rss action_rss =3D { .level =3D 0, .types =3D 0, .key_len =3D rss_key_len, .queue_num =3D m_num_rx_queues, .key =3D rss_key, .queue =3D rss_queue }; const struct rte_flow_action action[] =3D { { .type =3D RTE_FLOW_ACTION_TYPE_RSS, .conf =3D &action_rss }, { .type =3D RTE_FLOW_ACTION_TYPE_END } }; struct rte_flow_error flow_error; struct rte_flow* flow =3D rte_flow_create( m_portid, &attr, pattern, action, &flow_error); where m_num_rx_queues =3D 8, and rss_key and rss_key_len are our = enforced RSS key originally pulled from an X710. The rss_key_len =3D 52. After this I configure all the flow types: uint64_t rss_types[] =3D { ETH_RSS_FRAG_IPV4, ETH_RSS_NONFRAG_IPV4_TCP, ETH_RSS_NONFRAG_IPV4_UDP, ETH_RSS_NONFRAG_IPV4_SCTP, ETH_RSS_NONFRAG_IPV4_OTHER, ETH_RSS_FRAG_IPV6, ETH_RSS_NONFRAG_IPV6_TCP, ETH_RSS_NONFRAG_IPV6_UDP, ETH_RSS_NONFRAG_IPV6_SCTP, ETH_RSS_NONFRAG_IPV6_OTHER }; and for each type: const struct rte_flow_attr attr =3D { .group =3D 0, .priority =3D 0, .ingress =3D 1, .egress =3D 0, .transfer =3D 0, .reserved =3D 0 }; // Room for L2 to L4. struct rte_flow_item pattern[] =3D { { .type =3D RTE_FLOW_ITEM_TYPE_ETH }, { .type =3D RTE_FLOW_ITEM_TYPE_END }, { .type =3D RTE_FLOW_ITEM_TYPE_END }, { .type =3D RTE_FLOW_ITEM_TYPE_END } }; // Add L2/L3/L4 to pattern according to rss_type. const struct rte_flow_action_rss action_rss =3D { .func =3D RTE_ETH_HASH_FUNCTION_SYMMETRIC_TOEPLITZ, .level =3D 0, .types =3D rss_type, .key_len =3D rss_key_len, .queue_num =3D 0, .key =3D rss_key, .queue =3D NULL }; const struct rte_flow_action action[] =3D { { .type =3D RTE_FLOW_ACTION_TYPE_RSS, .conf =3D &action_rss }, { .type =3D RTE_FLOW_ACTION_TYPE_END } }; struct rte_flow_error flow_error; struct rte_flow* flow =3D rte_flow_create( m_portid, &attr, pattern, action, &flow_error); We also have a software Toeplitz calculator that agrees with the HW hash = value for both DPDK 20.11.1 and 20.11.3, so the hash calculation by the = HW seems to be ok. AFAICT, the above is similar to the RSS setup = instructions in https://doc.dpdk.org/guides-20.11/nics/i40e.html for = test-pmd, except that we give our own key as well. Some random facts: - Changing rss_queues to all 3's doesn't affect the distribution; all = the packets still go to queue 0. - I use Intel X710 for debugging and the observed behavior is from = there, but according to performance testing X722 also exhibits the same = problem. - My X710 fw versions are: i40e 0000:01:00.0: fw 8.4.66032 api 1.14 nvm = 8.40 0x8000aba4 1.2992.0. - A quick test on DPDK 20.11.2 indicates correct spread among all 8 RX = queues, so the problem is probably introduced in 20.11.3. Thanks, Antti-Pekka Liedes