From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E5F8CA034F for ; Wed, 10 Nov 2021 00:34:34 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5188A410F2; Wed, 10 Nov 2021 00:34:31 +0100 (CET) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mails.dpdk.org (Postfix) with ESMTP id 26C1B40142; Wed, 10 Nov 2021 00:34:29 +0100 (CET) Received: by mail-wm1-f48.google.com with SMTP id 71so569415wma.4; Tue, 09 Nov 2021 15:34:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=7xVwLomigo2RIxBXBympgWwLiDGv68KqGCByZguZ2eM=; b=ckj5WqoRy5OHOvOjW0Hesqzdx4hVg6RpKn11+bojd/q8Lf0w6aa16bybTLnS1KMieh 7WLYOjW5O719Jbur496hXSX22c9Q95sXjvwkG1M2y5GV3t8CJ9m5UYJXkl+cajiCcVxy X7+d5TF3/U3T3XTeCBOfarKHg3a+uA4cKgzMgkDSACpI+wZTL1SXjs17ryHG/YM+SQ/F YGRne7mzZ/DWEIcgPaeJ8MWjiXVwfeIVDYOGXJkouXMu3ov+rwi772AzlvVeseFDemRV MAWTLLNMvTJjZVBfvpz3k/mD2RB+BQoQc9kV7lhl3/DMxX4fYXq1uKgkz8R90FK19Z3b T8MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=7xVwLomigo2RIxBXBympgWwLiDGv68KqGCByZguZ2eM=; b=jgRwL7370mKzQqwKcfec0AswYI0nS0vNuGAv9TTwxxhkuGB6uvjfmAOln+4g4uJWfM 9VzKSwbmPREZYNu6fYLBF86PfkTvS3uQhSgSwzAaXX2meZ6KJPv/Qs0tFaz5cPh+Xa0Y XY9am+RO0bKIGnnc3Mk/kjAJihCIOXgBP5KLjYDGZ+43QVRGhoUlKXnla9LhI1WWfc1n xp7ggzMFJDH3kQz0+wkj+VND0994bJm6yjDWjyDwBR+96xIawesXsRjlK2ZiwQRtmpvn YKXqzn9ZyhbWLYeT4ftZy2XnipwnQOcVs/4KFT3HcSirobR185nRCuqbNcnRkRyPTKL6 nZ3g== X-Gm-Message-State: AOAM5317Jf2XvQ1atFFVyORQEEYVf/RGt/+FK9x2eFOBq4QW2K9gmVu8 7nQ/NXoJFh6jeDNYTcOAWpUy9NuTAymPDJ69cRF1WaLSmVA= X-Google-Smtp-Source: ABdhPJzpLfYAcB/cABfbTkyW34ixfWcP2QvVx1hYo7qCi+8WwjvidIvZ3d0v1E/AprnjoLroJVz+8a7SEELrofcJZzQ= X-Received: by 2002:a05:600c:4982:: with SMTP id h2mr11683195wmp.4.1636500868510; Tue, 09 Nov 2021 15:34:28 -0800 (PST) MIME-Version: 1.0 From: fwefew 4t4tg <7532yahoo@gmail.com> Date: Tue, 9 Nov 2021 18:34:17 -0500 Message-ID: Subject: RSS Hash suport in DPDK on Amazon ENA NICs broken? To: users@dpdk.org, dev@dpdk.org Content-Type: multipart/alternative; boundary="000000000000f138ff05d0638c2b" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000f138ff05d0638c2b Content-Type: text/plain; charset="UTF-8" On AWS EC2 I configure: - c5n.metal instance - Stock AWS AMI Ubuntu 20.04 LTS ($ cat /proc/version Linux version 5.4.0-1045-aws) - Apply the vfio pci driver from https://github.com/amzn/am zn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch - Using DPDK code from head of main branch. I built using ninja/meson - Add a 2nd ENA enabled NIC - Setup 2Mb huge pages, and bound the 2nd NIC with dpdk-devbind I run test-pmd with this command: $ sudo ./dpdk-testpmd --huge-dir=/mnt/huge -c 0xf -n 4 -- -i EAL: Detected CPU lcores: 72 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 1048576 kB hugepages reported EAL: VFIO support initialized EAL: Using IOMMU type 1 (Type 1) EAL: Ignore mapping IO port bar(3) EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0) TELEMETRY: No legacy callbacks, legacy socket not created Interactive-mode selected Error picking flow transfer proxy for port 0: Function not implemented - ignore testpmd: create a new mbuf pool : n=171456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. Configuring Port 0 (socket 0) Port 0: 16:9A:11:4A:5D:61 Checking link statuses... Done Error during enabling promiscuous mode for port 0: Operation not supported - ignore *Key point: I try to enable RSS hash offload for UDP or TCP. I get the following error shown in bold:* testpmd> port config all rss udp *Port 0 modified RSS hash function based on hardware support,requested:0x20820 configured:0x820*ena_rss_hash_set(): Setting RSS hash fields is not supported ena_rss_hash_update(): Failed to set RSS hash *Configuration of RSS hash at ethernet port 0 failed with error (95): Operation not supported.* testpmd> port config all rss tcp *Port 0 modified RSS hash function based on hardware support,requested:0x10410 configured:0x410*ena_rss_hash_set(): Setting RSS hash fields is not supported ena_rss_hash_update(): Failed to set RSS hash *Configuration of RSS hash at ethernet port 0 failed with error (95): Operation not supported.* If I run ethtool on the NIC before putting it down state and DPDK binding to vfio-pci I see an indication that there is a hash key, and RSS Hash works: ubuntu$ ethtool -x ens6 RX flow hash indirection table for ens6 with 32 RX ring(s): 0: 0 1 2 3 4 5 6 7 8: 8 9 10 11 12 13 14 15 16: 16 17 18 19 20 21 22 23 24: 24 25 26 27 28 29 30 31 32: 0 1 2 3 4 5 6 7 40: 8 9 10 11 12 13 14 15 48: 16 17 18 19 20 21 22 23 56: 24 25 26 27 28 29 30 31 64: 0 1 2 3 4 5 6 7 72: 8 9 10 11 12 13 14 15 80: 16 17 18 19 20 21 22 23 88: 24 25 26 27 28 29 30 31 96: 0 1 2 3 4 5 6 7 104: 8 9 10 11 12 13 14 15 112: 16 17 18 19 20 21 22 23 120: 24 25 26 27 28 29 30 31 RSS hash key: 80:d3:0f:c4:1d:8a:48:9e:7f:1d:18:91:d3:12:cb:38:87:fc:33:49:18:30:e4:1d:19:8f:9d:fd:c3:94:fe:f3:b2:f0:7f:e9:e9:9d:2d:af RSS hash function: toeplitz: on xor: off crc32: off What gives? Do I need to modify a constant somewhere in DPDK? At the risk of expanding scope, I am really trying to run eRPC which runs on top of DPDK. It fails in dpdk_init with a similar but slightly different error shown in bold in which the putative valid value is different than above: $ sudo ./hello_server 172.31.77.76:31850 20:536566 INFOR: eRPC Nexus: Launching 0 background threads. 20:536582 INFOR: eRPC Nexus: Launching session management thread on core 53. 20:536776 INFOR: eRPC Nexus: Created with management UDP port 31850, hostname 172.31.77.76. 20:536832 INFOR: DPDK transport for Rpc 0 initializing DPDK EAL. EAL: Detected CPU lcores: 72 EAL: Detected NUMA nodes: 2 EAL: Auto-detected process type: PRIMARY EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 1048576 kB hugepages reported EAL: VFIO support initialized EAL: Using IOMMU type 1 (Type 1) EAL: Ignore mapping IO port bar(3) EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0) TELEMETRY: No legacy callbacks, legacy socket not created 21:428883 WARNG: Running as primary DPDK process. eRPC DPDK daemon is not running. 21:428893 INFOR: DPDK transport for Rpc 0 got QP 0 21:428927 INFOR: Initializing port 0 with driver net_ena *Ethdev port_id=0 invalid rss_hf: 0x20820, valid value: 0xc30*terminate called after throwing an instance of 'std::runtime_error' what(): Ethdev configuration error: Invalid argument Aborted --000000000000f138ff05d0638c2b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On AWS EC2 I configure:

- c5n.metal instance
- St= ock AWS AMI Ubuntu 20.04 LTS ($ cat /proc/version
Linux ve= rsion 5.4.0-1045-aws)
- Apply the vfio = pci driver from=C2=A0https://= github.com/amzn/am zn-drivers/tree/master/userspace/dpdk/enav2-vfio-pat= ch
- Using DPDK code from head of main branch. I built using ninja/meson=
- Add a 2nd ENA enabled NIC=C2=A0
- Setup 2Mb huge pages, and bound = the 2nd NIC with dpdk-devbind

I run test-pmd with this command:
<= br>$ sudo ./dpdk-testpmd --huge-dir=3D/mnt/huge -c 0xf -n 4 -- -i
EAL: D= etected CPU lcores: 72
EAL: Detected NUMA nodes: 2
EAL: Detected stat= ic linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket=
EAL: Selected IOVA mode 'VA'
EAL: No available 1048576 kB hu= gepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type = 1 (Type 1)
EAL: Ignore mapping IO port bar(3)
EAL: Probe PCI driver: = net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0)
TELEMETRY: No legacy= callbacks, legacy socket not created
Interactive-mode selected
Error= picking flow transfer proxy for port 0: Function not implemented - ignore<= br>testpmd: create a new mbuf pool <mb_pool_0>: n=3D171456, size=3D21= 76, socket=3D0
testpmd: preferred mempool ops selected: ring_mp_mc
Wa= rning! port-topology=3Dpaired and odd forward ports number, the last port w= ill pair with itself.
Configuring Port 0 (socket 0)
Port 0: 16:9A:11:= 4A:5D:61
Checking link statuses...
Done
Error during enabling prom= iscuous mode for port 0: Operation not supported - ignore

Key poi= nt: I try to enable RSS hash offload for UDP or TCP. I get the following er= ror shown in bold:

testpmd> port config all rss udp
Por= t 0 modified RSS hash function based on hardware support,requested:0x20820 = configured:0x820
ena_rss_hash_set(): Setting RSS hash fields is not = supported
ena_rss_hash_update(): Failed to set RSS hash
Configurat= ion of RSS hash at ethernet port 0 failed with error (95): Operation not su= pported.
testpmd> port config all rss tcp
Port 0 modified R= SS hash function based on hardware support,requested:0x10410 configured:0x4= 10
ena_rss_hash_set(): Setting RSS hash fields is not supported
e= na_rss_hash_update(): Failed to set RSS hash
Configuration of RSS has= h at ethernet port 0 failed with error (95): Operation not supported.

If I run ethtool on the NIC before putting it down state a= nd DPDK binding to vfio-pci I see an indication that there is a hash key, a= nd RSS Hash works:

ubuntu$ ethtool -x ens6
RX flow hash indirecti= on table for ens6 with 32 RX ring(s):
=C2=A0 =C2=A0 0: =C2=A0 =C2=A0 =C2= =A00 =C2=A0 =C2=A0 1 =C2=A0 =C2=A0 2 =C2=A0 =C2=A0 3 =C2=A0 =C2=A0 4 =C2=A0= =C2=A0 5 =C2=A0 =C2=A0 6 =C2=A0 =C2=A0 7
=C2=A0 =C2=A0 8: =C2=A0 =C2=A0= =C2=A08 =C2=A0 =C2=A0 9 =C2=A0 =C2=A010 =C2=A0 =C2=A011 =C2=A0 =C2=A012 = =C2=A0 =C2=A013 =C2=A0 =C2=A014 =C2=A0 =C2=A015
=C2=A0 =C2=A016: =C2=A0 = =C2=A0 16 =C2=A0 =C2=A017 =C2=A0 =C2=A018 =C2=A0 =C2=A019 =C2=A0 =C2=A020 = =C2=A0 =C2=A021 =C2=A0 =C2=A022 =C2=A0 =C2=A023
=C2=A0 =C2=A024: =C2=A0 = =C2=A0 24 =C2=A0 =C2=A025 =C2=A0 =C2=A026 =C2=A0 =C2=A027 =C2=A0 =C2=A028 = =C2=A0 =C2=A029 =C2=A0 =C2=A030 =C2=A0 =C2=A031
=C2=A0 =C2=A032: =C2=A0 = =C2=A0 =C2=A00 =C2=A0 =C2=A0 1 =C2=A0 =C2=A0 2 =C2=A0 =C2=A0 3 =C2=A0 =C2= =A0 4 =C2=A0 =C2=A0 5 =C2=A0 =C2=A0 6 =C2=A0 =C2=A0 7
=C2=A0 =C2=A040: = =C2=A0 =C2=A0 =C2=A08 =C2=A0 =C2=A0 9 =C2=A0 =C2=A010 =C2=A0 =C2=A011 =C2= =A0 =C2=A012 =C2=A0 =C2=A013 =C2=A0 =C2=A014 =C2=A0 =C2=A015
=C2=A0 =C2= =A048: =C2=A0 =C2=A0 16 =C2=A0 =C2=A017 =C2=A0 =C2=A018 =C2=A0 =C2=A019 =C2= =A0 =C2=A020 =C2=A0 =C2=A021 =C2=A0 =C2=A022 =C2=A0 =C2=A023
=C2=A0 =C2= =A056: =C2=A0 =C2=A0 24 =C2=A0 =C2=A025 =C2=A0 =C2=A026 =C2=A0 =C2=A027 =C2= =A0 =C2=A028 =C2=A0 =C2=A029 =C2=A0 =C2=A030 =C2=A0 =C2=A031
=C2=A0 =C2= =A064: =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 1 =C2=A0 =C2=A0 2 =C2=A0 =C2=A0 = 3 =C2=A0 =C2=A0 4 =C2=A0 =C2=A0 5 =C2=A0 =C2=A0 6 =C2=A0 =C2=A0 7
=C2=A0= =C2=A072: =C2=A0 =C2=A0 =C2=A08 =C2=A0 =C2=A0 9 =C2=A0 =C2=A010 =C2=A0 =C2= =A011 =C2=A0 =C2=A012 =C2=A0 =C2=A013 =C2=A0 =C2=A014 =C2=A0 =C2=A015
= =C2=A0 =C2=A080: =C2=A0 =C2=A0 16 =C2=A0 =C2=A017 =C2=A0 =C2=A018 =C2=A0 = =C2=A019 =C2=A0 =C2=A020 =C2=A0 =C2=A021 =C2=A0 =C2=A022 =C2=A0 =C2=A023=C2=A0 =C2=A088: =C2=A0 =C2=A0 24 =C2=A0 =C2=A025 =C2=A0 =C2=A026 =C2=A0 = =C2=A027 =C2=A0 =C2=A028 =C2=A0 =C2=A029 =C2=A0 =C2=A030 =C2=A0 =C2=A031=C2=A0 =C2=A096: =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 1 =C2=A0 =C2=A0 2 =C2= =A0 =C2=A0 3 =C2=A0 =C2=A0 4 =C2=A0 =C2=A0 5 =C2=A0 =C2=A0 6 =C2=A0 =C2=A0 = 7
=C2=A0 104: =C2=A0 =C2=A0 =C2=A08 =C2=A0 =C2=A0 9 =C2=A0 =C2=A010 =C2= =A0 =C2=A011 =C2=A0 =C2=A012 =C2=A0 =C2=A013 =C2=A0 =C2=A014 =C2=A0 =C2=A01= 5
=C2=A0 112: =C2=A0 =C2=A0 16 =C2=A0 =C2=A017 =C2=A0 =C2=A018 =C2=A0 = =C2=A019 =C2=A0 =C2=A020 =C2=A0 =C2=A021 =C2=A0 =C2=A022 =C2=A0 =C2=A023=C2=A0 120: =C2=A0 =C2=A0 24 =C2=A0 =C2=A025 =C2=A0 =C2=A026 =C2=A0 =C2=A0= 27 =C2=A0 =C2=A028 =C2=A0 =C2=A029 =C2=A0 =C2=A030 =C2=A0 =C2=A031
RSS h= ash key:
80:d3:0f:c4:1d:8a:48:9e:7f:1d:18:91:d3:12:cb:38:87:fc:33:49:18:= 30:e4:1d:19:8f:9d:fd:c3:94:fe:f3:b2:f0:7f:e9:e9:9d:2d:af
RSS hash functi= on:
=C2=A0 =C2=A0 toeplitz: on
=C2=A0 =C2=A0 xor: off
=C2=A0 =C2= =A0 crc32: off

What gives? Do I need to modify a constant somewhere = in DPDK?

At the risk of expanding scope, I am really trying to run e= RPC which runs on top of DPDK. It fails in dpdk_init with a similar but sli= ghtly different error shown in bold in which the putative valid value is di= fferent than above:

$ sudo ./hello_server 172.31.77.76:31850
20:536566 INFOR: eRPC Nexus: Launchin= g 0 background threads.
20:536582 INFOR: eRPC Nexus: Launching session m= anagement thread on core 53.
20:536776 INFOR: eRPC Nexus: Created with m= anagement UDP port 31850, hostname 172.31.77.76.
20:536832 INFOR: DPDK t= ransport for Rpc 0 initializing DPDK EAL.
EAL: Detected CPU lcores: 72EAL: Detected NUMA nodes: 2
EAL: Auto-detected process type: PRIMARYEAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/ru= n/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No ava= ilable 1048576 kB hugepages reported
EAL: VFIO support initialized
EA= L: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(3)
EAL= : Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0)
= TELEMETRY: No legacy callbacks, legacy socket not created
21:428883 WARN= G: Running as primary DPDK process. eRPC DPDK daemon is not running.
21:= 428893 INFOR: DPDK transport for Rpc 0 got QP 0
21:428927 INFOR: Initial= izing port 0 with driver net_ena
Ethdev port_id=3D0 invalid rss_hf: 0= x20820, valid value: 0xc30
terminate called after throwing an instan= ce of 'std::runtime_error'
=C2=A0 what(): =C2=A0Ethdev configura= tion error: Invalid argument
Aborted
--000000000000f138ff05d0638c2b--