Linux version 5.4.0-1045-aws)
- Apply the vfio pci driver from https://github.com/amzn/am zn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch
- Using DPDK code from head of main branch. I built using ninja/meson
- Add a 2nd ENA enabled NIC
- Setup 2Mb huge pages, and bound the 2nd NIC with dpdk-devbind
I run test-pmd with this command:
$ sudo ./dpdk-testpmd --huge-dir=/mnt/huge -c 0xf -n 4 -- -i
EAL: Detected CPU lcores: 72
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(3)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Error picking flow transfer proxy for port 0: Function not implemented - ignore
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 16:9A:11:4A:5D:61
Checking link statuses...
Done
Error during enabling promiscuous mode for port 0: Operation not supported - ignore
Key point: I try to enable RSS hash offload for UDP or TCP. I get the following error shown in bold:testpmd> port config all rss udp
Port 0 modified RSS hash function based on hardware support,requested:0x20820 configured:0x820
ena_rss_hash_set(): Setting RSS hash fields is not supported
ena_rss_hash_update(): Failed to set RSS hash
Configuration of RSS hash at ethernet port 0 failed with error (95): Operation not supported.testpmd> port config all rss tcp
Port 0 modified RSS hash function based on hardware support,requested:0x10410 configured:0x410
ena_rss_hash_set(): Setting RSS hash fields is not supported
ena_rss_hash_update(): Failed to set RSS hash
Configuration of RSS hash at ethernet port 0 failed with error (95): Operation not supported.
If I run ethtool on the NIC before putting it down state and DPDK binding to vfio-pci I see an indication that there is a hash key, and RSS Hash works:
ubuntu$ ethtool -x ens6
RX flow hash indirection table for ens6 with 32 RX ring(s):
0: 0 1 2 3 4 5 6 7
8: 8 9 10 11 12 13 14 15
16: 16 17 18 19 20 21 22 23
24: 24 25 26 27 28 29 30 31
32: 0 1 2 3 4 5 6 7
40: 8 9 10 11 12 13 14 15
48: 16 17 18 19 20 21 22 23
56: 24 25 26 27 28 29 30 31
64: 0 1 2 3 4 5 6 7
72: 8 9 10 11 12 13 14 15
80: 16 17 18 19 20 21 22 23
88: 24 25 26 27 28 29 30 31
96: 0 1 2 3 4 5 6 7
104: 8 9 10 11 12 13 14 15
112: 16 17 18 19 20 21 22 23
120: 24 25 26 27 28 29 30 31
RSS hash key:
80:d3:0f:c4:1d:8a:48:9e:7f:1d:18:91:d3:12:cb:38:87:fc:33:49:18:30:e4:1d:19:8f:9d:fd:c3:94:fe:f3:b2:f0:7f:e9:e9:9d:2d:af
RSS hash function:
toeplitz: on
xor: off
crc32: off
What gives? Do I need to modify a constant somewhere in DPDK?
At the risk of expanding scope, I am really trying to run eRPC which runs on top of DPDK. It fails in dpdk_init with a similar but slightly different error shown in bold in which the putative valid value is different than above:
$ sudo ./hello_server
172.31.77.76:3185020:536566 INFOR: eRPC Nexus: Launching 0 background threads.
20:536582 INFOR: eRPC Nexus: Launching session management thread on core 53.
20:536776 INFOR: eRPC Nexus: Created with management UDP port 31850, hostname 172.31.77.76.
20:536832 INFOR: DPDK transport for Rpc 0 initializing DPDK EAL.
EAL: Detected CPU lcores: 72
EAL: Detected NUMA nodes: 2
EAL: Auto-detected process type: PRIMARY
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(3)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
21:428883 WARNG: Running as primary DPDK process. eRPC DPDK daemon is not running.
21:428893 INFOR: DPDK transport for Rpc 0 got QP 0
21:428927 INFOR: Initializing port 0 with driver net_ena
Ethdev port_id=0 invalid rss_hf: 0x20820, valid value: 0xc30
terminate called after throwing an instance of 'std::runtime_error'
what(): Ethdev configuration error: Invalid argument
Aborted