RSS Hash suport in DPDK on Amazon ENA NICs broken?

RSS Hash suport in DPDK on Amazon ENA NICs broken?

[fwefew 4t4tg]

[-- Attachment #1: Type: text/plain, Size: 5077 bytes --]

On AWS EC2 I configure:

- c5n.metal instance
- Stock AWS AMI Ubuntu 20.04 LTS ($ cat /proc/version
Linux version 5.4.0-1045-aws)
- Apply the vfio pci driver from https://github.com/amzn/am
zn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch
- Using DPDK code from head of main branch. I built using ninja/meson
- Add a 2nd ENA enabled NIC
- Setup 2Mb huge pages, and bound the 2nd NIC with dpdk-devbind

I run test-pmd with this command:

$ sudo ./dpdk-testpmd --huge-dir=/mnt/huge -c 0xf -n 4 -- -i
EAL: Detected CPU lcores: 72
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(3)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Error picking flow transfer proxy for port 0: Function not implemented -
ignore
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port
will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 16:9A:11:4A:5D:61
Checking link statuses...
Done
Error during enabling promiscuous mode for port 0: Operation not supported
- ignore

*Key point: I try to enable RSS hash offload for UDP or TCP. I get the
following error shown in bold:*

testpmd> port config all rss udp

*Port 0 modified RSS hash function based on hardware
support,requested:0x20820 configured:0x820*ena_rss_hash_set(): Setting RSS
hash fields is not supported
ena_rss_hash_update(): Failed to set RSS hash
*Configuration of RSS hash at ethernet port 0 failed with error (95):
Operation not supported.*
testpmd> port config all rss tcp

*Port 0 modified RSS hash function based on hardware
support,requested:0x10410 configured:0x410*ena_rss_hash_set(): Setting RSS
hash fields is not supported
ena_rss_hash_update(): Failed to set RSS hash

*Configuration of RSS hash at ethernet port 0 failed with error (95):
Operation not supported.*
If I run ethtool on the NIC before putting it down state and DPDK binding
to vfio-pci I see an indication that there is a hash key, and RSS Hash
works:

ubuntu$ ethtool -x ens6
RX flow hash indirection table for ens6 with 32 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      8     9    10    11    12    13    14    15
   16:     16    17    18    19    20    21    22    23
   24:     24    25    26    27    28    29    30    31
   32:      0     1     2     3     4     5     6     7
   40:      8     9    10    11    12    13    14    15
   48:     16    17    18    19    20    21    22    23
   56:     24    25    26    27    28    29    30    31
   64:      0     1     2     3     4     5     6     7
   72:      8     9    10    11    12    13    14    15
   80:     16    17    18    19    20    21    22    23
   88:     24    25    26    27    28    29    30    31
   96:      0     1     2     3     4     5     6     7
  104:      8     9    10    11    12    13    14    15
  112:     16    17    18    19    20    21    22    23
  120:     24    25    26    27    28    29    30    31
RSS hash key:
80:d3:0f:c4:1d:8a:48:9e:7f:1d:18:91:d3:12:cb:38:87:fc:33:49:18:30:e4:1d:19:8f:9d:fd:c3:94:fe:f3:b2:f0:7f:e9:e9:9d:2d:af
RSS hash function:
    toeplitz: on
    xor: off
    crc32: off

What gives? Do I need to modify a constant somewhere in DPDK?

At the risk of expanding scope, I am really trying to run eRPC which runs
on top of DPDK. It fails in dpdk_init with a similar but slightly different
error shown in bold in which the putative valid value is different than
above:

$ sudo ./hello_server 172.31.77.76:31850
20:536566 INFOR: eRPC Nexus: Launching 0 background threads.
20:536582 INFOR: eRPC Nexus: Launching session management thread on core 53.
20:536776 INFOR: eRPC Nexus: Created with management UDP port 31850,
hostname 172.31.77.76.
20:536832 INFOR: DPDK transport for Rpc 0 initializing DPDK EAL.
EAL: Detected CPU lcores: 72
EAL: Detected NUMA nodes: 2
EAL: Auto-detected process type: PRIMARY
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(3)
EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
21:428883 WARNG: Running as primary DPDK process. eRPC DPDK daemon is not
running.
21:428893 INFOR: DPDK transport for Rpc 0 got QP 0
21:428927 INFOR: Initializing port 0 with driver net_ena

*Ethdev port_id=0 invalid rss_hf: 0x20820, valid value: 0xc30*terminate
called after throwing an instance of 'std::runtime_error'
  what():  Ethdev configuration error: Invalid argument
Aborted

[-- Attachment #2: Type: text/html, Size: 5999 bytes --]

Re: [dpdk-dev] RSS Hash suport in DPDK on Amazon ENA NICs broken?

[Michał Krawczyk]

+Shai

śr., 10 lis 2021 o 00:34 fwefew 4t4tg <7532yahoo@gmail.com> napisał(a):
>
> On AWS EC2 I configure:
>
> - c5n.metal instance
> - Stock AWS AMI Ubuntu 20.04 LTS ($ cat /proc/version
> Linux version 5.4.0-1045-aws)
> - Apply the vfio pci driver from https://github.com/amzn/am
> zn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch
> - Using DPDK code from head of main branch. I built using ninja/meson
> - Add a 2nd ENA enabled NIC
> - Setup 2Mb huge pages, and bound the 2nd NIC with dpdk-devbind
>
> I run test-pmd with this command:
>
> $ sudo ./dpdk-testpmd --huge-dir=/mnt/huge -c 0xf -n 4 -- -i
> EAL: Detected CPU lcores: 72
> EAL: Detected NUMA nodes: 2
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: No available 1048576 kB hugepages reported
> EAL: VFIO support initialized
> EAL: Using IOMMU type 1 (Type 1)
> EAL: Ignore mapping IO port bar(3)
> EAL: Probe PCI driver: net_ena (1d0f:ec20) device: 0000:7f:00.0 (socket 0)
> TELEMETRY: No legacy callbacks, legacy socket not created
> Interactive-mode selected
> Error picking flow transfer proxy for port 0: Function not implemented -
> ignore
> testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> Warning! port-topology=paired and odd forward ports number, the last port
> will pair with itself.
> Configuring Port 0 (socket 0)
> Port 0: 16:9A:11:4A:5D:61
> Checking link statuses...
> Done
> Error during enabling promiscuous mode for port 0: Operation not supported
> - ignore
>
> *Key point: I try to enable RSS hash offload for UDP or TCP. I get the
> following error shown in bold:*
>
> testpmd> port config all rss udp
>
> *Port 0 modified RSS hash function based on hardware
> support,requested:0x20820 configured:0x820*ena_rss_hash_set(): Setting RSS
> hash fields is not supported
> ena_rss_hash_update(): Failed to set RSS hash
> *Configuration of RSS hash at ethernet port 0 failed with error (95):
> Operation not supported.*

As you can see, setting RSS hash fields is not supported by the
hardware. The driver provides this API call, but it cannot be
processed by the hardware as it lacks this feature.

If this API is not supported, the driver returns the defaults for
which the RSS HF is enabled in the FW:

#define ENA_ALL_RSS_HF (RTE_ETH_RSS_NONFRAG_IPV4_TCP |
RTE_ETH_RSS_NONFRAG_IPV4_UDP | \
RTE_ETH_RSS_NONFRAG_IPV6_TCP | RTE_ETH_RSS_NONFRAG_IPV6_UDP)

This is the 0xc30 value which you can see as 'valid value' in the
'hello_server' application.

Configuration of the RSS hash key should work although the RSS HF
cannot be changed. And RSS will be enabled for both the TCP and the
UDP by default - however it cannot be turned off.

Thanks,
Michal