RSS for ENA

RSS for ENA

[Fabio Fernandes]

[-- Attachment #1: Type: text/plain, Size: 1420 bytes --]

Hi,

I'm a bit puzzled why DPDK is reporting RSS to be not available on my ENA port (at AWS) when I'm expecting it to be.

My setup is:

> c7a.4xlarge
> [ENA_COM: ena_com_validate_version]ENA device version: 0.10
> [ENA_COM: ena_com_validate_version]ENA controller version: 0.0.1 implementation version 1
> 16-cores (hyper-threading disabled),
> 4 ENA interfaces/network adapters, but only one (with public IP) is bound to igb_uio
> DPDK 24.07
> 6.5.0-1020-aws,
> Ubuntu 22.04.4 LTS,

The device reports that 8 queues are available, but rte_eth_dev_info::rx_offload_capa does not have RTE_ETH_RX_OFFLOAD_RSS_HASH set after calling rte_eth_dev_info_get().

The flag value seems to be coming from here,
https://github.com/DPDK/dpdk/blob/b3485f4293997d35b6daecc3437bb0c183a51fb3/drivers/net/ena/ena_ethdev.c#L2402
Where the offload mask is:

> (ena_com_dev_get_features_ctx*) get_feat_ctx->offload
> tx = 3
> rx_supported = 7
> rx_enabled = 0

Isn't RSS supposed to be available on this ENA setup?

If not, how am I supposed to distribute inbound packets across those 8 receive queues/CPUs?

Some notes, log:

> at eth_ena_dev_init(), ena_use_large_llq_hdr() returned true
> ena_get_metrics_entries(): 0x6 customer metrics are supported
> ena_use_large_llq_hdr(): Recommended device entry size policy 4
> eth_ena_dev_init(): LLQ entry size 256B
> eth_ena_dev_init(): Placement policy: Low latency

Any advice?

Thanks,Fabio

[-- Attachment #2: Type: text/html, Size: 10044 bytes --]

Re: RSS for ENA

[Ivan Malov]

[-- Attachment #1: Type: text/plain, Size: 2112 bytes --]

Hi Fabio,

Have you considered to select MQ mode 'RSS' [1] via 'dev_conf.rxmode.mq_mode'
during 'rte_eth_dev_configure' [2] stage? Just to double-check.

Also, the 'RSS_HASH' flag could possibly mean the availability of 32-bit hash
value in received mbufs rather than support for RSS distribution as a whole.

[1] https://doc.dpdk.org/api-24.07/rte__ethdev_8h.html#a586b8e86131b4ec0ccaf464e847ccf3e
[2] https://doc.dpdk.org/api-24.07/rte__ethdev_8h.html#a1a7d3a20b102fee222541fda50fd87bd

Thank you.

On Sat, 10 Aug 2024, Fabio Fernandes wrote:

> Hi,
> I'm a bit puzzled why DPDK is reporting RSS to be not available on my ENA port (at AWS) when I'm expecting it to be.
> 
> My setup is:
>       c7a.4xlarge
>     [ENA_COM: ena_com_validate_version]ENA device version: 0.10
>     [ENA_COM: ena_com_validate_version]ENA controller version: 0.0.1 implementation version 1
> 16-cores (hyper-threading disabled),
> 4 ENA interfaces/network adapters, but only one (with public IP) is bound to igb_uio
> DPDK 24.07
> 6.5.0-1020-aws,
> Ubuntu 22.04.4 LTS,
> 
> 
> The device reports that 8 queues are available, but rte_eth_dev_info::rx_offload_capa does not have RTE_ETH_RX_OFFLOAD_RSS_HASH set after calling rte_eth_dev_info_get().
> 
> The flag value seems to be coming from here,
> https://github.com/DPDK/dpdk/blob/b3485f4293997d35b6daecc3437bb0c183a51fb3/drivers/net/ena/ena_ethdev.c#L2402
> Where the offload mask is:
>           (ena_com_dev_get_features_ctx*) get_feat_ctx->offload
>             tx = 3
>             rx_supported = 7
>             rx_enabled = 0
> 
> 
> Isn't RSS supposed to be available on this ENA setup?
> 
> If not, how am I supposed to distribute inbound packets across those 8 receive queues/CPUs?
> 
> Some notes, log:
>       at eth_ena_dev_init(), ena_use_large_llq_hdr() returned true
> ena_get_metrics_entries(): 0x6 customer metrics are supported
> ena_use_large_llq_hdr(): Recommended device entry size policy 4
> eth_ena_dev_init(): LLQ entry size 256B
> eth_ena_dev_init(): Placement policy: Low latency
> 
> 
> Any advice?
> 
> Thanks,
> Fabio
> 
> 
>

Re: RSS for ENA

[Stephen Hemminger]

On Sun, 11 Aug 2024 19:55:08 +0400 (+04)
Ivan Malov <ivan.malov@arknetworks.am> wrote:

> Hi Fabio,
> 
> Have you considered to select MQ mode 'RSS' [1] via 'dev_conf.rxmode.mq_mode'
> during 'rte_eth_dev_configure' [2] stage? Just to double-check.
> 
> Also, the 'RSS_HASH' flag could possibly mean the availability of 32-bit hash
> value in received mbufs rather than support for RSS distribution as a whole.
> 
> [1] https://doc.dpdk.org/api-24.07/rte__ethdev_8h.html#a586b8e86131b4ec0ccaf464e847ccf3e
> [2] https://doc.dpdk.org/api-24.07/rte__ethdev_8h.html#a1a7d3a20b102fee222541fda50fd87bd

Please do not top post on mailing lists.

> Hi,
> I'm a bit puzzled why DPDK is reporting RSS to be not available on my ENA port (at AWS) when I'm expecting it to be.
> 
> My setup is:
>       c7a.4xlarge
>     [ENA_COM: ena_com_validate_version]ENA device version: 0.10
>     [ENA_COM: ena_com_validate_version]ENA controller version: 0.0.1 implementation version 1
> 16-cores (hyper-threading disabled),
> 4 ENA interfaces/network adapters, but only one (with public IP) is bound to igb_uio
> DPDK 24.07
> 6.5.0-1020-aws,
> Ubuntu 22.04.4 LTS,
> 
> 
> The device reports that 8 queues are available, but rte_eth_dev_info::rx_offload_capa does not have RTE_ETH_RX_OFFLOAD_RSS_HASH set after calling rte_eth_dev_info_get().
> 
> The flag value seems to be coming from here,
> https://github.com/DPDK/dpdk/blob/b3485f4293997d35b6daecc3437bb0c183a51fb3/drivers/net/ena/ena_ethdev.c#L2402
> Where the offload mask is:
>           (ena_com_dev_get_features_ctx*) get_feat_ctx->offload
>             tx = 3
>             rx_supported = 7
>             rx_enabled = 0
> 
> 
> Isn't RSS supposed to be available on this ENA setup?
> 
> If not, how am I supposed to distribute inbound packets across those 8 receive queues/CPUs?
> 
> Some notes, log:
>       at eth_ena_dev_init(), ena_use_large_llq_hdr() returned true
> ena_get_metrics_entries(): 0x6 customer metrics are supported
> ena_use_large_llq_hdr(): Recommended device entry size policy 4
> eth_ena_dev_init(): LLQ entry size 256B
> eth_ena_dev_init(): Placement policy: Low latency
> 

Yes, the OFFLOAD_RSS_HASH flag indicates that the received mbuf will contain the hash value.
This driver only does it if multi-queue is enabled (which makes sense).


You need rxmode.mq_mode with RTE_ETH_MQ_RX_RSS_FLAG to make use of all 8 queues.

Re: RSS for ENA

[Fabio Fernandes]

Hi Ivan,

Thanks for your response. 

Yes I did remember that. I found the issue and it was a bit specific to ENA.

I had missed this part in the doc:
"When this mode is being activated, driver will automatically enable offload RTE_ETH_RX_OFFLOAD_RSS_HASH and pass RSS hash value to the mbuf."

https://github.com/amzn/amzndrivers/tree/master/userspace/dpdk#111-rss-feature

/Not/ specifying this flag during setup did the trick.

Thanks,
Fabio