[ixgbevf] Problem with RSS initial config after device init on X550 nic

[ixgbevf] Problem with RSS initial config after device init on X550 nic

[Edwin Brossette]

[-- Attachment #1: Type: text/plain, Size: 7255 bytes --]

Hello,

We recently ran into an issue with our product when working with an X550
nic with stable dpdk-23.11. We observed that all the incoming traffic was
directed only into a single queue.

The issue became evident after displaying the RSS reta which was fully
zeroed after device init, thus directing all traffic to rxq0. Moreover, RSS
hash key did not seem to be correctly initialized. Manually setting the
reta afterwards was enough to balance the incoming traffic between our
queues, which convinced me that the issue here was simply a matter of
correctly initializing the device on port start.

Looking into the pmd's code, I couldn't see any RSS configuration done vf
side at device startup, at least not in ixgbevf_dev_rx_init(). I've seen
ixgbe_dev_mq_rx_configure() was called during the pf's init and configured
RSS to be handled by vf if sriov was on, but this isn't enough to fully
configure RSS for the vf. (see code here:
https://git.dpdk.org/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx.c#n4644 )

I have also observed that all different models of nics using ixgbe did not
handle RSS in the same way. For example, for nics of the 82599 series, it
is written in their datasheet that, for IOV mode: "— Note that RSS is not
supported in IOV mode since there is only a single RSS hash function in the
hardware." On the contrary, x550 nics have special registers to handle RSS
in VF, like VFRSSRK and VFRETA, for example. I believe the RSS config not
being initialized for X550 nics might come from a slight misunderstanding
on this part.

Therefore, I can suggest a patch to add a call to ixgbe_rss_configure()
somewhere in ixgbevf_dev_rx_init() specifically for this model of nic.
Despite this function being named ixgbe_xxx instead of ixgbevf_xxx, it will
do correct initialization for RSS in vf mode because all functions to get
RSS-related registers such as ixgbe_reta_reg_get() or ixgbe_reta_size_get()
will check if the device is in vf or pf mode and fetch the appropriate
registers.

Here is a way to reproduce, on an X550 card:

Here are the nics I am using:
0000:08:00.0  ntfp1   ac:1f:6b:57:57:74  ixgbe    1x2.5 GT/s PCIe  1x2.5
GT/s PCIe  Intel Corporation Ethernet Connection X553 10 GbE SFP+
0000:08:00.1  ntfp2   ac:1f:6b:57:57:75  ixgbe    1x2.5 GT/s PCIe  1x2.5
GT/s PCIe  Intel Corporation Ethernet Connection X553 10 GbE SFP+
0000:08:10.0  eth0    d2:c4:fc:c5:c3:05  ixgbevf  1x2.5 GT/s PCIe
 0xUnknown        Intel Corporation X553 Virtual Function
0000:08:10.2  eth1    e2:a8:68:09:20:29  ixgbevf  1x2.5 GT/s PCIe
 0xUnknown        Intel Corporation X553 Virtual Function

    1) Starting up dpdk-testpmd:

sudo dpdk-hugepages.py --setup 2G;
dpdk-devbind --bind=vfio-pci 0000:08:10.0
dpdk-devbind --bind=vfio-pci 0000:08:10.2

dpdk-testpmd -a 0000:08:10.0 -a 0000:08:10.2 -- -i --rxq=2 --txq=2
--coremask=0xff0 --total-num-mbufs=250000
EAL: Detected CPU lcores: 12
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_ixgbe_vf (8086:15c5) device: 0000:08:10.0
(socket -1)
EAL: Probe PCI driver: net_ixgbe_vf (8086:15c5) device: 0000:08:10.2
(socket -1)
Interactive-mode selected
previous number of forwarding cores 1 - changed to number of configured
cores 8
Warning: NUMA should be configured manually by using --port-numa-config and
--ring-numa-config parameters along with --numa.
testpmd: create a new mbuf pool <mb_pool_0>: n=250000, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: 02:09:C0:9E:09:75
Configuring Port 1 (socket 0)
Port 1: 02:09:C0:76:6D:4B
Checking link statuses...
Done
testpmd>

    2) Display port info:

testpmd> show port info 0

********************* Infos for port 0  *********************
MAC address: 02:09:C0:B8:68:2F
Device name: 0000:08:10.0
Driver name: net_ixgbe_vf
Firmware-version: not available
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: down
Link speed: None
Link duplex: half-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: disabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
  strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 64
Supported RSS offload flow types:
  ipv4  ipv4-tcp  ipv4-udp  ipv6  ipv6-tcp  ipv6-udp  ipv6-ex
  ipv6-tcp-ex  ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 9728
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 2
Max possible RX queues: 4
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 2
Max possible TX queues: 4
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )
Device error handling mode: passive
Device private info:
  none

    3) Display RSS conf:

testpmd> show port 0 rss-hash
ixgbe_dev_rss_hash_conf_get(): <log>: rss is enabled
RSS functions:
  ipv4  ipv4-tcp  ipv6  ipv6-tcp
testpmd> show port 0 rss-hash key
ixgbe_dev_rss_hash_conf_get(): <log>: rss is enabled
RSS functions:
  ipv4  ipv4-tcp  ipv6  ipv6-tcp
RSS key:
88F1A05B9FFCD601333EB3FF4176AE8836B36D67D4013A4B75F25806D17078D08C1EF6A69FF29A78
testpmd> show port 0 rss reta
 [UINT16]: show port <port_id> rss reta <size> <mask0[,mask1]*>
testpmd> show port 0 rss-hash algorithm
ixgbe_dev_rss_hash_conf_get(): <log>: rss is enabled
RSS algorithm:
  default
testpmd> show port 0 rss reta 64 (0xfff)
RSS RETA configuration: hash index=0, queue=0
RSS RETA configuration: hash index=1, queue=0
RSS RETA configuration: hash index=2, queue=0
RSS RETA configuration: hash index=3, queue=0
RSS RETA configuration: hash index=4, queue=0
RSS RETA configuration: hash index=5, queue=0
RSS RETA configuration: hash index=6, queue=0
RSS RETA configuration: hash index=7, queue=0
RSS RETA configuration: hash index=8, queue=0
RSS RETA configuration: hash index=9, queue=0
RSS RETA configuration: hash index=10, queue=0
RSS RETA configuration: hash index=11, queue=0

Here you can see the reta is full of 0s, which is causing me the
performance issues.
The log appearing above "ixgbe_dev_rss_hash_conf_get(): <log>: rss is
enabled" is a custom log I added in ixgbe_dev_rss_hash_conf_get(), just
after the check that ixgbe_rss_enabled() didn't return 0 (see
https://git.dpdk.org/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx.c#n3670 ).
I also had occurrences where the RSS key and algorithm weren't set. In
these cases, test-pmd would tell me that RSS is disabled, even though it
was enabled and simply not configured. After adding my suggested patch, the
key set was the default key and the reta was correctly initialized
(alternating 1s and 0s).

[-- Attachment #2: Type: text/html, Size: 8009 bytes --]

[PATCH] net/ixgbevf: fix RSS init for x550 nics

[edwin.brossette]

From: Edwin Brossette <edwin.brossette@6wind.com>

Different Intel nics with the igxbe pmd do not handle RSS in the same
way when working with virtualization. While some nics like Intel 82599ES
only have a single RSS table in the device and leave all rss features to
be handled by the pf, some other nics like x550 let the vf handle RSS
features. This can lead to different behavior when rss is enabled
depending on the model of nic used.

In particular, it occurred that ixgbevf_dev_rx_init() do not initiate
rss parameters at device init, even if the multi-queue mode option is
set in the device configuration (ie: RTE_ETH_MQ_RX_RSS is set). Note
that this issue went unnoticed until now, probably because some nics do
not really have support for RSS in virtualization mode.

Thus, depending on the nic used, we can we find ourselves in a situation
where RSS is not configured despite being enabled. This will cause
serious performance issues because the RSS reta will be fully zeroed,
causing all packets to go only in the first queue and leaving all the
others empty.

By looking at ixgbe_reta_size_get(), we can see that only X550 nic
models have a non zero reta size set in vf mode. Thus add a call to
ixgbe_rss_configure() for these cards in ixgbevf_dev_rx_init() if the
option to enable RSS is set.

Fixes: f4d1598ee14f ("ixgbevf: support RSS config on x550")
Signed-off-by: Edwin Brossette <edwin.brossette@6wind.com>
---
 drivers/net/ixgbe/ixgbe_rxtx.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c
index 537aa2f68de8..0aa968f7e258 100644
--- a/drivers/net/ixgbe/ixgbe_rxtx.c
+++ b/drivers/net/ixgbe/ixgbe_rxtx.c
@@ -5873,6 +5873,25 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
 		IXGBE_PSRTYPE_RQPL_SHIFT;
 	IXGBE_WRITE_REG(hw, IXGBE_VFPSRTYPE, psrtype);
 
+	/* Initialize the rss for x550_vf cards if enabled */
+	switch (hw->mac.type) {
+	case ixgbe_mac_X550_vf:
+	case ixgbe_mac_X550EM_x_vf:
+	case ixgbe_mac_X550EM_a_vf:
+		switch (dev->data->dev_conf.rxmode.mq_mode) {
+		case RTE_ETH_MQ_RX_RSS:
+		case RTE_ETH_MQ_RX_DCB_RSS:
+		case RTE_ETH_MQ_RX_VMDQ_RSS:
+			ixgbe_rss_configure(dev);
+			break;
+		default:
+			break;
+		}
+		break;
+	default:
+		break;
+	}
+
 	ixgbe_set_rx_function(dev);
 
 	return 0;
-- 
2.35.0.4.g44a5d4affccf

Re: [PATCH] net/ixgbevf: fix RSS init for x550 nics

[Medvedkin, Vladimir]

On 15/02/2024 13:31, edwin.brossette@6wind.com wrote:
> From: Edwin Brossette <edwin.brossette@6wind.com>
>
> Different Intel nics with the igxbe pmd do not handle RSS in the same
> way when working with virtualization. While some nics like Intel 82599ES
> only have a single RSS table in the device and leave all rss features to
> be handled by the pf, some other nics like x550 let the vf handle RSS
> features. This can lead to different behavior when rss is enabled
> depending on the model of nic used.
>
> In particular, it occurred that ixgbevf_dev_rx_init() do not initiate
> rss parameters at device init, even if the multi-queue mode option is
> set in the device configuration (ie: RTE_ETH_MQ_RX_RSS is set). Note
> that this issue went unnoticed until now, probably because some nics do
> not really have support for RSS in virtualization mode.
>
> Thus, depending on the nic used, we can we find ourselves in a situation
> where RSS is not configured despite being enabled. This will cause
> serious performance issues because the RSS reta will be fully zeroed,
> causing all packets to go only in the first queue and leaving all the
> others empty.
>
> By looking at ixgbe_reta_size_get(), we can see that only X550 nic
> models have a non zero reta size set in vf mode. Thus add a call to
> ixgbe_rss_configure() for these cards in ixgbevf_dev_rx_init() if the
> option to enable RSS is set.
>
> Fixes: f4d1598ee14f ("ixgbevf: support RSS config on x550")
> Signed-off-by: Edwin Brossette <edwin.brossette@6wind.com>
> ---
>   drivers/net/ixgbe/ixgbe_rxtx.c | 19 +++++++++++++++++++
>   1 file changed, 19 insertions(+)
>
> diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c
> index 537aa2f68de8..0aa968f7e258 100644
> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
> @@ -5873,6 +5873,25 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
>   		IXGBE_PSRTYPE_RQPL_SHIFT;
>   	IXGBE_WRITE_REG(hw, IXGBE_VFPSRTYPE, psrtype);
>   
> +	/* Initialize the rss for x550_vf cards if enabled */
> +	switch (hw->mac.type) {
> +	case ixgbe_mac_X550_vf:
> +	case ixgbe_mac_X550EM_x_vf:
> +	case ixgbe_mac_X550EM_a_vf:
> +		switch (dev->data->dev_conf.rxmode.mq_mode) {
> +		case RTE_ETH_MQ_RX_RSS:
> +		case RTE_ETH_MQ_RX_DCB_RSS:
> +		case RTE_ETH_MQ_RX_VMDQ_RSS:
> +			ixgbe_rss_configure(dev);
> +			break;
> +		default:
> +			break;
> +		}
> +		break;
> +	default:
> +		break;
> +	}
> +
>   	ixgbe_set_rx_function(dev);
>   
>   	return 0;
Acked-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>

-- 
Regards,
Vladimir

Re: [PATCH] net/ixgbevf: fix RSS init for x550 nics

[Bruce Richardson]

On Tue, Feb 27, 2024 at 05:26:06PM +0000, Medvedkin, Vladimir wrote:
> 
> On 15/02/2024 13:31, edwin.brossette@6wind.com wrote:
> > From: Edwin Brossette <edwin.brossette@6wind.com>
> > 
> > Different Intel nics with the igxbe pmd do not handle RSS in the same
> > way when working with virtualization. While some nics like Intel 82599ES
> > only have a single RSS table in the device and leave all rss features to
> > be handled by the pf, some other nics like x550 let the vf handle RSS
> > features. This can lead to different behavior when rss is enabled
> > depending on the model of nic used.
> > 
> > In particular, it occurred that ixgbevf_dev_rx_init() do not initiate
> > rss parameters at device init, even if the multi-queue mode option is
> > set in the device configuration (ie: RTE_ETH_MQ_RX_RSS is set). Note
> > that this issue went unnoticed until now, probably because some nics do
> > not really have support for RSS in virtualization mode.
> > 
> > Thus, depending on the nic used, we can we find ourselves in a situation
> > where RSS is not configured despite being enabled. This will cause
> > serious performance issues because the RSS reta will be fully zeroed,
> > causing all packets to go only in the first queue and leaving all the
> > others empty.
> > 
> > By looking at ixgbe_reta_size_get(), we can see that only X550 nic
> > models have a non zero reta size set in vf mode. Thus add a call to
> > ixgbe_rss_configure() for these cards in ixgbevf_dev_rx_init() if the
> > option to enable RSS is set.
> > 
> > Fixes: f4d1598ee14f ("ixgbevf: support RSS config on x550")

+ Cc: stable@dpdk.org

> > Signed-off-by: Edwin Brossette <edwin.brossette@6wind.com>
> > ---
> Acked-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
> 

Applied to next-net-intel tree.
Thanks,
/Bruce