From: Edwin Brossette <edwin.brossette@6wind.com>
To: dev@dpdk.org
Cc: Olivier Matz <olivier.matz@6wind.com>
Subject: [ixgbevf] Problem with RSS initial config after device init on X550 nic
Date: Thu, 15 Feb 2024 14:12:44 +0100 [thread overview]
Message-ID: <CANDF9xDLFOMAniyGsDH6GEm_PCq0AVPBrraAw1epp5_9m1=bFw@mail.gmail.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 7255 bytes --]
Hello,
We recently ran into an issue with our product when working with an X550
nic with stable dpdk-23.11. We observed that all the incoming traffic was
directed only into a single queue.
The issue became evident after displaying the RSS reta which was fully
zeroed after device init, thus directing all traffic to rxq0. Moreover, RSS
hash key did not seem to be correctly initialized. Manually setting the
reta afterwards was enough to balance the incoming traffic between our
queues, which convinced me that the issue here was simply a matter of
correctly initializing the device on port start.
Looking into the pmd's code, I couldn't see any RSS configuration done vf
side at device startup, at least not in ixgbevf_dev_rx_init(). I've seen
ixgbe_dev_mq_rx_configure() was called during the pf's init and configured
RSS to be handled by vf if sriov was on, but this isn't enough to fully
configure RSS for the vf. (see code here:
https://git.dpdk.org/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx.c#n4644 )
I have also observed that all different models of nics using ixgbe did not
handle RSS in the same way. For example, for nics of the 82599 series, it
is written in their datasheet that, for IOV mode: "— Note that RSS is not
supported in IOV mode since there is only a single RSS hash function in the
hardware." On the contrary, x550 nics have special registers to handle RSS
in VF, like VFRSSRK and VFRETA, for example. I believe the RSS config not
being initialized for X550 nics might come from a slight misunderstanding
on this part.
Therefore, I can suggest a patch to add a call to ixgbe_rss_configure()
somewhere in ixgbevf_dev_rx_init() specifically for this model of nic.
Despite this function being named ixgbe_xxx instead of ixgbevf_xxx, it will
do correct initialization for RSS in vf mode because all functions to get
RSS-related registers such as ixgbe_reta_reg_get() or ixgbe_reta_size_get()
will check if the device is in vf or pf mode and fetch the appropriate
registers.
Here is a way to reproduce, on an X550 card:
Here are the nics I am using:
0000:08:00.0 ntfp1 ac:1f:6b:57:57:74 ixgbe 1x2.5 GT/s PCIe 1x2.5
GT/s PCIe Intel Corporation Ethernet Connection X553 10 GbE SFP+
0000:08:00.1 ntfp2 ac:1f:6b:57:57:75 ixgbe 1x2.5 GT/s PCIe 1x2.5
GT/s PCIe Intel Corporation Ethernet Connection X553 10 GbE SFP+
0000:08:10.0 eth0 d2:c4:fc:c5:c3:05 ixgbevf 1x2.5 GT/s PCIe
0xUnknown Intel Corporation X553 Virtual Function
0000:08:10.2 eth1 e2:a8:68:09:20:29 ixgbevf 1x2.5 GT/s PCIe
0xUnknown Intel Corporation X553 Virtual Function
1) Starting up dpdk-testpmd:
sudo dpdk-hugepages.py --setup 2G;
dpdk-devbind --bind=vfio-pci 0000:08:10.0
dpdk-devbind --bind=vfio-pci 0000:08:10.2
dpdk-testpmd -a 0000:08:10.0 -a 0000:08:10.2 -- -i --rxq=2 --txq=2
--coremask=0xff0 --total-num-mbufs=250000
EAL: Detected CPU lcores: 12
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_ixgbe_vf (8086:15c5) device: 0000:08:10.0
(socket -1)
EAL: Probe PCI driver: net_ixgbe_vf (8086:15c5) device: 0000:08:10.2
(socket -1)
Interactive-mode selected
previous number of forwarding cores 1 - changed to number of configured
cores 8
Warning: NUMA should be configured manually by using --port-numa-config and
--ring-numa-config parameters along with --numa.
testpmd: create a new mbuf pool <mb_pool_0>: n=250000, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: 02:09:C0:9E:09:75
Configuring Port 1 (socket 0)
Port 1: 02:09:C0:76:6D:4B
Checking link statuses...
Done
testpmd>
2) Display port info:
testpmd> show port info 0
********************* Infos for port 0 *********************
MAC address: 02:09:C0:B8:68:2F
Device name: 0000:08:10.0
Driver name: net_ixgbe_vf
Firmware-version: not available
Devargs:
Connect to socket: 0
memory allocation on the socket: 0
Link status: down
Link speed: None
Link duplex: half-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: disabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 4096
VLAN offload:
strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 64
Supported RSS offload flow types:
ipv4 ipv4-tcp ipv4-udp ipv6 ipv6-tcp ipv6-udp ipv6-ex
ipv6-tcp-ex ipv6-udp-ex
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 9728
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VMDq pools: 64
Current number of RX queues: 2
Max possible RX queues: 4
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 32
RXDs number alignment: 8
Current number of TX queues: 2
Max possible TX queues: 4
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 32
TXDs number alignment: 8
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )
Device error handling mode: passive
Device private info:
none
3) Display RSS conf:
testpmd> show port 0 rss-hash
ixgbe_dev_rss_hash_conf_get(): <log>: rss is enabled
RSS functions:
ipv4 ipv4-tcp ipv6 ipv6-tcp
testpmd> show port 0 rss-hash key
ixgbe_dev_rss_hash_conf_get(): <log>: rss is enabled
RSS functions:
ipv4 ipv4-tcp ipv6 ipv6-tcp
RSS key:
88F1A05B9FFCD601333EB3FF4176AE8836B36D67D4013A4B75F25806D17078D08C1EF6A69FF29A78
testpmd> show port 0 rss reta
[UINT16]: show port <port_id> rss reta <size> <mask0[,mask1]*>
testpmd> show port 0 rss-hash algorithm
ixgbe_dev_rss_hash_conf_get(): <log>: rss is enabled
RSS algorithm:
default
testpmd> show port 0 rss reta 64 (0xfff)
RSS RETA configuration: hash index=0, queue=0
RSS RETA configuration: hash index=1, queue=0
RSS RETA configuration: hash index=2, queue=0
RSS RETA configuration: hash index=3, queue=0
RSS RETA configuration: hash index=4, queue=0
RSS RETA configuration: hash index=5, queue=0
RSS RETA configuration: hash index=6, queue=0
RSS RETA configuration: hash index=7, queue=0
RSS RETA configuration: hash index=8, queue=0
RSS RETA configuration: hash index=9, queue=0
RSS RETA configuration: hash index=10, queue=0
RSS RETA configuration: hash index=11, queue=0
Here you can see the reta is full of 0s, which is causing me the
performance issues.
The log appearing above "ixgbe_dev_rss_hash_conf_get(): <log>: rss is
enabled" is a custom log I added in ixgbe_dev_rss_hash_conf_get(), just
after the check that ixgbe_rss_enabled() didn't return 0 (see
https://git.dpdk.org/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx.c#n3670 ).
I also had occurrences where the RSS key and algorithm weren't set. In
these cases, test-pmd would tell me that RSS is disabled, even though it
was enabled and simply not configured. After adding my suggested patch, the
key set was the default key and the reta was correctly initialized
(alternating 1s and 0s).
[-- Attachment #2: Type: text/html, Size: 8009 bytes --]
next reply other threads:[~2024-02-15 13:12 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-15 13:12 Edwin Brossette [this message]
2024-02-15 13:31 ` [PATCH] net/ixgbevf: fix RSS init for x550 nics edwin.brossette
2024-02-27 17:26 ` Medvedkin, Vladimir
2024-02-29 11:49 ` Bruce Richardson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CANDF9xDLFOMAniyGsDH6GEm_PCq0AVPBrraAw1epp5_9m1=bFw@mail.gmail.com' \
--to=edwin.brossette@6wind.com \
--cc=dev@dpdk.org \
--cc=olivier.matz@6wind.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).