* VMware run the DPDK with vmxnet3 NIC failed with the RSS config
@ 2025-04-16 8:23 Fangyin Hu
2025-04-16 13:59 ` Knight, Joshua
0 siblings, 1 reply; 4+ messages in thread
From: Fangyin Hu @ 2025-04-16 8:23 UTC (permalink / raw)
To: users
[-- Attachment #1: Type: text/plain, Size: 3735 bytes --]
Hi
My application developed with the DPDK, we found out one error message when my application run in the VMware platform.
[2025-04-15 03:01:09]:: --[vmxnet3_dev_start:961] hw->version: 6, rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_rss_configure:1434] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: vmxnet3_v4_rss_configure(): Set RSS fields (v4) failed: 1^M^M
[2025-04-15 03:01:09]:: vmxnet3_dev_start(): Failed to configure v4 RSS^M^M
From the debug logs, it seems that the vmxnet3 device failed to configure the v4 RSS.
We can found these informations from the DPDK's code snippet:
rss_hf = 3380 = 0xd34
So the RSS configure is this:
#define VMXNET3_RSS_OFFLOAD_ALL ( \
RTE_ETH_RSS_IPV4 | \
RTE_ETH_RSS_NONFRAG_IPV4_TCP | \
RTE_ETH_RSS_IPV6 | \
RTE_ETH_RSS_NONFRAG_IPV6_TCP)
#define VMXNET3_V4_RSS_MASK ( \
RTE_ETH_RSS_NONFRAG_IPV4_UDP | \
RTE_ETH_RSS_NONFRAG_IPV6_UDP)
The function call vmxnet3_rss_configure() is works fine but the function call vmxnet3_v4_rss_configure() failed:
if (VMXNET3_VERSION_GE_4(hw) &&
dev->data->dev_conf.rxmode.mq_mode == RTE_ETH_MQ_RX_RSS) {
/* Check for additional RSS */
ret = vmxnet3_v4_rss_configure(dev);
if (ret != VMXNET3_SUCCESS) {
PMD_INIT_LOG(ERR, "Failed to configure v4 RSS");
return ret;
}
}
/*
* Additional RSS configurations based on vmxnet v4+ APIs
*/
int
vmxnet3_v4_rss_configure(struct rte_eth_dev *dev)
{
struct vmxnet3_hw *hw = dev->data->dev_private;
Vmxnet3_DriverShared *shared = hw->shared;
Vmxnet3_CmdInfo *cmdInfo = &shared->cu.cmdInfo;
struct rte_eth_rss_conf *port_rss_conf;
uint64_t rss_hf;
uint32_t ret;
PMD_INIT_FUNC_TRACE();
cmdInfo->setRSSFields = 0;
port_rss_conf = &dev->data->dev_conf.rx_adv_conf.rss_conf;
printf("--[%s:%d] rss_hf: %llu --\n", __FUNCTION__, __LINE__, port_rss_conf->rss_hf);
if ((port_rss_conf->rss_hf & VMXNET3_MANDATORY_V4_RSS) !=
VMXNET3_MANDATORY_V4_RSS) {
PMD_INIT_LOG(WARNING, "RSS: IPv4/6 TCP is required for vmxnet3 v4 RSS,"
"automatically setting it");
port_rss_conf->rss_hf |= VMXNET3_MANDATORY_V4_RSS;
}
rss_hf = port_rss_conf->rss_hf &
(VMXNET3_V4_RSS_MASK | VMXNET3_RSS_OFFLOAD_ALL);
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_TCP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP4;
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_TCP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP6;
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_UDP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP4;
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_UDP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP6;
VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
VMXNET3_CMD_SET_RSS_FIELDS);
ret = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD);
if (ret != VMXNET3_SUCCESS) {
PMD_DRV_LOG(ERR, "Set RSS fields (v4) failed: %d", ret);
}
return ret;
}
It seems like the VMXNET3_WRITE_BAR1_REG failed, but don't know why this happened.
My DPDK version is 22.11.1 and the vmxnet3 driver version is 1.6.0.0-k-NAPI
BTW we just upgraded VM compatibility to ESXi 8.0 U2 and later (VM version 21), then our application run with this error logs.
The previous VM compatibility is ESXi 6.5 and later (VM version 13), and our application works fine. It is because the vmxnet3 hw version is 3, so it will not trigger to call the V4 RSS configure.
Could someone help to check this issue? Or maybe I missed something?
[-- Attachment #2: Type: text/html, Size: 14272 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: VMware run the DPDK with vmxnet3 NIC failed with the RSS config
2025-04-16 8:23 VMware run the DPDK with vmxnet3 NIC failed with the RSS config Fangyin Hu
@ 2025-04-16 13:59 ` Knight, Joshua
2025-04-18 7:49 ` Fangyin Hu
0 siblings, 1 reply; 4+ messages in thread
From: Knight, Joshua @ 2025-04-16 13:59 UTC (permalink / raw)
To: Fangyin Hu, users
[-- Attachment #1: Type: text/plain, Size: 5211 bytes --]
We’ve hit the same problem setting up RSS on vmxnet3 on newer esxi8 releases. In our case we would see the failure if we only had a single rx queue.
Snippet from my discussion with their support team:
> So to summarize, with VM Hardware version 14 in esxi 7 and 6.7, setting the RSS hash function did not generate an error in the driver. With VM Hardware version 14 in esxi 8U2, it generates an error in the driver.
I had a ticket opened with Broadcom and their response was they made a change in behavior where instead of silently rejecting the RSS configuration (since, really you don’t need RSS with a single queue although we still wanted it enabled) the nic now returns an error. The biggest problem I had with the change is they applied it to *earlier* vm hw versions, not just the latest/newest version.
Broadcom’s response to my ticket was:
>Regarding the difference in behavior you have observed with ESXi versions, there were some enhancements introduced in vSphere version 7.0U1 used along with VM hardware version >=14, it identifies the RSS misconfiguration with Rx Queue and throws an error message similar to what has been observed. In older versions, it was not erroring out so you will not observe any failures in the applications.
From: Fangyin Hu <FHu@SonicWall.com>
Date: Wednesday, April 16, 2025 at 4:23 AM
To: users@dpdk.org <users@dpdk.org>
Subject: VMware run the DPDK with vmxnet3 NIC failed with the RSS config
Hi
My application developed with the DPDK, we found out one error message when my application run in the VMware platform.
[2025-04-15 03:01:09]:: --[vmxnet3_dev_start:961] hw->version: 6, rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_rss_configure:1434] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: vmxnet3_v4_rss_configure(): Set RSS fields (v4) failed: 1^M^M
[2025-04-15 03:01:09]:: vmxnet3_dev_start(): Failed to configure v4 RSS^M^M
From the debug logs, it seems that the vmxnet3 device failed to configure the v4 RSS.
We can found these informations from the DPDK’s code snippet:
rss_hf = 3380 = 0xd34
So the RSS configure is this:
#define VMXNET3_RSS_OFFLOAD_ALL ( \
RTE_ETH_RSS_IPV4 | \
RTE_ETH_RSS_NONFRAG_IPV4_TCP | \
RTE_ETH_RSS_IPV6 | \
RTE_ETH_RSS_NONFRAG_IPV6_TCP)
#define VMXNET3_V4_RSS_MASK ( \
RTE_ETH_RSS_NONFRAG_IPV4_UDP | \
RTE_ETH_RSS_NONFRAG_IPV6_UDP)
The function call vmxnet3_rss_configure() is works fine but the function call vmxnet3_v4_rss_configure() failed:
if (VMXNET3_VERSION_GE_4(hw) &&
dev->data->dev_conf.rxmode.mq_mode == RTE_ETH_MQ_RX_RSS) {
/* Check for additional RSS */
ret = vmxnet3_v4_rss_configure(dev);
if (ret != VMXNET3_SUCCESS) {
PMD_INIT_LOG(ERR, "Failed to configure v4 RSS");
return ret;
}
}
/*
* Additional RSS configurations based on vmxnet v4+ APIs
*/
int
vmxnet3_v4_rss_configure(struct rte_eth_dev *dev)
{
struct vmxnet3_hw *hw = dev->data->dev_private;
Vmxnet3_DriverShared *shared = hw->shared;
Vmxnet3_CmdInfo *cmdInfo = &shared->cu.cmdInfo;
struct rte_eth_rss_conf *port_rss_conf;
uint64_t rss_hf;
uint32_t ret;
PMD_INIT_FUNC_TRACE();
cmdInfo->setRSSFields = 0;
port_rss_conf = &dev->data->dev_conf.rx_adv_conf.rss_conf;
printf("--[%s:%d] rss_hf: %llu --\n", __FUNCTION__, __LINE__, port_rss_conf->rss_hf);
if ((port_rss_conf->rss_hf & VMXNET3_MANDATORY_V4_RSS) !=
VMXNET3_MANDATORY_V4_RSS) {
PMD_INIT_LOG(WARNING, "RSS: IPv4/6 TCP is required for vmxnet3 v4 RSS,"
"automatically setting it");
port_rss_conf->rss_hf |= VMXNET3_MANDATORY_V4_RSS;
}
rss_hf = port_rss_conf->rss_hf &
(VMXNET3_V4_RSS_MASK | VMXNET3_RSS_OFFLOAD_ALL);
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_TCP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP4;
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_TCP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP6;
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_UDP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP4;
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_UDP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP6;
VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
VMXNET3_CMD_SET_RSS_FIELDS);
ret = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD);
if (ret != VMXNET3_SUCCESS) {
PMD_DRV_LOG(ERR, "Set RSS fields (v4) failed: %d", ret);
}
return ret;
}
It seems like the VMXNET3_WRITE_BAR1_REG failed, but don’t know why this happened.
My DPDK version is 22.11.1 and the vmxnet3 driver version is 1.6.0.0-k-NAPI
BTW we just upgraded VM compatibility to ESXi 8.0 U2 and later (VM version 21), then our application run with this error logs.
The previous VM compatibility is ESXi 6.5 and later (VM version 13), and our application works fine. It is because the vmxnet3 hw version is 3, so it will not trigger to call the V4 RSS configure.
Could someone help to check this issue? Or maybe I missed something?
[-- Attachment #2: Type: text/html, Size: 20604 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* RE: VMware run the DPDK with vmxnet3 NIC failed with the RSS config
2025-04-16 13:59 ` Knight, Joshua
@ 2025-04-18 7:49 ` Fangyin Hu
2025-04-22 12:49 ` spyroot
0 siblings, 1 reply; 4+ messages in thread
From: Fangyin Hu @ 2025-04-18 7:49 UTC (permalink / raw)
To: Knight, Joshua, users
[-- Attachment #1: Type: text/plain, Size: 7780 bytes --]
Hi, Knight
Thank you very much for your reply!
I check our codes, my application config the 1 RX queue for vmxnet3 device, so the call vmxnet3_v4_rss_configure() will failed due to the single RX queue.
I adjusted our codes and use 2 RX queue for vmxnet3 device, then retest my application aga in. Now my application works fine and here are some debug logs.
[2025-04-18 07:37:02]:: Try to configure port 0, RX Queue Number: 2, TX Queue Number: 1 Dev 0^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:310:init_options():DPDK interface (net_vmxnet3): 0^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:312:init_options(): multicast_en: 1^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:313:init_options(): num_rx_desc: 4096^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:314:init_options(): num_tx_desc: 512^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:315:init_options(): rx_drop_en: 0^M^M
[2025-04-18 07:37:02]:: odp_packet_io.c:439:odp_pktio_open():interface: 0, driver: dpdk^M^M
[2025-04-18 07:37:02]:: created pktio 1 idx( 0) (0)^M^M
[2025-04-18 07:37:02]:: ### starting data port pktio's...^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:2277:dpdk_setup_eth_tx():Mapping per TX queue statistics not supported: -95^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:2281:dpdk_setup_eth_tx():Mapped 0/16 TX counters^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:2315:dpdk_setup_eth_rx():Mapping per RX queue statistics not supported: -95^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:2319:dpdk_setup_eth_rx():Mapped 0/16 RX counters^M^M
[2025-04-18 07:37:02]:: --[vmxnet3_dev_start:963] hw->version: 6, rss_hf: 3380 --^M^M
[2025-04-18 07:37:02]:: --[vmxnet3_rss_configure:1430] rss_hf: 3380 --^M^M
[2025-04-18 07:37:02]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M
[2025-04-18 07:37:02]:: odp_packet_io.c:723:odp_pktio_start():interface: 0, input queues: 2, output queues: 1^M^M
[2025-04-18 07:37:03]:: pktio port 0 dpdk port 0 started!^M^M
So the next step we need to modify our codes, it will not config the RSS when the device RX queue number is 1
Thank you for your advise again and best wish to you!
From: Knight, Joshua <Joshua.Knight@netscout.com>
Sent: Wednesday, April 16, 2025 9:59 PM
To: Fangyin Hu <FHu@SonicWall.com>; users@dpdk.org
Subject: Re: VMware run the DPDK with vmxnet3 NIC failed with the RSS config
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We’ve hit the same problem setting up RSS on vmxnet3 on newer esxi8 releases. In our case we would see the failure if we only had a single rx queue.
Snippet from my discussion with their support team:
> So to summarize, with VM Hardware version 14 in esxi 7 and 6.7, setting the RSS hash function did not generate an error in the driver. With VM Hardware version 14 in esxi 8U2, it generates an error in the driver.
I had a ticket opened with Broadcom and their response was they made a change in behavior where instead of silently rejecting the RSS configuration (since, really you don’t need RSS with a single queue although we still wanted it enabled) the nic now returns an error. The biggest problem I had with the change is they applied it to *earlier* vm hw versions, not just the latest/newest version.
Broadcom’s response to my ticket was:
>Regarding the difference in behavior you have observed with ESXi versions, there were some enhancements introduced in vSphere version 7.0U1 used along with VM hardware version >=14, it identifies the RSS misconfiguration with Rx Queue and throws an error message similar to what has been observed. In older versions, it was not erroring out so you will not observe any failures in the applications.
From: Fangyin Hu <FHu@SonicWall.com<mailto:FHu@SonicWall.com>>
Date: Wednesday, April 16, 2025 at 4:23 AM
To: users@dpdk.org<mailto:users@dpdk.org> <users@dpdk.org<mailto:users@dpdk.org>>
Subject: VMware run the DPDK with vmxnet3 NIC failed with the RSS config
Hi
My application developed with the DPDK, we found out one error message when my application run in the VMware platform.
[2025-04-15 03:01:09]:: --[vmxnet3_dev_start:961] hw->version: 6, rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_rss_configure:1434] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: vmxnet3_v4_rss_configure(): Set RSS fields (v4) failed: 1^M^M
[2025-04-15 03:01:09]:: vmxnet3_dev_start(): Failed to configure v4 RSS^M^M
From the debug logs, it seems that the vmxnet3 device failed to configure the v4 RSS.
We can found these informations from the DPDK’s code snippet:
rss_hf = 3380 = 0xd34
So the RSS configure is this:
#define VMXNET3_RSS_OFFLOAD_ALL ( \
RTE_ETH_RSS_IPV4 | \
RTE_ETH_RSS_NONFRAG_IPV4_TCP | \
RTE_ETH_RSS_IPV6 | \
RTE_ETH_RSS_NONFRAG_IPV6_TCP)
#define VMXNET3_V4_RSS_MASK ( \
RTE_ETH_RSS_NONFRAG_IPV4_UDP | \
RTE_ETH_RSS_NONFRAG_IPV6_UDP)
The function call vmxnet3_rss_configure() is works fine but the function call vmxnet3_v4_rss_configure() failed:
if (VMXNET3_VERSION_GE_4(hw) &&
dev->data->dev_conf.rxmode.mq_mode == RTE_ETH_MQ_RX_RSS) {
/* Check for additional RSS */
ret = vmxnet3_v4_rss_configure(dev);
if (ret != VMXNET3_SUCCESS) {
PMD_INIT_LOG(ERR, "Failed to configure v4 RSS");
return ret;
}
}
/*
* Additional RSS configurations based on vmxnet v4+ APIs
*/
int
vmxnet3_v4_rss_configure(struct rte_eth_dev *dev)
{
struct vmxnet3_hw *hw = dev->data->dev_private;
Vmxnet3_DriverShared *shared = hw->shared;
Vmxnet3_CmdInfo *cmdInfo = &shared->cu.cmdInfo;
struct rte_eth_rss_conf *port_rss_conf;
uint64_t rss_hf;
uint32_t ret;
PMD_INIT_FUNC_TRACE();
cmdInfo->setRSSFields = 0;
port_rss_conf = &dev->data->dev_conf.rx_adv_conf.rss_conf;
printf("--[%s:%d] rss_hf: %llu --\n", __FUNCTION__, __LINE__, port_rss_conf->rss_hf);
if ((port_rss_conf->rss_hf & VMXNET3_MANDATORY_V4_RSS) !=
VMXNET3_MANDATORY_V4_RSS) {
PMD_INIT_LOG(WARNING, "RSS: IPv4/6 TCP is required for vmxnet3 v4 RSS,"
"automatically setting it");
port_rss_conf->rss_hf |= VMXNET3_MANDATORY_V4_RSS;
}
rss_hf = port_rss_conf->rss_hf &
(VMXNET3_V4_RSS_MASK | VMXNET3_RSS_OFFLOAD_ALL);
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_TCP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP4;
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_TCP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP6;
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_UDP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP4;
if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_UDP)
cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP6;
VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
VMXNET3_CMD_SET_RSS_FIELDS);
ret = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD);
if (ret != VMXNET3_SUCCESS) {
PMD_DRV_LOG(ERR, "Set RSS fields (v4) failed: %d", ret);
}
return ret;
}
It seems like the VMXNET3_WRITE_BAR1_REG failed, but don’t know why this happened.
My DPDK version is 22.11.1 and the vmxnet3 driver version is 1.6.0.0-k-NAPI
BTW we just upgraded VM compatibility to ESXi 8.0 U2 and later (VM version 21), then our application run with this error logs.
The previous VM compatibility is ESXi 6.5 and later (VM version 13), and our application works fine. It is because the vmxnet3 hw version is 3, so it will not trigger to call the V4 RSS configure.
Could someone help to check this issue? Or maybe I missed something?
[-- Attachment #2: Type: text/html, Size: 27237 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: VMware run the DPDK with vmxnet3 NIC failed with the RSS config
2025-04-18 7:49 ` Fangyin Hu
@ 2025-04-22 12:49 ` spyroot
0 siblings, 0 replies; 4+ messages in thread
From: spyroot @ 2025-04-22 12:49 UTC (permalink / raw)
To: Fangyin Hu; +Cc: Knight, Joshua, users
[-- Attachment #1: Type: text/plain, Size: 9931 bytes --]
I hit the same issue, and it was brute force. The first version of VMXNET3
is something that
has never been exposed directly. (It is implicit and based on the VM
Hardware version.A)
You can adjust it in the VM advanced settings. It makes sense to indicate
exactly which flag must be in the VM.
(Why? Sometimes, downgrading the VM Hardware version is not an option. )
While downgrading, the VMXNET3 version will run in compatibility mode.
(I.e., DPDK version old, nic new, RSS must X, etc, etc) There are a few
combinations where
You can not assume a single queue, or take a four-queue, or assume a
32-queue. (This combination depends on
Combination
NIC - VM - ESXi - DPDK version
So, asking a client to do four variations and find a combination is not a
good option.
(Note in kernel driver it's the same, you can use 32, but you have no idea
how to enable the VMXNET3 version that will allow you to use 32 queues)
Example.
You have 810 - VMXNET 3 poll mode and enable 16 queues. What do you do?
You have 810 - VMXNET 3 (the default queue in RSS 4). What do you do? ---
This will fail on 23/24/25.
You have 801 - VMXNET 3 - ESXi 8.0.2 and DPDK 22.xx app ( example people
use trex it build 22 or 23)
I just gave a few examples.
Kind Regards,
Mus
On Sat, Apr 19, 2025 at 11:15 AM Fangyin Hu <FHu@sonicwall.com> wrote:
> Hi, Knight
>
> Thank you very much for your reply!
>
> I check our codes, my application config the 1 RX queue for vmxnet3
> device, so the call vmxnet3_v4_rss_configure() will failed due to the
> single RX queue.
>
> I adjusted our codes and use 2 RX queue for vmxnet3 device, then retest my
> application aga in. Now my application works fine and here are some
> debug logs.
>
>
>
> [2025-04-18 07:37:02]:: Try to configure port 0, *RX Queue Number: 2*, TX
> Queue Number: 1 Dev 0^M^M
>
> [2025-04-18 07:37:02]:: pktio/dpdk.c:310:init_options():DPDK interface
> (net_vmxnet3): 0^M^M
>
> [2025-04-18 07:37:02]:: pktio/dpdk.c:312:init_options(): multicast_en:
> 1^M^M
>
> [2025-04-18 07:37:02]:: pktio/dpdk.c:313:init_options(): num_rx_desc:
> 4096^M^M
>
> [2025-04-18 07:37:02]:: pktio/dpdk.c:314:init_options(): num_tx_desc:
> 512^M^M
>
> [2025-04-18 07:37:02]:: pktio/dpdk.c:315:init_options(): rx_drop_en: 0^M^M
>
> [2025-04-18 07:37:02]:: odp_packet_io.c:439:odp_pktio_open():interface: 0,
> driver: dpdk^M^M
>
> [2025-04-18 07:37:02]:: created pktio 1 idx( 0) (0)^M^M
>
>
>
> [2025-04-18 07:37:02]:: ### starting data port pktio's...^M^M
>
> [2025-04-18 07:37:02]:: pktio/dpdk.c:2277:dpdk_setup_eth_tx():Mapping per
> TX queue statistics not supported: -95^M^M
>
> [2025-04-18 07:37:02]:: pktio/dpdk.c:2281:dpdk_setup_eth_tx():Mapped 0/16
> TX counters^M^M
>
> [2025-04-18 07:37:02]:: pktio/dpdk.c:2315:dpdk_setup_eth_rx():Mapping per
> RX queue statistics not supported: -95^M^M
>
> [2025-04-18 07:37:02]:: pktio/dpdk.c:2319:dpdk_setup_eth_rx():Mapped 0/16
> RX counters^M^M
>
> [2025-04-18 07:37:02]:: --[vmxnet3_dev_start:963] hw->version: 6, rss_hf:
> 3380 --^M^M
>
> [2025-04-18 07:37:02]:: --[vmxnet3_rss_configure:1430] rss_hf: 3380 --^M^M
>
> [2025-04-18 07:37:02]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380
> --^M^M
>
> [2025-04-18 07:37:02]:: odp_packet_io.c:723:odp_pktio_start():interface:
> 0, input queues: 2, output queues: 1^M^M
>
> [2025-04-18 07:37:03]:: pktio port 0 dpdk port 0 started!^M^M
>
>
>
> So the next step we need to modify our codes, it will not config the RSS
> when the device RX queue number is 1
>
>
>
> Thank you for your advise again and best wish to you!
>
>
>
> *From:* Knight, Joshua <Joshua.Knight@netscout.com>
> *Sent:* Wednesday, April 16, 2025 9:59 PM
> *To:* Fangyin Hu <FHu@SonicWall.com>; users@dpdk.org
> *Subject:* Re: VMware run the DPDK with vmxnet3 NIC failed with the RSS
> config
>
>
>
> *CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.*
>
>
>
> We’ve hit the same problem setting up RSS on vmxnet3 on newer esxi8
> releases. In our case we would see the failure if we only had a single rx
> queue.
>
> Snippet from my discussion with their support team:
> > So to summarize, with VM Hardware version 14 in esxi 7 and 6.7, setting
> the RSS hash function did not generate an error in the driver. With VM
> Hardware version 14 in esxi 8U2, it generates an error in the driver.
>
>
>
> I had a ticket opened with Broadcom and their response was they made a
> change in behavior where instead of silently rejecting the RSS
> configuration (since, really you don’t need RSS with a single queue
> although we still wanted it enabled) the nic now returns an error. The
> biggest problem I had with the change is they applied it to **earlier**
> vm hw versions, not just the latest/newest version.
>
>
>
> Broadcom’s response to my ticket was:
>
> >Regarding the difference in behavior you have observed with ESXi
> versions, there were some enhancements introduced in vSphere version 7.0U1
> used along with VM hardware version >=14, it identifies the RSS
> misconfiguration with Rx Queue and throws an error message similar to what
> has been observed. In older versions, it was not erroring out so you will
> not observe any failures in the applications.
>
>
>
>
>
>
>
> *From: *Fangyin Hu <FHu@SonicWall.com>
> *Date: *Wednesday, April 16, 2025 at 4:23 AM
> *To: *users@dpdk.org <users@dpdk.org>
> *Subject: *VMware run the DPDK with vmxnet3 NIC failed with the RSS config
>
> Hi
>
>
>
> My application developed with the DPDK, we found out one error message
> when my application run in the VMware platform.
>
>
>
> [2025-04-15 03:01:09]:: --[vmxnet3_dev_start:961] hw->version: 6, rss_hf:
> 3380 --^M^M
>
> [2025-04-15 03:01:09]:: --[vmxnet3_rss_configure:1434] rss_hf: 3380 --^M^M
>
> [2025-04-15 03:01:09]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380
> --^M^M
>
> [2025-04-15 03:01:09]:: vmxnet3_v4_rss_configure(): Set RSS fields (v4)
> failed: 1^M^M
>
> [2025-04-15 03:01:09]:: vmxnet3_dev_start(): Failed to configure v4 RSS^M^M
>
>
>
> From the debug logs, it seems that the vmxnet3 device failed to configure
> the v4 RSS.
>
>
>
> We can found these informations from the DPDK’s code snippet:
>
>
>
> rss_hf = 3380 = 0xd34
>
>
>
> So the RSS configure is this:
>
>
>
> #define VMXNET3_RSS_OFFLOAD_ALL ( \
>
> RTE_ETH_RSS_IPV4 | \
>
> RTE_ETH_RSS_NONFRAG_IPV4_TCP | \
>
> RTE_ETH_RSS_IPV6 | \
>
> RTE_ETH_RSS_NONFRAG_IPV6_TCP)
>
>
>
> #define VMXNET3_V4_RSS_MASK ( \
>
> RTE_ETH_RSS_NONFRAG_IPV4_UDP | \
>
> RTE_ETH_RSS_NONFRAG_IPV6_UDP)
>
>
>
> The function call vmxnet3_rss_configure() is works fine but the function
> call vmxnet3_v4_rss_configure() failed:
>
>
>
> if (VMXNET3_VERSION_GE_4(hw) &&
>
> dev->data->dev_conf.rxmode.mq_mode == RTE_ETH_MQ_RX_RSS) {
>
> /* Check for additional RSS */
>
> ret = vmxnet3_v4_rss_configure(dev);
>
> if (ret != VMXNET3_SUCCESS) {
>
> PMD_INIT_LOG(ERR, "Failed to configure v4 RSS");
>
> return ret;
>
> }
>
> }
>
>
>
> /*
>
> * Additional RSS configurations based on vmxnet v4+ APIs
>
> */
>
> int
>
> vmxnet3_v4_rss_configure(struct rte_eth_dev *dev)
>
> {
>
> struct vmxnet3_hw *hw = dev->data->dev_private;
>
> Vmxnet3_DriverShared *shared = hw->shared;
>
> Vmxnet3_CmdInfo *cmdInfo = &shared->cu.cmdInfo;
>
> struct rte_eth_rss_conf *port_rss_conf;
>
> uint64_t rss_hf;
>
> uint32_t ret;
>
>
>
> PMD_INIT_FUNC_TRACE();
>
>
>
> cmdInfo->setRSSFields = 0;
>
> port_rss_conf = &dev->data->dev_conf.rx_adv_conf.rss_conf;
>
>
>
> printf("--[%s:%d] rss_hf: %llu --\n", __FUNCTION__, __LINE__,
> port_rss_conf->rss_hf);
>
>
>
> if ((port_rss_conf->rss_hf & VMXNET3_MANDATORY_V4_RSS) !=
>
> VMXNET3_MANDATORY_V4_RSS) {
>
> PMD_INIT_LOG(WARNING, "RSS: IPv4/6 TCP is required for vmxnet3 v4
> RSS,"
>
> "automatically setting it");
>
> port_rss_conf->rss_hf |= VMXNET3_MANDATORY_V4_RSS;
>
> }
>
>
>
> rss_hf = port_rss_conf->rss_hf &
>
> (VMXNET3_V4_RSS_MASK | VMXNET3_RSS_OFFLOAD_ALL);
>
>
>
> if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_TCP)
>
> cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP4;
>
> if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_TCP)
>
> cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP6;
>
> if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_UDP)
>
> cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP4;
>
> if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_UDP)
>
> cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP6;
>
>
>
> VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
>
> VMXNET3_CMD_SET_RSS_FIELDS);
>
> ret = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD);
>
>
>
> if (ret != VMXNET3_SUCCESS) {
>
> PMD_DRV_LOG(ERR, "Set RSS fields (v4) failed: %d", ret);
>
> }
>
>
>
> return ret;
>
> }
>
>
>
> It seems like the VMXNET3_WRITE_BAR1_REG failed, but don’t know why this
> happened.
>
>
>
> My DPDK version is 22.11.1 and the vmxnet3 driver version is 1.6.0.0-k-NAPI
>
>
>
> BTW we just upgraded VM compatibility to *ESXi 8.0 U2 and later (VM
> version 21), *then our application run with this error logs.
>
>
>
> The previous VM compatibility is *ESXi 6.5 and later (VM version 13),*
> and our application works fine. It is because the vmxnet3 hw version is 3,
> so it will not trigger to call the V4 RSS configure.
>
>
>
> Could someone help to check this issue? Or maybe I missed something?
>
[-- Attachment #2: Type: text/html, Size: 26444 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-04-22 12:50 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-04-16 8:23 VMware run the DPDK with vmxnet3 NIC failed with the RSS config Fangyin Hu
2025-04-16 13:59 ` Knight, Joshua
2025-04-18 7:49 ` Fangyin Hu
2025-04-22 12:49 ` spyroot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).