VMware run the DPDK with vmxnet3 NIC failed with the RSS config

VMware run the DPDK with vmxnet3 NIC failed with the RSS config

[Fangyin Hu]

[-- Attachment #1: Type: text/plain, Size: 3735 bytes --]

Hi

My application developed with the DPDK, we found out one error message when my application run in the VMware platform.

[2025-04-15 03:01:09]:: --[vmxnet3_dev_start:961] hw->version: 6, rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_rss_configure:1434] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: vmxnet3_v4_rss_configure(): Set RSS fields (v4) failed: 1^M^M
[2025-04-15 03:01:09]:: vmxnet3_dev_start(): Failed to configure v4 RSS^M^M

From the debug logs, it seems that the vmxnet3 device failed to configure the v4 RSS.

We can found these informations from the DPDK's code snippet:

rss_hf = 3380 = 0xd34

So the RSS configure is this:

#define VMXNET3_RSS_OFFLOAD_ALL ( \
    RTE_ETH_RSS_IPV4 | \
    RTE_ETH_RSS_NONFRAG_IPV4_TCP | \
    RTE_ETH_RSS_IPV6 | \
    RTE_ETH_RSS_NONFRAG_IPV6_TCP)

#define VMXNET3_V4_RSS_MASK ( \
    RTE_ETH_RSS_NONFRAG_IPV4_UDP | \
RTE_ETH_RSS_NONFRAG_IPV6_UDP)

The function call vmxnet3_rss_configure() is works fine but the function call vmxnet3_v4_rss_configure() failed:

    if (VMXNET3_VERSION_GE_4(hw) &&
        dev->data->dev_conf.rxmode.mq_mode == RTE_ETH_MQ_RX_RSS) {
        /* Check for additional RSS  */
        ret = vmxnet3_v4_rss_configure(dev);
        if (ret != VMXNET3_SUCCESS) {
            PMD_INIT_LOG(ERR, "Failed to configure v4 RSS");
            return ret;
        }
}

/*
* Additional RSS configurations based on vmxnet v4+ APIs
*/
int
vmxnet3_v4_rss_configure(struct rte_eth_dev *dev)
{
    struct vmxnet3_hw *hw = dev->data->dev_private;
    Vmxnet3_DriverShared *shared = hw->shared;
    Vmxnet3_CmdInfo *cmdInfo = &shared->cu.cmdInfo;
    struct rte_eth_rss_conf *port_rss_conf;
    uint64_t rss_hf;
    uint32_t ret;

    PMD_INIT_FUNC_TRACE();

    cmdInfo->setRSSFields = 0;
    port_rss_conf = &dev->data->dev_conf.rx_adv_conf.rss_conf;

    printf("--[%s:%d] rss_hf: %llu --\n", __FUNCTION__, __LINE__, port_rss_conf->rss_hf);

    if ((port_rss_conf->rss_hf & VMXNET3_MANDATORY_V4_RSS) !=
        VMXNET3_MANDATORY_V4_RSS) {
        PMD_INIT_LOG(WARNING, "RSS: IPv4/6 TCP is required for vmxnet3 v4 RSS,"
                 "automatically setting it");
        port_rss_conf->rss_hf |= VMXNET3_MANDATORY_V4_RSS;
    }

    rss_hf = port_rss_conf->rss_hf &
        (VMXNET3_V4_RSS_MASK | VMXNET3_RSS_OFFLOAD_ALL);

    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_TCP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP4;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_TCP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP6;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_UDP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP4;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_UDP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP6;

    VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
                   VMXNET3_CMD_SET_RSS_FIELDS);
    ret = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD);

    if (ret != VMXNET3_SUCCESS) {
        PMD_DRV_LOG(ERR, "Set RSS fields (v4) failed: %d", ret);
    }

return ret;
}

It seems like the VMXNET3_WRITE_BAR1_REG failed, but don't know why this happened.

My DPDK version is 22.11.1 and the vmxnet3 driver version is 1.6.0.0-k-NAPI

BTW we just upgraded VM compatibility to ESXi 8.0 U2 and later (VM version 21), then our application run with this error logs.

The previous VM compatibility is ESXi 6.5 and later (VM version 13), and our application works fine. It is because the vmxnet3 hw version is 3, so it will not trigger to call the V4 RSS configure.

Could someone help to check this issue? Or maybe I missed something?

[-- Attachment #2: Type: text/html, Size: 14272 bytes --]

Re: VMware run the DPDK with vmxnet3 NIC failed with the RSS config

[Knight, Joshua]

[-- Attachment #1: Type: text/plain, Size: 5211 bytes --]

We’ve hit the same problem setting up RSS on vmxnet3 on newer esxi8 releases.  In our case we would see the failure if we only had a single rx queue.

Snippet from my discussion with their support team:
> So to summarize, with VM Hardware version 14 in esxi 7 and 6.7, setting the RSS hash function did not generate an error in the driver.  With VM Hardware version 14 in esxi 8U2, it generates an error in the driver.

I had a ticket opened with Broadcom and their response was they made a change in behavior where instead of silently rejecting the RSS configuration (since, really you don’t need RSS with a single queue although we still wanted it enabled) the nic now returns an error. The biggest problem I had with the change is they applied it to *earlier* vm hw versions, not just the latest/newest version.

Broadcom’s response to my ticket was:

>Regarding the difference in behavior you have observed with ESXi versions, there were some enhancements introduced in vSphere version 7.0U1 used along with VM hardware version >=14, it identifies the RSS misconfiguration with Rx Queue and throws an error message similar to what has been observed. In older versions, it was not erroring out so you will not observe any failures in the applications.



From: Fangyin Hu <FHu@SonicWall.com>
Date: Wednesday, April 16, 2025 at 4:23 AM
To: users@dpdk.org <users@dpdk.org>
Subject: VMware run the DPDK with vmxnet3 NIC failed with the RSS config
Hi

My application developed with the DPDK, we found out one error message when my application run in the VMware platform.

[2025-04-15 03:01:09]:: --[vmxnet3_dev_start:961] hw->version: 6, rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_rss_configure:1434] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: vmxnet3_v4_rss_configure(): Set RSS fields (v4) failed: 1^M^M
[2025-04-15 03:01:09]:: vmxnet3_dev_start(): Failed to configure v4 RSS^M^M

From the debug logs, it seems that the vmxnet3 device failed to configure the v4 RSS.

We can found these informations from the DPDK’s code snippet:

rss_hf = 3380 = 0xd34

So the RSS configure is this:

#define VMXNET3_RSS_OFFLOAD_ALL ( \
    RTE_ETH_RSS_IPV4 | \
    RTE_ETH_RSS_NONFRAG_IPV4_TCP | \
    RTE_ETH_RSS_IPV6 | \
    RTE_ETH_RSS_NONFRAG_IPV6_TCP)

#define VMXNET3_V4_RSS_MASK ( \
    RTE_ETH_RSS_NONFRAG_IPV4_UDP | \
RTE_ETH_RSS_NONFRAG_IPV6_UDP)

The function call vmxnet3_rss_configure() is works fine but the function call vmxnet3_v4_rss_configure() failed:

    if (VMXNET3_VERSION_GE_4(hw) &&
        dev->data->dev_conf.rxmode.mq_mode == RTE_ETH_MQ_RX_RSS) {
        /* Check for additional RSS  */
        ret = vmxnet3_v4_rss_configure(dev);
        if (ret != VMXNET3_SUCCESS) {
            PMD_INIT_LOG(ERR, "Failed to configure v4 RSS");
            return ret;
        }
}

/*
* Additional RSS configurations based on vmxnet v4+ APIs
*/
int
vmxnet3_v4_rss_configure(struct rte_eth_dev *dev)
{
    struct vmxnet3_hw *hw = dev->data->dev_private;
    Vmxnet3_DriverShared *shared = hw->shared;
    Vmxnet3_CmdInfo *cmdInfo = &shared->cu.cmdInfo;
    struct rte_eth_rss_conf *port_rss_conf;
    uint64_t rss_hf;
    uint32_t ret;

    PMD_INIT_FUNC_TRACE();

    cmdInfo->setRSSFields = 0;
    port_rss_conf = &dev->data->dev_conf.rx_adv_conf.rss_conf;

    printf("--[%s:%d] rss_hf: %llu --\n", __FUNCTION__, __LINE__, port_rss_conf->rss_hf);

    if ((port_rss_conf->rss_hf & VMXNET3_MANDATORY_V4_RSS) !=
        VMXNET3_MANDATORY_V4_RSS) {
        PMD_INIT_LOG(WARNING, "RSS: IPv4/6 TCP is required for vmxnet3 v4 RSS,"
                 "automatically setting it");
        port_rss_conf->rss_hf |= VMXNET3_MANDATORY_V4_RSS;
    }

    rss_hf = port_rss_conf->rss_hf &
        (VMXNET3_V4_RSS_MASK | VMXNET3_RSS_OFFLOAD_ALL);

    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_TCP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP4;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_TCP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP6;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_UDP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP4;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_UDP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP6;

    VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
                   VMXNET3_CMD_SET_RSS_FIELDS);
    ret = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD);

    if (ret != VMXNET3_SUCCESS) {
        PMD_DRV_LOG(ERR, "Set RSS fields (v4) failed: %d", ret);
    }

return ret;
}

It seems like the VMXNET3_WRITE_BAR1_REG failed, but don’t know why this happened.

My DPDK version is 22.11.1 and the vmxnet3 driver version is 1.6.0.0-k-NAPI

BTW we just upgraded VM compatibility to ESXi 8.0 U2 and later (VM version 21), then our application run with this error logs.

The previous VM compatibility is ESXi 6.5 and later (VM version 13), and our application works fine. It is because the vmxnet3 hw version is 3, so it will not trigger to call the V4 RSS configure.

Could someone help to check this issue? Or maybe I missed something?

[-- Attachment #2: Type: text/html, Size: 20604 bytes --]

RE: VMware run the DPDK with vmxnet3 NIC failed with the RSS config

[Fangyin Hu]

[-- Attachment #1: Type: text/plain, Size: 7780 bytes --]

Hi, Knight
Thank you very much for your reply!
I check our codes, my application config the 1 RX queue for vmxnet3 device, so the call vmxnet3_v4_rss_configure() will failed due to the single RX queue.
I adjusted our codes and use 2 RX queue for vmxnet3 device, then retest my application aga        in. Now my application works fine and here are some debug logs.

[2025-04-18 07:37:02]:: Try to configure port 0, RX Queue Number: 2, TX Queue Number: 1 Dev 0^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:310:init_options():DPDK interface (net_vmxnet3): 0^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:312:init_options():  multicast_en: 1^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:313:init_options():  num_rx_desc: 4096^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:314:init_options():  num_tx_desc: 512^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:315:init_options():  rx_drop_en: 0^M^M
[2025-04-18 07:37:02]:: odp_packet_io.c:439:odp_pktio_open():interface: 0, driver: dpdk^M^M
[2025-04-18 07:37:02]:: created pktio 1 idx( 0) (0)^M^M

[2025-04-18 07:37:02]:: ### starting data port pktio's...^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:2277:dpdk_setup_eth_tx():Mapping per TX queue statistics not supported: -95^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:2281:dpdk_setup_eth_tx():Mapped 0/16 TX counters^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:2315:dpdk_setup_eth_rx():Mapping per RX queue statistics not supported: -95^M^M
[2025-04-18 07:37:02]:: pktio/dpdk.c:2319:dpdk_setup_eth_rx():Mapped 0/16 RX counters^M^M
[2025-04-18 07:37:02]:: --[vmxnet3_dev_start:963] hw->version: 6, rss_hf: 3380 --^M^M
[2025-04-18 07:37:02]:: --[vmxnet3_rss_configure:1430] rss_hf: 3380 --^M^M
[2025-04-18 07:37:02]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M
[2025-04-18 07:37:02]:: odp_packet_io.c:723:odp_pktio_start():interface: 0, input queues: 2, output queues: 1^M^M
[2025-04-18 07:37:03]:: pktio port 0 dpdk port 0 started!^M^M

So the next step we need to modify our codes, it will not config the RSS when the device RX queue number is 1

Thank you for your advise again and best wish to you!

From: Knight, Joshua <Joshua.Knight@netscout.com>
Sent: Wednesday, April 16, 2025 9:59 PM
To: Fangyin Hu <FHu@SonicWall.com>; users@dpdk.org
Subject: Re: VMware run the DPDK with vmxnet3 NIC failed with the RSS config

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

We’ve hit the same problem setting up RSS on vmxnet3 on newer esxi8 releases.  In our case we would see the failure if we only had a single rx queue.

Snippet from my discussion with their support team:
> So to summarize, with VM Hardware version 14 in esxi 7 and 6.7, setting the RSS hash function did not generate an error in the driver.  With VM Hardware version 14 in esxi 8U2, it generates an error in the driver.

I had a ticket opened with Broadcom and their response was they made a change in behavior where instead of silently rejecting the RSS configuration (since, really you don’t need RSS with a single queue although we still wanted it enabled) the nic now returns an error. The biggest problem I had with the change is they applied it to *earlier* vm hw versions, not just the latest/newest version.

Broadcom’s response to my ticket was:

>Regarding the difference in behavior you have observed with ESXi versions, there were some enhancements introduced in vSphere version 7.0U1 used along with VM hardware version >=14, it identifies the RSS misconfiguration with Rx Queue and throws an error message similar to what has been observed. In older versions, it was not erroring out so you will not observe any failures in the applications.



From: Fangyin Hu <FHu@SonicWall.com<mailto:FHu@SonicWall.com>>
Date: Wednesday, April 16, 2025 at 4:23 AM
To: users@dpdk.org<mailto:users@dpdk.org> <users@dpdk.org<mailto:users@dpdk.org>>
Subject: VMware run the DPDK with vmxnet3 NIC failed with the RSS config
Hi

My application developed with the DPDK, we found out one error message when my application run in the VMware platform.

[2025-04-15 03:01:09]:: --[vmxnet3_dev_start:961] hw->version: 6, rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_rss_configure:1434] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: vmxnet3_v4_rss_configure(): Set RSS fields (v4) failed: 1^M^M
[2025-04-15 03:01:09]:: vmxnet3_dev_start(): Failed to configure v4 RSS^M^M

From the debug logs, it seems that the vmxnet3 device failed to configure the v4 RSS.

We can found these informations from the DPDK’s code snippet:

rss_hf = 3380 = 0xd34

So the RSS configure is this:

#define VMXNET3_RSS_OFFLOAD_ALL ( \
    RTE_ETH_RSS_IPV4 | \
    RTE_ETH_RSS_NONFRAG_IPV4_TCP | \
    RTE_ETH_RSS_IPV6 | \
    RTE_ETH_RSS_NONFRAG_IPV6_TCP)

#define VMXNET3_V4_RSS_MASK ( \
    RTE_ETH_RSS_NONFRAG_IPV4_UDP | \
RTE_ETH_RSS_NONFRAG_IPV6_UDP)

The function call vmxnet3_rss_configure() is works fine but the function call vmxnet3_v4_rss_configure() failed:

    if (VMXNET3_VERSION_GE_4(hw) &&
        dev->data->dev_conf.rxmode.mq_mode == RTE_ETH_MQ_RX_RSS) {
        /* Check for additional RSS  */
        ret = vmxnet3_v4_rss_configure(dev);
        if (ret != VMXNET3_SUCCESS) {
            PMD_INIT_LOG(ERR, "Failed to configure v4 RSS");
            return ret;
        }
}

/*
* Additional RSS configurations based on vmxnet v4+ APIs
*/
int
vmxnet3_v4_rss_configure(struct rte_eth_dev *dev)
{
    struct vmxnet3_hw *hw = dev->data->dev_private;
    Vmxnet3_DriverShared *shared = hw->shared;
    Vmxnet3_CmdInfo *cmdInfo = &shared->cu.cmdInfo;
    struct rte_eth_rss_conf *port_rss_conf;
    uint64_t rss_hf;
    uint32_t ret;

    PMD_INIT_FUNC_TRACE();

    cmdInfo->setRSSFields = 0;
    port_rss_conf = &dev->data->dev_conf.rx_adv_conf.rss_conf;

    printf("--[%s:%d] rss_hf: %llu --\n", __FUNCTION__, __LINE__, port_rss_conf->rss_hf);

    if ((port_rss_conf->rss_hf & VMXNET3_MANDATORY_V4_RSS) !=
        VMXNET3_MANDATORY_V4_RSS) {
        PMD_INIT_LOG(WARNING, "RSS: IPv4/6 TCP is required for vmxnet3 v4 RSS,"
                 "automatically setting it");
        port_rss_conf->rss_hf |= VMXNET3_MANDATORY_V4_RSS;
    }

    rss_hf = port_rss_conf->rss_hf &
        (VMXNET3_V4_RSS_MASK | VMXNET3_RSS_OFFLOAD_ALL);

    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_TCP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP4;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_TCP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP6;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_UDP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP4;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_UDP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP6;

    VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
                   VMXNET3_CMD_SET_RSS_FIELDS);
    ret = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD);

    if (ret != VMXNET3_SUCCESS) {
        PMD_DRV_LOG(ERR, "Set RSS fields (v4) failed: %d", ret);
    }

return ret;
}

It seems like the VMXNET3_WRITE_BAR1_REG failed, but don’t know why this happened.

My DPDK version is 22.11.1 and the vmxnet3 driver version is 1.6.0.0-k-NAPI

BTW we just upgraded VM compatibility to ESXi 8.0 U2 and later (VM version 21), then our application run with this error logs.

The previous VM compatibility is ESXi 6.5 and later (VM version 13), and our application works fine. It is because the vmxnet3 hw version is 3, so it will not trigger to call the V4 RSS configure.

Could someone help to check this issue? Or maybe I missed something?

[-- Attachment #2: Type: text/html, Size: 27237 bytes --]