Re: VMware run the DPDK with vmxnet3 NIC failed with the RSS config

["Knight, Joshua" <Joshua.Knight@netscout.com>]

[-- Attachment #1: Type: text/plain, Size: 5211 bytes --]

We’ve hit the same problem setting up RSS on vmxnet3 on newer esxi8 releases.  In our case we would see the failure if we only had a single rx queue.

Snippet from my discussion with their support team:
> So to summarize, with VM Hardware version 14 in esxi 7 and 6.7, setting the RSS hash function did not generate an error in the driver.  With VM Hardware version 14 in esxi 8U2, it generates an error in the driver.

I had a ticket opened with Broadcom and their response was they made a change in behavior where instead of silently rejecting the RSS configuration (since, really you don’t need RSS with a single queue although we still wanted it enabled) the nic now returns an error. The biggest problem I had with the change is they applied it to *earlier* vm hw versions, not just the latest/newest version.

Broadcom’s response to my ticket was:

>Regarding the difference in behavior you have observed with ESXi versions, there were some enhancements introduced in vSphere version 7.0U1 used along with VM hardware version >=14, it identifies the RSS misconfiguration with Rx Queue and throws an error message similar to what has been observed. In older versions, it was not erroring out so you will not observe any failures in the applications.



From: Fangyin Hu <FHu@SonicWall.com>
Date: Wednesday, April 16, 2025 at 4:23 AM
To: users@dpdk.org <users@dpdk.org>
Subject: VMware run the DPDK with vmxnet3 NIC failed with the RSS config
Hi

My application developed with the DPDK, we found out one error message when my application run in the VMware platform.

[2025-04-15 03:01:09]:: --[vmxnet3_dev_start:961] hw->version: 6, rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_rss_configure:1434] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: --[vmxnet3_v4_rss_configure:1381] rss_hf: 3380 --^M^M
[2025-04-15 03:01:09]:: vmxnet3_v4_rss_configure(): Set RSS fields (v4) failed: 1^M^M
[2025-04-15 03:01:09]:: vmxnet3_dev_start(): Failed to configure v4 RSS^M^M

From the debug logs, it seems that the vmxnet3 device failed to configure the v4 RSS.

We can found these informations from the DPDK’s code snippet:

rss_hf = 3380 = 0xd34

So the RSS configure is this:

#define VMXNET3_RSS_OFFLOAD_ALL ( \
    RTE_ETH_RSS_IPV4 | \
    RTE_ETH_RSS_NONFRAG_IPV4_TCP | \
    RTE_ETH_RSS_IPV6 | \
    RTE_ETH_RSS_NONFRAG_IPV6_TCP)

#define VMXNET3_V4_RSS_MASK ( \
    RTE_ETH_RSS_NONFRAG_IPV4_UDP | \
RTE_ETH_RSS_NONFRAG_IPV6_UDP)

The function call vmxnet3_rss_configure() is works fine but the function call vmxnet3_v4_rss_configure() failed:

    if (VMXNET3_VERSION_GE_4(hw) &&
        dev->data->dev_conf.rxmode.mq_mode == RTE_ETH_MQ_RX_RSS) {
        /* Check for additional RSS  */
        ret = vmxnet3_v4_rss_configure(dev);
        if (ret != VMXNET3_SUCCESS) {
            PMD_INIT_LOG(ERR, "Failed to configure v4 RSS");
            return ret;
        }
}

/*
* Additional RSS configurations based on vmxnet v4+ APIs
*/
int
vmxnet3_v4_rss_configure(struct rte_eth_dev *dev)
{
    struct vmxnet3_hw *hw = dev->data->dev_private;
    Vmxnet3_DriverShared *shared = hw->shared;
    Vmxnet3_CmdInfo *cmdInfo = &shared->cu.cmdInfo;
    struct rte_eth_rss_conf *port_rss_conf;
    uint64_t rss_hf;
    uint32_t ret;

    PMD_INIT_FUNC_TRACE();

    cmdInfo->setRSSFields = 0;
    port_rss_conf = &dev->data->dev_conf.rx_adv_conf.rss_conf;

    printf("--[%s:%d] rss_hf: %llu --\n", __FUNCTION__, __LINE__, port_rss_conf->rss_hf);

    if ((port_rss_conf->rss_hf & VMXNET3_MANDATORY_V4_RSS) !=
        VMXNET3_MANDATORY_V4_RSS) {
        PMD_INIT_LOG(WARNING, "RSS: IPv4/6 TCP is required for vmxnet3 v4 RSS,"
                 "automatically setting it");
        port_rss_conf->rss_hf |= VMXNET3_MANDATORY_V4_RSS;
    }

    rss_hf = port_rss_conf->rss_hf &
        (VMXNET3_V4_RSS_MASK | VMXNET3_RSS_OFFLOAD_ALL);

    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_TCP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP4;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_TCP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_TCPIP6;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV4_UDP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP4;
    if (rss_hf & RTE_ETH_RSS_NONFRAG_IPV6_UDP)
        cmdInfo->setRSSFields |= VMXNET3_RSS_FIELDS_UDPIP6;

    VMXNET3_WRITE_BAR1_REG(hw, VMXNET3_REG_CMD,
                   VMXNET3_CMD_SET_RSS_FIELDS);
    ret = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_CMD);

    if (ret != VMXNET3_SUCCESS) {
        PMD_DRV_LOG(ERR, "Set RSS fields (v4) failed: %d", ret);
    }

return ret;
}

It seems like the VMXNET3_WRITE_BAR1_REG failed, but don’t know why this happened.

My DPDK version is 22.11.1 and the vmxnet3 driver version is 1.6.0.0-k-NAPI

BTW we just upgraded VM compatibility to ESXi 8.0 U2 and later (VM version 21), then our application run with this error logs.

The previous VM compatibility is ESXi 6.5 and later (VM version 13), and our application works fine. It is because the vmxnet3 hw version is 3, so it will not trigger to call the V4 RSS configure.

Could someone help to check this issue? Or maybe I missed something?

[-- Attachment #2: Type: text/html, Size: 20604 bytes --]