[dpdk-users] vmxnet3 RSS implementation

[dpdk-users] vmxnet3 RSS implementation

[Jay Miller]

Greetings Programs,

We're having trouble getting RSS distribution working on vmxnet3.  It 
appears that no matter what we do from the hypervisor side that load 
balancing in vmxnet3 is only via mac address. We're not able to get 
distribution across queues based on the 5-tuple hash.

Certainly we don't see this limitation on KVM.  But we did notice that 
DPDK documentation seems to imply this:

>
>     36.2. Features and Limitations of VMXNET3 PMD
>
> In release 1.6.0, the VMXNET3 PMD provides the basic functionality of 
> packet reception and transmission. There are several options available 
> for filtering packets at VMXNET3 device level including:
>
>  1. *MAC Address based filtering*:
>       * Unicast, Broadcast, All Multicast modes - SUPPORTED BY DEFAULT
>       * Multicast with Multicast Filter table - NOT SUPPORTED
>       * Promiscuous mode - SUPPORTED
>       * *RSS based load balancing between queues - SUPPORTED*
>

Is this an expected limitation?  One could argue that it's not true RSS 
support if this is the case.

Thanks,

Jay Miller

-- 
Jay Miller | Tech Lead | Mobile: +1.781.258.7446
Oracle Communications Global Business Unit
100 Crosby Drive | Bedford, MA 01730

[dpdk-users] Support for more RSS hash types in vmxnet3

[Jay Miller]

It's clear that the vmxnet3 driver (even as of 18.08) supports just a 
subset of RSS hash types:

#define VMXNET3_RSS_OFFLOAD_ALL ( \
     ETH_RSS_IPV4 | \
     ETH_RSS_NONFRAG_IPV4_TCP | \
     ETH_RSS_IPV6 | \
     ETH_RSS_NONFRAG_IPV6_TCP)

Are there plans to add support for other hash types (like 
ETH_RSS_NONFRAG_IPV4_UDP), or is this an architectural limitation of 
vmxnet3?

The documentation could be more explicit about these limitations:

>>
>>     36.2. Features and Limitations of VMXNET3 PMD
>>
>> In release 1.6.0, the VMXNET3 PMD provides the basic functionality of 
>> packet reception and transmission. There are several options 
>> available for filtering packets at VMXNET3 device level including:
>>
>>  1. *MAC Address based filtering*:
>>       * Unicast, Broadcast, All Multicast modes - SUPPORTED BY DEFAULT
>>       * Multicast with Multicast Filter table - NOT SUPPORTED
>>       * Promiscuous mode - SUPPORTED
>>       * *RSS based load balancing between queues - SUPPORTED*
>>
>

Thanks,
J

-- 
Jay Miller | Mobile: +1.781.258.7446
Oracle Communications Global Business Unit
100 Crosby Drive | Bedford, MA 01730

Re: [dpdk-users] Support for more RSS hash types in vmxnet3

[Stephen Hemminger]

On Tue, 21 Aug 2018 13:57:08 -0400
Jay Miller <jay.miller@oracle.com> wrote:

> It's clear that the vmxnet3 driver (even as of 18.08) supports just a 
> subset of RSS hash types:
> 
> #define VMXNET3_RSS_OFFLOAD_ALL ( \
>      ETH_RSS_IPV4 | \
>      ETH_RSS_NONFRAG_IPV4_TCP | \
>      ETH_RSS_IPV6 | \
>      ETH_RSS_NONFRAG_IPV6_TCP)
> 
> Are there plans to add support for other hash types (like 
> ETH_RSS_NONFRAG_IPV4_UDP), or is this an architectural limitation of 
> vmxnet3?

Did you test, I suspect that VMWare supports that already.
The values in the API to the host are:

/* value of RxCompDesc.rssType */
enum {
	VMXNET3_RCD_RSS_TYPE_NONE     = 0,
	VMXNET3_RCD_RSS_TYPE_IPV4     = 1,
	VMXNET3_RCD_RSS_TYPE_TCPIPV4  = 2,
	VMXNET3_RCD_RSS_TYPE_IPV6     = 3,
	VMXNET3_RCD_RSS_TYPE_TCPIPV6  = 4,
};


> The documentation could be more explicit about these limitations:
> 
> >>
> >>     36.2. Features and Limitations of VMXNET3 PMD
> >>
> >> In release 1.6.0, the VMXNET3 PMD provides the basic functionality of 
> >> packet reception and transmission. There are several options 
> >> available for filtering packets at VMXNET3 device level including:
> >>
> >>  1. *MAC Address based filtering*:
> >>       * Unicast, Broadcast, All Multicast modes - SUPPORTED BY DEFAULT
> >>       * Multicast with Multicast Filter table - NOT SUPPORTED
> >>       * Promiscuous mode - SUPPORTED
> >>       * *RSS based load balancing between queues - SUPPORTED*
> >>  
> >  
> 
> Thanks,
> J
> 

Re: [dpdk-users] Support for more RSS hash types in vmxnet3

[Ferruh Yigit]

On 8/21/2018 6:57 PM, Jay Miller wrote:
> It's clear that the vmxnet3 driver (even as of 18.08) supports just a 
> subset of RSS hash types:
> 
> #define VMXNET3_RSS_OFFLOAD_ALL ( \
>      ETH_RSS_IPV4 | \
>      ETH_RSS_NONFRAG_IPV4_TCP | \
>      ETH_RSS_IPV6 | \
>      ETH_RSS_NONFRAG_IPV6_TCP)
> 
> Are there plans to add support for other hash types (like 
> ETH_RSS_NONFRAG_IPV4_UDP), or is this an architectural limitation of 
> vmxnet3?

Hi Yong,

Can you please double check if the driver reports all supported hash functions
correctly.

On v18.08, the RSS hf request from application changed from best effort to
strict requirement, meaning if an application request a hash function but driver
doesn't report it as supported API will return an error, that is why it is
important for PMD to report supported hf properly.

Thanks,
ferruh

> 
> The documentation could be more explicit about these limitations:
> 
>>>
>>>     36.2. Features and Limitations of VMXNET3 PMD
>>>
>>> In release 1.6.0, the VMXNET3 PMD provides the basic functionality of 
>>> packet reception and transmission. There are several options 
>>> available for filtering packets at VMXNET3 device level including:
>>>
>>>  1. *MAC Address based filtering*:
>>>       * Unicast, Broadcast, All Multicast modes - SUPPORTED BY DEFAULT
>>>       * Multicast with Multicast Filter table - NOT SUPPORTED
>>>       * Promiscuous mode - SUPPORTED
>>>       * *RSS based load balancing between queues - SUPPORTED*
>>>
>>
> 
> Thanks,
> J
> 

Re: [dpdk-users] Support for more RSS hash types in vmxnet3

[Yong Wang]

-----Original Message-----
From: Ferruh Yigit <ferruh.yigit@intel.com>
Date: Wednesday, August 22, 2018 at 2:55 AM
To: Jay Miller <jay.miller@oracle.com>, "users@dpdk.org" <users@dpdk.org>, Yong Wang <yongwang@vmware.com>
Subject: Re: [dpdk-users] Support for more RSS hash types in vmxnet3

    > On 8/21/2018 6:57 PM, Jay Miller wrote:
    >> It's clear that the vmxnet3 driver (even as of 18.08) supports just a 
    >> subset of RSS hash types:
    >> 
    >> #define VMXNET3_RSS_OFFLOAD_ALL ( \
    >>      ETH_RSS_IPV4 | \
    >>      ETH_RSS_NONFRAG_IPV4_TCP | \
    >>      ETH_RSS_IPV6 | \
    >>      ETH_RSS_NONFRAG_IPV6_TCP)
    >> 
    >> Are there plans to add support for other hash types (like 
    >> ETH_RSS_NONFRAG_IPV4_UDP), or is this an architectural limitation of 
    >> vmxnet3?
    
    > Hi Yong,
    
    > Can you please double check if the driver reports all supported hash functions
    > correctly.
    
    > On v18.08, the RSS hf request from application changed from best effort to
    > strict requirement, meaning if an application request a hash function but driver
    > doesn't report it as supported API will return an error, that is why it is
    > important for PMD to report supported hf properly.
    
    > Thanks,
    > ferruh

That's pretty much all the hash types supported by vmxnet3 by default up to version 3.  With version 4, UDP RSS will be supported but it's only supported on certain version of ESX.  Since v4 driver is not out yet, current VMXNET3_RSS_OFFLOAD_ALL should be good.

Re: [dpdk-users] Support for more RSS hash types in vmxnet3

[Iain Barker]

On 8/21/2018 6:57 PM, Jay Miller wrote:
> It's clear that the vmxnet3 driver (even as of 18.08) supports just a 
> subset of RSS hash types:
> 
> #define VMXNET3_RSS_OFFLOAD_ALL ( \
>      ETH_RSS_IPV4 | \
>      ETH_RSS_NONFRAG_IPV4_TCP | \
>      ETH_RSS_IPV6 | \
>      ETH_RSS_NONFRAG_IPV6_TCP)
> 
> Are there plans to add support for other hash types (like 
> ETH_RSS_NONFRAG_IPV4_UDP), or is this an architectural limitation of 
> vmxnet3?


On August 22, 2018 at 2:55 AM, Ferruh Yigit wrote:
    
> Hi Yong,
>    
> Can you please double check if the driver reports all supported hash functions
> correctly.
>    
> On v18.08, the RSS hf request from application changed from best effort to
> strict requirement, meaning if an application request a hash function but driver
> doesn't report it as supported API will return an error, that is why it is
> important for PMD to report supported hf properly.
>    
> Thanks,
> ferruh


On September 13, 2018 6:44 PM, Yong Wang wrote:

> That's pretty much all the hash types supported by vmxnet3 by default up to version 3.  
> With version 4, UDP RSS will be supported but it's only supported on certain version of ESX.  
> Since v4 driver is not out yet, current VMXNET3_RSS_OFFLOAD_ALL should be good.



Ferruh/Yong,

I notice that ESXi 6.7 has been released for some months now with support for VMXNET3 version 4 including with RSS for UDP
https://docs.vmware.com/en/vSphere/6.7/solutions/vSphere-6.7.2cd6d2a77980cc623caa6062f3c89362/GUID-C500585C0560D28B71180A40A4767C57.html

I'm surprised this wasn't already present in DPDK 19.02 given how long 6.7 has been available already.

Is it just a matter of changing the definition of VMXNET3_RSS_OFFLOAD_ALL  in the PMD to support this, or are other changes required?

thanks.
Iain

Re: [dpdk-users] Support for more RSS hash types in vmxnet3

[Yong Wang]

-----Original Message-----
From: Iain Barker <iain.barker@oracle.com> on behalf of Iain Barker <iain.barker@oracle.com>
Date: Monday, March 18, 2019 at 6:58 AM
To: Yong Wang <yongwang@vmware.com>, Ferruh Yigit <ferruh.yigit@intel.com>, Jay Miller <jay.miller@oracle.com>, "users@dpdk.org" <users@dpdk.org>
Subject: RE: [dpdk-users] Support for more RSS hash types in vmxnet3

    On 8/21/2018 6:57 PM, Jay Miller wrote:
    > It's clear that the vmxnet3 driver (even as of 18.08) supports just a 
    > subset of RSS hash types:
    > 
    > #define VMXNET3_RSS_OFFLOAD_ALL ( \
    >      ETH_RSS_IPV4 | \
    >      ETH_RSS_NONFRAG_IPV4_TCP | \
    >      ETH_RSS_IPV6 | \
    >      ETH_RSS_NONFRAG_IPV6_TCP)
    > 
    > Are there plans to add support for other hash types (like 
    > ETH_RSS_NONFRAG_IPV4_UDP), or is this an architectural limitation of 
    > vmxnet3?
    
    
    On August 22, 2018 at 2:55 AM, Ferruh Yigit wrote:
        
    > Hi Yong,
    >    
    > Can you please double check if the driver reports all supported hash functions
    > correctly.
    >    
    > On v18.08, the RSS hf request from application changed from best effort to
    > strict requirement, meaning if an application request a hash function but driver
    > doesn't report it as supported API will return an error, that is why it is
    > important for PMD to report supported hf properly.
    >    
    > Thanks,
    > ferruh
    
    
    On September 13, 2018 6:44 PM, Yong Wang wrote:
    
    > That's pretty much all the hash types supported by vmxnet3 by default up to version 3.  
    > With version 4, UDP RSS will be supported but it's only supported on certain version of ESX.  
    > Since v4 driver is not out yet, current VMXNET3_RSS_OFFLOAD_ALL should be good.
    
    
    
    Ferruh/Yong,
    
    I notice that ESXi 6.7 has been released for some months now with support for VMXNET3 version 4 including with RSS for UDP
    https://docs.vmware.com/en/vSphere/6.7/solutions/vSphere-6.7.2cd6d2a77980cc623caa6062f3c89362/GUID-C500585C0560D28B71180A40A4767C57.html
    
    I'm surprised this wasn't already present in DPDK 19.02 given how long 6.7 has been available already.
    
    Is it just a matter of changing the definition of VMXNET3_RSS_OFFLOAD_ALL  in the PMD to support this, or are other changes required?
    
    thanks.
    Iain

Iain,

We do plan to upstream the change to support the new RSS types in the near future.  Currently it's still under internal testing.

Thanks,
Yong