Supporting RSS with DPDK in a VM

Supporting RSS with DPDK in a VM

[Matheus Stolet]

Hello,

I am trying to run a DPDK application with RSS enabled so that I can 
have multiple rx queues. This application is running inside a VM. This 
VM is hosted by QEMU using KVM acceleration and OvS with DPDK and 
vhost-user are used in the backend. So to clarify things there are two 
DPDK portions to this. The first is the DPDK portion used by OvS that 
bypasses the host operating system. This is working fine. The other is a 
DPDK application inside the virtual machine that will bypass the guest 
operating system. This is where I am having trouble with.

When I set rte_eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS in my application 
I get the following errors:
   Warning: NIC does not support all requested RSS hash functions.
   virtio_dev_configure(): RSS support requested but not supported by the 
device
   Port0 dev_configure = -95

I setup my VM in QEMU to have mq=on and queues=10. I also set the number 
of rx_queues when creating the vhost port using ovs to 10. Before 
binding the interface to DPDK, I used ethtool to verify if the network 
interface was actually setup to have multiple queues.

Running the 'ethtool -l enps02' command yields the following output:
   Pre-set maximums:
   RX:             0
   TX:             0
   Other:          0
   Combined:       10
   Current hardware settings:
   RX:             0
   TX:             0
   Other:          0
   Combined:       10

 From my understanding the combined values indicate that the interface 
was properly setup to have multiple queues, so why am I getting the 
unsupported RSS error? Are there other configuration steps that I have 
to take to get this to work? Is RSS with DPDK not supported at all 
inside a VM at the moment? Perhaps the "Port0 dev_configure() = -95" 
error means something else? Without the receive side scaling turned on 
my application is not able to achieve the desired throughput and won't 
scale when I assign more cores to the application.

Versions:
VM
DPDK: 21.11.4
Kernel: 5.4.0-148-generic
Distribution: Ubuntu 20.04

Host
DPDK: 21.11.4
QEMU: 8.0.90
OvS: 3.0.5
Kernel: 5.15.111.1.amd64-smp
Distribution: Debian 11

Re: Supporting RSS with DPDK in a VM

[Thomas Monjalon]

Hello,

You can have packets distributed to multiple queues without RSS.

If you really wants to enable the RSS algorithms,
it seems not supported for now with vhost_user.
It can be enabled with vhost running in the kernel:
https://qemu.readthedocs.io/en/latest/devel/ebpf_rss.html


26/07/2023 00:58, Matheus Stolet:
> Hello,
> 
> I am trying to run a DPDK application with RSS enabled so that I can 
> have multiple rx queues. This application is running inside a VM. This 
> VM is hosted by QEMU using KVM acceleration and OvS with DPDK and 
> vhost-user are used in the backend. So to clarify things there are two 
> DPDK portions to this. The first is the DPDK portion used by OvS that 
> bypasses the host operating system. This is working fine. The other is a 
> DPDK application inside the virtual machine that will bypass the guest 
> operating system. This is where I am having trouble with.
> 
> When I set rte_eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS in my application 
> I get the following errors:
>    Warning: NIC does not support all requested RSS hash functions.
>    virtio_dev_configure(): RSS support requested but not supported by the 
> device
>    Port0 dev_configure = -95
> 
> I setup my VM in QEMU to have mq=on and queues=10. I also set the number 
> of rx_queues when creating the vhost port using ovs to 10. Before 
> binding the interface to DPDK, I used ethtool to verify if the network 
> interface was actually setup to have multiple queues.
> 
> Running the 'ethtool -l enps02' command yields the following output:
>    Pre-set maximums:
>    RX:             0
>    TX:             0
>    Other:          0
>    Combined:       10
>    Current hardware settings:
>    RX:             0
>    TX:             0
>    Other:          0
>    Combined:       10
> 
>  From my understanding the combined values indicate that the interface 
> was properly setup to have multiple queues, so why am I getting the 
> unsupported RSS error? Are there other configuration steps that I have 
> to take to get this to work? Is RSS with DPDK not supported at all 
> inside a VM at the moment? Perhaps the "Port0 dev_configure() = -95" 
> error means something else? Without the receive side scaling turned on 
> my application is not able to achieve the desired throughput and won't 
> scale when I assign more cores to the application.
> 
> Versions:
> VM
> DPDK: 21.11.4
> Kernel: 5.4.0-148-generic
> Distribution: Ubuntu 20.04
> 
> Host
> DPDK: 21.11.4
> QEMU: 8.0.90
> OvS: 3.0.5
> Kernel: 5.15.111.1.amd64-smp
> Distribution: Debian 11
> 

RE: Supporting RSS with DPDK in a VM

[Van Haaren, Harry]

> -----Original Message-----
> From: Matheus Stolet <mstolet@mpi-sws.org>
> Sent: Tuesday, July 25, 2023 11:59 PM
> To: users@dpdk.org
> Subject: Supporting RSS with DPDK in a VM
> 
> Hello,

Hi  Matheus,

> I am trying to run a DPDK application with RSS enabled so that I can
> have multiple rx queues. This application is running inside a VM. This
> VM is hosted by QEMU using KVM acceleration and OvS with DPDK and
> vhost-user are used in the backend. So to clarify things there are two
> DPDK portions to this. The first is the DPDK portion used by OvS that
> bypasses the host operating system. This is working fine. The other is a
> DPDK application inside the virtual machine that will bypass the guest
> operating system. This is where I am having trouble with.
> 
> When I set rte_eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS in my application
> I get the following errors:
>    Warning: NIC does not support all requested RSS hash functions.
>    virtio_dev_configure(): RSS support requested but not supported by the
> device
>    Port0 dev_configure = -95

When OVS sees multiple queues negotiated on a vhost port, it will transmit packets to
them. Depending on the "pmd thread" count in OVS, the mapping will be different,
but for best performance it is recommended to match the number of virtq's from
the guest to the number of OVS-pmd-threads on the host.

> I setup my VM in QEMU to have mq=on and queues=10. I also set the number
> of rx_queues when creating the vhost port using ovs to 10. Before
> binding the interface to DPDK, I used ethtool to verify if the network
> interface was actually setup to have multiple queues.
> 
> Running the 'ethtool -l enps02' command yields the following output:

Once you've bound interfaces to DPDK, ethtool no longer works,
and any configuration is reset to when DPDK initializes the port.
Ethtool cannot inspect DPDK ports/applications.

Although this mailing list will likely have experience, the OVS mailing list is likely a better location to
discuss OVS/DPDK/VM performance (I think you've posted there before).

Hope that helps, -Harry


>    Pre-set maximums:
>    RX:             0
>    TX:             0
>    Other:          0
>    Combined:       10
>    Current hardware settings:
>    RX:             0
>    TX:             0
>    Other:          0
>    Combined:       10
> 
>  From my understanding the combined values indicate that the interface
> was properly setup to have multiple queues, so why am I getting the
> unsupported RSS error? Are there other configuration steps that I have
> to take to get this to work? Is RSS with DPDK not supported at all
> inside a VM at the moment? Perhaps the "Port0 dev_configure() = -95"
> error means something else? Without the receive side scaling turned on
> my application is not able to achieve the desired throughput and won't
> scale when I assign more cores to the application.
> 
> Versions:
> VM
> DPDK: 21.11.4
> Kernel: 5.4.0-148-generic
> Distribution: Ubuntu 20.04
> 
> Host
> DPDK: 21.11.4
> QEMU: 8.0.90
> OvS: 3.0.5
> Kernel: 5.15.111.1.amd64-smp
> Distribution: Debian 11

Re: Supporting RSS with DPDK in a VM

[Maxime Coquelin]

Hi Matheus,

On 7/28/23 16:12, Thomas Monjalon wrote:
> Hello,
> 
> You can have packets distributed to multiple queues without RSS.
> 
> If you really wants to enable the RSS algorithms,
> it seems not supported for now with vhost_user.
> It can be enabled with vhost running in the kernel:
> https://qemu.readthedocs.io/en/latest/devel/ebpf_rss.html

Thomas is right, DPDK Virtio PMD supports RSS, but DPDK Vhost-user does
not (yet).

However, as you seem to be using a fairly recent OVS-DPDK version, you 
can enable Hash-based Tx packet steering mode on the vhostuser ports.

It has been introduced in OVS v2.17:
https://docs.openvswitch.org/en/latest/topics/userspace-tx-steering/

It is usefull if you have less PMD threads in OVS than the number of
queues in your Virtio device. Traffic will be distributed on the queues
based on the 5-tuple hash computed for OVS switching. The downside is
that this is not configurable as RSS though (Key not configurable, no
filters, ...).

Regards,
Maxime

> 
> 26/07/2023 00:58, Matheus Stolet:
>> Hello,
>>
>> I am trying to run a DPDK application with RSS enabled so that I can
>> have multiple rx queues. This application is running inside a VM. This
>> VM is hosted by QEMU using KVM acceleration and OvS with DPDK and
>> vhost-user are used in the backend. So to clarify things there are two
>> DPDK portions to this. The first is the DPDK portion used by OvS that
>> bypasses the host operating system. This is working fine. The other is a
>> DPDK application inside the virtual machine that will bypass the guest
>> operating system. This is where I am having trouble with.
>>
>> When I set rte_eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS in my application
>> I get the following errors:
>>     Warning: NIC does not support all requested RSS hash functions.
>>     virtio_dev_configure(): RSS support requested but not supported by the
>> device
>>     Port0 dev_configure = -95
>>
>> I setup my VM in QEMU to have mq=on and queues=10. I also set the number
>> of rx_queues when creating the vhost port using ovs to 10. Before
>> binding the interface to DPDK, I used ethtool to verify if the network
>> interface was actually setup to have multiple queues.
>>
>> Running the 'ethtool -l enps02' command yields the following output:
>>     Pre-set maximums:
>>     RX:             0
>>     TX:             0
>>     Other:          0
>>     Combined:       10
>>     Current hardware settings:
>>     RX:             0
>>     TX:             0
>>     Other:          0
>>     Combined:       10
>>
>>   From my understanding the combined values indicate that the interface
>> was properly setup to have multiple queues, so why am I getting the
>> unsupported RSS error? Are there other configuration steps that I have
>> to take to get this to work? Is RSS with DPDK not supported at all
>> inside a VM at the moment? Perhaps the "Port0 dev_configure() = -95"
>> error means something else? Without the receive side scaling turned on
>> my application is not able to achieve the desired throughput and won't
>> scale when I assign more cores to the application.
>>
>> Versions:
>> VM
>> DPDK: 21.11.4
>> Kernel: 5.4.0-148-generic
>> Distribution: Ubuntu 20.04
>>
>> Host
>> DPDK: 21.11.4
>> QEMU: 8.0.90
>> OvS: 3.0.5
>> Kernel: 5.15.111.1.amd64-smp
>> Distribution: Debian 11
>>
> 
> 
> 
> 
> 

Re: Supporting RSS with DPDK in a VM

[Matheus Stolet]

Sorry it this is a duplicate, but I believe that last time when I 
replied I forgot to cc the mailing list. First of all, thank you for the 
helpful info, but I have one more question!

How do I get the packets distributed to multiple queues without RSS? I 
turned off  RSS by setting ETH_MQ_RX_NONE. I configured multiple queues 
to be used by QEMU and OvS. Inside my DPDK application I checked the 
nb_rx_queues value inside the rte_eth_dev_info struct and it says I have 
10 rx_queues and max_rx_queues for the virtio device that was bound to 
the dpdk application, so at least this initial configuration seems to be 
working and DPDK is detecting multiple queues. The issue I am having is 
that I am only receiving packets in rx queue 0. From some other similar 
questions it seems like that is the default and specified behaviour, so 
how do I also get the other rx queues to be used? I found some vague 
stuff about using the rte_flow API. I can see how the queue action can 
be used to move packets to specific queues. These packets could even be 
matched based on IP address and TCP ports, but I don't see how to 
somewhat uniformly distribute them between queues without using a hash 
and modulo operator (I believe the rte_flow API doesn't support that).

On 2023-07-28 16:12, Thomas Monjalon wrote:
> Hello,
> 
> You can have packets distributed to multiple queues without RSS.
> 
> If you really wants to enable the RSS algorithms,
> it seems not supported for now with vhost_user.
> It can be enabled with vhost running in the kernel:
> https://qemu.readthedocs.io/en/latest/devel/ebpf_rss.html
> 
> 
> 26/07/2023 00:58, Matheus Stolet:
>> Hello,
>> 
>> I am trying to run a DPDK application with RSS enabled so that I can
>> have multiple rx queues. This application is running inside a VM. This
>> VM is hosted by QEMU using KVM acceleration and OvS with DPDK and
>> vhost-user are used in the backend. So to clarify things there are two
>> DPDK portions to this. The first is the DPDK portion used by OvS that
>> bypasses the host operating system. This is working fine. The other is 
>> a
>> DPDK application inside the virtual machine that will bypass the guest
>> operating system. This is where I am having trouble with.
>> 
>> When I set rte_eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS in my 
>> application
>> I get the following errors:
>>    Warning: NIC does not support all requested RSS hash functions.
>>    virtio_dev_configure(): RSS support requested but not supported by 
>> the
>> device
>>    Port0 dev_configure = -95
>> 
>> I setup my VM in QEMU to have mq=on and queues=10. I also set the 
>> number
>> of rx_queues when creating the vhost port using ovs to 10. Before
>> binding the interface to DPDK, I used ethtool to verify if the network
>> interface was actually setup to have multiple queues.
>> 
>> Running the 'ethtool -l enps02' command yields the following output:
>>    Pre-set maximums:
>>    RX:             0
>>    TX:             0
>>    Other:          0
>>    Combined:       10
>>    Current hardware settings:
>>    RX:             0
>>    TX:             0
>>    Other:          0
>>    Combined:       10
>> 
>>  From my understanding the combined values indicate that the interface
>> was properly setup to have multiple queues, so why am I getting the
>> unsupported RSS error? Are there other configuration steps that I have
>> to take to get this to work? Is RSS with DPDK not supported at all
>> inside a VM at the moment? Perhaps the "Port0 dev_configure() = -95"
>> error means something else? Without the receive side scaling turned on
>> my application is not able to achieve the desired throughput and won't
>> scale when I assign more cores to the application.
>> 
>> Versions:
>> VM
>> DPDK: 21.11.4
>> Kernel: 5.4.0-148-generic
>> Distribution: Ubuntu 20.04
>> 
>> Host
>> DPDK: 21.11.4
>> QEMU: 8.0.90
>> OvS: 3.0.5
>> Kernel: 5.15.111.1.amd64-smp
>> Distribution: Debian 11
>>