[dpdk-dev] Very low performance with l2fwd in a VM with PCI Passthrough

[dpdk-dev] Very low performance with l2fwd in a VM with PCI Passthrough

[Surya Nimmagadda]

Hi,

I am seeing very low throughput when I run l2fwd in a VM. 

When I send 1g data, I could see only about 10mbps of traffic being received by the VM. 

I dont see this problem when running l2fwd on the host. I could get 10gbps traffic in and out of both the ports.

My setup details are

- Traffic coming in on a 10G interface(eth0) and going out on another 10G interface(eth1)
- Both 10G NICs are 82599
- Created virtual functions eth5 and eth8 on eth0 and eth1 respectively
- eth5 and eth8 are mapped to eth0 and eth1 in a VM, with device type as e1000 in Passthru mode.
- Host OS : Centos 6.2, Guest OS : Ubuntu 12.04

<…>
Network devices using IGB_UIO driver
====================================
0000:00:07.0 '82540EM Gigabit Ethernet Controller' drv=igb_uio unused=e1000
0000:00:08.0 '82540EM Gigabit Ethernet Controller' drv=igb_uio unused=e1000
<…>

The counters on the virtual interface (eth5/eth8) show all 10g traffic received, but I see very 10mbps worth traffic at the l2fwd app counters on the VM.

I also see the following logs in the VM dmesg output.

[75399.491215] irq 0xb not handled
[75400.142025] irq 0xb not handled
[75400.142150] irq 0xb not handled
[75400.142913] irq 0xb not handled
[75400.142920] irq 0xb not handled

Has anyone seen this issue? Am I missing anything?

Thanks,
Surya

Re: [dpdk-dev] Very low performance with l2fwd in a VM with PCI Passthrough

[Surya Nimmagadda]

When change the device emulation in the NIC to virtio, I could get 900mbps when sending 1g of input. Good so far.

But when I increased the traffic rate to 10g, output became very turbulent - ranging from 0 to 150mbps.

I tried rate limiting on the vfs using the following commands, but didnt help.

ip link set eth0 vf 0 rate 1000
ip link set eth1 vf 0 rate 1000

Are there any recommended settings for using DPDK over virtual function in a VM?

Thanks,
Surya

On Mar 6, 2014, at 7:08 PM, Surya Nimmagadda <nscsekhar@juniper.net> wrote:

> Hi,
> 
> I am seeing very low throughput when I run l2fwd in a VM. 
> 
> When I send 1g data, I could see only about 10mbps of traffic being received by the VM. 
> 
> I dont see this problem when running l2fwd on the host. I could get 10gbps traffic in and out of both the ports.
> 
> My setup details are
> 
> - Traffic coming in on a 10G interface(eth0) and going out on another 10G interface(eth1)
> - Both 10G NICs are 82599
> - Created virtual functions eth5 and eth8 on eth0 and eth1 respectively
> - eth5 and eth8 are mapped to eth0 and eth1 in a VM, with device type as e1000 in Passthru mode.
> - Host OS : Centos 6.2, Guest OS : Ubuntu 12.04
> 
> <…>
> Network devices using IGB_UIO driver
> ====================================
> 0000:00:07.0 '82540EM Gigabit Ethernet Controller' drv=igb_uio unused=e1000
> 0000:00:08.0 '82540EM Gigabit Ethernet Controller' drv=igb_uio unused=e1000
> <…>
> 
> The counters on the virtual interface (eth5/eth8) show all 10g traffic received, but I see very 10mbps worth traffic at the l2fwd app counters on the VM.
> 
> I also see the following logs in the VM dmesg output.
> 
> [75399.491215] irq 0xb not handled
> [75400.142025] irq 0xb not handled
> [75400.142150] irq 0xb not handled
> [75400.142913] irq 0xb not handled
> [75400.142920] irq 0xb not handled
> 
> Has anyone seen this issue? Am I missing anything?
> 
> Thanks,
> Surya
> 
> 

Re: [dpdk-dev] Very low performance with l2fwd in a VM with PCI Passthrough

[Richardson, Bruce]

> My setup details are
> 
> - Traffic coming in on a 10G interface(eth0) and going out on another 10G
> interface(eth1)
> - Both 10G NICs are 82599
> - Created virtual functions eth5 and eth8 on eth0 and eth1 respectively
> - eth5 and eth8 are mapped to eth0 and eth1 in a VM, with device type as
> e1000 in Passthru mode.

If the NIC in the VM is showing up as an e1000 then you have not passed the VF through to the VM at all, instead you are using an emulated NIC, which will be very, very slow in comparison - if nothing else, it's an emulated 1G NIC, not a 10G one. 
To set up a VM and use the virtual functions from the 10G NIC, I suggest referring to Chapter 9 of the Programmers Guide available at http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/intel-dpdk-programmers-guide.html . Instructions for setting up the VM to use VFs are covered by section 9.2

Re: [dpdk-dev] Very low performance with l2fwd in a VM with PCI Passthrough

[Surya Nimmagadda]

>    To set up a VM and use the virtual functions from the 10G NIC, I suggest referring to Chapter 9 of the Programmers Guide available at http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/intel-dpdk-      programmers-guide.html . Instructions for setting up the VM to use VFs are covered by section 9.2

Thanks a lot. This helped me out.

Surya
________________________________________
From: Richardson, Bruce <bruce.richardson@intel.com>
Sent: Friday, March 7, 2014 1:27 AM
To: Surya Nimmagadda; dev@dpdk.org
Subject: RE: [dpdk-dev] Very low performance with l2fwd in a VM with PCI        Passthrough

> My setup details are
>
> - Traffic coming in on a 10G interface(eth0) and going out on another 10G
> interface(eth1)
> - Both 10G NICs are 82599
> - Created virtual functions eth5 and eth8 on eth0 and eth1 respectively
> - eth5 and eth8 are mapped to eth0 and eth1 in a VM, with device type as
> e1000 in Passthru mode.

If the NIC in the VM is showing up as an e1000 then you have not passed the VF through to the VM at all, instead you are using an emulated NIC, which will be very, very slow in comparison - if nothing else, it's an emulated 1G NIC, not a 10G one.
To set up a VM and use the virtual functions from the 10G NIC, I suggest referring to Chapter 9 of the Programmers Guide available at http://www.intel.com/content/www/us/en/intelligent-systems/intel-technology/intel-dpdk-programmers-guide.html . Instructions for setting up the VM to use VFs are covered by section 9.2