[dpdk-users] How to wring performance out of virtio?

[dpdk-users] How to wring performance out of virtio?

[Matt Laswell]

Hey everybody,

My application relies on some control plane apps that need to talk to the
network through ports that are controlled by DPDK.  Historically, we've
used KNI for that, which lets the control plane applications use the normal
Linux network stack and interact with it with common tools.  Recently, I've
been looking into leaving KNI and moving to virtio for the same purpose.
However, I find that I can't move traffic through it as quickly as I could
with KNI.

Specifically, the relevant portions of my dataplane application move data
around roughly like this:
NIC<->PMD<->packet ingestor (recognizes control plane traffic) <->
KNI/virtio thread <-> KNI/virtio

What I find is that if my control plane application is mostly sending
traffic out to the network, I can achieve roughly the same bandwidth
through virtio that I did through KNI.  If I'm primarily reading from the
network, however, I only get about half of the bandwidth I used to get from
KNI.  Also, I see congestion when enqueueing for virtio.  The congestion
isn't alleviated by increasing queue sizes, so I presume I'm not dealing
with occasional bursts, but general slowness.

Obviously, optimization is a complex subject, and I'm only describing a
small corner of a more complex codebase.  But, for those of you who have
more experience with virtio under DPDK than I do, if you were presented
with a problem like this, what is the handful of questions you ask, or big
wins you'd seek out first?

A few details about my application:

   - Physical hardware, with isolated CPU cores for the dataplane, not a VM
   - We're using the virtio library that comes with Ubuntu, not a
   custom/DPDK one
   - Checksum generation and validation are enabled at the NIC, but TSO and
   LRO are not, as they don't play nicely with the rest of our application

Thanks in advance,
- Matt