Re: [dpdk-dev] Appropriate DPDK data structures for TCP sockets

[Avi Kivity <avi@cloudius-systems.com>]

On 02/23/2015 11:16 PM, Matthew Hall wrote:
> On Mon, Feb 23, 2015 at 08:48:57AM -0600, Matt Laswell wrote:
>> Apologies in advance for likely being a bit long-winded.
> Long winded is great, helps me get context.
>
>> First, you really need to take cache performance into account when you're
>> choosing a data structure.  Something like a balanced tree can seem awfully
>> appealing at first blush
> Agreed. I did some amount of DPDK stuff before but without TCP. This is why I
> was figuring a packet-hash is better than a tree.
>
>> Second, rather than synchronizing (perhaps with locks, perhaps with
>> lockless data structures), it's often beneficial to create multiple
>> threads, each of which holds a fraction of your connection tracking data.
> Yes, I REALLY REALLY REALLY wanted to do RSS. But the virtio-net and other
> VM's don't support RSS, unlike the classic PCIe NIC's. In order to get the
> community to use my app I have to give them a "batteries included"
> environment, where the system can still work even with no RSS.

For an example of a tcp stack on top of dpdk please see seastar [1]. It 
supports hardware RSS, software RSS, or a combination (if the number of 
hardware queues is smaller than the number of cores).

>> Third, it's very worthwhile to have a cache for the most recently accessed
>> connection.  First, because network traffic is bursty, and you'll
>> frequently see multiple packets from the same connection in succession.
>> Second, because it can make life easier for your application code.  If you
>> have multiple places that need to access connection data, you don't have to
>> worry so much about the cost of repeated searches.  Again, this may or may
>> not matter for your particular application.  But for ones I've worked on,
>> it's been a win.
> Yes, this sounds like a really good idea. One advantage in my product, I am
> only doing TCP Syslog, so I don't have an arbitrary zillion connections like
> FW or IPS would want. I could cap it at something like 8192 or 16384 and be
> good enough for some time until a better solution is worked out.
>
> I could make some capped array or linked list of the X most recent ones for
> cheap access. It's just socket pointers so it doesn't hardly cost anything to
> copy a couple pointers into a cache and quickly invalidate when the connection
> closes.

A simple per-core hash table is sufficient in our experience.  Yes, you 
will take a cache miss, but it's not the end of the world.


[1] https://github.com/cloudius-systems/seastar