Re: [dpdk-dev] [PATCH RFC] librte_reorder: new reorder library

[Matthew Hall <mhall@mhcomputing.net>]

On Thu, Oct 09, 2014 at 10:14:21AM +0100, Bruce Richardson wrote:
> Hi Matthew,
> 
> What you are doing will indeed work, and it's the way the vast majority of 
> the sample apps are written. However, this will not always work for everyone 
> else, sadly.
> 
> First off, with RSS, there are a number of limitations. On the 1G and 10G 
> NICs RSS works only with IP traffic, and won't work in cases with other 
> protocols or where IP is encapsulated in anything other than a single VLAN.  
> Those cases need software load distribution. As well as this, you have very 
> little control over where flows get put, as the separation into queues 
> (which go to cores), is only done on the low seven bits. For applications 
> which work with a small number of flows, e.g. where multiple flows are 
> contained inside a single tunnel, you get a get a large flow imbalance, 
> where you get far more traffic coming to one queue/core than to another.  
> Again in this instance, software load balancing is needed.
> 
> Secondly, then, based off that, it is entirely possible when doing software 
> load balancing to strictly process packets for a flow in order - and indeed 
> this is what the existing packet distributor does. However, for certain 
> types of flow where processing of packets for that flow can be done in 
> parallel, forcing things to be done serially can slow things down. As well 
> as this, there can sometimes be requirements for the load balancing between 
> cores to be done as fairly as possible so that it is guaranteed that all 
> cores have approx the same load, irrespective of the number of input flows.  
> In these cases, having the option to blindly distribute traffic to cores and 
> then reorder packets on TX is the best way to ensure even load distribution.  
> It's not going to be for everyone, but it's good to have the option - and 
> there are a number of people doing things this way already.
> 
> Lastly, there is also the assumption being made that all flows are 
> independent, which again may not always be the case. If you need ordering 
> across flows and to share load between cores then reordering on transmission 
> is the only way to do things.
> 
> Hope this helps,
> 
> Regards,
> /Bruce

Bruce,

This explanation is of excellent quality.

It would be nice if it could be made into a whitepaper about the different 
L2-L7 acceleration technologies available in the Intel NICs, popular VNICs 
(virtio-net and vmxnet3), Intel CPUs, and DPDK code, all working together. Or 
incorporated into such a document if it already exists.

Without things like this it's very hard to understand when and how to enable 
the different accelerations can be used together, when they'll work, and when 
they won't work.

For example, I didn't know RSS only worked on IP... I was assuming it would do 
a consistent-hash of MAC's for non-IP packets at least... also, when it 
doesn't know what to do, does it send them to the default queue, or a random 
FIFO RX queue picks it up or what?

Matthew.