Re: [dpdk-dev] IXGBE RX packet loss with 5+ cores

[Alexander Duyck <alexander.duyck@gmail.com>]

On 10/13/2015 07:47 AM, Sanford, Robert wrote:
>>>> [Robert:]
>>>> 1. The 82599 device supports up to 128 queues. Why do we see trouble
>>>> with as few as 5 queues? What could limit the system (and one port
>>>> controlled by 5+ cores) from receiving at line-rate without loss?
>>>>
>>>> 2. As far as we can tell, the RX path only touches the device
>>>> registers when it updates a Receive Descriptor Tail register (RDT[n]),
>>>> roughly every rx_free_thresh packets. Is there a big difference
>>>> between one core doing this and N cores doing it 1/N as often?
>>> [Stephen:]
>>> As you add cores, there is more traffic on the PCI bus from each core
>>> polling. There is a fix number of PCI bus transactions per second
>>> possible.
>>> Each core is increasing the number of useless (empty) transactions.
>> [Bruce:]
>> The polling for packets by the core should not be using PCI bandwidth
>> directly,
>> as the ixgbe driver (and other drivers) check for the DD bit being set on
>> the
>> descriptor in memory/cache.
> I was preparing to reply with the same point.
>
>>> [Stephen:] Why do you think adding more cores will help?
> We're using run-to-completion and sometimes spend too many cycles per pkt.
> We realize that we need to move to io+workers model, but wanted a better
> understanding of the dynamics involved here.
>
>
>
>> [Bruce:] However, using an increased number of queues can
>> use PCI bandwidth in other ways, for instance, with more queues you
>> reduce the
>> amount of descriptor coalescing that can be done by the NICs, so that
>> instead of
>> having a single transaction of 4 descriptors to one queue, the NIC may
>> instead
>> have to do 4 transactions each writing 1 descriptor to 4 different
>> queues. This
>> is possibly why sending all traffic to a single queue works ok - the
>> polling on
>> the other queues is still being done, but has little effect.
> Brilliant! This idea did not occur to me.

You can actually make the throughput regression disappear by altering 
the traffic pattern you are testing with.  In the past I have found that 
sending traffic in bursts where 4 frames belong to the same queue before 
moving to the next one essentially eliminated the dropped packets due to 
PCIe bandwidth limitations.  The trick is you need to have the Rx 
descriptor processing work in batches so that you can get multiple 
descriptors processed for each PCIe read/write.

- Alex