Re: [dpdk-dev] DCA

[Matthew Hall <mhall@mhcomputing.net>]

On Tue, Apr 21, 2015 at 10:27:48AM +0100, Bruce Richardson wrote:
> Can you perhaps comment on the use-case where you find this binding 
> limiting? Modern platforms have multiple NUMA nodes, but they also generally 
> have PCI slots connected to those multiple NUMA nodes also, so that you can 
> have your NIC ports similarly NUMA partitionned?

Hi Bruce,

I was wondering if you have tried to do this on COTS (commerical 
off-the-shelf) hardware before. What I found each time I tried it was that 
PCIe slots are not very evenly distributed across the NUMA nodes unlike what 
you'd expect.

Sometimes the PCIe lanes on CPU 0 get partly used up by Super IO or other 
integrated peripherals. Other times the motherboards give you 2 x8 when you 
needed 1 x16 or they give you a bundh of x4 when you needed x8, etc.

It's actually pretty difficult to find the mapping, for one, and even when you 
do, even harder to get the right slots for your cards and so on. In the ixgbe 
kernel driver you'll sometimes get some cryptic debug prints when it's been 
munged and performance will suffer. But in the ixgbe PMD driver you're on your 
own mostly.

Matthew.