Re: [dpdk-dev] DCA

[Bruce Richardson <bruce.richardson@intel.com>]

On Tue, Apr 21, 2015 at 10:44:54AM -0700, Matthew Hall wrote:
> On Tue, Apr 21, 2015 at 10:27:48AM +0100, Bruce Richardson wrote:
> > Can you perhaps comment on the use-case where you find this binding 
> > limiting? Modern platforms have multiple NUMA nodes, but they also generally 
> > have PCI slots connected to those multiple NUMA nodes also, so that you can 
> > have your NIC ports similarly NUMA partitionned?
> 
> Hi Bruce,
> 
> I was wondering if you have tried to do this on COTS (commerical 
> off-the-shelf) hardware before. What I found each time I tried it was that 
> PCIe slots are not very evenly distributed across the NUMA nodes unlike what 
> you'd expect.
> 

I doubt I've tried it on regular commercial boards as much as you guys have,
though it does happen!

> Sometimes the PCIe lanes on CPU 0 get partly used up by Super IO or other 
> integrated peripherals. Other times the motherboards give you 2 x8 when you 
> needed 1 x16 or they give you a bundh of x4 when you needed x8, etc.

Point taken!

> 
> It's actually pretty difficult to find the mapping, for one, and even when you 
> do, even harder to get the right slots for your cards and so on. In the ixgbe 
> kernel driver you'll sometimes get some cryptic debug prints when it's been 
> munged and performance will suffer. But in the ixgbe PMD driver you're on your 
> own mostly.

It was to try and make the NUMA mapping of PCI clearer that we added in the
printing of the NUMA node on PCI scan:

EAL: PCI device 0000:86:00.0 on NUMA socket 1
EAL:   probe driver: 8086:154a rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7fb452f04000
EAL:   PCI memory mapped at 0x7fb453004000

Is there something more than this you feel we could do in the PMD to help with
slot identification?

/Bruce

> 
> Matthew.