Re: [dpdk-dev] Performance hit - NICs on different CPU sockets

["Wiles, Keith" <keith.wiles@intel.com>]

On 6/16/16, 10:16 AM, "Take Ceara" <dumitru.ceara@gmail.com> wrote:

>On Thu, Jun 16, 2016 at 4:58 PM, Wiles, Keith <keith.wiles@intel.com> wrote:
>>
>> From the output below it appears the x710 devices 01:00.[0-3] are on socket 0
>> And the x710 devices 02:00.[0-3] sit on socket 1.
>>
>
>I assume there's a mistake here. The x710 devices on socket 0 are:
>$ lspci | grep -ie "01:.*x710"
>01:00.0 Ethernet controller: Intel Corporation Ethernet Controller
>X710 for 10GbE SFP+ (rev 01)
>01:00.1 Ethernet controller: Intel Corporation Ethernet Controller
>X710 for 10GbE SFP+ (rev 01)
>01:00.2 Ethernet controller: Intel Corporation Ethernet Controller
>X710 for 10GbE SFP+ (rev 01)
>01:00.3 Ethernet controller: Intel Corporation Ethernet Controller
>X710 for 10GbE SFP+ (rev 01)
>
>and the X710 devices on socket 1 are:
>$ lspci | grep -ie "81:.*x710"
>81:00.0 Ethernet controller: Intel Corporation Ethernet Controller
>X710 for 10GbE SFP+ (rev 01)
>81:00.1 Ethernet controller: Intel Corporation Ethernet Controller
>X710 for 10GbE SFP+ (rev 01)
>81:00.2 Ethernet controller: Intel Corporation Ethernet Controller
>X710 for 10GbE SFP+ (rev 01)
>81:00.3 Ethernet controller: Intel Corporation Ethernet Controller
>X710 for 10GbE SFP+ (rev 01)

Yes, you are correct I miss-read the lspci output.

>
>> This means the ports on 01.00.xx should be handled by socket 0 CPUs and 02:00.xx should be handled by Socket 1. I can not tell if that is the case for you here. The CPUs or lcores from the cpu_layout.py should help understand the layout.
>>
>
>That was the first scenario I tried:
>- assign 16 CPUs from socket 0 to port 0 (01:00.3)
>- assign 16 CPUs from socket 1 to port 1 (81:00.3)
>
>Our performance measurements show then a setup rate of 1.6M sess/s
>which is less then half of what I get when i install both X710 on
>socket 1 and use only 16 CPUs from socket 1 for both ports.

Right now I do not know what the issue is with the system. Could be too many Rx/Tx ring pairs per port and limiting the memory in the NICs, which is why you get better performance when you have 8 core per port. I am not really seeing the whole picture and how DPDK is configured to help more. Sorry.

Maybe seeing the DPDK command line would help.

++Keith

>
>I double checked the cpu layout. We also have our own CLI and warnings
>when using cores that are not on the same socket as the port they're
>assigned too so the mapping should be fine.
>
>Thanks,
>Dumitru
>