[dpdk-users] lcore stack location and NUMA nodes

[dpdk-users] lcore stack location and NUMA nodes

[N. Benes]

Hi all,

is it ensured that the stacks of lcore threads with set affinity are
located on the same NUMA node as the assigned CPU core?

So far, I understand that e.g. `rte_eal_mp_remote_launch()` does not
actually create threads but just communicates the function to execute to
already existing threads (via pipes). These threads are created during
initialisation of the EAL:

https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal.c#n1164

The threads execute a state machine `eal_thread_loop()` where the
affinity is changed/set for the corresponding lcore/thread:

https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal_thread.c#n119

However, couldn't this lead to the stacks of some threads to be still
located on the (different) NUMA node where `pthread_create()` was called
from, i.e. possible decreased performance?
I did not find an explicit relocation of thread stacks.
(I also did not find anything related in the glibc/nptl code)

One possible fix may be to change the affinity of the main/master thread
before calling `pthread_create()` for the slaves and to reset the
affinity of the master thread after all slave threads are created.

Cheers!

Re: [dpdk-users] lcore stack location and NUMA nodes

[Stephen Hemminger]

On Tue, 14 May 2019 14:50:00 +0000
"N. Benes" <nbenes@eso.org> wrote:

> Hi all,
> 
> is it ensured that the stacks of lcore threads with set affinity are
> located on the same NUMA node as the assigned CPU core?

No stacks are not directly affinitied to NUMA node.
Stacks live in regular (virtual) memory and the operating system manages
VM for processes/threads. The Linux MM subsystem may try and choose a page
that is on the same node as where thread is scheduled.

You can't DMA off of the stack and stack is highly cached data so
so it shouldn't matter much for a normal application.

> 
> So far, I understand that e.g. `rte_eal_mp_remote_launch()` does not
> actually create threads but just communicates the function to execute to
> already existing threads (via pipes). These threads are created during
> initialisation of the EAL:
> 
> https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal.c#n1164
> 
> The threads execute a state machine `eal_thread_loop()` where the
> affinity is changed/set for the corresponding lcore/thread:
> 
> https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal_thread.c#n119
> 
> However, couldn't this lead to the stacks of some threads to be still
> located on the (different) NUMA node where `pthread_create()` was called
> from, i.e. possible decreased performance?
> I did not find an explicit relocation of thread stacks.
> (I also did not find anything related in the glibc/nptl code)
> 
> One possible fix may be to change the affinity of the main/master thread
> before calling `pthread_create()` for the slaves and to reset the
> affinity of the master thread after all slave threads are created.
> 
> Cheers!