DPDK usage discussions
 help / color / mirror / Atom feed
* [dpdk-users] lcore stack location and NUMA nodes
@ 2019-05-14 14:50 N. Benes
  2019-05-14 15:20 ` Stephen Hemminger
  0 siblings, 1 reply; 2+ messages in thread
From: N. Benes @ 2019-05-14 14:50 UTC (permalink / raw)
  To: users

Hi all,

is it ensured that the stacks of lcore threads with set affinity are
located on the same NUMA node as the assigned CPU core?

So far, I understand that e.g. `rte_eal_mp_remote_launch()` does not
actually create threads but just communicates the function to execute to
already existing threads (via pipes). These threads are created during
initialisation of the EAL:

https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal.c#n1164

The threads execute a state machine `eal_thread_loop()` where the
affinity is changed/set for the corresponding lcore/thread:

https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal_thread.c#n119

However, couldn't this lead to the stacks of some threads to be still
located on the (different) NUMA node where `pthread_create()` was called
from, i.e. possible decreased performance?
I did not find an explicit relocation of thread stacks.
(I also did not find anything related in the glibc/nptl code)

One possible fix may be to change the affinity of the main/master thread
before calling `pthread_create()` for the slaves and to reset the
affinity of the master thread after all slave threads are created.

Cheers!

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [dpdk-users] lcore stack location and NUMA nodes
  2019-05-14 14:50 [dpdk-users] lcore stack location and NUMA nodes N. Benes
@ 2019-05-14 15:20 ` Stephen Hemminger
  0 siblings, 0 replies; 2+ messages in thread
From: Stephen Hemminger @ 2019-05-14 15:20 UTC (permalink / raw)
  To: N. Benes; +Cc: users

On Tue, 14 May 2019 14:50:00 +0000
"N. Benes" <nbenes@eso.org> wrote:

> Hi all,
> 
> is it ensured that the stacks of lcore threads with set affinity are
> located on the same NUMA node as the assigned CPU core?

No stacks are not directly affinitied to NUMA node.
Stacks live in regular (virtual) memory and the operating system manages
VM for processes/threads. The Linux MM subsystem may try and choose a page
that is on the same node as where thread is scheduled.

You can't DMA off of the stack and stack is highly cached data so
so it shouldn't matter much for a normal application.

> 
> So far, I understand that e.g. `rte_eal_mp_remote_launch()` does not
> actually create threads but just communicates the function to execute to
> already existing threads (via pipes). These threads are created during
> initialisation of the EAL:
> 
> https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal.c#n1164
> 
> The threads execute a state machine `eal_thread_loop()` where the
> affinity is changed/set for the corresponding lcore/thread:
> 
> https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal_thread.c#n119
> 
> However, couldn't this lead to the stacks of some threads to be still
> located on the (different) NUMA node where `pthread_create()` was called
> from, i.e. possible decreased performance?
> I did not find an explicit relocation of thread stacks.
> (I also did not find anything related in the glibc/nptl code)
> 
> One possible fix may be to change the affinity of the main/master thread
> before calling `pthread_create()` for the slaves and to reset the
> affinity of the master thread after all slave threads are created.
> 
> Cheers!


^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2019-05-14 15:20 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-14 14:50 [dpdk-users] lcore stack location and NUMA nodes N. Benes
2019-05-14 15:20 ` Stephen Hemminger

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).