DPDK usage discussions
 help / color / Atom feed
* [dpdk-users] lcore stack location and NUMA nodes
@ 2019-05-14 14:50 N. Benes
  2019-05-14 15:20 ` Stephen Hemminger
  0 siblings, 1 reply; 2+ messages in thread
From: N. Benes @ 2019-05-14 14:50 UTC (permalink / raw)
  To: users

Hi all,

is it ensured that the stacks of lcore threads with set affinity are
located on the same NUMA node as the assigned CPU core?

So far, I understand that e.g. `rte_eal_mp_remote_launch()` does not
actually create threads but just communicates the function to execute to
already existing threads (via pipes). These threads are created during
initialisation of the EAL:

https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal.c#n1164

The threads execute a state machine `eal_thread_loop()` where the
affinity is changed/set for the corresponding lcore/thread:

https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal_thread.c#n119

However, couldn't this lead to the stacks of some threads to be still
located on the (different) NUMA node where `pthread_create()` was called
from, i.e. possible decreased performance?
I did not find an explicit relocation of thread stacks.
(I also did not find anything related in the glibc/nptl code)

One possible fix may be to change the affinity of the main/master thread
before calling `pthread_create()` for the slaves and to reset the
affinity of the master thread after all slave threads are created.

Cheers!

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [dpdk-users] lcore stack location and NUMA nodes
  2019-05-14 14:50 [dpdk-users] lcore stack location and NUMA nodes N. Benes
@ 2019-05-14 15:20 ` Stephen Hemminger
  0 siblings, 0 replies; 2+ messages in thread
From: Stephen Hemminger @ 2019-05-14 15:20 UTC (permalink / raw)
  To: N. Benes; +Cc: users

On Tue, 14 May 2019 14:50:00 +0000
"N. Benes" <nbenes@eso.org> wrote:

> Hi all,
> 
> is it ensured that the stacks of lcore threads with set affinity are
> located on the same NUMA node as the assigned CPU core?

No stacks are not directly affinitied to NUMA node.
Stacks live in regular (virtual) memory and the operating system manages
VM for processes/threads. The Linux MM subsystem may try and choose a page
that is on the same node as where thread is scheduled.

You can't DMA off of the stack and stack is highly cached data so
so it shouldn't matter much for a normal application.

> 
> So far, I understand that e.g. `rte_eal_mp_remote_launch()` does not
> actually create threads but just communicates the function to execute to
> already existing threads (via pipes). These threads are created during
> initialisation of the EAL:
> 
> https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal.c#n1164
> 
> The threads execute a state machine `eal_thread_loop()` where the
> affinity is changed/set for the corresponding lcore/thread:
> 
> https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal_thread.c#n119
> 
> However, couldn't this lead to the stacks of some threads to be still
> located on the (different) NUMA node where `pthread_create()` was called
> from, i.e. possible decreased performance?
> I did not find an explicit relocation of thread stacks.
> (I also did not find anything related in the glibc/nptl code)
> 
> One possible fix may be to change the affinity of the main/master thread
> before calling `pthread_create()` for the slaves and to reset the
> affinity of the master thread after all slave threads are created.
> 
> Cheers!


^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, back to index

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-14 14:50 [dpdk-users] lcore stack location and NUMA nodes N. Benes
2019-05-14 15:20 ` Stephen Hemminger

DPDK usage discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/ public-inbox