From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id AF6F3A00E6 for ; Tue, 14 May 2019 17:20:56 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E64A35B16; Tue, 14 May 2019 17:20:55 +0200 (CEST) Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by dpdk.org (Postfix) with ESMTP id B33A0548B for ; Tue, 14 May 2019 17:20:54 +0200 (CEST) Received: by mail-pl1-f170.google.com with SMTP id f97so3328933plb.5 for ; Tue, 14 May 2019 08:20:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LJnGUvZdgd6s+B+EVlQGecs4JAgUZuD2KU5kNzH5LaM=; b=GbdbQkYeGs+9FSx5Gy8YGdL40Ps73xFeQ70BWWc6lpOO3vCNNMmF/7JR/W9muC4N+E iICY9kwwadzMsixi1Ponnv2vakHiiWsiUi8hAnkVVOvsGq8X1W5BjJ1CdtK7YMM+DaaI bw3BUjcUTm6XfvzS9cw4tNYICwBHCZAMs2sdKDzvkKdUYNOr/gQYzvptPISD/jlidiGp FByCKUq0zLYCWYng3BVxpjHptmcTVB2LQmeagXEOWZU2+fvdA0jZsFIVRmaUYXJ+yTmj mlEFEuFPnHzMkYpaZ8K7LY+eAcpM0Hx1ZbDS+cD3p5SCDY7lE6YRaMGfzXqLZhcBScLj DvuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LJnGUvZdgd6s+B+EVlQGecs4JAgUZuD2KU5kNzH5LaM=; b=MJTarGC8nwXr6zfRpkh9OfCGlS7Jax4kkjLvmweAZWJ8SX4UGITNuEtwFqfh1S5Nzl vMLqzzxzOz6p3V5FQJsLJrJYvwVD2Kaqa84cvjWreCzqg8i5y6fn+rfEwUL2vREzBgGo YFLnMH5QhWO56zUxqUSOgvGqE9a8jKwoCxrAMr7UYA8qTLADBKttWb35KOswFUqeU5CH HM7JVJRZB9yYbAvxAQ1MOT6dCINCVYEGBRNa7hwbtYZEvGPSXokVWjquZ8DUzSC6qLo5 K2pCkGT6BzMajVOqKPTEd8k/WZK/XFYLka1G8/qVgYziL3E5+Mk7MP0L1p4E5fq5aSgX kCLg== X-Gm-Message-State: APjAAAVwWoIk2ybw1ocyUvwNBgbyftLPWVMLmKtkJdHyFHDaoHg8y8VT fi2tvuo6QQnItJ2TDyy7Im4m+A== X-Google-Smtp-Source: APXvYqyeIOnU5bdurirdUvZCncmFzTdIELuC2JgeP+/OULipNSvh8u68vJz+s2ta0u84KcYd7r80xQ== X-Received: by 2002:a17:902:2869:: with SMTP id e96mr14501525plb.150.1557847252942; Tue, 14 May 2019 08:20:52 -0700 (PDT) Received: from hermes.lan (204-195-22-127.wavecable.com. [204.195.22.127]) by smtp.gmail.com with ESMTPSA id q128sm21661892pfb.164.2019.05.14.08.20.52 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 14 May 2019 08:20:52 -0700 (PDT) Date: Tue, 14 May 2019 08:20:45 -0700 From: Stephen Hemminger To: "N. Benes" Cc: users@dpdk.org Message-ID: <20190514082045.3b10cddb@hermes.lan> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-users] lcore stack location and NUMA nodes X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" On Tue, 14 May 2019 14:50:00 +0000 "N. Benes" wrote: > Hi all, > > is it ensured that the stacks of lcore threads with set affinity are > located on the same NUMA node as the assigned CPU core? No stacks are not directly affinitied to NUMA node. Stacks live in regular (virtual) memory and the operating system manages VM for processes/threads. The Linux MM subsystem may try and choose a page that is on the same node as where thread is scheduled. You can't DMA off of the stack and stack is highly cached data so so it shouldn't matter much for a normal application. > > So far, I understand that e.g. `rte_eal_mp_remote_launch()` does not > actually create threads but just communicates the function to execute to > already existing threads (via pipes). These threads are created during > initialisation of the EAL: > > https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal.c#n1164 > > The threads execute a state machine `eal_thread_loop()` where the > affinity is changed/set for the corresponding lcore/thread: > > https://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal_thread.c#n119 > > However, couldn't this lead to the stacks of some threads to be still > located on the (different) NUMA node where `pthread_create()` was called > from, i.e. possible decreased performance? > I did not find an explicit relocation of thread stacks. > (I also did not find anything related in the glibc/nptl code) > > One possible fix may be to change the affinity of the main/master thread > before calling `pthread_create()` for the slaves and to reset the > affinity of the master thread after all slave threads are created. > > Cheers!