From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <stephen@networkplumber.org> Received: from mail-pa0-f51.google.com (mail-pa0-f51.google.com [209.85.220.51]) by dpdk.org (Postfix) with ESMTP id 81255C31E for <dev@dpdk.org>; Wed, 22 Apr 2015 17:41:24 +0200 (CEST) Received: by pabtp1 with SMTP id tp1so276148100pab.2 for <dev@dpdk.org>; Wed, 22 Apr 2015 08:41:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-type:content-transfer-encoding; bh=JLOnVQQ99EtcJRQHSDIq3lE+hspOaeDM0X5OlY99JIo=; b=IvDQPJJQJ7HdARahgGuKScZWHp8b46H2/IJAR1xWUJ2c0pdga289Fm1M3iJEha4+BX 4y84CGdVZpz3Pdqz6Chl2vacK7ug/Qw0aoM+Bu6zEQtm/4CgCAMo+QhSDsiMyrVtW+na B5B3dPcjK18e0bbBXYj+mJOPCkcv1e+e1K+TU9RPx0+QcZPceM9Al3Wd1XMwMuDaO3bu ZPn09/jQaol0iPqOzhp5eES4Ic219GHcPlwbK6IZ1MVzvDpM7bPFMZxnZDq43bp3bw2S EDycnhgAxsGaC9d6Q1E8RdiSxO0KUbqNNxkR1+zCU5Lp1PZKSKclwDuPGvDWz8/VxmFZ zcFQ== X-Gm-Message-State: ALoCoQlMWhtNfscZ/NmO30frCCHbnMsOi1fL3RXccqyLlmelzjvUS6LE8+y6ahVeNCmeHoQk4yVX X-Received: by 10.68.248.39 with SMTP id yj7mr48226990pbc.58.1429717283831; Wed, 22 Apr 2015 08:41:23 -0700 (PDT) Received: from urahara (static-50-53-82-155.bvtn.or.frontiernet.net. [50.53.82.155]) by mx.google.com with ESMTPSA id fu7sm5519930pac.26.2015.04.22.08.41.23 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Apr 2015 08:41:23 -0700 (PDT) Date: Wed, 22 Apr 2015 08:41:27 -0700 From: Stephen Hemminger <stephen@networkplumber.org> To: Bruce Richardson <bruce.richardson@intel.com> Message-ID: <20150422084127.7a04d0b0@urahara> In-Reply-To: <20150422133416.GA8676@bricha3-MOBL3> References: <SNT148-W27CC4C58D714B7FEBCD7ACF6EE0@phx.gbl> <E115CCD9D858EF4F90C690B0DCB4D897272961FD@IRSMSX108.ger.corp.intel.com> <SNT148-W7973933D48CAD040A45339F6EE0@phx.gbl> <20150422133416.GA8676@bricha3-MOBL3> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: "dev@dpdk.org" <dev@dpdk.org> Subject: Re: [dpdk-dev] problem: Cannot allocate memzone for ethernet port data X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Wed, 22 Apr 2015 15:41:25 -0000 On Wed, 22 Apr 2015 14:34:17 +0100 Bruce Richardson <bruce.richardson@intel.com> wrote: > On Wed, Apr 22, 2015 at 07:36:59PM +0800, ChenXiaodong wrote: > > Thanks for you reply! There is actually only one NUMA node. here is what the commands return: > > > > Hi, > > there seems to be a mismatch in terms of the numa node the memory is reported > as being on (node 0) and the numa node that the cores are detected as being on > - as reported by the physical package id value. That's why the app is failing > as it's looking for the memory on the wrong numa node. > Can you try applying the patch I've just posted to change how we detect > the numa node of a core, and see if it helps in your case. > > thanks, > /Bruce This issue has been around a long time. Older kernels (pre 3.3) don't have a good solution and are really broken. Some old history: > On Fri, 31 May 2013 11:29:09 +0200 > Damien Millescamps <damien.millescamps at 6wind.com> wrote: > > > On 05/30/2013 07:12 PM, Stephen Hemminger wrote: > > > One fix not included is the NUMA cpu assignment. The original Intel code > > > in 1.2 incorrectly used /proc/cpuinfo to try and assign CPU's to NUMA socket. > > > The problem is that /proc/cpuinfo physical_id corresponds to what the BIOS > > > tells the kernel and is intended for messages only. For example, on our > > > Dell boxes the first CPU and only CPU is reported as physical_id 1! > > > The fix is to use sysfs instead, > > Hi Stephen, > > > > Are you using the > > /sys/devices/system/node/nodeX/cpuX/topology/physical_package_id special > > file ? > > > > If so, then it is only usable starting from Kernel 3.3 according to this > > fix: > > http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/commit/?id=64be4c1c2428e148de6081af235e2418e6a66dda > > > > The value returned for kernel prior to 3.3 is really not better than the > > one from /proc/cpuinfo, so that's kind of a robbing Peter to pay Paul > > example... > > > > There is obviously a problem with the NUMA node ID detection right now > > since both /proc/cpuinfo and /sys can return incorrect values, however > > the node and cpu numbering is always good in the kernel boot log from > > what I know. So there might be a better way to find the real node ID > > whatever the kernel version used. > > > > Cheers, > > You need to use /sys/devices/system/cpu/cpuN/topology/physical_package_id > as documented in kernel (Documentation/cputopology.txt). > This was confirmed by several kernel developers including Andi Kleen from > Intel. The value in /proc/cpuinfo comes from the APCI tables and is the value > reported by the BIOS. There are machines that report socket 1 and 2. > Haven't played with older kernels, but the topology information in sysfs > has existed since 2.6.16. > > The final Intel solution was to use physical_packate_id and fall back > to /proc/cpuinfo as last resort.