From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 1F7EA7CBD for ; Thu, 22 Mar 2018 10:04:36 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Mar 2018 02:04:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.48,343,1517904000"; d="scan'208";a="44327015" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.252.52.175]) ([10.252.52.175]) by orsmga002.jf.intel.com with ESMTP; 22 Mar 2018 02:04:34 -0700 To: gowrishankar muthukrishnan Cc: dev@dpdk.org, Bruce Richardson , Chao Zhu References: <18deafea-5662-88ef-2ddc-3a1970d67405@linux.vnet.ibm.com> From: "Burakov, Anatoly" Message-ID: <38ca660a-3cce-1599-f52e-04a9b878774d@intel.com> Date: Thu, 22 Mar 2018 09:04:32 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [PATCH 18.05 v4] eal: add function to return number of detected sockets X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Mar 2018 09:04:37 -0000 On 22-Mar-18 5:16 AM, gowrishankar muthukrishnan wrote: > On Wednesday 21 March 2018 03:54 PM, Burakov, Anatoly wrote: >> >>>> +    config->numa_node_count = max_socket_id + 1; >>> >>> In some IBM servers, socket ID number does not seem to be in >>> sequence. For an instance, 0 and 8 for a 2 node server. >>> >>> In this case, numa_node_count would mislead users if wrongly >>> understood by its variable name IMO (see below) >>>> +    RTE_LOG(INFO, EAL, "Detected %u NUMA nodes\n", >>>> config->numa_node_count); >>> >>> For an instance, reading above message would tell 'EAL detected 8 >>> nodes' in my server, but actually there are only two nodes. >>> >>> Could its name better be 'numa_node_id_max' ?. Also, we store in >>> actual count of numa nodes in _count variable. >>> >>> Also, there could be a case when there is no local memory available >>> to a numa node too. >>> >>> Thanks, >>> Gowrishankar >> >> The point of this patchset is to (pre)allocate memory only on existing >> sockets. >> >> If we don't know how many sockets there are, we are forced to >> preallocate VA space per each *possible* NUMA node - that is, reserve >> e.g. 8x128G of memory, 6 of which will go unused on a 2-socket system. >> We can't know if there is no memory on socket in advance, but we can >> at least avoid preallocating VA space for sockets that don't exist in >> the first place. >> > > Sounds good Anatoly. > May be, sysfs/ might help to confirm if a numa node has local memory ?. We can't go to sysfs every time we want to allocate memory, and we can't really depend on what sysfs tells us about availability of hugepages on a particular socket (assuming that's what you meant by "confirm if a numa node has local memory"). User may modify hugepage numbers for each socket at runtime, and suddenly we do (or don't) have memory on local socket. Therefore i think a better approach would be - if a socket exists (that is, if we can find lcores on that socket, even if they're not active), assume it has/had/will have memory, and store it as a valid socket id. I'll respin a v5 with changes outlined below then. Thanks! > > Anyway, for the context of this particular patch (return numa nodes), > below approach you mentioned is good. > >> How about we store all possible socket id's instead? e.g. something like: >> >> static int numa_node_ids[MAX_NUMA_NODES]; >> <...> >> int rte_eal_cpu_init() { >>     int sockets[RTE_MAX_LCORE]; >>     <...> >>     for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { >>         core_to_socket[lcore_id] = socket; > sockets[lcore_id] = eal_cpu_socket_id(lcore_id); >>     } >>     <...> >>     qsort(sockets); >>     <...> >>     // store all unique sockets in numa_node_ids in ascending order > > Just thinking that, is there a purpose of retaining a numa ID which does > not have local memory attached ? > but sockets[] is suppose to reflect all available nodes though (and > assuming, its calling place to ensure > for the existence of numa local memory). > > >> } >> <...> >> >> on a 2 socket system we then get: >> >> rte_num_sockets() => return 2 >> rte_get_socket_id(int idx) => return numa_node_ids[idx] > rte_get_socket_mem(idx) might help to validate for local memory existence ? > >> >> Would that be suitable? >> > > Thanks, > Gowrishankar > > -- Thanks, Anatoly