From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-we0-f175.google.com (mail-we0-f175.google.com [74.125.82.175]) by dpdk.org (Postfix) with ESMTP id 36E3F6934 for ; Fri, 9 May 2014 15:30:49 +0200 (CEST) Received: by mail-we0-f175.google.com with SMTP id t61so4014687wes.34 for ; Fri, 09 May 2014 06:30:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-type:content-transfer-encoding; bh=caRG7IFXREq9jEQdAo5J91NdzCbQkyZ3rsbdlzH7kGU=; b=NUsR/IGWQhLzK+tD4IAdzDmRuLX4u7M5fdFDXNuZV4thlYtp4D8fyCJ2Ee4J9Okiuh 45zdNCvpuccqGVgMpifaWvSXCiU0PvFY13woae/QXRdTaWWsNA9trPdLfExT67KNowFt ddoGSMhfRAd6PtzHtOgjcFGR3RWGP294u1aBmR0Y5/0kNWEc1WsxIsUjaYWJ6/DYI4i4 fcqjca2iFekiXijYWzqQrHkO5YT34U/Xd6bLVb2N5SVG6AHWAG/5JelVVtSZ/kdibkrZ gfSO+7347XlF70RBSVOS27HH7PX+tkKyj0Dq9kQqo1V2TlLcVwiS3hcd2rDaJD8mcbXS jhMA== X-Gm-Message-State: ALoCoQnNISdkLBVNDuFPpTTbffJGZacgNFavMy/RT3HYuvTmajUBSb82ATE6YvviX1TSga2jv7Zf X-Received: by 10.180.109.69 with SMTP id hq5mr3447823wib.30.1399642255398; Fri, 09 May 2014 06:30:55 -0700 (PDT) Received: from alcyon.dev.6wind.com (6wind.net2.nerim.net. [213.41.180.237]) by mx.google.com with ESMTPSA id vc2sm5505537wjc.2.2014.05.09.06.30.53 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 09 May 2014 06:30:54 -0700 (PDT) From: David Marchand To: dev@dpdk.org Date: Fri, 9 May 2014 15:30:42 +0200 Message-Id: <1399642242-19725-1-git-send-email-david.marchand@6wind.com> X-Mailer: git-send-email 1.7.10.4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v2] eal: change default per socket memory allocation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 May 2014 13:30:49 -0000 From: Didier Pallard Currently, if there is more memory in hugepages than the amount requested by dpdk application, the memory is allocated by taking as much memory as possible from each socket, starting from first one. For example if a system is configured with 8 GB in 2 sockets (4 GB per socket), and dpdk is requesting only 4GB of memory, all memory will be taken in socket 0 (that have exactly 4GB of free hugepages) even if some cores are configured on socket 1, and there are free hugepages on socket 1... Change this behaviour to allocate memory on all sockets where some cores are configured, spreading the memory amongst sockets using following ratio per socket: N° of cores configured on the socket / Total number of configured cores * requested memory This algorithm is used when memory amount is specified globally using -m option. Per socket memory allocation can always be done using --socket-mem option. Changes included in v2: - only update linux implementation as bsd looks not to be ready for numa - if new algorithm fails, then defaults to previous behaviour Signed-off-by: Didier Pallard Signed-off-by: David Marchand --- lib/librte_eal/linuxapp/eal/eal_memory.c | 50 +++++++++++++++++++++++++++--- 1 file changed, 45 insertions(+), 5 deletions(-) diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c index 73a6394..471dcfd 100644 --- a/lib/librte_eal/linuxapp/eal/eal_memory.c +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c @@ -881,13 +881,53 @@ calc_num_pages_per_socket(uint64_t * memory, if (num_hp_info == 0) return -1; - for (socket = 0; socket < RTE_MAX_NUMA_NODES && total_mem != 0; socket++) { - /* if specific memory amounts per socket weren't requested */ - if (internal_config.force_sockets == 0) { + /* if specific memory amounts per socket weren't requested */ + if (internal_config.force_sockets == 0) { + int cpu_per_socket[RTE_MAX_NUMA_NODES]; + size_t default_size, total_size; + unsigned lcore_id; + + /* Compute number of cores per socket */ + memset(cpu_per_socket, 0, sizeof(cpu_per_socket)); + RTE_LCORE_FOREACH(lcore_id) { + cpu_per_socket[rte_lcore_to_socket_id(lcore_id)]++; + } + + /* + * Automatically spread requested memory amongst detected sockets according + * to number of cores from cpu mask present on each socket + */ + total_size = internal_config.memory; + for (socket = 0; socket < RTE_MAX_NUMA_NODES && total_size != 0; socket++) { + + /* Set memory amount per socket */ + default_size = (internal_config.memory * cpu_per_socket[socket]) + / rte_lcore_count(); + + /* Limit to maximum available memory on socket */ + default_size = RTE_MIN(default_size, get_socket_mem_size(socket)); + + /* Update sizes */ + memory[socket] = default_size; + total_size -= default_size; + } + + /* + * If some memory is remaining, try to allocate it by getting all + * available memory from sockets, one after the other + */ + for (socket = 0; socket < RTE_MAX_NUMA_NODES && total_size != 0; socket++) { /* take whatever is available */ - memory[socket] = RTE_MIN(get_socket_mem_size(socket), - total_mem); + default_size = RTE_MIN(get_socket_mem_size(socket) - memory[socket], + total_size); + + /* Update sizes */ + memory[socket] += default_size; + total_size -= default_size; } + } + + for (socket = 0; socket < RTE_MAX_NUMA_NODES && total_mem != 0; socket++) { /* skips if the memory on specific socket wasn't requested */ for (i = 0; i < num_hp_info && memory[socket] != 0; i++){ hp_used[i].hugedir = hp_info[i].hugedir; -- 1.7.10.4