From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id B0762235 for ; Mon, 16 Jul 2018 12:35:23 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jul 2018 03:35:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,360,1526367600"; d="scan'208";a="240632554" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.237.220.102]) ([10.237.220.102]) by orsmga005.jf.intel.com with ESMTP; 16 Jul 2018 03:35:13 -0700 To: "Kumar, Ravi1" , "dev@dpdk.org" References: <0cde5e38-a27f-cf08-2fce-5494d774c4b3@intel.com> From: "Burakov, Anatoly" Message-ID: <8498d7f8-e82e-6c7b-c21a-b8bfe6ffa315@intel.com> Date: Mon, 16 Jul 2018 11:35:11 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jul 2018 10:35:24 -0000 On 14-Jul-18 10:44 AM, Kumar, Ravi1 wrote: > > Memory setup with 2M pages works with the default configuration.  With > the default configuration and 2M hugepages > > 1.            Total amount of memory for each NUMA zone does not exceed > 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE). > > 2.            Total number of segment lists per NUMA is limited to 32768 > (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).   This constraint is met for each numa > zone.  This is the limiting factor for memory per numa with 2M hugepages > and the default configuration. > > 3.            The data structures are capable of supporting 64G of > memory for each numa zone (32768 segments * 2M hugepagesize). > > 4.            8 NUMA zones * 64G = 512G.   Therefore the total for all > numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB). > > 5.            Resources are capable of allocating up to 64G per NUMA > zone.  Things will work as long as there are enough 2M hugepages  to > cover the memory  needs of the DPDK applications AND no memory zone > needs more than 64G. > > With the default configuration and 1G hugepages > > 1.            Total amount of memory for each NUMA zone is limited to > 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE).  This constraint is hit for each > numa zone.  This is the limiting factor for memory per numa. > > 2.            Total number of segment lists (128) does not exceed 32768 > (CONFIG_RTE_MAX_MEMSEG_PER_TYPE).    There are 128 segments per NUMA. > > 3.            The data structures are capable of supporting 128G of > memory for each numa zone (128 segments * 1G hugepagesize).     However, > only the first four NUMA zones get initialized before we hit > CONFIG_RTE_MAX_MEM_MB (512G). > > 4.            The total for all numa zones is limited to 512G > (CONFIG_RTE_MAX_MEM_MB).  This  limit is  hit after configuring the > first four NUMA zones (4 x 128G = 512G).   The rest of the NUMA zones > cannot allocate memory. > > Apparently, it is intended to support max 8 NUMAs by default > (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it can > only support up to 4 NUMAs. > > Possible workarounds when using 1G hugepages: > > 1.            Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit of > 64G per NUMA zone).  This is probably the best option unless you need a > lot of memory in any given NUMA. > > 2.            Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576. Hi Ravi, OK this makes it much clearer, thanks! I think the first one should be done. I think 64G per NUMA node is still a reasonable amount of memory and it makes the default work (i think we can go as far as reducing this limit to 32G per type!), and whoever has issues with it can change CONFIG_RTE_MAX_MEM_MB_PER_TYPE or CONFIG_RTE_MAX_MEM_MB for their use case. That's what these options are there for :) -- Thanks, Anatoly