From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 7ACCD5F34 for ; Thu, 28 Jun 2018 10:42:54 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 28 Jun 2018 01:42:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,282,1526367600"; d="scan'208";a="67989448" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.237.220.28]) ([10.237.220.28]) by fmsmga001.fm.intel.com with ESMTP; 28 Jun 2018 01:42:52 -0700 To: "Kumar, Ravi1" , "dev@dpdk.org" References: From: "Burakov, Anatoly" Message-ID: <0cde5e38-a27f-cf08-2fce-5494d774c4b3@intel.com> Date: Thu, 28 Jun 2018 09:42:51 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 08:42:54 -0000 On 28-Jun-18 8:03 AM, Kumar, Ravi1 wrote: >> On 22-Jun-18 5:37 PM, Kumar, Ravi1 wrote: >>> Hi, >>> >>> As the memory subsystem in DPDK 18.05 is reworked, it has introduced a problem for AMD EPYC 2P platforms. >>> The issue is that DPDK 18.05 only works with up to 4 NUMAs. For AMD EPYC 2P platforms, DPDK now only works with P0 (NUMA 0-3) and does not work with P1 (NUMA 4-7). >>> >>> The problem can be fixed by reducing some of the default settings of the memory subsystem. >>> >>> To solve this issue: >>> - We can create our own config file for our integrated 10G NIC, that is for amd_xgbe PMD. This will make amd_xgbe immune to this problem. >>> - However, when any other NIC (Intel, Mellanox, Cavium or Broadcom etc.) is plugged into NUMA 4-7, the problem will still be exposed. >>> - If we only fix it in "config/common_base", it will cover all cases. >>> >>> Our current workaround is: >>> Edit config file "./config/common_base" and change the following line >>> CONFIG_RTE_MAX_MEM_MB_PER_TYPE=131072 >>> TO >>> CONFIG_RTE_MAX_MEM_MB_PER_TYPE=65536 >>> >>> Any better solution for this issue is welcome. >>> >>> We would appreciate if this issue can be fixed in the next release (18.08) so the STOCK version of DPDK works on AMD EPYC 2P platforms. >>> >>> Regards, >>> Ravi >>> >> >> Hi Ravi, >> >> What is the reason behind this limitation? Is it too much virtual memory being preallocated? >> >> -- >> Thanks, >> Anatoly >> > Hi Anatoly, > > We believe this is true. By default, too much virtual memory is being preallocated. The result is it can only support up to 4 NUMAs. > > Our workaround is to reduce the amount of preallocated virtual memory by half, so to support up to 8 NUMAs. > > Regards, > Ravi > I assume you see a bunch of failed mmap() calls with ENOMEM? In general, changing base config that way is an OK change, and it won't even be an ABI break since this memory is allocated at runtime. I just want to make sure that we fix the underlying problem, rather than the symptom. -- Thanks, Anatoly