From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from rcdn-iport-5.cisco.com (rcdn-iport-5.cisco.com [173.37.86.76]) by dpdk.org (Postfix) with ESMTP id D6610B475 for ; Thu, 5 Feb 2015 16:41:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=14593; q=dns/txt; s=iport; t=1423150887; x=1424360487; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=OlSctjIH9wS26ulSnb9RNeZov6n1lODAd2Z3BeR3P+g=; b=a9fckFDqfIgCdhL+ukyM7JgBlvmBBk8KN2lcX47dJvhR6V22JcR1lw+T Ps+6SSjFu/vyOH9ZjImYNSqCJqfF1To+xBfHTp4pRByWry2f3bFt/8cg5 WiisieU0greWxJc7JQVdKxJ8mpbxp5oc0R+PiF7hoJqZrOYHCnzxeWC0d Q=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ak8FAPSN01StJA2I/2dsb2JhbABagmQigSsEyDkCgSVDAQEBAQF9hAwBAQEDATo/BQsCAQgOCh4QMiUCBA4FiCUIAdZ9AQEBAQEBAQEBAQEBAQEBAQEBAQEBF48hJDMHgxaBEwWPFIkqgReOFYM+IoF/H4FQb4EDQX4BAQE X-IronPort-AV: E=Sophos;i="5.09,524,1418083200"; d="scan'208";a="393708226" Received: from alln-core-3.cisco.com ([173.36.13.136]) by rcdn-iport-5.cisco.com with ESMTP; 05 Feb 2015 15:41:25 +0000 Received: from xhc-rcd-x13.cisco.com (xhc-rcd-x13.cisco.com [173.37.183.87]) by alln-core-3.cisco.com (8.14.5/8.14.5) with ESMTP id t15FfP6M007989 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Thu, 5 Feb 2015 15:41:25 GMT Received: from xmb-aln-x01.cisco.com ([fe80::747b:83e1:9755:d453]) by xhc-rcd-x13.cisco.com ([173.37.183.87]) with mapi id 14.03.0195.001; Thu, 5 Feb 2015 09:41:25 -0600 From: "Damjan Marion (damarion)" To: Neil Horman Thread-Topic: [dpdk-dev] mmap fails with more than 40000 hugepages Thread-Index: AQHQQTthrW63bl6xlkWr3i+nt5poHJziaVwAgAAFpoCAABvOAIAAC7GA Date: Thu, 5 Feb 2015 15:41:25 +0000 Message-ID: <6F7BF2F5-FBE9-4A52-86E6-813E2BE1840A@cisco.com> References: <736BD68D-C5DF-4883-A720-DAD8A2A866BE@cisco.com> <20150205125948.GC28355@hmsreliant.think-freely.org> <2F27D954-A245-45DC-A1BE-0CA3E17AAD3B@cisco.com> <20150205145932.GD28355@hmsreliant.think-freely.org> In-Reply-To: <20150205145932.GD28355@hmsreliant.think-freely.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.55.15.242] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] mmap fails with more than 40000 hugepages X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 15:41:27 -0000 > On 05 Feb 2015, at 15:59, Neil Horman wrote: >=20 > On Thu, Feb 05, 2015 at 01:20:01PM +0000, Damjan Marion (damarion) wrote: >>=20 >>> On 05 Feb 2015, at 13:59, Neil Horman wrote: >>>=20 >>> On Thu, Feb 05, 2015 at 12:00:48PM +0000, Damjan Marion (damarion) wrot= e: >>>> Hi, >>>>=20 >>>> I have system with 2 NUMA nodes and 256G RAM total. I noticed that DPD= K crashes in rte_eal_init() >>>> when number of available hugepages is around 40000 or above. >>>> Everything works fine with lower values (i.e. 30000). >>>>=20 >>>> I also tried with allocating 40000 on node0 and 0 on node1, same crash= happens. >>>>=20 >>>>=20 >>>> Any idea what might be causing this? >>>>=20 >>>> Thanks, >>>>=20 >>>> Damjan >>>>=20 >>>>=20 >>>> $ cat /sys/devices/system/node/node[01]/hugepages/hugepages-2048kB/nr_= hugepages >>>> 20000 >>>> 20000 >>>>=20 >>>> $ grep -i huge /proc/meminfo >>>> AnonHugePages: 706560 kB >>>> HugePages_Total: 40000 >>>> HugePages_Free: 40000 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>>=20 >>> Whats your shmmax value set to? 40000 2MB hugepages is way above the de= fault >>> setting for how much shared ram a system will allow. I've not done the= math on >>> your logs below, but judging by the size of some of the mapped segments= , I'm >>> betting your hitting the default limit of 4GB. >>=20 >> $ cat /proc/sys/kernel/shmmax >> 33554432 >>=20 >> $ sysctl -w kernel.shmmax=3D8589934592 >> kernel.shmmax =3D 8589934592 >>=20 >> same crash :( >>=20 >> Thanks, >>=20 >> Damjan >=20 > What about the shmmni and shmmax values. The shmmax value will also need= to be > set to at least 80G (more if you have other shared memory needs), and shm= mni > will need to be larger than 40,000 to handle all the segments your creati= ng. > Neil Hmm, if that is the reason, then 30000 will also not work. as both values a= re set by default to much much lower value. Anyway, I tried: $ sysctl kernel.shmmni kernel.shmmni =3D 100000 $ sysctl kernel.shmmax kernel.shmmax =3D 274877906944 $ cat /sys/devices/system/node/node[01]/hugepages/hugepages-2048kB/nr_hugep= ages 20000 20000 $ sudo ~/src/dpdk/x86_64-native-linuxapp-gcc/app/testpmd -l 5-7 -n 3 --sock= et-mem 512,512 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Detected lcore 4 as core 4 on socket 0 EAL: Detected lcore 5 as core 5 on socket 0 EAL: Detected lcore 6 as core 6 on socket 0 EAL: Detected lcore 7 as core 7 on socket 0 EAL: Detected lcore 8 as core 0 on socket 1 EAL: Detected lcore 9 as core 1 on socket 1 EAL: Detected lcore 10 as core 2 on socket 1 EAL: Detected lcore 11 as core 3 on socket 1 EAL: Detected lcore 12 as core 4 on socket 1 EAL: Detected lcore 13 as core 5 on socket 1 EAL: Detected lcore 14 as core 6 on socket 1 EAL: Detected lcore 15 as core 7 on socket 1 EAL: Detected lcore 16 as core 0 on socket 0 EAL: Detected lcore 17 as core 1 on socket 0 EAL: Detected lcore 18 as core 2 on socket 0 EAL: Detected lcore 19 as core 3 on socket 0 EAL: Detected lcore 20 as core 4 on socket 0 EAL: Detected lcore 21 as core 5 on socket 0 EAL: Detected lcore 22 as core 6 on socket 0 EAL: Detected lcore 23 as core 7 on socket 0 EAL: Detected lcore 24 as core 0 on socket 1 EAL: Detected lcore 25 as core 1 on socket 1 EAL: Detected lcore 26 as core 2 on socket 1 EAL: Detected lcore 27 as core 3 on socket 1 EAL: Detected lcore 28 as core 4 on socket 1 EAL: Detected lcore 29 as core 5 on socket 1 EAL: Detected lcore 30 as core 6 on socket 1 EAL: Detected lcore 31 as core 7 on socket 1 EAL: Support maximum 128 logical core(s) by configuration. EAL: Detected 32 lcore(s) EAL: VFIO modules not all loaded, skip VFIO support... EAL: Setting up memory... EAL: Ask a virtual area of 0x80000000 bytes EAL: Virtual area found at 0x7fe280000000 (size =3D 0x80000000) EAL: Ask a virtual area of 0x80000000 bytes EAL: Virtual area found at 0x7fe1c0000000 (size =3D 0x80000000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe335c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x9400000 bytes EAL: Virtual area found at 0x7fe32c600000 (size =3D 0x9400000) EAL: Ask a virtual area of 0x5c00000 bytes EAL: Virtual area found at 0x7fe326800000 (size =3D 0x5c00000) EAL: Ask a virtual area of 0x38e00000 bytes EAL: Virtual area found at 0x7fe247000000 (size =3D 0x38e00000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe326400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x8000000 bytes EAL: Virtual area found at 0x7fe31e200000 (size =3D 0x8000000) EAL: Ask a virtual area of 0x815800000 bytes EAL: Virtual area found at 0x7fd9aa600000 (size =3D 0x815800000) EAL: Ask a virtual area of 0x1f400000 bytes EAL: Virtual area found at 0x7fd98b000000 (size =3D 0x1f400000) EAL: Ask a virtual area of 0xfd400000 bytes EAL: Virtual area found at 0x7fd88da00000 (size =3D 0xfd400000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31de00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31da00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31d600000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31d200000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31ce00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31ca00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31c600000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31c200000 (size =3D 0x200000) EAL: Ask a virtual area of 0x400000 bytes EAL: Virtual area found at 0x7fe31bc00000 (size =3D 0x400000) EAL: Ask a virtual area of 0x400000 bytes EAL: Virtual area found at 0x7fe31b600000 (size =3D 0x400000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31b200000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31ae00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31aa00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31a600000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe31a200000 (size =3D 0x200000) EAL: Ask a virtual area of 0x400000 bytes EAL: Virtual area found at 0x7fe319c00000 (size =3D 0x400000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe319800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe319400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe319000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe318c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe318800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe318400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe318000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe317c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe317800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe317400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe317000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe316c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe316800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe316400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe316000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe315c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe315800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe315400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe315000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe314c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe314800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe314400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe314000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe313c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe313800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe313400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe313000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe312c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe312800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe312400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe312000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe311c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe311800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe311400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe311000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe310c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe310800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe310400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x400000 bytes EAL: Virtual area found at 0x7fe30fe00000 (size =3D 0x400000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30fa00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30f600000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30f200000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30ee00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30ea00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30e600000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30e200000 (size =3D 0x200000) EAL: Ask a virtual area of 0x400000 bytes EAL: Virtual area found at 0x7fe30dc00000 (size =3D 0x400000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30d800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30d400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30d000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30cc00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30c800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30c400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30c000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30bc00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30b800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30b400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30b000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30ac00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30a800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30a400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe30a000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe309c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe309800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe309400000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe309000000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe308c00000 (size =3D 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fe308800000 (size =3D 0x200000) EAL: Ask a virtual area of 0x2f800000 bytes EAL: Virtual area found at 0x7fd85e000000 (size =3D 0x2f800000) EAL: Ask a virtual area of 0x7400000 bytes EAL: Virtual area found at 0x7fe301200000 (size =3D 0x7400000) EAL: Ask a virtual area of 0x9b5400000 bytes EAL: Virtual area found at 0x7fcea8a00000 (size =3D 0x9b5400000) EAL: map_all_hugepages(): mmap failed: Cannot allocate memory EAL: Failed to remap 2 MB pages PANIC in rte_eal_init(): Cannot init memory