From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <i.maximets@samsung.com> Received: from mailout3.w1.samsung.com (mailout3.w1.samsung.com [210.118.77.13]) by dpdk.org (Postfix) with ESMTP id 51213378E; Thu, 16 Feb 2017 14:56:03 +0100 (CET) Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout3.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OLH00M1J01D1710@mailout3.w1.samsung.com>; Thu, 16 Feb 2017 13:56:01 +0000 (GMT) Received: from eusmges5.samsung.com (unknown [203.254.199.245]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170216135600eucas1p27c6031e17b8fa7deb136ee36a90100af~jyQ05qpKd1438814388eucas1p2W; Thu, 16 Feb 2017 13:56:00 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges5.samsung.com (EUCPMTA) with SMTP id 30.9D.17477.07FA5A85; Thu, 16 Feb 2017 13:56:00 +0000 (GMT) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170216135559eucas1p11eb7f2d5c72dd851254750485d723c6e~jyQ0RmPhA3021130211eucas1p17; Thu, 16 Feb 2017 13:55:59 +0000 (GMT) X-AuditID: cbfec7f5-f79d06d000004445-fb-58a5af70b372 Received: from eusync1.samsung.com ( [203.254.199.211]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 98.66.10233.47FA5A85; Thu, 16 Feb 2017 13:56:04 +0000 (GMT) Received: from [106.109.129.180] by eusync1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OLH00KME01A5390@eusync1.samsung.com>; Thu, 16 Feb 2017 13:55:59 +0000 (GMT) To: "Tan, Jianfeng" <jianfeng.tan@intel.com>, "dev@dpdk.org" <dev@dpdk.org>, David Marchand <david.marchand@6wind.com>, "Gonzalez Monroy, Sergio" <sergio.gonzalez.monroy@intel.com> Cc: Heetae Ahn <heetae82.ahn@samsung.com>, Yuanhan Liu <yuanhan.liu@linux.intel.com>, Neil Horman <nhorman@tuxdriver.com>, "Pei, Yulong" <yulong.pei@intel.com>, "stable@dpdk.org" <stable@dpdk.org> From: Ilya Maximets <i.maximets@samsung.com> Message-id: <920faebb-7042-2223-26ac-84e6dd02b13e@samsung.com> Date: Thu, 16 Feb 2017 16:55:57 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-version: 1.0 In-reply-to: <ED26CBA2FAD1BF48A8719AEF02201E3651138958@SHSMSX103.ccr.corp.intel.com> Content-type: text/plain; charset=windows-1252 Content-transfer-encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0hTcRjG+e+cnR1Hi9Oc7UULaVAfQqdRwcFECyRGEdmHSvpSSw/O2ubY 1LRQFJea90sXWWJKTm3Tltf2wUinOTfFtNTM8hKGmF0oLS8plttR8Nvv4X3e5+WBl8SENVxv MkYdx2jVcqWE4OOt3Sv9/hqLMSLwXpWIrs3KxOkf88859P2FDzx6KHOFR+c8/E3QY+kOgq4t 7EL0etYaj35XOMClF6eWsON82WDBRyT7W2Hkyh63feHIyh3nZPnNJiQbWjKjcOISPziKUcYk MNqAkCt8xfigmavpgcTZwhpOKlr2zEYeJFBHoCGri8fybhiYsBDZiE8KKSOC2+UVOCsWEPx8 k8bb2uhbXdt0VSPo+fRy0zWLoGUh1+3ypIKgtTef4xqIKCuC6bkOzCUwqgdBXaoFc7kIyg+c 5i7kYgEVAhPLTVwX49R+KJtpcLMXFQHW0VyC9eyC5ZIJ3MUe1AVYnetz52BUIJTe7eSw7AtN dd/dx4By8sCcpt9YJjfEXmhsx9gOYdBf4EAse8KcvXmz2x54W5KDs3wLiloyeGyOHkFl0TKX HYSC8/3w5rGdUNz6AGPzBZCVIWQtMrA9MxIsn4Dp13nufCH1GUFePVOIfA3b6hi2VTBsq1CB MBMSMfE6VTSjOyrVyVW6eHW0NDJW1Yg23qd33f7HiozdQTZEkUiyQzDLVEUIufIEXZLKhoDE JCKBot4YIRREyZNuMtrYy9p4JaOzIR8Sl4gFbRVDF4VUtDyOuc4wGka7NeWQHt6p6NW1cfpp n99E3Vk8t+2XwyJWPFrc1Xl6jKPvaPh2cqrZORYsuzOsKEuvSj6WP+nRKTdNvfg66V0mPVw8 GWI3evr5JM8QwoL2GXHkvDLDfzQl75/9vLg6W/PEJ6Cx0qkOczKhBxzSfUtTJv6N8FNeZ6yl 04mc9KsphhH9iHElWYLrFPJDBzGtTv4ffXghIDoDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrKIsWRmVeSWpSXmKPExsVy+t/xy7ol65dGGEz6zGKxoqOdxeLdp+1M FtM+32a3uNL+k92ie/YXNotbzSfZLFZMOMJo8a/jD7vF9QkXWC2+PfjO7MDlcbH/DqPHrwVL WT0W73nJ5DHvZKBH35ZVjB5Xvq9mDGCLcrPJSE1MSS1SSM1Lzk/JzEu3VQoNcdO1UFLIS8xN tVWK0PUNCVJSKEvMKQXyjAzQgINzgHuwkr5dglvG3YurWQtOSFS8mLCcqYHxh3AXIyeHhICJ xJnff9ggbDGJC/fWA9lcHEICSxglZq7dxAzhvGCUuLzuFjNIlbCAlcS2031MIAkRgR2MEueu 32SFqHrGKPH6NUSGWeAUo8Sqc6/ZQVrYBHQkTq0+wghi8wrYSdz7sZkVxGYRUJWY82wjmC0q ECEx/+kqJogaQYkfk++xdDFycHAKhEnMPyYEYjIL6Encv6gFUsEsIC+xec1b5gmMArOQNMxC qJqFpGoBI/MqRpHU0uLc9NxiI73ixNzi0rx0veT83E2MwCjcduznlh2MXe+CDzEKcDAq8fA6 ZCyJEGJNLCuuzD3EKMHBrCTCm7F2aYQQb0piZVVqUX58UWlOavEhRlOgDyYyS4km5wMTRF5J vKGJobmloZGxhYW5kZGSOO/UD1fChQTSE0tSs1NTC1KLYPqYODilGhgv2D+4q2LldmBjvPmD yU/n5l0VV/q16eXitrub2663L1H9+O5P39nzeiuj3wdqfvJr4fm475j8Vq6bCTbLQrw4J4TH L5mV/j287ZPxioiMaK3p7VZmjX8fVm42nx4yzXJ77/eQV5FN328UNrydn6k538XiS1umrveC /ZHOcoE2AnyfKq06VO4rsRRnJBpqMRcVJwIA9B2xI9gCAAA= X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170216135559eucas1p11eb7f2d5c72dd851254750485d723c6e X-Msg-Generator: CA X-Sender-IP: 182.198.249.180 X-Local-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G+yCvOyEseyghOyekBtFbmdpbmVlcg==?= X-Global-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G1NhbXN1bmcgRWxlY3Ryb25pY3MbRW5naW5lZXI=?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= CMS-TYPE: 201P X-HopCount: 7 X-CMS-RootMailID: 20170216130139eucas1p2512567d6f5db9eaac5ee840b56bf920a X-RootMTR: 20170216130139eucas1p2512567d6f5db9eaac5ee840b56bf920a References: <CGME20170216130139eucas1p2512567d6f5db9eaac5ee840b56bf920a@eucas1p2.samsung.com> <1487250070-13973-1-git-send-email-i.maximets@samsung.com> <ED26CBA2FAD1BF48A8719AEF02201E3651138958@SHSMSX103.ccr.corp.intel.com> Subject: Re: [dpdk-dev] [PATCH] mem: balanced allocation of hugepages X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Thu, 16 Feb 2017 13:56:03 -0000 Hi, On 16.02.2017 16:26, Tan, Jianfeng wrote: > Hi, > >> -----Original Message----- >> From: Ilya Maximets [mailto:i.maximets@samsung.com] >> Sent: Thursday, February 16, 2017 9:01 PM >> To: dev@dpdk.org; David Marchand; Gonzalez Monroy, Sergio >> Cc: Heetae Ahn; Yuanhan Liu; Tan, Jianfeng; Neil Horman; Pei, Yulong; Ilya >> Maximets; stable@dpdk.org >> Subject: [PATCH] mem: balanced allocation of hugepages >> >> Currently EAL allocates hugepages one by one not paying >> attention from which NUMA node allocation was done. >> >> Such behaviour leads to allocation failure if number of >> available hugepages for application limited by cgroups >> or hugetlbfs and memory requested not only from the first >> socket. >> >> Example: >> # 90 x 1GB hugepages availavle in a system >> >> cgcreate -g hugetlb:/test >> # Limit to 32GB of hugepages >> cgset -r hugetlb.1GB.limit_in_bytes=34359738368 test >> # Request 4GB from each of 2 sockets >> cgexec -g hugetlb:test testpmd --socket-mem=4096,4096 ... >> >> EAL: SIGBUS: Cannot mmap more hugepages of size 1024 MB >> EAL: 32 not 90 hugepages of size 1024 MB allocated >> EAL: Not enough memory available on socket 1! >> Requested: 4096MB, available: 0MB >> PANIC in rte_eal_init(): >> Cannot init memory >> >> This happens beacause all allocated pages are >> on socket 0. > > For such an use case, why not just use "numactl --interleave=0,1 <DPDK app> xxx"? Unfortunately, interleave policy doesn't work for me. I suspect kernel configuration blocks this or I don't understand something in kernel internals. I'm using 3.10 rt kernel from rhel7. I tried to set up MPOL_INTERLEAVE in code and it doesn't work for me. Your example with numactl doesn't work too: # Limited to 8GB of hugepages cgexec -g hugetlb:test testpmd --socket-mem=4096,4096 EAL: Setting up physically contiguous memory... EAL: SIGBUS: Cannot mmap more hugepages of size 1024 MB EAL: 8 not 90 hugepages of size 1024 MB allocated EAL: Hugepage /dev/hugepages/rtemap_0 is on socket 0 EAL: Hugepage /dev/hugepages/rtemap_1 is on socket 0 EAL: Hugepage /dev/hugepages/rtemap_2 is on socket 0 EAL: Hugepage /dev/hugepages/rtemap_3 is on socket 0 EAL: Hugepage /dev/hugepages/rtemap_4 is on socket 0 EAL: Hugepage /dev/hugepages/rtemap_5 is on socket 0 EAL: Hugepage /dev/hugepages/rtemap_6 is on socket 0 EAL: Hugepage /dev/hugepages/rtemap_7 is on socket 0 EAL: Not enough memory available on socket 1! Requested: 4096MB, available: 0MB PANIC in rte_eal_init(): Cannot init memory Also, using numactl will affect all the allocations in application. This may cause additional unexpected issues. > > Do you see use case like --socket-mem 2048,1024 and only three 1GB-hugepage are allowed? This case will work with my patch. But the opposite one '--socket-mem=1024,2048' will fail. To be clear, we need to allocate all required memory at first from each numa node and then allocate all other available pages in round-robin fashion. But such solution looks a little ugly. What do you think? Best regards, Ilya Maximets.