From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout3.w1.samsung.com (mailout3.w1.samsung.com [210.118.77.13]) by dpdk.org (Postfix) with ESMTP id B90BE378E; Thu, 16 Feb 2017 14:57:40 +0100 (CET) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout3.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OLH00L3Z043KR10@mailout3.w1.samsung.com>; Thu, 16 Feb 2017 13:57:39 +0000 (GMT) Received: from eusmges5.samsung.com (unknown [203.254.199.245]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170216135738eucas1p11fe9fe199e328585bb6f5a1c31cc0e23~jySQPXhI42451524515eucas1p1B; Thu, 16 Feb 2017 13:57:38 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges5.samsung.com (EUCPMTA) with SMTP id 2F.AD.17477.2DFA5A85; Thu, 16 Feb 2017 13:57:38 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170216135737eucas1p28ef1ccb03d5dcc3bef789750a7d7969e~jySPidTNk2679426794eucas1p2E; Thu, 16 Feb 2017 13:57:37 +0000 (GMT) X-AuditID: cbfec7f5-f79d06d000004445-d4-58a5afd2f2a7 Received: from eusync1.samsung.com ( [203.254.199.211]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id 34.63.06687.220B5A85; Thu, 16 Feb 2017 13:58:58 +0000 (GMT) Received: from [106.109.129.180] by eusync1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OLH00KPI0405390@eusync1.samsung.com>; Thu, 16 Feb 2017 13:57:37 +0000 (GMT) To: "Tan, Jianfeng" , "dev@dpdk.org" , David Marchand , "Gonzalez Monroy, Sergio" Cc: Heetae Ahn , Yuanhan Liu , Neil Horman , "Pei, Yulong" , "stable@dpdk.org" From: Ilya Maximets Message-id: <52ccca5d-6b83-b8b0-66e0-bdc6ac000798@samsung.com> Date: Thu, 16 Feb 2017 16:57:36 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-version: 1.0 In-reply-to: <920faebb-7042-2223-26ac-84e6dd02b13e@samsung.com> Content-type: text/plain; charset=windows-1252 Content-transfer-encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0hTYRjHeXfOzo7L1WleejJFmt0QM4WCg5QW9WHYh+qDKGmXoScduWmb s9Q+LBWd5jXFTEE0nXjFUluGRjqHl1mmuTQtpJREpZS8J2J5PBP89nve5/88f/4PL4mJK/hO pFwZw6iUskgJIcQNXX/7T35q0Ad5VSY50lW6VJyeW3jNowsWvwpoS+pfAf24eImgx5J6Cboq x4ToTd2GgB7JGeDTK99XsfNC6WD2NyRdL9XzpeVtMzxpSe81aVZzDZJaVmvRVeK68GwYEymP ZVSnfG8LI8ZWK7DoYacHY//yCC0yOqQjGxKo09C7XCLg2BEGxhuIdCQkxZQewZvpD3y2IaYW ETQN+e8MTD0fsooqEYwsjaJ0RG4V0wjycVZjR/mAoS+Lx2rsqRYEk7MdGFtgVA+COm0DxqoI ygPMtSbEsojyhbk/1dtuOHUUUiamCJYdqCBo+ZJBcJr9sJY3jrNmNpQfzBeGsM8Y5QWF+Z08 jl2hqe73thdQZgFUtr/DWD1QLtDYjnEBLsFCeSKfYzuY7W62pneGNF0Hj+MEyH2VIuD2JCMo y12zDviBefSz1WwvPDE8te4XgS5FzEmkYHyhJzi+AJMfMwXcsZ7xIE2fzMtBrkW74hTtylC0 K0MpwmqQPaNRK8IZ9RlPtUyh1ijDPUOjFI1o6+/0bXYvtyB9l48RUSSS2IqmmYogMV8Wq45T GBGQmMReFFGvDxKLwmRx8Ywq6pZKE8mojegQiUsOiNpKLYFiKlwWw9xlmGhGtdPlkTZOWqSr s5TtaQ0craq/43Z/OEsZGvzyiP6KwTnAPzUx/sbDVs/46J/y2hOmjPeWFo+ZzX1khGGis+Cm g0uAzGQp0y5eVE5dHszshGwy0xxiQcW4klnx7o/t1PzoOIYvJb+Vu51bv5dyXDFfe7BH8Kj6 F3RoNhIYd+/DtgPmee1CsARXR8i83TGVWvYf6rCqZTcDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrKIsWRmVeSWpSXmKPExsVy+t/xy7pKG5ZGGFz8pG6xoqOdxeLdp+1M FtM+32a3uNL+k92ie/YXNotbzSfZLFZMOMJo8a/jD7vF9QkXWC2+PfjO7MDlcbH/DqPHrwVL WT0W73nJ5DHvZKBH35ZVjB5Xvq9mDGCLcrPJSE1MSS1SSM1Lzk/JzEu3VQoNcdO1UFLIS8xN tVWK0PUNCVJSKEvMKQXyjAzQgINzgHuwkr5dglvGre9LmAuuSVXc+j+ZrYHxkGgXIyeHhICJ xLNFl9kgbDGJC/fWA9lcHEICSxgl1q/oZIRwXjBKHD74hgmkSljASmLb6T4mkISIwA5GiXPX b7JCVM1mktjRfpgdxGEWOMUoserca3aQFjYBHYlTq48wgti8AnYS7z6uZAWxWQRUJdoePQNb LioQITH/6SomiBpBiR+T77F0MXJwcArYS7yfEQNiMgvoSdy/qAVSwSwgL7F5zVvmCYwCs5A0 zEKomoWkagEj8ypGkdTS4tz03GJDveLE3OLSvHS95PzcTYzAKNx27OfmHYyXNgYfYhTgYFTi 4X2RuiRCiDWxrLgy9xCjBAezkghvxtqlEUK8KYmVValF+fFFpTmpxYcYTYE+mMgsJZqcD0wQ eSXxhiaG5paGRsYWFuZGRkrivCUfroQLCaQnlqRmp6YWpBbB9DFxcEo1MCYFNPzUuXTm+wwr uekm87UPft67Oabd529uro2F/WLZ3bPzNRT2y9xNXZJyK3Rio7e1lpKl6L+p6xf3SJc9+pS9 /BvL+092fK9NmZjf60o9aKpROuKgdJhVIN832ve4w5y1x6/PcJPeeuzrTrHZMnx2uVYc/Ke3 lJyQ32kWVfhSZdUun5VTFZVYijMSDbWYi4oTAWNERuXYAgAA X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170216135737eucas1p28ef1ccb03d5dcc3bef789750a7d7969e X-Msg-Generator: CA X-Sender-IP: 182.198.249.179 X-Local-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G+yCvOyEseyghOyekBtFbmdpbmVlcg==?= X-Global-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G1NhbXN1bmcgRWxlY3Ryb25pY3MbRW5naW5lZXI=?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= CMS-TYPE: 201P X-HopCount: 7 X-CMS-RootMailID: 20170216130139eucas1p2512567d6f5db9eaac5ee840b56bf920a X-RootMTR: 20170216130139eucas1p2512567d6f5db9eaac5ee840b56bf920a References: <1487250070-13973-1-git-send-email-i.maximets@samsung.com> <920faebb-7042-2223-26ac-84e6dd02b13e@samsung.com> Subject: Re: [dpdk-dev] [PATCH] mem: balanced allocation of hugepages X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Feb 2017 13:57:41 -0000 On 16.02.2017 16:55, Ilya Maximets wrote: > Hi, > > On 16.02.2017 16:26, Tan, Jianfeng wrote: >> Hi, >> >>> -----Original Message----- >>> From: Ilya Maximets [mailto:i.maximets@samsung.com] >>> Sent: Thursday, February 16, 2017 9:01 PM >>> To: dev@dpdk.org; David Marchand; Gonzalez Monroy, Sergio >>> Cc: Heetae Ahn; Yuanhan Liu; Tan, Jianfeng; Neil Horman; Pei, Yulong; Ilya >>> Maximets; stable@dpdk.org >>> Subject: [PATCH] mem: balanced allocation of hugepages >>> >>> Currently EAL allocates hugepages one by one not paying >>> attention from which NUMA node allocation was done. >>> >>> Such behaviour leads to allocation failure if number of >>> available hugepages for application limited by cgroups >>> or hugetlbfs and memory requested not only from the first >>> socket. >>> >>> Example: >>> # 90 x 1GB hugepages availavle in a system >>> >>> cgcreate -g hugetlb:/test >>> # Limit to 32GB of hugepages >>> cgset -r hugetlb.1GB.limit_in_bytes=34359738368 test >>> # Request 4GB from each of 2 sockets >>> cgexec -g hugetlb:test testpmd --socket-mem=4096,4096 ... >>> >>> EAL: SIGBUS: Cannot mmap more hugepages of size 1024 MB >>> EAL: 32 not 90 hugepages of size 1024 MB allocated >>> EAL: Not enough memory available on socket 1! >>> Requested: 4096MB, available: 0MB >>> PANIC in rte_eal_init(): >>> Cannot init memory >>> >>> This happens beacause all allocated pages are >>> on socket 0. >> >> For such an use case, why not just use "numactl --interleave=0,1 xxx"? > > Unfortunately, interleave policy doesn't work for me. I suspect kernel configuration > blocks this or I don't understand something in kernel internals. > I'm using 3.10 rt kernel from rhel7. > > I tried to set up MPOL_INTERLEAVE in code and it doesn't work for me. Your example > with numactl doesn't work too: > > # Limited to 8GB of hugepages > cgexec -g hugetlb:test testpmd --socket-mem=4096,4096 Sorry, cgexec -g hugetlb:test numactl --interleave=0,1 ./testpmd --socket-mem=4096,4096 .. > > EAL: Setting up physically contiguous memory... > EAL: SIGBUS: Cannot mmap more hugepages of size 1024 MB > EAL: 8 not 90 hugepages of size 1024 MB allocated > EAL: Hugepage /dev/hugepages/rtemap_0 is on socket 0 > EAL: Hugepage /dev/hugepages/rtemap_1 is on socket 0 > EAL: Hugepage /dev/hugepages/rtemap_2 is on socket 0 > EAL: Hugepage /dev/hugepages/rtemap_3 is on socket 0 > EAL: Hugepage /dev/hugepages/rtemap_4 is on socket 0 > EAL: Hugepage /dev/hugepages/rtemap_5 is on socket 0 > EAL: Hugepage /dev/hugepages/rtemap_6 is on socket 0 > EAL: Hugepage /dev/hugepages/rtemap_7 is on socket 0 > EAL: Not enough memory available on socket 1! Requested: 4096MB, available: 0MB > PANIC in rte_eal_init(): > Cannot init memory > > Also, using numactl will affect all the allocations in application. This may > cause additional unexpected issues. > >> >> Do you see use case like --socket-mem 2048,1024 and only three 1GB-hugepage are allowed? > > This case will work with my patch. > But the opposite one '--socket-mem=1024,2048' will fail. > To be clear, we need to allocate all required memory at first > from each numa node and then allocate all other available pages > in round-robin fashion. But such solution looks a little ugly. > > What do you think? > > Best regards, Ilya Maximets. > >