From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 17971A053B; Thu, 6 Feb 2020 14:53:59 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 32D561C0BE; Thu, 6 Feb 2020 14:53:58 +0100 (CET) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id B323D1C0B9; Thu, 6 Feb 2020 14:53:56 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Feb 2020 05:53:55 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,409,1574150400"; d="scan'208";a="264609706" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.237.220.124]) ([10.237.220.124]) by fmsmga002.fm.intel.com with ESMTP; 06 Feb 2020 05:53:54 -0800 To: Byonggon Chun , dev@dpdk.org, users@dpdk.org References: <20191219130620.30ade5fa@hermes.lan> From: "Burakov, Anatoly" Message-ID: <9424a8d3-25cc-52e2-4cf0-9eb81817f8fb@intel.com> Date: Thu, 6 Feb 2020 13:53:54 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.4.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [dpdk-users] Should ''shmget" not be used to consume hugepages in DPDK? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 22-Dec-19 3:33 PM, Byonggon Chun wrote: > x-posting to dev mailing list. > > Hi all. > I'm Kubernetes contributors and I'm working to make container isolation of > hugepages that allows us to set hugepages limit per container cgroup. > (At this point, limits are set on pod level cgroup even though we asked > hugepages as the container level resource) > > I tore down testPMD and some parts of DPDK lib and I got a question after i > found there is no usage of "shmget" in DPDK. > > My question is that Should "shmget" not be used to consume hugepages in > DPDK? > And here is following questions: > 1) If we don't have to use "shmget", Why? Does it affect performance? > 2) If I use "shmget" to get hugepages, should I call "mlock" syscall for it? > > For more details, as I know there are three ways to consume hugepages in > kubernetes. > 1) shmget with SHM_HUGETLB > 2) mmap with hugetlbs filebacking > 3) mmap with MAP_ANONYMOUS | MAP_HUGETLB > > And I found that testPMD calls mlock syscall when it maps an anonymous > hugepages or external allocated > hugepages.https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916 > The reason we're not using shmget is not because it's "legacy" or something else. It's because it doesn't give the guarantees that we want to have in DPDK. Namely, that the same shared object in memory is mapped at the same addresses. Last time i checked, shmget doesn't allow to map things at a specific address - each process will have its own pointer to shared memory, and accesses to shared memory by pointer are not valid across process boundaries. This is contrary to DPDK's goals, because we want to avoid address translation when working with multiple processes (in that sense, DPDK's multiprocess is basically like having multiple threads). > Thanks. > > > > > > On Fri, Dec 20, 2019 at 9:42 PM Byonggon Chun > wrote: > >>> shmget is a legacy Unix API and there is no point in using it. >> >> Yeah, I agree with it, >> I also prefer to use mmap with hugetlbfs in a DPDK container. >> >> The reason why I started this mail thread is some DPDK users still use >> shmget to consume hugepages, and I just wanted to find a good rationale to >> convince them to use mmap. >> >> But, at this point, I have only one rationale : shmget is a legacy UINIX >> API. >> >> On Fri, Dec 20, 2019 at 6:06 AM Stephen Hemminger < >> stephen@networkplumber.org> wrote: >> >>> On Fri, 20 Dec 2019 01:23:50 +0900 >>> Byonggon Chun wrote: >>> >>>> Hi all. >>>> I'm Kubernetes contributors and I'm working to make container isolation >>> of >>>> hugepages that allows us to set hugepages limit per container cgroup. >>>> (At this point, limits are set on pod level cgroup even though we asked >>>> hugepages as the container level resource) >>>> >>>> I tore down testPMD and some parts of DPDK lib and I got a question >>> after i >>>> found there is no usage of "shmget" in DPDK. >>>> >>>> My question is that Should "shmget" not be used to consume hugepages in >>>> DPDK? >>>> And here is following questions: >>>> 1) If we don't have to use "shmget", Why? Does it affect performance? >>>> 2) If I use "shmget" to get hugepages, should I call "mlock" syscall >>> for it? >>>> >>>> For more details, as I know there are three ways to consume hugepages in >>>> kubernetes. >>>> 1) shmget with SHM_HUGETLB >>>> 2) mmap with hugetlbs filebacking >>>> 3) mmap with MAP_ANONYMOUS | MAP_HUGETLB >>>> >>>> And I found that testPMD calls mlock syscall when it maps an anonymous >>>> hugepages or external allocated hugepages. >>>> >>> https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896 >>>> >>> https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916 >>>> >>>> Thanks. >>> >>> shmget is a legacy Unix API and there is no point in using it. >>> For new applications libhugetlbfs is preferable. >>> >> > -- Thanks, Anatoly