DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: Byonggon Chun <byonggonchun@gmail.com>, dev@dpdk.org, users@dpdk.org
Subject: Re: [dpdk-dev] [dpdk-users] Should ''shmget" not be used to consume hugepages in DPDK?
Date: Thu, 6 Feb 2020 13:53:54 +0000	[thread overview]
Message-ID: <9424a8d3-25cc-52e2-4cf0-9eb81817f8fb@intel.com> (raw)
In-Reply-To: <CACh8f0NaCdXTybyXHPh_DG9V_1Bthx=82hGrz6XDKrm1FjfQEQ@mail.gmail.com>

On 22-Dec-19 3:33 PM, Byonggon Chun wrote:
> x-posting to dev mailing list.
> 
> Hi all.
> I'm Kubernetes contributors and I'm working to make container isolation of
> hugepages that allows us to set hugepages limit per container cgroup.
> (At this point, limits are set on pod level cgroup even though we asked
> hugepages as the container level resource)
> 
> I tore down testPMD and some parts of DPDK lib and I got a question after i
> found there is no usage of "shmget" in DPDK.
> 
> My question is that Should "shmget" not be used to consume hugepages in
> DPDK?
> And here is following questions:
> 1) If we don't have to use "shmget", Why? Does it affect performance?
> 2) If I use "shmget" to get hugepages, should I call "mlock" syscall for it?
> 
> For more details, as I know there are three ways to consume hugepages in
> kubernetes.
> 1) shmget with SHM_HUGETLB
> 2) mmap with hugetlbs filebacking
> 3) mmap with MAP_ANONYMOUS | MAP_HUGETLB
> 
> And I found that testPMD calls mlock syscall when it maps an anonymous
> hugepages or external allocated
> hugepages.https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916
> 

The reason we're not using shmget is not because it's "legacy" or 
something else. It's because it doesn't give the guarantees that we want 
to have in DPDK. Namely, that the same shared object in memory is mapped 
at the same addresses. Last time i checked, shmget doesn't allow to map 
things at a specific address - each process will have its own pointer to 
shared memory, and accesses to shared memory by pointer are not valid 
across process boundaries. This is contrary to DPDK's goals, because we 
want to avoid address translation when working with multiple processes 
(in that sense, DPDK's multiprocess is basically like having multiple 
threads).

> Thanks.
> 
> 
> 
> 
> 
> On Fri, Dec 20, 2019 at 9:42 PM Byonggon Chun <byonggonchun@gmail.com>
> wrote:
> 
>>> shmget is a legacy Unix API and there is no point in using it.
>>
>> Yeah, I agree with it,
>> I also prefer to use mmap with hugetlbfs in a DPDK container.
>>
>> The reason why I started this mail thread is some DPDK users still use
>> shmget to consume hugepages, and I just wanted to find a good rationale to
>> convince them to use mmap.
>>
>> But, at this point, I have only one rationale : shmget is a legacy UINIX
>> API.
>>
>> On Fri, Dec 20, 2019 at 6:06 AM Stephen Hemminger <
>> stephen@networkplumber.org> wrote:
>>
>>> On Fri, 20 Dec 2019 01:23:50 +0900
>>> Byonggon Chun <byonggonchun@gmail.com> wrote:
>>>
>>>> Hi all.
>>>> I'm Kubernetes contributors and I'm working to make container isolation
>>> of
>>>> hugepages that allows us to set hugepages limit per container cgroup.
>>>> (At this point, limits are set on pod level cgroup even though we asked
>>>> hugepages as the container level resource)
>>>>
>>>> I tore down testPMD and some parts of DPDK lib and I got a question
>>> after i
>>>> found there is no usage of "shmget" in DPDK.
>>>>
>>>> My question is that Should "shmget" not be used to consume hugepages in
>>>> DPDK?
>>>> And here is following questions:
>>>> 1) If we don't have to use "shmget", Why? Does it affect performance?
>>>> 2) If I use "shmget" to get hugepages, should I call "mlock" syscall
>>> for it?
>>>>
>>>> For more details, as I know there are three ways to consume hugepages in
>>>> kubernetes.
>>>> 1) shmget with SHM_HUGETLB
>>>> 2) mmap with hugetlbs filebacking
>>>> 3) mmap with MAP_ANONYMOUS | MAP_HUGETLB
>>>>
>>>> And I found that testPMD calls mlock syscall when it maps an anonymous
>>>> hugepages or external allocated hugepages.
>>>>
>>> https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896
>>>>
>>> https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916
>>>>
>>>> Thanks.
>>>
>>> shmget is a legacy Unix API and there is no point in using it.
>>> For new applications libhugetlbfs is preferable.
>>>
>>
> 


-- 
Thanks,
Anatoly

      reply	other threads:[~2020-02-06 13:53 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CACh8f0M_W9NvAgqp-vc-KR57hXO6=tYeWbfqQNG+NytYxp6AsQ@mail.gmail.com>
     [not found] ` <20191219130620.30ade5fa@hermes.lan>
     [not found]   ` <CACh8f0OnKRo2nPt6pw8e5zJP366LXqxVN3Cgg5e1tTZHqC46_Q@mail.gmail.com>
2019-12-22 15:33     ` Byonggon Chun
2020-02-06 13:53       ` Burakov, Anatoly [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9424a8d3-25cc-52e2-4cf0-9eb81817f8fb@intel.com \
    --to=anatoly.burakov@intel.com \
    --cc=byonggonchun@gmail.com \
    --cc=dev@dpdk.org \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).