Re: [dpdk-dev] [dpdk-users] Should ''shmget" not be used to consume hugepages in DPDK?

Re: [dpdk-dev] [dpdk-users] Should ''shmget" not be used to consume hugepages in DPDK?

[Byonggon Chun]

x-posting to dev mailing list.

Hi all.
I'm Kubernetes contributors and I'm working to make container isolation of
hugepages that allows us to set hugepages limit per container cgroup.
(At this point, limits are set on pod level cgroup even though we asked
hugepages as the container level resource)

I tore down testPMD and some parts of DPDK lib and I got a question after i
found there is no usage of "shmget" in DPDK.

My question is that Should "shmget" not be used to consume hugepages in
DPDK?
And here is following questions:
1) If we don't have to use "shmget", Why? Does it affect performance?
2) If I use "shmget" to get hugepages, should I call "mlock" syscall for it?

For more details, as I know there are three ways to consume hugepages in
kubernetes.
1) shmget with SHM_HUGETLB
2) mmap with hugetlbs filebacking
3) mmap with MAP_ANONYMOUS | MAP_HUGETLB

And I found that testPMD calls mlock syscall when it maps an anonymous
hugepages or external allocated
hugepages.https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916

Thanks.





On Fri, Dec 20, 2019 at 9:42 PM Byonggon Chun <byonggonchun@gmail.com>
wrote:

> > shmget is a legacy Unix API and there is no point in using it.
>
> Yeah, I agree with it,
> I also prefer to use mmap with hugetlbfs in a DPDK container.
>
> The reason why I started this mail thread is some DPDK users still use
> shmget to consume hugepages, and I just wanted to find a good rationale to
> convince them to use mmap.
>
> But, at this point, I have only one rationale : shmget is a legacy UINIX
> API.
>
> On Fri, Dec 20, 2019 at 6:06 AM Stephen Hemminger <
> stephen@networkplumber.org> wrote:
>
>> On Fri, 20 Dec 2019 01:23:50 +0900
>> Byonggon Chun <byonggonchun@gmail.com> wrote:
>>
>> > Hi all.
>> > I'm Kubernetes contributors and I'm working to make container isolation
>> of
>> > hugepages that allows us to set hugepages limit per container cgroup.
>> > (At this point, limits are set on pod level cgroup even though we asked
>> > hugepages as the container level resource)
>> >
>> > I tore down testPMD and some parts of DPDK lib and I got a question
>> after i
>> > found there is no usage of "shmget" in DPDK.
>> >
>> > My question is that Should "shmget" not be used to consume hugepages in
>> > DPDK?
>> > And here is following questions:
>> > 1) If we don't have to use "shmget", Why? Does it affect performance?
>> > 2) If I use "shmget" to get hugepages, should I call "mlock" syscall
>> for it?
>> >
>> > For more details, as I know there are three ways to consume hugepages in
>> > kubernetes.
>> > 1) shmget with SHM_HUGETLB
>> > 2) mmap with hugetlbs filebacking
>> > 3) mmap with MAP_ANONYMOUS | MAP_HUGETLB
>> >
>> > And I found that testPMD calls mlock syscall when it maps an anonymous
>> > hugepages or external allocated hugepages.
>> >
>> https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896
>> >
>> https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916
>> >
>> > Thanks.
>>
>> shmget is a legacy Unix API and there is no point in using it.
>> For new applications libhugetlbfs is preferable.
>>
>

Re: [dpdk-dev] [dpdk-users] Should ''shmget" not be used to consume hugepages in DPDK?

[Burakov, Anatoly]

On 22-Dec-19 3:33 PM, Byonggon Chun wrote:
> x-posting to dev mailing list.
> 
> Hi all.
> I'm Kubernetes contributors and I'm working to make container isolation of
> hugepages that allows us to set hugepages limit per container cgroup.
> (At this point, limits are set on pod level cgroup even though we asked
> hugepages as the container level resource)
> 
> I tore down testPMD and some parts of DPDK lib and I got a question after i
> found there is no usage of "shmget" in DPDK.
> 
> My question is that Should "shmget" not be used to consume hugepages in
> DPDK?
> And here is following questions:
> 1) If we don't have to use "shmget", Why? Does it affect performance?
> 2) If I use "shmget" to get hugepages, should I call "mlock" syscall for it?
> 
> For more details, as I know there are three ways to consume hugepages in
> kubernetes.
> 1) shmget with SHM_HUGETLB
> 2) mmap with hugetlbs filebacking
> 3) mmap with MAP_ANONYMOUS | MAP_HUGETLB
> 
> And I found that testPMD calls mlock syscall when it maps an anonymous
> hugepages or external allocated
> hugepages.https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916
> 

The reason we're not using shmget is not because it's "legacy" or 
something else. It's because it doesn't give the guarantees that we want 
to have in DPDK. Namely, that the same shared object in memory is mapped 
at the same addresses. Last time i checked, shmget doesn't allow to map 
things at a specific address - each process will have its own pointer to 
shared memory, and accesses to shared memory by pointer are not valid 
across process boundaries. This is contrary to DPDK's goals, because we 
want to avoid address translation when working with multiple processes 
(in that sense, DPDK's multiprocess is basically like having multiple 
threads).

> Thanks.
> 
> 
> 
> 
> 
> On Fri, Dec 20, 2019 at 9:42 PM Byonggon Chun <byonggonchun@gmail.com>
> wrote:
> 
>>> shmget is a legacy Unix API and there is no point in using it.
>>
>> Yeah, I agree with it,
>> I also prefer to use mmap with hugetlbfs in a DPDK container.
>>
>> The reason why I started this mail thread is some DPDK users still use
>> shmget to consume hugepages, and I just wanted to find a good rationale to
>> convince them to use mmap.
>>
>> But, at this point, I have only one rationale : shmget is a legacy UINIX
>> API.
>>
>> On Fri, Dec 20, 2019 at 6:06 AM Stephen Hemminger <
>> stephen@networkplumber.org> wrote:
>>
>>> On Fri, 20 Dec 2019 01:23:50 +0900
>>> Byonggon Chun <byonggonchun@gmail.com> wrote:
>>>
>>>> Hi all.
>>>> I'm Kubernetes contributors and I'm working to make container isolation
>>> of
>>>> hugepages that allows us to set hugepages limit per container cgroup.
>>>> (At this point, limits are set on pod level cgroup even though we asked
>>>> hugepages as the container level resource)
>>>>
>>>> I tore down testPMD and some parts of DPDK lib and I got a question
>>> after i
>>>> found there is no usage of "shmget" in DPDK.
>>>>
>>>> My question is that Should "shmget" not be used to consume hugepages in
>>>> DPDK?
>>>> And here is following questions:
>>>> 1) If we don't have to use "shmget", Why? Does it affect performance?
>>>> 2) If I use "shmget" to get hugepages, should I call "mlock" syscall
>>> for it?
>>>>
>>>> For more details, as I know there are three ways to consume hugepages in
>>>> kubernetes.
>>>> 1) shmget with SHM_HUGETLB
>>>> 2) mmap with hugetlbs filebacking
>>>> 3) mmap with MAP_ANONYMOUS | MAP_HUGETLB
>>>>
>>>> And I found that testPMD calls mlock syscall when it maps an anonymous
>>>> hugepages or external allocated hugepages.
>>>>
>>> https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L896
>>>>
>>> https://github.com/DPDK/dpdk/blob/924e55fb340623f03fdf2ff7fbcfd78819d1db25/app/test-pmd/testpmd.c#L916
>>>>
>>>> Thanks.
>>>
>>> shmget is a legacy Unix API and there is no point in using it.
>>> For new applications libhugetlbfs is preferable.
>>>
>>
> 


-- 
Thanks,
Anatoly