[dpdk-users] can we reserve hugepage and not release

[dpdk-users] can we reserve hugepage and not release

[Jiany Wu]

Hello,

I met a scenario that, need to start and stop the container many times for
the hugepage. But after several times container start and stop, the
hugepage is not able to reserve.
Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
Is there anyway to make sure the hugepage is still kept continuous? Thanks
indeed.

Dave

Re: [dpdk-users] can we reserve hugepage and not release

[Thomas Monjalon]

18/09/2021 04:37, Jiany Wu:
> Hello,
> 
> I met a scenario that, need to start and stop the container many times for
> the hugepage. But after several times container start and stop, the
> hugepage is not able to reserve.
> Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> Is there anyway to make sure the hugepage is still kept continuous? Thanks
> indeed.

Interesting question.
I think we need to address it in the DPDK documentation.

Anatoly, Stephen, Bruce, any advice please?

RE: [dpdk-users] can we reserve hugepage and not release

[Burakov, Anatoly]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 11:04 AM
> To: Jiany Wu <wujianyue000@gmail.com>
> Cc: users@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; olivier.matz@6wind.com;
> dmitry.kozliuk@gmail.com; stephen@networkplumber.org; Mcnamara, John
> <john.mcnamara@intel.com>
> Subject: Re: [dpdk-users] can we reserve hugepage and not release
> 
> 18/09/2021 04:37, Jiany Wu:
> > Hello,
> >
> > I met a scenario that, need to start and stop the container many times
> > for the hugepage. But after several times container start and stop,
> > the hugepage is not able to reserve.
> > Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> > Is there anyway to make sure the hugepage is still kept continuous?
> > Thanks indeed.
> 
> Interesting question.
> I think we need to address it in the DPDK documentation.
> 
> Anatoly, Stephen, Bruce, any advice please?
> 

Hi,

From description, I don't quite understand what's the issue here. Is the problem about "contiguousness of memory", or is it about inability to reserve more hugepages? How are hugepages assigned to your container? Have you tried using --in-memory mode?

Thanks,
Anatoly

Re: [dpdk-users] can we reserve hugepage and not release

[Thomas Monjalon]

29/09/2021 12:16, Burakov, Anatoly:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 18/09/2021 04:37, Jiany Wu:
> > > Hello,
> > >
> > > I met a scenario that, need to start and stop the container many times
> > > for the hugepage. But after several times container start and stop,
> > > the hugepage is not able to reserve.
> > > Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> > > Is there anyway to make sure the hugepage is still kept continuous?
> > > Thanks indeed.
> > 
> > Interesting question.
> > I think we need to address it in the DPDK documentation.
> > 
> > Anatoly, Stephen, Bruce, any advice please?
> > 
> 
> Hi,
> 
> From description, I don't quite understand what's the issue here. Is the problem about "contiguousness of memory", or is it about inability to reserve more hugepages?

I think the issue is that sometimes some pages are not properly released,
so we cannot reserve them again.
That's something I experienced myself.
Any trick to reset hugepages state?

> How are hugepages assigned to your container?
> Have you tried using --in-memory mode?

RE: [dpdk-users] can we reserve hugepage and not release

[Burakov, Anatoly]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 12:11 PM
> To: Jiany Wu <wujianyue000@gmail.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>
> Cc: users@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> olivier.matz@6wind.com; dmitry.kozliuk@gmail.com;
> stephen@networkplumber.org; Mcnamara, John
> <john.mcnamara@intel.com>
> Subject: Re: [dpdk-users] can we reserve hugepage and not release
> 
> 29/09/2021 12:16, Burakov, Anatoly:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 18/09/2021 04:37, Jiany Wu:
> > > > Hello,
> > > >
> > > > I met a scenario that, need to start and stop the container many
> > > > times for the hugepage. But after several times container start
> > > > and stop, the hugepage is not able to reserve.
> > > > Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> > > > Is there anyway to make sure the hugepage is still kept continuous?
> > > > Thanks indeed.
> > >
> > > Interesting question.
> > > I think we need to address it in the DPDK documentation.
> > >
> > > Anatoly, Stephen, Bruce, any advice please?
> > >
> >
> > Hi,
> >
> > From description, I don't quite understand what's the issue here. Is the
> problem about "contiguousness of memory", or is it about inability to reserve
> more hugepages?
> 
> I think the issue is that sometimes some pages are not properly released, so
> we cannot reserve them again.
> That's something I experienced myself.
> Any trick to reset hugepages state?

[[AB]] 
Under normal circumstances, the hugepage files would just be deleted and recreated the next time a primary process initializes even if there are leftover pages.

That said, assuming the pages are not "released" because DPDK has left a few files behind and a new container doesn't have permissions to delete them (or can't for some other reason), then there is very little DPDK can do other than track down all memory leaks. For example, using RX callback feature is intentionally causing a memory leak because by default the API will not free the callback structure as we cannot guarantee that there are no Dataplane threads still using that structure. Individual drivers/libraries might also leak memory due to e.g. secondary process usage. For example, in the general case, if something is meant to be shared by primary and secondary, but primary cannot free() it because it might still be used by secondaries, and secondaries have no way of verifying whether the resource is being used by other processes, or can be freed.

Put it simply, sometimes DPDK may leak memory either out of necessity, or even intentionally, to avoid potential use-after-free errors, and there's no general way to solve this problem that I'm aware of. This is the kind of stuff --in-memory mode was introduced to address, as when hugepages are in memory, there can never be leaks because we never touch the filesystem in the first place, so if there is a recommended way to address this at all, it would be --in-memory mode. 

> 
> > How are hugepages assigned to your container?
> > Have you tried using --in-memory mode?
> 
> 

Re: [dpdk-users] can we reserve hugepage and not release

[Jiany Wu]

Thanks for your explanation. Will try in memory mode.;)

Burakov, Anatoly <anatoly.burakov@intel.com>于2021年9月29日 周三下午8:40写道：

> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Wednesday, September 29, 2021 12:11 PM
> > To: Jiany Wu <wujianyue000@gmail.com>; Burakov, Anatoly
> > <anatoly.burakov@intel.com>
> > Cc: users@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> > olivier.matz@6wind.com; dmitry.kozliuk@gmail.com;
> > stephen@networkplumber.org; Mcnamara, John
> > <john.mcnamara@intel.com>
> > Subject: Re: [dpdk-users] can we reserve hugepage and not release
> >
> > 29/09/2021 12:16, Burakov, Anatoly:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 18/09/2021 04:37, Jiany Wu:
> > > > > Hello,
> > > > >
> > > > > I met a scenario that, need to start and stop the container many
> > > > > times for the hugepage. But after several times container start
> > > > > and stop, the hugepage is not able to reserve.
> > > > > Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> > > > > Is there anyway to make sure the hugepage is still kept continuous?
> > > > > Thanks indeed.
> > > >
> > > > Interesting question.
> > > > I think we need to address it in the DPDK documentation.
> > > >
> > > > Anatoly, Stephen, Bruce, any advice please?
> > > >
> > >
> > > Hi,
> > >
> > > From description, I don't quite understand what's the issue here. Is
> the
> > problem about "contiguousness of memory", or is it about inability to
> reserve
> > more hugepages?
> >
> > I think the issue is that sometimes some pages are not properly
> released, so
> > we cannot reserve them again.
> > That's something I experienced myself.
> > Any trick to reset hugepages state?
>
> [[AB]]
> Under normal circumstances, the hugepage files would just be deleted and
> recreated the next time a primary process initializes even if there are
> leftover pages.
>
> That said, assuming the pages are not "released" because DPDK has left a
> few files behind and a new container doesn't have permissions to delete
> them (or can't for some other reason), then there is very little DPDK can
> do other than track down all memory leaks. For example, using RX callback
> feature is intentionally causing a memory leak because by default the API
> will not free the callback structure as we cannot guarantee that there are
> no Dataplane threads still using that structure. Individual
> drivers/libraries might also leak memory due to e.g. secondary process
> usage. For example, in the general case, if something is meant to be shared
> by primary and secondary, but primary cannot free() it because it might
> still be used by secondaries, and secondaries have no way of verifying
> whether the resource is being used by other processes, or can be freed.
>
> Put it simply, sometimes DPDK may leak memory either out of necessity, or
> even intentionally, to avoid potential use-after-free errors, and there's
> no general way to solve this problem that I'm aware of. This is the kind of
> stuff --in-memory mode was introduced to address, as when hugepages are in
> memory, there can never be leaks because we never touch the filesystem in
> the first place, so if there is a recommended way to address this at all,
> it would be --in-memory mode.
>
> >
> > > How are hugepages assigned to your container?
> > > Have you tried using --in-memory mode?
> >
> >
>
>