DPDK usage discussions
 help / color / mirror / Atom feed
* [dpdk-users] can we reserve hugepage and not release
@ 2021-09-18  2:37 Jiany Wu
  2021-09-29 10:04 ` Thomas Monjalon
  0 siblings, 1 reply; 6+ messages in thread
From: Jiany Wu @ 2021-09-18  2:37 UTC (permalink / raw)
  To: users

Hello,

I met a scenario that, need to start and stop the container many times for
the hugepage. But after several times container start and stop, the
hugepage is not able to reserve.
Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
Is there anyway to make sure the hugepage is still kept continuous? Thanks
indeed.

Dave

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-users] can we reserve hugepage and not release
  2021-09-18  2:37 [dpdk-users] can we reserve hugepage and not release Jiany Wu
@ 2021-09-29 10:04 ` Thomas Monjalon
  2021-09-29 10:16   ` Burakov, Anatoly
  0 siblings, 1 reply; 6+ messages in thread
From: Thomas Monjalon @ 2021-09-29 10:04 UTC (permalink / raw)
  To: Jiany Wu
  Cc: users, anatoly.burakov, bruce.richardson, olivier.matz,
	dmitry.kozliuk, stephen, john.mcnamara

18/09/2021 04:37, Jiany Wu:
> Hello,
> 
> I met a scenario that, need to start and stop the container many times for
> the hugepage. But after several times container start and stop, the
> hugepage is not able to reserve.
> Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> Is there anyway to make sure the hugepage is still kept continuous? Thanks
> indeed.

Interesting question.
I think we need to address it in the DPDK documentation.

Anatoly, Stephen, Bruce, any advice please?



^ permalink raw reply	[flat|nested] 6+ messages in thread

* RE: [dpdk-users] can we reserve hugepage and not release
  2021-09-29 10:04 ` Thomas Monjalon
@ 2021-09-29 10:16   ` Burakov, Anatoly
  2021-09-29 11:11     ` Thomas Monjalon
  0 siblings, 1 reply; 6+ messages in thread
From: Burakov, Anatoly @ 2021-09-29 10:16 UTC (permalink / raw)
  To: Thomas Monjalon, Jiany Wu
  Cc: users, Richardson, Bruce, olivier.matz, dmitry.kozliuk, stephen,
	Mcnamara, John

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 11:04 AM
> To: Jiany Wu <wujianyue000@gmail.com>
> Cc: users@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; olivier.matz@6wind.com;
> dmitry.kozliuk@gmail.com; stephen@networkplumber.org; Mcnamara, John
> <john.mcnamara@intel.com>
> Subject: Re: [dpdk-users] can we reserve hugepage and not release
> 
> 18/09/2021 04:37, Jiany Wu:
> > Hello,
> >
> > I met a scenario that, need to start and stop the container many times
> > for the hugepage. But after several times container start and stop,
> > the hugepage is not able to reserve.
> > Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> > Is there anyway to make sure the hugepage is still kept continuous?
> > Thanks indeed.
> 
> Interesting question.
> I think we need to address it in the DPDK documentation.
> 
> Anatoly, Stephen, Bruce, any advice please?
> 

Hi,

From description, I don't quite understand what's the issue here. Is the problem about "contiguousness of memory", or is it about inability to reserve more hugepages? How are hugepages assigned to your container? Have you tried using --in-memory mode?

Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-users] can we reserve hugepage and not release
  2021-09-29 10:16   ` Burakov, Anatoly
@ 2021-09-29 11:11     ` Thomas Monjalon
  2021-09-29 12:40       ` Burakov, Anatoly
  0 siblings, 1 reply; 6+ messages in thread
From: Thomas Monjalon @ 2021-09-29 11:11 UTC (permalink / raw)
  To: Jiany Wu, Burakov, Anatoly
  Cc: users, Richardson, Bruce, olivier.matz, dmitry.kozliuk, stephen,
	Mcnamara, John

29/09/2021 12:16, Burakov, Anatoly:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 18/09/2021 04:37, Jiany Wu:
> > > Hello,
> > >
> > > I met a scenario that, need to start and stop the container many times
> > > for the hugepage. But after several times container start and stop,
> > > the hugepage is not able to reserve.
> > > Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> > > Is there anyway to make sure the hugepage is still kept continuous?
> > > Thanks indeed.
> > 
> > Interesting question.
> > I think we need to address it in the DPDK documentation.
> > 
> > Anatoly, Stephen, Bruce, any advice please?
> > 
> 
> Hi,
> 
> From description, I don't quite understand what's the issue here. Is the problem about "contiguousness of memory", or is it about inability to reserve more hugepages?

I think the issue is that sometimes some pages are not properly released,
so we cannot reserve them again.
That's something I experienced myself.
Any trick to reset hugepages state?

> How are hugepages assigned to your container?
> Have you tried using --in-memory mode?




^ permalink raw reply	[flat|nested] 6+ messages in thread

* RE: [dpdk-users] can we reserve hugepage and not release
  2021-09-29 11:11     ` Thomas Monjalon
@ 2021-09-29 12:40       ` Burakov, Anatoly
  2021-10-05 15:56         ` Jiany Wu
  0 siblings, 1 reply; 6+ messages in thread
From: Burakov, Anatoly @ 2021-09-29 12:40 UTC (permalink / raw)
  To: Thomas Monjalon, Jiany Wu
  Cc: users, Richardson, Bruce, olivier.matz, dmitry.kozliuk, stephen,
	Mcnamara, John

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 12:11 PM
> To: Jiany Wu <wujianyue000@gmail.com>; Burakov, Anatoly
> <anatoly.burakov@intel.com>
> Cc: users@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> olivier.matz@6wind.com; dmitry.kozliuk@gmail.com;
> stephen@networkplumber.org; Mcnamara, John
> <john.mcnamara@intel.com>
> Subject: Re: [dpdk-users] can we reserve hugepage and not release
> 
> 29/09/2021 12:16, Burakov, Anatoly:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 18/09/2021 04:37, Jiany Wu:
> > > > Hello,
> > > >
> > > > I met a scenario that, need to start and stop the container many
> > > > times for the hugepage. But after several times container start
> > > > and stop, the hugepage is not able to reserve.
> > > > Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> > > > Is there anyway to make sure the hugepage is still kept continuous?
> > > > Thanks indeed.
> > >
> > > Interesting question.
> > > I think we need to address it in the DPDK documentation.
> > >
> > > Anatoly, Stephen, Bruce, any advice please?
> > >
> >
> > Hi,
> >
> > From description, I don't quite understand what's the issue here. Is the
> problem about "contiguousness of memory", or is it about inability to reserve
> more hugepages?
> 
> I think the issue is that sometimes some pages are not properly released, so
> we cannot reserve them again.
> That's something I experienced myself.
> Any trick to reset hugepages state?

[[AB]] 
Under normal circumstances, the hugepage files would just be deleted and recreated the next time a primary process initializes even if there are leftover pages.

That said, assuming the pages are not "released" because DPDK has left a few files behind and a new container doesn't have permissions to delete them (or can't for some other reason), then there is very little DPDK can do other than track down all memory leaks. For example, using RX callback feature is intentionally causing a memory leak because by default the API will not free the callback structure as we cannot guarantee that there are no Dataplane threads still using that structure. Individual drivers/libraries might also leak memory due to e.g. secondary process usage. For example, in the general case, if something is meant to be shared by primary and secondary, but primary cannot free() it because it might still be used by secondaries, and secondaries have no way of verifying whether the resource is being used by other processes, or can be freed.

Put it simply, sometimes DPDK may leak memory either out of necessity, or even intentionally, to avoid potential use-after-free errors, and there's no general way to solve this problem that I'm aware of. This is the kind of stuff --in-memory mode was introduced to address, as when hugepages are in memory, there can never be leaks because we never touch the filesystem in the first place, so if there is a recommended way to address this at all, it would be --in-memory mode. 

> 
> > How are hugepages assigned to your container?
> > Have you tried using --in-memory mode?
> 
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-users] can we reserve hugepage and not release
  2021-09-29 12:40       ` Burakov, Anatoly
@ 2021-10-05 15:56         ` Jiany Wu
  0 siblings, 0 replies; 6+ messages in thread
From: Jiany Wu @ 2021-10-05 15:56 UTC (permalink / raw)
  To: Burakov, Anatoly
  Cc: Mcnamara, John, Richardson, Bruce, Thomas Monjalon,
	dmitry.kozliuk, olivier.matz, stephen, users

Thanks for your explanation. Will try in memory mode.;)

Burakov, Anatoly <anatoly.burakov@intel.com>于2021年9月29日 周三下午8:40写道:

> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Wednesday, September 29, 2021 12:11 PM
> > To: Jiany Wu <wujianyue000@gmail.com>; Burakov, Anatoly
> > <anatoly.burakov@intel.com>
> > Cc: users@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> > olivier.matz@6wind.com; dmitry.kozliuk@gmail.com;
> > stephen@networkplumber.org; Mcnamara, John
> > <john.mcnamara@intel.com>
> > Subject: Re: [dpdk-users] can we reserve hugepage and not release
> >
> > 29/09/2021 12:16, Burakov, Anatoly:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 18/09/2021 04:37, Jiany Wu:
> > > > > Hello,
> > > > >
> > > > > I met a scenario that, need to start and stop the container many
> > > > > times for the hugepage. But after several times container start
> > > > > and stop, the hugepage is not able to reserve.
> > > > > Hugepage size is 2MB, and HW only support 2MB, can't support 1GB.
> > > > > Is there anyway to make sure the hugepage is still kept continuous?
> > > > > Thanks indeed.
> > > >
> > > > Interesting question.
> > > > I think we need to address it in the DPDK documentation.
> > > >
> > > > Anatoly, Stephen, Bruce, any advice please?
> > > >
> > >
> > > Hi,
> > >
> > > From description, I don't quite understand what's the issue here. Is
> the
> > problem about "contiguousness of memory", or is it about inability to
> reserve
> > more hugepages?
> >
> > I think the issue is that sometimes some pages are not properly
> released, so
> > we cannot reserve them again.
> > That's something I experienced myself.
> > Any trick to reset hugepages state?
>
> [[AB]]
> Under normal circumstances, the hugepage files would just be deleted and
> recreated the next time a primary process initializes even if there are
> leftover pages.
>
> That said, assuming the pages are not "released" because DPDK has left a
> few files behind and a new container doesn't have permissions to delete
> them (or can't for some other reason), then there is very little DPDK can
> do other than track down all memory leaks. For example, using RX callback
> feature is intentionally causing a memory leak because by default the API
> will not free the callback structure as we cannot guarantee that there are
> no Dataplane threads still using that structure. Individual
> drivers/libraries might also leak memory due to e.g. secondary process
> usage. For example, in the general case, if something is meant to be shared
> by primary and secondary, but primary cannot free() it because it might
> still be used by secondaries, and secondaries have no way of verifying
> whether the resource is being used by other processes, or can be freed.
>
> Put it simply, sometimes DPDK may leak memory either out of necessity, or
> even intentionally, to avoid potential use-after-free errors, and there's
> no general way to solve this problem that I'm aware of. This is the kind of
> stuff --in-memory mode was introduced to address, as when hugepages are in
> memory, there can never be leaks because we never touch the filesystem in
> the first place, so if there is a recommended way to address this at all,
> it would be --in-memory mode.
>
> >
> > > How are hugepages assigned to your container?
> > > Have you tried using --in-memory mode?
> >
> >
>
>

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-10-05 15:56 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-18  2:37 [dpdk-users] can we reserve hugepage and not release Jiany Wu
2021-09-29 10:04 ` Thomas Monjalon
2021-09-29 10:16   ` Burakov, Anatoly
2021-09-29 11:11     ` Thomas Monjalon
2021-09-29 12:40       ` Burakov, Anatoly
2021-10-05 15:56         ` Jiany Wu

DPDK usage discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git