Re: [dpdk-dev] Big spike in DPDK VSZ

[David Marchand <david.marchand@redhat.com>]

Hello Anatoly,

On Tue, Feb 4, 2020 at 11:23 AM Burakov, Anatoly
<anatoly.burakov@intel.com> wrote:
>
> On 30-Jan-20 8:51 AM, David Marchand wrote:
> > On Thu, Jan 30, 2020 at 8:48 AM siddarth rai <siddsr@gmail.com> wrote:
> >> I have been using DPDK 19.08 and I notice the process VSZ is huge.
> >>
> >> I tried running the test PMD. It takes 64G VSZ and if I use the
> >> '--in-memory' option it takes up to 188G.
> >>
> >> Is there anyway to disable allocation of such huge VSZ in DPDK ?
> >
> > *Disclaimer* I don't know the arcanes of the mem subsystem.
> >
> > I suppose this is due to the memory allocator in dpdk that reserves
> > unused virtual space (for memory hotplug + multiprocess).
>
> Yes, that's correct. In order to guarantee memory reservation succeeding
> at all times, we need to reserve all possible memory in advance.
> Otherwise we may end up in a situation where primary process has
> allocated a page, but the secondary can't map it because the address
> space is already occupied by something else.
>
> >
> > If this is the case, maybe we could do something to enhance the
> > situation for applications that won't care about multiprocess.
> > Like inform dpdk that the application won't use multiprocess and skip
> > those reservations.
>
> You're welcome to try this, but i assure you, avoiding these
> reservations is a lot of work, because you'd be adding a yet another
> path to an already overly complex allocator :)

Went and looked at the EAL options we have.
I understand that the --in-memory (essentially the --no-shconf
functionality) option should be enough.
I can see that Siddarth uses it.

But the problem is that the memory allocator still reserves big va
areas for memory hotplug.


>
> >
> > Or another idea would be to limit those reservations to what is passed
> > via --socket-limit.
> >
> > Anatoly?
>
> I have a patchset in the works that does this and was planning to submit
> it to 19.08, but things got in the way and it's still sitting there
> collecting bit rot. This may be reason enough to resurrect it and finish
> it up :)

Does this patchset contain more than just taking --socket-limit into account?


--
David Marchand