Re: [dpdk-dev] very high VIRT memory usage

["Burakov, Anatoly" <anatoly.burakov@intel.com>]

On 09-Jun-20 4:35 PM, Stephen Hemminger wrote:
> On Tue, 9 Jun 2020 14:39:54 +0100
> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote:
> 
>> On 09-Jun-20 2:13 PM, Ferruh Yigit wrote:
>>> On 6/9/2020 1:46 PM, Burakov, Anatoly wrote:
>>>> On 08-Jun-20 12:03 PM, Francesco wrote:
>>>>> Hi all,
>>>>> I upgraded an old DPDK-based app which was using DPDK 17.11 to latest DPDK
>>>>> 20.05 and I noticed that if I look  at "top" I see that the VIRT memory
>>>>> taken by my application is now 256.1GB while before it was <1GB.
>>>>>
>>>>> I've seen this same behavior with also "testpmd" example... is this a known
>>>>> issue with latest DPDK versions?
>>>>> Can I tweak some setting to have VIRT memory usage more or less similar to
>>>>> RSS ?
>>>>>
>>>>> I forgot to add I'm working on Linux, Centos7
>>>>>
>>>>> Thanks,
>>>>> Francesco Montorsi
>>>>>   
>>>>
>>>> There was a discussion on this not too long ago, but i can't seem to
>>>> find it for some reason.
>>>
>>> Can it be "Big spike in DPDK VSZ" ?
>>> http://inbox.dpdk.org/dev/CAGxAMwD6Wtfi=C2Txwjfk0zhFvRzeqBu7mFfE8ayh=EJi2aU-A@mail.gmail.com/#t
>>>    
>>
>> Yes, that's the one, thanks Ferruh :)
>>
>>>> Anyway, long story short, that's not a bug,
>>>> that's by design.
>>>>
>>>> Since 18.11 (or 18.05 to be precise), there is a new memory subsystem in
>>>> DPDK that allows growing and shrinking DPDK memory usage at runtime.
>>>> That means, you can start with zero hugepages preallocated, and then
>>>> allocate as you go, letting the memory subsystem decide how much memory
>>>> you need.
>>>>
>>>> The catch is that all of this hugepage memory is allocated into
>>>> somewhere, some virtual address space. And *that* address space is
>>>> preallocated at startup, to allow for secondary processes to duplicate
>>>> primary process's address space exactly, and allow dynamic allocation of
>>>> *shared* memory at runtime.
>>>>
>>>> This memory will show up in top et al. but the truth is, it's zero cost,
>>>> because it's anonymous memory. It isn't actually taking up any RAM. It
>>>> will show up in dumps (20.05 has already fixed that issue, and the fixes
>>>> will probably be backported to stable, including 18.11), so unless you
>>>> have a very specific problem, i don't think that's anything you should
>>>> be concerned about.
> 
> The one concern is for cases like cgroup memory accounting thinking
> the process is huge and OOM killing it.
> 

Is there any way to know the *actual* memory usage of the process (i.e. 
not including anonymous memory)?

-- 
Thanks,
Anatoly