[dpdk-dev] Hugepages not being deleted

[dpdk-dev] Hugepages not being deleted

[Michael Santana Francisco]

Hello,

I am currently working on a patch to fix the eal_flags_autotest test as 
it currently fails on many platforms.
I have made some progress, however I stumbled upon a possible issue with 
EAL and hugepages.
Looking at the code and some documentation it appears to me that 
hupepages are supposed to be automatically deleted on dynamic memory 
mode as the dpdk process exits.
The test however reports that this is not happening.

This can be shown by:

bash# export DPDK_TEST=eal_flags_autotest
bash# ./build/app/test/dpdk-test
...
Error - hugepage files for memtest1 were not deleted!
Error in test_file_prefix()
Test Failed
bash# ls /dev/hugepages/ #hugetlbfs is mounted on /dev/hugepages
memtest1map_0  memtest1map_1  memtest1map_2  memtest1map_3 
memtest1map_4  memtest1map_5  memtest1map_6  memtest1map_7 
memtest1map_8  rtemap_0

To me it appears that the hugepages are in fact not being deleted correctly.
Is this an anomaly or is anyone else seeing this issue as well?

Michael Santana


Note, if you are running on a system with less than 8 cores please see 
patch
https://github.com/Maickii/dpdk-2/commit/7cfad856611e3ded4050f670ec11d1b2e17851d8.patch

[dpdk-dev] Hugepages not being deleted

[Michael Santana Francisco]

Hello,

I am currently working on a patch to fix the eal_flags_autotest test as 
it currently fails on many platforms.
I have made some progress, however I stumbled upon a possible issue with 
EAL and hugepages.
Looking at the code and some documentation it appears to me that 
hupepages are supposed to be automatically deleted on dynamic memory 
mode as the dpdk process exits.
The test however reports that this is not happening.

This can be shown by:

bash# export DPDK_TEST=eal_flags_autotest
bash# ./build/app/test/dpdk-test
...
Error - hugepage files for memtest1 were not deleted!
Error in test_file_prefix()
Test Failed
bash# ls /dev/hugepages/ #hugetlbfs is mounted on /dev/hugepages
memtest1map_0  memtest1map_1  memtest1map_2  memtest1map_3 
memtest1map_4  memtest1map_5  memtest1map_6  memtest1map_7 
memtest1map_8  rtemap_0

To me it appears that the hugepages are in fact not being deleted correctly.
Is this an anomaly or is anyone else seeing this issue as well?

Michael Santana


Note, if you are running on a system with less than 8 cores please see 
patch
https://github.com/Maickii/dpdk-2/commit/7cfad856611e3ded4050f670ec11d1b2e17851d8.patch

Re: [dpdk-dev] Hugepages not being deleted

[Michael Santana Francisco]

ping
> Hello,
>
> I am currently working on a patch to fix the eal_flags_autotest test 
> as it currently fails on many platforms.
> I have made some progress, however I stumbled upon a possible issue 
> with EAL and hugepages.
> Looking at the code and some documentation it appears to me that 
> hupepages are supposed to be automatically deleted on dynamic memory 
> mode as the dpdk process exits.
> The test however reports that this is not happening.
>
> This can be shown by:
>
> bash# export DPDK_TEST=eal_flags_autotest
> bash# ./build/app/test/dpdk-test
> ...
> Error - hugepage files for memtest1 were not deleted!
> Error in test_file_prefix()
> Test Failed
> bash# ls /dev/hugepages/ #hugetlbfs is mounted on /dev/hugepages
> memtest1map_0  memtest1map_1  memtest1map_2  memtest1map_3 
> memtest1map_4  memtest1map_5  memtest1map_6  memtest1map_7 
> memtest1map_8  rtemap_0
>
> To me it appears that the hugepages are in fact not being deleted 
> correctly.
> Is this an anomaly or is anyone else seeing this issue as well?
>
> Michael Santana
>
>
> Note, if you are running on a system with less than 8 cores please see 
> patch
> https://github.com/Maickii/dpdk-2/commit/7cfad856611e3ded4050f670ec11d1b2e17851d8.patch 
>

Re: [dpdk-dev] Hugepages not being deleted

[Michael Santana Francisco]

ping
> Hello,
>
> I am currently working on a patch to fix the eal_flags_autotest test 
> as it currently fails on many platforms.
> I have made some progress, however I stumbled upon a possible issue 
> with EAL and hugepages.
> Looking at the code and some documentation it appears to me that 
> hupepages are supposed to be automatically deleted on dynamic memory 
> mode as the dpdk process exits.
> The test however reports that this is not happening.
>
> This can be shown by:
>
> bash# export DPDK_TEST=eal_flags_autotest
> bash# ./build/app/test/dpdk-test
> ...
> Error - hugepage files for memtest1 were not deleted!
> Error in test_file_prefix()
> Test Failed
> bash# ls /dev/hugepages/ #hugetlbfs is mounted on /dev/hugepages
> memtest1map_0  memtest1map_1  memtest1map_2  memtest1map_3 
> memtest1map_4  memtest1map_5  memtest1map_6  memtest1map_7 
> memtest1map_8  rtemap_0
>
> To me it appears that the hugepages are in fact not being deleted 
> correctly.
> Is this an anomaly or is anyone else seeing this issue as well?
>
> Michael Santana
>
>
> Note, if you are running on a system with less than 8 cores please see 
> patch
> https://github.com/Maickii/dpdk-2/commit/7cfad856611e3ded4050f670ec11d1b2e17851d8.patch 
>

Re: [dpdk-dev] Hugepages not being deleted

[Burakov, Anatoly]

On 23-Apr-19 4:15 PM, Michael Santana Francisco wrote:
> Hello,
> 
> I am currently working on a patch to fix the eal_flags_autotest test as 
> it currently fails on many platforms.
> I have made some progress, however I stumbled upon a possible issue with 
> EAL and hugepages.
> Looking at the code and some documentation it appears to me that 
> hupepages are supposed to be automatically deleted on dynamic memory 
> mode as the dpdk process exits.
> The test however reports that this is not happening.

Apologies, this email has slipped through the cracks.

Hugepages not being deleted can happen in certain circumstances (such as 
drivers allocating things and not freeing them later), and there's 
little DPDK EAL can do to fix it - if the memory is used, it's not safe 
to just pull it out from under whoever is using it.

The best you can do is find what is allocating that memory, and fix the 
problem at the source.

> 
> This can be shown by:
> 
> bash# export DPDK_TEST=eal_flags_autotest
> bash# ./build/app/test/dpdk-test
> ...
> Error - hugepage files for memtest1 were not deleted!
> Error in test_file_prefix()
> Test Failed
> bash# ls /dev/hugepages/ #hugetlbfs is mounted on /dev/hugepages
> memtest1map_0  memtest1map_1  memtest1map_2  memtest1map_3 
> memtest1map_4  memtest1map_5  memtest1map_6  memtest1map_7 
> memtest1map_8  rtemap_0
> 
> To me it appears that the hugepages are in fact not being deleted 
> correctly.
> Is this an anomaly or is anyone else seeing this issue as well?
> 
> Michael Santana
> 
> 
> Note, if you are running on a system with less than 8 cores please see 
> patch
> https://github.com/Maickii/dpdk-2/commit/7cfad856611e3ded4050f670ec11d1b2e17851d8.patch 
> 
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] Hugepages not being deleted

[Burakov, Anatoly]

On 23-Apr-19 4:15 PM, Michael Santana Francisco wrote:
> Hello,
> 
> I am currently working on a patch to fix the eal_flags_autotest test as 
> it currently fails on many platforms.
> I have made some progress, however I stumbled upon a possible issue with 
> EAL and hugepages.
> Looking at the code and some documentation it appears to me that 
> hupepages are supposed to be automatically deleted on dynamic memory 
> mode as the dpdk process exits.
> The test however reports that this is not happening.

Apologies, this email has slipped through the cracks.

Hugepages not being deleted can happen in certain circumstances (such as 
drivers allocating things and not freeing them later), and there's 
little DPDK EAL can do to fix it - if the memory is used, it's not safe 
to just pull it out from under whoever is using it.

The best you can do is find what is allocating that memory, and fix the 
problem at the source.

> 
> This can be shown by:
> 
> bash# export DPDK_TEST=eal_flags_autotest
> bash# ./build/app/test/dpdk-test
> ...
> Error - hugepage files for memtest1 were not deleted!
> Error in test_file_prefix()
> Test Failed
> bash# ls /dev/hugepages/ #hugetlbfs is mounted on /dev/hugepages
> memtest1map_0  memtest1map_1  memtest1map_2  memtest1map_3 
> memtest1map_4  memtest1map_5  memtest1map_6  memtest1map_7 
> memtest1map_8  rtemap_0
> 
> To me it appears that the hugepages are in fact not being deleted 
> correctly.
> Is this an anomaly or is anyone else seeing this issue as well?
> 
> Michael Santana
> 
> 
> Note, if you are running on a system with less than 8 cores please see 
> patch
> https://github.com/Maickii/dpdk-2/commit/7cfad856611e3ded4050f670ec11d1b2e17851d8.patch 
> 
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] Hugepages not being deleted

[David Marchand]

Hello Anatoly,

Not sure what the issue is at the moment.
I too have an error on the eal_flags_autotest but what is worrying me is
that it works fine in 18.11.1 and 19.02.
Is there anything that rings a bell?

Thanks.

-- 
David Marchand

Re: [dpdk-dev] Hugepages not being deleted

[Burakov, Anatoly]

On 17-May-19 3:06 PM, David Marchand wrote:
> Hello Anatoly,
> 
> Not sure what the issue is at the moment.
> I too have an error on the eal_flags_autotest but what is worrying me is 
> that it works fine in 18.11.1 and 19.02.
> Is there anything that rings a bell?
> 
> Thanks.
> 
> -- 
> David Marchand

Not to me. Last i checked they were passing.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] Hugepages not being deleted

[David Marchand]

On Mon, May 20, 2019 at 12:43 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 17-May-19 3:06 PM, David Marchand wrote:
> > Hello Anatoly,
> >
> > Not sure what the issue is at the moment.
> > I too have an error on the eal_flags_autotest but what is worrying me is
> > that it works fine in 18.11.1 and 19.02.
> > Is there anything that rings a bell?
> >
> > Thanks.
> >
> > --
> > David Marchand
>
> Not to me. Last i checked they were passing.
>

Bisected it, I will send a fix.


-- 
David Marchand

Re: [dpdk-dev] Hugepages not being deleted

[Burakov, Anatoly]

On 20-May-19 12:09 PM, David Marchand wrote:
> On Mon, May 20, 2019 at 12:43 PM Burakov, Anatoly 
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> 
>     On 17-May-19 3:06 PM, David Marchand wrote:
>      > Hello Anatoly,
>      >
>      > Not sure what the issue is at the moment.
>      > I too have an error on the eal_flags_autotest but what is
>     worrying me is
>      > that it works fine in 18.11.1 and 19.02.
>      > Is there anything that rings a bell?
>      >
>      > Thanks.
>      >
>      > --
>      > David Marchand
> 
>     Not to me. Last i checked they were passing.
> 
> 
> Bisected it, I will send a fix.
> 
> 
> -- 
> David Marchand

I've bisected it too. I think Erik (author of original patch) already is 
working on the problem. I don't think it's easily (and safely) solvable. 
You can find more info in below threads:

http://patches.dpdk.org/patch/53191/
http://patches.dpdk.org/patch/53268/

-- 
Thanks,
Anatoly

Re: [dpdk-dev] Hugepages not being deleted

[David Marchand]

On Mon, May 20, 2019 at 1:45 PM Burakov, Anatoly <anatoly.burakov@intel.com>
wrote:

> On 20-May-19 12:09 PM, David Marchand wrote:
> > On Mon, May 20, 2019 at 12:43 PM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> >
> >     On 17-May-19 3:06 PM, David Marchand wrote:
> >      > Hello Anatoly,
> >      >
> >      > Not sure what the issue is at the moment.
> >      > I too have an error on the eal_flags_autotest but what is
> >     worrying me is
> >      > that it works fine in 18.11.1 and 19.02.
> >      > Is there anything that rings a bell?
> >      >
> >      > Thanks.
> >      >
> >      > --
> >      > David Marchand
> >
> >     Not to me. Last i checked they were passing.
> >
> >
> > Bisected it, I will send a fix.
> >
> >
> > --
> > David Marchand
>
> I've bisected it too. I think Erik (author of original patch) already is
> working on the problem. I don't think it's easily (and safely) solvable.
> You can find more info in below threads:
>
> http://patches.dpdk.org/patch/53191/
> http://patches.dpdk.org/patch/53268/


Thanks, indeed I was on this issue.


-- 
David Marchand

Re: [dpdk-dev] Hugepages not being deleted

[Carrillo, Erik G]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Monday, May 20, 2019 6:45 AM
> To: David Marchand <david.marchand@redhat.com>
> Cc: Michael Santana <msantana@redhat.com>; dev <dev@dpdk.org>;
> Carrillo, Erik G <erik.g.carrillo@intel.com>
> Subject: Re: [dpdk-dev] Hugepages not being deleted
> 
> On 20-May-19 12:09 PM, David Marchand wrote:
> > On Mon, May 20, 2019 at 12:43 PM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
> >
> >     On 17-May-19 3:06 PM, David Marchand wrote:
> >      > Hello Anatoly,
> >      >
> >      > Not sure what the issue is at the moment.
> >      > I too have an error on the eal_flags_autotest but what is
> >     worrying me is
> >      > that it works fine in 18.11.1 and 19.02.
> >      > Is there anything that rings a bell?
> >      >
> >      > Thanks.
> >      >
> >      > --
> >      > David Marchand
> >
> >     Not to me. Last i checked they were passing.
> >
> >
> > Bisected it, I will send a fix.
> >
> >
> > --
> > David Marchand
> 
> I've bisected it too. I think Erik (author of original patch) already is working on
> the problem. I don't think it's easily (and safely) solvable.
> You can find more info in below threads:
> 
> http://patches.dpdk.org/patch/53191/
> http://patches.dpdk.org/patch/53268/
> 

Yes, there a memzone that needs to be properly freed at finalize/cleanup time in the timer library.  One way to free it safely was explored in the following:

http://patchwork.dpdk.org/patch/53334/

but that solution has ABI implications that were undesirable.  Still looking for a better way.

Thanks,
Erik

> --
> Thanks,
> Anatoly