[dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed

[dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed

[Yan, Xiaoping (NSB - CN/Hangzhou)]

Hi,

Before I run pdump, 21MB is used in /tmp
tmpfs                        63G   21M   63G   1% /tmp
Then I run pdump with such command:
dpdk-pdump  -c 0x4001000400100 -a 0000:03:00.6 -a 0000:03:06.1 --legacy-mem --base-virtaddr 0x2aaa2aa000 --file-prefix l2rt --  --pdump port=1,queue=*,rx-dev=/tmp/l2biprt.pcap,tx-dev=/tmp/l2biprt.pcap,mbuf-size=10240,total-num-mbufs=10000
After stop, it generates a 14MB file
              -rw-rw-r--. 1 9999 9999  14M Aug 23 03:58 l2biprt.pcap
And 35M is used in /tmp
              tmpfs                        63G   35M   63G   1% /tmp
And fuser show there are still several users of this file. (all these pids are of running dpdk primary and secondary processes )
              # fuser /tmp/l2biprt.pcap
/tmp/l2biprt.pcap:     139   342   347   434
Then after I rm the file (l2biprt.pcap), disk space in tmp is not freed (still 35MB used)
              rm l2biprt.pcap
              df -h
              tmpfs                        63G   35M   63G   1% /tmp


It seems it goes like this:
             pdump start->pdump send vdev hotplug add request->primary and secondary process calls pmd_pcap_probe() which opens the pcap file->
              pdump stop-> pdump send vdev hotplug remove request-> primary and secondary process does not close the pcap file

To properly close the pcap file and release disk space, it seems we should:

  *   somehow close the pcap file in "vdev hotplug remove", or
  *   don't open the pcap file in "vdev hotplug add"( pmd_pcap_probe()), it seems pcap eth_dev_start will open the pcap file in case it was not opened.

Any comment is appreciated, thank you.

Best regards
Yan Xiaoping

Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed

[Ferruh Yigit]

On 8/23/2021 5:14 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote:
> Hi,
> 
> Before I run pdump, 21MB is used in /tmp
> tmpfs                        63G   21M   63G   1% /tmp
> Then I run pdump with such command:
> dpdk-pdump  -c 0x4001000400100 -a 0000:03:00.6 -a 0000:03:06.1 --legacy-mem --base-virtaddr 0x2aaa2aa000 --file-prefix l2rt --  --pdump port=1,queue=*,rx-dev=/tmp/l2biprt.pcap,tx-dev=/tmp/l2biprt.pcap,mbuf-size=10240,total-num-mbufs=10000
> After stop, it generates a 14MB file
>               -rw-rw-r--. 1 9999 9999  14M Aug 23 03:58 l2biprt.pcap
> And 35M is used in /tmp
>               tmpfs                        63G   35M   63G   1% /tmp
> And fuser show there are still several users of this file. (all these pids are of running dpdk primary and secondary processes )
>               # fuser /tmp/l2biprt.pcap
> /tmp/l2biprt.pcap:     139   342   347   434
> Then after I rm the file (l2biprt.pcap), disk space in tmp is not freed (still 35MB used)
>               rm l2biprt.pcap
>               df -h
>               tmpfs                        63G   35M   63G   1% /tmp
> 
> 
> It seems it goes like this:
>              pdump start->pdump send vdev hotplug add request->primary and secondary process calls pmd_pcap_probe() which opens the pcap file->
>               pdump stop-> pdump send vdev hotplug remove request-> primary and secondary process does not close the pcap file
> 
> To properly close the pcap file and release disk space, it seems we should:
> 
>   *   somehow close the pcap file in "vdev hotplug remove", or
>   *   don't open the pcap file in "vdev hotplug add"( pmd_pcap_probe()), it seems pcap eth_dev_start will open the pcap file in case it was not opened.
> 
> Any comment is appreciated, thank you.


Hi Yan,

I can't reproduce the behavior you mentioned, for my case the .pcap file is
closed when pdump is terminated and there is no active user of the file after
that point.

Removing the pcap device cause closing the .pcap files.

When you terminate pdump application it will remove the pcap devices, and since
it is a secondary process a message sent to primary to remove it too, making
both process closing the pcap files.

When I use testpmd as primary application, I am getting following log from
testpmd when pdump terminated:
"Port 2 is closed"
Can you please test with testpmd? You can use '--log-level=*:debug' eal
parameter to get more log for multi process communication.

Btw, what is your DPDK version? I tested with latest DPDK (21.11.0-rc0).

Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed

[Yan, Xiaoping (NSB - CN/Hangzhou)]

Hi,

I'm using dpdk 20.11. (it's a bit difficult for me to upgrade my applications to latest dpdk)

I tested dpdk-pdump with dpdk-tespmd, result is same (disk space not freed.)
	[root@up-0:/]
	# ps aux
	USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
	root          78 99.1  0.0 67407568 102268 pts/0 SLl+ 07:15   2:32 dpdk-testpmd --base-virtaddr 0
	root         144  0.3  0.0 67332948 7708 pts/1   Sl+  07:17   0:00 dpdk-pdump -l 14 -a 0000:b3:02

	# fuser /tmp/l2biprt.pcap
	/tmp/l2biprt.pcap:      78   144

When stop pdump, indeed testpmd prints:
	EAL: msg: eal_dev_mp_request
	eth_dev_close(): Closing pcap ethdev on NUMA socket 0
	Port 1 is closed
	EAL: reply: eal_dev_mp_request
But testpmd does not close the file:
	[root@up-0:/]
	# fuser /tmp/l2biprt.pcap
	/tmp/l2biprt.pcap:      78

And remove the pcap file, disk space is not freed:
	# df -h /tmp/
	Filesystem      Size  Used Avail Use% Mounted on
	tmpfs            94G  900K   94G   1% /tmp
	[root@up-0:/]
	# ls -lh /tmp/
	total 900K
	-rw-r--r--. 1 root root 897K Aug 24 07:18 l2biprt.pcap
	drwxrwxrwx. 2 root root   18 Aug 24 07:14 nes
	[root@up-0:/]
	# rm /tmp/l2biprt.pcap
	[root@up-0:/]
	# ls -lh /tmp/
	total 0
	drwxrwxrwx. 2 root root 18 Aug 24 07:14 nes
	[root@up-0:/]
	# df -h /tmp/
	Filesystem      Size  Used Avail Use% Mounted on
	tmpfs            94G  900K   94G   1% /tmp

Could you please kindly check the same for your test?
It seems the testpmd close pcap device via pmd_pcap_remove(), but pcap_dump_close is not called. 
Could you please kindly help to check how testpmd in your case close the pcap file (if possible with function call stack?)?

Thank you.


Best regards
Yan Xiaoping

-----Original Message-----
From: Ferruh Yigit <ferruh.yigit@intel.com> 
Sent: 2021年8月23日 18:04
To: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com>; dev@dpdk.org
Cc: Reshma Pattan <reshma.pattan@intel.com>
Subject: Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed

On 8/23/2021 5:14 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote:
> Hi,
> 
> Before I run pdump, 21MB is used in /tmp
> tmpfs                        63G   21M   63G   1% /tmp
> Then I run pdump with such command:
> dpdk-pdump  -c 0x4001000400100 -a 0000:03:00.6 -a 0000:03:06.1 
> --legacy-mem --base-virtaddr 0x2aaa2aa000 --file-prefix l2rt --  
> --pdump 
> port=1,queue=*,rx-dev=/tmp/l2biprt.pcap,tx-dev=/tmp/l2biprt.pcap,mbuf-
> size=10240,total-num-mbufs=10000
> After stop, it generates a 14MB file
>               -rw-rw-r--. 1 9999 9999  14M Aug 23 03:58 l2biprt.pcap 
> And 35M is used in /tmp
>               tmpfs                        63G   35M   63G   1% /tmp
> And fuser show there are still several users of this file. (all these pids are of running dpdk primary and secondary processes )
>               # fuser /tmp/l2biprt.pcap
> /tmp/l2biprt.pcap:     139   342   347   434
> Then after I rm the file (l2biprt.pcap), disk space in tmp is not freed (still 35MB used)
>               rm l2biprt.pcap
>               df -h
>               tmpfs                        63G   35M   63G   1% /tmp
> 
> 
> It seems it goes like this:
>              pdump start->pdump send vdev hotplug add request->primary and secondary process calls pmd_pcap_probe() which opens the pcap file->
>               pdump stop-> pdump send vdev hotplug remove request-> 
> primary and secondary process does not close the pcap file
> 
> To properly close the pcap file and release disk space, it seems we should:
> 
>   *   somehow close the pcap file in "vdev hotplug remove", or
>   *   don't open the pcap file in "vdev hotplug add"( pmd_pcap_probe()), it seems pcap eth_dev_start will open the pcap file in case it was not opened.
> 
> Any comment is appreciated, thank you.


Hi Yan,

I can't reproduce the behavior you mentioned, for my case the .pcap file is closed when pdump is terminated and there is no active user of the file after that point.

Removing the pcap device cause closing the .pcap files.

When you terminate pdump application it will remove the pcap devices, and since it is a secondary process a message sent to primary to remove it too, making both process closing the pcap files.

When I use testpmd as primary application, I am getting following log from testpmd when pdump terminated:
"Port 2 is closed"
Can you please test with testpmd? You can use '--log-level=*:debug' eal parameter to get more log for multi process communication.

Btw, what is your DPDK version? I tested with latest DPDK (21.11.0-rc0).

Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed

[Ferruh Yigit]

(Please avoid top-post, I have updated the order of the messages. Please find
the reply at the bottom of the email.)

>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@intel.com> 
>> Sent: 2021年8月23日 18:04
>> To: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com>; dev@dpdk.org
>> Cc: Reshma Pattan <reshma.pattan@intel.com>
>> Subject: Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed
>> 
>> On 8/23/2021 5:14 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote:
>>> Hi,
>>>
>>> Before I run pdump, 21MB is used in /tmp
>>> tmpfs                        63G   21M   63G   1% /tmp
>>> Then I run pdump with such command:
>>> dpdk-pdump  -c 0x4001000400100 -a 0000:03:00.6 -a 0000:03:06.1 
>>> --legacy-mem --base-virtaddr 0x2aaa2aa000 --file-prefix l2rt --  
>>> --pdump 
>>> port=1,queue=*,rx-dev=/tmp/l2biprt.pcap,tx-dev=/tmp/l2biprt.pcap,mbuf-
>>> size=10240,total-num-mbufs=10000
>>> After stop, it generates a 14MB file
>>>               -rw-rw-r--. 1 9999 9999  14M Aug 23 03:58 l2biprt.pcap 
>>> And 35M is used in /tmp
>>>               tmpfs                        63G   35M   63G   1% /tmp
>>> And fuser show there are still several users of this file. (all these pids are of running dpdk primary and secondary processes )
>>>               # fuser /tmp/l2biprt.pcap
>>> /tmp/l2biprt.pcap:     139   342   347   434
>>> Then after I rm the file (l2biprt.pcap), disk space in tmp is not freed (still 35MB used)
>>>               rm l2biprt.pcap
>>>               df -h
>>>               tmpfs                        63G   35M   63G   1% /tmp
>>>
>>>
>>> It seems it goes like this:
>>>              pdump start->pdump send vdev hotplug add request->primary and secondary process calls pmd_pcap_probe() which opens the pcap file->
>>>               pdump stop-> pdump send vdev hotplug remove request-> 
>>> primary and secondary process does not close the pcap file
>>>
>>> To properly close the pcap file and release disk space, it seems we should:
>>>
>>>   *   somehow close the pcap file in "vdev hotplug remove", or
>>>   *   don't open the pcap file in "vdev hotplug add"( pmd_pcap_probe()), it seems pcap eth_dev_start will open the pcap file in case it was not opened.
>>>
>>> Any comment is appreciated, thank you.
>> 
>> 
>> Hi Yan,
>> 
>> I can't reproduce the behavior you mentioned, for my case the .pcap file is closed when pdump is terminated and there is no active user of the file after that point.
>> 
>> Removing the pcap device cause closing the .pcap files.
>> 
>> When you terminate pdump application it will remove the pcap devices, and since it is a secondary process a message sent to primary to remove it too, making both process closing the pcap files.
>> 
>> When I use testpmd as primary application, I am getting following log from testpmd when pdump terminated:
>> "Port 2 is closed"
>> Can you please test with testpmd? You can use '--log-level=*:debug' eal parameter to get more log for multi process communication.
>> 
>> Btw, what is your DPDK version? I tested with latest DPDK (21.11.0-rc0).
>> 
>> 
> On 8/24/2021 8:36 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote:
> Hi,
> 
> I'm using dpdk 20.11. (it's a bit difficult for me to upgrade my applications to latest dpdk)
> 
> I tested dpdk-pdump with dpdk-tespmd, result is same (disk space not freed.)
> 	[root@up-0:/]
> 	# ps aux
> 	USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> 	root          78 99.1  0.0 67407568 102268 pts/0 SLl+ 07:15   2:32 dpdk-testpmd --base-virtaddr 0
> 	root         144  0.3  0.0 67332948 7708 pts/1   Sl+  07:17   0:00 dpdk-pdump -l 14 -a 0000:b3:02
> 
> 	# fuser /tmp/l2biprt.pcap
> 	/tmp/l2biprt.pcap:      78   144
> 
> When stop pdump, indeed testpmd prints:
> 	EAL: msg: eal_dev_mp_request
> 	eth_dev_close(): Closing pcap ethdev on NUMA socket 0
> 	Port 1 is closed
> 	EAL: reply: eal_dev_mp_request
> But testpmd does not close the file:
> 	[root@up-0:/]
> 	# fuser /tmp/l2biprt.pcap
> 	/tmp/l2biprt.pcap:      78
> 
> And remove the pcap file, disk space is not freed:
> 	# df -h /tmp/
> 	Filesystem      Size  Used Avail Use% Mounted on
> 	tmpfs            94G  900K   94G   1% /tmp
> 	[root@up-0:/]
> 	# ls -lh /tmp/
> 	total 900K
> 	-rw-r--r--. 1 root root 897K Aug 24 07:18 l2biprt.pcap
> 	drwxrwxrwx. 2 root root   18 Aug 24 07:14 nes
> 	[root@up-0:/]
> 	# rm /tmp/l2biprt.pcap
> 	[root@up-0:/]
> 	# ls -lh /tmp/
> 	total 0
> 	drwxrwxrwx. 2 root root 18 Aug 24 07:14 nes
> 	[root@up-0:/]
> 	# df -h /tmp/
> 	Filesystem      Size  Used Avail Use% Mounted on
> 	tmpfs            94G  900K   94G   1% /tmp
> 
> Could you please kindly check the same for your test?
> It seems the testpmd close pcap device via pmd_pcap_remove(), but pcap_dump_close is not called. 
> Could you please kindly help to check how testpmd in your case close the pcap file (if possible with function call stack?)?
> 
> Thank you.
> 
> 

I can reproduce the issue with v20.11. The problems seems fixed by commit [1] in
the latest DPDK, the commit title gives the hint ;)

Backporting that commit can solve the issue.

Regards,
ferruh


[1]
Fixes: e412a138db91 ("net/pcap: fix file descriptor leak on close")