Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed

[Ferruh Yigit <ferruh.yigit@intel.com>]

(Please avoid top-post, I have updated the order of the messages. Please find
the reply at the bottom of the email.)

>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@intel.com> 
>> Sent: 2021年8月23日 18:04
>> To: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com>; dev@dpdk.org
>> Cc: Reshma Pattan <reshma.pattan@intel.com>
>> Subject: Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed
>> 
>> On 8/23/2021 5:14 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote:
>>> Hi,
>>>
>>> Before I run pdump, 21MB is used in /tmp
>>> tmpfs                        63G   21M   63G   1% /tmp
>>> Then I run pdump with such command:
>>> dpdk-pdump  -c 0x4001000400100 -a 0000:03:00.6 -a 0000:03:06.1 
>>> --legacy-mem --base-virtaddr 0x2aaa2aa000 --file-prefix l2rt --  
>>> --pdump 
>>> port=1,queue=*,rx-dev=/tmp/l2biprt.pcap,tx-dev=/tmp/l2biprt.pcap,mbuf-
>>> size=10240,total-num-mbufs=10000
>>> After stop, it generates a 14MB file
>>>               -rw-rw-r--. 1 9999 9999  14M Aug 23 03:58 l2biprt.pcap 
>>> And 35M is used in /tmp
>>>               tmpfs                        63G   35M   63G   1% /tmp
>>> And fuser show there are still several users of this file. (all these pids are of running dpdk primary and secondary processes )
>>>               # fuser /tmp/l2biprt.pcap
>>> /tmp/l2biprt.pcap:     139   342   347   434
>>> Then after I rm the file (l2biprt.pcap), disk space in tmp is not freed (still 35MB used)
>>>               rm l2biprt.pcap
>>>               df -h
>>>               tmpfs                        63G   35M   63G   1% /tmp
>>>
>>>
>>> It seems it goes like this:
>>>              pdump start->pdump send vdev hotplug add request->primary and secondary process calls pmd_pcap_probe() which opens the pcap file->
>>>               pdump stop-> pdump send vdev hotplug remove request-> 
>>> primary and secondary process does not close the pcap file
>>>
>>> To properly close the pcap file and release disk space, it seems we should:
>>>
>>>   *   somehow close the pcap file in "vdev hotplug remove", or
>>>   *   don't open the pcap file in "vdev hotplug add"( pmd_pcap_probe()), it seems pcap eth_dev_start will open the pcap file in case it was not opened.
>>>
>>> Any comment is appreciated, thank you.
>> 
>> 
>> Hi Yan,
>> 
>> I can't reproduce the behavior you mentioned, for my case the .pcap file is closed when pdump is terminated and there is no active user of the file after that point.
>> 
>> Removing the pcap device cause closing the .pcap files.
>> 
>> When you terminate pdump application it will remove the pcap devices, and since it is a secondary process a message sent to primary to remove it too, making both process closing the pcap files.
>> 
>> When I use testpmd as primary application, I am getting following log from testpmd when pdump terminated:
>> "Port 2 is closed"
>> Can you please test with testpmd? You can use '--log-level=*:debug' eal parameter to get more log for multi process communication.
>> 
>> Btw, what is your DPDK version? I tested with latest DPDK (21.11.0-rc0).
>> 
>> 
> On 8/24/2021 8:36 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote:
> Hi,
> 
> I'm using dpdk 20.11. (it's a bit difficult for me to upgrade my applications to latest dpdk)
> 
> I tested dpdk-pdump with dpdk-tespmd, result is same (disk space not freed.)
> 	[root@up-0:/]
> 	# ps aux
> 	USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> 	root          78 99.1  0.0 67407568 102268 pts/0 SLl+ 07:15   2:32 dpdk-testpmd --base-virtaddr 0
> 	root         144  0.3  0.0 67332948 7708 pts/1   Sl+  07:17   0:00 dpdk-pdump -l 14 -a 0000:b3:02
> 
> 	# fuser /tmp/l2biprt.pcap
> 	/tmp/l2biprt.pcap:      78   144
> 
> When stop pdump, indeed testpmd prints:
> 	EAL: msg: eal_dev_mp_request
> 	eth_dev_close(): Closing pcap ethdev on NUMA socket 0
> 	Port 1 is closed
> 	EAL: reply: eal_dev_mp_request
> But testpmd does not close the file:
> 	[root@up-0:/]
> 	# fuser /tmp/l2biprt.pcap
> 	/tmp/l2biprt.pcap:      78
> 
> And remove the pcap file, disk space is not freed:
> 	# df -h /tmp/
> 	Filesystem      Size  Used Avail Use% Mounted on
> 	tmpfs            94G  900K   94G   1% /tmp
> 	[root@up-0:/]
> 	# ls -lh /tmp/
> 	total 900K
> 	-rw-r--r--. 1 root root 897K Aug 24 07:18 l2biprt.pcap
> 	drwxrwxrwx. 2 root root   18 Aug 24 07:14 nes
> 	[root@up-0:/]
> 	# rm /tmp/l2biprt.pcap
> 	[root@up-0:/]
> 	# ls -lh /tmp/
> 	total 0
> 	drwxrwxrwx. 2 root root 18 Aug 24 07:14 nes
> 	[root@up-0:/]
> 	# df -h /tmp/
> 	Filesystem      Size  Used Avail Use% Mounted on
> 	tmpfs            94G  900K   94G   1% /tmp
> 
> Could you please kindly check the same for your test?
> It seems the testpmd close pcap device via pmd_pcap_remove(), but pcap_dump_close is not called. 
> Could you please kindly help to check how testpmd in your case close the pcap file (if possible with function call stack?)?
> 
> Thank you.
> 
> 

I can reproduce the issue with v20.11. The problems seems fixed by commit [1] in
the latest DPDK, the commit title gives the hint ;)

Backporting that commit can solve the issue.

Regards,
ferruh


[1]
Fixes: e412a138db91 ("net/pcap: fix file descriptor leak on close")