* [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed
@ 2021-08-23 4:14 Yan, Xiaoping (NSB - CN/Hangzhou)
2021-08-23 10:04 ` Ferruh Yigit
0 siblings, 1 reply; 4+ messages in thread
From: Yan, Xiaoping (NSB - CN/Hangzhou) @ 2021-08-23 4:14 UTC (permalink / raw)
To: dev
Hi,
Before I run pdump, 21MB is used in /tmp
tmpfs 63G 21M 63G 1% /tmp
Then I run pdump with such command:
dpdk-pdump -c 0x4001000400100 -a 0000:03:00.6 -a 0000:03:06.1 --legacy-mem --base-virtaddr 0x2aaa2aa000 --file-prefix l2rt -- --pdump port=1,queue=*,rx-dev=/tmp/l2biprt.pcap,tx-dev=/tmp/l2biprt.pcap,mbuf-size=10240,total-num-mbufs=10000
After stop, it generates a 14MB file
-rw-rw-r--. 1 9999 9999 14M Aug 23 03:58 l2biprt.pcap
And 35M is used in /tmp
tmpfs 63G 35M 63G 1% /tmp
And fuser show there are still several users of this file. (all these pids are of running dpdk primary and secondary processes )
# fuser /tmp/l2biprt.pcap
/tmp/l2biprt.pcap: 139 342 347 434
Then after I rm the file (l2biprt.pcap), disk space in tmp is not freed (still 35MB used)
rm l2biprt.pcap
df -h
tmpfs 63G 35M 63G 1% /tmp
It seems it goes like this:
pdump start->pdump send vdev hotplug add request->primary and secondary process calls pmd_pcap_probe() which opens the pcap file->
pdump stop-> pdump send vdev hotplug remove request-> primary and secondary process does not close the pcap file
To properly close the pcap file and release disk space, it seems we should:
* somehow close the pcap file in "vdev hotplug remove", or
* don't open the pcap file in "vdev hotplug add"( pmd_pcap_probe()), it seems pcap eth_dev_start will open the pcap file in case it was not opened.
Any comment is appreciated, thank you.
Best regards
Yan Xiaoping
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed 2021-08-23 4:14 [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed Yan, Xiaoping (NSB - CN/Hangzhou) @ 2021-08-23 10:04 ` Ferruh Yigit 2021-08-24 7:36 ` Yan, Xiaoping (NSB - CN/Hangzhou) 0 siblings, 1 reply; 4+ messages in thread From: Ferruh Yigit @ 2021-08-23 10:04 UTC (permalink / raw) To: Yan, Xiaoping (NSB - CN/Hangzhou), dev; +Cc: Reshma Pattan On 8/23/2021 5:14 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote: > Hi, > > Before I run pdump, 21MB is used in /tmp > tmpfs 63G 21M 63G 1% /tmp > Then I run pdump with such command: > dpdk-pdump -c 0x4001000400100 -a 0000:03:00.6 -a 0000:03:06.1 --legacy-mem --base-virtaddr 0x2aaa2aa000 --file-prefix l2rt -- --pdump port=1,queue=*,rx-dev=/tmp/l2biprt.pcap,tx-dev=/tmp/l2biprt.pcap,mbuf-size=10240,total-num-mbufs=10000 > After stop, it generates a 14MB file > -rw-rw-r--. 1 9999 9999 14M Aug 23 03:58 l2biprt.pcap > And 35M is used in /tmp > tmpfs 63G 35M 63G 1% /tmp > And fuser show there are still several users of this file. (all these pids are of running dpdk primary and secondary processes ) > # fuser /tmp/l2biprt.pcap > /tmp/l2biprt.pcap: 139 342 347 434 > Then after I rm the file (l2biprt.pcap), disk space in tmp is not freed (still 35MB used) > rm l2biprt.pcap > df -h > tmpfs 63G 35M 63G 1% /tmp > > > It seems it goes like this: > pdump start->pdump send vdev hotplug add request->primary and secondary process calls pmd_pcap_probe() which opens the pcap file-> > pdump stop-> pdump send vdev hotplug remove request-> primary and secondary process does not close the pcap file > > To properly close the pcap file and release disk space, it seems we should: > > * somehow close the pcap file in "vdev hotplug remove", or > * don't open the pcap file in "vdev hotplug add"( pmd_pcap_probe()), it seems pcap eth_dev_start will open the pcap file in case it was not opened. > > Any comment is appreciated, thank you. Hi Yan, I can't reproduce the behavior you mentioned, for my case the .pcap file is closed when pdump is terminated and there is no active user of the file after that point. Removing the pcap device cause closing the .pcap files. When you terminate pdump application it will remove the pcap devices, and since it is a secondary process a message sent to primary to remove it too, making both process closing the pcap files. When I use testpmd as primary application, I am getting following log from testpmd when pdump terminated: "Port 2 is closed" Can you please test with testpmd? You can use '--log-level=*:debug' eal parameter to get more log for multi process communication. Btw, what is your DPDK version? I tested with latest DPDK (21.11.0-rc0). ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed 2021-08-23 10:04 ` Ferruh Yigit @ 2021-08-24 7:36 ` Yan, Xiaoping (NSB - CN/Hangzhou) 2021-08-24 8:59 ` Ferruh Yigit 0 siblings, 1 reply; 4+ messages in thread From: Yan, Xiaoping (NSB - CN/Hangzhou) @ 2021-08-24 7:36 UTC (permalink / raw) To: Ferruh Yigit, dev; +Cc: Reshma Pattan Hi, I'm using dpdk 20.11. (it's a bit difficult for me to upgrade my applications to latest dpdk) I tested dpdk-pdump with dpdk-tespmd, result is same (disk space not freed.) [root@up-0:/] # ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 78 99.1 0.0 67407568 102268 pts/0 SLl+ 07:15 2:32 dpdk-testpmd --base-virtaddr 0 root 144 0.3 0.0 67332948 7708 pts/1 Sl+ 07:17 0:00 dpdk-pdump -l 14 -a 0000:b3:02 # fuser /tmp/l2biprt.pcap /tmp/l2biprt.pcap: 78 144 When stop pdump, indeed testpmd prints: EAL: msg: eal_dev_mp_request eth_dev_close(): Closing pcap ethdev on NUMA socket 0 Port 1 is closed EAL: reply: eal_dev_mp_request But testpmd does not close the file: [root@up-0:/] # fuser /tmp/l2biprt.pcap /tmp/l2biprt.pcap: 78 And remove the pcap file, disk space is not freed: # df -h /tmp/ Filesystem Size Used Avail Use% Mounted on tmpfs 94G 900K 94G 1% /tmp [root@up-0:/] # ls -lh /tmp/ total 900K -rw-r--r--. 1 root root 897K Aug 24 07:18 l2biprt.pcap drwxrwxrwx. 2 root root 18 Aug 24 07:14 nes [root@up-0:/] # rm /tmp/l2biprt.pcap [root@up-0:/] # ls -lh /tmp/ total 0 drwxrwxrwx. 2 root root 18 Aug 24 07:14 nes [root@up-0:/] # df -h /tmp/ Filesystem Size Used Avail Use% Mounted on tmpfs 94G 900K 94G 1% /tmp Could you please kindly check the same for your test? It seems the testpmd close pcap device via pmd_pcap_remove(), but pcap_dump_close is not called. Could you please kindly help to check how testpmd in your case close the pcap file (if possible with function call stack?)? Thank you. Best regards Yan Xiaoping -----Original Message----- From: Ferruh Yigit <ferruh.yigit@intel.com> Sent: 2021年8月23日 18:04 To: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com>; dev@dpdk.org Cc: Reshma Pattan <reshma.pattan@intel.com> Subject: Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed On 8/23/2021 5:14 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote: > Hi, > > Before I run pdump, 21MB is used in /tmp > tmpfs 63G 21M 63G 1% /tmp > Then I run pdump with such command: > dpdk-pdump -c 0x4001000400100 -a 0000:03:00.6 -a 0000:03:06.1 > --legacy-mem --base-virtaddr 0x2aaa2aa000 --file-prefix l2rt -- > --pdump > port=1,queue=*,rx-dev=/tmp/l2biprt.pcap,tx-dev=/tmp/l2biprt.pcap,mbuf- > size=10240,total-num-mbufs=10000 > After stop, it generates a 14MB file > -rw-rw-r--. 1 9999 9999 14M Aug 23 03:58 l2biprt.pcap > And 35M is used in /tmp > tmpfs 63G 35M 63G 1% /tmp > And fuser show there are still several users of this file. (all these pids are of running dpdk primary and secondary processes ) > # fuser /tmp/l2biprt.pcap > /tmp/l2biprt.pcap: 139 342 347 434 > Then after I rm the file (l2biprt.pcap), disk space in tmp is not freed (still 35MB used) > rm l2biprt.pcap > df -h > tmpfs 63G 35M 63G 1% /tmp > > > It seems it goes like this: > pdump start->pdump send vdev hotplug add request->primary and secondary process calls pmd_pcap_probe() which opens the pcap file-> > pdump stop-> pdump send vdev hotplug remove request-> > primary and secondary process does not close the pcap file > > To properly close the pcap file and release disk space, it seems we should: > > * somehow close the pcap file in "vdev hotplug remove", or > * don't open the pcap file in "vdev hotplug add"( pmd_pcap_probe()), it seems pcap eth_dev_start will open the pcap file in case it was not opened. > > Any comment is appreciated, thank you. Hi Yan, I can't reproduce the behavior you mentioned, for my case the .pcap file is closed when pdump is terminated and there is no active user of the file after that point. Removing the pcap device cause closing the .pcap files. When you terminate pdump application it will remove the pcap devices, and since it is a secondary process a message sent to primary to remove it too, making both process closing the pcap files. When I use testpmd as primary application, I am getting following log from testpmd when pdump terminated: "Port 2 is closed" Can you please test with testpmd? You can use '--log-level=*:debug' eal parameter to get more log for multi process communication. Btw, what is your DPDK version? I tested with latest DPDK (21.11.0-rc0). ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed 2021-08-24 7:36 ` Yan, Xiaoping (NSB - CN/Hangzhou) @ 2021-08-24 8:59 ` Ferruh Yigit 0 siblings, 0 replies; 4+ messages in thread From: Ferruh Yigit @ 2021-08-24 8:59 UTC (permalink / raw) To: Yan, Xiaoping (NSB - CN/Hangzhou), dev; +Cc: Reshma Pattan (Please avoid top-post, I have updated the order of the messages. Please find the reply at the bottom of the email.) >> -----Original Message----- >> From: Ferruh Yigit <ferruh.yigit@intel.com> >> Sent: 2021年8月23日 18:04 >> To: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com>; dev@dpdk.org >> Cc: Reshma Pattan <reshma.pattan@intel.com> >> Subject: Re: [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed >> >> On 8/23/2021 5:14 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote: >>> Hi, >>> >>> Before I run pdump, 21MB is used in /tmp >>> tmpfs 63G 21M 63G 1% /tmp >>> Then I run pdump with such command: >>> dpdk-pdump -c 0x4001000400100 -a 0000:03:00.6 -a 0000:03:06.1 >>> --legacy-mem --base-virtaddr 0x2aaa2aa000 --file-prefix l2rt -- >>> --pdump >>> port=1,queue=*,rx-dev=/tmp/l2biprt.pcap,tx-dev=/tmp/l2biprt.pcap,mbuf- >>> size=10240,total-num-mbufs=10000 >>> After stop, it generates a 14MB file >>> -rw-rw-r--. 1 9999 9999 14M Aug 23 03:58 l2biprt.pcap >>> And 35M is used in /tmp >>> tmpfs 63G 35M 63G 1% /tmp >>> And fuser show there are still several users of this file. (all these pids are of running dpdk primary and secondary processes ) >>> # fuser /tmp/l2biprt.pcap >>> /tmp/l2biprt.pcap: 139 342 347 434 >>> Then after I rm the file (l2biprt.pcap), disk space in tmp is not freed (still 35MB used) >>> rm l2biprt.pcap >>> df -h >>> tmpfs 63G 35M 63G 1% /tmp >>> >>> >>> It seems it goes like this: >>> pdump start->pdump send vdev hotplug add request->primary and secondary process calls pmd_pcap_probe() which opens the pcap file-> >>> pdump stop-> pdump send vdev hotplug remove request-> >>> primary and secondary process does not close the pcap file >>> >>> To properly close the pcap file and release disk space, it seems we should: >>> >>> * somehow close the pcap file in "vdev hotplug remove", or >>> * don't open the pcap file in "vdev hotplug add"( pmd_pcap_probe()), it seems pcap eth_dev_start will open the pcap file in case it was not opened. >>> >>> Any comment is appreciated, thank you. >> >> >> Hi Yan, >> >> I can't reproduce the behavior you mentioned, for my case the .pcap file is closed when pdump is terminated and there is no active user of the file after that point. >> >> Removing the pcap device cause closing the .pcap files. >> >> When you terminate pdump application it will remove the pcap devices, and since it is a secondary process a message sent to primary to remove it too, making both process closing the pcap files. >> >> When I use testpmd as primary application, I am getting following log from testpmd when pdump terminated: >> "Port 2 is closed" >> Can you please test with testpmd? You can use '--log-level=*:debug' eal parameter to get more log for multi process communication. >> >> Btw, what is your DPDK version? I tested with latest DPDK (21.11.0-rc0). >> >> > On 8/24/2021 8:36 AM, Yan, Xiaoping (NSB - CN/Hangzhou) wrote: > Hi, > > I'm using dpdk 20.11. (it's a bit difficult for me to upgrade my applications to latest dpdk) > > I tested dpdk-pdump with dpdk-tespmd, result is same (disk space not freed.) > [root@up-0:/] > # ps aux > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 78 99.1 0.0 67407568 102268 pts/0 SLl+ 07:15 2:32 dpdk-testpmd --base-virtaddr 0 > root 144 0.3 0.0 67332948 7708 pts/1 Sl+ 07:17 0:00 dpdk-pdump -l 14 -a 0000:b3:02 > > # fuser /tmp/l2biprt.pcap > /tmp/l2biprt.pcap: 78 144 > > When stop pdump, indeed testpmd prints: > EAL: msg: eal_dev_mp_request > eth_dev_close(): Closing pcap ethdev on NUMA socket 0 > Port 1 is closed > EAL: reply: eal_dev_mp_request > But testpmd does not close the file: > [root@up-0:/] > # fuser /tmp/l2biprt.pcap > /tmp/l2biprt.pcap: 78 > > And remove the pcap file, disk space is not freed: > # df -h /tmp/ > Filesystem Size Used Avail Use% Mounted on > tmpfs 94G 900K 94G 1% /tmp > [root@up-0:/] > # ls -lh /tmp/ > total 900K > -rw-r--r--. 1 root root 897K Aug 24 07:18 l2biprt.pcap > drwxrwxrwx. 2 root root 18 Aug 24 07:14 nes > [root@up-0:/] > # rm /tmp/l2biprt.pcap > [root@up-0:/] > # ls -lh /tmp/ > total 0 > drwxrwxrwx. 2 root root 18 Aug 24 07:14 nes > [root@up-0:/] > # df -h /tmp/ > Filesystem Size Used Avail Use% Mounted on > tmpfs 94G 900K 94G 1% /tmp > > Could you please kindly check the same for your test? > It seems the testpmd close pcap device via pmd_pcap_remove(), but pcap_dump_close is not called. > Could you please kindly help to check how testpmd in your case close the pcap file (if possible with function call stack?)? > > Thank you. > > I can reproduce the issue with v20.11. The problems seems fixed by commit [1] in the latest DPDK, the commit title gives the hint ;) Backporting that commit can solve the issue. Regards, ferruh [1] Fixes: e412a138db91 ("net/pcap: fix file descriptor leak on close") ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-08-24 9:00 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-08-23 4:14 [dpdk-dev] remove dpdk-pdump pcap file, disk space is not freed Yan, Xiaoping (NSB - CN/Hangzhou) 2021-08-23 10:04 ` Ferruh Yigit 2021-08-24 7:36 ` Yan, Xiaoping (NSB - CN/Hangzhou) 2021-08-24 8:59 ` Ferruh Yigit
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).