Re: [dpdk-dev] [PATCH v2] eal: clean up unused files on initialization

Re: [dpdk-dev] [PATCH v2] eal: clean up unused files on initialization

[Eads, Gage]

Worked in my testing.

Tested-by: Gage Eads <gage.eads@intel.com>

Thanks,
Gage

Re: [dpdk-dev] [PATCH v2] eal: clean up unused files on initialization

[Thomas Monjalon]

13/11/2018 16:54, Anatoly Burakov:
> When creating process data structures, EAL will create many files
> in EAL runtime directory. Because we allow multiple secondary
> processes to run, each secondary process gets their own unique
> file. With many secondary processes running and exiting on the
> system, runtime directory will, over time, create enormous amounts
> of sockets, fbarray files and other stuff that just sits there
> unused because the process that allocated it has died a long time
> ago. This may lead to exhaustion of disk (or RAM) space in the
> runtime directory.
> 
> Fix this by removing every unlocked file at initialization that
> matches either socket or fbarray naming convention. We cannot be
> sure of any other files, so we'll leave them alone. Also, remove
> similar code from mp socket code.
> 
> We do it at the end of init, rather than at the beginning, because
> secondary process will use primary process' data structures even
> if the primary itself has died, and we don't want to remove those
> before we lock them.
> 
> Bugzilla ID: 106
> 
> Cc: stable@dpdk.org
> 
> Reported-by: Vipin Varghese <vipin.varghese@intel.com>
> 
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>

Applied, thanks

Re: [dpdk-dev] [PATCH v2] eal: clean up unused files on initialization

[Burakov, Anatoly]

On 14-Nov-18 3:44 AM, Thomas Monjalon wrote:
> 14/11/2018 04:24, Varghese, Vipin:
>> Tested-by: Vipin Varghese <vipin.varghese@intel.com>
>>
>> <snipped>
>>   
>>>>> When creating process data structures, EAL will create many files in
>>>>> EAL runtime directory. Because we allow multiple secondary processes
>>>>> to run, each secondary process gets their own unique file. With many
>>>>> secondary processes running and exiting on the system, runtime
>>>>> directory will, over time, create enormous amounts of sockets,
>>>>> fbarray files and other stuff that just sits there unused because the
>>>>> process that allocated it has died a long time ago. This may lead to
>>>>> exhaustion of disk (or RAM) space in the runtime directory.
>>>>>
>>>>> Fix this by removing every unlocked file at initialization that
>>>>> matches either socket or fbarray naming convention. We cannot be sure
>>>>> of any other files, so we'll leave them alone. Also, remove similar
>>>>> code from mp socket code.
>>>>>
>>>>> We do it at the end of init, rather than at the beginning, because
>>>>> secondary process will use primary process' data structures even if
>>>>> the primary itself has died, and we don't want to remove those before
>>>>> we lock them.
>>>>>
>>>>> Bugzilla ID: 106
>>>>>
>>>>> Cc: stable@dpdk.org
>>>>>
>>>>> Reported-by: Vipin Varghese <vipin.varghese@intel.com>
>>>>>
>>>>> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>
>> Thanks Anatoly for the patch which clean-ups the tmpfs. This unblock the client from critical stopper too.
>>
>>>>
>>>> I feel it is too big and too late for 18.11.
>>>> Can we move it to 19.02?
>>>
>>>   From maintainer's point of view, i agree that it's too risky to merge into 18.11
>>> at this stage. My input should probably stop there, but Vipin (the original bug
>>> reporter) may have other thoughts on this matter.
>>
>> Hi Thomas, without the fix it affects both dpdk and non dpdk application use a host or VM. My suggestion to have the fix in and port to 18.11 LTS too.
> 
> It is changing a behaviour.
> I propose to test it on 19.02 and backport it in 18.11.1.
> 
> Any other opinion?
> 

That would probably be the best compromise IMO.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2] eal: clean up unused files on initialization

[Thomas Monjalon]

14/11/2018 04:24, Varghese, Vipin:
> Tested-by: Vipin Varghese <vipin.varghese@intel.com>
> 
> <snipped>
>  
> > >> When creating process data structures, EAL will create many files in
> > >> EAL runtime directory. Because we allow multiple secondary processes
> > >> to run, each secondary process gets their own unique file. With many
> > >> secondary processes running and exiting on the system, runtime
> > >> directory will, over time, create enormous amounts of sockets,
> > >> fbarray files and other stuff that just sits there unused because the
> > >> process that allocated it has died a long time ago. This may lead to
> > >> exhaustion of disk (or RAM) space in the runtime directory.
> > >>
> > >> Fix this by removing every unlocked file at initialization that
> > >> matches either socket or fbarray naming convention. We cannot be sure
> > >> of any other files, so we'll leave them alone. Also, remove similar
> > >> code from mp socket code.
> > >>
> > >> We do it at the end of init, rather than at the beginning, because
> > >> secondary process will use primary process' data structures even if
> > >> the primary itself has died, and we don't want to remove those before
> > >> we lock them.
> > >>
> > >> Bugzilla ID: 106
> > >>
> > >> Cc: stable@dpdk.org
> > >>
> > >> Reported-by: Vipin Varghese <vipin.varghese@intel.com>
> > >>
> > >> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> 
> Thanks Anatoly for the patch which clean-ups the tmpfs. This unblock the client from critical stopper too.
> 
> > >
> > > I feel it is too big and too late for 18.11.
> > > Can we move it to 19.02?
> > 
> >  From maintainer's point of view, i agree that it's too risky to merge into 18.11
> > at this stage. My input should probably stop there, but Vipin (the original bug
> > reporter) may have other thoughts on this matter.
> 
> Hi Thomas, without the fix it affects both dpdk and non dpdk application use a host or VM. My suggestion to have the fix in and port to 18.11 LTS too.

It is changing a behaviour.
I propose to test it on 19.02 and backport it in 18.11.1.

Any other opinion?

Re: [dpdk-dev] [PATCH v2] eal: clean up unused files on initialization

[Varghese, Vipin]

Tested-by: Vipin Varghese <vipin.varghese@intel.com>

<snipped>
 
> >> When creating process data structures, EAL will create many files in
> >> EAL runtime directory. Because we allow multiple secondary processes
> >> to run, each secondary process gets their own unique file. With many
> >> secondary processes running and exiting on the system, runtime
> >> directory will, over time, create enormous amounts of sockets,
> >> fbarray files and other stuff that just sits there unused because the
> >> process that allocated it has died a long time ago. This may lead to
> >> exhaustion of disk (or RAM) space in the runtime directory.
> >>
> >> Fix this by removing every unlocked file at initialization that
> >> matches either socket or fbarray naming convention. We cannot be sure
> >> of any other files, so we'll leave them alone. Also, remove similar
> >> code from mp socket code.
> >>
> >> We do it at the end of init, rather than at the beginning, because
> >> secondary process will use primary process' data structures even if
> >> the primary itself has died, and we don't want to remove those before
> >> we lock them.
> >>
> >> Bugzilla ID: 106
> >>
> >> Cc: stable@dpdk.org
> >>
> >> Reported-by: Vipin Varghese <vipin.varghese@intel.com>
> >>
> >> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>

Thanks Anatoly for the patch which clean-ups the tmpfs. This unblock the client from critical stopper too.

> >
> > I feel it is too big and too late for 18.11.
> > Can we move it to 19.02?
> >
> 
>  From maintainer's point of view, i agree that it's too risky to merge into 18.11
> at this stage. My input should probably stop there, but Vipin (the original bug
> reporter) may have other thoughts on this matter.
> 
> --
> Thanks,
> Anatoly

Hi Thomas, without the fix it affects both dpdk and non dpdk application use a host or VM. My suggestion to have the fix in and port to 18.11 LTS too.

Thanks
Vipin Varghese

Re: [dpdk-dev] [PATCH v2] eal: clean up unused files on initialization

[Burakov, Anatoly]

On 13-Nov-18 4:57 PM, Thomas Monjalon wrote:
> 13/11/2018 16:54, Anatoly Burakov:
>> When creating process data structures, EAL will create many files
>> in EAL runtime directory. Because we allow multiple secondary
>> processes to run, each secondary process gets their own unique
>> file. With many secondary processes running and exiting on the
>> system, runtime directory will, over time, create enormous amounts
>> of sockets, fbarray files and other stuff that just sits there
>> unused because the process that allocated it has died a long time
>> ago. This may lead to exhaustion of disk (or RAM) space in the
>> runtime directory.
>>
>> Fix this by removing every unlocked file at initialization that
>> matches either socket or fbarray naming convention. We cannot be
>> sure of any other files, so we'll leave them alone. Also, remove
>> similar code from mp socket code.
>>
>> We do it at the end of init, rather than at the beginning, because
>> secondary process will use primary process' data structures even
>> if the primary itself has died, and we don't want to remove those
>> before we lock them.
>>
>> Bugzilla ID: 106
>>
>> Cc: stable@dpdk.org
>>
>> Reported-by: Vipin Varghese <vipin.varghese@intel.com>
>>
>> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> 
> I feel it is too big and too late for 18.11.
> Can we move it to 19.02?
> 

 From maintainer's point of view, i agree that it's too risky to merge 
into 18.11 at this stage. My input should probably stop there, but Vipin 
(the original bug reporter) may have other thoughts on this matter.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2] eal: clean up unused files on initialization

[Thomas Monjalon]

13/11/2018 16:54, Anatoly Burakov:
> When creating process data structures, EAL will create many files
> in EAL runtime directory. Because we allow multiple secondary
> processes to run, each secondary process gets their own unique
> file. With many secondary processes running and exiting on the
> system, runtime directory will, over time, create enormous amounts
> of sockets, fbarray files and other stuff that just sits there
> unused because the process that allocated it has died a long time
> ago. This may lead to exhaustion of disk (or RAM) space in the
> runtime directory.
> 
> Fix this by removing every unlocked file at initialization that
> matches either socket or fbarray naming convention. We cannot be
> sure of any other files, so we'll leave them alone. Also, remove
> similar code from mp socket code.
> 
> We do it at the end of init, rather than at the beginning, because
> secondary process will use primary process' data structures even
> if the primary itself has died, and we don't want to remove those
> before we lock them.
> 
> Bugzilla ID: 106
> 
> Cc: stable@dpdk.org
> 
> Reported-by: Vipin Varghese <vipin.varghese@intel.com>
> 
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>

I feel it is too big and too late for 18.11.
Can we move it to 19.02?

[dpdk-dev] [PATCH v2] eal: clean up unused files on initialization

[Anatoly Burakov]

When creating process data structures, EAL will create many files
in EAL runtime directory. Because we allow multiple secondary
processes to run, each secondary process gets their own unique
file. With many secondary processes running and exiting on the
system, runtime directory will, over time, create enormous amounts
of sockets, fbarray files and other stuff that just sits there
unused because the process that allocated it has died a long time
ago. This may lead to exhaustion of disk (or RAM) space in the
runtime directory.

Fix this by removing every unlocked file at initialization that
matches either socket or fbarray naming convention. We cannot be
sure of any other files, so we'll leave them alone. Also, remove
similar code from mp socket code.

We do it at the end of init, rather than at the beginning, because
secondary process will use primary process' data structures even
if the primary itself has died, and we don't want to remove those
before we lock them.

Bugzilla ID: 106

Cc: stable@dpdk.org

Reported-by: Vipin Varghese <vipin.varghese@intel.com>

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---

Notes:
    v2: fix FreeBSD compile

 lib/librte_eal/bsdapp/eal/eal.c         | 100 ++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_proc.c |  30 -------
 lib/librte_eal/common/eal_filesystem.h  |   3 +
 lib/librte_eal/linuxapp/eal/eal.c       |  99 +++++++++++++++++++++++
 4 files changed, 202 insertions(+), 30 deletions(-)

diff --git a/lib/librte_eal/bsdapp/eal/eal.c b/lib/librte_eal/bsdapp/eal/eal.c
index 508cbc46f..99be6b554 100644
--- a/lib/librte_eal/bsdapp/eal/eal.c
+++ b/lib/librte_eal/bsdapp/eal/eal.c
@@ -3,6 +3,8 @@
  * Copyright(c) 2014 6WIND S.A.
  */
 
+#include <dirent.h>
+#include <fnmatch.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <stdint.h>
@@ -141,6 +143,92 @@ eal_create_runtime_dir(void)
 	return 0;
 }
 
+int
+eal_clean_runtime_dir(void)
+{
+	DIR *dir;
+	struct dirent *dirent;
+	int dir_fd, fd, lck_result;
+	static const char * const filters[] = {
+		"fbarray_*",
+		"mp_socket_*"
+	};
+
+	/* open directory */
+	dir = opendir(runtime_dir);
+	if (!dir) {
+		RTE_LOG(ERR, EAL, "Unable to open runtime directory %s\n",
+				runtime_dir);
+		goto error;
+	}
+	dir_fd = dirfd(dir);
+
+	/* lock the directory before doing anything, to avoid races */
+	if (flock(dir_fd, LOCK_EX) < 0) {
+		RTE_LOG(ERR, EAL, "Unable to lock runtime directory %s\n",
+			runtime_dir);
+		goto error;
+	}
+
+	dirent = readdir(dir);
+	if (!dirent) {
+		RTE_LOG(ERR, EAL, "Unable to read runtime directory %s\n",
+				runtime_dir);
+		goto error;
+	}
+
+	while (dirent != NULL) {
+		unsigned int f_idx;
+		bool skip = true;
+
+		/* skip files that don't match the patterns */
+		for (f_idx = 0; f_idx < RTE_DIM(filters); f_idx++) {
+			const char *filter = filters[f_idx];
+
+			if (fnmatch(filter, dirent->d_name, 0) == 0) {
+				skip = false;
+				break;
+			}
+		}
+		if (skip) {
+			dirent = readdir(dir);
+			continue;
+		}
+
+		/* try and lock the file */
+		fd = openat(dir_fd, dirent->d_name, O_RDONLY);
+
+		/* skip to next file */
+		if (fd == -1) {
+			dirent = readdir(dir);
+			continue;
+		}
+
+		/* non-blocking lock */
+		lck_result = flock(fd, LOCK_EX | LOCK_NB);
+
+		/* if lock succeeds, remove the file */
+		if (lck_result != -1)
+			unlinkat(dir_fd, dirent->d_name, 0);
+		close(fd);
+		dirent = readdir(dir);
+	}
+
+	/* closedir closes dir_fd and drops the lock */
+	closedir(dir);
+	return 0;
+
+error:
+	if (dir)
+		closedir(dir);
+
+	RTE_LOG(ERR, EAL, "Error while clearing runtime dir: %s\n",
+		strerror(errno));
+
+	return -1;
+}
+
+
 const char *
 rte_eal_get_runtime_dir(void)
 {
@@ -805,6 +893,18 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	/*
+	 * clean up unused files in runtime directory. we do this at the end of
+	 * init and not at the beginning because we want to clean stuff up
+	 * whether we are primary or secondary process, but we cannot remove
+	 * primary process' files because secondary should be able to run even
+	 * if primary process is dead.
+	 */
+	if (eal_clean_runtime_dir() < 0) {
+		rte_eal_init_alert("Cannot clear runtime directory\n");
+		return -1;
+	}
+
 	rte_eal_mcfg_complete();
 
 	/* Call each registered callback, if enabled */
diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
index 97663d3ba..e66d7fafb 100644
--- a/lib/librte_eal/common/eal_common_proc.c
+++ b/lib/librte_eal/common/eal_common_proc.c
@@ -542,29 +542,6 @@ open_socket_fd(void)
 	return mp_fd;
 }
 
-static int
-unlink_sockets(const char *filter)
-{
-	int dir_fd;
-	DIR *mp_dir;
-	struct dirent *ent;
-
-	mp_dir = opendir(mp_dir_path);
-	if (!mp_dir) {
-		RTE_LOG(ERR, EAL, "Unable to open directory %s\n", mp_dir_path);
-		return -1;
-	}
-	dir_fd = dirfd(mp_dir);
-
-	while ((ent = readdir(mp_dir))) {
-		if (fnmatch(filter, ent->d_name, 0) == 0)
-			unlinkat(dir_fd, ent->d_name, 0);
-	}
-
-	closedir(mp_dir);
-	return 0;
-}
-
 int
 rte_mp_channel_init(void)
 {
@@ -603,13 +580,6 @@ rte_mp_channel_init(void)
 		return -1;
 	}
 
-	if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
-			unlink_sockets(mp_filter)) {
-		RTE_LOG(ERR, EAL, "failed to unlink mp sockets\n");
-		close(dir_fd);
-		return -1;
-	}
-
 	if (open_socket_fd() < 0) {
 		close(dir_fd);
 		return -1;
diff --git a/lib/librte_eal/common/eal_filesystem.h b/lib/librte_eal/common/eal_filesystem.h
index b3e8ae5ea..1528282cf 100644
--- a/lib/librte_eal/common/eal_filesystem.h
+++ b/lib/librte_eal/common/eal_filesystem.h
@@ -25,6 +25,9 @@
 int
 eal_create_runtime_dir(void);
 
+int
+eal_clean_runtime_dir(void);
+
 /* returns runtime dir */
 const char *
 rte_eal_get_runtime_dir(void);
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 361744d40..a1bc02dfa 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -13,7 +13,9 @@
 #include <syslog.h>
 #include <getopt.h>
 #include <sys/file.h>
+#include <dirent.h>
 #include <fcntl.h>
+#include <fnmatch.h>
 #include <stddef.h>
 #include <errno.h>
 #include <limits.h>
@@ -149,6 +151,91 @@ eal_create_runtime_dir(void)
 	return 0;
 }
 
+int
+eal_clean_runtime_dir(void)
+{
+	DIR *dir;
+	struct dirent *dirent;
+	int dir_fd, fd, lck_result;
+	static const char * const filters[] = {
+		"fbarray_*",
+		"mp_socket_*"
+	};
+
+	/* open directory */
+	dir = opendir(runtime_dir);
+	if (!dir) {
+		RTE_LOG(ERR, EAL, "Unable to open runtime directory %s\n",
+				runtime_dir);
+		goto error;
+	}
+	dir_fd = dirfd(dir);
+
+	/* lock the directory before doing anything, to avoid races */
+	if (flock(dir_fd, LOCK_EX) < 0) {
+		RTE_LOG(ERR, EAL, "Unable to lock runtime directory %s\n",
+			runtime_dir);
+		goto error;
+	}
+
+	dirent = readdir(dir);
+	if (!dirent) {
+		RTE_LOG(ERR, EAL, "Unable to read runtime directory %s\n",
+				runtime_dir);
+		goto error;
+	}
+
+	while (dirent != NULL) {
+		unsigned int f_idx;
+		bool skip = true;
+
+		/* skip files that don't match the patterns */
+		for (f_idx = 0; f_idx < RTE_DIM(filters); f_idx++) {
+			const char *filter = filters[f_idx];
+
+			if (fnmatch(filter, dirent->d_name, 0) == 0) {
+				skip = false;
+				break;
+			}
+		}
+		if (skip) {
+			dirent = readdir(dir);
+			continue;
+		}
+
+		/* try and lock the file */
+		fd = openat(dir_fd, dirent->d_name, O_RDONLY);
+
+		/* skip to next file */
+		if (fd == -1) {
+			dirent = readdir(dir);
+			continue;
+		}
+
+		/* non-blocking lock */
+		lck_result = flock(fd, LOCK_EX | LOCK_NB);
+
+		/* if lock succeeds, remove the file */
+		if (lck_result != -1)
+			unlinkat(dir_fd, dirent->d_name, 0);
+		close(fd);
+		dirent = readdir(dir);
+	}
+
+	/* closedir closes dir_fd and drops the lock */
+	closedir(dir);
+	return 0;
+
+error:
+	if (dir)
+		closedir(dir);
+
+	RTE_LOG(ERR, EAL, "Error while clearing runtime dir: %s\n",
+		strerror(errno));
+
+	return -1;
+}
+
 const char *
 rte_eal_get_runtime_dir(void)
 {
@@ -1096,6 +1183,18 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	/*
+	 * clean up unused files in runtime directory. we do this at the end of
+	 * init and not at the beginning because we want to clean stuff up
+	 * whether we are primary or secondary process, but we cannot remove
+	 * primary process' files because secondary should be able to run even
+	 * if primary process is dead.
+	 */
+	if (eal_clean_runtime_dir() < 0) {
+		rte_eal_init_alert("Cannot clear runtime directory\n");
+		return -1;
+	}
+
 	rte_eal_mcfg_complete();
 
 	/* Call each registered callback, if enabled */
-- 
2.17.1