[RFC] eal_debug: do not use malloc in rte_dump_stack

[RFC] eal_debug: do not use malloc in rte_dump_stack

[Stephen Hemminger]

The glibc backtrace_symbols() calls malloc which makes it
dangerous to use rte_dump_stack() in a signal handler that
is handling errors that maybe due to memory corruption.

Instead, use dladdr() to lookup up symbols incrementally.

The format of the messages is based on what X org server
has been doing for many years. It changes from bottom up
to top down order.

Bugzilla ID: 929
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 lib/eal/linux/eal_debug.c | 45 ++++++++++++++++++++++++++++-----------
 1 file changed, 32 insertions(+), 13 deletions(-)

diff --git a/lib/eal/linux/eal_debug.c b/lib/eal/linux/eal_debug.c
index 64dab4e0da24..bf232f72f402 100644
--- a/lib/eal/linux/eal_debug.c
+++ b/lib/eal/linux/eal_debug.c
@@ -4,6 +4,7 @@
 
 #ifdef RTE_BACKTRACE
 #include <execinfo.h>
+#include <dlfcn.h>
 #endif
 #include <stdarg.h>
 #include <signal.h>
@@ -18,26 +19,44 @@
 
 #define BACKTRACE_SIZE 256
 
-/* dump the stack of the calling core */
+/* Dump the stack of the calling core
+ *
+ * Note: this requires some careful usage in order to
+ * stay safe in case where called from a signal
+ * handler and the malloc pool may be corrupted.
+ */
 void rte_dump_stack(void)
 {
 #ifdef RTE_BACKTRACE
 	void *func[BACKTRACE_SIZE];
-	char **symb = NULL;
-	int size;
+	int i, size;
 
 	size = backtrace(func, BACKTRACE_SIZE);
-	symb = backtrace_symbols(func, size);
-
-	if (symb == NULL)
-		return;
 
-	while (size > 0) {
-		rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
-			"%d: [%s]\n", size, symb[size - 1]);
-		size --;
+	for (i = 0; i < size; i++) {
+		void *pc = func[i];
+		const char *fname;
+		Dl_info info;
+
+		if (dladdr(pc, &info) == 0) {
+			rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
+				"%d: ?? [%p]\n", i, pc);
+			continue;
+		}
+
+		fname = (info.dli_fname && *info.dli_fname) ? info.dli_fname : "(vdso)";
+		if (info.dli_saddr != NULL)
+			rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
+				"%d: %s (%s+%#tx) [%p]\n",
+				i, fname, info.dli_sname,
+				(ptrdiff_t)((uintptr_t)pc - (uintptr_t)info.dli_saddr),
+				pc);
+		else
+			rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
+				"%d: %s (%p+%#tx) [%p]\n",
+				i, fname, info.dli_fbase,
+				(ptrdiff_t)((uintptr_t)pc - (uintptr_t)info.dli_fbase),
+				pc);
 	}
-
	free(symb);
 #endif /* RTE_BACKTRACE */
 }
-- 
2.34.1

RE: [RFC] eal_debug: do not use malloc in rte_dump_stack

[Morten Brørup]

> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Saturday, 29 January 2022 02.11
> 
> The glibc backtrace_symbols() calls malloc which makes it
> dangerous to use rte_dump_stack() in a signal handler that
> is handling errors that maybe due to memory corruption.

Yes. We have experienced that problem with backtrace_symbols(); so as a workaround, our failure signal handler dumps all other information first, and calls backtrace_symbols() last, in case it crashes.

> 
> Instead, use dladdr() to lookup up symbols incrementally.

I took a brief look at the dladdr() source code, and it looks good to me.

> 
> The format of the messages is based on what X org server
> has been doing for many years. It changes from bottom up
> to top down order.

Good idea. Seems more logical.

> 
> Bugzilla ID: 929
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
>  lib/eal/linux/eal_debug.c | 45 ++++++++++++++++++++++++++++-----------
>  1 file changed, 32 insertions(+), 13 deletions(-)
> 
> diff --git a/lib/eal/linux/eal_debug.c b/lib/eal/linux/eal_debug.c
> index 64dab4e0da24..bf232f72f402 100644
> --- a/lib/eal/linux/eal_debug.c
> +++ b/lib/eal/linux/eal_debug.c
> @@ -4,6 +4,7 @@
> 
>  #ifdef RTE_BACKTRACE
>  #include <execinfo.h>
> +#include <dlfcn.h>
>  #endif
>  #include <stdarg.h>
>  #include <signal.h>
> @@ -18,26 +19,44 @@
> 
>  #define BACKTRACE_SIZE 256
> 
> -/* dump the stack of the calling core */
> +/* Dump the stack of the calling core
> + *
> + * Note: this requires some careful usage in order to
> + * stay safe in case where called from a signal
> + * handler and the malloc pool may be corrupted.
> + */
>  void rte_dump_stack(void)
>  {
>  #ifdef RTE_BACKTRACE
>  	void *func[BACKTRACE_SIZE];
> -	char **symb = NULL;
> -	int size;
> +	int i, size;
> 
>  	size = backtrace(func, BACKTRACE_SIZE);
> -	symb = backtrace_symbols(func, size);
> -
> -	if (symb == NULL)
> -		return;
> 
> -	while (size > 0) {
> -		rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
> -			"%d: [%s]\n", size, symb[size - 1]);
> -		size --;
> +	for (i = 0; i < size; i++) {
> +		void *pc = func[i];
> +		const char *fname;
> +		Dl_info info;
> +
> +		if (dladdr(pc, &info) == 0) {
> +			rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
> +				"%d: ?? [%p]\n", i, pc);
> +			continue;
> +		}
> +
> +		fname = (info.dli_fname && *info.dli_fname) ?
> info.dli_fname : "(vdso)";
> +		if (info.dli_saddr != NULL)
> +			rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
> +				"%d: %s (%s+%#tx) [%p]\n",
> +				i, fname, info.dli_sname,
> +				(ptrdiff_t)((uintptr_t)pc -
> (uintptr_t)info.dli_saddr),
> +				pc);
> +		else
> +			rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
> +				"%d: %s (%p+%#tx) [%p]\n",
> +				i, fname, info.dli_fbase,
> +				(ptrdiff_t)((uintptr_t)pc -
> (uintptr_t)info.dli_fbase),
> +				pc);
>  	}
> -
> 	free(symb);

Probably something is lost in formatting here, but free(symb) must also be removed.

>  #endif /* RTE_BACKTRACE */
>  }
> --
> 2.34.1
> 

Great improvement, Stephen!

Acked-by: Morten Brørup <mb@smartsharesystems.com>

[PATCH v2 0/2] rte_dump_stack: improvements

[Stephen Hemminger]

This is update to earlier RFC. Add some more comments and changes
to have common code for Linux and FreeBSD

Stephen Hemminger (2):
  eal_debug: do not use malloc in rte_dump_stack
  eal: common rte_dump_stack for both Linux and FreeBSD

 lib/eal/freebsd/eal_debug.c | 43 ------------------------
 lib/eal/freebsd/meson.build |  1 -
 lib/eal/linux/eal_debug.c   | 43 ------------------------
 lib/eal/linux/meson.build   |  1 -
 lib/eal/unix/eal_debug.c    | 65 +++++++++++++++++++++++++++++++++++++
 lib/eal/unix/meson.build    |  5 +--
 6 files changed, 68 insertions(+), 90 deletions(-)
 delete mode 100644 lib/eal/freebsd/eal_debug.c
 delete mode 100644 lib/eal/linux/eal_debug.c
 create mode 100644 lib/eal/unix/eal_debug.c

-- 
2.34.1

[PATCH v2 1/2] eal_debug: do not use malloc in rte_dump_stack

[Stephen Hemminger]

The glibc backtrace_symbols() calls malloc which makes it
dangerous to use rte_dump_stack() in a signal handler that
is handling errors that maybe due to memory corruption.

Instead, use dladdr() to lookup up symbols incrementally.

The format of the messages is based on what X org server
has been doing for many years. It changes from bottom up
to top down order.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 lib/eal/linux/eal_debug.c | 46 +++++++++++++++++++++++++++++----------
 1 file changed, 34 insertions(+), 12 deletions(-)

diff --git a/lib/eal/linux/eal_debug.c b/lib/eal/linux/eal_debug.c
index 64dab4e0da24..51f24e3cccdf 100644
--- a/lib/eal/linux/eal_debug.c
+++ b/lib/eal/linux/eal_debug.c
@@ -4,6 +4,7 @@
 
 #ifdef RTE_BACKTRACE
 #include <execinfo.h>
+#include <dlfcn.h>
 #endif
 #include <stdarg.h>
 #include <signal.h>
@@ -18,26 +19,47 @@
 
 #define BACKTRACE_SIZE 256
 
-/* dump the stack of the calling core */
+/* Dump the stack of the calling core
+ *
+ * Note: this requires limiting what functions are used in this code to stay safe
+ * when called from a signal handler and malloc pool maybe corrupt.
+ */
 void rte_dump_stack(void)
 {
 #ifdef RTE_BACKTRACE
 	void *func[BACKTRACE_SIZE];
-	char **symb = NULL;
-	int size;
+	int i, size;
 
 	size = backtrace(func, BACKTRACE_SIZE);
-	symb = backtrace_symbols(func, size);
 
-	if (symb == NULL)
-		return;
+	for (i = 0; i < size; i++) {
+		void *pc = func[i];
+		const char *fname;
+		Dl_info info;
 
-	while (size > 0) {
-		rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
-			"%d: [%s]\n", size, symb[size - 1]);
-		size --;
-	}
+		if (dladdr(pc, &info) == 0) {
+			/* If symbol information not found print in hex */
+			rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
+				"%d: ?? [%p]\n", i, pc);
+			continue;
+		}
 
-	free(symb);
+		/* Is file name known? */
+		fname = (info.dli_fname && *info.dli_fname) ? info.dli_fname : "(vdso)";
+
+		/* Is symbol name known? */
+		if (info.dli_sname != NULL)
+			rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
+				"%d: %s (%s+%#tx) [%p]\n",
+				i, fname, info.dli_sname,
+				(ptrdiff_t)((uintptr_t)pc - (uintptr_t)info.dli_saddr),
+				pc);
+		else
+			rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
+				"%d: %s (%p+%#tx) [%p]\n",
+				i, fname, info.dli_fbase,
+				(ptrdiff_t)((uintptr_t)pc - (uintptr_t)info.dli_fbase),
+				pc);
+	}
 #endif /* RTE_BACKTRACE */
 }
-- 
2.34.1

Re: [PATCH v2 1/2] eal_debug: do not use malloc in rte_dump_stack

[Thomas Monjalon]

12/02/2022 19:44, Stephen Hemminger:
> The glibc backtrace_symbols() calls malloc which makes it
> dangerous to use rte_dump_stack() in a signal handler that
> is handling errors that maybe due to memory corruption.
> 
> Instead, use dladdr() to lookup up symbols incrementally.
> 
> The format of the messages is based on what X org server
> has been doing for many years. It changes from bottom up
> to top down order.
> 
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

Morten acked the RFC.
Is there any significant change?

Re: [PATCH v2 1/2] eal_debug: do not use malloc in rte_dump_stack

[Stephen Hemminger]

On Sun, 13 Feb 2022 12:41:59 +0100
Thomas Monjalon <thomas@monjalon.net> wrote:

> 12/02/2022 19:44, Stephen Hemminger:
> > The glibc backtrace_symbols() calls malloc which makes it
> > dangerous to use rte_dump_stack() in a signal handler that
> > is handling errors that maybe due to memory corruption.
> > 
> > Instead, use dladdr() to lookup up symbols incrementally.
> > 
> > The format of the messages is based on what X org server
> > has been doing for many years. It changes from bottom up
> > to top down order.
> > 
> > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>  
> 
> Morten acked the RFC.
> Is there any significant change?
> 
> 
> 

The change was making it not RFC.
Lets make a common code for this FreeBSD should have same code.

[PATCH v2 2/2] eal: common rte_dump_stack for both Linux and FreeBSD

[Stephen Hemminger]

The core functions used by rte_dump_stack() are backtrace_symbols
and dladdr. Both of those functions are the same in Linux and
FreeBSD so the code for decoding stack should be common.

Also, the filenames unix/meson.build are now resorted.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 lib/eal/freebsd/eal_debug.c         | 43 -----------------------------
 lib/eal/freebsd/meson.build         |  1 -
 lib/eal/linux/meson.build           |  1 -
 lib/eal/{linux => unix}/eal_debug.c |  0
 lib/eal/unix/meson.build            |  5 ++--
 5 files changed, 3 insertions(+), 47 deletions(-)
 delete mode 100644 lib/eal/freebsd/eal_debug.c
 rename lib/eal/{linux => unix}/eal_debug.c (100%)

diff --git a/lib/eal/freebsd/eal_debug.c b/lib/eal/freebsd/eal_debug.c
deleted file mode 100644
index 64dab4e0da24..000000000000
--- a/lib/eal/freebsd/eal_debug.c
+++ /dev/null
@@ -1,43 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifdef RTE_BACKTRACE
-#include <execinfo.h>
-#endif
-#include <stdarg.h>
-#include <signal.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include <stdint.h>
-
-#include <rte_log.h>
-#include <rte_debug.h>
-#include <rte_common.h>
-#include <rte_eal.h>
-
-#define BACKTRACE_SIZE 256
-
-/* dump the stack of the calling core */
-void rte_dump_stack(void)
-{
-#ifdef RTE_BACKTRACE
-	void *func[BACKTRACE_SIZE];
-	char **symb = NULL;
-	int size;
-
-	size = backtrace(func, BACKTRACE_SIZE);
-	symb = backtrace_symbols(func, size);
-
-	if (symb == NULL)
-		return;
-
-	while (size > 0) {
-		rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
-			"%d: [%s]\n", size, symb[size - 1]);
-		size --;
-	}
-
-	free(symb);
-#endif /* RTE_BACKTRACE */
-}
diff --git a/lib/eal/freebsd/meson.build b/lib/eal/freebsd/meson.build
index 398ceab71d03..85cca5a096ca 100644
--- a/lib/eal/freebsd/meson.build
+++ b/lib/eal/freebsd/meson.build
@@ -7,7 +7,6 @@ sources += files(
         'eal.c',
         'eal_alarm.c',
         'eal_cpuflags.c',
-        'eal_debug.c',
         'eal_dev.c',
         'eal_hugepage_info.c',
         'eal_interrupts.c',
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 65f2ac6b4798..3cccfa36c0a4 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -7,7 +7,6 @@ sources += files(
         'eal.c',
         'eal_alarm.c',
         'eal_cpuflags.c',
-        'eal_debug.c',
         'eal_dev.c',
         'eal_hugepage_info.c',
         'eal_interrupts.c',
diff --git a/lib/eal/linux/eal_debug.c b/lib/eal/unix/eal_debug.c
similarity index 100%
rename from lib/eal/linux/eal_debug.c
rename to lib/eal/unix/eal_debug.c
diff --git a/lib/eal/unix/meson.build b/lib/eal/unix/meson.build
index a22ea7cabc46..9f724bfd5d7a 100644
--- a/lib/eal/unix/meson.build
+++ b/lib/eal/unix/meson.build
@@ -2,10 +2,11 @@
 # Copyright(c) 2020 Dmitry Kozlyuk
 
 sources += files(
+        'eal_debug.c',
         'eal_file.c',
+        'eal_filesystem.c',
+        'eal_firmware.c',
         'eal_unix_memory.c',
         'eal_unix_timer.c',
-        'eal_firmware.c',
-        'eal_filesystem.c',
         'rte_thread.c',
 )
-- 
2.34.1

RE: [PATCH v2 0/2] rte_dump_stack: improvements

[Morten Brørup]

> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Saturday, 12 February 2022 19.45
> 
> This is update to earlier RFC. Add some more comments and changes
> to have common code for Linux and FreeBSD
> 
> Stephen Hemminger (2):
>   eal_debug: do not use malloc in rte_dump_stack
>   eal: common rte_dump_stack for both Linux and FreeBSD
> 
>  lib/eal/freebsd/eal_debug.c | 43 ------------------------
>  lib/eal/freebsd/meson.build |  1 -
>  lib/eal/linux/eal_debug.c   | 43 ------------------------
>  lib/eal/linux/meson.build   |  1 -
>  lib/eal/unix/eal_debug.c    | 65 +++++++++++++++++++++++++++++++++++++
>  lib/eal/unix/meson.build    |  5 +--
>  6 files changed, 68 insertions(+), 90 deletions(-)
>  delete mode 100644 lib/eal/freebsd/eal_debug.c
>  delete mode 100644 lib/eal/linux/eal_debug.c
>  create mode 100644 lib/eal/unix/eal_debug.c
> 
> --
> 2.34.1
> 

The dladdr() man page mentions that linking with -ldl is required; I assume this is already part of the DPDK EAL build system?

For the series,
Acked-by: Morten Brørup <mb@smartsharesystems.com>

Re: [PATCH v2 0/2] rte_dump_stack: improvements

[Bruce Richardson]

On Mon, Feb 14, 2022 at 12:10:30PM +0100, Morten Brørup wrote:
> > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > Sent: Saturday, 12 February 2022 19.45
> > 
> > This is update to earlier RFC. Add some more comments and changes
> > to have common code for Linux and FreeBSD
> > 
> > Stephen Hemminger (2):
> >   eal_debug: do not use malloc in rte_dump_stack
> >   eal: common rte_dump_stack for both Linux and FreeBSD
> > 
> >  lib/eal/freebsd/eal_debug.c | 43 ------------------------
> >  lib/eal/freebsd/meson.build |  1 -
> >  lib/eal/linux/eal_debug.c   | 43 ------------------------
> >  lib/eal/linux/meson.build   |  1 -
> >  lib/eal/unix/eal_debug.c    | 65 +++++++++++++++++++++++++++++++++++++
> >  lib/eal/unix/meson.build    |  5 +--
> >  6 files changed, 68 insertions(+), 90 deletions(-)
> >  delete mode 100644 lib/eal/freebsd/eal_debug.c
> >  delete mode 100644 lib/eal/linux/eal_debug.c
> >  create mode 100644 lib/eal/unix/eal_debug.c
> > 
> > --
> > 2.34.1
> > 
> 
> The dladdr() man page mentions that linking with -ldl is required; I assume this is already part of the DPDK EAL build system?
>
Yes, we should be already linking against dl because of the use of dlopen
for loading drivers dynamically. 

Re: [PATCH v2 0/2] rte_dump_stack: improvements

[David Marchand]

On Sat, Feb 12, 2022 at 7:44 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> This is update to earlier RFC. Add some more comments and changes
> to have common code for Linux and FreeBSD
>
> Stephen Hemminger (2):
>   eal_debug: do not use malloc in rte_dump_stack
>   eal: common rte_dump_stack for both Linux and FreeBSD
>
>  lib/eal/freebsd/eal_debug.c | 43 ------------------------
>  lib/eal/freebsd/meson.build |  1 -
>  lib/eal/linux/eal_debug.c   | 43 ------------------------
>  lib/eal/linux/meson.build   |  1 -
>  lib/eal/unix/eal_debug.c    | 65 +++++++++++++++++++++++++++++++++++++
>  lib/eal/unix/meson.build    |  5 +--
>  6 files changed, 68 insertions(+), 90 deletions(-)
>  delete mode 100644 lib/eal/freebsd/eal_debug.c
>  delete mode 100644 lib/eal/linux/eal_debug.c
>  create mode 100644 lib/eal/unix/eal_debug.c

Strange to change only the Linux implementation as a first patch, then
merge implementations in a second time effectively changing FreeBSD
implementation in what is presented in commitlog as a factorisation
cleanup.
Please invert the patches.

Besides, the series does not compile on current main.
It's probably a result of the header inclusion cleanup we had in
v22.03, but I prefer you check.


Thanks.

-- 
David Marchand

Re: [PATCH v2 0/2] rte_dump_stack: improvements

[Stephen Hemminger]

On Thu, 7 Apr 2022 14:45:07 +0200
David Marchand <david.marchand@redhat.com> wrote:

> On Sat, Feb 12, 2022 at 7:44 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > This is update to earlier RFC. Add some more comments and changes
> > to have common code for Linux and FreeBSD
> >
> > Stephen Hemminger (2):
> >   eal_debug: do not use malloc in rte_dump_stack
> >   eal: common rte_dump_stack for both Linux and FreeBSD
> >
> >  lib/eal/freebsd/eal_debug.c | 43 ------------------------
> >  lib/eal/freebsd/meson.build |  1 -
> >  lib/eal/linux/eal_debug.c   | 43 ------------------------
> >  lib/eal/linux/meson.build   |  1 -
> >  lib/eal/unix/eal_debug.c    | 65 +++++++++++++++++++++++++++++++++++++
> >  lib/eal/unix/meson.build    |  5 +--
> >  6 files changed, 68 insertions(+), 90 deletions(-)
> >  delete mode 100644 lib/eal/freebsd/eal_debug.c
> >  delete mode 100644 lib/eal/linux/eal_debug.c
> >  create mode 100644 lib/eal/unix/eal_debug.c  
> 
> Strange to change only the Linux implementation as a first patch, then
> merge implementations in a second time effectively changing FreeBSD
> implementation in what is presented in commitlog as a factorisation
> cleanup.
> Please invert the patches.
> 
> Besides, the series does not compile on current main.
> It's probably a result of the header inclusion cleanup we had in
> v22.03, but I prefer you check.
> 
> 
> Thanks.


As I looked at it more, there was more there.
Turns out that printf and therefore rte_log() is not signal safe.
There is a version of backtrace_symbols_fd that just uses writev() on  glibc for Linux
so that is a better alternative, but format changes.

But the BSD version of backtrace_symbols_fd uses printf and is therefore not signal safe.
Not sure if that matters.

[PATCH v3] rte_dump_stack: make in async signal safe

[Stephen Hemminger]

rte_dump_stack needs to be usable in situations when a bug is
encountered and from signal handlers (such as SEGV).

Glibc backtrace_symbols() calls malloc which makes it
dangerous in a signal handler that is handling errors that maybe
due to memory corruption. Additionally, rte_log() is unsafe because
syslog() is not signal safe; printf() is also documented as
not being safe.

This version formats message and uses writev for each line in a manner
similar to what glibc version of backtrace_symbols_fd() does. The
FreeBSD version of backtrace_symbols_fd() is not signal safe.

Sample output:

0: ./build/app/dpdk-testpmd (rte_dump_stack+0x2b) [560a6e9c002b]
1: ./build/app/dpdk-testpmd (main+0xad) [560a6decd5ad]
2: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xcd) [7fd43d3e27fd]
3: ./build/app/dpdk-testpmd (_start+0x2a) [560a6e83628a]

Bugzilla ID: 929
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
v3
  - merge previous two patches into one
    common Linux/FreeBSD code.
  - rewrite the code to not use functions which are not
    documented to be signal safe.

 lib/eal/freebsd/eal_debug.c |  43 -------------
 lib/eal/freebsd/meson.build |   1 -
 lib/eal/include/rte_debug.h |   2 +-
 lib/eal/linux/eal_debug.c   |  38 -----------
 lib/eal/linux/meson.build   |   1 -
 lib/eal/unix/eal_debug.c    | 123 ++++++++++++++++++++++++++++++++++++
 lib/eal/unix/meson.build    |   1 +
 7 files changed, 125 insertions(+), 84 deletions(-)
 delete mode 100644 lib/eal/freebsd/eal_debug.c
 delete mode 100644 lib/eal/linux/eal_debug.c
 create mode 100644 lib/eal/unix/eal_debug.c

diff --git a/lib/eal/freebsd/eal_debug.c b/lib/eal/freebsd/eal_debug.c
deleted file mode 100644
index 64dab4e0da24..000000000000
--- a/lib/eal/freebsd/eal_debug.c
+++ /dev/null
@@ -1,43 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifdef RTE_BACKTRACE
-#include <execinfo.h>
-#endif
-#include <stdarg.h>
-#include <signal.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include <stdint.h>
-
-#include <rte_log.h>
-#include <rte_debug.h>
-#include <rte_common.h>
-#include <rte_eal.h>
-
-#define BACKTRACE_SIZE 256
-
-/* dump the stack of the calling core */
-void rte_dump_stack(void)
-{
-#ifdef RTE_BACKTRACE
-	void *func[BACKTRACE_SIZE];
-	char **symb = NULL;
-	int size;
-
-	size = backtrace(func, BACKTRACE_SIZE);
-	symb = backtrace_symbols(func, size);
-
-	if (symb == NULL)
-		return;
-
-	while (size > 0) {
-		rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
-			"%d: [%s]\n", size, symb[size - 1]);
-		size --;
-	}
-
-	free(symb);
-#endif /* RTE_BACKTRACE */
-}
diff --git a/lib/eal/freebsd/meson.build b/lib/eal/freebsd/meson.build
index 398ceab71d03..85cca5a096ca 100644
--- a/lib/eal/freebsd/meson.build
+++ b/lib/eal/freebsd/meson.build
@@ -7,7 +7,6 @@ sources += files(
         'eal.c',
         'eal_alarm.c',
         'eal_cpuflags.c',
-        'eal_debug.c',
         'eal_dev.c',
         'eal_hugepage_info.c',
         'eal_interrupts.c',
diff --git a/lib/eal/include/rte_debug.h b/lib/eal/include/rte_debug.h
index c4bc71ce28f5..2c4b94a7c9bf 100644
--- a/lib/eal/include/rte_debug.h
+++ b/lib/eal/include/rte_debug.h
@@ -22,7 +22,7 @@ extern "C" {
 #endif
 
 /**
- * Dump the stack of the calling core to the console.
+ * Dump the stack of the calling core to the standard error.
  */
 void rte_dump_stack(void);
 
diff --git a/lib/eal/linux/eal_debug.c b/lib/eal/linux/eal_debug.c
deleted file mode 100644
index b0ecf5a9dcde..000000000000
--- a/lib/eal/linux/eal_debug.c
+++ /dev/null
@@ -1,38 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifdef RTE_BACKTRACE
-#include <execinfo.h>
-#endif
-#include <stdlib.h>
-#include <stdio.h>
-
-#include <rte_log.h>
-#include <rte_debug.h>
-
-#define BACKTRACE_SIZE 256
-
-/* dump the stack of the calling core */
-void rte_dump_stack(void)
-{
-#ifdef RTE_BACKTRACE
-	void *func[BACKTRACE_SIZE];
-	char **symb = NULL;
-	int size;
-
-	size = backtrace(func, BACKTRACE_SIZE);
-	symb = backtrace_symbols(func, size);
-
-	if (symb == NULL)
-		return;
-
-	while (size > 0) {
-		rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
-			"%d: [%s]\n", size, symb[size - 1]);
-		size --;
-	}
-
-	free(symb);
-#endif /* RTE_BACKTRACE */
-}
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 65f2ac6b4798..3cccfa36c0a4 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -7,7 +7,6 @@ sources += files(
         'eal.c',
         'eal_alarm.c',
         'eal_cpuflags.c',
-        'eal_debug.c',
         'eal_dev.c',
         'eal_hugepage_info.c',
         'eal_interrupts.c',
diff --git a/lib/eal/unix/eal_debug.c b/lib/eal/unix/eal_debug.c
new file mode 100644
index 000000000000..79b6f2fea2b9
--- /dev/null
+++ b/lib/eal/unix/eal_debug.c
@@ -0,0 +1,123 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include <rte_debug.h>
+
+
+#ifdef RTE_BACKTRACE
+
+#include <dlfcn.h>
+#include <execinfo.h>
+#include <string.h>
+#include <sys/uio.h>
+#include <unistd.h>
+
+#define BACKTRACE_SIZE 256
+
+/*
+ * Convert number to string and return start of string.
+ * Note: string does not start at beginning of buffer.
+ */
+static char *safe_itoa(long val, char *buf, size_t len, unsigned int radix)
+{
+	char *bp = buf + len;
+	static const char hexdigit[] = "0123456789abcdef";
+
+	*--bp = '\0'; /* Null terminate the string */
+	do {
+		/* if buffer is not big enough, then truncate */
+		if (bp == buf)
+			return bp;
+
+		*--bp = hexdigit[val % radix];
+		val /= radix;
+	} while (val != 0);
+
+	return bp;
+}
+
+
+/* Dump the stack of the calling core
+ *
+ * To be safe in signal handler requires limiting what functions are
+ * used in this code since may be called from inside libc or
+ * when malloc poll is corrupt.
+ *
+ * Most of libc is therefore not safe, include RTE_LOG (calls syslog);
+ * backtrace_symbols (calls malloc), etc.
+ */
+void rte_dump_stack(void)
+{
+	void *func[BACKTRACE_SIZE];
+	Dl_info info;
+	char buf1[8], buf2[32], buf3[32], buf4[32];
+	struct iovec iov[10];
+	int i, size;
+
+	size = backtrace(func, BACKTRACE_SIZE);
+
+	for (i = 0; i < size; i++) {
+		struct iovec *io = iov;
+		char *str;
+		uintptr_t base;
+		long offset;
+		void *pc = func[i];
+
+/* Macro to put string onto set of iovecs
+ * cast is to suppress warnings about lose of const qualifier
+ */
+#define PUSH_IOV(io, str) {					\
+		(io)->iov_base = (char *)(uintptr_t)str;	\
+		(io)->iov_len = strlen(str);			\
+		++io; }
+
+		/* output stack frame number */
+		str = safe_itoa(i, buf1, sizeof(buf1), 10);
+		PUSH_IOV(io, str);	/* iov[0] */
+		PUSH_IOV(io, ": ");	/* iov[1] */
+
+		/* Lookup the symbol information */
+		if (dladdr (pc, &info) == 0) {
+			PUSH_IOV(io, "?? [");
+		} else {
+			const char *fname;
+
+			if (info.dli_fname && *info.dli_fname)
+				fname = info.dli_fname;
+			else
+				fname = "(vdso)";
+			PUSH_IOV(io, fname);	/* iov[2] */
+			PUSH_IOV(io, " (");	/* iov[3] */
+
+			if (info.dli_saddr != NULL) {
+				PUSH_IOV(io, info.dli_sname);	/* iov[4] */
+				base = (uintptr_t)info.dli_saddr;
+			} else {
+				str = safe_itoa((unsigned long)info.dli_fbase,
+						buf3, sizeof(buf3), 16);
+				PUSH_IOV(io, str);
+				base = (uintptr_t)info.dli_fbase;
+			}
+
+			PUSH_IOV(io, "+0x");	/* iov[5] */
+
+			offset = (uintptr_t)pc - base;
+			str = safe_itoa(offset, buf4, sizeof(buf4), 16);
+			PUSH_IOV(io, str);	/* iov[6] */
+
+			PUSH_IOV(io, ") [");	/* iov[7] */
+		}
+
+		str = safe_itoa((unsigned long)pc, buf2, sizeof(buf2), 16);
+		PUSH_IOV(io, str);	/* iov[8] */
+		PUSH_IOV(io, "]\n");	/* iov[9] */
+
+		if (writev(STDERR_FILENO, iov, io - iov) < 0)
+			break;
+	}
+}
+#else
+/* stub if not enabled */
+void rte_dump_stack(void) { }
+#endif /* RTE_BACKTRACE */
diff --git a/lib/eal/unix/meson.build b/lib/eal/unix/meson.build
index a22ea7cabc46..9c59f3d31693 100644
--- a/lib/eal/unix/meson.build
+++ b/lib/eal/unix/meson.build
@@ -2,6 +2,7 @@
 # Copyright(c) 2020 Dmitry Kozlyuk
 
 sources += files(
+        'eal_debug.c',
         'eal_file.c',
         'eal_unix_memory.c',
         'eal_unix_timer.c',
-- 
2.35.1

[PATCH v4] rte_dump_stack: make in async signal safe

[Stephen Hemminger]

rte_dump_stack() needs to be usable in situations when a bug is
encountered and from signal handlers (such as SEGV).

Glibc backtrace_symbols() calls malloc which makes it
dangerous in a signal handler that is handling errors that maybe
due to memory corruption. Additionally, rte_log() is unsafe because
syslog() is not signal safe; printf() is also documented as
not being safe.

This version formats message and uses writev for each line in a manner
similar to what glibc version of backtrace_symbols_fd() does. The
FreeBSD version of backtrace_symbols_fd() is not signal safe.

Sample output:

0: ./build/app/dpdk-testpmd (rte_dump_stack+0x2b) [560a6e9c002b]
1: ./build/app/dpdk-testpmd (main+0xad) [560a6decd5ad]
2: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xcd) [7fd43d3e27fd]
3: ./build/app/dpdk-testpmd (_start+0x2a) [560a6e83628a]

Bugzilla ID: 929
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
v4 
  - fix whitespace report from checkpatch
v3
  - merge previous two patches into one
    common Linux/FreeBSD code.
  - rewrite the code to not use functions which are not
    documented to be signal safe.
 lib/eal/freebsd/eal_debug.c |  43 -------------
 lib/eal/freebsd/meson.build |   1 -
 lib/eal/include/rte_debug.h |   2 +-
 lib/eal/linux/eal_debug.c   |  38 -----------
 lib/eal/linux/meson.build   |   1 -
 lib/eal/unix/eal_debug.c    | 123 ++++++++++++++++++++++++++++++++++++
 lib/eal/unix/meson.build    |   1 +
 7 files changed, 125 insertions(+), 84 deletions(-)
 delete mode 100644 lib/eal/freebsd/eal_debug.c
 delete mode 100644 lib/eal/linux/eal_debug.c
 create mode 100644 lib/eal/unix/eal_debug.c

diff --git a/lib/eal/freebsd/eal_debug.c b/lib/eal/freebsd/eal_debug.c
deleted file mode 100644
index 64dab4e0da24..000000000000
--- a/lib/eal/freebsd/eal_debug.c
+++ /dev/null
@@ -1,43 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifdef RTE_BACKTRACE
-#include <execinfo.h>
-#endif
-#include <stdarg.h>
-#include <signal.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include <stdint.h>
-
-#include <rte_log.h>
-#include <rte_debug.h>
-#include <rte_common.h>
-#include <rte_eal.h>
-
-#define BACKTRACE_SIZE 256
-
-/* dump the stack of the calling core */
-void rte_dump_stack(void)
-{
-#ifdef RTE_BACKTRACE
-	void *func[BACKTRACE_SIZE];
-	char **symb = NULL;
-	int size;
-
-	size = backtrace(func, BACKTRACE_SIZE);
-	symb = backtrace_symbols(func, size);
-
-	if (symb == NULL)
-		return;
-
-	while (size > 0) {
-		rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
-			"%d: [%s]\n", size, symb[size - 1]);
-		size --;
-	}
-
-	free(symb);
-#endif /* RTE_BACKTRACE */
-}
diff --git a/lib/eal/freebsd/meson.build b/lib/eal/freebsd/meson.build
index 398ceab71d03..85cca5a096ca 100644
--- a/lib/eal/freebsd/meson.build
+++ b/lib/eal/freebsd/meson.build
@@ -7,7 +7,6 @@ sources += files(
         'eal.c',
         'eal_alarm.c',
         'eal_cpuflags.c',
-        'eal_debug.c',
         'eal_dev.c',
         'eal_hugepage_info.c',
         'eal_interrupts.c',
diff --git a/lib/eal/include/rte_debug.h b/lib/eal/include/rte_debug.h
index c4bc71ce28f5..2c4b94a7c9bf 100644
--- a/lib/eal/include/rte_debug.h
+++ b/lib/eal/include/rte_debug.h
@@ -22,7 +22,7 @@ extern "C" {
 #endif
 
 /**
- * Dump the stack of the calling core to the console.
+ * Dump the stack of the calling core to the standard error.
  */
 void rte_dump_stack(void);
 
diff --git a/lib/eal/linux/eal_debug.c b/lib/eal/linux/eal_debug.c
deleted file mode 100644
index b0ecf5a9dcde..000000000000
--- a/lib/eal/linux/eal_debug.c
+++ /dev/null
@@ -1,38 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifdef RTE_BACKTRACE
-#include <execinfo.h>
-#endif
-#include <stdlib.h>
-#include <stdio.h>
-
-#include <rte_log.h>
-#include <rte_debug.h>
-
-#define BACKTRACE_SIZE 256
-
-/* dump the stack of the calling core */
-void rte_dump_stack(void)
-{
-#ifdef RTE_BACKTRACE
-	void *func[BACKTRACE_SIZE];
-	char **symb = NULL;
-	int size;
-
-	size = backtrace(func, BACKTRACE_SIZE);
-	symb = backtrace_symbols(func, size);
-
-	if (symb == NULL)
-		return;
-
-	while (size > 0) {
-		rte_log(RTE_LOG_ERR, RTE_LOGTYPE_EAL,
-			"%d: [%s]\n", size, symb[size - 1]);
-		size --;
-	}
-
-	free(symb);
-#endif /* RTE_BACKTRACE */
-}
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 65f2ac6b4798..3cccfa36c0a4 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -7,7 +7,6 @@ sources += files(
         'eal.c',
         'eal_alarm.c',
         'eal_cpuflags.c',
-        'eal_debug.c',
         'eal_dev.c',
         'eal_hugepage_info.c',
         'eal_interrupts.c',
diff --git a/lib/eal/unix/eal_debug.c b/lib/eal/unix/eal_debug.c
new file mode 100644
index 000000000000..dea7372af2f8
--- /dev/null
+++ b/lib/eal/unix/eal_debug.c
@@ -0,0 +1,123 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#include <rte_debug.h>
+
+
+#ifdef RTE_BACKTRACE
+
+#include <dlfcn.h>
+#include <execinfo.h>
+#include <string.h>
+#include <sys/uio.h>
+#include <unistd.h>
+
+#define BACKTRACE_SIZE 256
+
+/*
+ * Convert number to string and return start of string.
+ * Note: string does not start at beginning of buffer.
+ */
+static char *safe_itoa(long val, char *buf, size_t len, unsigned int radix)
+{
+	char *bp = buf + len;
+	static const char hexdigit[] = "0123456789abcdef";
+
+	*--bp = '\0'; /* Null terminate the string */
+	do {
+		/* if buffer is not big enough, then truncate */
+		if (bp == buf)
+			return bp;
+
+		*--bp = hexdigit[val % radix];
+		val /= radix;
+	} while (val != 0);
+
+	return bp;
+}
+
+
+/* Dump the stack of the calling core
+ *
+ * To be safe in signal handler requires limiting what functions are
+ * used in this code since may be called from inside libc or
+ * when malloc poll is corrupt.
+ *
+ * Most of libc is therefore not safe, include RTE_LOG (calls syslog);
+ * backtrace_symbols (calls malloc), etc.
+ */
+void rte_dump_stack(void)
+{
+	void *func[BACKTRACE_SIZE];
+	Dl_info info;
+	char buf1[8], buf2[32], buf3[32], buf4[32];
+	struct iovec iov[10];
+	int i, size;
+
+	size = backtrace(func, BACKTRACE_SIZE);
+
+	for (i = 0; i < size; i++) {
+		struct iovec *io = iov;
+		char *str;
+		uintptr_t base;
+		long offset;
+		void *pc = func[i];
+
+/* Macro to put string onto set of iovecs
+ * cast is to suppress warnings about lose of const qualifier
+ */
+#define PUSH_IOV(io, str) {					\
+		(io)->iov_base = (char *)(uintptr_t)str;	\
+		(io)->iov_len = strlen(str);			\
+		++io; }
+
+		/* output stack frame number */
+		str = safe_itoa(i, buf1, sizeof(buf1), 10);
+		PUSH_IOV(io, str);	/* iov[0] */
+		PUSH_IOV(io, ": ");	/* iov[1] */
+
+		/* Lookup the symbol information */
+		if (dladdr(pc, &info) == 0) {
+			PUSH_IOV(io, "?? [");
+		} else {
+			const char *fname;
+
+			if (info.dli_fname && *info.dli_fname)
+				fname = info.dli_fname;
+			else
+				fname = "(vdso)";
+			PUSH_IOV(io, fname);	/* iov[2] */
+			PUSH_IOV(io, " (");	/* iov[3] */
+
+			if (info.dli_saddr != NULL) {
+				PUSH_IOV(io, info.dli_sname);	/* iov[4] */
+				base = (uintptr_t)info.dli_saddr;
+			} else {
+				str = safe_itoa((unsigned long)info.dli_fbase,
+						buf3, sizeof(buf3), 16);
+				PUSH_IOV(io, str);
+				base = (uintptr_t)info.dli_fbase;
+			}
+
+			PUSH_IOV(io, "+0x");	/* iov[5] */
+
+			offset = (uintptr_t)pc - base;
+			str = safe_itoa(offset, buf4, sizeof(buf4), 16);
+			PUSH_IOV(io, str);	/* iov[6] */
+
+			PUSH_IOV(io, ") [");	/* iov[7] */
+		}
+
+		str = safe_itoa((unsigned long)pc, buf2, sizeof(buf2), 16);
+		PUSH_IOV(io, str);	/* iov[8] */
+		PUSH_IOV(io, "]\n");	/* iov[9] */
+
+		if (writev(STDERR_FILENO, iov, io - iov) < 0)
+			break;
+	}
+}
+#else
+/* stub if not enabled */
+void rte_dump_stack(void) { }
+#endif /* RTE_BACKTRACE */
diff --git a/lib/eal/unix/meson.build b/lib/eal/unix/meson.build
index 781505ca9061..cc7d67dd321d 100644
--- a/lib/eal/unix/meson.build
+++ b/lib/eal/unix/meson.build
@@ -2,6 +2,7 @@
 # Copyright(c) 2020 Dmitry Kozlyuk
 
 sources += files(
+        'eal_debug.c',
         'eal_file.c',
         'eal_filesystem.c',
         'eal_firmware.c',
-- 
2.35.1

Re: [PATCH v4] rte_dump_stack: make in async signal safe

[David Marchand]

On Thu, Apr 14, 2022 at 10:19 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> rte_dump_stack() needs to be usable in situations when a bug is
> encountered and from signal handlers (such as SEGV).
>
> Glibc backtrace_symbols() calls malloc which makes it
> dangerous in a signal handler that is handling errors that maybe
> due to memory corruption. Additionally, rte_log() is unsafe because
> syslog() is not signal safe; printf() is also documented as
> not being safe.
>
> This version formats message and uses writev for each line in a manner
> similar to what glibc version of backtrace_symbols_fd() does. The
> FreeBSD version of backtrace_symbols_fd() is not signal safe.
>
> Sample output:
>
> 0: ./build/app/dpdk-testpmd (rte_dump_stack+0x2b) [560a6e9c002b]
> 1: ./build/app/dpdk-testpmd (main+0xad) [560a6decd5ad]
> 2: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xcd) [7fd43d3e27fd]
> 3: ./build/app/dpdk-testpmd (_start+0x2a) [560a6e83628a]
>
> Bugzilla ID: 929
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: David Marchand <david.marchand@redhat.com>

Applied, thanks.


-- 
David Marchand