DPDK patches and discussions
 help / color / mirror / Atom feed
From: Stephen Hemminger <stephen@networkplumber.org>
To: anatoly.burakov@intel.com
Cc: dev@dpdk.org, stable@dpdk.org,
	Stephen Hemminger <stephen@networkplumber.org>
Subject: [PATCH] eal: fix data race in multi-process support
Date: Fri, 17 Dec 2021 10:29:22 -0800	[thread overview]
Message-ID: <20211217182922.159503-1-stephen@networkplumber.org> (raw)
In-Reply-To: <20211217181649.154972-1-stephen@networkplumber.org>

If DPDK is built with thread sanitizer it reports a race
in setting of multiprocess file descriptor. The fix is to
use atomic operations when updating mp_fd.

Simple example:
$ dpdk-testpmd -l 1-3 --no-huge
...
EAL: Error - exiting with code: 1
  Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory
==================
WARNING: ThreadSanitizer: data race (pid=83054)
  Write of size 4 at 0x55e3b7fce450 by main thread:
    #0 rte_mp_channel_cleanup <null> (dpdk-testpmd+0x160d79c)
    #1 rte_eal_cleanup <null> (dpdk-testpmd+0x1614fb5)
    #2 rte_exit <null> (dpdk-testpmd+0x15ec97a)
    #3 mbuf_pool_create.cold <null> (dpdk-testpmd+0x242e1a)
    #4 main <null> (dpdk-testpmd+0x5ab05d)

  Previous read of size 4 at 0x55e3b7fce450 by thread T2:
    #0 mp_handle <null> (dpdk-testpmd+0x160c979)
    #1 ctrl_thread_init <null> (dpdk-testpmd+0x15ff76e)

  As if synchronized via sleep:
    #0 nanosleep ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:362 (libtsan.so.0+0x5cd8e)
    #1 get_tsc_freq <null> (dpdk-testpmd+0x1622889)
    #2 set_tsc_freq <null> (dpdk-testpmd+0x15ffb9c)
    #3 rte_eal_timer_init <null> (dpdk-testpmd+0x1622a34)
    #4 rte_eal_init.cold <null> (dpdk-testpmd+0x26b314)
    #5 main <null> (dpdk-testpmd+0x5aab45)

  Location is global 'mp_fd' of size 4 at 0x55e3b7fce450 (dpdk-testpmd+0x0000027c7450)

  Thread T2 'rte_mp_handle' (tid=83057, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x58ba2)
    #1 rte_ctrl_thread_create <null> (dpdk-testpmd+0x15ff870)
    #2 rte_mp_channel_init.cold <null> (dpdk-testpmd+0x269986)
    #3 rte_eal_init <null> (dpdk-testpmd+0x1615b28)
    #4 main <null> (dpdk-testpmd+0x5aab45)

SUMMARY: ThreadSanitizer: data race (/home/shemminger/DPDK/main/build/app/dpdk-testpmd+0x160d79c) in rte_mp_channel_cleanup
==================
ThreadSanitizer: reported 1 warnings

Fixes: bacaa2754017 ("eal: add channel for multi-process communication")
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

---
v2 - fix the mp socket bind

 lib/eal/common/eal_common_proc.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/lib/eal/common/eal_common_proc.c b/lib/eal/common/eal_common_proc.c
index ebd0f6673b8b..72c7e8f536af 100644
--- a/lib/eal/common/eal_common_proc.c
+++ b/lib/eal/common/eal_common_proc.c
@@ -262,7 +262,7 @@ rte_mp_action_unregister(const char *name)
 }
 
 static int
-read_msg(struct mp_msg_internal *m, struct sockaddr_un *s)
+read_msg(int fd, struct mp_msg_internal *m, struct sockaddr_un *s)
 {
 	int msglen;
 	struct iovec iov;
@@ -282,7 +282,7 @@ read_msg(struct mp_msg_internal *m, struct sockaddr_un *s)
 	msgh.msg_control = control;
 	msgh.msg_controllen = sizeof(control);
 
-	msglen = recvmsg(mp_fd, &msgh, 0);
+	msglen = recvmsg(fd, &msgh, 0);
 	if (msglen < 0) {
 		RTE_LOG(ERR, EAL, "recvmsg failed, %s\n", strerror(errno));
 		return -1;
@@ -383,9 +383,10 @@ mp_handle(void *arg __rte_unused)
 {
 	struct mp_msg_internal msg;
 	struct sockaddr_un sa;
+	int fd;
 
-	while (mp_fd >= 0) {
-		if (read_msg(&msg, &sa) == 0)
+	while ((fd = __atomic_load_n(&mp_fd, __ATOMIC_RELAXED)) >= 0) {
+		if (read_msg(fd, &msg, &sa) == 0)
 			process_msg(&msg, &sa);
 	}
 
@@ -626,9 +627,8 @@ rte_mp_channel_init(void)
 			NULL, mp_handle, NULL) < 0) {
 		RTE_LOG(ERR, EAL, "failed to create mp thread: %s\n",
 			strerror(errno));
-		close(mp_fd);
 		close(dir_fd);
-		mp_fd = -1;
+		close(__atomic_exchange_n(&mp_fd, -1, __ATOMIC_RELAXED));
 		return -1;
 	}
 
@@ -644,11 +644,10 @@ rte_mp_channel_cleanup(void)
 {
 	int fd;
 
-	if (mp_fd < 0)
+	fd = __atomic_exchange_n(&mp_fd, -1, __ATOMIC_RELAXED);
+	if (fd < 0)
 		return;
 
-	fd = mp_fd;
-	mp_fd = -1;
 	pthread_cancel(mp_handle_tid);
 	pthread_join(mp_handle_tid, NULL);
 	close_socket_fd(fd);
-- 
2.30.2


  reply	other threads:[~2021-12-17 18:29 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-12-17 18:16 Stephen Hemminger
2021-12-17 18:29 ` Stephen Hemminger [this message]
2022-02-13 11:39   ` Thomas Monjalon
2022-04-14 20:28     ` Stephen Hemminger
2022-04-20 15:13   ` Burakov, Anatoly
2022-09-06 16:45   ` [PATCH v2] " Stephen Hemminger
2022-09-07  0:31     ` fengchengwen
2022-10-09 23:53     ` Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20211217182922.159503-1-stephen@networkplumber.org \
    --to=stephen@networkplumber.org \
    --cc=anatoly.burakov@intel.com \
    --cc=dev@dpdk.org \
    --cc=stable@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).