From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DC9C5A04A5; Fri, 17 Dec 2021 19:16:55 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 382104067C; Fri, 17 Dec 2021 19:16:55 +0100 (CET) Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by mails.dpdk.org (Postfix) with ESMTP id A44004013F for ; Fri, 17 Dec 2021 19:16:53 +0100 (CET) Received: by mail-pl1-f169.google.com with SMTP id v19so2537245plo.7 for ; Fri, 17 Dec 2021 10:16:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=mXoOeyN/QzBsynRhJVawGN1VupdmbOaH59OuNe84+2k=; b=fsTzIOGuulWFE7785KiEUibodr8IEfuo5kK9OjMWd5wUj99F322UobEXdJfCRrzmQN ZlKvUea7yFQqBymlySHWqfrICEPJh6HZmXYb5dBB8e3UyZvrdJYoQVOcunDa1tMnF89A NucN6Wl2A3BrBheZurrRgfNO1PTLoTcJjCePD+QgM6ZEtoN+jW9NPd9TipXRrv19s+lP 2q6DzMtxeQamATVg47OeVIu5zJT2yJnjvD5czeeUfwu7r8zpucIKIkc3qSKgZrXUXwOL M7xFSB/g/qPoRBktTEC+RNAzPe6+OEV9bhMP19WHkvrvuy9+DUULvARsOEEx+rleJzMz FDog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=mXoOeyN/QzBsynRhJVawGN1VupdmbOaH59OuNe84+2k=; b=iJdcyoEq5TkEVLoePrt0f92Qgo/YvzeBbpGUVVCF8pyowUX/QaeZli4JmH68S1OAy5 M17HynTv5k/zqx8byPBweBahwCtEcq+eCWD0bybFj+yNLg2aNg2E5V+mJyGC90xgjno7 MYuJI6hyzaan2bx0xdhILk/jj2mvK40Tl54XKzgnrZCElTm8nTAhMCAM6KggO5Ai9kEv sYNUH0n+8rUY0TprE6woV9p2NtXXiylZ7vtoOsCMGkmvL+qA1xQwIf61YUAVT97UZ3NZ j4l5z5VX85N58yC0j//xlj5H6Ttbpl9KH3hf/zGb/+M2fApqbbR71xshypPZCwjEPF3r sEVQ== X-Gm-Message-State: AOAM530hR6pqMoIc2VhQkkjswBDIzVOChTsj5iwoIlJTp/iGTNajAU3n fR+InrbI6pIL/TzJf4eOiTcv9w== X-Google-Smtp-Source: ABdhPJxD04AHeZIZr3P24CWawZtjJgAFuPd9S5Rb9T5vgqvo3x3h3ynJRuDcxxZYUJKtel+hpP81Fg== X-Received: by 2002:a17:902:f781:b0:148:eb1f:f775 with SMTP id q1-20020a170902f78100b00148eb1ff775mr1213125pln.164.1639765012673; Fri, 17 Dec 2021 10:16:52 -0800 (PST) Received: from hermes.local (204-195-112-199.wavecable.com. [204.195.112.199]) by smtp.gmail.com with ESMTPSA id w19sm9008881pga.80.2021.12.17.10.16.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Dec 2021 10:16:52 -0800 (PST) From: Stephen Hemminger To: anatoly.burakov@intel.com Cc: stable@dpdk.org, dev@dpdk.org, Stephen Hemminger Subject: [PATCH] eal: fix data race in multi-process support Date: Fri, 17 Dec 2021 10:16:49 -0800 Message-Id: <20211217181649.154972-1-stephen@networkplumber.org> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org If DPDK is built with thread sanitizer it reports a race in setting of multiprocess file descriptor. The fix is to use atomic operations when updating mp_fd. Simple example: $ dpdk-testpmd -l 1-3 --no-huge EAL: Detected CPU lcores: 16 EAL: Detected NUMA nodes: 1 EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem EAL: Detected static linkage of DPDK EAL: Multi-process socket /run/user/1000/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' testpmd: No probed ethernet devices testpmd: create a new mbuf pool : n=163456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc EAL: Error - exiting with code: 1 Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory ================== WARNING: ThreadSanitizer: data race (pid=83054) Write of size 4 at 0x55e3b7fce450 by main thread: #0 rte_mp_channel_cleanup (dpdk-testpmd+0x160d79c) #1 rte_eal_cleanup (dpdk-testpmd+0x1614fb5) #2 rte_exit (dpdk-testpmd+0x15ec97a) #3 mbuf_pool_create.cold (dpdk-testpmd+0x242e1a) #4 main (dpdk-testpmd+0x5ab05d) Previous read of size 4 at 0x55e3b7fce450 by thread T2: #0 mp_handle (dpdk-testpmd+0x160c979) #1 ctrl_thread_init (dpdk-testpmd+0x15ff76e) As if synchronized via sleep: #0 nanosleep ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:362 (libtsan.so.0+0x5cd8e) #1 get_tsc_freq (dpdk-testpmd+0x1622889) #2 set_tsc_freq (dpdk-testpmd+0x15ffb9c) #3 rte_eal_timer_init (dpdk-testpmd+0x1622a34) #4 rte_eal_init.cold (dpdk-testpmd+0x26b314) #5 main (dpdk-testpmd+0x5aab45) Location is global 'mp_fd' of size 4 at 0x55e3b7fce450 (dpdk-testpmd+0x0000027c7450) Thread T2 'rte_mp_handle' (tid=83057, running) created by main thread at: #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x58ba2) #1 rte_ctrl_thread_create (dpdk-testpmd+0x15ff870) #2 rte_mp_channel_init.cold (dpdk-testpmd+0x269986) #3 rte_eal_init (dpdk-testpmd+0x1615b28) #4 main (dpdk-testpmd+0x5aab45) SUMMARY: ThreadSanitizer: data race (/home/shemminger/DPDK/main/build/app/dpdk-testpmd+0x160d79c) in rte_mp_channel_cleanup ================== ThreadSanitizer: reported 1 warnings Signed-off-by: Stephen Hemminger --- lib/eal/common/eal_common_proc.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/lib/eal/common/eal_common_proc.c b/lib/eal/common/eal_common_proc.c index ebd0f6673b8b..4e6c1af59833 100644 --- a/lib/eal/common/eal_common_proc.c +++ b/lib/eal/common/eal_common_proc.c @@ -262,7 +262,7 @@ rte_mp_action_unregister(const char *name) } static int -read_msg(struct mp_msg_internal *m, struct sockaddr_un *s) +read_msg(int fd, struct mp_msg_internal *m, struct sockaddr_un *s) { int msglen; struct iovec iov; @@ -282,7 +282,7 @@ read_msg(struct mp_msg_internal *m, struct sockaddr_un *s) msgh.msg_control = control; msgh.msg_controllen = sizeof(control); - msglen = recvmsg(mp_fd, &msgh, 0); + msglen = recvmsg(fd, &msgh, 0); if (msglen < 0) { RTE_LOG(ERR, EAL, "recvmsg failed, %s\n", strerror(errno)); return -1; @@ -383,9 +383,10 @@ mp_handle(void *arg __rte_unused) { struct mp_msg_internal msg; struct sockaddr_un sa; + int fd; - while (mp_fd >= 0) { - if (read_msg(&msg, &sa) == 0) + while ((fd = __atomic_load_n(&mp_fd, __ATOMIC_RELAXED)) >= 0) { + if (read_msg(fd, &msg, &sa) == 0) process_msg(&msg, &sa); } @@ -537,14 +538,15 @@ static int open_socket_fd(void) { struct sockaddr_un un; + int fd; peer_name[0] = '\0'; if (rte_eal_process_type() == RTE_PROC_SECONDARY) snprintf(peer_name, sizeof(peer_name), "%d_%"PRIx64, getpid(), rte_rdtsc()); - mp_fd = socket(AF_UNIX, SOCK_DGRAM, 0); - if (mp_fd < 0) { + fd = socket(AF_UNIX, SOCK_DGRAM, 0); + if (fd < 0) { RTE_LOG(ERR, EAL, "failed to create unix socket\n"); return -1; } @@ -559,12 +561,13 @@ open_socket_fd(void) if (bind(mp_fd, (struct sockaddr *)&un, sizeof(un)) < 0) { RTE_LOG(ERR, EAL, "failed to bind %s: %s\n", un.sun_path, strerror(errno)); - close(mp_fd); + close(fd); return -1; } RTE_LOG(INFO, EAL, "Multi-process socket %s\n", un.sun_path); - return mp_fd; + __atomic_store_n(&mp_fd, fd, __ATOMIC_RELAXED); + return fd; } static void @@ -626,9 +629,8 @@ rte_mp_channel_init(void) NULL, mp_handle, NULL) < 0) { RTE_LOG(ERR, EAL, "failed to create mp thread: %s\n", strerror(errno)); - close(mp_fd); close(dir_fd); - mp_fd = -1; + close(__atomic_exchange_n(&mp_fd, -1, __ATOMIC_RELAXED)); return -1; } @@ -644,11 +646,10 @@ rte_mp_channel_cleanup(void) { int fd; - if (mp_fd < 0) + fd = __atomic_exchange_n(&mp_fd, -1, __ATOMIC_RELAXED); + if (fd < 0) return; - fd = mp_fd; - mp_fd = -1; pthread_cancel(mp_handle_tid); pthread_join(mp_handle_tid, NULL); close_socket_fd(fd); -- 2.30.2